2024-08-30T23:12:48.943 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-08-30T23:12:48.964 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-30T23:12:49.057 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173 branch: wip-bharath1-testing-2024-08-30-1604 description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} email: skanta@redhat.com first_in_suite: false job_id: '7882173' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath1-testing-2024-08-30-1604 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 24432b660539783e58f890fcff372342c034f6ec ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 24432b660539783e58f890fcff372342c034f6ec selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-bharath1-testing-2024-08-30-1604 sha1: 24432b660539783e58f890fcff372342c034f6ec owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 1274 sha1: 24432b660539783e58f890fcff372342c034f6ec sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath1-testing-2024-08-30-1604 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_24432b660539783e58f890fcff372342c034f6ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 24432b660539783e58f890fcff372342c034f6ec targets: smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4pbQN5rhPueYv4Yr3IuCsq07GCcRCavJZgKArOCXIcsKMskHdkepqgm4V24f2lhFn93vpnDfd1qeoS6UMpJoU= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIi6T1j4zXOXQvQybfJQhu6vyEKCNhyUfuE5zLwGt/Gr6C9SEBPdD5hQ16u/tcmgOkkBgRBbbxrGj2y2BOXlFBI= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMGk5yP/ILQUeiq1xbhYl9xUHE7J7MYg/FPsARUb4a93R/68jSCWLTXJva7APzZ6+utQbP0SJ7hcjgIlTohnSbc= tasks: - install: branch: squid exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-08-30_17:51:23 tube: smithi user: skanta verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-30T23:12:49.057 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_24432b660539783e58f890fcff372342c034f6ec/qa; will attempt to use it 2024-08-30T23:12:49.058 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_24432b660539783e58f890fcff372342c034f6ec/qa/tasks 2024-08-30T23:12:49.058 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-30T23:12:49.059 INFO:teuthology.task.internal:Saving configuration 2024-08-30T23:12:49.076 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-30T23:12:49.083 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-30T23:12:49.114 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-30 23:07:49.051250', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4pbQN5rhPueYv4Yr3IuCsq07GCcRCavJZgKArOCXIcsKMskHdkepqgm4V24f2lhFn93vpnDfd1qeoS6UMpJoU='} 2024-08-30T23:12:49.144 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-30 23:07:49.049511', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIi6T1j4zXOXQvQybfJQhu6vyEKCNhyUfuE5zLwGt/Gr6C9SEBPdD5hQ16u/tcmgOkkBgRBbbxrGj2y2BOXlFBI='} 2024-08-30T23:12:49.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-30 23:07:49.047543', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMGk5yP/ILQUeiq1xbhYl9xUHE7J7MYg/FPsARUb4a93R/68jSCWLTXJva7APzZ6+utQbP0SJ7hcjgIlTohnSbc='} 2024-08-30T23:12:49.174 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-30T23:12:49.181 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-08-30T23:12:49.181 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-08-30T23:12:49.181 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-08-30T23:12:49.181 INFO:teuthology.run_tasks:Running task console_log... 2024-08-30T23:12:49.296 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0772fbfbe0>, signals=[15]) 2024-08-30T23:12:49.296 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-30T23:12:49.301 INFO:teuthology.task.internal:Opening connections... 2024-08-30T23:12:49.301 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-08-30T23:12:49.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T23:12:49.387 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-08-30T23:12:49.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T23:12:49.487 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-08-30T23:12:49.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T23:12:49.572 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-30T23:12:49.579 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-08-30T23:12:49.597 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-08-30T23:12:49.597 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-08-30T23:12:49.655 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="9" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="9" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el9" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:LOGO="fedora-logo-icon" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-30T23:12:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-30T23:12:49.657 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-08-30T23:12:49.695 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-08-30T23:12:49.713 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-08-30T23:12:49.714 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-08-30T23:12:49.774 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-30T23:12:49.775 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-08-30T23:12:49.775 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-30T23:12:49.775 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-30T23:12:49.775 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-30T23:12:49.775 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-08-30T23:12:49.812 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-08-30T23:12:49.836 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-08-30T23:12:49.836 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-08-30T23:12:49.902 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-08-30T23:12:49.902 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-08-30T23:12:49.902 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-08-30T23:12:49.902 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-08-30T23:12:49.902 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-08-30T23:12:49.902 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-30T23:12:49.903 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-30T23:12:49.904 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-08-30T23:12:49.937 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-30T23:12:49.985 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-30T23:12:50.003 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-30T23:12:50.003 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-08-30T23:12:50.007 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-08-30T23:12:50.011 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-08-30T23:12:50.030 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-30T23:12:50.036 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-30T23:12:50.037 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-08-30T23:12:50.065 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-08-30T23:12:50.068 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-08-30T23:12:50.102 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-30T23:12:50.209 INFO:teuthology.run_tasks:Running task kernel... 2024-08-30T23:12:50.225 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-30T23:12:50.225 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-30T23:12:50.225 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-30T23:12:50.226 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-30T23:12:50.226 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-30T23:12:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:12:50.244 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-08-30T23:12:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:12:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:12:50.247 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-08-30T23:12:50.247 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-08-30T23:12:50.301 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-503.el9.x86_64 2024-08-30T23:12:50.302 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-503.el9.x86_64 2024-08-30T23:12:50.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-08-30T23:12:50.304 INFO:teuthology.orchestra.run.smithi106.stdout:5.14.0-503.el9.x86_64 2024-08-30T23:12:50.304 INFO:teuthology.task.kernel:Running kernel on smithi106: 5.14.0-503.el9.x86_64 2024-08-30T23:12:50.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-08-30T23:12:50.305 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-503.el9.x86_64 2024-08-30T23:12:50.306 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-503.el9.x86_64 2024-08-30T23:12:50.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-08-30T23:12:52.607 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-08-30T23:12:52.688 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-08-30T23:12:53.833 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-08-30T23:12:54.622 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-08-30T23:12:55.791 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-08-30T23:12:56.101 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-08-30T23:12:59.231 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.6 MB 00:00 2024-08-30T23:13:00.435 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.6 MB 00:00 2024-08-30T23:13:00.682 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 18 kB 00:00 2024-08-30T23:13:01.217 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.6 MB 00:01 2024-08-30T23:13:01.453 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 35 MB/s | 22 MB 00:00 2024-08-30T23:13:01.894 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 18 kB 00:00 2024-08-30T23:13:02.681 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 34 MB/s | 22 MB 00:00 2024-08-30T23:13:03.484 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 18 kB 00:01 2024-08-30T23:13:04.246 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 35 MB/s | 22 MB 00:00 2024-08-30T23:13:06.900 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-30T23:13:08.105 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-30T23:13:08.982 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-30T23:13:08.982 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-30T23:13:09.063 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:13:09.064 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:13:09.064 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:13:09.172 DEBUG:teuthology.orchestra.run.smithi106:> echo no | sudo yum reinstall kernel || true 2024-08-30T23:13:09.702 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-30T23:13:09.711 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:03 ago on Fri 30 Aug 2024 11:13:06 PM UTC. 2024-08-30T23:13:09.843 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout:Reinstalling: 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:13:09.852 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:13:09.853 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.9 M 2024-08-30T23:13:09.853 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 0 2024-08-30T23:13:09.853 INFO:teuthology.orchestra.run.smithi106.stderr:Operation aborted. 2024-08-30T23:13:09.906 INFO:teuthology.orchestra.run.smithi106.stdout:Is this ok [y/N]: 2024-08-30T23:13:09.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum reinstall -y kernel || true 2024-08-30T23:13:10.195 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-30T23:13:10.195 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-30T23:13:10.277 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:13:10.278 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:13:10.278 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:13:10.394 DEBUG:teuthology.orchestra.run.smithi136:> echo no | sudo yum reinstall kernel || true 2024-08-30T23:13:10.442 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:04 ago on Fri 30 Aug 2024 11:13:06 PM UTC. 2024-08-30T23:13:10.570 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:Reinstalling: 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.9 M 2024-08-30T23:13:10.571 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 0 2024-08-30T23:13:10.572 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:13:10.935 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:02 ago on Fri 30 Aug 2024 11:13:08 PM UTC. 2024-08-30T23:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-503.el9.x86_64.rpm 5.3 MB/s | 1.9 MB 00:00 2024-08-30T23:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:13:11.028 INFO:teuthology.orchestra.run.smithi106.stdout:Total 4.1 MB/s | 1.9 MB 00:00 2024-08-30T23:13:11.057 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:13:11.080 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:13:11.086 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:13:11.086 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-30T23:13:11.086 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:13:11.086 INFO:teuthology.orchestra.run.smithi136.stdout:Reinstalling: 2024-08-30T23:13:11.086 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 1.9 M 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 0 2024-08-30T23:13:11.087 INFO:teuthology.orchestra.run.smithi136.stderr:Operation aborted. 2024-08-30T23:13:11.144 INFO:teuthology.orchestra.run.smithi136.stdout:Is this ok [y/N]: 2024-08-30T23:13:11.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum reinstall -y kernel || true 2024-08-30T23:13:11.287 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:13:11.287 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:13:11.318 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:13:11.318 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:13:11.485 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:13:11.580 INFO:teuthology.orchestra.run.smithi106.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T23:13:11.580 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:11.685 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:03 ago on Fri 30 Aug 2024 11:13:08 PM UTC. 2024-08-30T23:13:11.800 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-30T23:13:11.800 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-08-30T23:13:11.812 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:13:11.813 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:13:11.813 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-30T23:13:11.813 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:13:11.813 INFO:teuthology.orchestra.run.smithi136.stdout:Reinstalling: 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 1.9 M 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 0 2024-08-30T23:13:11.814 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:13:11.883 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:13:11.884 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:13:11.884 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:13:12.005 DEBUG:teuthology.orchestra.run.smithi173:> echo no | sudo yum reinstall kernel || true 2024-08-30T23:13:12.265 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-503.el9.x86_64.rpm 5.4 MB/s | 1.9 MB 00:00 2024-08-30T23:13:12.266 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:13:12.266 INFO:teuthology.orchestra.run.smithi136.stdout:Total 4.1 MB/s | 1.9 MB 00:00 2024-08-30T23:13:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T23:13:12.294 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:13:12.519 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:13:12.520 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:13:12.550 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:13:12.550 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:13:12.587 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:03 ago on Fri 30 Aug 2024 11:13:09 PM UTC. 2024-08-30T23:13:12.715 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:13:12.742 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout:Reinstalling: 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:13:12.753 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 1.9 M 2024-08-30T23:13:12.754 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 0 2024-08-30T23:13:12.754 INFO:teuthology.orchestra.run.smithi173.stderr:Operation aborted. 2024-08-30T23:13:12.792 INFO:teuthology.orchestra.run.smithi136.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T23:13:12.793 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:12.809 INFO:teuthology.orchestra.run.smithi173.stdout:Is this ok [y/N]: 2024-08-30T23:13:12.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum reinstall -y kernel || true 2024-08-30T23:13:12.997 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:12.997 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:13:12.997 INFO:teuthology.orchestra.run.smithi106.stdout:Reinstalled: 2024-08-30T23:13:12.997 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-30T23:13:12.997 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:13:12.997 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:13:13.193 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-08-30T23:13:13.292 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-30T23:13:13.293 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-30T23:13:13.293 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-30T23:13:13.293 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-30T23:13:13.293 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-30T23:13:13.328 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-08-30T23:13:13.348 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:04 ago on Fri 30 Aug 2024 11:13:09 PM UTC. 2024-08-30T23:13:13.350 DEBUG:teuthology.parallel:result is None 2024-08-30T23:13:13.476 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout:Reinstalling: 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:13:13.477 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 1.9 M 2024-08-30T23:13:13.478 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 0 2024-08-30T23:13:13.478 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:13:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T23:13:14.070 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-503.el9.x86_64.rpm 5.6 MB/s | 1.9 MB 00:00 2024-08-30T23:13:14.070 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:13:14.070 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.2 MB/s | 1.9 MB 00:00 2024-08-30T23:13:14.099 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:13:14.224 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:14.224 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:13:14.224 INFO:teuthology.orchestra.run.smithi136.stdout:Reinstalled: 2024-08-30T23:13:14.224 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-30T23:13:14.224 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:13:14.224 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:13:14.319 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:13:14.319 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:13:14.350 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:13:14.350 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:13:14.406 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-08-30T23:13:14.465 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-30T23:13:14.466 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-30T23:13:14.466 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-30T23:13:14.466 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-30T23:13:14.466 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-30T23:13:14.499 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-08-30T23:13:14.518 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:13:14.519 DEBUG:teuthology.parallel:result is None 2024-08-30T23:13:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T23:13:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:15.299 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:15.299 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-08-30T23:13:16.033 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-08-30T23:13:16.034 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:13:16.034 INFO:teuthology.orchestra.run.smithi173.stdout:Reinstalled: 2024-08-30T23:13:16.034 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-503.el9.x86_64 2024-08-30T23:13:16.034 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:13:16.034 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:13:16.234 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-08-30T23:13:16.332 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-503.el9.x86_64 2024-08-30T23:13:16.333 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-08-30T23:13:16.333 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-08-30T23:13:16.333 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-08-30T23:13:16.333 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-30T23:13:16.368 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-08-30T23:13:16.388 DEBUG:teuthology.parallel:result is None 2024-08-30T23:13:16.389 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-30T23:13:16.397 INFO:teuthology.task.internal:Creating test directory... 2024-08-30T23:13:16.398 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-30T23:13:16.400 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-30T23:13:16.403 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-30T23:13:16.444 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-30T23:13:16.450 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-30T23:13:16.510 INFO:teuthology.task.internal:Creating archive directory... 2024-08-30T23:13:16.510 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-30T23:13:16.513 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-30T23:13:16.515 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-30T23:13:16.551 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-30T23:13:16.557 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-30T23:13:16.557 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-30T23:13:16.589 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-30T23:13:16.592 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-30T23:13:16.619 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T23:13:16.629 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T23:13:16.636 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T23:13:16.647 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T23:13:16.649 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T23:13:16.661 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-30T23:13:16.663 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-30T23:13:16.669 INFO:teuthology.task.internal:Configuring sudo... 2024-08-30T23:13:16.669 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-30T23:13:16.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-30T23:13:16.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-30T23:13:16.701 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-30T23:13:16.710 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-30T23:13:16.710 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-30T23:13:16.740 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-30T23:13:16.743 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-30T23:13:16.758 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T23:13:16.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T23:13:16.888 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T23:13:16.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T23:13:17.012 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:13:17.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-30T23:13:17.078 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T23:13:17.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T23:13:17.173 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T23:13:17.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T23:13:17.303 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:13:17.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-30T23:13:17.370 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T23:13:17.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-30T23:13:17.466 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T23:13:17.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-30T23:13:17.591 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:13:17.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-30T23:13:17.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-08-30T23:13:17.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-08-30T23:13:17.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-08-30T23:13:17.693 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T23:13:17.701 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T23:13:17.734 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-30T23:13:18.144 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-30T23:13:18.151 INFO:teuthology.task.internal:Starting timer... 2024-08-30T23:13:18.151 INFO:teuthology.run_tasks:Running task pcp... 2024-08-30T23:13:18.160 INFO:teuthology.run_tasks:Running task selinux... 2024-08-30T23:13:18.168 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-08-30T23:13:18.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-08-30T23:13:18.246 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-08-30T23:13:18.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-08-30T23:13:18.293 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-08-30T23:13:18.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-08-30T23:13:18.356 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-08-30T23:13:18.359 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-30T23:13:18.359 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-08-30T23:13:18.384 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-08-30T23:13:18.385 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-08-30T23:13:18.409 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-08-30T23:13:18.409 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-08-30T23:13:18.503 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-08-30T23:13:18.503 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-08-30T23:13:18.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-30T23:13:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:13:18.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-30T23:13:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:13:18.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-30T23:13:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:13:18.583 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-30T23:13:18.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-08-30T23:13:18.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-08-30T23:13:18.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-08-30T23:13:18.699 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-30T23:13:18.709 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-30T23:13:18.719 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-30T23:13:18.720 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi106.front.sepia.ceph.com,smithi136.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-30T23:15:03.935 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-08-30T23:15:03.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-08-30T23:15:03.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T23:15:04.030 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-08-30T23:15:04.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-08-30T23:15:04.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-30T23:15:04.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T23:15:04.214 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-08-30T23:15:04.296 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-08-30T23:15:04.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-08-30T23:15:04.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-30T23:15:04.386 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-08-30T23:15:04.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-08-30T23:15:04.474 INFO:teuthology.run_tasks:Running task clock... 2024-08-30T23:15:04.484 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-30T23:15:04.484 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-30T23:15:04.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T23:15:04.487 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-30T23:15:04.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T23:15:04.490 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-30T23:15:04.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-30T23:15:04.515 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-30T23:15:04.518 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-30T23:15:04.529 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-30T23:15:04.534 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-30T23:15:04.544 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-30T23:15:04.554 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-08-30T23:15:04.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-08-30T23:15:04.559 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-30T23:15:04.565 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-08-30T23:15:04.569 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-08-30T23:15:04.579 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-30T23:15:04.584 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-30T23:15:04.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-08-30T23:15:04.593 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-30T23:15:04.595 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-08-30T23:15:04.598 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-30T23:15:04.609 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-30T23:15:04.623 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-30T23:15:04.649 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-08-30T23:15:04.652 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T23:15:04.652 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-08-30T23:15:04.652 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.652 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.652 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.652 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.661 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-08-30T23:15:04.664 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T23:15:04.664 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-08-30T23:15:04.665 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.665 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.665 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.665 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.679 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-08-30T23:15:04.683 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-30T23:15:04.683 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-08-30T23:15:04.684 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.684 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.684 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.684 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-30T23:15:04.684 INFO:teuthology.run_tasks:Running task install... 2024-08-30T23:15:04.692 DEBUG:teuthology.task.install:project ceph 2024-08-30T23:15:04.692 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '24432b660539783e58f890fcff372342c034f6ec'}} 2024-08-30T23:15:04.692 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '24432b660539783e58f890fcff372342c034f6ec'} 2024-08-30T23:15:04.693 INFO:teuthology.task.install:Using flavor: default 2024-08-30T23:15:04.700 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-30T23:15:04.700 INFO:teuthology.task.install:extra packages: [] 2024-08-30T23:15:04.701 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '24432b660539783e58f890fcff372342c034f6ec', 'tag': None, 'wait_for_package': False} 2024-08-30T23:15:04.701 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:15:04.701 INFO:teuthology.packaging:ref: None 2024-08-30T23:15:04.701 INFO:teuthology.packaging:tag: None 2024-08-30T23:15:04.702 INFO:teuthology.packaging:branch: squid 2024-08-30T23:15:04.702 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:15:04.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-30T23:15:04.705 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '24432b660539783e58f890fcff372342c034f6ec', 'tag': None, 'wait_for_package': False} 2024-08-30T23:15:04.705 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:15:04.705 INFO:teuthology.packaging:ref: None 2024-08-30T23:15:04.705 INFO:teuthology.packaging:tag: None 2024-08-30T23:15:04.705 INFO:teuthology.packaging:branch: squid 2024-08-30T23:15:04.705 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:15:04.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-30T23:15:04.707 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '24432b660539783e58f890fcff372342c034f6ec', 'tag': None, 'wait_for_package': False} 2024-08-30T23:15:04.707 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:15:04.707 INFO:teuthology.packaging:ref: None 2024-08-30T23:15:04.707 INFO:teuthology.packaging:tag: None 2024-08-30T23:15:04.707 INFO:teuthology.packaging:branch: squid 2024-08-30T23:15:04.707 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:15:04.708 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-30T23:15:04.954 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/ 2024-08-30T23:15:04.954 INFO:teuthology.task.install.rpm:Package version is 19.1.1-242.g244a32d3 2024-08-30T23:15:04.987 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/ 2024-08-30T23:15:04.988 INFO:teuthology.task.install.rpm:Package version is 19.1.1-242.g244a32d3 2024-08-30T23:15:05.007 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/ 2024-08-30T23:15:05.007 INFO:teuthology.task.install.rpm:Package version is 19.1.1-242.g244a32d3 2024-08-30T23:15:05.087 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-30T23:15:05.088 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:15:05.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-30T23:15:05.104 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-30T23:15:05.105 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:15:05.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-30T23:15:05.114 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-30T23:15:05.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:15:05.114 INFO:teuthology.packaging:ref: None 2024-08-30T23:15:05.114 INFO:teuthology.packaging:tag: None 2024-08-30T23:15:05.115 INFO:teuthology.packaging:branch: squid 2024-08-30T23:15:05.115 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:15:05.115 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-30T23:15:05.128 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/244a32d3583bb0aa259286ff61157888a54a3ca5/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-30T23:15:05.129 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:15:05.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-30T23:15:05.133 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-30T23:15:05.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:15:05.133 INFO:teuthology.packaging:ref: None 2024-08-30T23:15:05.133 INFO:teuthology.packaging:tag: None 2024-08-30T23:15:05.133 INFO:teuthology.packaging:branch: squid 2024-08-30T23:15:05.133 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:15:05.133 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-30T23:15:05.155 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-30T23:15:05.156 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:15:05.156 INFO:teuthology.packaging:ref: None 2024-08-30T23:15:05.156 INFO:teuthology.packaging:tag: None 2024-08-30T23:15:05.156 INFO:teuthology.packaging:branch: squid 2024-08-30T23:15:05.156 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:15:05.156 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-30T23:15:05.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-30T23:15:05.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-30T23:15:05.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-30T23:15:05.256 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-30T23:15:05.277 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-30T23:15:05.300 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-30T23:15:05.322 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-08-30T23:15:05.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-08-30T23:15:05.342 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-08-30T23:15:05.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-08-30T23:15:05.366 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-08-30T23:15:05.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-08-30T23:15:05.546 INFO:teuthology.orchestra.run.smithi106.stdout:45 files removed 2024-08-30T23:15:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:45 files removed 2024-08-30T23:15:05.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw 2024-08-30T23:15:05.592 INFO:teuthology.orchestra.run.smithi136.stdout:45 files removed 2024-08-30T23:15:05.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-08-30T23:15:05.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-08-30T23:15:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 298 kB/s | 83 kB 00:00 2024-08-30T23:15:06.159 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 287 kB/s | 83 kB 00:00 2024-08-30T23:15:06.194 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 277 kB/s | 83 kB 00:00 2024-08-30T23:15:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-08-30T23:15:06.435 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 50 kB/s | 12 kB 00:00 2024-08-30T23:15:06.478 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-08-30T23:15:06.694 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-30T23:15:06.711 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-30T23:15:06.753 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-08-30T23:15:07.212 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-08-30T23:15:07.278 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-08-30T23:15:08.511 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-08-30T23:15:09.129 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-08-30T23:15:09.290 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-08-30T23:15:10.504 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-08-30T23:15:13.446 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.6 MB 00:00 2024-08-30T23:15:14.964 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.6 MB 00:00 2024-08-30T23:15:14.997 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 18 kB 00:00 2024-08-30T23:15:15.759 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 35 MB/s | 22 MB 00:00 2024-08-30T23:15:16.689 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 18 kB 00:00 2024-08-30T23:15:17.526 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 32 MB/s | 22 MB 00:00 2024-08-30T23:15:17.688 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.6 MB 00:04 2024-08-30T23:15:19.355 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-08-30T23:15:20.435 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 23 MB/s | 22 MB 00:00 2024-08-30T23:15:21.098 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-30T23:15:22.933 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-30T23:15:23.415 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.1.1-242.g244a32d3.el9 ceph 11 M 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.4 M 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.2 M 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.1.1-242.g244a32d3.el9 ceph 5.4 M 2024-08-30T23:15:23.418 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.1.1-242.g244a32d3.el9 ceph 21 M 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.1.1-242.g244a32d3.el9 ceph 25 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 753 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 506 k 2024-08-30T23:15:23.419 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 5.3 M 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 45 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.1.1-242.g244a32d3.el9 ceph 136 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.1.1-242.g244a32d3.el9 ceph 162 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.1.1-242.g244a32d3.el9 ceph 320 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 301 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.1.1-242.g244a32d3.el9 ceph 99 k 2024-08-30T23:15:23.420 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:15:23.421 INFO:teuthology.orchestra.run.smithi136.stdout:Install 30 Packages 2024-08-30T23:15:23.422 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-08-30T23:15:23.422 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:23.422 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 60 M 2024-08-30T23:15:23.422 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:15:24.357 INFO:teuthology.orchestra.run.smithi136.stdout:(1/32): ceph-base-19.1.1-242.g244a32d3.el9.x86_ 9.8 MB/s | 5.4 MB 00:00 2024-08-30T23:15:24.432 INFO:teuthology.orchestra.run.smithi136.stdout:(2/32): ceph-selinux-19.1.1-242.g244a32d3.el9.x 337 kB/s | 25 kB 00:00 2024-08-30T23:15:24.608 INFO:teuthology.orchestra.run.smithi136.stdout:(3/32): ceph-radosgw-19.1.1-242.g244a32d3.el9.x 13 MB/s | 11 MB 00:00 2024-08-30T23:15:24.641 INFO:teuthology.orchestra.run.smithi136.stdout:(4/32): libcephfs2-19.1.1-242.g244a32d3.el9.x86 3.5 MB/s | 753 kB 00:00 2024-08-30T23:15:24.666 INFO:teuthology.orchestra.run.smithi136.stdout:(5/32): libradosstriper1-19.1.1-242.g244a32d3.e 8.5 MB/s | 506 kB 00:00 2024-08-30T23:15:24.750 INFO:teuthology.orchestra.run.smithi136.stdout:(6/32): python3-ceph-argparse-19.1.1-242.g244a3 543 kB/s | 45 kB 00:00 2024-08-30T23:15:24.850 INFO:teuthology.orchestra.run.smithi136.stdout:(7/32): python3-ceph-common-19.1.1-242.g244a32d 1.3 MB/s | 136 kB 00:00 2024-08-30T23:15:24.942 INFO:teuthology.orchestra.run.smithi136.stdout:(8/32): python3-cephfs-19.1.1-242.g244a32d3.el9 1.7 MB/s | 162 kB 00:00 2024-08-30T23:15:25.151 INFO:teuthology.orchestra.run.smithi136.stdout:(9/32): ceph-common-19.1.1-242.g244a32d3.el9.x8 16 MB/s | 21 MB 00:01 2024-08-30T23:15:25.243 INFO:teuthology.orchestra.run.smithi136.stdout:(10/32): librgw2-19.1.1-242.g244a32d3.el9.x86_6 8.9 MB/s | 5.3 MB 00:00 2024-08-30T23:15:25.257 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.1.1-242.g244a32d3.el9 ceph 11 M 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.4 M 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.2 M 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-30T23:15:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.1.1-242.g244a32d3.el9 ceph 5.4 M 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.1.1-242.g244a32d3.el9 ceph 21 M 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.1.1-242.g244a32d3.el9 ceph 25 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 753 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 506 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 5.3 M 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-30T23:15:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 45 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.1.1-242.g244a32d3.el9 ceph 136 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.1.1-242.g244a32d3.el9 ceph 162 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.1.1-242.g244a32d3.el9 ceph 320 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 301 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.1.1-242.g244a32d3.el9 ceph 99 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-30T23:15:25.262 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout:Install 30 Packages 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:25.263 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 60 M 2024-08-30T23:15:25.264 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:15:25.267 INFO:teuthology.orchestra.run.smithi136.stdout:(11/32): python3-rados-19.1.1-242.g244a32d3.el9 984 kB/s | 320 kB 00:00 2024-08-30T23:15:25.293 INFO:teuthology.orchestra.run.smithi136.stdout:(12/32): python3-rgw-19.1.1-242.g244a32d3.el9.x 1.9 MB/s | 99 kB 00:00 2024-08-30T23:15:25.318 INFO:teuthology.orchestra.run.smithi136.stdout:(13/32): python3-rbd-19.1.1-242.g244a32d3.el9.x 1.8 MB/s | 301 kB 00:00 2024-08-30T23:15:25.469 INFO:teuthology.orchestra.run.smithi136.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 533 kB/s | 107 kB 00:00 2024-08-30T23:15:25.494 INFO:teuthology.orchestra.run.smithi136.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-08-30T23:15:25.519 INFO:teuthology.orchestra.run.smithi136.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 710 kB/s | 160 kB 00:00 2024-08-30T23:15:25.569 INFO:teuthology.orchestra.run.smithi136.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 845 kB/s | 42 kB 00:00 2024-08-30T23:15:25.603 INFO:teuthology.orchestra.run.smithi136.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-08-30T23:15:25.645 INFO:teuthology.orchestra.run.smithi136.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-08-30T23:15:25.670 INFO:teuthology.orchestra.run.smithi136.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 536 kB/s | 36 kB 00:00 2024-08-30T23:15:25.704 INFO:teuthology.orchestra.run.smithi136.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-08-30T23:15:25.729 INFO:teuthology.orchestra.run.smithi136.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-08-30T23:15:25.774 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-08-30T23:15:25.788 INFO:teuthology.orchestra.run.smithi136.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-08-30T23:15:25.813 INFO:teuthology.orchestra.run.smithi136.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-30T23:15:25.838 INFO:teuthology.orchestra.run.smithi136.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 277 kB/s | 26 kB 00:00 2024-08-30T23:15:25.864 INFO:teuthology.orchestra.run.smithi136.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-30T23:15:25.897 INFO:teuthology.orchestra.run.smithi136.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-08-30T23:15:25.926 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-base-19.1.1-242.g244a32d3.el9.x86_ 10 MB/s | 5.4 MB 00:00 2024-08-30T23:15:25.956 INFO:teuthology.orchestra.run.smithi136.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-08-30T23:15:25.993 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-selinux-19.1.1-242.g244a32d3.el9.x 378 kB/s | 25 kB 00:00 2024-08-30T23:15:25.997 INFO:teuthology.orchestra.run.smithi136.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-30T23:15:26.081 INFO:teuthology.orchestra.run.smithi136.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-08-30T23:15:26.093 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): libcephfs2-19.1.1-242.g244a32d3.el9.x86 7.4 MB/s | 753 kB 00:00 2024-08-30T23:15:26.210 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): ceph-radosgw-19.1.1-242.g244a32d3.el9.x 13 MB/s | 11 MB 00:00 2024-08-30T23:15:26.235 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): libradosstriper1-19.1.1-242.g244a32d3.e 3.5 MB/s | 506 kB 00:00 2024-08-30T23:15:26.353 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-ceph-argparse-19.1.1-242.g244a3 383 kB/s | 45 kB 00:00 2024-08-30T23:15:26.364 INFO:teuthology.orchestra.run.smithi136.stdout:(31/32): librbd1-19.1.1-242.g244a32d3.el9.x86_6 8.6 MB/s | 3.2 MB 00:00 2024-08-30T23:15:26.419 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): python3-ceph-common-19.1.1-242.g244a32d 2.0 MB/s | 136 kB 00:00 2024-08-30T23:15:26.431 INFO:teuthology.orchestra.run.smithi136.stdout:(32/32): librados2-19.1.1-242.g244a32d3.el9.x86 7.2 MB/s | 3.4 MB 00:00 2024-08-30T23:15:26.436 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:15:26.437 INFO:teuthology.orchestra.run.smithi136.stdout:Total 20 MB/s | 60 MB 00:03 2024-08-30T23:15:26.503 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): librgw2-19.1.1-242.g244a32d3.el9.x86_64 18 MB/s | 5.3 MB 00:00 2024-08-30T23:15:26.528 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-cephfs-19.1.1-242.g244a32d3.el9 1.5 MB/s | 162 kB 00:00 2024-08-30T23:15:26.536 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:15:26.553 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-rados-19.1.1-242.g244a32d3.el9 6.3 MB/s | 320 kB 00:00 2024-08-30T23:15:26.571 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:15:26.571 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:15:26.578 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-rbd-19.1.1-242.g244a32d3.el9.x 5.8 MB/s | 301 kB 00:00 2024-08-30T23:15:26.604 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-rgw-19.1.1-242.g244a32d3.el9.x 1.9 MB/s | 99 kB 00:00 2024-08-30T23:15:26.804 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): ceph-common-19.1.1-242.g244a32d3.el9.x 15 MB/s | 21 MB 00:01 2024-08-30T23:15:27.045 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:15:27.046 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:15:27.138 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 136 kB/s | 45 kB 00:00 2024-08-30T23:15:27.163 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 183 kB/s | 107 kB 00:00 2024-08-30T23:15:27.230 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 256 kB/s | 160 kB 00:00 2024-08-30T23:15:27.339 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-08-30T23:15:27.447 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-08-30T23:15:27.523 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-08-30T23:15:27.540 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 390 kB/s | 36 kB 00:00 2024-08-30T23:15:27.590 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 6.1 MB/s | 309 kB 00:00 2024-08-30T23:15:27.624 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-08-30T23:15:27.666 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-08-30T23:15:27.691 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-08-30T23:15:27.716 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2024-08-30T23:15:27.741 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-30T23:15:27.775 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.1 MB/s | 191 kB 00:00 2024-08-30T23:15:27.876 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-08-30T23:15:27.909 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-08-30T23:15:27.951 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-08-30T23:15:28.116 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:19.1.1-242.g244a32d3.el9 ceph 11 M 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.4 M 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.2 M 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:19.1.1-242.g244a32d3.el9 ceph 5.4 M 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:19.1.1-242.g244a32d3.el9 ceph 21 M 2024-08-30T23:15:28.119 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:19.1.1-242.g244a32d3.el9 ceph 25 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 753 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 506 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:19.1.1-242.g244a32d3.el9 ceph 5.3 M 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-30T23:15:28.120 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 45 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:19.1.1-242.g244a32d3.el9 ceph 136 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:19.1.1-242.g244a32d3.el9 ceph 162 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:19.1.1-242.g244a32d3.el9 ceph 320 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 301 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:19.1.1-242.g244a32d3.el9 ceph 99 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-08-30T23:15:28.121 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout:Install 30 Packages 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 2 Packages 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 60 M 2024-08-30T23:15:28.122 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:15:28.135 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): librbd1-19.1.1-242.g244a32d3.el9.x86_6 14 MB/s | 3.2 MB 00:00 2024-08-30T23:15:28.193 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): librados2-19.1.1-242.g244a32d3.el9.x86 11 MB/s | 3.4 MB 00:00 2024-08-30T23:15:28.198 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:15:28.198 INFO:teuthology.orchestra.run.smithi173.stdout:Total 21 MB/s | 60 MB 00:02 2024-08-30T23:15:28.296 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:15:28.333 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:15:28.333 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:15:28.664 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:15:28.740 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-30T23:15:28.769 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-30T23:15:28.799 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:15:28.800 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:15:28.803 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-30T23:15:28.808 INFO:teuthology.orchestra.run.smithi106.stdout:(1/32): ceph-base-19.1.1-242.g244a32d3.el9.x86_ 8.9 MB/s | 5.4 MB 00:00 2024-08-30T23:15:28.825 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-30T23:15:28.848 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-30T23:15:28.859 INFO:teuthology.orchestra.run.smithi106.stdout:(2/32): ceph-selinux-19.1.1-242.g244a32d3.el9.x 503 kB/s | 25 kB 00:00 2024-08-30T23:15:28.959 INFO:teuthology.orchestra.run.smithi106.stdout:(3/32): libcephfs2-19.1.1-242.g244a32d3.el9.x86 7.4 MB/s | 753 kB 00:00 2024-08-30T23:15:29.059 INFO:teuthology.orchestra.run.smithi106.stdout:(4/32): libradosstriper1-19.1.1-242.g244a32d3.e 4.9 MB/s | 506 kB 00:00 2024-08-30T23:15:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-30T23:15:29.185 INFO:teuthology.orchestra.run.smithi106.stdout:(5/32): ceph-radosgw-19.1.1-242.g244a32d3.el9.x 11 MB/s | 11 MB 00:00 2024-08-30T23:15:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-30T23:15:29.393 INFO:teuthology.orchestra.run.smithi106.stdout:(6/32): ceph-common-19.1.1-242.g244a32d3.el9.x8 18 MB/s | 21 MB 00:01 2024-08-30T23:15:29.418 INFO:teuthology.orchestra.run.smithi106.stdout:(7/32): python3-ceph-argparse-19.1.1-242.g244a3 193 kB/s | 45 kB 00:00 2024-08-30T23:15:29.444 INFO:teuthology.orchestra.run.smithi106.stdout:(8/32): python3-ceph-common-19.1.1-242.g244a32d 2.7 MB/s | 136 kB 00:00 2024-08-30T23:15:29.477 INFO:teuthology.orchestra.run.smithi106.stdout:(9/32): python3-cephfs-19.1.1-242.g244a32d3.el9 2.7 MB/s | 162 kB 00:00 2024-08-30T23:15:29.503 INFO:teuthology.orchestra.run.smithi106.stdout:(10/32): python3-rados-19.1.1-242.g244a32d3.el9 5.3 MB/s | 320 kB 00:00 2024-08-30T23:15:29.528 INFO:teuthology.orchestra.run.smithi106.stdout:(11/32): python3-rbd-19.1.1-242.g244a32d3.el9.x 5.9 MB/s | 301 kB 00:00 2024-08-30T23:15:29.543 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-30T23:15:29.553 INFO:teuthology.orchestra.run.smithi106.stdout:(12/32): python3-rgw-19.1.1-242.g244a32d3.el9.x 1.9 MB/s | 99 kB 00:00 2024-08-30T23:15:29.637 INFO:teuthology.orchestra.run.smithi106.stdout:(13/32): librgw2-19.1.1-242.g244a32d3.el9.x86_6 9.3 MB/s | 5.3 MB 00:00 2024-08-30T23:15:29.646 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:29.738 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:29.836 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 8/34 2024-08-30T23:15:29.864 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 9/34 2024-08-30T23:15:29.910 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 9/34 2024-08-30T23:15:29.939 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-30T23:15:29.979 INFO:teuthology.orchestra.run.smithi106.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 237 kB/s | 107 kB 00:00 2024-08-30T23:15:29.982 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x 11/34 2024-08-30T23:15:30.004 INFO:teuthology.orchestra.run.smithi106.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 123 kB/s | 45 kB 00:00 2024-08-30T23:15:30.060 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 12/34 2024-08-30T23:15:30.083 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:30.105 INFO:teuthology.orchestra.run.smithi106.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 291 kB/s | 160 kB 00:00 2024-08-30T23:15:30.138 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:30.172 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-30T23:15:30.221 INFO:teuthology.orchestra.run.smithi106.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-08-30T23:15:30.253 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:15:30.280 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-08-30T23:15:30.306 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-30T23:15:30.344 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-30T23:15:30.346 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-30T23:15:30.347 INFO:teuthology.orchestra.run.smithi106.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 853 kB/s | 292 kB 00:00 2024-08-30T23:15:30.372 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-30T23:15:30.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-30T23:15:30.398 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-30T23:15:30.405 INFO:teuthology.orchestra.run.smithi106.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-08-30T23:15:30.410 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-30T23:15:30.422 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-30T23:15:30.439 INFO:teuthology.orchestra.run.smithi106.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 221 kB 00:00 2024-08-30T23:15:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-30T23:15:30.448 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-30T23:15:30.464 INFO:teuthology.orchestra.run.smithi106.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 306 kB/s | 36 kB 00:00 2024-08-30T23:15:30.506 INFO:teuthology.orchestra.run.smithi106.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-08-30T23:15:30.540 INFO:teuthology.orchestra.run.smithi106.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-08-30T23:15:30.565 INFO:teuthology.orchestra.run.smithi106.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-30T23:15:30.582 INFO:teuthology.orchestra.run.smithi106.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 353 kB/s | 26 kB 00:00 2024-08-30T23:15:30.607 INFO:teuthology.orchestra.run.smithi106.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-08-30T23:15:30.641 INFO:teuthology.orchestra.run.smithi106.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-08-30T23:15:30.699 INFO:teuthology.orchestra.run.smithi106.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-08-30T23:15:30.713 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-30T23:15:30.750 INFO:teuthology.orchestra.run.smithi106.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-30T23:15:30.751 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-30T23:15:30.767 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-30T23:15:30.797 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 22/34 2024-08-30T23:15:30.825 INFO:teuthology.orchestra.run.smithi106.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-08-30T23:15:30.894 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 22/34 2024-08-30T23:15:30.926 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 23/34 2024-08-30T23:15:30.962 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-30T23:15:30.967 INFO:teuthology.orchestra.run.smithi106.stdout:(31/32): librados2-19.1.1-242.g244a32d3.el9.x86 13 MB/s | 3.4 MB 00:00 2024-08-30T23:15:31.050 INFO:teuthology.orchestra.run.smithi106.stdout:(32/32): librbd1-19.1.1-242.g244a32d3.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-08-30T23:15:31.056 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:15:31.056 INFO:teuthology.orchestra.run.smithi106.stdout:Total 21 MB/s | 60 MB 00:02 2024-08-30T23:15:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-30T23:15:31.160 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:15:31.196 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:15:31.196 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:15:31.218 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 8/34 2024-08-30T23:15:31.441 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-30T23:15:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 9/34 2024-08-30T23:15:31.466 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 26/34 2024-08-30T23:15:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 9/34 2024-08-30T23:15:31.520 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 26/34 2024-08-30T23:15:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-30T23:15:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 27/34 2024-08-30T23:15:31.605 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x 11/34 2024-08-30T23:15:31.618 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86 28/34 2024-08-30T23:15:31.662 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:15:31.662 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:15:31.673 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 12/34 2024-08-30T23:15:31.696 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:31.761 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:31.795 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-30T23:15:31.903 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-08-30T23:15:31.929 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-30T23:15:31.969 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-30T23:15:32.003 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-30T23:15:32.033 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-30T23:15:32.065 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-30T23:15:32.379 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-30T23:15:32.420 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 22/34 2024-08-30T23:15:32.510 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 22/34 2024-08-30T23:15:32.541 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 23/34 2024-08-30T23:15:32.577 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-30T23:15:33.059 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-30T23:15:33.081 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 26/34 2024-08-30T23:15:33.152 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 26/34 2024-08-30T23:15:33.229 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 27/34 2024-08-30T23:15:33.244 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:15:33.267 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86 28/34 2024-08-30T23:15:33.334 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-30T23:15:33.372 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-30T23:15:33.397 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-30T23:15:33.420 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-30T23:15:33.446 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-30T23:15:33.767 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-30T23:15:33.842 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-30T23:15:33.891 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:33.927 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:34.230 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-30T23:15:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:34.399 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:34.467 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:34.486 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 8/34 2024-08-30T23:15:34.515 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 30/34 2024-08-30T23:15:34.518 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 9/34 2024-08-30T23:15:34.580 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 9/34 2024-08-30T23:15:34.608 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 30/34 2024-08-30T23:15:34.608 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-30T23:15:34.608 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-30T23:15:34.608 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:34.609 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-30T23:15:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:34.652 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x 11/34 2024-08-30T23:15:34.721 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 12/34 2024-08-30T23:15:34.745 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-30T23:15:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-08-30T23:15:34.994 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-30T23:15:35.035 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-30T23:15:35.068 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-30T23:15:35.097 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-30T23:15:35.129 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-30T23:15:35.438 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-30T23:15:35.476 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 22/34 2024-08-30T23:15:35.565 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 22/34 2024-08-30T23:15:35.579 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:35.606 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 23/34 2024-08-30T23:15:35.641 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-30T23:15:35.754 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:36.120 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-30T23:15:36.146 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 26/34 2024-08-30T23:15:36.218 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 26/34 2024-08-30T23:15:36.295 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 27/34 2024-08-30T23:15:36.298 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:36.324 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86 28/34 2024-08-30T23:15:36.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 30/34 2024-08-30T23:15:36.426 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 30/34 2024-08-30T23:15:36.426 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-30T23:15:36.426 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-30T23:15:36.426 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:36.460 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:38.630 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:38.673 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:39.215 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 29/34 2024-08-30T23:15:39.259 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 30/34 2024-08-30T23:15:39.350 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 30/34 2024-08-30T23:15:39.350 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-30T23:15:39.350 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-30T23:15:39.350 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:39.377 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-08-30T23:15:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 32/34 2024-08-30T23:15:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 32/34 2024-08-30T23:15:45.810 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:15:45.810 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-30T23:15:45.810 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T23:15:45.810 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T23:15:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-30T23:15:45.904 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-30T23:15:45.904 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:46.605 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:46.605 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-08-30T23:15:46.605 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-08-30T23:15:46.605 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-08-30T23:15:46.605 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-08-30T23:15:46.606 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-08-30T23:15:46.606 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-08-30T23:15:46.606 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-08-30T23:15:46.606 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:47.182 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 1/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 3/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 4/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 5/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 6/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x 8/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86 9/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 10/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 11/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 12/34 2024-08-30T23:15:47.183 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-30T23:15:47.184 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-08-30T23:15:47.185 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-30T23:15:47.186 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:47.186 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-30T23:15:47.186 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 33/34 2024-08-30T23:15:47.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 32/34 2024-08-30T23:15:47.610 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 32/34 2024-08-30T23:15:47.611 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:15:47.611 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-30T23:15:47.611 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T23:15:47.611 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T23:15:47.611 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:47.613 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-30T23:15:47.688 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-30T23:15:47.688 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:47.974 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-30T23:15:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-20.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:47.977 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:15:48.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-08-30T23:15:48.893 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:27 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 1/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 3/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 4/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 5/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 6/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x 8/34 2024-08-30T23:15:48.901 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86 9/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 10/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 11/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 12/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-30T23:15:48.902 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-30T23:15:48.903 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 33/34 2024-08-30T23:15:49.037 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.1.1-242.g244a32d3.el9 ceph 49 M 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.1.1-242.g244a32d3.el9 ceph 163 k 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-30T23:15:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 49 M 2024-08-30T23:15:49.039 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-08-30T23:15:49.040 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:15:49.248 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:49.248 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-08-30T23:15:49.248 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-08-30T23:15:49.249 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-08-30T23:15:49.249 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-08-30T23:15:49.249 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-08-30T23:15:49.249 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /root 2024-08-30T23:15:49.249 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-08-30T23:15:49.249 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:49.522 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-08-30T23:15:49.589 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): libcephsqlite-19.1.1-242.g244a32d3.el9.x 554 kB/s | 163 kB 00:00 2024-08-30T23:15:49.615 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 689 kB/s | 64 kB 00:00 2024-08-30T23:15:49.640 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:49.640 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:49.641 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-08-30T23:15:49.641 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.641 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.641 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:15:49.641 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-30T23:15:49.641 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-30T23:15:49.642 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.643 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-20.el9.x86_64 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:49.644 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:15:49.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-08-30T23:15:50.269 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 32/34 2024-08-30T23:15:50.294 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 32/34 2024-08-30T23:15:50.294 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:15:50.295 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-30T23:15:50.295 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T23:15:50.295 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-30T23:15:50.295 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:50.296 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-30T23:15:50.379 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-30T23:15:50.380 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:50.516 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:28 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:15:50.661 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:15:50.661 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:15:50.661 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-08-30T23:15:50.661 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:15:50.661 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.1.1-242.g244a32d3.el9 ceph 49 M 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.1.1-242.g244a32d3.el9 ceph 163 k 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-08-30T23:15:50.662 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:15:50.663 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 49 M 2024-08-30T23:15:50.663 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-08-30T23:15:50.663 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:15:50.976 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): ceph-test-19.1.1-242.g244a32d3.el9.x86_6 29 MB/s | 49 MB 00:01 2024-08-30T23:15:50.979 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:15:50.979 INFO:teuthology.orchestra.run.smithi136.stdout:Total 25 MB/s | 49 MB 00:01 2024-08-30T23:15:50.987 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): libcephsqlite-19.1.1-242.g244a32d3.el9.x 755 kB/s | 163 kB 00:00 2024-08-30T23:15:51.007 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:15:51.027 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:15:51.027 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:15:51.404 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:15:51.404 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 1/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 3/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 4/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 5/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 6/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 7/34 2024-08-30T23:15:51.618 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x 8/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86 9/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 10/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 11/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 12/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 13/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-30T23:15:51.619 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-30T23:15:51.620 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-30T23:15:51.620 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-30T23:15:51.620 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-30T23:15:51.620 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-30T23:15:51.620 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-30T23:15:51.621 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-30T23:15:51.621 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-08-30T23:15:51.621 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-30T23:15:51.621 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:19.1.1-242.g244a32d3.el9.x86_64 31/34 2024-08-30T23:15:51.621 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-30T23:15:51.621 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 33/34 2024-08-30T23:15:51.644 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 97 kB/s | 64 kB 00:00 2024-08-30T23:15:51.795 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 297 kB/s | 304 kB 00:01 2024-08-30T23:15:51.924 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:15:52.011 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-30T23:15:52.051 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-30T23:15:52.079 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 3/4 2024-08-30T23:15:52.357 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): ceph-test-19.1.1-242.g244a32d3.el9.x86_6 31 MB/s | 49 MB 00:01 2024-08-30T23:15:52.359 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:15:52.359 INFO:teuthology.orchestra.run.smithi173.stdout:Total 29 MB/s | 49 MB 00:01 2024-08-30T23:15:52.369 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-30T23:15:52.369 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.370 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-30T23:15:52.371 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: re2-1:20211101-20.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:52.372 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:15:52.386 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:15:52.407 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:15:52.407 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:15:52.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-test 2024-08-30T23:15:52.788 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:15:52.788 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:15:53.230 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:28 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:15:53.319 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:15:53.378 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:19.1.1-242.g244a32d3.el9 ceph 49 M 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:19.1.1-242.g244a32d3.el9 ceph 163 k 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:15:53.379 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:15:53.380 INFO:teuthology.orchestra.run.smithi106.stdout:Install 4 Packages 2024-08-30T23:15:53.380 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:15:53.380 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 49 M 2024-08-30T23:15:53.380 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 209 M 2024-08-30T23:15:53.380 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:15:53.384 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-30T23:15:53.424 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-30T23:15:53.452 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 3/4 2024-08-30T23:15:53.644 INFO:teuthology.orchestra.run.smithi106.stdout:(1/4): libcephsqlite-19.1.1-242.g244a32d3.el9.x 752 kB/s | 163 kB 00:00 2024-08-30T23:15:53.703 INFO:teuthology.orchestra.run.smithi106.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-08-30T23:15:53.821 INFO:teuthology.orchestra.run.smithi106.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 361 kB/s | 64 kB 00:00 2024-08-30T23:15:55.063 INFO:teuthology.orchestra.run.smithi106.stdout:(4/4): ceph-test-19.1.1-242.g244a32d3.el9.x86_6 30 MB/s | 49 MB 00:01 2024-08-30T23:15:55.065 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:15:55.065 INFO:teuthology.orchestra.run.smithi106.stdout:Total 29 MB/s | 49 MB 00:01 2024-08-30T23:15:55.093 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:15:55.112 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:15:55.112 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:15:55.497 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:15:55.498 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:15:56.027 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:15:56.099 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-30T23:15:56.139 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-30T23:15:56.168 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 3/4 2024-08-30T23:15:56.645 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 3/4 2024-08-30T23:15:56.718 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 4/4 2024-08-30T23:15:58.035 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 3/4 2024-08-30T23:15:58.173 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 4/4 2024-08-30T23:15:59.195 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 4/4 2024-08-30T23:15:59.195 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 1/4 2024-08-30T23:15:59.195 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 2/4 2024-08-30T23:15:59.195 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-30T23:15:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-30T23:15:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:59.524 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:15:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:15:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-30T23:15:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-30T23:15:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:15:59.525 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:15:59.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-08-30T23:16:00.357 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:39 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 4/4 2024-08-30T23:16:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 1/4 2024-08-30T23:16:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 2/4 2024-08-30T23:16:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-30T23:16:00.492 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:00.494 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:00.494 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.1.1-242.g244a32d3.el9 ceph 6.5 k 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.1.1-242.g244a32d3.el9 ceph 2.4 M 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.1.1-242.g244a32d3.el9 ceph 1.5 M 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.1.1-242.g244a32d3.el9 ceph-noarch 249 k 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.1.1-242.g244a32d3.el9 ceph 4.7 M 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 17 M 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-30T23:16:00.495 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-30T23:16:00.496 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-30T23:16:00.497 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout:Install 37 Packages 2024-08-30T23:16:00.498 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:00.499 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 30 M 2024-08-30T23:16:00.499 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 108 M 2024-08-30T23:16:00.499 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:00.733 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 3/4 2024-08-30T23:16:00.778 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-30T23:16:00.778 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:00.778 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:00.779 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:00.779 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:00.779 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-30T23:16:00.779 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-30T23:16:00.779 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:00.779 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:00.805 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 4/4 2024-08-30T23:16:00.854 INFO:teuthology.orchestra.run.smithi136.stdout:(1/37): ceph-19.1.1-242.g244a32d3.el9.x86_64.rp 49 kB/s | 6.5 kB 00:00 2024-08-30T23:16:01.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-08-30T23:16:01.146 INFO:teuthology.orchestra.run.smithi136.stdout:(2/37): ceph-mds-19.1.1-242.g244a32d3.el9.x86_6 5.6 MB/s | 2.4 MB 00:00 2024-08-30T23:16:01.213 INFO:teuthology.orchestra.run.smithi136.stdout:(3/37): ceph-mgr-19.1.1-242.g244a32d3.el9.x86_6 3.1 MB/s | 1.5 MB 00:00 2024-08-30T23:16:01.263 INFO:teuthology.orchestra.run.smithi136.stdout:(4/37): ceph-mgr-modules-core-19.1.1-242.g244a3 4.9 MB/s | 249 kB 00:00 2024-08-30T23:16:01.472 INFO:teuthology.orchestra.run.smithi136.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-08-30T23:16:01.622 INFO:teuthology.orchestra.run.smithi136.stdout:(6/37): ceph-mon-19.1.1-242.g244a32d3.el9.x86_6 6.1 MB/s | 4.7 MB 00:00 2024-08-30T23:16:01.645 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:39 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:01.656 INFO:teuthology.orchestra.run.smithi136.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-08-30T23:16:01.798 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.1.1-242.g244a32d3.el9 ceph 6.5 k 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.1.1-242.g244a32d3.el9 ceph 2.4 M 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.1.1-242.g244a32d3.el9 ceph 1.5 M 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.1.1-242.g244a32d3.el9 ceph-noarch 249 k 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.1.1-242.g244a32d3.el9 ceph 4.7 M 2024-08-30T23:16:01.801 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 17 M 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-30T23:16:01.802 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-30T23:16:01.803 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout:Install 37 Packages 2024-08-30T23:16:01.804 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:01.805 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 30 M 2024-08-30T23:16:01.805 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 108 M 2024-08-30T23:16:01.805 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:01.856 INFO:teuthology.orchestra.run.smithi136.stdout:(8/37): ceph-osd-19.1.1-242.g244a32d3.el9.x86_6 24 MB/s | 17 MB 00:00 2024-08-30T23:16:01.873 INFO:teuthology.orchestra.run.smithi136.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 623 kB/s | 135 kB 00:00 2024-08-30T23:16:01.936 INFO:teuthology.orchestra.run.smithi136.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.4 MB/s | 215 kB 00:00 2024-08-30T23:16:01.961 INFO:teuthology.orchestra.run.smithi136.stdout:(11/37): python3-ply-3.11-14.el9.noarch.rpm 314 kB/s | 106 kB 00:00 2024-08-30T23:16:02.028 INFO:teuthology.orchestra.run.smithi136.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 728 kB/s | 125 kB 00:00 2024-08-30T23:16:02.145 INFO:teuthology.orchestra.run.smithi136.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 298 kB/s | 35 kB 00:00 2024-08-30T23:16:02.162 INFO:teuthology.orchestra.run.smithi136.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-08-30T23:16:02.187 INFO:teuthology.orchestra.run.smithi136.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 751 kB/s | 188 kB 00:00 2024-08-30T23:16:02.212 INFO:teuthology.orchestra.run.smithi136.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-30T23:16:02.238 INFO:teuthology.orchestra.run.smithi136.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-08-30T23:16:02.263 INFO:teuthology.orchestra.run.smithi136.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 654 kB/s | 77 kB 00:00 2024-08-30T23:16:02.280 INFO:teuthology.orchestra.run.smithi136.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 331 kB/s | 22 kB 00:00 2024-08-30T23:16:02.330 INFO:teuthology.orchestra.run.smithi136.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-08-30T23:16:02.355 INFO:teuthology.orchestra.run.smithi136.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-08-30T23:16:02.406 INFO:teuthology.orchestra.run.smithi136.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-08-30T23:16:02.431 INFO:teuthology.orchestra.run.smithi136.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 703 kB/s | 18 kB 00:00 2024-08-30T23:16:02.457 INFO:teuthology.orchestra.run.smithi136.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-08-30T23:16:02.460 INFO:teuthology.orchestra.run.smithi173.stdout:(1/37): ceph-19.1.1-242.g244a32d3.el9.x86_64.rp 48 kB/s | 6.5 kB 00:00 2024-08-30T23:16:02.482 INFO:teuthology.orchestra.run.smithi136.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-08-30T23:16:02.507 INFO:teuthology.orchestra.run.smithi136.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-08-30T23:16:02.533 INFO:teuthology.orchestra.run.smithi136.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-08-30T23:16:02.558 INFO:teuthology.orchestra.run.smithi136.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-08-30T23:16:02.584 INFO:teuthology.orchestra.run.smithi136.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-08-30T23:16:02.608 INFO:teuthology.orchestra.run.smithi136.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-08-30T23:16:02.634 INFO:teuthology.orchestra.run.smithi136.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-08-30T23:16:02.659 INFO:teuthology.orchestra.run.smithi136.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-30T23:16:02.677 INFO:teuthology.orchestra.run.smithi173.stdout:(2/37): ceph-mds-19.1.1-242.g244a32d3.el9.x86_6 6.8 MB/s | 2.4 MB 00:00 2024-08-30T23:16:02.684 INFO:teuthology.orchestra.run.smithi136.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-30T23:16:02.710 INFO:teuthology.orchestra.run.smithi136.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-30T23:16:02.710 INFO:teuthology.orchestra.run.smithi173.stdout:(3/37): ceph-mgr-19.1.1-242.g244a32d3.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2024-08-30T23:16:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-30T23:16:02.760 INFO:teuthology.orchestra.run.smithi136.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-08-30T23:16:02.786 INFO:teuthology.orchestra.run.smithi136.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-30T23:16:02.790 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:02.791 INFO:teuthology.orchestra.run.smithi136.stdout:Total 13 MB/s | 30 MB 00:02 2024-08-30T23:16:02.836 INFO:teuthology.orchestra.run.smithi173.stdout:(4/37): ceph-mgr-modules-core-19.1.1-242.g244a3 1.9 MB/s | 249 kB 00:00 2024-08-30T23:16:02.911 INFO:teuthology.orchestra.run.smithi173.stdout:(5/37): ceph-mon-19.1.1-242.g244a32d3.el9.x86_6 10 MB/s | 4.7 MB 00:00 2024-08-30T23:16:02.941 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:02.981 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:02.981 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:03.153 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 4/4 2024-08-30T23:16:03.153 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 1/4 2024-08-30T23:16:03.153 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 2/4 2024-08-30T23:16:03.153 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-30T23:16:03.228 INFO:teuthology.orchestra.run.smithi173.stdout:(6/37): ceph-osd-19.1.1-242.g244a32d3.el9.x86_6 31 MB/s | 17 MB 00:00 2024-08-30T23:16:03.278 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:03.280 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:03.479 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-30T23:16:03.479 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:03.479 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:03.479 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:03.479 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:03.479 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-30T23:16:03.480 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-30T23:16:03.480 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:03.480 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:03.703 INFO:teuthology.orchestra.run.smithi173.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 292 kB/s | 253 kB 00:00 2024-08-30T23:16:03.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph 2024-08-30T23:16:03.962 INFO:teuthology.orchestra.run.smithi173.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 147 kB/s | 106 kB 00:00 2024-08-30T23:16:03.995 INFO:teuthology.orchestra.run.smithi173.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-08-30T23:16:04.336 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:39 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:04.421 INFO:teuthology.orchestra.run.smithi173.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 189 kB/s | 135 kB 00:00 2024-08-30T23:16:04.480 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:04.482 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:19.1.1-242.g244a32d3.el9 ceph 6.5 k 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:19.1.1-242.g244a32d3.el9 ceph 2.4 M 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:19.1.1-242.g244a32d3.el9 ceph 1.5 M 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:19.1.1-242.g244a32d3.el9 ceph-noarch 249 k 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:19.1.1-242.g244a32d3.el9 ceph 4.7 M 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 17 M 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-30T23:16:04.483 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-30T23:16:04.484 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-30T23:16:04.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout:Install 37 Packages 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:04.486 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 30 M 2024-08-30T23:16:04.487 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 108 M 2024-08-30T23:16:04.487 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:04.579 INFO:teuthology.orchestra.run.smithi173.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-08-30T23:16:04.647 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:04.671 INFO:teuthology.orchestra.run.smithi173.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-08-30T23:16:04.696 INFO:teuthology.orchestra.run.smithi173.stdout:(13/37): python3-requests-2.25.1-8.el9.noarch.r 170 kB/s | 125 kB 00:00 2024-08-30T23:16:04.713 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-30T23:16:04.722 INFO:teuthology.orchestra.run.smithi173.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 705 kB/s | 35 kB 00:00 2024-08-30T23:16:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-30T23:16:04.747 INFO:teuthology.orchestra.run.smithi173.stdout:(15/37): python3-urllib3-1.26.5-6.el9.noarch.rp 286 kB/s | 215 kB 00:00 2024-08-30T23:16:04.772 INFO:teuthology.orchestra.run.smithi173.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-30T23:16:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-30T23:16:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-30T23:16:04.856 INFO:teuthology.orchestra.run.smithi173.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 484 kB/s | 77 kB 00:00 2024-08-30T23:16:04.875 INFO:teuthology.orchestra.run.smithi106.stdout:(1/37): ceph-19.1.1-242.g244a32d3.el9.x86_64.rp 56 kB/s | 6.5 kB 00:00 2024-08-30T23:16:04.881 INFO:teuthology.orchestra.run.smithi173.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 312 kB/s | 42 kB 00:00 2024-08-30T23:16:04.918 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-30T23:16:04.923 INFO:teuthology.orchestra.run.smithi173.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-08-30T23:16:04.948 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-30T23:16:04.948 INFO:teuthology.orchestra.run.smithi173.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-08-30T23:16:04.982 INFO:teuthology.orchestra.run.smithi173.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-08-30T23:16:05.007 INFO:teuthology.orchestra.run.smithi173.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-08-30T23:16:05.033 INFO:teuthology.orchestra.run.smithi173.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 713 kB/s | 18 kB 00:00 2024-08-30T23:16:05.048 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-30T23:16:05.058 INFO:teuthology.orchestra.run.smithi173.stdout:(24/37): lua-devel-5.4.4-4.el9.x86_64.rpm 78 kB/s | 22 kB 00:00 2024-08-30T23:16:05.083 INFO:teuthology.orchestra.run.smithi173.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-08-30T23:16:05.083 INFO:teuthology.orchestra.run.smithi106.stdout:(2/37): ceph-mds-19.1.1-242.g244a32d3.el9.x86_6 7.3 MB/s | 2.4 MB 00:00 2024-08-30T23:16:05.117 INFO:teuthology.orchestra.run.smithi173.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-08-30T23:16:05.117 INFO:teuthology.orchestra.run.smithi106.stdout:(3/37): ceph-mgr-19.1.1-242.g244a32d3.el9.x86_6 4.2 MB/s | 1.5 MB 00:00 2024-08-30T23:16:05.126 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-30T23:16:05.142 INFO:teuthology.orchestra.run.smithi173.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 385 kB/s | 20 kB 00:00 2024-08-30T23:16:05.167 INFO:teuthology.orchestra.run.smithi173.stdout:(28/37): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-08-30T23:16:05.167 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-08-30T23:16:05.192 INFO:teuthology.orchestra.run.smithi106.stdout:(4/37): ceph-mgr-modules-core-19.1.1-242.g244a3 3.3 MB/s | 249 kB 00:00 2024-08-30T23:16:05.193 INFO:teuthology.orchestra.run.smithi173.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 629 kB/s | 46 kB 00:00 2024-08-30T23:16:05.210 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-30T23:16:05.218 INFO:teuthology.orchestra.run.smithi173.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-08-30T23:16:05.234 INFO:teuthology.orchestra.run.smithi173.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-08-30T23:16:05.240 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-30T23:16:05.259 INFO:teuthology.orchestra.run.smithi106.stdout:(5/37): ceph-mon-19.1.1-242.g244a32d3.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-08-30T23:16:05.260 INFO:teuthology.orchestra.run.smithi173.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-08-30T23:16:05.275 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-30T23:16:05.285 INFO:teuthology.orchestra.run.smithi173.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-08-30T23:16:05.308 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-30T23:16:05.310 INFO:teuthology.orchestra.run.smithi173.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-08-30T23:16:05.327 INFO:teuthology.orchestra.run.smithi173.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-08-30T23:16:05.349 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-30T23:16:05.352 INFO:teuthology.orchestra.run.smithi173.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-08-30T23:16:05.384 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-30T23:16:05.403 INFO:teuthology.orchestra.run.smithi173.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-08-30T23:16:05.408 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:05.408 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.4 MB/s | 30 MB 00:03 2024-08-30T23:16:05.418 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-30T23:16:05.435 INFO:teuthology.orchestra.run.smithi106.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-08-30T23:16:05.467 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-30T23:16:05.493 INFO:teuthology.orchestra.run.smithi106.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-08-30T23:16:05.499 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-30T23:16:05.560 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-30T23:16:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:(8/37): ceph-osd-19.1.1-242.g244a32d3.el9.x86_6 29 MB/s | 17 MB 00:00 2024-08-30T23:16:05.696 INFO:teuthology.orchestra.run.smithi106.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 674 kB/s | 135 kB 00:00 2024-08-30T23:16:05.711 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-30T23:16:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-08-30T23:16:05.753 INFO:teuthology.orchestra.run.smithi106.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-08-30T23:16:05.791 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-30T23:16:05.852 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-30T23:16:05.878 INFO:teuthology.orchestra.run.smithi106.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-08-30T23:16:05.891 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-30T23:16:05.898 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:05.899 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:05.903 INFO:teuthology.orchestra.run.smithi106.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-08-30T23:16:05.929 INFO:teuthology.orchestra.run.smithi106.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 939 kB/s | 188 kB 00:00 2024-08-30T23:16:05.948 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-30T23:16:05.954 INFO:teuthology.orchestra.run.smithi106.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-08-30T23:16:05.971 INFO:teuthology.orchestra.run.smithi106.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-08-30T23:16:05.988 INFO:teuthology.orchestra.run.smithi106.stdout:(17/37): python3-markupsafe-1.1.1-12.el9.x86_64 317 kB/s | 35 kB 00:00 2024-08-30T23:16:06.004 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-30T23:16:06.005 INFO:teuthology.orchestra.run.smithi106.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-08-30T23:16:06.047 INFO:teuthology.orchestra.run.smithi106.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-08-30T23:16:06.072 INFO:teuthology.orchestra.run.smithi106.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 221 kB/s | 22 kB 00:00 2024-08-30T23:16:06.106 INFO:teuthology.orchestra.run.smithi106.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-08-30T23:16:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 458 kB/s | 11 kB 00:00 2024-08-30T23:16:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-08-30T23:16:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-08-30T23:16:06.190 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-30T23:16:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-08-30T23:16:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-08-30T23:16:06.245 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-30T23:16:06.257 INFO:teuthology.orchestra.run.smithi106.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-08-30T23:16:06.282 INFO:teuthology.orchestra.run.smithi106.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-08-30T23:16:06.324 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-30T23:16:06.324 INFO:teuthology.orchestra.run.smithi106.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-08-30T23:16:06.358 INFO:teuthology.orchestra.run.smithi106.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-08-30T23:16:06.383 INFO:teuthology.orchestra.run.smithi106.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-08-30T23:16:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-30T23:16:06.412 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-30T23:16:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-08-30T23:16:06.453 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-30T23:16:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-08-30T23:16:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-08-30T23:16:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-08-30T23:16:06.535 INFO:teuthology.orchestra.run.smithi106.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-30T23:16:06.538 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:06.538 INFO:teuthology.orchestra.run.smithi106.stdout:Total 15 MB/s | 30 MB 00:02 2024-08-30T23:16:06.570 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-30T23:16:06.691 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:06.731 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:06.731 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:06.742 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.n 32/37 2024-08-30T23:16:06.772 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 33/37 2024-08-30T23:16:06.798 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 33/37 2024-08-30T23:16:06.798 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:06.798 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-30T23:16:06.798 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T23:16:06.798 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T23:16:06.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:07.025 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:07.026 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:07.259 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:07.314 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-30T23:16:07.346 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-30T23:16:07.374 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-30T23:16:07.446 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-30T23:16:07.499 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-30T23:16:07.538 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-30T23:16:07.623 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-30T23:16:07.712 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-30T23:16:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-08-30T23:16:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-30T23:16:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-30T23:16:07.850 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-30T23:16:07.883 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-30T23:16:07.931 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-30T23:16:07.967 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-30T23:16:07.993 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-30T23:16:08.040 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-30T23:16:08.073 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-30T23:16:08.148 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-30T23:16:08.223 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 34/37 2024-08-30T23:16:08.251 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 34/37 2024-08-30T23:16:08.251 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:08.251 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-30T23:16:08.251 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T23:16:08.251 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T23:16:08.252 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-30T23:16:08.365 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-30T23:16:08.393 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:08.429 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-30T23:16:08.446 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-30T23:16:08.465 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-30T23:16:08.487 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-30T23:16:08.522 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-30T23:16:08.524 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-30T23:16:08.586 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-30T23:16:08.594 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-30T23:16:08.631 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-30T23:16:08.672 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-30T23:16:08.691 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 35/37 2024-08-30T23:16:08.714 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 35/37 2024-08-30T23:16:08.715 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:08.715 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-30T23:16:08.715 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T23:16:08.715 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T23:16:08.715 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:08.771 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-30T23:16:08.772 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-30T23:16:08.819 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-30T23:16:08.858 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-30T23:16:08.897 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-30T23:16:08.907 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-08-30T23:16:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 36/37 2024-08-30T23:16:08.943 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-30T23:16:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 36/37 2024-08-30T23:16:08.965 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:08.965 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-30T23:16:08.965 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T23:16:08.965 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T23:16:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:08.973 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-30T23:16:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-30T23:16:09.007 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-30T23:16:09.018 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-30T23:16:09.028 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.1.1-242.g244a32d3.el9.x86_64 37/37 2024-08-30T23:16:09.041 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-30T23:16:09.090 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-30T23:16:09.124 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-30T23:16:09.135 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-30T23:16:09.167 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-30T23:16:09.215 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-30T23:16:09.247 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-30T23:16:09.290 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.n 32/37 2024-08-30T23:16:09.312 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 33/37 2024-08-30T23:16:09.320 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-30T23:16:09.338 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 33/37 2024-08-30T23:16:09.338 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:09.338 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-30T23:16:09.338 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T23:16:09.338 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T23:16:09.338 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:09.457 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-30T23:16:09.538 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-30T23:16:09.592 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-30T23:16:09.639 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-30T23:16:09.696 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-30T23:16:09.751 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-30T23:16:09.934 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-30T23:16:09.995 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-30T23:16:10.079 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-30T23:16:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-30T23:16:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-30T23:16:10.324 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-30T23:16:10.489 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.n 32/37 2024-08-30T23:16:10.519 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 33/37 2024-08-30T23:16:10.546 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 33/37 2024-08-30T23:16:10.546 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:10.546 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-30T23:16:10.546 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T23:16:10.546 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-30T23:16:10.546 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:19.1.1-242.g244a32d3.el9.x86_64 37/37 2024-08-30T23:16:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.1.1-242.g244a32d3.el9.x86_64 1/37 2024-08-30T23:16:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 2/37 2024-08-30T23:16:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 3/37 2024-08-30T23:16:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 4/37 2024-08-30T23:16:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 5/37 2024-08-30T23:16:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.n 6/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-30T23:16:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-30T23:16:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-30T23:16:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-30T23:16:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-30T23:16:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-30T23:16:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-30T23:16:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-30T23:16:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-30T23:16:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-30T23:16:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-30T23:16:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-30T23:16:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-30T23:16:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-30T23:16:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-30T23:16:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-30T23:16:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-30T23:16:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-30T23:16:10.768 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 34/37 2024-08-30T23:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 34/37 2024-08-30T23:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-30T23:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T23:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T23:16:10.790 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:11.230 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 35/37 2024-08-30T23:16:11.253 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 35/37 2024-08-30T23:16:11.253 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:11.253 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-30T23:16:11.253 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T23:16:11.253 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T23:16:11.254 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:11.480 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 36/37 2024-08-30T23:16:11.504 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 36/37 2024-08-30T23:16:11.504 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:11.504 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-30T23:16:11.504 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T23:16:11.504 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T23:16:11.504 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:11.567 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.1.1-242.g244a32d3.el9.x86_64 37/37 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.noarch 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:11.785 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-30T23:16:11.786 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-30T23:16:11.787 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-30T23:16:11.788 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-30T23:16:11.788 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-30T23:16:11.788 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-30T23:16:11.788 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:11.788 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 34/37 2024-08-30T23:16:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 34/37 2024-08-30T23:16:12.006 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:12.006 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-30T23:16:12.006 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T23:16:12.006 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-30T23:16:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:12.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-08-30T23:16:12.446 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 35/37 2024-08-30T23:16:12.470 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 35/37 2024-08-30T23:16:12.470 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:12.470 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-30T23:16:12.470 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T23:16:12.470 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-30T23:16:12.470 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:12.705 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 36/37 2024-08-30T23:16:12.714 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:51 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:12.729 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 36/37 2024-08-30T23:16:12.729 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:12.729 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-30T23:16:12.729 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T23:16:12.729 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-30T23:16:12.729 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:12.792 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:19.1.1-242.g244a32d3.el9.x86_64 37/37 2024-08-30T23:16:12.803 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:12.844 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:12.845 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:12.845 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:12.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.1.1-242.g244a32d3.el9.x86_64 37/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.1.1-242.g244a32d3.el9.x86_64 1/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 2/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 3/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 4/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 5/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.n 6/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-30T23:16:13.238 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-30T23:16:13.239 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-08-30T23:16:13.240 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-30T23:16:13.466 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:52 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:13.596 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.1.1-242.g244a32d3.el9 ceph-noarch 765 k 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 765 k 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 771 k 2024-08-30T23:16:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:13.908 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-19.1.1-242.g244a32d3.el9.noarch.rpm 2.4 MB/s | 765 kB 00:00 2024-08-30T23:16:13.909 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:13.909 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.4 MB/s | 765 kB 00:00 2024-08-30T23:16:13.909 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:13.914 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:13.914 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:13.920 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:13.920 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:13.990 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:14.242 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.noarch 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-30T23:16:14.243 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-30T23:16:14.244 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-30T23:16:14.245 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-30T23:16:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-30T23:16:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-30T23:16:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-30T23:16:14.246 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:14.246 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:14.386 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-2:19.1.1-242.g244a32d3.el9.x86_64 37/37 2024-08-30T23:16:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:19.1.1-242.g244a32d3.el9.x86_64 1/37 2024-08-30T23:16:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 2/37 2024-08-30T23:16:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 3/37 2024-08-30T23:16:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 4/37 2024-08-30T23:16:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 5/37 2024-08-30T23:16:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.n 6/37 2024-08-30T23:16:14.388 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-30T23:16:14.388 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-30T23:16:14.388 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-30T23:16:14.388 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-30T23:16:14.388 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-30T23:16:14.388 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-30T23:16:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-30T23:16:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-30T23:16:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-30T23:16:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-30T23:16:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-30T23:16:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-30T23:16:14.390 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-30T23:16:14.390 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-30T23:16:14.390 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-30T23:16:14.390 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-30T23:16:14.390 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-30T23:16:14.390 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-30T23:16:14.391 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-30T23:16:14.392 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-30T23:16:14.392 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-30T23:16:14.392 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-30T23:16:14.392 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-08-30T23:16:14.392 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-30T23:16:14.419 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:14.512 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:14.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-08-30T23:16:14.850 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:15.092 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:15.092 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 2024-08-30T23:16:15.093 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:15.093 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:15.178 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:53 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:15.267 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:15.308 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:15.309 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:15.309 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:15.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-08-30T23:16:15.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:19.1.1-242.g244a32d3.el9.noarch 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: lua-5.4.4-4.el9.x86_64 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-30T23:16:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply-3.11-14.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-30T23:16:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.8-2.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:15.385 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:15.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-base 2024-08-30T23:16:16.101 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:53 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.1.1-242.g244a32d3.el9 ceph-noarch 765 k 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:16.102 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 765 k 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 771 k 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:54 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-242.g244a32d3.el9 ceph 145 k 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:16.103 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:16.104 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:16.104 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 145 k 2024-08-30T23:16:16.104 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 436 k 2024-08-30T23:16:16.104 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:16.243 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-19.1.1-242.g244a32d 735 kB/s | 145 kB 00:00 2024-08-30T23:16:16.243 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:16.244 INFO:teuthology.orchestra.run.smithi136.stdout:Total 730 kB/s | 145 kB 00:00 2024-08-30T23:16:16.244 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:16.251 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:16.251 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:16.308 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:16.308 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:16.313 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.1.1-242.g244a32d3.el9.noarch.rpm 3.0 MB/s | 765 kB 00:00 2024-08-30T23:16:16.313 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:16.314 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.0 MB/s | 765 kB 00:00 2024-08-30T23:16:16.314 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:16.319 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:16.319 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:16.325 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:16.325 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:16.459 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:16.544 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:16.569 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:16.569 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:16.569 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-30T23:16:16.569 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:16.653 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:51 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:16.741 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-base-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:16.782 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:16.783 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:16.783 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:16.849 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:16.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install cephadm 2024-08-30T23:16:16.940 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:17.299 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:17.405 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:52 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:17.478 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:17.478 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:17.478 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:17.478 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:17.479 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:17.479 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:17.535 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:17.536 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:17.536 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-08-30T23:16:17.536 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:17.536 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:17.536 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:19.1.1-242.g244a32d3.el9 ceph-noarch 765 k 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 765 k 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 771 k 2024-08-30T23:16:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:17.542 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 2024-08-30T23:16:17.542 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:17.542 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:17.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-08-30T23:16:17.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-08-30T23:16:17.808 INFO:teuthology.orchestra.run.smithi106.stdout:cephadm-19.1.1-242.g244a32d3.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-08-30T23:16:17.808 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:17.809 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.7 MB/s | 765 kB 00:00 2024-08-30T23:16:17.809 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:17.814 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:17.814 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:17.820 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:17.820 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:17.890 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:18.253 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:57 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:18.325 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:56 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:18.346 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:18.360 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:18.388 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:18.389 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:18.389 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:18.444 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:18.463 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-242.g244a32d3.el9 ceph 145 k 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:18.464 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-08-30T23:16:18.465 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 436 k 2024-08-30T23:16:18.465 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:18.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-08-30T23:16:18.647 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.1.1-242.g244a32d 798 kB/s | 145 kB 00:00 2024-08-30T23:16:18.647 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:18.648 INFO:teuthology.orchestra.run.smithi173.stdout:Total 792 kB/s | 145 kB 00:00 2024-08-30T23:16:18.648 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:18.655 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:18.655 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:18.716 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:18.716 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:18.795 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:18.853 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:18.914 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:18.939 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:18.939 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:18.939 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-30T23:16:18.939 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:19.028 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:58 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:19.042 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:19.1.1-242.g244a32d3.el9.noarch 1/1 2024-08-30T23:16:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:19.043 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:19.1.1-242.g244a32d3.el9.noarch 2024-08-30T23:16:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:19.043 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:19.158 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 896 k 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-30T23:16:19.159 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 976 k 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.9 M 2024-08-30T23:16:19.160 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:19.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-immutable-object-cache 2024-08-30T23:16:19.339 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 596 kB/s | 80 kB 00:00 2024-08-30T23:16:19.581 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-19.1.1-242.g244a32d3.el9.x86_6 2.3 MB/s | 896 kB 00:00 2024-08-30T23:16:19.581 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:19.582 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.3 MB/s | 976 kB 00:00 2024-08-30T23:16:19.593 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:19.615 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:19.615 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:19.687 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:19.688 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:19.804 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:54 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:19.831 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:19.831 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:19.831 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:19.866 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:19.938 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:19.938 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:19.938 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-08-30T23:16:19.938 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:19.938 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-242.g244a32d3.el9 ceph 145 k 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 145 k 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 436 k 2024-08-30T23:16:19.939 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:20.007 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-30T23:16:20.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-08-30T23:16:20.081 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2/2 2024-08-30T23:16:20.131 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-immutable-object-cache-19.1.1-242.g244a32d 758 kB/s | 145 kB 00:00 2024-08-30T23:16:20.131 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:20.132 INFO:teuthology.orchestra.run.smithi106.stdout:Total 752 kB/s | 145 kB 00:00 2024-08-30T23:16:20.132 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:20.139 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:20.139 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:20.197 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:20.197 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:20.342 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:20.432 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:20.457 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:20.457 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-30T23:16:20.457 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:20.585 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:58 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:20.678 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:20.720 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:20.721 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:20.721 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:20.762 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2/2 2024-08-30T23:16:20.762 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/2 2024-08-30T23:16:20.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-08-30T23:16:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-30T23:16:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:21.031 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-30T23:16:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:21.031 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:21.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-08-30T23:16:21.330 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:59 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.e 1/1 2024-08-30T23:16:21.381 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:21.381 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:21.382 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:21.382 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:21.460 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:21.460 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 896 k 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-08-30T23:16:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 976 k 2024-08-30T23:16:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.9 M 2024-08-30T23:16:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:21.609 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 631 kB/s | 80 kB 00:00 2024-08-30T23:16:21.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr 2024-08-30T23:16:21.841 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:00 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:21.859 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-19.1.1-242.g244a32d3.el9.x86_6 2.3 MB/s | 896 kB 00:00 2024-08-30T23:16:21.860 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:21.860 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 976 kB 00:00 2024-08-30T23:16:21.871 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:21.893 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:21.893 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:21.965 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:21.965 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:21.976 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:21.977 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-30T23:16:21.977 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.1.1-242.g244a32d3.el9 ceph 127 k 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-08-30T23:16:21.978 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:22.144 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:22.159 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-19.1.1-242.g244a32d3.el9.x86_64. 701 kB/s | 127 kB 00:00 2024-08-30T23:16:22.159 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:22.160 INFO:teuthology.orchestra.run.smithi136.stdout:Total 696 kB/s | 127 kB 00:00 2024-08-30T23:16:22.160 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:22.165 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:22.165 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:22.167 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:57 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:22.222 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:22.222 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:22.259 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-mgr-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:22.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-30T23:16:22.301 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:22.302 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:22.302 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:22.354 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2/2 2024-08-30T23:16:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:22.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-fuse 2024-08-30T23:16:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:22.873 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:22.922 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:57 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:23.053 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:23.053 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 896 k 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout:Install 2 Packages 2024-08-30T23:16:23.054 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:23.055 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 976 k 2024-08-30T23:16:23.055 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 2.9 M 2024-08-30T23:16:23.055 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:23.083 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2/2 2024-08-30T23:16:23.083 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/2 2024-08-30T23:16:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:23.100 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:23.101 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:23.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-08-30T23:16:23.347 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-30T23:16:23.347 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:23.347 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:23.347 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-30T23:16:23.347 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:23.347 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:23.468 INFO:teuthology.orchestra.run.smithi106.stdout:(1/2): ceph-fuse-19.1.1-242.g244a32d3.el9.x86_6 2.4 MB/s | 896 kB 00:00 2024-08-30T23:16:23.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-08-30T23:16:23.742 INFO:teuthology.orchestra.run.smithi106.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 124 kB/s | 80 kB 00:00 2024-08-30T23:16:23.744 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:23.744 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.4 MB/s | 976 kB 00:00 2024-08-30T23:16:23.756 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:23.779 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:23.779 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:23.852 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:23.853 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:23.898 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:02 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:23.988 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:24.026 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:24.030 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:24.031 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:24.031 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:24.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-08-30T23:16:24.177 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-30T23:16:24.180 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:02 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:24.237 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2/2 2024-08-30T23:16:24.316 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.1.1-242.g244a32d3.el9 ceph 127 k 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:24.317 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:24.318 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 127 k 2024-08-30T23:16:24.318 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-08-30T23:16:24.318 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:24.530 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.1.1-242.g244a32d3.el9.x86_64. 599 kB/s | 127 kB 00:00 2024-08-30T23:16:24.530 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:24.530 INFO:teuthology.orchestra.run.smithi173.stdout:Total 594 kB/s | 127 kB 00:00 2024-08-30T23:16:24.531 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:24.536 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:24.536 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:24.592 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:24.592 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:24.710 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:03 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:24.735 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:24.840 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.1.1-242.g244a32d3.el9 ceph 32 k 2024-08-30T23:16:24.841 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 32 k 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 142 k 2024-08-30T23:16:24.842 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:24.969 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2/2 2024-08-30T23:16:24.970 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/2 2024-08-30T23:16:24.988 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-19.1.1-242.g244a32d3.el9.x86_64 217 kB/s | 32 kB 00:00 2024-08-30T23:16:24.988 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:24.989 INFO:teuthology.orchestra.run.smithi136.stdout:Total 214 kB/s | 32 kB 00:00 2024-08-30T23:16:24.989 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:24.992 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:24.992 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:24.997 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:24.997 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:25.087 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:25.182 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:25.246 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-30T23:16:25.246 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:25.246 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:25.246 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-30T23:16:25.246 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:25.246 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:25.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados-devel 2024-08-30T23:16:25.495 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:25.496 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:25.496 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:25.496 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:25.496 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:25.496 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:25.535 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:25.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-08-30T23:16:25.737 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:25.738 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:25.738 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:25.738 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:25.738 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:25.738 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:25.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-08-30T23:16:26.053 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:01 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:26.190 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:19.1.1-242.g244a32d3.el9 ceph 127 k 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:26.191 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:26.192 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:26.192 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:26.192 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:26.192 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 127 k 2024-08-30T23:16:26.192 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 456 k 2024-08-30T23:16:26.192 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:26.262 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:04 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:26.353 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:26.374 INFO:teuthology.orchestra.run.smithi106.stdout:librados-devel-19.1.1-242.g244a32d3.el9.x86_64. 696 kB/s | 127 kB 00:00 2024-08-30T23:16:26.374 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:26.375 INFO:teuthology.orchestra.run.smithi106.stdout:Total 689 kB/s | 127 kB 00:00 2024-08-30T23:16:26.375 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:26.380 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:26.380 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:26.395 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:26.396 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:26.437 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:26.437 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:26.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-08-30T23:16:26.486 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:05 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:26.574 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:26.576 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:26.617 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:26.618 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:26.618 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:26.676 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:26.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-08-30T23:16:27.042 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:05 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:27.073 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:27.173 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-08-30T23:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:27.174 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.1.1-242.g244a32d3.el9 ceph 32 k 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 32 k 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 142 k 2024-08-30T23:16:27.175 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:27.235 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:06 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:27.324 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:27.326 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.1.1-242.g244a32d3.el9.x86_64 209 kB/s | 32 kB 00:00 2024-08-30T23:16:27.327 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:27.327 INFO:teuthology.orchestra.run.smithi173.stdout:Total 207 kB/s | 32 kB 00:00 2024-08-30T23:16:27.327 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:27.331 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:27.331 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:27.335 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:27.335 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:27.347 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:27.347 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:27.347 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:27.347 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:27.347 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:27.347 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:27.366 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:27.367 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:27.367 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:27.424 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:27.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-08-30T23:16:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:27.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs2 2024-08-30T23:16:27.866 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:27.998 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:06 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:28.078 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:28.078 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:28.079 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:28.087 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:28.129 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:28.130 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:28.130 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:28.138 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:03 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:28.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-08-30T23:16:28.229 INFO:teuthology.orchestra.run.smithi106.stdout:Package libcephfs2-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:28.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-08-30T23:16:28.271 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:28.272 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:28.272 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:28.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs-devel 2024-08-30T23:16:28.762 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:07 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:28.788 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:06 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:28.852 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:28.876 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:28.895 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:28.896 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:28.896 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:28.906 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:03 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:28.919 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:28.920 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:28.920 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:28.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-08-30T23:16:28.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-08-30T23:16:29.035 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:29.035 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:19.1.1-242.g244a32d3.el9 ceph 32 k 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 32 k 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 142 k 2024-08-30T23:16:29.036 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:29.185 INFO:teuthology.orchestra.run.smithi106.stdout:libcephfs-devel-19.1.1-242.g244a32d3.el9.x86_64 213 kB/s | 32 kB 00:00 2024-08-30T23:16:29.186 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:29.186 INFO:teuthology.orchestra.run.smithi106.stdout:Total 211 kB/s | 32 kB 00:00 2024-08-30T23:16:29.186 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:29.189 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:29.190 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:29.193 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:29.194 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:29.378 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:29.530 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:08 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:29.531 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:29.619 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:29.620 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:29.661 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:29.661 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:29.662 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:29.662 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:29.662 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:29.662 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:29.724 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:29.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-08-30T23:16:29.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-08-30T23:16:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:29.945 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:29.945 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:30.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados2 2024-08-30T23:16:30.282 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:08 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:30.296 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:09 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:30.371 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:30.385 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:30.413 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:30.414 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:30.414 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:30.427 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:30.428 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-08-30T23:16:30.428 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:30.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-08-30T23:16:30.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-08-30T23:16:30.669 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:05 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:30.759 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:30.800 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:30.802 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:30.802 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:30.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librbd1 2024-08-30T23:16:31.031 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:09 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:31.073 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:10 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:31.121 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:31.162 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:31.163 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:31.163 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:31.203 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:31.204 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:31.204 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-30T23:16:31.204 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:31.204 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:31.204 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 85 k 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 85 k 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 231 k 2024-08-30T23:16:31.205 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:31.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-08-30T23:16:31.391 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-19.1.1-242.g244a32d3.el9.x86_64.rpm 458 kB/s | 85 kB 00:00 2024-08-30T23:16:31.392 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:31.392 INFO:teuthology.orchestra.run.smithi136.stdout:Total 455 kB/s | 85 kB 00:00 2024-08-30T23:16:31.392 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:31.399 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:31.399 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:31.422 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:06 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:31.457 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:31.457 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:31.512 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:31.553 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:31.554 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:31.554 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:31.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rados 2024-08-30T23:16:31.685 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:31.796 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:09 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:31.892 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:31.937 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:31.938 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:31.938 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:32.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-08-30T23:16:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:32.168 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:07 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:32.257 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rados-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:32.300 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:32.301 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:32.301 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:32.303 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:32.303 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:32.303 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:32.303 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:32.303 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:32.303 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:32.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rgw 2024-08-30T23:16:32.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-08-30T23:16:32.654 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:10 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:32.745 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:32.786 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:32.787 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-08-30T23:16:32.788 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:32.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-08-30T23:16:32.920 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:07 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:33.008 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rgw-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:33.049 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:33.050 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:33.050 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:33.113 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:12 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:33.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-cephfs 2024-08-30T23:16:33.247 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.1 M 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:33.248 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:33.249 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:33.249 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.1 M 2024-08-30T23:16:33.249 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 13 M 2024-08-30T23:16:33.249 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:33.412 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:11 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:33.543 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 85 k 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:33.544 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:33.545 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-08-30T23:16:33.545 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-08-30T23:16:33.545 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:33.645 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-19.1.1-242.g244a32d3.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2024-08-30T23:16:33.645 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:33.645 INFO:teuthology.orchestra.run.smithi136.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-08-30T23:16:33.646 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:33.654 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:33.654 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:33.685 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:08 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:33.729 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:33.729 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:33.730 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.1.1-242.g244a32d3.el9.x86_64.rpm 460 kB/s | 85 kB 00:00 2024-08-30T23:16:33.730 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:33.731 INFO:teuthology.orchestra.run.smithi173.stdout:Total 457 kB/s | 85 kB 00:00 2024-08-30T23:16:33.731 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:33.738 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:33.738 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:33.778 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-cephfs-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:33.794 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:33.795 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:33.823 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:33.824 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:33.824 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:33.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rbd 2024-08-30T23:16:33.928 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:34.031 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:34.161 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:34.245 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:34.245 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-30T23:16:34.245 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T23:16:34.245 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T23:16:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:34.434 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:34.481 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:09 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:34.573 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rbd-2:19.1.1-242.g244a32d3.el9.x86_64 is already installed. 2024-08-30T23:16:34.617 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:34.618 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-08-30T23:16:34.618 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:34.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-fuse 2024-08-30T23:16:34.695 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:34.696 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:34.696 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:34.696 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:34.696 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:34.696 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:34.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-08-30T23:16:35.253 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:10 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:35.344 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:35.345 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:35.345 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:35.345 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:35.345 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:35.345 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:35.383 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:19.1.1-242.g244a32d3.el9 ceph 85 k 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:35.384 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:35.385 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:35.385 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:35.385 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 85 k 2024-08-30T23:16:35.385 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 231 k 2024-08-30T23:16:35.385 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:35.471 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:13 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:35.572 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-fuse-19.1.1-242.g244a32d3.el9.x86_64.rpm 454 kB/s | 85 kB 00:00 2024-08-30T23:16:35.573 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:35.573 INFO:teuthology.orchestra.run.smithi106.stdout:Total 450 kB/s | 85 kB 00:00 2024-08-30T23:16:35.573 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:35.580 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:35.580 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:35.607 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:35.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-08-30T23:16:35.608 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.1 M 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 13 M 2024-08-30T23:16:35.609 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:35.637 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:35.637 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:35.777 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:35.866 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:36.017 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.1.1-242.g244a32d3.el9.x86_64.rpm 7.6 MB/s | 3.1 MB 00:00 2024-08-30T23:16:36.017 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:36.018 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.6 MB/s | 3.1 MB 00:00 2024-08-30T23:16:36.018 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:36.027 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:36.027 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:36.101 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:36.101 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:36.169 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:15 ago on Fri 30 Aug 2024 11:15:21 PM UTC. 2024-08-30T23:16:36.265 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:36.305 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-08-30T23:16:36.305 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 170 k 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 170 k 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 494 k 2024-08-30T23:16:36.306 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-08-30T23:16:36.507 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-19.1.1-242.g244a32d3.el9.x86_64.rpm 848 kB/s | 170 kB 00:00 2024-08-30T23:16:36.508 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:36.508 INFO:teuthology.orchestra.run.smithi136.stdout:Total 843 kB/s | 170 kB 00:00 2024-08-30T23:16:36.508 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-08-30T23:16:36.511 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:36.511 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:36.511 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:36.511 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:36.511 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:36.511 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:36.516 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-08-30T23:16:36.516 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-08-30T23:16:36.526 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:36.574 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-08-30T23:16:36.575 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-08-30T23:16:36.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:36.634 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:36.635 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:36.635 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-30T23:16:36.635 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T23:16:36.635 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T23:16:36.635 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-08-30T23:16:36.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-mirror 2024-08-30T23:16:36.811 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:37.237 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:37.298 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:12 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:37.429 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:37.429 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:19.1.1-242.g244a32d3.el9 ceph 3.1 M 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:37.430 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:37.431 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 3.1 M 2024-08-30T23:16:37.431 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 13 M 2024-08-30T23:16:37.431 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:37.487 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:37.488 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:37.488 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-08-30T23:16:37.488 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:37.488 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:16:37.488 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-08-30T23:16:37.711 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:37.711 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:37.711 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:37.711 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:37.712 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:37.712 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:37.754 DEBUG:teuthology.parallel:result is None 2024-08-30T23:16:37.825 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-mirror-19.1.1-242.g244a32d3.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-08-30T23:16:37.826 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:37.827 INFO:teuthology.orchestra.run.smithi106.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-08-30T23:16:37.827 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:37.835 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:37.836 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:37.910 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:37.910 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:37.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-08-30T23:16:38.336 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:38.393 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:38.418 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:38.418 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-30T23:16:38.418 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-30T23:16:38.418 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T23:16:38.418 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-30T23:16:38.418 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:38.506 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:16 ago on Fri 30 Aug 2024 11:15:22 PM UTC. 2024-08-30T23:16:38.641 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-08-30T23:16:38.642 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:38.642 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-08-30T23:16:38.642 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 170 k 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 170 k 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 494 k 2024-08-30T23:16:38.643 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-08-30T23:16:38.878 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.1.1-242.g244a32d3.el9.x86_64.rpm 726 kB/s | 170 kB 00:00 2024-08-30T23:16:38.879 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:38.879 INFO:teuthology.orchestra.run.smithi173.stdout:Total 721 kB/s | 170 kB 00:00 2024-08-30T23:16:38.879 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-08-30T23:16:38.887 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-08-30T23:16:38.887 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-08-30T23:16:38.945 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-08-30T23:16:38.945 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-08-30T23:16:39.098 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-08-30T23:16:39.189 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:39.509 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:39.510 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:39.510 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:39.510 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:39.510 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:39.510 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:39.595 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:39.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-nbd 2024-08-30T23:16:39.861 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:39.862 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:39.862 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-08-30T23:16:39.862 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:39.862 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:16:39.862 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-08-30T23:16:40.089 DEBUG:teuthology.parallel:result is None 2024-08-30T23:16:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:15 ago on Fri 30 Aug 2024 11:15:25 PM UTC. 2024-08-30T23:16:40.410 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:19.1.1-242.g244a32d3.el9 ceph 170 k 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-08-30T23:16:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-08-30T23:16:40.412 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:40.412 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 170 k 2024-08-30T23:16:40.412 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 494 k 2024-08-30T23:16:40.412 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-08-30T23:16:40.628 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-nbd-19.1.1-242.g244a32d3.el9.x86_64.rpm 789 kB/s | 170 kB 00:00 2024-08-30T23:16:40.629 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:16:40.629 INFO:teuthology.orchestra.run.smithi106.stdout:Total 783 kB/s | 170 kB 00:00 2024-08-30T23:16:40.629 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-08-30T23:16:40.637 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-08-30T23:16:40.637 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-08-30T23:16:40.693 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-08-30T23:16:40.694 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-08-30T23:16:40.835 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-08-30T23:16:40.921 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:41.301 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:41.594 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 1/1 2024-08-30T23:16:41.594 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:41.594 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-08-30T23:16:41.594 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:19.1.1-242.g244a32d3.el9.x86_64 2024-08-30T23:16:41.594 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:16:41.594 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-08-30T23:16:41.823 DEBUG:teuthology.parallel:result is None 2024-08-30T23:16:41.824 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:16:41.824 INFO:teuthology.packaging:ref: None 2024-08-30T23:16:41.824 INFO:teuthology.packaging:tag: None 2024-08-30T23:16:41.824 INFO:teuthology.packaging:branch: squid 2024-08-30T23:16:41.824 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:16:41.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-30T23:16:42.042 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-30T23:16:42.063 INFO:teuthology.orchestra.run.smithi106.stdout:19.1.1-242.g244a32d3.el9 2024-08-30T23:16:42.063 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-242.g244a32d3.el9 2024-08-30T23:16:42.063 INFO:teuthology.task.install:The correct ceph version 19.1.1-242.g244a32d3 is installed. 2024-08-30T23:16:42.065 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:16:42.065 INFO:teuthology.packaging:ref: None 2024-08-30T23:16:42.065 INFO:teuthology.packaging:tag: None 2024-08-30T23:16:42.065 INFO:teuthology.packaging:branch: squid 2024-08-30T23:16:42.065 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:16:42.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-30T23:16:42.267 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-30T23:16:42.292 INFO:teuthology.orchestra.run.smithi136.stdout:19.1.1-242.g244a32d3.el9 2024-08-30T23:16:42.292 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-242.g244a32d3.el9 2024-08-30T23:16:42.292 INFO:teuthology.task.install:The correct ceph version 19.1.1-242.g244a32d3 is installed. 2024-08-30T23:16:42.294 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-30T23:16:42.295 INFO:teuthology.packaging:ref: None 2024-08-30T23:16:42.295 INFO:teuthology.packaging:tag: None 2024-08-30T23:16:42.295 INFO:teuthology.packaging:branch: squid 2024-08-30T23:16:42.295 INFO:teuthology.packaging:sha1: 24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:16:42.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-08-30T23:16:42.498 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-30T23:16:42.524 INFO:teuthology.orchestra.run.smithi173.stdout:19.1.1-242.g244a32d3.el9 2024-08-30T23:16:42.524 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-242.g244a32d3.el9 2024-08-30T23:16:42.524 INFO:teuthology.task.install:The correct ceph version 19.1.1-242.g244a32d3 is installed. 2024-08-30T23:16:42.525 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-30T23:16:42.526 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:16:42.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-30T23:16:42.553 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:16:42.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-30T23:16:42.583 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:16:42.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-30T23:16:42.613 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-30T23:16:42.613 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:16:42.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-08-30T23:16:42.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-30T23:16:42.703 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:16:42.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-08-30T23:16:42.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-30T23:16:42.801 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:16:42.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-08-30T23:16:42.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-30T23:16:42.902 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-30T23:16:42.903 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:16:42.903 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-30T23:16:42.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-30T23:16:42.991 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:16:42.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-30T23:16:43.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-30T23:16:43.089 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:16:43.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-30T23:16:43.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-30T23:16:43.184 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-30T23:16:43.184 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:16:43.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-08-30T23:16:43.209 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-30T23:16:43.273 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:16:43.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-08-30T23:16:43.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-30T23:16:43.370 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:16:43.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-08-30T23:16:43.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-30T23:16:43.466 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-30T23:16:43.552 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '24432b660539783e58f890fcff372342c034f6ec'} 2024-08-30T23:16:43.552 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:16:43.553 INFO:tasks.cephadm:Cluster fsid is eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:16:43.553 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-30T23:16:43.554 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.106', 'mon.b': '172.21.15.136', 'mon.c': '172.21.15.173'} 2024-08-30T23:16:43.554 INFO:tasks.cephadm:First mon is mon.a on smithi106 2024-08-30T23:16:43.554 INFO:tasks.cephadm:First mgr is y 2024-08-30T23:16:43.554 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-30T23:16:43.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-08-30T23:16:43.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-08-30T23:16:43.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-08-30T23:16:43.639 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-08-30T23:16:43.639 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:16:43.797 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '24432b660539783e58f890fcff372342c034f6ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82659/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4625-g24432b66', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4625.g24432b66'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-30 12:50:33.948506', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-bharath1-testing-2024-08-30-1604', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-08-30T23:16:43.945 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-bharath1-testing-2024-08-30-1604, sha1 24432b660539783e58f890fcff372342c034f6ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:16:43.947 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/x86_64/flavors/default/cephadm 2024-08-30T23:16:43.948 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/x86_64/flavors/default/cephadm 2024-08-30T23:16:43.948 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:44.319 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r--. 1 ubuntu ubuntu 802580 Aug 30 23:16 /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:44.319 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:44.701 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 802580 Aug 30 23:16 /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:44.701 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-bharath1-testing-2024-08-30-1604/24432b660539783e58f890fcff372342c034f6ec/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:51.201 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 802580 Aug 30 23:16 /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:51.202 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:51.222 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:51.241 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-30T23:16:51.273 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec on all hosts... 2024-08-30T23:16:51.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec pull 2024-08-30T23:16:51.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec pull 2024-08-30T23:16:51.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec pull 2024-08-30T23:16:52.873 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec... 2024-08-30T23:16:52.879 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec... 2024-08-30T23:16:52.903 INFO:teuthology.orchestra.run.smithi106.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec... 2024-08-30T23:17:25.614 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:17:25.614 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.3.0-4625-g24432b66 (24432b660539783e58f890fcff372342c034f6ec) squid (dev)", 2024-08-30T23:17:25.614 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60", 2024-08-30T23:17:25.614 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-08-30T23:17:25.614 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8" 2024-08-30T23:17:25.615 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:17:25.615 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:17:45.390 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:17:45.391 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.3.0-4625-g24432b66 (24432b660539783e58f890fcff372342c034f6ec) squid (dev)", 2024-08-30T23:17:45.391 INFO:teuthology.orchestra.run.smithi106.stdout: "image_id": "fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60", 2024-08-30T23:17:45.391 INFO:teuthology.orchestra.run.smithi106.stdout: "repo_digests": [ 2024-08-30T23:17:45.391 INFO:teuthology.orchestra.run.smithi106.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8" 2024-08-30T23:17:45.391 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:17:45.391 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.3.0-4625-g24432b66 (24432b660539783e58f890fcff372342c034f6ec) squid (dev)", 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60", 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8" 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:17:47.586 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:17:47.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-08-30T23:17:47.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-08-30T23:17:47.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-08-30T23:17:47.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-08-30T23:17:47.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-08-30T23:17:47.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-08-30T23:17:47.785 INFO:tasks.cephadm:Writing seed config... 2024-08-30T23:17:47.786 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-08-30T23:17:47.787 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-30T23:17:47.788 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-30T23:17:47.789 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-08-30T23:17:47.789 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-08-30T23:17:47.789 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-08-30T23:17:47.789 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-08-30T23:17:47.792 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-08-30T23:17:47.793 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-08-30T23:17:47.793 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-08-30T23:17:47.793 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-08-30T23:17:47.793 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-30T23:17:47.793 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-08-30T23:17:47.794 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:17:47.794 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-30T23:17:47.811 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eb936aa2-6725-11ef-bcd4-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-08-30T23:17:47.812 DEBUG:teuthology.orchestra.run.smithi106:mon.a> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a.service 2024-08-30T23:17:47.854 DEBUG:teuthology.orchestra.run.smithi106:mgr.y> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y.service 2024-08-30T23:17:47.897 INFO:tasks.cephadm:Bootstrapping... 2024-08-30T23:17:47.897 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec -v bootstrap --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.106 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-30T23:17:48.100 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-08-30T23:17:48.100 INFO:teuthology.orchestra.run.smithi106.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec', '-v', 'bootstrap', '--fsid', 'eb936aa2-6725-11ef-bcd4-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.106', '--skip-admin-label'] 2024-08-30T23:17:48.100 INFO:teuthology.orchestra.run.smithi106.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-30T23:17:48.100 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying podman|docker is present... 2024-08-30T23:17:48.124 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 5.2.2 2024-08-30T23:17:48.125 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying lvm2 is present... 2024-08-30T23:17:48.125 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying time synchronization is in place... 2024-08-30T23:17:48.132 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-30T23:17:48.132 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-30T23:17:48.139 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-30T23:17:48.139 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-08-30T23:17:48.146 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout enabled 2024-08-30T23:17:48.153 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout active 2024-08-30T23:17:48.153 INFO:teuthology.orchestra.run.smithi106.stdout:Unit chronyd.service is enabled and running 2024-08-30T23:17:48.153 INFO:teuthology.orchestra.run.smithi106.stdout:Repeating the final host check... 2024-08-30T23:17:48.175 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout 5.2.2 2024-08-30T23:17:48.175 INFO:teuthology.orchestra.run.smithi106.stdout:podman (/bin/podman) version 5.2.2 is present 2024-08-30T23:17:48.175 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl is present 2024-08-30T23:17:48.175 INFO:teuthology.orchestra.run.smithi106.stdout:lvcreate is present 2024-08-30T23:17:48.182 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-30T23:17:48.182 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-30T23:17:48.188 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-30T23:17:48.188 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-08-30T23:17:48.195 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout enabled 2024-08-30T23:17:48.201 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout active 2024-08-30T23:17:48.201 INFO:teuthology.orchestra.run.smithi106.stdout:Unit chronyd.service is enabled and running 2024-08-30T23:17:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:Host looks OK 2024-08-30T23:17:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:Cluster fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:17:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:Acquiring lock 139993998826560 on /run/cephadm/eb936aa2-6725-11ef-bcd4-c7b262605968.lock 2024-08-30T23:17:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:Lock 139993998826560 acquired on /run/cephadm/eb936aa2-6725-11ef-bcd4-c7b262605968.lock 2024-08-30T23:17:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying IP 172.21.15.106 port 3300 ... 2024-08-30T23:17:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying IP 172.21.15.106 port 6789 ... 2024-08-30T23:17:48.203 INFO:teuthology.orchestra.run.smithi106.stdout:Base mon IP(s) is [172.21.15.106:3300, 172.21.15.106:6789], mon addrv is [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-08-30T23:17:48.206 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.106 metric 100 2024-08-30T23:17:48.206 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.106 metric 100 2024-08-30T23:17:48.208 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-30T23:17:48.209 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-08-30T23:17:48.209 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-08-30T23:17:48.211 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-30T23:17:48.211 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-30T23:17:48.211 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-30T23:17:48.211 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-08-30T23:17:48.212 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce6d/64 scope link noprefixroute 2024-08-30T23:17:48.212 INFO:teuthology.orchestra.run.smithi106.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-30T23:17:48.212 INFO:teuthology.orchestra.run.smithi106.stdout:Mon IP `172.21.15.106` is in CIDR network `172.21.0.0/20` 2024-08-30T23:17:48.212 INFO:teuthology.orchestra.run.smithi106.stdout:Mon IP `172.21.15.106` is in CIDR network `172.21.0.0/20` 2024-08-30T23:17:48.212 INFO:teuthology.orchestra.run.smithi106.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-30T23:17:48.213 INFO:teuthology.orchestra.run.smithi106.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-30T23:17:48.213 INFO:teuthology.orchestra.run.smithi106.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec... 2024-08-30T23:17:48.618 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stdout fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 2024-08-30T23:17:48.618 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec... 2024-08-30T23:17:48.618 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Getting image source signatures 2024-08-30T23:17:48.618 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying blob sha256:7ba2cc7b91768c5345ac65baebb7871cf8e706bc4fee598be049ce3b65839dd2 2024-08-30T23:17:48.618 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-30T23:17:48.618 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Copying config sha256:fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 2024-08-30T23:17:48.619 INFO:teuthology.orchestra.run.smithi106.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-30T23:17:49.449 INFO:teuthology.orchestra.run.smithi106.stdout:ceph: stdout ceph version 19.3.0-4625-g24432b66 (24432b660539783e58f890fcff372342c034f6ec) squid (dev) 2024-08-30T23:17:49.449 INFO:teuthology.orchestra.run.smithi106.stdout:Ceph version: ceph version 19.3.0-4625-g24432b66 (24432b660539783e58f890fcff372342c034f6ec) squid (dev) 2024-08-30T23:17:49.449 INFO:teuthology.orchestra.run.smithi106.stdout:Extracting ceph user uid/gid from container image... 2024-08-30T23:17:50.428 INFO:teuthology.orchestra.run.smithi106.stdout:stat: stdout 167 167 2024-08-30T23:17:50.428 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial keys... 2024-08-30T23:17:51.300 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQAeU9JmXeZpKxAAMK9coXIpOiO6kemKo0N/XQ== 2024-08-30T23:17:52.130 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQAfU9JmoKTOJBAAi7l77aQhZi46MDG1PDFhjQ== 2024-08-30T23:17:53.037 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph-authtool: stdout AQAgU9JmAoPEGhAAa6Bolo0JIFlr6SnCfY/x6A== 2024-08-30T23:17:53.038 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial monmap... 2024-08-30T23:17:53.977 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:monmaptool for a [v2:172.21.15.106:3300,v1:172.21.15.106:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:setting min_mon_release = quincy 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: set fsid to eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:17:53.978 INFO:teuthology.orchestra.run.smithi106.stdout:Creating mon... 2024-08-30T23:17:58.525 INFO:teuthology.orchestra.run.smithi106.stdout:create mon.a on 2024-08-30T23:17:59.155 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-30T23:17:59.492 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-30T23:18:00.049 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968.target. 2024-08-30T23:18:00.049 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968.target. 2024-08-30T23:18:00.921 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a 2024-08-30T23:18:00.921 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a.service: Unit ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a.service not loaded. 2024-08-30T23:18:01.279 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968.target.wants/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a.service → /etc/systemd/system/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@.service. 2024-08-30T23:18:01.850 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:01 smithi106 podman[16221]: 2024-08-30 23:18:01.679583704 +0000 UTC m=+0.015486963 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:18:01.850 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:01 smithi106 podman[16221]: 2024-08-30 23:18:01.85119972 +0000 UTC m=+0.187102966 container create 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-08-30T23:18:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:02 smithi106 podman[16221]: 2024-08-30 23:18:02.045102304 +0000 UTC m=+0.381005555 container init 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2) 2024-08-30T23:18:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:02 smithi106 podman[16221]: 2024-08-30 23:18:02.048181802 +0000 UTC m=+0.384085050 container start 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-08-30T23:18:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:02 smithi106 ceph-mon[16240]: mkfs eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:02 smithi106 ceph-mon[16240]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-30T23:18:02.142 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-30T23:18:02.142 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-08-30T23:18:02.149 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-30T23:18:02.149 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-08-30T23:18:02.149 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-08-30T23:18:02.149 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to enable service . firewalld.service is not available 2024-08-30T23:18:02.149 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mon to start... 2024-08-30T23:18:02.149 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mon... 2024-08-30T23:18:02.456 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:02 smithi106 bash[16221]: 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c 2024-08-30T23:18:02.456 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:02 smithi106 systemd[1]: Started Ceph mon.a for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout cluster: 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout id: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout services: 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.523902s) 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:03.147 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout data: 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout pgs: 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:mon is available 2024-08-30T23:18:03.148 INFO:teuthology.orchestra.run.smithi106.stdout:Assimilating anything we can from ceph.conf... 2024-08-30T23:18:03.260 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-30T23:18:03.260 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: monmap epoch 1 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: last_changed 2024-08-30T23:17:53.378775+0000 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: min_mon_release 19 (squid) 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: election_strategy: 1 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: fsmap 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: osdmap e1: 0 total, 0 up, 0 in 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: mgrmap e1: no daemons active 2024-08-30T23:18:03.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:03 smithi106 ceph-mon[16240]: from='client.? 172.21.15.106:0/2369249072' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-08-30T23:18:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:04 smithi106 ceph-mon[16240]: from='client.? 172.21.15.106:0/126711804' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-08-30T23:18:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:04 smithi106 ceph-mon[16240]: from='client.? 172.21.15.106:0/126711804' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-30T23:18:04.378 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [global] 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout fsid = eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [osd] 2024-08-30T23:18:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-30T23:18:04.380 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-30T23:18:04.380 INFO:teuthology.orchestra.run.smithi106.stdout:Generating new minimal ceph.conf... 2024-08-30T23:18:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:05 smithi106 ceph-mon[16240]: from='client.? 172.21.15.106:0/207914395' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:18:05.368 INFO:teuthology.orchestra.run.smithi106.stdout:Restarting the monitor... 2024-08-30T23:18:05.634 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:05 smithi106 systemd[1]: Stopping Ceph mon.a for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:18:06.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:05 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16236]: 2024-08-30T23:18:05.633+0000 7f02744cd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-30T23:18:06.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:05 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16236]: 2024-08-30T23:18:05.633+0000 7f02744cd640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-30T23:18:06.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:05 smithi106 podman[16481]: 2024-08-30 23:18:05.79768592 +0000 UTC m=+0.275641455 container died 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-08-30T23:18:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:06 smithi106 podman[16481]: 2024-08-30 23:18:06.141799578 +0000 UTC m=+0.619755111 container cleanup 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-30T23:18:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:06 smithi106 bash[16481]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a 2024-08-30T23:18:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:06 smithi106 podman[16494]: 2024-08-30 23:18:06.818076681 +0000 UTC m=+1.018302648 container remove 35430979dc26f750941b55cbc2066a3efa486d3322d35cd02b294b59f8c3e04c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:18:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:06 smithi106 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-30T23:18:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:06 smithi106 systemd[1]: Stopped Ceph mon.a for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:18:07.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:06 smithi106 systemd[1]: Starting Ceph mon.a for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:18:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:07 smithi106 podman[16583]: 2024-08-30 23:18:07.05781751 +0000 UTC m=+0.015587749 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:18:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:08 smithi106 podman[16583]: 2024-08-30 23:18:08.331346654 +0000 UTC m=+1.289116880 container create dc9bafd601de2e62d8690b245c3158e7348976e75b83dd7f4346ef1c9682602e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True) 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 podman[16583]: 2024-08-30 23:18:10.349567496 +0000 UTC m=+3.307337728 container init dc9bafd601de2e62d8690b245c3158e7348976e75b83dd7f4346ef1c9682602e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 podman[16583]: 2024-08-30 23:18:10.353514005 +0000 UTC m=+3.311284236 container start dc9bafd601de2e62d8690b245c3158e7348976e75b83dd7f4346ef1c9682602e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: set uid:gid to 167:167 (ceph:ceph) 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: ceph version 19.3.0-4625-g24432b66 (24432b660539783e58f890fcff372342c034f6ec) squid (dev), process ceph-mon, pid 2 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: pidfile_write: ignore empty --pid-file 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: load: jerasure load: lrc 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: RocksDB version: 7.9.2 2024-08-30T23:18:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Git sha 0 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Compile date 2024-08-30 11:46:54 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: DB SUMMARY 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: DB Session ID: N5Q1DGE55IMVJ58NAX5W 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: CURRENT file: CURRENT 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: IDENTITY file: IDENTITY 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 102306 ; 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.error_if_exists: 0 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.create_if_missing: 0 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.paranoid_checks: 1 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-30T23:18:10.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-30T23:18:10.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.env: 0x563268e38160 2024-08-30T23:18:10.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.fs: PosixFileSystem 2024-08-30T23:18:10.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.info_log: 0x56326b1f9880 2024-08-30T23:18:10.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_file_opening_threads: 16 2024-08-30T23:18:10.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.statistics: (nil) 2024-08-30T23:18:10.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.use_fsync: 0 2024-08-30T23:18:10.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_log_file_size: 0 2024-08-30T23:18:10.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.keep_log_file_num: 1000 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.recycle_log_file_num: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.allow_fallocate: 1 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.allow_mmap_reads: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.allow_mmap_writes: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.use_direct_reads: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.create_missing_column_families: 0 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.db_log_dir: 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.wal_dir: 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-30T23:18:10.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.advise_random_on_open: 1 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.db_write_buffer_size: 0 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.write_buffer_manager: 0x56326b1fce60 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.rate_limiter: (nil) 2024-08-30T23:18:10.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.wal_recovery_mode: 2 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.enable_thread_tracking: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.enable_pipelined_write: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.unordered_write: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.row_cache: None 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.wal_filter: None 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.allow_ingest_behind: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.two_write_queues: 0 2024-08-30T23:18:10.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.manual_wal_flush: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.wal_compression: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.atomic_flush: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.log_readahead_size: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.best_efforts_recovery: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.allow_data_in_errors: 0 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.db_host_id: __hostname__ 2024-08-30T23:18:10.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_background_jobs: 2 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_background_compactions: -1 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_subcompactions: 1 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_total_wal_size: 0 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_open_files: -1 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bytes_per_sync: 0 2024-08-30T23:18:10.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_readahead_size: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_background_flushes: -1 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Compression algorithms supported: 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kZSTD supported: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kXpressCompression supported: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kBZip2Compression supported: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kLZ4Compression supported: 1 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kZlibCompression supported: 1 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kLZ4HCCompression supported: 1 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: kSnappyCompression supported: 1 2024-08-30T23:18:10.617 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.merge_operator: 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_filter: None 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_filter_factory: None 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.sst_partitioner_factory: None 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56326b1f94e0) 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout: cache_index_and_filter_blocks: 1 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-30T23:18:10.618 INFO:journalctl@ceph.mon.a.smithi106.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: pin_top_level_index_and_filter: 1 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_type: 0 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: data_block_index_type: 0 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_shortening: 1 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: checksum: 4 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: no_block_cache: 0 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache: 0x56326b1ec850 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_name: BinnedLRUCache 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_options: 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: capacity : 536870912 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: num_shard_bits : 4 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: strict_capacity_limit : 0 2024-08-30T23:18:10.619 INFO:journalctl@ceph.mon.a.smithi106.stdout: high_pri_pool_ratio: 0.000 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_cache_compressed: (nil) 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: persistent_cache: (nil) 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_size: 4096 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_size_deviation: 10 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_restart_interval: 16 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: index_block_restart_interval: 1 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: metadata_block_size: 4096 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: partition_filters: 0 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: use_delta_encoding: 1 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: filter_policy: bloomfilter 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: whole_key_filtering: 1 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: verify_compression: 0 2024-08-30T23:18:10.620 INFO:journalctl@ceph.mon.a.smithi106.stdout: read_amp_bytes_per_bit: 0 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: format_version: 5 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: enable_index_compression: 1 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: block_align: 0 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: max_auto_readahead_size: 262144 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: prepopulate_block_cache: 0 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: initial_auto_readahead_size: 8192 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout: num_file_reads_for_auto_readahead: 2 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.write_buffer_size: 33554432 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_write_buffer_number: 2 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression: NoCompression 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression: Disabled 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.prefix_extractor: nullptr 2024-08-30T23:18:10.621 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.num_levels: 7 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-30T23:18:10.622 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.level: 32767 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.strategy: 0 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.enabled: false 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-30T23:18:10.623 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.target_file_size_base: 67108864 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-30T23:18:10.624 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.arena_block_size: 1048576 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.disable_auto_compactions: 0 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-30T23:18:10.625 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.inplace_update_support: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.bloom_locality: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.max_successive_merges: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-30T23:18:10.626 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.paranoid_file_checks: 0 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.force_consistency_checks: 1 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.report_bg_io_stats: 0 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.ttl: 2592000 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.enable_blob_files: false 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.min_blob_size: 0 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.blob_file_size: 268435456 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-30T23:18:10.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.blob_file_starting_level: 0 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0855ba01-da94-410b-9d62-6ebdce3a78a7 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725059890384784, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725059890385936, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 97553, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 95536, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13537, "raw_average_key_size": 50, "raw_value_size": 88194, "raw_average_value_size": 327, "num_data_blocks": 12, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725059890, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0855ba01-da94-410b-9d62-6ebdce3a78a7", "db_session_id": "N5Q1DGE55IMVJ58NAX5W", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725059890386053, "job": 1, "event": "recovery_finished"} 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56326b21ce00 2024-08-30T23:18:10.628 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: DB pointer 0x56326b31c000 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** DB Stats ** 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** Compaction Stats [default] ** 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-30T23:18:10.629 INFO:journalctl@ceph.mon.a.smithi106.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: L0 2/0 97.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 121.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: Sum 2/0 97.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 121.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 121.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** Compaction Stats [default] ** 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 121.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-30T23:18:10.630 INFO:journalctl@ceph.mon.a.smithi106.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: Cumulative compaction: 0.00 GB write, 25.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: Interval compaction: 0.00 GB write, 25.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: Block cache BinnedLRUCache@0x56326b1ec850#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: starting mon.a rank 0 at public addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] at bind addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???) e1 preinit fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:10.631 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).mds e1 new map 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).mds e1 print_map 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: e1 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: btime 2024-08-30T23:18:02:086399+0000 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: legacy client fscid: -1 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout: No filesystems configured 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-30T23:18:10.632 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: monmap epoch 1 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: last_changed 2024-08-30T23:17:53.378775+0000 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: min_mon_release 19 (squid) 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: election_strategy: 1 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: fsmap 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: osdmap e1: 0 total, 0 up, 0 in 2024-08-30T23:18:10.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 ceph-mon[16603]: mgrmap e1: no daemons active 2024-08-30T23:18:10.634 INFO:teuthology.orchestra.run.smithi106.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-30T23:18:10.946 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 bash[16583]: dc9bafd601de2e62d8690b245c3158e7348976e75b83dd7f4346ef1c9682602e 2024-08-30T23:18:10.946 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:10 smithi106 systemd[1]: Started Ceph mon.a for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:18:11.661 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-30T23:18:11.682 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-30T23:18:11.682 INFO:teuthology.orchestra.run.smithi106.stdout:Creating mgr... 2024-08-30T23:18:11.683 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-30T23:18:11.683 INFO:teuthology.orchestra.run.smithi106.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-30T23:18:11.945 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y 2024-08-30T23:18:11.945 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y.service: Unit ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y.service not loaded. 2024-08-30T23:18:12.170 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968.target.wants/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@.service. 2024-08-30T23:18:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1995378273' entity='client.admin' 2024-08-30T23:18:12.357 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 systemd[1]: Starting Ceph mgr.y for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:18:12.762 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 podman[16814]: 2024-08-30 23:18:12.366893128 +0000 UTC m=+0.015607449 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec 2024-08-30T23:18:12.762 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 podman[16814]: 2024-08-30 23:18:12.510682844 +0000 UTC m=+0.159397194 container create 7eb86b7aeb2eecc76a6e1da77889e041c914e104ea188afa27a84ce26e4cf655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:18:12.762 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 podman[16814]: 2024-08-30 23:18:12.696248646 +0000 UTC m=+0.344962954 container init 7eb86b7aeb2eecc76a6e1da77889e041c914e104ea188afa27a84ce26e4cf655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-30T23:18:12.763 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 podman[16814]: 2024-08-30 23:18:12.699114693 +0000 UTC m=+0.347829014 container start 7eb86b7aeb2eecc76a6e1da77889e041c914e104ea188afa27a84ce26e4cf655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-30T23:18:12.763 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 bash[16814]: 7eb86b7aeb2eecc76a6e1da77889e041c914e104ea188afa27a84ce26e4cf655 2024-08-30T23:18:12.776 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-30T23:18:12.776 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-08-30T23:18:12.783 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-30T23:18:12.783 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-08-30T23:18:12.783 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-08-30T23:18:12.784 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to enable service . firewalld.service is not available 2024-08-30T23:18:12.792 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-30T23:18:12.793 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-08-30T23:18:12.801 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-30T23:18:12.801 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-08-30T23:18:12.801 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-08-30T23:18:12.801 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-30T23:18:12.801 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr to start... 2024-08-30T23:18:12.801 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr... 2024-08-30T23:18:13.019 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:12 smithi106 systemd[1]: Started Ceph mgr.y for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:18:13.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2406574601' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-30T23:18:13.929 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:13.935 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:13.935 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:18:13.935 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-08-30T23:18:13.935 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-30T23:18:13.936 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.937 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "btime": "2024-08-30T23:18:02:086399+0000", 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-30T23:18:13.938 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:13.939 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-08-30T23:18:02.087118+0000", 2024-08-30T23:18:13.940 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-30T23:18:13.940 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:13.940 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-30T23:18:13.940 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:13.940 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (1/15)... 2024-08-30T23:18:16.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2018267714' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-30T23:18:17.097 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:17.097 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:17.097 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-30T23:18:17.098 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-30T23:18:17.099 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-30T23:18:17.102 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-30T23:18:17.102 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.102 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "btime": "2024-08-30T23:18:02:086399+0000", 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-30T23:18:17.103 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-08-30T23:18:02.087118+0000", 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-30T23:18:17.104 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:17.105 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-30T23:18:17.105 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:17.105 INFO:teuthology.orchestra.run.smithi106.stdout:mgr not available, waiting (2/15)... 2024-08-30T23:18:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: Activating manager daemon y 2024-08-30T23:18:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: mgrmap e2: y(active, starting, since 0.00343611s) 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: Manager daemon y is now available 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' 2024-08-30T23:18:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:17 smithi106 ceph-mon[16603]: from='mgr.14100 172.21.15.106:0/3422304058' entity='mgr.y' 2024-08-30T23:18:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:19 smithi106 ceph-mon[16603]: mgrmap e3: y(active, since 1.00696s) 2024-08-30T23:18:20.552 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:20.554 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:20.554 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "health": { 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 0 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "a" 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-30T23:18:20.555 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-30T23:18:20.556 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-30T23:18:20.559 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.559 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "btime": "2024-08-30T23:18:02:086399+0000", 2024-08-30T23:18:20.560 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "restful" 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ], 2024-08-30T23:18:20.561 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "modified": "2024-08-30T23:18:02.087118+0000", 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout }, 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:20.562 INFO:teuthology.orchestra.run.smithi106.stdout:mgr is available 2024-08-30T23:18:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:20 smithi106 ceph-mon[16603]: mgrmap e4: y(active, since 2s) 2024-08-30T23:18:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/381502439' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-30T23:18:21.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:21.767 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [global] 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout fsid = eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.106:3300,v1:172.21.15.106:6789] 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout [osd] 2024-08-30T23:18:21.768 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-30T23:18:21.769 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-30T23:18:21.769 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling cephadm module... 2024-08-30T23:18:21.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:21 smithi106 ceph-mon[16603]: from='client.14115 172.21.15.106:0/1547271129' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-08-30T23:18:23.077 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1108154825' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-08-30T23:18:23.078 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:22 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: ignoring --setuser ceph since I am not root 2024-08-30T23:18:23.078 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:22 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: ignoring --setgroup ceph since I am not root 2024-08-30T23:18:24.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1108154825' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-30T23:18:24.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:23 smithi106 ceph-mon[16603]: mgrmap e5: y(active, since 5s) 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for the mgr to restart... 2024-08-30T23:18:24.654 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr epoch 5... 2024-08-30T23:18:24.902 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/393936004' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: Active manager daemon y restarted 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: Activating manager daemon y 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: osdmap e2: 0 total, 0 up, 0 in 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: mgrmap e6: y(active, starting, since 0.00329934s) 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-30T23:18:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-30T23:18:28.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-30T23:18:28.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-30T23:18:28.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:27 smithi106 ceph-mon[16603]: Manager daemon y is now available 2024-08-30T23:18:29.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:29.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: Found migration_current of "None". Setting to last migration. 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: mgrmap e7: y(active, since 1.00543s) 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='client.14123 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='client.14123 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-30T23:18:29.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:18:29.013+0000 7f609336c640 -1 log_channel(cephadm) log [ERR] : [30/Aug/2024:23:18:29] ENGINE Error in HTTPServer.serve 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: Traceback (most recent call last): 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self._connections.run(self.expiration_interval) 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self._run(expiration_interval) 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: new_conn = self._from_server_socket(self.server.socket) 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-30T23:18:29.318 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: s = self.context.wrap_socket( 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: return self.sslsocket_class._create( 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self.do_handshake() 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self._sslobj.do_handshake() 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-30T23:18:29.319 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:18:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-30T23:18:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-30T23:18:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:mgr epoch 5 is available 2024-08-30T23:18:29.320 INFO:teuthology.orchestra.run.smithi106.stdout:Setting orchestrator backend to cephadm... 2024-08-30T23:18:30.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:28] ENGINE Bus STARTING 2024-08-30T23:18:30.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:29] ENGINE Serving on https://172.21.15.106:7150 2024-08-30T23:18:30.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:29] ENGINE Error in HTTPServer.serve 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: Traceback (most recent call last): 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: self._connections.run(self.expiration_interval) 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: self._run(expiration_interval) 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: s = self.context.wrap_socket( 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: return self.sslsocket_class._create( 2024-08-30T23:18:30.348 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout: self.do_handshake() 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout: self._sslobj.do_handshake() 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:29] ENGINE Serving on http://172.21.15.106:8765 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:29] ENGINE Bus STARTED 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:30.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:30 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:18:31.168 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:31 smithi106 ceph-mon[16603]: from='client.14131 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:31.168 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:31 smithi106 ceph-mon[16603]: mgrmap e8: y(active, since 2s) 2024-08-30T23:18:31.713 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-30T23:18:31.714 INFO:teuthology.orchestra.run.smithi106.stdout:Generating ssh key... 2024-08-30T23:18:32.377 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-mon[16603]: from='client.14133 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: Generating public/private rsa key pair. 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: Your identification has been saved in /tmp/tmpnc1ziotm/key 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: Your public key has been saved in /tmp/tmpnc1ziotm/key.pub 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: The key fingerprint is: 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: SHA256:kwu76vkWV2b3yHJ7n30CuXVqvb4hJXzqBSQdIP3L9kA ceph-eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: The key's randomart image is: 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: +---[RSA 3072]----+ 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | ..... | 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | ... . | 2024-08-30T23:18:32.684 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | ..o | 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | .+ =E | 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | . S+ ooB.o | 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | .o.o. *=O .| 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | .o. o.Bo* | 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | ... + B.*| 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: | .++o +.B*| 2024-08-30T23:18:32.685 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:32 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: +----[SHA256]-----+ 2024-08-30T23:18:33.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:33 smithi106 ceph-mon[16603]: from='client.14135 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:33.511 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:33 smithi106 ceph-mon[16603]: Generating ssh key... 2024-08-30T23:18:33.512 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:33 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:33.512 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:33 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:34.068 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:18:34.069 INFO:teuthology.orchestra.run.smithi106.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-30T23:18:34.069 INFO:teuthology.orchestra.run.smithi106.stdout:Adding key to root@localhost authorized_keys... 2024-08-30T23:18:34.069 INFO:teuthology.orchestra.run.smithi106.stdout:Adding host smithi106... 2024-08-30T23:18:34.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:34 smithi106 ceph-mon[16603]: from='client.14137 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:35.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:35 smithi106 ceph-mon[16603]: from='client.14139 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "addr": "172.21.15.106", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:36 smithi106 ceph-mon[16603]: Deploying cephadm binary to smithi106 2024-08-30T23:18:38.368 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Added host 'smithi106' with addr '172.21.15.106' 2024-08-30T23:18:38.368 INFO:teuthology.orchestra.run.smithi106.stdout:Deploying unmanaged mon service... 2024-08-30T23:18:38.986 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:38 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:38.986 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:38 smithi106 ceph-mon[16603]: Added host smithi106 2024-08-30T23:18:38.987 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:38 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:18:39.810 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-30T23:18:39.810 INFO:teuthology.orchestra.run.smithi106.stdout:Deploying unmanaged mgr service... 2024-08-30T23:18:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:39 smithi106 ceph-mon[16603]: from='client.14141 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:39 smithi106 ceph-mon[16603]: Saving service mon spec with placement count:5 2024-08-30T23:18:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:39 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:39 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:18:41.043 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-30T23:18:41.836 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:41 smithi106 ceph-mon[16603]: from='client.14143 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:41.836 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:41 smithi106 ceph-mon[16603]: Saving service mgr spec with placement count:2 2024-08-30T23:18:41.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:41 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:43.185 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:42 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:43.185 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:42 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:43.185 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3361145385' entity='client.admin' 2024-08-30T23:18:44.777 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3844001323' entity='client.admin' 2024-08-30T23:18:44.777 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:44 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:44.780 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling the dashboard module... 2024-08-30T23:18:46.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:46 smithi106 ceph-mon[16603]: from='mgr.14119 172.21.15.106:0/3987656035' entity='mgr.y' 2024-08-30T23:18:46.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2395738255' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-08-30T23:18:46.449 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:46 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: ignoring --setuser ceph since I am not root 2024-08-30T23:18:46.449 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:46 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: ignoring --setgroup ceph since I am not root 2024-08-30T23:18:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2395738255' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-30T23:18:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:47 smithi106 ceph-mon[16603]: mgrmap e9: y(active, since 18s) 2024-08-30T23:18:48.225 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:48.225 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-30T23:18:48.225 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-30T23:18:48.226 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-30T23:18:48.226 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-30T23:18:48.226 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:48.226 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for the mgr to restart... 2024-08-30T23:18:48.226 INFO:teuthology.orchestra.run.smithi106.stdout:Waiting for mgr epoch 9... 2024-08-30T23:18:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3078104682' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: Active manager daemon y restarted 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: Activating manager daemon y 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: osdmap e3: 0 total, 0 up, 0 in 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: mgrmap e10: y(active, starting, since 0.00345144s) 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-30T23:18:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-30T23:18:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-30T23:18:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-30T23:18:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:51 smithi106 ceph-mon[16603]: Manager daemon y is now available 2024-08-30T23:18:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:18:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-30T23:18:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-30T23:18:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:52 smithi106 ceph-mon[16603]: mgrmap e11: y(active, since 1.00542s) 2024-08-30T23:18:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:52 smithi106 ceph-mon[16603]: from='client.14155 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-30T23:18:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:52 smithi106 ceph-mon[16603]: from='client.14155 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-30T23:18:52.983 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout { 2024-08-30T23:18:52.983 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-30T23:18:52.983 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-30T23:18:52.983 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout } 2024-08-30T23:18:52.983 INFO:teuthology.orchestra.run.smithi106.stdout:mgr epoch 9 is available 2024-08-30T23:18:52.983 INFO:teuthology.orchestra.run.smithi106.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-08-30T23:18:53.292 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:18:53.210+0000 7f220a204640 -1 log_channel(cephadm) log [ERR] : [30/Aug/2024:23:18:53] ENGINE Error in HTTPServer.serve 2024-08-30T23:18:53.292 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: Traceback (most recent call last): 2024-08-30T23:18:53.292 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self._connections.run(self.expiration_interval) 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self._run(expiration_interval) 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: new_conn = self._from_server_socket(self.server.socket) 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: s = self.context.wrap_socket( 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: return self.sslsocket_class._create( 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-30T23:18:53.293 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self.do_handshake() 2024-08-30T23:18:53.294 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-30T23:18:53.294 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: self._sslobj.do_handshake() 2024-08-30T23:18:53.294 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-30T23:18:53.294 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:18:53.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:52] ENGINE Bus STARTING 2024-08-30T23:18:53.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:53] ENGINE Serving on http://172.21.15.106:8765 2024-08-30T23:18:53.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:53] ENGINE Serving on https://172.21.15.106:7150 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:53] ENGINE Bus STARTED 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:53 smithi106 ceph-mon[16603]: [30/Aug/2024:23:18:53] ENGINE Error in HTTPServer.serve 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: Traceback (most recent call last): 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: self._connections.run(self.expiration_interval) 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: self._run(expiration_interval) 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-30T23:18:53.561 INFO:journalctl@ceph.mon.a.smithi106.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: s = self.context.wrap_socket( 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: return self.sslsocket_class._create( 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: self.do_handshake() 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: self._sslobj.do_handshake() 2024-08-30T23:18:53.562 INFO:journalctl@ceph.mon.a.smithi106.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-30T23:18:54.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:54 smithi106 ceph-mon[16603]: mgrmap e12: y(active, since 2s) 2024-08-30T23:18:54.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:54 smithi106 ceph-mon[16603]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:54.604 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:18:55.157 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE/DCCAuSgAwIBAgIUJeqJOGugrt72h0kUFUHhNF2Wad8wDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDgzMDIzMTg1NFoXDTM0\nMDgzMTIzMTg1NFowGDEWMBQGA1UEAwwNMTcyLjIxLjE1LjEwNjCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBALpOHB0xSjZJY2UuWO8uOoADb1Fdn3vuZ586\nPNIUaoGngiKB/Tl7xgL0+KkpEXE3stTgDFnS8H0F1ngvAOYeUm8554JXUOrD/PyT\nW7WHjTuRsXOHmanmQt+pUS0N5kSO436x6FrjEMnLDv/ATDLHgBNU0C5w78ffin+T\nVaOy5ut7zGIUsrihALIDNvt10eciFEtYueLJWDbOSaAGFo6osIglMR7by1qXXk3T\nfjQwS2N0HSKwiHzVPOFXWDaGATp2hf3WtSYU5PYS09vCzNM8y76iuSMKniuIhn6u\nsLQIUH4/GM1Sn/+0d/HNFknVc22JJgjI9+tJ6UF5ky8V8nk08S3ThV+jCYgVsqbL\n4OKv9N89+a6+WuRkYmDJGpVH7cKDsbXwsd8LZ+Hiu/XzjDWyJalFjdnvcpTm3IL3\nV8JGU4cficjBKExd16peNQDvNTte+WN20XPajdFhYmf10Srrh4vsEiwdq3BY++vu\nLa9Erc+Bi3onWkchVJcuPZWVr9S77MLxO0KqKDkXRX6dLidsDfemRtPcbBX05s1H\neYmQLwsP+SJ3G5ZyoOMgbSvzMMHWfPUTiT81u5J6jTmBlowED4D99wERzH9SH27O\nVnBSG8sjbC/dBO4Gdxs6LWlhhDO6RdQdoNrQYAe1KCkb4Ijgu03nIflIIBjpJYV2\n4WEqcaPFAgMBAAGjPzA9MC0GA1UdEQQmMCSCCXNtaXRoaTEwNoIRZGFzaGJvYXJk\nX3NlcnZlcnOHBKwVD2owDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAgEA\newjVwVo0z3H0xacbNvL/dF81k+EJtvoyctG2FMRz1/sdg89w35+xjpX2WAuZSAIU\nF8eGpvsxMk2SqT9e5FsHcKVY0TN4xhvMeFOX0y/79S7VnW+UzlY4wxT9vn9L9q73\ndEKSTLShR61WZQbA9Uycm6085P8lv0lzCmBHjlOLeW1xENzpznxEZJdI5jr+d3h/\nU9Q8vouy+kqHpHFwHXVHEC+YH0oaYXGq+roY41Nz8G3M0+J+VzaE3iXxRTax8KoD\nwLhv7rywxwDwUIwUObsUq0QIbxdGnSnjYhZacEinvBtKqWcugAf/W0xa9rWq4Ekz\n4glwKHYyDsLhuaTlX5rz1yzTEuEM0IrSWjaDRC6qZzOFcvOUV8L/daVInX3TgaAr\nVSIz6IKs002XhEXnE5cvJZSUaOtnrUfzBeTpAxiUWUao7OkXBtAKXFUF2kqgE6xV\nms4HE55ZIEjPisPHxVsFI2T1q744zVNahH6cc3pX4RL/s73TrEWoiIdkw7ZGufs6\nkcuzcka6inlMvakETteqjTMWFzeN9FIx8z4iH9ZZbRB226/k8gG8bysSvSDGL2/y\nDVnh8ei3JLxZxtDZz0ZFY6ZeF7stFdFqgQsrSTqf9a3XcgQOq/+XiZYcPkwh9ayQ\nSkPkjUc26n8TiK//Ms82RCJMsZacZHPOm/ts6V8V+uU=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJJwIBAAKCAgEAuk4cHTFKNkljZS5Y7y46gANvUV2fe+5nnzo80hRqgaeCIoH9\nOXvGAvT4qSkRcTey1OAMWdLwfQXWeC8A5h5SbznngldQ6sP8/JNbtYeNO5Gxc4eZ\nqeZC36lRLQ3mRI7jfrHoWuMQycsO/8BMMseAE1TQLnDvx9+Kf5NVo7Lm63vMYhSy\nuKEAsgM2+3XR5yIUS1i54slYNs5JoAYWjqiwiCUxHtvLWpdeTdN+NDBLY3QdIrCI\nfNU84VdYNoYBOnaF/da1JhTk9hLT28LM0zzLvqK5IwqeK4iGfq6wtAhQfj8YzVKf\n/7R38c0WSdVzbYkmCMj360npQXmTLxXyeTTxLdOFX6MJiBWypsvg4q/03z35rr5a\n5GRiYMkalUftwoOxtfCx3wtn4eK79fOMNbIlqUWN2e9ylObcgvdXwkZThx+JyMEo\nTF3Xql41AO81O175Y3bRc9qN0WFiZ/XRKuuHi+wSLB2rcFj76+4tr0Stz4GLeida\nRyFUly49lZWv1LvswvE7QqooORdFfp0uJ2wN96ZG09xsFfTmzUd5iZAvCw/5Incb\nlnKg4yBtK/MwwdZ89ROJPzW7knqNOYGWjAQPgP33ARHMf1Ifbs5WcFIbyyNsL90E\n7gZ3GzotaWGEM7pF1B2g2tBgB7UoKRvgiOC7Tech+UggGOklhXbhYSpxo8UCAwEA\nAQKCAgAF0TSNhiF5anc2YekYUXWK07tLwMRJvAWeR3+TVlyGVIhQCDEI9MrCWDXU\nylnSEtl8TiFJ7g02SigKiU/aNO4sAj8upwf5cfSEn8adohrl1kRtMA8QlVush36S\nYpG4aExwLfaXB+oD8wQyi3rq8acXyMPV/2QX9eyUKTHjsk1a4vsHFlBc9XdZF5fw\nPXCNaqypjElZlyHbAZFI9DJgx/b4eU4nAYPq/eOi7cyjV91da0hm8eOBH64Ood84\n7Jw94BTfpS7zfnQxFCDAsZIbB8Cbi5HY/JbjExiSp2UQagYMEG+iJvH6AP+Rxu6V\nkEWnZkCQG6j/aXJlafoPiVGQjqJNLI3UgT3ZwT42z7jbpZULz0cZFZ21nJ2ZyUE9\nKw/CT1dvUOwOKvX4skMcZA/sejBndshB/BxAME8VUtNWu3PKdKgiQUoMSszbbaDw\n2oaa3bY7+00WSLJSpDJQ4g3didUyxEtizVDb6He5tUbnfx9SgQJl/g0Iize635TK\nTauE9ArvvQ+1qc8sEBJYDn0uSuHoLXqoFGtqX2ueoitsnpfRwRDHubIUtVbz8PAX\nHoTFZZ9Gtfqom6v98kf8ta/aUy/+8jzRiNUv9HzY8XIKtfVvLz8cJX98JBqieYg6\ndObBhwwkH+1M533QSCCGJTvgqdpAS9di7Vq1p8zoCiCdBc7L7QKCAQEA/W4RI2SY\n8plCJJbhooNz+UhtnUsZnD4zAwKBFxt2SSvb/mkI3htPjMEd78nIWCK1SJHI4q0c\n8lsSPvH1AfLUQgt8XG7CYj14qbQfsR7UcexyNdLvIOIcJjGi2aq2GLvlHmOzYWTD\nEI4bpAXfzpMZEOQ7dRDpEGVw5zRINIB/8tEfgc8PtpL2X5P9pOS5cbCx/fB3iQgp\n65mplJKUP4sK1Z7ekoU2s2eTEMhOZR61j2HmpV8OxtKGMfjKiA6OE7OlI13x1hc8\nYS1wsLBBuSali4Mch9w++8cFVHUR1Ht0UuJ4UL87Y/aHQesdDV9XTOqfgDHElVP8\nfHqFLUdDKB0JmwKCAQEAvDHHdoz/mN2Wc7SAxKFoucybPPw0vntdv2CLFU7k9x/6\nWEeIXJH2bgYFcloRAyn8jiM1/gWGZZr3IfvwB6NOWjrBJNfdseYPiIE8jNgiv2A3\nE72PIsOEtcutgLjLA75GKcdwLrQHkMidlNkbpR+jbL7jkOxCTchl9QL6a9c5JrKp\n5ivuX1ulecztPtszVsolFTJYRXEqxkoWrEO6DmgopnUFfbHN3gqSq68Ig4iyXR9v\nWi+JbDVIYbEf6NKCaqsSvpAUp/9j14haUibvFl/3QyNS5EehCkHXE0tWiKuzoI80\nwpHe4Eqr84iobClNN4rbF/SSWUUVUDMXF/A1zyUOHwKCAQA7G1Gp9ItrgW0YxYvo\ncEo7KLZQCjT16NfS6uGatw/2buFHS84Cp1H9XQkTZPdx/t5TB5VXsNJgf/cYcSwY\ngT8aDD27/SUe7oFW6yi9EZiw3VmhzglxnOq9ybPWmmcQC7qbRNuQaoTINqyhq7EQ\nJ+SfuOtZGVGSPmZ/tgTlIdc1laqIhvd+OHZ3wjdVHvdVWcWZmVz3u8yzI0qW2Tz0\nOQSU4BG/EIy6ePym3GDUWr4zq7y0cyAqJO2heAocg3CFjc+FGSMPMKonbAatXt5l\nRWs0x6Z+1VJnPHq0wP0glo6BykoPYxqB7PyM1j4XYVYmHU6IApN71HO9wIGRFt7F\n20fjAoIBAFkWd4X3WUfUHdTILeznO9w4r4Q/H/jMDaT63txHbLxPtwLzWen4bt/G\nme14FXWZ+qSOppCJrQWMRlEgIpB7DixIhAYc6iW88r178F38MvbFZU1C/u9F/xzQ\nPmIhGHm9KRm9/60cLDitv2snG58fSw2/j5qbPQf1Rc6mikVirPtUuHjtcvpROBRg\nni6TNcpwBv3xxzeYStdzWyhRJ1KKnDOwUdkewoXKG4SKZ01c+JJ8vwnCVxuSXaFh\nZ+mfPX2hXk43ho/AplLxiVA6fhPpefOeVS8beXCRmm7Xyy8HmRD4udEhnJAwyRWQ\nK6qIIrWYVf8RF/aKbXx5swTyAYb906ECggEAMEk7U21DFK4WhuH2l2uzYQuewUZ9\n/cfiM8eJqvfGOGHuctJZBJgs3RWqfPtIQZFQHHMoPPean4yGFpsw35dsKppIDO4N\nEJXB18cRYlJtVuZtFAMea9ehBef+1Z0V72fSb54vfIjiigCP7gQ28SPvTyfdTZSb\ngClsrkiVVGnScSoUM+axSVtTV9qFWufKa+IFCPuWyU0Qov8WabDw2BbRjSI7W65R\nqOQmk1PRtaEcrbtpnLxRhTLyUHKFXgynnINjf/YQ2vuRLyR/nQIpu4Y+kkNrmlbg\n+S23Gnr+htLqXDVTYqWRcXCdrObNb22wQRJb3pI/xnWUFRPrVVM/F2N1jQ==\n-----END RSA PRIVATE KEY-----\n"} 2024-08-30T23:18:55.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:18:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:56 smithi106 ceph-mon[16603]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:18:57.296 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-08-30T23:18:58.522 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-08-30T23:18:58.522 INFO:teuthology.orchestra.run.smithi106.stdout:Creating initial admin user... 2024-08-30T23:18:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:58 smithi106 ceph-mon[16603]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:18:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:58 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:00.100 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:59 smithi106 ceph-mon[16603]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:00.100 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:18:59 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:00.102 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ZWTsNeJs..Tckjns.MjIQeUU8Uayf0ODTBZZ1t8ZjJOD2qeNrscHK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725059939, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-30T23:19:00.103 INFO:teuthology.orchestra.run.smithi106.stdout:Fetching dashboard port number... 2024-08-30T23:19:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/883092012' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-08-30T23:19:01.301 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stdout 8443 2024-08-30T23:19:01.312 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-30T23:19:01.312 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout disabled 2024-08-30T23:19:01.320 INFO:teuthology.orchestra.run.smithi106.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-30T23:19:01.320 INFO:teuthology.orchestra.run.smithi106.stdout:systemctl: stdout inactive 2024-08-30T23:19:01.320 INFO:teuthology.orchestra.run.smithi106.stdout:firewalld.service is not enabled 2024-08-30T23:19:01.321 INFO:teuthology.orchestra.run.smithi106.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-30T23:19:01.322 INFO:teuthology.orchestra.run.smithi106.stdout:Ceph Dashboard is now available at: 2024-08-30T23:19:01.322 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:01.322 INFO:teuthology.orchestra.run.smithi106.stdout: URL: https://smithi106:8443/ 2024-08-30T23:19:01.322 INFO:teuthology.orchestra.run.smithi106.stdout: User: admin 2024-08-30T23:19:01.322 INFO:teuthology.orchestra.run.smithi106.stdout: Password: fkrv3nz19e 2024-08-30T23:19:01.322 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:01.323 INFO:teuthology.orchestra.run.smithi106.stdout:Saving cluster configuration to /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config directory 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout:Or, if you are only running a single cluster on this host: 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.661 INFO:teuthology.orchestra.run.smithi106.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: ceph telemetry on 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout:For more information see: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:02.662 INFO:teuthology.orchestra.run.smithi106.stdout:Bootstrap complete. 2024-08-30T23:19:02.703 INFO:tasks.cephadm:Fetching config... 2024-08-30T23:19:02.704 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:02.704 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-30T23:19:02.720 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-30T23:19:02.720 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:02.720 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-30T23:19:02.776 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-30T23:19:02.776 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:02.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-30T23:19:02.841 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-30T23:19:02.842 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:02.842 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-30T23:19:02.897 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-30T23:19:02.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-30T23:19:02.973 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:02.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-30T23:19:03.026 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:03.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-30T23:19:03.081 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE/nBNPzAevetjUZlfPsqn7HWy0yI1n9i1IwaJhJiv1aJQFynjfqK+991KvMLAllhXy9fReFJ93LbBn9DKzKpnHFIgJeH4tjYFsfjb6ixucT1PN2ztgTgRQMEBluR+kq7k/mnxmNxNpzAWFYhUJCmnoVmv40GgiWEkOdtdzR8H3MapFnQo2T0rIbD/Aetfd7p5A1pGY3MXltVi2OuUJ1xQc4lpPPQb/YeixvQYuEXCEFzTmgWaZqtusorfU5VPVbhZqlDlf8iaAlG4GodRi6mSUK8GtgPnGWftnykWzO2epOJYFWDI2SOO3vfjQeHVzls8rS17Cr0zVWyM0DToEIsEE6vLhud3Wfu9W6yPVKzqhbq8HvOo5ki8H+nz9DhnfmDxH4y2O8bfcq7abAxsZLZfN/53a1Rm3bYe829Eu1JrNcZzCSTBnf0lXA/KXyJb7tPG6FWpEMGMpAtk9xHkSo6CkEU/ho+z7adEgI1kTvxEXuIhf73s7JXi02YFXu9rjec= ceph-eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:03.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-30T23:19:03.341 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2199418683' entity='client.admin' 2024-08-30T23:19:05.169 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2956192645' entity='client.admin' 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: mgrmap e13: y(active, since 12s) 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"} : dispatch 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:05.170 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:05.197 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-30T23:19:05.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-30T23:19:05.728 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: Updating smithi106:/etc/ceph/ceph.conf 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: Updating smithi106:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: Updating smithi106:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:07.638 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2024-08-30T23:19:07.639 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:19:07.639 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2024-08-30T23:19:07.657 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:19:07.658 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:07.716 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2024-08-30T23:19:07.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch host add smithi136 2024-08-30T23:19:07.956 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:09 smithi106 ceph-mon[16603]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:10 smithi106 ceph-mon[16603]: Deploying cephadm binary to smithi136 2024-08-30T23:19:11.718 INFO:teuthology.orchestra.run.smithi106.stdout:Added host 'smithi136' with addr '172.21.15.136' 2024-08-30T23:19:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:12 smithi106 ceph-mon[16603]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:12.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-08-30T23:19:12.653 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:13.298 INFO:teuthology.orchestra.run.smithi106.stdout:[{"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2024-08-30T23:19:13.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:13 smithi106 ceph-mon[16603]: Added host smithi136 2024-08-30T23:19:13.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:13.299 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:13.929 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-08-30T23:19:13.930 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:19:13.930 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-08-30T23:19:13.949 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:19:13.949 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:14.007 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-08-30T23:19:14.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch host add smithi173 2024-08-30T23:19:14.247 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:14.507 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:14 smithi106 ceph-mon[16603]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:15.569 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:15 smithi106 ceph-mon[16603]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:19:16.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:16.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:16.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:16.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:16.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:16.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:16.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"} : dispatch 2024-08-30T23:19:16.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:16.550 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:17 smithi106 ceph-mon[16603]: Deploying cephadm binary to smithi173 2024-08-30T23:19:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:17 smithi106 ceph-mon[16603]: Updating smithi136:/etc/ceph/ceph.conf 2024-08-30T23:19:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:17 smithi106 ceph-mon[16603]: Updating smithi136:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:18.138 INFO:teuthology.orchestra.run.smithi106.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-08-30T23:19:18.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:18.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: Updating smithi136:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-30T23:19:18.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:18.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:18.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:18.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:18.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:18.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:18.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-08-30T23:19:19.050 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:19.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:19 smithi106 ceph-mon[16603]: Added host smithi173 2024-08-30T23:19:19.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:19.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:19.707 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:19.708 INFO:teuthology.orchestra.run.smithi106.stdout:[{"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-08-30T23:19:20.299 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-30T23:19:20.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd crush tunables default 2024-08-30T23:19:20.535 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:20 smithi106 ceph-mon[16603]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:21.265 INFO:teuthology.orchestra.run.smithi106.stderr:adjusted tunables profile to default 2024-08-30T23:19:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:21 smithi106 ceph-mon[16603]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:19:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1690723776' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-08-30T23:19:21.897 INFO:tasks.cephadm:Adding mon.a on smithi106 2024-08-30T23:19:21.897 INFO:tasks.cephadm:Adding mon.b on smithi136 2024-08-30T23:19:21.897 INFO:tasks.cephadm:Adding mon.c on smithi173 2024-08-30T23:19:21.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch apply mon '3;smithi106:172.21.15.106=a;smithi136:172.21.15.136=b;smithi173:172.21.15.173=c' 2024-08-30T23:19:22.466 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-30T23:19:22.512 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-08-30T23:19:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:22 smithi106 ceph-mon[16603]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1690723776' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-30T23:19:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:22 smithi106 ceph-mon[16603]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:23.191 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-08-30T23:19:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"} : dispatch 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: Updating smithi173:/etc/ceph/ceph.conf 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: Updating smithi173:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: Updating smithi173:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi106:172.21.15.106=a;smithi136:172.21.15.136=b;smithi173:172.21.15.173=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: Saving service mon spec with placement smithi106:172.21.15.106=a;smithi136:172.21.15.136=b;smithi173:172.21.15.173=c;count:3 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:23.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:23 smithi106 ceph-mon[16603]: Deploying daemon mon.c on smithi173 2024-08-30T23:19:23.916 DEBUG:teuthology.orchestra.run.smithi136:mon.b> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.b.service 2024-08-30T23:19:23.919 DEBUG:teuthology.orchestra.run.smithi173:mon.c> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.c.service 2024-08-30T23:19:23.922 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-30T23:19:23.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-30T23:19:24.915 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:25 smithi106 ceph-mon[16603]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:25.826 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:19:25.826 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","modified":"2024-08-30T23:17:53.378775Z","created":"2024-08-30T23:17:53.378775Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-30T23:19:25.827 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-08-30T23:19:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2570347788' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:27.508 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-30T23:19:27.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-30T23:19:27.764 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:19:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:27 smithi106 ceph-mon[16603]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:28.515 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:19:28.515 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","modified":"2024-08-30T23:17:53.378775Z","created":"2024-08-30T23:17:53.378775Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-30T23:19:28.516 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-08-30T23:19:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2531669719' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:28 smithi173 systemd[1]: Started Ceph mon.c for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:19:30.507 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-30T23:19:30.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-30T23:19:30.767 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:19:34.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:34.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: mon.a calling monitor election 2024-08-30T23:19:34.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: mon.c calling monitor election 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: monmap epoch 2 2024-08-30T23:19:34.278 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: last_changed 2024-08-30T23:19:28.837648+0000 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: min_mon_release 19 (squid) 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: election_strategy: 1 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: fsmap 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: mgrmap e13: y(active, since 42s) 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: overall HEALTH_OK 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:34.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:34.280 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:34.280 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: mon.a calling monitor election 2024-08-30T23:19:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: mon.c calling monitor election 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: monmap epoch 2 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: last_changed 2024-08-30T23:19:28.837648+0000 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: min_mon_release 19 (squid) 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: election_strategy: 1 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:34.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: fsmap 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: mgrmap e13: y(active, since 42s) 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: overall HEALTH_OK 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:34.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:34.457 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:19:34.457 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","modified":"2024-08-30T23:19:28.837648Z","created":"2024-08-30T23:17:53.378775Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-30T23:19:34.458 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-08-30T23:19:35.029 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:34 smithi173 ceph-mon[20554]: Deploying daemon mon.b on smithi136 2024-08-30T23:19:35.029 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/2213921411' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:35.029 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:34 smithi106 ceph-mon[16603]: Deploying daemon mon.b on smithi136 2024-08-30T23:19:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2213921411' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:36.058 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-30T23:19:36.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-30T23:19:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:35 smithi106 ceph-mon[16603]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:36.107 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:19:35 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:19:35.841+0000 7f2243404640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-30T23:19:36.310 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:19:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:35 smithi173 ceph-mon[20554]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:19:37.031 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","modified":"2024-08-30T23:19:28.837648Z","created":"2024-08-30T23:17:53.378775Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-30T23:19:37.031 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-08-30T23:19:37.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/2717880419' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2717880419' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:37 smithi136 ceph-mon[20207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725059977605346, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2200807, "oldest_snapshot_seqno": -1} 2024-08-30T23:19:38.615 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-30T23:19:38.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-08-30T23:19:38.865 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:19:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: mon.a calling monitor election 2024-08-30T23:19:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: mon.c calling monitor election 2024-08-30T23:19:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: monmap epoch 3 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: last_changed 2024-08-30T23:19:37.609829+0000 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: min_mon_release 19 (squid) 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: election_strategy: 1 2024-08-30T23:19:43.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: 2: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: fsmap 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: mgrmap e13: y(active, since 51s) 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: mon.a calling monitor election 2024-08-30T23:19:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: mon.c calling monitor election 2024-08-30T23:19:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: monmap epoch 3 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: last_changed 2024-08-30T23:19:37.609829+0000 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: min_mon_release 19 (squid) 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: election_strategy: 1 2024-08-30T23:19:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: 2: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: fsmap 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: mgrmap e13: y(active, since 51s) 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:43.142 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:19:43.143 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","modified":"2024-08-30T23:19:37.609829Z","created":"2024-08-30T23:17:53.378775Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-30T23:19:43.143 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-08-30T23:19:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/193473991' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:43 smithi173 ceph-mon[20554]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:44.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/193473991' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:44.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:43 smithi106 ceph-mon[16603]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:44.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.471 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-30T23:19:44.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph config generate-minimal-conf 2024-08-30T23:19:44.736 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: mon.a calling monitor election 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: mon.c calling monitor election 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:44.899 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: monmap epoch 3 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: last_changed 2024-08-30T23:19:37.609829+0000 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: min_mon_release 19 (squid) 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: election_strategy: 1 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: 2: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-08-30T23:19:44.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: fsmap 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: mgrmap e13: y(active, since 51s) 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: mon.b (rank 2) addr [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] is down (out of quorum) 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/193473991' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:19:44.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:44.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:45.367 INFO:teuthology.orchestra.run.smithi106.stdout:# minimal ceph.conf for eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:45.367 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-08-30T23:19:45.367 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:45.367 INFO:teuthology.orchestra.run.smithi106.stdout: mon_host = [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: mon.b calling monitor election 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Updating smithi173:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Updating smithi136:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Updating smithi106:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: mon.b calling monitor election 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: mon.c calling monitor election 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: mon.a calling monitor election 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-30T23:19:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: monmap epoch 3 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: last_changed 2024-08-30T23:19:37.609829+0000 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: min_mon_release 19 (squid) 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: election_strategy: 1 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: 2: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: fsmap 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: mgrmap e13: y(active, since 53s) 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-30T23:19:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: overall HEALTH_OK 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Reconfiguring mon.a (unknown last config time)... 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: Reconfiguring daemon mon.a on smithi106 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/524948170' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: mon.b calling monitor election 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Updating smithi173:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Updating smithi136:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Updating smithi106:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: mon.b calling monitor election 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: mon.c calling monitor election 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: mon.a calling monitor election 2024-08-30T23:19:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: monmap epoch 3 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: last_changed 2024-08-30T23:19:37.609829+0000 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: min_mon_release 19 (squid) 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: election_strategy: 1 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: 2: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: fsmap 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: mgrmap e13: y(active, since 53s) 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-30T23:19:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: overall HEALTH_OK 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Reconfiguring mon.a (unknown last config time)... 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: Reconfiguring daemon mon.a on smithi106 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/524948170' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: mon.b calling monitor election 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Updating smithi173:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Updating smithi136:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Updating smithi106:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/config/ceph.conf 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: mon.b calling monitor election 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: mon.c calling monitor election 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: mon.a calling monitor election 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-30T23:19:45.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: monmap epoch 3 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: last_changed 2024-08-30T23:19:37.609829+0000 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: created 2024-08-30T23:17:53.378775+0000 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: min_mon_release 19 (squid) 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: election_strategy: 1 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: 0: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.a 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: 1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.c 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: 2: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: fsmap 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: osdmap e4: 0 total, 0 up, 0 in 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: mgrmap e13: y(active, since 53s) 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-30T23:19:45.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: overall HEALTH_OK 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Reconfiguring mon.a (unknown last config time)... 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: Reconfiguring daemon mon.a on smithi106 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:45.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/524948170' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:46.092 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-30T23:19:46.092 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:46.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-08-30T23:19:46.121 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:46.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:46.186 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:19:46.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-08-30T23:19:46.214 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:19:46.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:46.281 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:19:46.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-08-30T23:19:46.312 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:19:46.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-30T23:19:46.380 INFO:tasks.cephadm:Adding mgr.y on smithi106 2024-08-30T23:19:46.380 INFO:tasks.cephadm:Adding mgr.x on smithi136 2024-08-30T23:19:46.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch apply mgr '2;smithi106=y;smithi136=x' 2024-08-30T23:19:46.668 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:19:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:46.857 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:19:46 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:19:46.613+0000 7f2243404640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-30T23:19:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:47.330 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-08-30T23:19:47.899 DEBUG:teuthology.orchestra.run.smithi136:mgr.x> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.x.service 2024-08-30T23:19:47.902 DEBUG:tasks.cephadm:set 0 configs 2024-08-30T23:19:47.902 INFO:tasks.cephadm:Deploying OSDs... 2024-08-30T23:19:47.902 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:19:47.903 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-08-30T23:19:47.922 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-30T23:19:47.923 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-08-30 23:19:06.019013928 +0000 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-08-30 23:14:44.462172419 +0000 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-08-30 23:14:44.462172419 +0000 2024-08-30T23:19:47.982 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-08-30 23:14:44.462172419 +0000 2024-08-30T23:19:47.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-30T23:19:48.051 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-08-30T23:19:48.051 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-08-30T23:19:48.051 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000299487 s, 1.7 MB/s 2024-08-30T23:19:48.053 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-30T23:19:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: Reconfiguring daemon mon.b on smithi136 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.112 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-08-30 23:19:06.051013167 +0000 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-08-30 23:14:44.797162163 +0000 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-08-30 23:14:44.797162163 +0000 2024-08-30T23:19:48.169 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-08-30 23:14:44.797162163 +0000 2024-08-30T23:19:48.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-30T23:19:48.234 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-08-30T23:19:48.235 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-08-30T23:19:48.235 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00023013 s, 2.2 MB/s 2024-08-30T23:19:48.236 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: Reconfiguring daemon mon.b on smithi136 2024-08-30T23:19:48.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:48.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:48.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:48.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:48.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.293 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-08-30T23:19:48.349 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-08-30 23:19:06.081012454 +0000 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-08-30 23:14:45.117152370 +0000 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-08-30 23:14:45.117152370 +0000 2024-08-30T23:19:48.350 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-08-30 23:14:45.117152370 +0000 2024-08-30T23:19:48.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: Reconfiguring daemon mon.b on smithi136 2024-08-30T23:19:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:19:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:19:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:19:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:48.381 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-08-30T23:19:48.381 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-08-30T23:19:48.381 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000229223 s, 2.2 MB/s 2024-08-30T23:19:48.382 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-30T23:19:48.440 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-08-30 23:19:06.112011717 +0000 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-08-30 23:14:45.448142241 +0000 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-08-30 23:14:45.448142241 +0000 2024-08-30T23:19:48.497 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-08-30 23:14:45.448142241 +0000 2024-08-30T23:19:48.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-30T23:19:48.560 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-08-30T23:19:48.560 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-08-30T23:19:48.560 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000205519 s, 2.5 MB/s 2024-08-30T23:19:48.561 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-30T23:19:48.618 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:19:48.618 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-08-30T23:19:48.634 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-30T23:19:48.635 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-08-30T23:19:48.690 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-30 23:19:37.086145512 +0000 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-30 23:14:45.198883534 +0000 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-30 23:14:45.198883534 +0000 2024-08-30T23:19:48.691 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-30 23:14:45.198883534 +0000 2024-08-30T23:19:48.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-30T23:19:48.754 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-30T23:19:48.754 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-30T23:19:48.754 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000239003 s, 2.1 MB/s 2024-08-30T23:19:48.755 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-30T23:19:48.812 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-30 23:19:37.086145512 +0000 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-30 23:14:45.523875464 +0000 2024-08-30T23:19:48.867 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-30 23:14:45.523875464 +0000 2024-08-30T23:19:48.868 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-30 23:14:45.523875464 +0000 2024-08-30T23:19:48.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-30T23:19:48.930 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-30T23:19:48.930 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-30T23:19:48.930 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000186464 s, 2.7 MB/s 2024-08-30T23:19:48.931 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-30T23:19:48.987 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-08-30T23:19:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-30T23:19:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:49.042 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-08-30T23:19:49.042 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:49.042 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:49.043 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-30 23:19:37.087145490 +0000 2024-08-30T23:19:49.043 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-30 23:14:45.873866774 +0000 2024-08-30T23:19:49.043 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-30 23:14:45.873866774 +0000 2024-08-30T23:19:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-30 23:14:45.873866774 +0000 2024-08-30T23:19:49.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-30T23:19:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-30T23:19:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-30T23:19:49.109 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000236953 s, 2.2 MB/s 2024-08-30T23:19:49.109 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-30T23:19:49.165 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-08-30T23:19:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-30T23:19:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:49.220 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-30T23:19:49.220 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:49.220 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:49.221 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-08-30 23:19:37.087145490 +0000 2024-08-30T23:19:49.221 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-08-30 23:14:46.200858655 +0000 2024-08-30T23:19:49.221 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-08-30 23:14:46.200858655 +0000 2024-08-30T23:19:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-08-30 23:14:46.200858655 +0000 2024-08-30T23:19:49.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi106=y;smithi136=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: Saving service mgr spec with placement smithi106=y;smithi136=x;count:2 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:49.281 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:49.283 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-08-30T23:19:49.283 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-08-30T23:19:49.283 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000261467 s, 2.0 MB/s 2024-08-30T23:19:49.284 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi106=y;smithi136=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: Saving service mgr spec with placement smithi106=y;smithi136=x;count:2 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:49.341 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:19:49.341 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-08-30T23:19:49.357 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-30T23:19:49.357 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-08-30T23:19:49.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi106=y;smithi136=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:49.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: Saving service mgr spec with placement smithi106=y;smithi136=x;count:2 2024-08-30T23:19:49.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:49.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:49.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-30T23:19:49.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-30T23:19:49.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:49.413 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-08-30 23:19:27.912040265 +0000 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-08-30 23:14:45.345779206 +0000 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-08-30 23:14:45.345779206 +0000 2024-08-30T23:19:49.414 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-08-30 23:14:45.345779206 +0000 2024-08-30T23:19:49.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-30T23:19:49.478 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-08-30T23:19:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-08-30T23:19:49.479 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000215887 s, 2.4 MB/s 2024-08-30T23:19:49.480 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-30T23:19:49.536 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-08-30T23:19:49.591 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-30T23:19:49.591 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:49.591 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-08-30T23:19:49.591 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:49.592 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:49.592 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-08-30 23:19:27.913040244 +0000 2024-08-30T23:19:49.592 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-08-30 23:14:45.713771209 +0000 2024-08-30T23:19:49.592 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-08-30 23:14:45.713771209 +0000 2024-08-30T23:19:49.592 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-08-30 23:14:45.713771209 +0000 2024-08-30T23:19:49.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-30T23:19:49.656 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-08-30T23:19:49.656 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-08-30T23:19:49.656 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000236538 s, 2.2 MB/s 2024-08-30T23:19:49.657 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-30T23:19:49.713 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-08-30 23:19:27.913040244 +0000 2024-08-30T23:19:49.769 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-08-30 23:14:46.062763625 +0000 2024-08-30T23:19:49.770 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-08-30 23:14:46.062763625 +0000 2024-08-30T23:19:49.770 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-08-30 23:14:46.062763625 +0000 2024-08-30T23:19:49.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-30T23:19:49.832 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-08-30T23:19:49.832 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-08-30T23:19:49.832 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000298894 s, 1.7 MB/s 2024-08-30T23:19:49.834 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-30T23:19:49.890 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-08-30T23:19:49.945 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-08-30 23:19:27.913040244 +0000 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-08-30 23:14:46.390756498 +0000 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-08-30 23:14:46.390756498 +0000 2024-08-30T23:19:49.946 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-08-30 23:14:46.390756498 +0000 2024-08-30T23:19:49.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-30T23:19:50.011 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-08-30T23:19:50.012 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-08-30T23:19:50.012 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000202044 s, 2.5 MB/s 2024-08-30T23:19:50.012 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-30T23:19:50.068 INFO:tasks.cephadm:Deploying osd.0 on smithi106 with /dev/vg_nvme/lv_4... 2024-08-30T23:19:50.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: Reconfiguring mon.c (monmap changed)... 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: Reconfiguring daemon mon.c on smithi173 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-30T23:19:50.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-30T23:19:50.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-30T23:19:50.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: Reconfiguring mon.c (monmap changed)... 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: Reconfiguring daemon mon.c on smithi173 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:50.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:50.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-30T23:19:50.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-30T23:19:50.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-30T23:19:50.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:50.342 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: Reconfiguring mon.c (monmap changed)... 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: Reconfiguring daemon mon.c on smithi173 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-30T23:19:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-30T23:19:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-30T23:19:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:51 smithi136 ceph-mon[20207]: Deploying daemon mgr.x on smithi136 2024-08-30T23:19:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:51 smithi173 ceph-mon[20554]: Deploying daemon mgr.x on smithi136 2024-08-30T23:19:51.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:51 smithi106 ceph-mon[16603]: Deploying daemon mgr.x on smithi136 2024-08-30T23:19:51.708 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 systemd[1]: Starting Ceph mgr.x for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:19:52.038 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 podman[21558]: 2024-08-30 23:19:51.709073516 +0000 UTC m=+0.107876577 container create 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-08-30T23:19:52.039 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 podman[21558]: 2024-08-30 23:19:51.616693319 +0000 UTC m=+0.015496395 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:19:52.039 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 podman[21558]: 2024-08-30 23:19:51.869511515 +0000 UTC m=+0.268314573 container init 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-30T23:19:52.039 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 podman[21558]: 2024-08-30 23:19:51.872244522 +0000 UTC m=+0.271047590 container start 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-30T23:19:52.039 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 ceph-mgr[21577]: -- 172.21.15.136:0/2521929377 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9a97850e0 con 0x55a9a9763000 2024-08-30T23:19:52.039 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 bash[21558]: 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 2024-08-30T23:19:52.039 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 30 23:19:51 smithi136 systemd[1]: Started Ceph mgr.x for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:19:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:52 smithi136 ceph-mon[20207]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:52 smithi173 ceph-mon[20554]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:52 smithi106 ceph-mon[16603]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:52.658 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:19:52.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_4 2024-08-30T23:19:52.924 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:19:53.292 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:53.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: Reconfiguring mgr.y (unknown last config time)... 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: Reconfiguring daemon mgr.y on smithi106 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:19:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: Reconfiguring mgr.y (unknown last config time)... 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: Reconfiguring daemon mgr.y on smithi106 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:19:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: Reconfiguring mgr.y (unknown last config time)... 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: Reconfiguring daemon mgr.y on smithi106 2024-08-30T23:19:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:19:54.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:19:54.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:55.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:55.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:55.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:55.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:55.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:19:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:19:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:19:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:19:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:56 smithi106 ceph-mon[16603]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:56.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:56 smithi136 ceph-mon[20207]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:56.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:56 smithi173 ceph-mon[20554]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/809561873' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9"} : dispatch 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/809561873' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9"}]': finished 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: osdmap e5: 1 total, 0 up, 1 in 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: Standby manager daemon x started 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-30T23:19:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:57.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1446245014' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:19:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/809561873' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9"} : dispatch 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/809561873' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9"}]': finished 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: osdmap e5: 1 total, 0 up, 1 in 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: Standby manager daemon x started 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1446245014' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:19:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:19:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/809561873' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9"} : dispatch 2024-08-30T23:19:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/809561873' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9"}]': finished 2024-08-30T23:19:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: osdmap e5: 1 total, 0 up, 1 in 2024-08-30T23:19:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:19:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: Standby manager daemon x started 2024-08-30T23:19:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-30T23:19:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-30T23:19:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-30T23:19:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='mgr.? 172.21.15.136:0/272997976' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-30T23:19:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:19:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1446245014' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:19:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:58 smithi173 ceph-mon[20554]: mgrmap e14: y(active, since 66s), standbys: x 2024-08-30T23:19:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:58 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-30T23:19:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:58 smithi106 ceph-mon[16603]: mgrmap e14: y(active, since 66s), standbys: x 2024-08-30T23:19:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:58 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-30T23:19:59.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:58 smithi136 ceph-mon[20207]: mgrmap e14: y(active, since 66s), standbys: x 2024-08-30T23:19:59.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:58 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-30T23:20:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:19:59 smithi173 ceph-mon[20554]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:00.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:19:59 smithi106 ceph-mon[16603]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:19:59 smithi136 ceph-mon[20207]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:00.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:00 smithi106 ceph-mon[16603]: overall HEALTH_OK 2024-08-30T23:20:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:00 smithi173 ceph-mon[20554]: overall HEALTH_OK 2024-08-30T23:20:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:00 smithi136 ceph-mon[20207]: overall HEALTH_OK 2024-08-30T23:20:02.016 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:01 smithi106 ceph-mon[16603]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:01 smithi173 ceph-mon[20554]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:01 smithi136 ceph-mon[20207]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:03.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:03 smithi106 ceph-mon[16603]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:03 smithi136 ceph-mon[20207]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:03 smithi173 ceph-mon[20554]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-30T23:20:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:04 smithi106 ceph-mon[16603]: Deploying daemon osd.0 on smithi106 2024-08-30T23:20:04.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-30T23:20:04.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:04.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:04 smithi136 ceph-mon[20207]: Deploying daemon osd.0 on smithi106 2024-08-30T23:20:04.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-30T23:20:04.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:04.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:04 smithi173 ceph-mon[20554]: Deploying daemon osd.0 on smithi106 2024-08-30T23:20:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:05 smithi106 ceph-mon[16603]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:05 smithi136 ceph-mon[20207]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:05 smithi173 ceph-mon[20554]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:07.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:07 smithi106 ceph-mon[16603]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:07 smithi136 ceph-mon[20207]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:07 smithi173 ceph-mon[20554]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:09 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:09 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:09 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:09.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:09 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:09 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:09 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:09 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:09 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:09 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:10.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:10 smithi136 ceph-mon[20207]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:10 smithi173 ceph-mon[20554]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:10.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:10 smithi106 ceph-mon[16603]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:11.538 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:11 smithi106 ceph-mon[16603]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:11 smithi136 ceph-mon[20207]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:11 smithi173 ceph-mon[20554]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:13.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:13.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:13.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.056 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 0 on host 'smithi106' 2024-08-30T23:20:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:13.756 DEBUG:teuthology.orchestra.run.smithi106:osd.0> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.0.service 2024-08-30T23:20:13.759 INFO:tasks.cephadm:Deploying osd.1 on smithi106 with /dev/vg_nvme/lv_3... 2024-08-30T23:20:13.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-30T23:20:14.040 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:14.040 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:14.040 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:14.041 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:13 smithi106 ceph-mon[16603]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:13 smithi136 ceph-mon[20207]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:13 smithi173 ceph-mon[20554]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:14.515 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:20:14.857 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:20:14 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0[30291]: 2024-08-30T23:20:14.426+0000 7f4bcb55f640 -1 osd.0 0 log_to_monitors true 2024-08-30T23:20:15.147 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:15 smithi106 ceph-mon[16603]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-30T23:20:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:15 smithi173 ceph-mon[20554]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-30T23:20:15.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:15 smithi136 ceph-mon[20207]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-30T23:20:16.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:16 smithi136 ceph-mon[20207]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-30T23:20:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:16 smithi136 ceph-mon[20207]: osdmap e6: 1 total, 0 up, 1 in 2024-08-30T23:20:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:16 smithi136 ceph-mon[20207]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:20:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:16 smithi136 ceph-mon[20207]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:16 smithi173 ceph-mon[20554]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-30T23:20:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:16 smithi173 ceph-mon[20554]: osdmap e6: 1 total, 0 up, 1 in 2024-08-30T23:20:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:16 smithi173 ceph-mon[20554]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:20:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:16 smithi173 ceph-mon[20554]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-mon[16603]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-30T23:20:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-mon[16603]: osdmap e6: 1 total, 0 up, 1 in 2024-08-30T23:20:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-mon[16603]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:20:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-mon[16603]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:16.857 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0[30291]: 2024-08-30T23:20:16.502+0000 7f4bc74e2640 -1 osd.0 0 waiting for initial osdmap 2024-08-30T23:20:16.857 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:20:16 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0[30291]: 2024-08-30T23:20:16.505+0000 7f4bbf301640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:20:17.183 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:17 smithi106 ceph-mon[16603]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:20:17.183 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:17 smithi106 ceph-mon[16603]: osdmap e7: 1 total, 0 up, 1 in 2024-08-30T23:20:17.183 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.183 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.183 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:17 smithi106 ceph-mon[16603]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' 2024-08-30T23:20:17.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:17 smithi136 ceph-mon[20207]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:20:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:17 smithi136 ceph-mon[20207]: osdmap e7: 1 total, 0 up, 1 in 2024-08-30T23:20:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:17 smithi136 ceph-mon[20207]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' 2024-08-30T23:20:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:17 smithi173 ceph-mon[20554]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:20:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:17 smithi173 ceph-mon[20554]: osdmap e7: 1 total, 0 up, 1 in 2024-08-30T23:20:17.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:17 smithi173 ceph-mon[20554]: from='osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896]' entity='osd.0' 2024-08-30T23:20:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:18.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi106 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi106 to 17088M 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896] boot 2024-08-30T23:20:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: osdmap e8: 1 total, 1 up, 1 in 2024-08-30T23:20:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi106 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi106 to 17088M 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:18.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:18.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:18.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896] boot 2024-08-30T23:20:18.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: osdmap e8: 1 total, 1 up, 1 in 2024-08-30T23:20:18.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:18.568 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:20:18.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi106 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi106 to 17088M 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-30T23:20:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: osd.0 [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896] boot 2024-08-30T23:20:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: osdmap e8: 1 total, 1 up, 1 in 2024-08-30T23:20:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-30T23:20:18.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_3 2024-08-30T23:20:18.835 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:20:20.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:20 smithi106 ceph-mon[16603]: osdmap e9: 1 total, 1 up, 1 in 2024-08-30T23:20:20.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:20 smithi106 ceph-mon[16603]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:20.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:20:20.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:20:20.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:20.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:20 smithi136 ceph-mon[20207]: osdmap e9: 1 total, 1 up, 1 in 2024-08-30T23:20:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:20 smithi136 ceph-mon[20207]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:20:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:20:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:20 smithi173 ceph-mon[20554]: osdmap e9: 1 total, 1 up, 1 in 2024-08-30T23:20:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:20 smithi173 ceph-mon[20554]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:20:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:20:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:21 smithi136 ceph-mon[20207]: from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:20:21.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:21 smithi173 ceph-mon[20554]: from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:20:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:21 smithi106 ceph-mon[16603]: from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:20:22.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:22 smithi106 ceph-mon[16603]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:22.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1564893707' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53"} : dispatch 2024-08-30T23:20:22.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1564893707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53"}]': finished 2024-08-30T23:20:22.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:22 smithi106 ceph-mon[16603]: osdmap e10: 2 total, 1 up, 2 in 2024-08-30T23:20:22.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:22.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:22 smithi136 ceph-mon[20207]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1564893707' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53"} : dispatch 2024-08-30T23:20:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1564893707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53"}]': finished 2024-08-30T23:20:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:22 smithi136 ceph-mon[20207]: osdmap e10: 2 total, 1 up, 2 in 2024-08-30T23:20:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:22.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:22 smithi173 ceph-mon[20554]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:22.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1564893707' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53"} : dispatch 2024-08-30T23:20:22.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1564893707' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53"}]': finished 2024-08-30T23:20:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:22 smithi173 ceph-mon[20554]: osdmap e10: 2 total, 1 up, 2 in 2024-08-30T23:20:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:23.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4287732116' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:20:23.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4287732116' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:20:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4287732116' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:20:24.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:24 smithi136 ceph-mon[20207]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:24.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:24 smithi173 ceph-mon[20554]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:24 smithi106 ceph-mon[16603]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:25 smithi106 ceph-mon[16603]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:25 smithi136 ceph-mon[20207]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:25 smithi173 ceph-mon[20554]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:27.556 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:27 smithi106 ceph-mon[16603]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:27 smithi136 ceph-mon[20207]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:27 smithi173 ceph-mon[20554]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-30T23:20:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:29.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-30T23:20:29.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:29.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-30T23:20:29.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:30 smithi106 ceph-mon[16603]: Deploying daemon osd.1 on smithi106 2024-08-30T23:20:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:30 smithi106 ceph-mon[16603]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:30 smithi136 ceph-mon[20207]: Deploying daemon osd.1 on smithi106 2024-08-30T23:20:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:30 smithi136 ceph-mon[20207]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:30.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:30 smithi173 ceph-mon[20554]: Deploying daemon osd.1 on smithi106 2024-08-30T23:20:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:30 smithi173 ceph-mon[20554]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:31.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:31 smithi106 ceph-mon[16603]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:31 smithi136 ceph-mon[20207]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:31 smithi173 ceph-mon[20554]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:33.519 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:33 smithi106 ceph-mon[16603]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:33 smithi136 ceph-mon[20207]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:33 smithi173 ceph-mon[20554]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:34.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:34.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:34.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:34.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:34.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:34.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:34.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:34.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:34.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:35.706 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:35 smithi136 ceph-mon[20207]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:35.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:35 smithi173 ceph-mon[20554]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:35 smithi106 ceph-mon[16603]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:38.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:38.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:38.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:38 smithi106 ceph-mon[16603]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:38.314 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 1 on host 'smithi106' 2024-08-30T23:20:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:38 smithi173 ceph-mon[20554]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:38 smithi136 ceph-mon[20207]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:39.012 DEBUG:teuthology.orchestra.run.smithi106:osd.1> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1.service 2024-08-30T23:20:39.015 INFO:tasks.cephadm:Deploying osd.2 on smithi106 with /dev/vg_nvme/lv_2... 2024-08-30T23:20:39.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-30T23:20:39.266 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:39 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:39.266 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:39 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:39.267 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:39 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:39.267 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:39 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-30T23:20:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:39 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:39 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:39 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:39 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-30T23:20:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:39 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:20:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:39 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:39 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:39 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-30T23:20:39.662 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:20:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:40 smithi173 ceph-mon[20554]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:40 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-30T23:20:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:40 smithi173 ceph-mon[20554]: osdmap e11: 2 total, 1 up, 2 in 2024-08-30T23:20:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:40 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:20:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:40 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-mon[16603]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-30T23:20:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-mon[16603]: osdmap e11: 2 total, 1 up, 2 in 2024-08-30T23:20:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:20:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:40 smithi136 ceph-mon[20207]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:40 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-30T23:20:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:40 smithi136 ceph-mon[20207]: osdmap e11: 2 total, 1 up, 2 in 2024-08-30T23:20:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:40 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:20:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:40 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.082 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[35847]: 2024-08-30T23:20:40.756+0000 7f66a9fb0640 -1 osd.1 0 waiting for initial osdmap 2024-08-30T23:20:41.082 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:20:40 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[35847]: 2024-08-30T23:20:40.761+0000 7f66a1dcf640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:20:41.334 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:41 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:20:41.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:41 smithi106 ceph-mon[16603]: osdmap e12: 2 total, 1 up, 2 in 2024-08-30T23:20:41.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:41 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' 2024-08-30T23:20:41.335 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:41 smithi106 ceph-mon[16603]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:41 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:20:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:41 smithi173 ceph-mon[20554]: osdmap e12: 2 total, 1 up, 2 in 2024-08-30T23:20:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:41 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' 2024-08-30T23:20:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:41 smithi173 ceph-mon[20554]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:41 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:20:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:41 smithi136 ceph-mon[20207]: osdmap e12: 2 total, 1 up, 2 in 2024-08-30T23:20:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:41 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252]' entity='osd.1' 2024-08-30T23:20:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:41 smithi136 ceph-mon[20207]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-30T23:20:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:42 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:20:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:42 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:20:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:42 smithi173 ceph-mon[20554]: osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252] boot 2024-08-30T23:20:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:42 smithi173 ceph-mon[20554]: osdmap e13: 2 total, 2 up, 2 in 2024-08-30T23:20:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:42 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:20:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:42 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:20:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:42 smithi106 ceph-mon[16603]: osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252] boot 2024-08-30T23:20:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:42 smithi106 ceph-mon[16603]: osdmap e13: 2 total, 2 up, 2 in 2024-08-30T23:20:42.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:42 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:20:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:42 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:20:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:42 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:42 smithi136 ceph-mon[20207]: osd.1 [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252] boot 2024-08-30T23:20:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:42 smithi136 ceph-mon[20207]: osdmap e13: 2 total, 2 up, 2 in 2024-08-30T23:20:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:42 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:20:44.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi106 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi106 to 8544M 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: osdmap e14: 2 total, 2 up, 2 in 2024-08-30T23:20:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:43 smithi136 ceph-mon[20207]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi106 2024-08-30T23:20:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi106 to 8544M 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: osdmap e14: 2 total, 2 up, 2 in 2024-08-30T23:20:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:43 smithi173 ceph-mon[20554]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi106 2024-08-30T23:20:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi106 to 8544M 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: osdmap e14: 2 total, 2 up, 2 in 2024-08-30T23:20:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:43 smithi106 ceph-mon[16603]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:44.157 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:20:44.186 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_2 2024-08-30T23:20:44.431 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:20:45.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:20:45.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:20:45.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:45.519 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:20:45.520 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:20:45.520 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:20:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:20:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:46 smithi173 ceph-mon[20554]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:20:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:46 smithi173 ceph-mon[20554]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:46.422 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:46 smithi106 ceph-mon[16603]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:20:46.422 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:46 smithi106 ceph-mon[16603]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:46.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:46 smithi136 ceph-mon[20207]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:20:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:46 smithi136 ceph-mon[20207]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:47.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:47 smithi106 ceph-mon[16603]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:47 smithi136 ceph-mon[20207]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:47 smithi173 ceph-mon[20554]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/156570708' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219"} : dispatch 2024-08-30T23:20:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/156570708' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219"}]': finished 2024-08-30T23:20:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:48 smithi106 ceph-mon[16603]: osdmap e15: 3 total, 2 up, 3 in 2024-08-30T23:20:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:48 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:20:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/996160461' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:20:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/156570708' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219"} : dispatch 2024-08-30T23:20:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/156570708' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219"}]': finished 2024-08-30T23:20:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:48 smithi136 ceph-mon[20207]: osdmap e15: 3 total, 2 up, 3 in 2024-08-30T23:20:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:48 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:20:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/996160461' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:20:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/156570708' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219"} : dispatch 2024-08-30T23:20:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/156570708' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219"}]': finished 2024-08-30T23:20:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:48 smithi173 ceph-mon[20554]: osdmap e15: 3 total, 2 up, 3 in 2024-08-30T23:20:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:48 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:20:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/996160461' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:20:49.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:49 smithi106 ceph-mon[16603]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:49 smithi136 ceph-mon[20207]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:49 smithi173 ceph-mon[20554]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:51.648 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:51 smithi106 ceph-mon[16603]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:51.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:51 smithi136 ceph-mon[20207]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:51 smithi173 ceph-mon[20554]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:53 smithi106 ceph-mon[16603]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:53 smithi136 ceph-mon[20207]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:53 smithi173 ceph-mon[20554]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-30T23:20:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-30T23:20:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-30T23:20:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:20:55.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:55 smithi106 ceph-mon[16603]: Deploying daemon osd.2 on smithi106 2024-08-30T23:20:55.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:55 smithi106 ceph-mon[16603]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:55 smithi136 ceph-mon[20207]: Deploying daemon osd.2 on smithi106 2024-08-30T23:20:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:55 smithi136 ceph-mon[20207]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:55 smithi173 ceph-mon[20554]: Deploying daemon osd.2 on smithi106 2024-08-30T23:20:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:55 smithi173 ceph-mon[20554]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:57 smithi106 ceph-mon[16603]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:57 smithi136 ceph-mon[20207]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:57 smithi173 ceph-mon[20554]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:20:59 smithi136 ceph-mon[20207]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:20:59 smithi173 ceph-mon[20554]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:20:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:20:59 smithi106 ceph-mon[16603]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:01 smithi106 ceph-mon[16603]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:01 smithi136 ceph-mon[20207]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:01 smithi173 ceph-mon[20554]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:03.277 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 2 on host 'smithi106' 2024-08-30T23:21:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:03.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:03.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:03.891 DEBUG:teuthology.orchestra.run.smithi106:osd.2> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.2.service 2024-08-30T23:21:03.894 INFO:tasks.cephadm:Deploying osd.3 on smithi106 with /dev/vg_nvme/lv_1... 2024-08-30T23:21:03.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-30T23:21:04.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:04.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:04 smithi106 ceph-mon[16603]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:04.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:04.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:04.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:04.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:04 smithi136 ceph-mon[20207]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:04 smithi173 ceph-mon[20554]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:04.857 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:04 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2[41630]: 2024-08-30T23:21:04.583+0000 7f93034ef640 -1 osd.2 0 log_to_monitors true 2024-08-30T23:21:05.134 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:21:05.333 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:05 smithi106 ceph-mon[16603]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-30T23:21:05.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:05 smithi136 ceph-mon[20207]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-30T23:21:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:05 smithi173 ceph-mon[20554]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-30T23:21:06.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:06 smithi136 ceph-mon[20207]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-30T23:21:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:06 smithi136 ceph-mon[20207]: osdmap e16: 3 total, 2 up, 3 in 2024-08-30T23:21:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:06 smithi136 ceph-mon[20207]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:21:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:06 smithi136 ceph-mon[20207]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:06 smithi173 ceph-mon[20554]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-30T23:21:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:06 smithi173 ceph-mon[20554]: osdmap e16: 3 total, 2 up, 3 in 2024-08-30T23:21:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:06 smithi173 ceph-mon[20554]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:21:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:06 smithi173 ceph-mon[20554]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-mon[16603]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-30T23:21:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-mon[16603]: osdmap e16: 3 total, 2 up, 3 in 2024-08-30T23:21:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-mon[16603]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:21:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-mon[16603]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:07.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2[41630]: 2024-08-30T23:21:06.690+0000 7f92ff472640 -1 osd.2 0 waiting for initial osdmap 2024-08-30T23:21:07.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:06 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2[41630]: 2024-08-30T23:21:06.693+0000 7f92f7291640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:21:07.436 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:21:07.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: osdmap e17: 3 total, 2 up, 3 in 2024-08-30T23:21:07.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' 2024-08-30T23:21:07.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.437 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi106 to 5696M 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:07.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:07.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:07.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:07.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: osdmap e17: 3 total, 2 up, 3 in 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi106 to 5696M 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:07.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: osdmap e17: 3 total, 2 up, 3 in 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052]' entity='osd.2' 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi106 to 5696M 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:07.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:07.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:08.286 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:21:08.287 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:21:08.287 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:08 smithi106 ceph-mon[16603]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:08.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:21:08.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:21:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:08 smithi136 ceph-mon[20207]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:08 smithi136 ceph-mon[20207]: osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052] boot 2024-08-30T23:21:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:08 smithi136 ceph-mon[20207]: osdmap e18: 3 total, 3 up, 3 in 2024-08-30T23:21:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:21:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:21:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:08 smithi173 ceph-mon[20554]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-30T23:21:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:08 smithi173 ceph-mon[20554]: osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052] boot 2024-08-30T23:21:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:08 smithi173 ceph-mon[20554]: osdmap e18: 3 total, 3 up, 3 in 2024-08-30T23:21:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:08.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:08 smithi106 ceph-mon[16603]: osd.2 [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052] boot 2024-08-30T23:21:08.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:08 smithi106 ceph-mon[16603]: osdmap e18: 3 total, 3 up, 3 in 2024-08-30T23:21:08.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-30T23:21:08.588 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:21:08.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi106:vg_nvme/lv_1 2024-08-30T23:21:08.859 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:21:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:10 smithi173 ceph-mon[20554]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:10 smithi173 ceph-mon[20554]: osdmap e19: 3 total, 3 up, 3 in 2024-08-30T23:21:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:10.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:21:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:10 smithi106 ceph-mon[16603]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:10 smithi106 ceph-mon[16603]: osdmap e19: 3 total, 3 up, 3 in 2024-08-30T23:21:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:21:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:10 smithi136 ceph-mon[20207]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:10 smithi136 ceph-mon[20207]: osdmap e19: 3 total, 3 up, 3 in 2024-08-30T23:21:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:21:11.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:11 smithi106 ceph-mon[16603]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:11.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:11 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-30T23:21:11.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:11 smithi106 ceph-mon[16603]: osdmap e20: 3 total, 3 up, 3 in 2024-08-30T23:21:11.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:11 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:21:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:11 smithi173 ceph-mon[20554]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:11 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-30T23:21:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:11 smithi173 ceph-mon[20554]: osdmap e20: 3 total, 3 up, 3 in 2024-08-30T23:21:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:11 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:21:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:11 smithi136 ceph-mon[20207]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:11 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-30T23:21:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:11 smithi136 ceph-mon[20207]: osdmap e20: 3 total, 3 up, 3 in 2024-08-30T23:21:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:11 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45661]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45661]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45664]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45664]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:12.297 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 30 23:21:11 smithi106 sudo[45664]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:12.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: osdmap e21: 3 total, 3 up, 3 in 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/895893100' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f"} : dispatch 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/895893100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f"}]': finished 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: osdmap e22: 4 total, 3 up, 4 in 2024-08-30T23:21:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: osdmap e21: 3 total, 3 up, 3 in 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/895893100' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f"} : dispatch 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/895893100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f"}]': finished 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: osdmap e22: 4 total, 3 up, 4 in 2024-08-30T23:21:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: osdmap e21: 3 total, 3 up, 3 in 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/895893100' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f"} : dispatch 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/895893100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f"}]': finished 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: osdmap e22: 4 total, 3 up, 4 in 2024-08-30T23:21:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45744]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45744]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45744]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45747]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45747]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45747]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45738]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45738]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45738]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45738]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.108 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45741]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.109 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45741]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45750]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45750]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45753]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45753]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:12 smithi106 sudo[45753]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 sudo[22523]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 sudo[22523]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 sudo[22523]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:12 smithi173 sudo[22523]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 sudo[22526]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 sudo[22526]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 sudo[22526]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 sudo[22526]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:13.299 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1825656773' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: osdmap e23: 4 total, 3 up, 4 in 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:13.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:13.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:13.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:13.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:13 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1825656773' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: osdmap e23: 4 total, 3 up, 4 in 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:13 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1825656773' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: osdmap e23: 4 total, 3 up, 4 in 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:13.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:13.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:13.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:13.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:13.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:13.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:13 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:14.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:14 smithi173 ceph-mon[20554]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:14.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:14 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:21:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:14 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:21:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:14 smithi173 ceph-mon[20554]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-30T23:21:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:14 smithi106 ceph-mon[16603]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:14 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:21:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:14 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:21:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:14 smithi106 ceph-mon[16603]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-30T23:21:14.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 ceph-mon[20207]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:14.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:21:14.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:21:14.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 ceph-mon[20207]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-30T23:21:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22258]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-30T23:21:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22258]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22258]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22258]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22261]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-30T23:21:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22261]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-30T23:21:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22261]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-30T23:21:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:14 smithi136 sudo[22261]: pam_unix(sudo:session): session closed for user root 2024-08-30T23:21:15.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:15 smithi106 ceph-mon[16603]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:15 smithi136 ceph-mon[20207]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:15 smithi173 ceph-mon[20554]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:17 smithi106 ceph-mon[16603]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:17 smithi136 ceph-mon[20207]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:17 smithi173 ceph-mon[20554]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:19.384 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:19 smithi106 ceph-mon[16603]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 244 KiB/s wr, 15 op/s 2024-08-30T23:21:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:19 smithi136 ceph-mon[20207]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 244 KiB/s wr, 15 op/s 2024-08-30T23:21:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:19 smithi173 ceph-mon[20554]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 244 KiB/s wr, 15 op/s 2024-08-30T23:21:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-30T23:21:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:20 smithi106 ceph-mon[16603]: Deploying daemon osd.3 on smithi106 2024-08-30T23:21:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-30T23:21:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:20 smithi136 ceph-mon[20207]: Deploying daemon osd.3 on smithi106 2024-08-30T23:21:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-30T23:21:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:20 smithi173 ceph-mon[20554]: Deploying daemon osd.3 on smithi106 2024-08-30T23:21:21.367 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:21 smithi106 ceph-mon[16603]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 186 KiB/s wr, 11 op/s 2024-08-30T23:21:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:21 smithi136 ceph-mon[20207]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 186 KiB/s wr, 11 op/s 2024-08-30T23:21:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:21 smithi173 ceph-mon[20554]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 186 KiB/s wr, 11 op/s 2024-08-30T23:21:23.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:23 smithi106 ceph-mon[16603]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 150 KiB/s wr, 9 op/s 2024-08-30T23:21:23.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:23 smithi136 ceph-mon[20207]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 150 KiB/s wr, 9 op/s 2024-08-30T23:21:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:23 smithi173 ceph-mon[20554]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 150 KiB/s wr, 9 op/s 2024-08-30T23:21:24.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:24.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:24.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:25.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:25 smithi136 ceph-mon[20207]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:26.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:25 smithi106 ceph-mon[16603]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:25 smithi173 ceph-mon[20554]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:27 smithi106 ceph-mon[16603]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:27 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:27 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:27 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:27.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:27 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:27 smithi136 ceph-mon[20207]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:27 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:27 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:27 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:27 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:27 smithi173 ceph-mon[20554]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:27 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:27 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:27 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:27 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:28.025 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 3 on host 'smithi106' 2024-08-30T23:21:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:28 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:28 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:28 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:28 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.660 DEBUG:teuthology.orchestra.run.smithi106:osd.3> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.3.service 2024-08-30T23:21:28.662 INFO:tasks.cephadm:Deploying osd.4 on smithi136 with /dev/vg_nvme/lv_4... 2024-08-30T23:21:28.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-30T23:21:28.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:28 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:28 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:28.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:28 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:28 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:28 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:28 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:28 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:28 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:28.913 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:21:29.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:29 smithi106 ceph-mon[16603]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:29.788 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:29 smithi136 ceph-mon[20207]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:29 smithi173 ceph-mon[20554]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 130 KiB/s wr, 8 op/s 2024-08-30T23:21:30.105 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 30 23:21:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-30T23:21:29.938+0000 7ff37479f640 -1 osd.3 0 log_to_monitors true 2024-08-30T23:21:30.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:30 smithi106 ceph-mon[16603]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-30T23:21:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:30 smithi136 ceph-mon[20207]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-30T23:21:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:30 smithi173 ceph-mon[20554]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-30T23:21:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:21:31.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-08-30T23:21:31.387 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:21:31.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:31 smithi136 ceph-mon[20207]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-30T23:21:31.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:31 smithi136 ceph-mon[20207]: osdmap e24: 4 total, 3 up, 4 in 2024-08-30T23:21:31.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:31 smithi136 ceph-mon[20207]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:21:31.657 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:31.658 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:31 smithi136 ceph-mon[20207]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:31 smithi173 ceph-mon[20554]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-30T23:21:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:31 smithi173 ceph-mon[20554]: osdmap e24: 4 total, 3 up, 4 in 2024-08-30T23:21:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:31 smithi173 ceph-mon[20554]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:21:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:31 smithi173 ceph-mon[20554]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-mon[16603]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-30T23:21:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-mon[16603]: osdmap e24: 4 total, 3 up, 4 in 2024-08-30T23:21:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-mon[16603]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:21:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-mon[16603]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:31.858 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-30T23:21:31.389+0000 7ff370722640 -1 osd.3 0 waiting for initial osdmap 2024-08-30T23:21:31.858 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 30 23:21:31 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-30T23:21:31.395+0000 7ff36c549640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: osdmap e25: 4 total, 3 up, 4 in 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.668 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi106 to 4272M 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.669 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.670 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.671 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: osdmap e25: 4 total, 3 up, 4 in 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi106 to 4272M 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-08-30T23:21:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: osdmap e25: 4 total, 3 up, 4 in 2024-08-30T23:21:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi106 to 4272M 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:32.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:32.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:21:32.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:21:32.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:21:32.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:32.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:32.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:32.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:33.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:21:33.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:21:33.662 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:33.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435] boot 2024-08-30T23:21:33.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: osdmap e26: 4 total, 4 up, 4 in 2024-08-30T23:21:33.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:33.663 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:33 smithi136 ceph-mon[20207]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:21:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:21:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435] boot 2024-08-30T23:21:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: osdmap e26: 4 total, 4 up, 4 in 2024-08-30T23:21:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:33 smithi173 ceph-mon[20554]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: osd.3 [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435] boot 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: osdmap e26: 4 total, 4 up, 4 in 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-30T23:21:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:33 smithi106 ceph-mon[16603]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-08-30T23:21:34.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:21:34.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:21:34.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: osdmap e27: 4 total, 4 up, 4 in 2024-08-30T23:21:34.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/875921822' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"} : dispatch 2024-08-30T23:21:34.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"} : dispatch 2024-08-30T23:21:34.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"}]': finished 2024-08-30T23:21:34.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: osdmap e28: 5 total, 4 up, 5 in 2024-08-30T23:21:34.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: osdmap e27: 4 total, 4 up, 4 in 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/875921822' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"} : dispatch 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"} : dispatch 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"}]': finished 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: osdmap e28: 5 total, 4 up, 5 in 2024-08-30T23:21:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: osdmap e27: 4 total, 4 up, 4 in 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/875921822' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"} : dispatch 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"} : dispatch 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729"}]': finished 2024-08-30T23:21:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: osdmap e28: 5 total, 4 up, 5 in 2024-08-30T23:21:34.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/2707099312' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:35 smithi136 ceph-mon[20207]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/2707099312' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:35 smithi173 ceph-mon[20554]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/2707099312' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:35 smithi106 ceph-mon[16603]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:37 smithi106 ceph-mon[16603]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:37 smithi136 ceph-mon[20207]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:37 smithi173 ceph-mon[20554]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:39.456 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:39 smithi136 ceph-mon[20207]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:39 smithi173 ceph-mon[20554]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:39 smithi106 ceph-mon[16603]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:41 smithi173 ceph-mon[20554]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-30T23:21:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:41 smithi106 ceph-mon[16603]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-30T23:21:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:41.732 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:41 smithi136 ceph-mon[20207]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:41.732 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-30T23:21:41.732 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:42.629 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:42 smithi136 ceph-mon[20207]: Deploying daemon osd.4 on smithi136 2024-08-30T23:21:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:42 smithi173 ceph-mon[20554]: Deploying daemon osd.4 on smithi136 2024-08-30T23:21:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:42 smithi106 ceph-mon[16603]: Deploying daemon osd.4 on smithi136 2024-08-30T23:21:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:43 smithi136 ceph-mon[20207]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:43 smithi173 ceph-mon[20554]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:43 smithi106 ceph-mon[16603]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:45 smithi106 ceph-mon[16603]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:45.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:45 smithi136 ceph-mon[20207]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:45.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:45 smithi173 ceph-mon[20554]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:46.802 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:46.802 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:46.802 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:47.763 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:47 smithi136 ceph-mon[20207]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:47 smithi173 ceph-mon[20554]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:47 smithi106 ceph-mon[16603]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:49.726 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 4 on host 'smithi136' 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:21:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:50.327 DEBUG:teuthology.orchestra.run.smithi136:osd.4> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.4.service 2024-08-30T23:21:50.330 INFO:tasks.cephadm:Deploying osd.5 on smithi136 with /dev/vg_nvme/lv_3... 2024-08-30T23:21:50.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-30T23:21:51.458 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:21:51.716 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:21:51.716 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:21:51.716 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:51 smithi136 ceph-mon[20207]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:51.717 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 30 23:21:51 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-30T23:21:51.429+0000 7fee2f95a640 -1 osd.4 0 log_to_monitors true 2024-08-30T23:21:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:21:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:21:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:51 smithi173 ceph-mon[20554]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:21:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:21:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:51 smithi106 ceph-mon[16603]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:52.516 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:52 smithi136 ceph-mon[20207]: from='osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-30T23:21:52.516 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:52 smithi136 ceph-mon[20207]: osdmap e29: 5 total, 4 up, 5 in 2024-08-30T23:21:52.516 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:52.516 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:52 smithi136 ceph-mon[20207]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-30T23:21:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:52 smithi173 ceph-mon[20554]: from='osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-30T23:21:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:52 smithi173 ceph-mon[20554]: osdmap e29: 5 total, 4 up, 5 in 2024-08-30T23:21:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:52 smithi173 ceph-mon[20554]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-30T23:21:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:52 smithi106 ceph-mon[16603]: from='osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-30T23:21:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:52 smithi106 ceph-mon[16603]: osdmap e29: 5 total, 4 up, 5 in 2024-08-30T23:21:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:52 smithi106 ceph-mon[16603]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-30T23:21:53.722 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-30T23:21:53.447+0000 7fee2b8dd640 -1 osd.4 0 waiting for initial osdmap 2024-08-30T23:21:53.722 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-30T23:21:53.453+0000 7fee27704640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:21:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-30T23:21:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:21:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: osdmap e30: 5 total, 4 up, 5 in 2024-08-30T23:21:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:21:53.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi136 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi136 to 17088M 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:53.723 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:53 smithi136 ceph-mon[20207]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:21:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: osdmap e30: 5 total, 4 up, 5 in 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi136 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi136 to 17088M 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:53.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:53.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:53 smithi173 ceph-mon[20554]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:21:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-30T23:21:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:21:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: osdmap e30: 5 total, 4 up, 5 in 2024-08-30T23:21:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:21:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi136 2024-08-30T23:21:53.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:21:53.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi136 to 17088M 2024-08-30T23:21:53.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:53.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:21:53.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:21:53.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-30T23:21:53.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:53 smithi106 ceph-mon[16603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:21:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: osdmap e31: 5 total, 4 up, 5 in 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341] boot 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: osdmap e32: 5 total, 5 up, 5 in 2024-08-30T23:21:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:21:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:21:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: osdmap e31: 5 total, 4 up, 5 in 2024-08-30T23:21:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341] boot 2024-08-30T23:21:54.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: osdmap e32: 5 total, 5 up, 5 in 2024-08-30T23:21:54.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:21:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:21:54.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: osdmap e31: 5 total, 4 up, 5 in 2024-08-30T23:21:54.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: osd.4 [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341] boot 2024-08-30T23:21:54.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: osdmap e32: 5 total, 5 up, 5 in 2024-08-30T23:21:54.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-30T23:21:54.860 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:21:54.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-08-30T23:21:55.130 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:21:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:55 smithi136 ceph-mon[20207]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:21:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:55 smithi136 ceph-mon[20207]: osdmap e33: 5 total, 5 up, 5 in 2024-08-30T23:21:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:55 smithi173 ceph-mon[20554]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:21:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:55 smithi173 ceph-mon[20554]: osdmap e33: 5 total, 5 up, 5 in 2024-08-30T23:21:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:55 smithi106 ceph-mon[16603]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:21:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:55 smithi106 ceph-mon[16603]: osdmap e33: 5 total, 5 up, 5 in 2024-08-30T23:21:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:56 smithi106 ceph-mon[16603]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:56 smithi106 ceph-mon[16603]: osdmap e34: 5 total, 5 up, 5 in 2024-08-30T23:21:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:56 smithi136 ceph-mon[20207]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:56 smithi136 ceph-mon[20207]: osdmap e34: 5 total, 5 up, 5 in 2024-08-30T23:21:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:56 smithi173 ceph-mon[20554]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:21:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:21:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:21:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:21:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:56 smithi173 ceph-mon[20554]: osdmap e34: 5 total, 5 up, 5 in 2024-08-30T23:21:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:57 smithi136 ceph-mon[20207]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:21:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:57 smithi173 ceph-mon[20554]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:21:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:57 smithi106 ceph-mon[16603]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:21:58.758 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/1948939052' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"} : dispatch 2024-08-30T23:21:58.758 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"} : dispatch 2024-08-30T23:21:58.758 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"}]': finished 2024-08-30T23:21:58.758 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:58 smithi136 ceph-mon[20207]: osdmap e35: 6 total, 5 up, 6 in 2024-08-30T23:21:58.758 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:58 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:21:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/1948939052' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"} : dispatch 2024-08-30T23:21:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"} : dispatch 2024-08-30T23:21:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"}]': finished 2024-08-30T23:21:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:58 smithi173 ceph-mon[20554]: osdmap e35: 6 total, 5 up, 6 in 2024-08-30T23:21:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:58 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:21:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/1948939052' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"} : dispatch 2024-08-30T23:21:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"} : dispatch 2024-08-30T23:21:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3"}]': finished 2024-08-30T23:21:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:58 smithi106 ceph-mon[16603]: osdmap e35: 6 total, 5 up, 6 in 2024-08-30T23:21:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:58 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:21:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/207437462' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:21:59 smithi136 ceph-mon[20207]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-30T23:21:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/207437462' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:21:59 smithi173 ceph-mon[20554]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-30T23:21:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/207437462' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:21:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:21:59 smithi106 ceph-mon[16603]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-08-30T23:22:01.707 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:01 smithi136 ceph-mon[20207]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:22:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:01 smithi173 ceph-mon[20554]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:22:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:01 smithi106 ceph-mon[16603]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:22:03.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:03 smithi136 ceph-mon[20207]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:22:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:03 smithi173 ceph-mon[20554]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:22:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:03 smithi106 ceph-mon[16603]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:22:05.458 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:05 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-30T23:22:05.458 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:05 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:05 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-30T23:22:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:05 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-30T23:22:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:06 smithi136 ceph-mon[20207]: Deploying daemon osd.5 on smithi136 2024-08-30T23:22:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:06 smithi136 ceph-mon[20207]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-30T23:22:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:06 smithi173 ceph-mon[20554]: Deploying daemon osd.5 on smithi136 2024-08-30T23:22:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:06 smithi173 ceph-mon[20554]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-30T23:22:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:06 smithi106 ceph-mon[16603]: Deploying daemon osd.5 on smithi136 2024-08-30T23:22:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:06 smithi106 ceph-mon[16603]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-30T23:22:08.447 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:22:08.448 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:22:08.448 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:08 smithi136 ceph-mon[20207]: osdmap e36: 6 total, 5 up, 6 in 2024-08-30T23:22:08.448 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:08.448 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:08 smithi136 ceph-mon[20207]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:22:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:22:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:08 smithi173 ceph-mon[20554]: osdmap e36: 6 total, 5 up, 6 in 2024-08-30T23:22:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:08 smithi173 ceph-mon[20554]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:22:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:22:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:08 smithi106 ceph-mon[16603]: osdmap e36: 6 total, 5 up, 6 in 2024-08-30T23:22:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:08 smithi106 ceph-mon[16603]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:09.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:09 smithi136 ceph-mon[20207]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:09 smithi173 ceph-mon[20554]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:09 smithi106 ceph-mon[16603]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:11.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:11 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:11.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:11 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:11.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:11 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:11.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:11 smithi136 ceph-mon[20207]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:11 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:11 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:11 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:11 smithi173 ceph-mon[20554]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:11 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:11 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:11.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:11 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:11.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:11 smithi106 ceph-mon[16603]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:13.523 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:13 smithi136 ceph-mon[20207]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:13.523 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 5 on host 'smithi136' 2024-08-30T23:22:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:13 smithi173 ceph-mon[20554]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:13 smithi106 ceph-mon[16603]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:14.236 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service 2024-08-30T23:22:14.238 INFO:tasks.cephadm:Deploying osd.6 on smithi136 with /dev/vg_nvme/lv_2... 2024-08-30T23:22:14.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-30T23:22:14.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:14.971 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:22:15.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:15 smithi136 ceph-mon[20207]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:15.548 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:22:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[30446]: 2024-08-30T23:22:15.494+0000 7f6ea4b66640 -1 osd.5 0 log_to_monitors true 2024-08-30T23:22:15.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:15 smithi106 ceph-mon[16603]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:15 smithi173 ceph-mon[20554]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:16.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:16 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:22:16.627 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:16 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:22:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:16 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:22:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:16 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:22:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:16 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:22:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:16 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:22:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:22:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-mon[20207]: osdmap e37: 6 total, 5 up, 6 in 2024-08-30T23:22:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-mon[20207]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:17.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[30446]: 2024-08-30T23:22:17.535+0000 7f6ea0ae9640 -1 osd.5 0 waiting for initial osdmap 2024-08-30T23:22:17.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:22:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[30446]: 2024-08-30T23:22:17.539+0000 7f6e9c910640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:22:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:17 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:22:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:17 smithi173 ceph-mon[20554]: osdmap e37: 6 total, 5 up, 6 in 2024-08-30T23:22:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:17 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:17 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:17 smithi173 ceph-mon[20554]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:17 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:22:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:17 smithi106 ceph-mon[16603]: osdmap e37: 6 total, 5 up, 6 in 2024-08-30T23:22:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:17 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:17 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:17 smithi106 ceph-mon[16603]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: osdmap e38: 6 total, 5 up, 6 in 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi136 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi136 to 8544M 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:18.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:22:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: osdmap e38: 6 total, 5 up, 6 in 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi136 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi136 to 8544M 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: osdmap e38: 6 total, 5 up, 6 in 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi136 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi136 to 8544M 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:18.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:19.134 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:22:19.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-08-30T23:22:19.404 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:22:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:19 smithi136 ceph-mon[20207]: osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833] boot 2024-08-30T23:22:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:19 smithi136 ceph-mon[20207]: osdmap e39: 6 total, 6 up, 6 in 2024-08-30T23:22:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:19 smithi136 ceph-mon[20207]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:19 smithi173 ceph-mon[20554]: osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833] boot 2024-08-30T23:22:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:19 smithi173 ceph-mon[20554]: osdmap e39: 6 total, 6 up, 6 in 2024-08-30T23:22:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:19 smithi173 ceph-mon[20554]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:19 smithi106 ceph-mon[16603]: osd.5 [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833] boot 2024-08-30T23:22:19.929 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:19 smithi106 ceph-mon[16603]: osdmap e39: 6 total, 6 up, 6 in 2024-08-30T23:22:19.929 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:22:19.929 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:19 smithi106 ceph-mon[16603]: pgmap v133: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:20.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:20 smithi136 ceph-mon[20207]: osdmap e40: 6 total, 6 up, 6 in 2024-08-30T23:22:20.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:20 smithi136 ceph-mon[20207]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:22:20.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:22:20.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:22:20.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:20.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:20 smithi136 ceph-mon[20207]: osdmap e41: 6 total, 6 up, 6 in 2024-08-30T23:22:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:20 smithi173 ceph-mon[20554]: osdmap e40: 6 total, 6 up, 6 in 2024-08-30T23:22:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:20 smithi173 ceph-mon[20554]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:22:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:22:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:22:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:20 smithi173 ceph-mon[20554]: osdmap e41: 6 total, 6 up, 6 in 2024-08-30T23:22:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:20 smithi106 ceph-mon[16603]: osdmap e40: 6 total, 6 up, 6 in 2024-08-30T23:22:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:20 smithi106 ceph-mon[16603]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:22:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:22:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:22:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:20 smithi106 ceph-mon[16603]: osdmap e41: 6 total, 6 up, 6 in 2024-08-30T23:22:21.748 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:21 smithi136 ceph-mon[20207]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:21 smithi173 ceph-mon[20554]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:21 smithi106 ceph-mon[16603]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/3158189890' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"} : dispatch 2024-08-30T23:22:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:22 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"} : dispatch 2024-08-30T23:22:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:22 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"}]': finished 2024-08-30T23:22:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:22 smithi173 ceph-mon[20554]: osdmap e42: 7 total, 6 up, 7 in 2024-08-30T23:22:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/3158189890' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"} : dispatch 2024-08-30T23:22:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:22 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"} : dispatch 2024-08-30T23:22:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:22 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"}]': finished 2024-08-30T23:22:22.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:22 smithi106 ceph-mon[16603]: osdmap e42: 7 total, 6 up, 7 in 2024-08-30T23:22:22.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:22.926 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/3158189890' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"} : dispatch 2024-08-30T23:22:22.926 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:22 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"} : dispatch 2024-08-30T23:22:22.926 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:22 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84"}]': finished 2024-08-30T23:22:22.926 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:22 smithi136 ceph-mon[20207]: osdmap e42: 7 total, 6 up, 7 in 2024-08-30T23:22:22.926 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/4223845042' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:22:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:23 smithi173 ceph-mon[20554]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:23 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:22:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:23 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:22:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/4223845042' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:22:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:23 smithi106 ceph-mon[16603]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:23 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:22:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:23 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:22:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/4223845042' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:22:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:23 smithi136 ceph-mon[20207]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:23 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:22:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:23 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:22:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:25 smithi136 ceph-mon[20207]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:22:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:25 smithi173 ceph-mon[20554]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:22:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:25 smithi106 ceph-mon[16603]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:22:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:27 smithi136 ceph-mon[20207]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-08-30T23:22:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:27 smithi173 ceph-mon[20554]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-08-30T23:22:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:27 smithi106 ceph-mon[16603]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-08-30T23:22:29.439 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:29 smithi136 ceph-mon[20207]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-08-30T23:22:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:29 smithi173 ceph-mon[20554]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-08-30T23:22:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:29 smithi106 ceph-mon[16603]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-08-30T23:22:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-30T23:22:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:30 smithi136 ceph-mon[20207]: Deploying daemon osd.6 on smithi136 2024-08-30T23:22:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-30T23:22:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:30 smithi173 ceph-mon[20554]: Deploying daemon osd.6 on smithi136 2024-08-30T23:22:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-30T23:22:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:30 smithi106 ceph-mon[16603]: Deploying daemon osd.6 on smithi136 2024-08-30T23:22:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:31 smithi136 ceph-mon[20207]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-30T23:22:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:31 smithi173 ceph-mon[20554]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-30T23:22:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:31 smithi106 ceph-mon[16603]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-30T23:22:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:33 smithi136 ceph-mon[20207]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-08-30T23:22:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:33 smithi173 ceph-mon[20554]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-08-30T23:22:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:33 smithi106 ceph-mon[16603]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 47 KiB/s, 0 objects/s recovering 2024-08-30T23:22:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:35.981 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:35 smithi136 ceph-mon[20207]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-30T23:22:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:35 smithi173 ceph-mon[20554]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-30T23:22:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:35 smithi106 ceph-mon[16603]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-30T23:22:36.995 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:22:36.995 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:22:36.995 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:22:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:22:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:22:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:22:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:22:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:22:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:22:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:37 smithi136 ceph-mon[20207]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:37 smithi173 ceph-mon[20554]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:37 smithi106 ceph-mon[16603]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.346 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 6 on host 'smithi136' 2024-08-30T23:22:38.962 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:38.963 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:38.963 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:38.963 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.963 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:38.963 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.963 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:38.989 DEBUG:teuthology.orchestra.run.smithi136:osd.6> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.6.service 2024-08-30T23:22:38.992 INFO:tasks.cephadm:Deploying osd.7 on smithi136 with /dev/vg_nvme/lv_1... 2024-08-30T23:22:38.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-30T23:22:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:39 smithi136 ceph-mon[20207]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:39 smithi136 ceph-mon[20207]: from='osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-30T23:22:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:39 smithi136 ceph-mon[20207]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-30T23:22:40.040 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 30 23:22:39 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-30T23:22:39.731+0000 7ff63daa3640 -1 osd.6 0 log_to_monitors true 2024-08-30T23:22:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:39 smithi173 ceph-mon[20554]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:39 smithi173 ceph-mon[20554]: from='osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-30T23:22:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:39 smithi173 ceph-mon[20554]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-30T23:22:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:39 smithi106 ceph-mon[16603]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:39 smithi106 ceph-mon[16603]: from='osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-30T23:22:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:39 smithi106 ceph-mon[16603]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-30T23:22:40.281 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:22:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:40 smithi136 ceph-mon[20207]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-30T23:22:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:40 smithi136 ceph-mon[20207]: osdmap e43: 7 total, 6 up, 7 in 2024-08-30T23:22:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:40 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:40 smithi136 ceph-mon[20207]: from='osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:40 smithi136 ceph-mon[20207]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:40 smithi173 ceph-mon[20554]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-30T23:22:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:40 smithi173 ceph-mon[20554]: osdmap e43: 7 total, 6 up, 7 in 2024-08-30T23:22:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:40 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:40 smithi173 ceph-mon[20554]: from='osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:40 smithi173 ceph-mon[20554]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:40 smithi106 ceph-mon[16603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-30T23:22:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:40 smithi106 ceph-mon[16603]: osdmap e43: 7 total, 6 up, 7 in 2024-08-30T23:22:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:40 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:40 smithi106 ceph-mon[16603]: from='osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:40 smithi106 ceph-mon[16603]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:22:41.286 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-30T23:22:41.032+0000 7ff639a26640 -1 osd.6 0 waiting for initial osdmap 2024-08-30T23:22:41.286 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-30T23:22:41.038+0000 7ff631845640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:22:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: osdmap e44: 7 total, 6 up, 7 in 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: from='osd.6 ' entity='osd.6' 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865] boot 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: osdmap e45: 7 total, 7 up, 7 in 2024-08-30T23:22:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:22:42.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: osdmap e44: 7 total, 6 up, 7 in 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: from='osd.6 ' entity='osd.6' 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865] boot 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: osdmap e45: 7 total, 7 up, 7 in 2024-08-30T23:22:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:22:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: osdmap e44: 7 total, 6 up, 7 in 2024-08-30T23:22:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: from='osd.6 ' entity='osd.6' 2024-08-30T23:22:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-08-30T23:22:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: osd.6 [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865] boot 2024-08-30T23:22:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: osdmap e45: 7 total, 7 up, 7 in 2024-08-30T23:22:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi136 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi136 to 5696M 2024-08-30T23:22:43.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:43.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:43.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: osdmap e46: 7 total, 7 up, 7 in 2024-08-30T23:22:43.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:43 smithi136 ceph-mon[20207]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:22:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi136 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi136 to 5696M 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: osdmap e46: 7 total, 7 up, 7 in 2024-08-30T23:22:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:43 smithi106 ceph-mon[16603]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:43.954 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-08-30T23:22:43.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-08-30T23:22:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi136 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi136 to 5696M 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:22:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:22:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: osdmap e46: 7 total, 7 up, 7 in 2024-08-30T23:22:44.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:43 smithi173 ceph-mon[20554]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:44.221 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:22:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:44 smithi136 ceph-mon[20207]: osdmap e47: 7 total, 7 up, 7 in 2024-08-30T23:22:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:44 smithi173 ceph-mon[20554]: osdmap e47: 7 total, 7 up, 7 in 2024-08-30T23:22:45.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:44 smithi106 ceph-mon[16603]: osdmap e47: 7 total, 7 up, 7 in 2024-08-30T23:22:45.938 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:45 smithi136 ceph-mon[20207]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:22:45.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:22:45.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:22:45.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:45.939 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:45 smithi136 ceph-mon[20207]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:45 smithi173 ceph-mon[20554]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:22:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:22:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:22:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:45 smithi173 ceph-mon[20554]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:45 smithi106 ceph-mon[16603]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:22:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:22:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:22:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:45 smithi106 ceph-mon[16603]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:22:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:22:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:46 smithi136 ceph-mon[20207]: osdmap e48: 7 total, 7 up, 7 in 2024-08-30T23:22:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:22:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:22:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:46 smithi173 ceph-mon[20554]: osdmap e48: 7 total, 7 up, 7 in 2024-08-30T23:22:47.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:22:47.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:22:47.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:46 smithi106 ceph-mon[16603]: osdmap e48: 7 total, 7 up, 7 in 2024-08-30T23:22:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/2641514383' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"} : dispatch 2024-08-30T23:22:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"} : dispatch 2024-08-30T23:22:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"}]': finished 2024-08-30T23:22:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: osdmap e49: 8 total, 7 up, 8 in 2024-08-30T23:22:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/813949795' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:22:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/2641514383' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"} : dispatch 2024-08-30T23:22:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"} : dispatch 2024-08-30T23:22:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"}]': finished 2024-08-30T23:22:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: osdmap e49: 8 total, 7 up, 8 in 2024-08-30T23:22:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/813949795' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:22:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/2641514383' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"} : dispatch 2024-08-30T23:22:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"} : dispatch 2024-08-30T23:22:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b"}]': finished 2024-08-30T23:22:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: osdmap e49: 8 total, 7 up, 8 in 2024-08-30T23:22:48.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:48.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:48.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/813949795' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:22:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:49 smithi136 ceph-mon[20207]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:49 smithi173 ceph-mon[20554]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:49 smithi106 ceph-mon[16603]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:51 smithi136 ceph-mon[20207]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:51 smithi173 ceph-mon[20554]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:51 smithi106 ceph-mon[16603]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:53.640 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:53 smithi136 ceph-mon[20207]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:53 smithi173 ceph-mon[20554]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:53 smithi106 ceph-mon[16603]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-30T23:22:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-30T23:22:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:55.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-30T23:22:55.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:22:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:55 smithi173 ceph-mon[20554]: Deploying daemon osd.7 on smithi136 2024-08-30T23:22:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:55 smithi173 ceph-mon[20554]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:55 smithi106 ceph-mon[16603]: Deploying daemon osd.7 on smithi136 2024-08-30T23:22:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:55 smithi106 ceph-mon[16603]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:56.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:55 smithi136 ceph-mon[20207]: Deploying daemon osd.7 on smithi136 2024-08-30T23:22:56.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:55 smithi136 ceph-mon[20207]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:22:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:22:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:56 smithi173 ceph-mon[20554]: osdmap e50: 8 total, 7 up, 8 in 2024-08-30T23:22:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:22:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:22:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:56 smithi106 ceph-mon[16603]: osdmap e50: 8 total, 7 up, 8 in 2024-08-30T23:22:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:22:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:22:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:56 smithi136 ceph-mon[20207]: osdmap e50: 8 total, 7 up, 8 in 2024-08-30T23:22:57.241 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:57.968 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:22:57.968 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:22:57.968 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:57 smithi136 ceph-mon[20207]: osdmap e51: 8 total, 7 up, 8 in 2024-08-30T23:22:57.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:57.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:57 smithi136 ceph-mon[20207]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:22:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:22:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:57 smithi106 ceph-mon[16603]: osdmap e51: 8 total, 7 up, 8 in 2024-08-30T23:22:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:57 smithi106 ceph-mon[16603]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:22:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:22:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:57 smithi173 ceph-mon[20554]: osdmap e51: 8 total, 7 up, 8 in 2024-08-30T23:22:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:22:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:57 smithi173 ceph-mon[20554]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:59.640 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:22:59 smithi136 ceph-mon[20207]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:22:59 smithi173 ceph-mon[20554]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:22:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:22:59 smithi106 ceph-mon[16603]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:00.605 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:00.605 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:00.605 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:01.652 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:01 smithi136 ceph-mon[20207]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:01 smithi173 ceph-mon[20554]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:01 smithi106 ceph-mon[16603]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:03.259 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 7 on host 'smithi136' 2024-08-30T23:23:03.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:03.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:03.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:03.900 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:03.901 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:03.902 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:03 smithi136 ceph-mon[20207]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:03.903 DEBUG:teuthology.orchestra.run.smithi136:osd.7> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7.service 2024-08-30T23:23:03.905 INFO:tasks.cephadm:Deploying osd.8 on smithi173 with /dev/vg_nvme/lv_4... 2024-08-30T23:23:03.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-30T23:23:04.155 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:23:04.275 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:04.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:04.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:04.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:04.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.277 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:03 smithi173 ceph-mon[20554]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:04.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:03 smithi106 ceph-mon[16603]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:05.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:05 smithi136 ceph-mon[20207]: from='osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-30T23:23:05.399 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:05 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-30T23:23:05.399 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:23:05 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[42416]: 2024-08-30T23:23:05.178+0000 7f7eed341640 -1 osd.7 0 log_to_monitors true 2024-08-30T23:23:05.552 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:05 smithi173 ceph-mon[20554]: from='osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-30T23:23:05.552 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:05 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-30T23:23:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:05 smithi106 ceph-mon[16603]: from='osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-30T23:23:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:05 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-30T23:23:06.534 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[42416]: 2024-08-30T23:23:06.267+0000 7f7ee92c4640 -1 osd.7 0 waiting for initial osdmap 2024-08-30T23:23:06.534 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[42416]: 2024-08-30T23:23:06.275+0000 7f7ee50eb640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:23:06.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-30T23:23:06.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: osdmap e52: 8 total, 7 up, 8 in 2024-08-30T23:23:06.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:06.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: from='osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:23:06.535 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:23:06.535 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:06.535 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:23:06.535 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: osdmap e53: 8 total, 7 up, 8 in 2024-08-30T23:23:06.535 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: osdmap e52: 8 total, 7 up, 8 in 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: from='osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: osdmap e53: 8 total, 7 up, 8 in 2024-08-30T23:23:06.565 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:06.568 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:23:06.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: osdmap e52: 8 total, 7 up, 8 in 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: from='osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: osdmap e53: 8 total, 7 up, 8 in 2024-08-30T23:23:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:06.845 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:23:07.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513] boot 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: osdmap e54: 8 total, 8 up, 8 in 2024-08-30T23:23:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:07.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:07.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513] boot 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: osdmap e54: 8 total, 8 up, 8 in 2024-08-30T23:23:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: osd.7 [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513] boot 2024-08-30T23:23:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: osdmap e54: 8 total, 8 up, 8 in 2024-08-30T23:23:07.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-30T23:23:08.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi136 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi136 to 4272M 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:08 smithi136 ceph-mon[20207]: osdmap e55: 8 total, 8 up, 8 in 2024-08-30T23:23:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi136 2024-08-30T23:23:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi136 to 4272M 2024-08-30T23:23:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:08 smithi173 ceph-mon[20554]: osdmap e55: 8 total, 8 up, 8 in 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi136 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi136 to 4272M 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:08.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:08 smithi106 ceph-mon[16603]: osdmap e55: 8 total, 8 up, 8 in 2024-08-30T23:23:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:09 smithi173 ceph-mon[20554]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:09 smithi173 ceph-mon[20554]: osdmap e56: 8 total, 8 up, 8 in 2024-08-30T23:23:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:09 smithi136 ceph-mon[20207]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:09 smithi136 ceph-mon[20207]: osdmap e56: 8 total, 8 up, 8 in 2024-08-30T23:23:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:09 smithi106 ceph-mon[16603]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:09 smithi106 ceph-mon[16603]: osdmap e56: 8 total, 8 up, 8 in 2024-08-30T23:23:10.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:10 smithi136 ceph-mon[20207]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:10.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/3024878804' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"} : dispatch 2024-08-30T23:23:10.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:10 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"} : dispatch 2024-08-30T23:23:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:10 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"}]': finished 2024-08-30T23:23:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:10 smithi136 ceph-mon[20207]: osdmap e57: 9 total, 8 up, 9 in 2024-08-30T23:23:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:10 smithi173 ceph-mon[20554]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/3024878804' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"} : dispatch 2024-08-30T23:23:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:10 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"} : dispatch 2024-08-30T23:23:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:10 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"}]': finished 2024-08-30T23:23:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:10 smithi173 ceph-mon[20554]: osdmap e57: 9 total, 8 up, 9 in 2024-08-30T23:23:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:10 smithi106 ceph-mon[16603]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/3024878804' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"} : dispatch 2024-08-30T23:23:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:10 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"} : dispatch 2024-08-30T23:23:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:10 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "046e019e-1e33-403c-9577-757c7d546f83"}]': finished 2024-08-30T23:23:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:10 smithi106 ceph-mon[16603]: osdmap e57: 9 total, 8 up, 9 in 2024-08-30T23:23:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:11.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:11 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:23:11.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/2794784046' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:23:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:11 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:23:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/2794784046' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:23:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:11 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:23:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2794784046' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:23:12.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:12 smithi173 ceph-mon[20554]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:12 smithi173 ceph-mon[20554]: osdmap e58: 9 total, 8 up, 9 in 2024-08-30T23:23:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:12 smithi106 ceph-mon[16603]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:12 smithi106 ceph-mon[16603]: osdmap e58: 9 total, 8 up, 9 in 2024-08-30T23:23:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:12 smithi136 ceph-mon[20207]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:12 smithi136 ceph-mon[20207]: osdmap e58: 9 total, 8 up, 9 in 2024-08-30T23:23:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:13 smithi173 ceph-mon[20554]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-08-30T23:23:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:13 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-30T23:23:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:13 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:23:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:13 smithi136 ceph-mon[20207]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-08-30T23:23:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:13 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-30T23:23:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:13 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:23:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:13 smithi106 ceph-mon[16603]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-08-30T23:23:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:13 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-30T23:23:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:13 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:23:15.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:15 smithi173 ceph-mon[20554]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:23:15.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:15 smithi106 ceph-mon[16603]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:23:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:15 smithi136 ceph-mon[20207]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:23:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-30T23:23:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-30T23:23:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-30T23:23:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:18.097 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:17 smithi173 ceph-mon[20554]: Deploying daemon osd.8 on smithi173 2024-08-30T23:23:18.098 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:17 smithi173 ceph-mon[20554]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:23:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:17 smithi106 ceph-mon[16603]: Deploying daemon osd.8 on smithi173 2024-08-30T23:23:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:17 smithi106 ceph-mon[16603]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:23:18.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:17 smithi136 ceph-mon[20207]: Deploying daemon osd.8 on smithi173 2024-08-30T23:23:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:17 smithi136 ceph-mon[20207]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:23:19.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:19 smithi173 ceph-mon[20554]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-08-30T23:23:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:19 smithi136 ceph-mon[20207]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-08-30T23:23:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:19 smithi106 ceph-mon[16603]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-08-30T23:23:21.722 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:21 smithi173 ceph-mon[20554]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-30T23:23:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:21 smithi136 ceph-mon[20207]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-30T23:23:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:21 smithi106 ceph-mon[16603]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-30T23:23:22.624 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:22.624 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:22.624 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:23.501 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:23 smithi173 ceph-mon[20554]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-08-30T23:23:23.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:23 smithi136 ceph-mon[20207]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-08-30T23:23:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:23 smithi106 ceph-mon[16603]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s, 0 objects/s recovering 2024-08-30T23:23:25.342 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 8 on host 'smithi173' 2024-08-30T23:23:25.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:25.961 DEBUG:teuthology.orchestra.run.smithi173:osd.8> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.8.service 2024-08-30T23:23:25.963 INFO:tasks.cephadm:Deploying osd.9 on smithi173 with /dev/vg_nvme/lv_3... 2024-08-30T23:23:25.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-30T23:23:26.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:26 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:26 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:26 smithi173 ceph-mon[20554]: osdmap e59: 9 total, 8 up, 9 in 2024-08-30T23:23:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:26 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:26 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:26 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:26 smithi136 ceph-mon[20207]: osdmap e59: 9 total, 8 up, 9 in 2024-08-30T23:23:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:26 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:26 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:26 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:26 smithi106 ceph-mon[16603]: osdmap e59: 9 total, 8 up, 9 in 2024-08-30T23:23:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:26 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:27.280 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 30 23:23:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-30T23:23:26.979+0000 7f71d7351640 -1 osd.8 0 log_to_monitors true 2024-08-30T23:23:27.487 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:23:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:27 smithi173 ceph-mon[20554]: from='osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-30T23:23:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:27 smithi173 ceph-mon[20554]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-30T23:23:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:27 smithi173 ceph-mon[20554]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:27 smithi136 ceph-mon[20207]: from='osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-30T23:23:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:27 smithi136 ceph-mon[20207]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-30T23:23:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:27 smithi136 ceph-mon[20207]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:27 smithi106 ceph-mon[16603]: from='osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-30T23:23:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:27 smithi106 ceph-mon[16603]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-30T23:23:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:27 smithi106 ceph-mon[16603]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:28.434 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-mon[20554]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-30T23:23:28.434 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-mon[20554]: osdmap e60: 9 total, 8 up, 9 in 2024-08-30T23:23:28.434 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:28.434 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-mon[20554]: from='osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:28.434 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-mon[20554]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:28 smithi136 ceph-mon[20207]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-30T23:23:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:28 smithi136 ceph-mon[20207]: osdmap e60: 9 total, 8 up, 9 in 2024-08-30T23:23:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:28 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:28 smithi136 ceph-mon[20207]: from='osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:28 smithi136 ceph-mon[20207]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:28 smithi106 ceph-mon[16603]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-30T23:23:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:28 smithi106 ceph-mon[16603]: osdmap e60: 9 total, 8 up, 9 in 2024-08-30T23:23:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:28 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:28 smithi106 ceph-mon[16603]: from='osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:28 smithi106 ceph-mon[16603]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:29.073 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-30T23:23:28.815+0000 7f71d32d4640 -1 osd.8 0 waiting for initial osdmap 2024-08-30T23:23:29.073 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 30 23:23:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-30T23:23:28.821+0000 7f71cb0f3640 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:23:29.459 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:23:29.459 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: osdmap e61: 9 total, 8 up, 9 in 2024-08-30T23:23:29.459 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:29.459 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:29.459 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi173 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi173 to 21184M 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: from='osd.8 ' entity='osd.8' 2024-08-30T23:23:29.460 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:29 smithi173 ceph-mon[20554]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: osdmap e61: 9 total, 8 up, 9 in 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi173 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi173 to 21184M 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: from='osd.8 ' entity='osd.8' 2024-08-30T23:23:29.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:29 smithi136 ceph-mon[20207]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:23:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: osdmap e61: 9 total, 8 up, 9 in 2024-08-30T23:23:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi173 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi173 to 21184M 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: from='osd.8 ' entity='osd.8' 2024-08-30T23:23:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:29 smithi106 ceph-mon[16603]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-08-30T23:23:30.498 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:23:30.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-08-30T23:23:30.765 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:23:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:30 smithi136 ceph-mon[20207]: osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876] boot 2024-08-30T23:23:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:30 smithi136 ceph-mon[20207]: osdmap e62: 9 total, 9 up, 9 in 2024-08-30T23:23:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:30 smithi173 ceph-mon[20554]: osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876] boot 2024-08-30T23:23:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:30 smithi173 ceph-mon[20554]: osdmap e62: 9 total, 9 up, 9 in 2024-08-30T23:23:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:30 smithi106 ceph-mon[16603]: osd.8 [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876] boot 2024-08-30T23:23:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:30 smithi106 ceph-mon[16603]: osdmap e62: 9 total, 9 up, 9 in 2024-08-30T23:23:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-30T23:23:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:31 smithi136 ceph-mon[20207]: osdmap e63: 9 total, 9 up, 9 in 2024-08-30T23:23:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:31 smithi136 ceph-mon[20207]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:31.799 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:31 smithi173 ceph-mon[20554]: osdmap e63: 9 total, 9 up, 9 in 2024-08-30T23:23:31.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:31 smithi173 ceph-mon[20554]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:31.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:31.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:31.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:31 smithi106 ceph-mon[16603]: osdmap e63: 9 total, 9 up, 9 in 2024-08-30T23:23:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:31 smithi106 ceph-mon[16603]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:32.714 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:32 smithi173 ceph-mon[20554]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:32.714 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:32 smithi173 ceph-mon[20554]: osdmap e64: 9 total, 9 up, 9 in 2024-08-30T23:23:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:32 smithi136 ceph-mon[20207]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:32 smithi136 ceph-mon[20207]: osdmap e64: 9 total, 9 up, 9 in 2024-08-30T23:23:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:32 smithi106 ceph-mon[16603]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:32 smithi106 ceph-mon[16603]: osdmap e64: 9 total, 9 up, 9 in 2024-08-30T23:23:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:33 smithi136 ceph-mon[20207]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-08-30T23:23:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:33 smithi173 ceph-mon[20554]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-08-30T23:23:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:33 smithi106 ceph-mon[16603]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-08-30T23:23:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/572750120' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"} : dispatch 2024-08-30T23:23:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"} : dispatch 2024-08-30T23:23:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"}]': finished 2024-08-30T23:23:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:34 smithi136 ceph-mon[20207]: osdmap e65: 10 total, 9 up, 10 in 2024-08-30T23:23:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/219720964' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:23:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/572750120' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"} : dispatch 2024-08-30T23:23:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"} : dispatch 2024-08-30T23:23:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"}]': finished 2024-08-30T23:23:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:34 smithi173 ceph-mon[20554]: osdmap e65: 10 total, 9 up, 10 in 2024-08-30T23:23:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/219720964' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:23:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/572750120' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"} : dispatch 2024-08-30T23:23:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"} : dispatch 2024-08-30T23:23:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c"}]': finished 2024-08-30T23:23:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:34 smithi106 ceph-mon[16603]: osdmap e65: 10 total, 9 up, 10 in 2024-08-30T23:23:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/219720964' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:23:35.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:35 smithi173 ceph-mon[20554]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-30T23:23:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:35 smithi136 ceph-mon[20207]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-30T23:23:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:35 smithi106 ceph-mon[16603]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-08-30T23:23:37.913 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:37 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:37.914 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:37 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:37.914 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:37 smithi173 ceph-mon[20554]: osdmap e66: 10 total, 9 up, 10 in 2024-08-30T23:23:37.914 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:37.914 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:37 smithi173 ceph-mon[20554]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:23:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:37 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:37 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:37 smithi136 ceph-mon[20207]: osdmap e66: 10 total, 9 up, 10 in 2024-08-30T23:23:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:37 smithi136 ceph-mon[20207]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:23:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:37 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:37 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:37 smithi106 ceph-mon[16603]: osdmap e66: 10 total, 9 up, 10 in 2024-08-30T23:23:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:37 smithi106 ceph-mon[16603]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-30T23:23:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:39 smithi173 ceph-mon[20554]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:23:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:39 smithi136 ceph-mon[20207]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:23:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:39 smithi106 ceph-mon[16603]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:23:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-30T23:23:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-30T23:23:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-30T23:23:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:42.429 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:42 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:23:42.429 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:42 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:23:42.429 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:42 smithi173 ceph-mon[20554]: Deploying daemon osd.9 on smithi173 2024-08-30T23:23:42.429 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:42 smithi173 ceph-mon[20554]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:42 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:23:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:42 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:23:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:42 smithi136 ceph-mon[20207]: Deploying daemon osd.9 on smithi173 2024-08-30T23:23:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:42 smithi136 ceph-mon[20207]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:42 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:23:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:42 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:23:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:42 smithi106 ceph-mon[16603]: Deploying daemon osd.9 on smithi173 2024-08-30T23:23:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:42 smithi106 ceph-mon[16603]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:43.653 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:43 smithi173 ceph-mon[20554]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:43 smithi136 ceph-mon[20207]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:43 smithi106 ceph-mon[16603]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:45.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:45 smithi106 ceph-mon[16603]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:45.714 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:45 smithi173 ceph-mon[20554]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:45.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:45 smithi136 ceph-mon[20207]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:46.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:47.756 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:47 smithi173 ceph-mon[20554]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:47 smithi136 ceph-mon[20207]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:47 smithi106 ceph-mon[16603]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:48.682 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:48.682 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:48.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:48.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:49.668 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: osdmap e67: 10 total, 9 up, 10 in 2024-08-30T23:23:49.668 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:49.668 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.668 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.668 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:49.669 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:49.669 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.669 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:49 smithi173 ceph-mon[20554]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: osdmap e67: 10 total, 9 up, 10 in 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:49 smithi136 ceph-mon[20207]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: osdmap e67: 10 total, 9 up, 10 in 2024-08-30T23:23:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:49.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:49.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:49.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:49 smithi106 ceph-mon[16603]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:50.297 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 9 on host 'smithi173' 2024-08-30T23:23:50.946 DEBUG:teuthology.orchestra.run.smithi173:osd.9> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.9.service 2024-08-30T23:23:50.949 INFO:tasks.cephadm:Deploying osd.10 on smithi173 with /dev/vg_nvme/lv_2... 2024-08-30T23:23:50.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-30T23:23:51.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:51.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:51.189 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:23:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:51.663 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:23:51.783 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 30 23:23:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-30T23:23:51.550+0000 7f569af40640 -1 osd.9 0 log_to_monitors true 2024-08-30T23:23:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:51 smithi173 ceph-mon[20554]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:51 smithi173 ceph-mon[20554]: from='osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-30T23:23:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:51 smithi173 ceph-mon[20554]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-30T23:23:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:51 smithi136 ceph-mon[20207]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:51 smithi136 ceph-mon[20207]: from='osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-30T23:23:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:51 smithi136 ceph-mon[20207]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-30T23:23:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:51 smithi106 ceph-mon[16603]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:51 smithi106 ceph-mon[16603]: from='osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-30T23:23:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:51 smithi106 ceph-mon[16603]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-30T23:23:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:53 smithi136 ceph-mon[20207]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-30T23:23:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:53 smithi136 ceph-mon[20207]: osdmap e68: 10 total, 9 up, 10 in 2024-08-30T23:23:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:53 smithi136 ceph-mon[20207]: from='osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:53 smithi136 ceph-mon[20207]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-mon[20554]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-30T23:23:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-mon[20554]: osdmap e68: 10 total, 9 up, 10 in 2024-08-30T23:23:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-mon[20554]: from='osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-mon[20554]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:53 smithi106 ceph-mon[16603]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-30T23:23:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:53 smithi106 ceph-mon[16603]: osdmap e68: 10 total, 9 up, 10 in 2024-08-30T23:23:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:53 smithi106 ceph-mon[16603]: from='osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:53 smithi106 ceph-mon[16603]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:23:53.823 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-30T23:23:53.404+0000 7f5696ec3640 -1 osd.9 0 waiting for initial osdmap 2024-08-30T23:23:53.824 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 30 23:23:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-30T23:23:53.412+0000 7f568ece2640 -1 osd.9 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:23:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:54 smithi136 ceph-mon[20207]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:23:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:54 smithi136 ceph-mon[20207]: osdmap e69: 10 total, 9 up, 10 in 2024-08-30T23:23:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:54 smithi136 ceph-mon[20207]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:54 smithi136 ceph-mon[20207]: from='osd.9 ' entity='osd.9' 2024-08-30T23:23:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:54 smithi173 ceph-mon[20554]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:23:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:54 smithi173 ceph-mon[20554]: osdmap e69: 10 total, 9 up, 10 in 2024-08-30T23:23:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:54 smithi173 ceph-mon[20554]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:54 smithi173 ceph-mon[20554]: from='osd.9 ' entity='osd.9' 2024-08-30T23:23:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:54 smithi106 ceph-mon[16603]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:23:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:54 smithi106 ceph-mon[16603]: osdmap e69: 10 total, 9 up, 10 in 2024-08-30T23:23:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:54 smithi106 ceph-mon[16603]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-30T23:23:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:54 smithi106 ceph-mon[16603]: from='osd.9 ' entity='osd.9' 2024-08-30T23:23:55.140 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222] boot 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: osdmap e70: 10 total, 10 up, 10 in 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.141 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:55.142 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:55.142 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222] boot 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: osdmap e70: 10 total, 10 up, 10 in 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:55.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:55.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: osd.9 [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222] boot 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: osdmap e70: 10 total, 10 up, 10 in 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-30T23:23:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:23:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:23:56.196 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:56 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi173 2024-08-30T23:23:56.196 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:56 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi173 to 10592M 2024-08-30T23:23:56.196 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:56 smithi173 ceph-mon[20554]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:23:56.197 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:56 smithi173 ceph-mon[20554]: osdmap e71: 10 total, 10 up, 10 in 2024-08-30T23:23:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:23:56.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-08-30T23:23:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:56 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi173 2024-08-30T23:23:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:56 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi173 to 10592M 2024-08-30T23:23:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:56 smithi136 ceph-mon[20207]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:23:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:56 smithi136 ceph-mon[20207]: osdmap e71: 10 total, 10 up, 10 in 2024-08-30T23:23:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:56 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi173 2024-08-30T23:23:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:56 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi173 to 10592M 2024-08-30T23:23:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:56 smithi106 ceph-mon[16603]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:23:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:56 smithi106 ceph-mon[16603]: osdmap e71: 10 total, 10 up, 10 in 2024-08-30T23:23:56.466 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:23:57.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:57.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:57.503 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:23:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:23:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:23:58.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:58 smithi173 ceph-mon[20554]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:58.414 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:58 smithi173 ceph-mon[20554]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:23:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:58 smithi136 ceph-mon[20207]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:58 smithi136 ceph-mon[20207]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:23:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:58 smithi106 ceph-mon[16603]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:23:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:58 smithi106 ceph-mon[16603]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:23:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:59 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:23:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:23:59 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:23:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:59 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:23:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:23:59 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:23:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:59 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:23:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:23:59 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: osdmap e72: 10 total, 10 up, 10 in 2024-08-30T23:24:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/2832286330' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"} : dispatch 2024-08-30T23:24:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"} : dispatch 2024-08-30T23:24:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"}]': finished 2024-08-30T23:24:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: osdmap e73: 11 total, 10 up, 11 in 2024-08-30T23:24:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/2227820396' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:24:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: osdmap e72: 10 total, 10 up, 10 in 2024-08-30T23:24:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/2832286330' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"} : dispatch 2024-08-30T23:24:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"} : dispatch 2024-08-30T23:24:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"}]': finished 2024-08-30T23:24:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: osdmap e73: 11 total, 10 up, 11 in 2024-08-30T23:24:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/2227820396' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: osdmap e72: 10 total, 10 up, 10 in 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2832286330' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"} : dispatch 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"} : dispatch 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2"}]': finished 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: osdmap e73: 11 total, 10 up, 11 in 2024-08-30T23:24:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:00.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/2227820396' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:24:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:01 smithi136 ceph-mon[20207]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:01 smithi173 ceph-mon[20554]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:01 smithi106 ceph-mon[16603]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:03.512 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:03 smithi173 ceph-mon[20554]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:03 smithi136 ceph-mon[20207]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:03 smithi106 ceph-mon[16603]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:05 smithi173 ceph-mon[20554]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:05 smithi136 ceph-mon[20207]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:05 smithi106 ceph-mon[16603]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-30T23:24:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-30T23:24:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:07.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-30T23:24:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:07 smithi106 ceph-mon[16603]: Deploying daemon osd.10 on smithi173 2024-08-30T23:24:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:07 smithi106 ceph-mon[16603]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:08.206 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:07 smithi173 ceph-mon[20554]: Deploying daemon osd.10 on smithi173 2024-08-30T23:24:08.206 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:07 smithi173 ceph-mon[20554]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:08.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:07 smithi136 ceph-mon[20207]: Deploying daemon osd.10 on smithi173 2024-08-30T23:24:08.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:07 smithi136 ceph-mon[20207]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:09.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:09 smithi173 ceph-mon[20554]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:09 smithi136 ceph-mon[20207]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:09 smithi106 ceph-mon[16603]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:11.675 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:11 smithi173 ceph-mon[20554]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:11 smithi136 ceph-mon[20207]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:11 smithi106 ceph-mon[16603]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:12.712 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:12.712 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:12.713 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:13.522 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:13 smithi173 ceph-mon[20554]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:13 smithi136 ceph-mon[20207]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:13 smithi106 ceph-mon[16603]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:15.125 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 10 on host 'smithi173' 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.755 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:15.756 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:15.756 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:15 smithi173 ceph-mon[20554]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:15.787 DEBUG:teuthology.orchestra.run.smithi173:osd.10> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.10.service 2024-08-30T23:24:15.789 INFO:tasks.cephadm:Deploying osd.11 on smithi173 with /dev/vg_nvme/lv_1... 2024-08-30T23:24:15.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:15 smithi136 ceph-mon[20207]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:15 smithi106 ceph-mon[16603]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:16.439 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:24:17.431 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:17 smithi173 ceph-mon[20554]: osdmap e74: 11 total, 10 up, 11 in 2024-08-30T23:24:17.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:17.432 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 30 23:24:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-30T23:24:17.254+0000 7f6022c74640 -1 osd.10 0 log_to_monitors true 2024-08-30T23:24:17.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:17 smithi136 ceph-mon[20207]: osdmap e74: 11 total, 10 up, 11 in 2024-08-30T23:24:17.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:17 smithi106 ceph-mon[16603]: osdmap e74: 11 total, 10 up, 11 in 2024-08-30T23:24:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: osdmap e75: 11 total, 10 up, 11 in 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: from='osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-30T23:24:18.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:18 smithi173 ceph-mon[20554]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:18.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: osdmap e75: 11 total, 10 up, 11 in 2024-08-30T23:24:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: from='osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-30T23:24:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-30T23:24:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:18 smithi136 ceph-mon[20207]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: osdmap e75: 11 total, 10 up, 11 in 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: from='osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-30T23:24:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:18 smithi106 ceph-mon[16603]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:19.525 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-30T23:24:19.161+0000 7f601ebf7640 -1 osd.10 0 waiting for initial osdmap 2024-08-30T23:24:19.525 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-30T23:24:19.167+0000 7f601aa1e640 -1 osd.10 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:24:19.525 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-mon[20554]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-30T23:24:19.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-mon[20554]: from='osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:19.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-mon[20554]: osdmap e76: 11 total, 10 up, 11 in 2024-08-30T23:24:19.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:19.526 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:19 smithi173 ceph-mon[20554]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:19 smithi136 ceph-mon[20207]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-30T23:24:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:19 smithi136 ceph-mon[20207]: from='osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:19 smithi136 ceph-mon[20207]: osdmap e76: 11 total, 10 up, 11 in 2024-08-30T23:24:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:19 smithi136 ceph-mon[20207]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:19 smithi106 ceph-mon[16603]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-30T23:24:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:19 smithi106 ceph-mon[16603]: from='osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:19 smithi106 ceph-mon[16603]: osdmap e76: 11 total, 10 up, 11 in 2024-08-30T23:24:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:19 smithi106 ceph-mon[16603]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: osdmap e77: 11 total, 10 up, 11 in 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:20.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: osdmap e77: 11 total, 10 up, 11 in 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:20.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: osdmap e77: 11 total, 10 up, 11 in 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:20.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:21.128 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-08-30T23:24:21.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-08-30T23:24:21.404 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:24:21.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:21.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi173 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi173 to 7061M 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808] boot 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: osdmap e78: 11 total, 11 up, 11 in 2024-08-30T23:24:21.518 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:21 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi173 2024-08-30T23:24:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi173 to 7061M 2024-08-30T23:24:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808] boot 2024-08-30T23:24:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: osdmap e78: 11 total, 11 up, 11 in 2024-08-30T23:24:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:21 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi173 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi173 to 7061M 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: osd.10 [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808] boot 2024-08-30T23:24:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: osdmap e78: 11 total, 11 up, 11 in 2024-08-30T23:24:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:21 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-30T23:24:22.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:22 smithi173 ceph-mon[20554]: osdmap e79: 11 total, 11 up, 11 in 2024-08-30T23:24:22.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:22 smithi173 ceph-mon[20554]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:22.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:24:22.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:24:22.487 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:22.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:22 smithi136 ceph-mon[20207]: osdmap e79: 11 total, 11 up, 11 in 2024-08-30T23:24:22.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:22 smithi136 ceph-mon[20207]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:22.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:24:22.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:24:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:22 smithi106 ceph-mon[16603]: osdmap e79: 11 total, 11 up, 11 in 2024-08-30T23:24:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:22 smithi106 ceph-mon[16603]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-30T23:24:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-30T23:24:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:23.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:23 smithi173 ceph-mon[20554]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:24:23.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:23 smithi136 ceph-mon[20207]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:24:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:23 smithi106 ceph-mon[16603]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:24:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:24 smithi173 ceph-mon[20554]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:24.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:24 smithi136 ceph-mon[20207]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:24 smithi106 ceph-mon[16603]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:25.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/1808329066' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"} : dispatch 2024-08-30T23:24:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:25 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"} : dispatch 2024-08-30T23:24:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:25 smithi136 ceph-mon[20207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"}]': finished 2024-08-30T23:24:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:25 smithi136 ceph-mon[20207]: osdmap e80: 12 total, 11 up, 12 in 2024-08-30T23:24:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:25 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/4202024098' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:24:25.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/1808329066' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"} : dispatch 2024-08-30T23:24:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:25 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"} : dispatch 2024-08-30T23:24:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:25 smithi173 ceph-mon[20554]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"}]': finished 2024-08-30T23:24:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:25 smithi173 ceph-mon[20554]: osdmap e80: 12 total, 11 up, 12 in 2024-08-30T23:24:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:25 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/4202024098' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:24:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/1808329066' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"} : dispatch 2024-08-30T23:24:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:25 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"} : dispatch 2024-08-30T23:24:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:25 smithi106 ceph-mon[16603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b"}]': finished 2024-08-30T23:24:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:25 smithi106 ceph-mon[16603]: osdmap e80: 12 total, 11 up, 12 in 2024-08-30T23:24:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:25 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/4202024098' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-30T23:24:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:26 smithi136 ceph-mon[20207]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:26.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:26 smithi173 ceph-mon[20554]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:26 smithi106 ceph-mon[16603]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:27 smithi136 ceph-mon[20207]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:27 smithi173 ceph-mon[20554]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:27 smithi106 ceph-mon[16603]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:29.683 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:29 smithi173 ceph-mon[20554]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:29 smithi136 ceph-mon[20207]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:29 smithi106 ceph-mon[16603]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:31.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:31 smithi173 ceph-mon[20554]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:31 smithi136 ceph-mon[20207]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:31 smithi106 ceph-mon[16603]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-30T23:24:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:32 smithi136 ceph-mon[20207]: Deploying daemon osd.11 on smithi173 2024-08-30T23:24:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-30T23:24:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:32 smithi173 ceph-mon[20554]: Deploying daemon osd.11 on smithi173 2024-08-30T23:24:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-30T23:24:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:32 smithi106 ceph-mon[16603]: Deploying daemon osd.11 on smithi173 2024-08-30T23:24:33.756 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:33.756 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:33.757 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:33 smithi173 ceph-mon[20554]: osdmap e81: 12 total, 11 up, 12 in 2024-08-30T23:24:33.757 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:33.757 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:33 smithi173 ceph-mon[20554]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:33 smithi136 ceph-mon[20207]: osdmap e81: 12 total, 11 up, 12 in 2024-08-30T23:24:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:33 smithi136 ceph-mon[20207]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:33 smithi106 ceph-mon[16603]: osdmap e81: 12 total, 11 up, 12 in 2024-08-30T23:24:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:33 smithi106 ceph-mon[16603]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:34.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:34.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:35.656 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:35 smithi173 ceph-mon[20554]: osdmap e82: 12 total, 11 up, 12 in 2024-08-30T23:24:35.656 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:35.656 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:35 smithi173 ceph-mon[20554]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:35 smithi136 ceph-mon[20207]: osdmap e82: 12 total, 11 up, 12 in 2024-08-30T23:24:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:35 smithi136 ceph-mon[20207]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:35 smithi106 ceph-mon[16603]: osdmap e82: 12 total, 11 up, 12 in 2024-08-30T23:24:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:35 smithi106 ceph-mon[16603]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:36.704 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:36.704 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:36.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:37 smithi173 ceph-mon[20554]: osdmap e83: 12 total, 11 up, 12 in 2024-08-30T23:24:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:37 smithi173 ceph-mon[20554]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:37 smithi136 ceph-mon[20207]: osdmap e83: 12 total, 11 up, 12 in 2024-08-30T23:24:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:37 smithi136 ceph-mon[20207]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:37 smithi106 ceph-mon[16603]: osdmap e83: 12 total, 11 up, 12 in 2024-08-30T23:24:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:37 smithi106 ceph-mon[16603]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:39.778 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:39 smithi173 ceph-mon[20554]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:39 smithi136 ceph-mon[20207]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:39 smithi106 ceph-mon[16603]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:40.655 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 11 on host 'smithi173' 2024-08-30T23:24:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.293 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.294 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.320 DEBUG:teuthology.orchestra.run.smithi173:osd.11> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.11.service 2024-08-30T23:24:41.323 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-08-30T23:24:41.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:41.579 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:42.206 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:24:42.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:42 smithi136 ceph-mon[20207]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:42 smithi173 ceph-mon[20554]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:42.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 30 23:24:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-30T23:24:42.065+0000 7fcfbfdfe640 -1 osd.11 0 log_to_monitors true 2024-08-30T23:24:42.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:42 smithi106 ceph-mon[16603]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:42.951 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":83,"num_osds":12,"num_up_osds":11,"osd_up_since":1725060260,"num_in_osds":12,"osd_in_since":1725060264,"num_remapped_pgs":0} 2024-08-30T23:24:43.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:43 smithi136 ceph-mon[20207]: from='osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-30T23:24:43.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:43 smithi136 ceph-mon[20207]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-30T23:24:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2496835437' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:43 smithi173 ceph-mon[20554]: from='osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-30T23:24:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:43 smithi173 ceph-mon[20554]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-30T23:24:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2496835437' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:43 smithi106 ceph-mon[16603]: from='osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-30T23:24:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:43 smithi106 ceph-mon[16603]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-30T23:24:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2496835437' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:43.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-08-30T23:24:44.226 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-30T23:24:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: osdmap e84: 12 total, 11 up, 12 in 2024-08-30T23:24:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: from='osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-30T23:24:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: osdmap e84: 12 total, 11 up, 12 in 2024-08-30T23:24:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: from='osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:44.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-30T23:24:44.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: osdmap e84: 12 total, 11 up, 12 in 2024-08-30T23:24:44.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:44.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: from='osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:44.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]} : dispatch 2024-08-30T23:24:44.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:44.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:44 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:44.823 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-30T23:24:44.464+0000 7fcfbc595640 -1 osd.11 0 waiting for initial osdmap 2024-08-30T23:24:44.823 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 30 23:24:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-30T23:24:44.472+0000 7fcfb3ba0640 -1 osd.11 85 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:24:44.880 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: Detected new or changed devices on smithi173 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: osdmap e85: 12 total, 11 up, 12 in 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: Adjusting osd_memory_target on smithi173 to 5296M 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:45.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='osd.11 ' entity='osd.11' 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=613588543,config=3313370793,health=3897446751,kv=2099994583}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3204783607,logm=2245168801}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317309684}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604108220}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=516368861}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1425223388}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433706896}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969732887}) 2024-08-30T23:24:45.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2450436671}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1752032520}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629268430}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257145365}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=932068802}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082555680}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=544087975}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386229325}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4203836887}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=56} crc {logm=3152203246,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=3480739260,mon_config_key=2931756324}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4293735304}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=414483389}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=51} crc {mon_config_key=1990699620,monmap=1193225519,osd_metadata=2378647436,osd_pg_creating=2000802784,osdmap=1313300139}) 2024-08-30T23:24:45.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=3320000875}) 2024-08-30T23:24:45.294 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2564389338' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: Detected new or changed devices on smithi173 2024-08-30T23:24:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:24:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: osdmap e85: 12 total, 11 up, 12 in 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: Adjusting osd_memory_target on smithi173 to 5296M 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='osd.11 ' entity='osd.11' 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=613588543,config=3313370793,health=3897446751,kv=2099994583}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3204783607,logm=2245168801}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317309684}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604108220}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=516368861}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1425223388}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433706896}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969732887}) 2024-08-30T23:24:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2450436671}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1752032520}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629268430}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257145365}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=932068802}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082555680}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=544087975}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386229325}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4203836887}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=56} crc {logm=3152203246,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=3480739260,mon_config_key=2931756324}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4293735304}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=414483389}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=51} crc {mon_config_key=1990699620,monmap=1193225519,osd_metadata=2378647436,osd_pg_creating=2000802784,osdmap=1313300139}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=3320000875}) 2024-08-30T23:24:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2564389338' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: Detected new or changed devices on smithi173 2024-08-30T23:24:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: osdmap e85: 12 total, 11 up, 12 in 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: Adjusting osd_memory_target on smithi173 to 5296M 2024-08-30T23:24:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='osd.11 ' entity='osd.11' 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=613588543,config=3313370793,health=3897446751,kv=2099994583}) 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=3204783607,logm=2245168801}) 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1317309684}) 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=604108220}) 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=516368861}) 2024-08-30T23:24:45.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1425223388}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3433706896}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969732887}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2450436671}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1752032520}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629268430}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257145365}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=932068802}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082555680}) 2024-08-30T23:24:45.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=544087975}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386229325}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4203836887}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=5,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=56} crc {logm=3152203246,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=3480739260,mon_config_key=2931756324}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4293735304}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=414483389}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=11,osd_pg_creating=1,osdmap=51} crc {mon_config_key=1990699620,monmap=1193225519,osd_metadata=2378647436,osd_pg_creating=2000802784,osdmap=1313300139}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=68} crc {osdmap=3320000875}) 2024-08-30T23:24:45.361 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2564389338' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:45.609 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":85,"num_osds":12,"num_up_osds":11,"osd_up_since":1725060260,"num_in_osds":12,"osd_in_since":1725060264,"num_remapped_pgs":0} 2024-08-30T23:24:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084] boot 2024-08-30T23:24:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: osdmap e86: 12 total, 12 up, 12 in 2024-08-30T23:24:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084] boot 2024-08-30T23:24:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: osdmap e86: 12 total, 12 up, 12 in 2024-08-30T23:24:46.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:46.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-08-30T23:24:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: osd.11 [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084] boot 2024-08-30T23:24:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: osdmap e86: 12 total, 12 up, 12 in 2024-08-30T23:24:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-30T23:24:46.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-08-30T23:24:46.875 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:47.487 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:24:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:47 smithi136 ceph-mon[20207]: osdmap e87: 12 total, 12 up, 12 in 2024-08-30T23:24:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:47 smithi136 ceph-mon[20207]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:47 smithi173 ceph-mon[20554]: osdmap e87: 12 total, 12 up, 12 in 2024-08-30T23:24:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:47 smithi173 ceph-mon[20554]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:47.845 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:47 smithi106 ceph-mon[16603]: osdmap e87: 12 total, 12 up, 12 in 2024-08-30T23:24:47.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:47 smithi106 ceph-mon[16603]: pgmap v255: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:48.183 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":88,"num_osds":12,"num_up_osds":12,"osd_up_since":1725060285,"num_in_osds":12,"osd_in_since":1725060264,"num_remapped_pgs":0} 2024-08-30T23:24:48.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-30T23:24:48.428 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:48.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:48 smithi106 ceph-mon[16603]: osdmap e88: 12 total, 12 up, 12 in 2024-08-30T23:24:48.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2114185167' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:48 smithi136 ceph-mon[20207]: osdmap e88: 12 total, 12 up, 12 in 2024-08-30T23:24:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2114185167' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:48 smithi173 ceph-mon[20554]: osdmap e88: 12 total, 12 up, 12 in 2024-08-30T23:24:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2114185167' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-30T23:24:49.036 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:24:49.036 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":88,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","created":"2024-08-30T23:18:02.086618+0000","modified":"2024-08-30T23:24:47.471885+0000","last_up_change":"2024-08-30T23:24:45.463540+0000","last_in_change":"2024-08-30T23:24:24.349570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T23:21:09.548123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48096b59-4ae9-43da-a0c2-a689ad9fe2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6803","nonce":3938533896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6805","nonce":3938533896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6809","nonce":3938533896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6807","nonce":3938533896}]},"public_addr":"172.21.15.106:6803/3938533896","cluster_addr":"172.21.15.106:6805/3938533896","heartbeat_back_addr":"172.21.15.106:6809/3938533896","heartbeat_front_addr":"172.21.15.106:6807/3938533896","state":["exists","up"]},{"osd":1,"uuid":"0b7f3c0d-489c-48b4-91ca-fa8da6baef53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6811","nonce":1861847252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6813","nonce":1861847252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6817","nonce":1861847252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6815","nonce":1861847252}]},"public_addr":"172.21.15.106:6811/1861847252","cluster_addr":"172.21.15.106:6813/1861847252","heartbeat_back_addr":"172.21.15.106:6817/1861847252","heartbeat_front_addr":"172.21.15.106:6815/1861847252","state":["exists","up"]},{"osd":2,"uuid":"b2453c8c-444c-4a5c-9c37-36f6f0213219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6819","nonce":2791969052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6821","nonce":2791969052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6825","nonce":2791969052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6823","nonce":2791969052}]},"public_addr":"172.21.15.106:6819/2791969052","cluster_addr":"172.21.15.106:6821/2791969052","heartbeat_back_addr":"172.21.15.106:6825/2791969052","heartbeat_front_addr":"172.21.15.106:6823/2791969052","state":["exists","up"]},{"osd":3,"uuid":"d1778ea7-0e45-4646-8fd2-b0f7c9e3086f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6827","nonce":2233735435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6829","nonce":2233735435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6833","nonce":2233735435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6831","nonce":2233735435}]},"public_addr":"172.21.15.106:6827/2233735435","cluster_addr":"172.21.15.106:6829/2233735435","heartbeat_back_addr":"172.21.15.106:6833/2233735435","heartbeat_front_addr":"172.21.15.106:6831/2233735435","state":["exists","up"]},{"osd":4,"uuid":"0335aeb0-87b1-4cff-9351-c489ad188729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6801","nonce":2130223341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6803","nonce":2130223341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6807","nonce":2130223341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6805","nonce":2130223341}]},"public_addr":"172.21.15.136:6801/2130223341","cluster_addr":"172.21.15.136:6803/2130223341","heartbeat_back_addr":"172.21.15.136:6807/2130223341","heartbeat_front_addr":"172.21.15.136:6805/2130223341","state":["exists","up"]},{"osd":5,"uuid":"dc2279d3-fcec-4934-8a75-9befec1bbbe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6809","nonce":2282248833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6811","nonce":2282248833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6815","nonce":2282248833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6813","nonce":2282248833}]},"public_addr":"172.21.15.136:6809/2282248833","cluster_addr":"172.21.15.136:6811/2282248833","heartbeat_back_addr":"172.21.15.136:6815/2282248833","heartbeat_front_addr":"172.21.15.136:6813/2282248833","state":["exists","up"]},{"osd":6,"uuid":"dddb0cd9-8f57-4836-96fd-f66925b36c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6817","nonce":2133537865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6819","nonce":2133537865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6823","nonce":2133537865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6821","nonce":2133537865}]},"public_addr":"172.21.15.136:6817/2133537865","cluster_addr":"172.21.15.136:6819/2133537865","heartbeat_back_addr":"172.21.15.136:6823/2133537865","heartbeat_front_addr":"172.21.15.136:6821/2133537865","state":["exists","up"]},{"osd":7,"uuid":"b9894fb4-6f34-47cd-81cd-9e7d9758360b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6825","nonce":1537958513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6827","nonce":1537958513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6831","nonce":1537958513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6829","nonce":1537958513}]},"public_addr":"172.21.15.136:6825/1537958513","cluster_addr":"172.21.15.136:6827/1537958513","heartbeat_back_addr":"172.21.15.136:6831/1537958513","heartbeat_front_addr":"172.21.15.136:6829/1537958513","state":["exists","up"]},{"osd":8,"uuid":"046e019e-1e33-403c-9577-757c7d546f83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6801","nonce":526702876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6803","nonce":526702876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6807","nonce":526702876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6805","nonce":526702876}]},"public_addr":"172.21.15.173:6801/526702876","cluster_addr":"172.21.15.173:6803/526702876","heartbeat_back_addr":"172.21.15.173:6807/526702876","heartbeat_front_addr":"172.21.15.173:6805/526702876","state":["exists","up"]},{"osd":9,"uuid":"ea9ca538-e1da-4687-8b4f-542c0fc3737c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6809","nonce":478092222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6811","nonce":478092222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6815","nonce":478092222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6813","nonce":478092222}]},"public_addr":"172.21.15.173:6809/478092222","cluster_addr":"172.21.15.173:6811/478092222","heartbeat_back_addr":"172.21.15.173:6815/478092222","heartbeat_front_addr":"172.21.15.173:6813/478092222","state":["exists","up"]},{"osd":10,"uuid":"085c32cd-dde4-4a54-bb0d-9182075040c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6817","nonce":225763808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6819","nonce":225763808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6823","nonce":225763808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6821","nonce":225763808}]},"public_addr":"172.21.15.173:6817/225763808","cluster_addr":"172.21.15.173:6819/225763808","heartbeat_back_addr":"172.21.15.173:6823/225763808","heartbeat_front_addr":"172.21.15.173:6821/225763808","state":["exists","up"]},{"osd":11,"uuid":"6bd9ce25-ce61-4998-af79-26c4ebe2ce3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6825","nonce":2219423084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6827","nonce":2219423084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6831","nonce":2219423084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6829","nonce":2219423084}]},"public_addr":"172.21.15.173:6825/2219423084","cluster_addr":"172.21.15.173:6827/2219423084","heartbeat_back_addr":"172.21.15.173:6831/2219423084","heartbeat_front_addr":"172.21.15.173:6829/2219423084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:15.299261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:17.060892+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:31.745168+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:58.603251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:19.598499+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:36.617251+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:47.565219+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:33.944281+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:35.919418+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:52.533468+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:18.276907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:43.112306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3708473409":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/1031615522":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3817969167":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3688603346":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6801/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3790397394":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:0/1787333807":"2024-08-31T23:18:27.701240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T23:24:49.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3785873689' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:24:49.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:49 smithi106 ceph-mon[16603]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:49.641 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-30T23:24:49.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-30T23:24:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3785873689' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:24:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:49 smithi136 ceph-mon[20207]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3785873689' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:24:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:49 smithi173 ceph-mon[20554]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:49.884 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:50.537 INFO:teuthology.orchestra.run.smithi106.stdout:pg_num: 1 2024-08-30T23:24:50.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:50 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:24:50.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:50 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:24:50.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:50 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:24:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:50 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:24:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:50 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:24:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:50 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:24:51.316 INFO:tasks.cephadm:Setting up client nodes... 2024-08-30T23:24:51.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-30T23:24:51.562 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/957941453' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-30T23:24:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:51 smithi106 ceph-mon[16603]: pgmap v258: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:51.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/957941453' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-30T23:24:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:51 smithi136 ceph-mon[20207]: pgmap v258: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/957941453' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-30T23:24:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:51 smithi173 ceph-mon[20554]: pgmap v258: 1 pgs: 1 peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:24:52.295 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0] 2024-08-30T23:24:52.295 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDEVNJmeoiDERAAnSfIN6itqPx00Sz6Q0Z21Q== 2024-08-30T23:24:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/348352262' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:52.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/348352262' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:52.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/348352262' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:52.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/348352262' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/348352262' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/348352262' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:53.048 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-08-30T23:24:53.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-30T23:24:53.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-30T23:24:53.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-30T23:24:53.347 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:24:53.774 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:53 smithi136 ceph-mon[20207]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:24:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:53 smithi173 ceph-mon[20554]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:24:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:53 smithi106 ceph-mon[16603]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-30T23:24:54.128 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-08-30T23:24:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDGVNJmIv9JBxAAtuO9WOeCO0tdyhraTFThoQ== 2024-08-30T23:24:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.136:0/288037198' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:54 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:54 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.136:0/288037198' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:54 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:54 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:54.833 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-08-30T23:24:54.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-08-30T23:24:54.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-30T23:24:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.136:0/288037198' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:54 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:54 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:54.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-30T23:24:55.130 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:24:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:55 smithi136 ceph-mon[20207]: osdmap e89: 12 total, 12 up, 12 in 2024-08-30T23:24:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:55 smithi136 ceph-mon[20207]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:24:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:55 smithi173 ceph-mon[20554]: osdmap e89: 12 total, 12 up, 12 in 2024-08-30T23:24:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:55 smithi173 ceph-mon[20554]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:24:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:55 smithi106 ceph-mon[16603]: osdmap e89: 12 total, 12 up, 12 in 2024-08-30T23:24:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:55 smithi106 ceph-mon[16603]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:24:55.861 INFO:teuthology.orchestra.run.smithi173.stdout:[client.2] 2024-08-30T23:24:55.861 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQDHVNJmAxAAMxAA1/eiJP5Qvs9TIATjM4YzgQ== 2024-08-30T23:24:56.574 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-08-30T23:24:56.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-08-30T23:24:56.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-08-30T23:24:56.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph config log 1 --format=json 2024-08-30T23:24:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:24:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:24:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:56 smithi173 ceph-mon[20554]: osdmap e90: 12 total, 12 up, 12 in 2024-08-30T23:24:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.173:0/699398840' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:24:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:24:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:56 smithi106 ceph-mon[16603]: osdmap e90: 12 total, 12 up, 12 in 2024-08-30T23:24:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.173:0/699398840' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:56.867 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:24:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:24:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:56 smithi136 ceph-mon[20207]: osdmap e90: 12 total, 12 up, 12 in 2024-08-30T23:24:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.173:0/699398840' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-30T23:24:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-30T23:24:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:24:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:57 smithi173 ceph-mon[20554]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:24:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4076993967' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-30T23:24:57.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:57 smithi106 ceph-mon[16603]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:24:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4076993967' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-30T23:24:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:57 smithi136 ceph-mon[20207]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:24:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4076993967' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-30T23:24:58.173 INFO:teuthology.orchestra.run.smithi106.stdout:[{"version":28,"timestamp":"2024-08-30T23:24:44.465789+0000","name":"","changes":[{"name":"osd.11/osd_mclock_max_capacity_iops_ssd","new_value":"58724.455498"}]}] 2024-08-30T23:24:58.173 INFO:tasks.ceph_manager:config epoch is 28 2024-08-30T23:24:58.174 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-30T23:24:58.174 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-30T23:24:58.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph mgr dump --format=json 2024-08-30T23:24:58.413 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:24:59.099 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:24:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:24:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/86647845' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-30T23:24:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:24:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/86647845' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-30T23:24:59.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:24:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/86647845' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-30T23:24:59.730 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":15,"flags":0,"active_gid":14151,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3641968899},{"type":"v1","addr":"172.21.15.106:6801","nonce":3641968899}]},"active_addr":"172.21.15.106:6801/3641968899","active_change":"2024-08-30T23:18:51.239769+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24104,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.106:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":2007975942}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3573563326}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":1598400485}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.106:0","nonce":3076126701}]}]} 2024-08-30T23:24:59.735 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-30T23:24:59.735 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-30T23:24:59.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-30T23:24:59.977 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:00 smithi106 ceph-mon[16603]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:25:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:00 smithi136 ceph-mon[20207]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:25:00.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:00 smithi173 ceph-mon[20554]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-30T23:25:00.630 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:00.630 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":90,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","created":"2024-08-30T23:18:02.086618+0000","modified":"2024-08-30T23:24:55.555356+0000","last_up_change":"2024-08-30T23:24:45.463540+0000","last_in_change":"2024-08-30T23:24:24.349570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T23:21:09.548123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48096b59-4ae9-43da-a0c2-a689ad9fe2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6803","nonce":3938533896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6805","nonce":3938533896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6809","nonce":3938533896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6807","nonce":3938533896}]},"public_addr":"172.21.15.106:6803/3938533896","cluster_addr":"172.21.15.106:6805/3938533896","heartbeat_back_addr":"172.21.15.106:6809/3938533896","heartbeat_front_addr":"172.21.15.106:6807/3938533896","state":["exists","up"]},{"osd":1,"uuid":"0b7f3c0d-489c-48b4-91ca-fa8da6baef53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6811","nonce":1861847252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6813","nonce":1861847252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6817","nonce":1861847252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6815","nonce":1861847252}]},"public_addr":"172.21.15.106:6811/1861847252","cluster_addr":"172.21.15.106:6813/1861847252","heartbeat_back_addr":"172.21.15.106:6817/1861847252","heartbeat_front_addr":"172.21.15.106:6815/1861847252","state":["exists","up"]},{"osd":2,"uuid":"b2453c8c-444c-4a5c-9c37-36f6f0213219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6819","nonce":2791969052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6821","nonce":2791969052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6825","nonce":2791969052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6823","nonce":2791969052}]},"public_addr":"172.21.15.106:6819/2791969052","cluster_addr":"172.21.15.106:6821/2791969052","heartbeat_back_addr":"172.21.15.106:6825/2791969052","heartbeat_front_addr":"172.21.15.106:6823/2791969052","state":["exists","up"]},{"osd":3,"uuid":"d1778ea7-0e45-4646-8fd2-b0f7c9e3086f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6827","nonce":2233735435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6829","nonce":2233735435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6833","nonce":2233735435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6831","nonce":2233735435}]},"public_addr":"172.21.15.106:6827/2233735435","cluster_addr":"172.21.15.106:6829/2233735435","heartbeat_back_addr":"172.21.15.106:6833/2233735435","heartbeat_front_addr":"172.21.15.106:6831/2233735435","state":["exists","up"]},{"osd":4,"uuid":"0335aeb0-87b1-4cff-9351-c489ad188729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6801","nonce":2130223341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6803","nonce":2130223341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6807","nonce":2130223341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6805","nonce":2130223341}]},"public_addr":"172.21.15.136:6801/2130223341","cluster_addr":"172.21.15.136:6803/2130223341","heartbeat_back_addr":"172.21.15.136:6807/2130223341","heartbeat_front_addr":"172.21.15.136:6805/2130223341","state":["exists","up"]},{"osd":5,"uuid":"dc2279d3-fcec-4934-8a75-9befec1bbbe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6809","nonce":2282248833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6811","nonce":2282248833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6815","nonce":2282248833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6813","nonce":2282248833}]},"public_addr":"172.21.15.136:6809/2282248833","cluster_addr":"172.21.15.136:6811/2282248833","heartbeat_back_addr":"172.21.15.136:6815/2282248833","heartbeat_front_addr":"172.21.15.136:6813/2282248833","state":["exists","up"]},{"osd":6,"uuid":"dddb0cd9-8f57-4836-96fd-f66925b36c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6817","nonce":2133537865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6819","nonce":2133537865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6823","nonce":2133537865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6821","nonce":2133537865}]},"public_addr":"172.21.15.136:6817/2133537865","cluster_addr":"172.21.15.136:6819/2133537865","heartbeat_back_addr":"172.21.15.136:6823/2133537865","heartbeat_front_addr":"172.21.15.136:6821/2133537865","state":["exists","up"]},{"osd":7,"uuid":"b9894fb4-6f34-47cd-81cd-9e7d9758360b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6825","nonce":1537958513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6827","nonce":1537958513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6831","nonce":1537958513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6829","nonce":1537958513}]},"public_addr":"172.21.15.136:6825/1537958513","cluster_addr":"172.21.15.136:6827/1537958513","heartbeat_back_addr":"172.21.15.136:6831/1537958513","heartbeat_front_addr":"172.21.15.136:6829/1537958513","state":["exists","up"]},{"osd":8,"uuid":"046e019e-1e33-403c-9577-757c7d546f83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6801","nonce":526702876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6803","nonce":526702876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6807","nonce":526702876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6805","nonce":526702876}]},"public_addr":"172.21.15.173:6801/526702876","cluster_addr":"172.21.15.173:6803/526702876","heartbeat_back_addr":"172.21.15.173:6807/526702876","heartbeat_front_addr":"172.21.15.173:6805/526702876","state":["exists","up"]},{"osd":9,"uuid":"ea9ca538-e1da-4687-8b4f-542c0fc3737c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6809","nonce":478092222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6811","nonce":478092222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6815","nonce":478092222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6813","nonce":478092222}]},"public_addr":"172.21.15.173:6809/478092222","cluster_addr":"172.21.15.173:6811/478092222","heartbeat_back_addr":"172.21.15.173:6815/478092222","heartbeat_front_addr":"172.21.15.173:6813/478092222","state":["exists","up"]},{"osd":10,"uuid":"085c32cd-dde4-4a54-bb0d-9182075040c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6817","nonce":225763808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6819","nonce":225763808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6823","nonce":225763808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6821","nonce":225763808}]},"public_addr":"172.21.15.173:6817/225763808","cluster_addr":"172.21.15.173:6819/225763808","heartbeat_back_addr":"172.21.15.173:6823/225763808","heartbeat_front_addr":"172.21.15.173:6821/225763808","state":["exists","up"]},{"osd":11,"uuid":"6bd9ce25-ce61-4998-af79-26c4ebe2ce3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6825","nonce":2219423084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6827","nonce":2219423084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6831","nonce":2219423084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6829","nonce":2219423084}]},"public_addr":"172.21.15.173:6825/2219423084","cluster_addr":"172.21.15.173:6827/2219423084","heartbeat_back_addr":"172.21.15.173:6831/2219423084","heartbeat_front_addr":"172.21.15.173:6829/2219423084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:15.299261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:17.060892+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:31.745168+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:58.603251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:19.598499+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:53.516024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:55.504630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:33.944281+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:35.919418+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:52.533468+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:18.276907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:43.112306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3708473409":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/1031615522":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3817969167":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3688603346":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6801/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3790397394":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:0/1787333807":"2024-08-31T23:18:27.701240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T23:25:01.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2093866028' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:25:01.354 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-30T23:25:01.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-30T23:25:01.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2093866028' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:25:01.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2093866028' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:25:01.596 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:02.203 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":91,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","created":"2024-08-30T23:18:02.086618+0000","modified":"2024-08-30T23:25:02.154372+0000","last_up_change":"2024-08-30T23:24:45.463540+0000","last_in_change":"2024-08-30T23:24:24.349570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T23:21:09.548123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48096b59-4ae9-43da-a0c2-a689ad9fe2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6803","nonce":3938533896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6805","nonce":3938533896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6809","nonce":3938533896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6807","nonce":3938533896}]},"public_addr":"172.21.15.106:6803/3938533896","cluster_addr":"172.21.15.106:6805/3938533896","heartbeat_back_addr":"172.21.15.106:6809/3938533896","heartbeat_front_addr":"172.21.15.106:6807/3938533896","state":["exists","up"]},{"osd":1,"uuid":"0b7f3c0d-489c-48b4-91ca-fa8da6baef53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6811","nonce":1861847252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6813","nonce":1861847252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6817","nonce":1861847252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":1861847252},{"type":"v1","addr":"172.21.15.106:6815","nonce":1861847252}]},"public_addr":"172.21.15.106:6811/1861847252","cluster_addr":"172.21.15.106:6813/1861847252","heartbeat_back_addr":"172.21.15.106:6817/1861847252","heartbeat_front_addr":"172.21.15.106:6815/1861847252","state":["exists","up"]},{"osd":2,"uuid":"b2453c8c-444c-4a5c-9c37-36f6f0213219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6819","nonce":2791969052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6821","nonce":2791969052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6825","nonce":2791969052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6823","nonce":2791969052}]},"public_addr":"172.21.15.106:6819/2791969052","cluster_addr":"172.21.15.106:6821/2791969052","heartbeat_back_addr":"172.21.15.106:6825/2791969052","heartbeat_front_addr":"172.21.15.106:6823/2791969052","state":["exists","up"]},{"osd":3,"uuid":"d1778ea7-0e45-4646-8fd2-b0f7c9e3086f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6827","nonce":2233735435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6829","nonce":2233735435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6833","nonce":2233735435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6831","nonce":2233735435}]},"public_addr":"172.21.15.106:6827/2233735435","cluster_addr":"172.21.15.106:6829/2233735435","heartbeat_back_addr":"172.21.15.106:6833/2233735435","heartbeat_front_addr":"172.21.15.106:6831/2233735435","state":["exists","up"]},{"osd":4,"uuid":"0335aeb0-87b1-4cff-9351-c489ad188729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6801","nonce":2130223341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6803","nonce":2130223341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6807","nonce":2130223341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6805","nonce":2130223341}]},"public_addr":"172.21.15.136:6801/2130223341","cluster_addr":"172.21.15.136:6803/2130223341","heartbeat_back_addr":"172.21.15.136:6807/2130223341","heartbeat_front_addr":"172.21.15.136:6805/2130223341","state":["exists","up"]},{"osd":5,"uuid":"dc2279d3-fcec-4934-8a75-9befec1bbbe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6809","nonce":2282248833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6811","nonce":2282248833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6815","nonce":2282248833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2282248833},{"type":"v1","addr":"172.21.15.136:6813","nonce":2282248833}]},"public_addr":"172.21.15.136:6809/2282248833","cluster_addr":"172.21.15.136:6811/2282248833","heartbeat_back_addr":"172.21.15.136:6815/2282248833","heartbeat_front_addr":"172.21.15.136:6813/2282248833","state":["exists","up"]},{"osd":6,"uuid":"dddb0cd9-8f57-4836-96fd-f66925b36c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6817","nonce":2133537865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6819","nonce":2133537865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6823","nonce":2133537865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6821","nonce":2133537865}]},"public_addr":"172.21.15.136:6817/2133537865","cluster_addr":"172.21.15.136:6819/2133537865","heartbeat_back_addr":"172.21.15.136:6823/2133537865","heartbeat_front_addr":"172.21.15.136:6821/2133537865","state":["exists","up"]},{"osd":7,"uuid":"b9894fb4-6f34-47cd-81cd-9e7d9758360b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6825","nonce":1537958513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6827","nonce":1537958513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6831","nonce":1537958513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1537958513},{"type":"v1","addr":"172.21.15.136:6829","nonce":1537958513}]},"public_addr":"172.21.15.136:6825/1537958513","cluster_addr":"172.21.15.136:6827/1537958513","heartbeat_back_addr":"172.21.15.136:6831/1537958513","heartbeat_front_addr":"172.21.15.136:6829/1537958513","state":["exists","up"]},{"osd":8,"uuid":"046e019e-1e33-403c-9577-757c7d546f83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6801","nonce":526702876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6803","nonce":526702876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6807","nonce":526702876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6805","nonce":526702876}]},"public_addr":"172.21.15.173:6801/526702876","cluster_addr":"172.21.15.173:6803/526702876","heartbeat_back_addr":"172.21.15.173:6807/526702876","heartbeat_front_addr":"172.21.15.173:6805/526702876","state":["exists","up"]},{"osd":9,"uuid":"ea9ca538-e1da-4687-8b4f-542c0fc3737c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6809","nonce":478092222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6811","nonce":478092222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6815","nonce":478092222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6813","nonce":478092222}]},"public_addr":"172.21.15.173:6809/478092222","cluster_addr":"172.21.15.173:6811/478092222","heartbeat_back_addr":"172.21.15.173:6815/478092222","heartbeat_front_addr":"172.21.15.173:6813/478092222","state":["exists","up"]},{"osd":10,"uuid":"085c32cd-dde4-4a54-bb0d-9182075040c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6817","nonce":225763808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6819","nonce":225763808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6823","nonce":225763808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6821","nonce":225763808}]},"public_addr":"172.21.15.173:6817/225763808","cluster_addr":"172.21.15.173:6819/225763808","heartbeat_back_addr":"172.21.15.173:6823/225763808","heartbeat_front_addr":"172.21.15.173:6821/225763808","state":["exists","up"]},{"osd":11,"uuid":"6bd9ce25-ce61-4998-af79-26c4ebe2ce3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6825","nonce":2219423084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6827","nonce":2219423084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6831","nonce":2219423084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6829","nonce":2219423084}]},"public_addr":"172.21.15.173:6825/2219423084","cluster_addr":"172.21.15.173:6827/2219423084","heartbeat_back_addr":"172.21.15.173:6831/2219423084","heartbeat_front_addr":"172.21.15.173:6829/2219423084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:15.299261+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:17.060892+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:31.745168+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:23:58.603251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:19.598499+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:53.516024+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:55.504630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:33.944281+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:35.919418+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:25:02.039113+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:18.276907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-30T23:24:43.112306+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3708473409":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/1031615522":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3817969167":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3688603346":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6801/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3790397394":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:0/1787333807":"2024-08-31T23:18:27.701240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-30T23:25:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:02 smithi106 ceph-mon[16603]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:02.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:02 smithi136 ceph-mon[20207]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:02.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:02 smithi173 ceph-mon[20554]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:02.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-30T23:25:02.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-30T23:25:02.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-30T23:25:02.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-30T23:25:02.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-30T23:25:02.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-30T23:25:02.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-30T23:25:02.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-30T23:25:02.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-30T23:25:02.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-30T23:25:02.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-30T23:25:02.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-30T23:25:03.533 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:03 smithi136 ceph-mon[20207]: osdmap e91: 12 total, 12 up, 12 in 2024-08-30T23:25:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/201254316' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:25:03.542 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:03 smithi173 ceph-mon[20554]: osdmap e91: 12 total, 12 up, 12 in 2024-08-30T23:25:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/201254316' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:25:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:03 smithi106 ceph-mon[16603]: osdmap e91: 12 total, 12 up, 12 in 2024-08-30T23:25:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/201254316' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-30T23:25:04.276 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:04 smithi106 ceph-mon[16603]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:04.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:04 smithi136 ceph-mon[20207]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:04 smithi173 ceph-mon[20554]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:05.330 INFO:teuthology.orchestra.run.smithi106.stdout:193273528351 2024-08-30T23:25:05.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-30T23:25:05.331 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.331 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.332 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.335 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.335 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.335 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.345 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.350 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.350 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:05 smithi106 ceph-mon[16603]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:05.696 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:05 smithi136 ceph-mon[20207]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:05.728 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:05.732 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:05 smithi173 ceph-mon[20554]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:167503724579 2024-08-30T23:25:05.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-30T23:25:06.839 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:07.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:07 smithi106 ceph-mon[16603]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:07 smithi136 ceph-mon[20207]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:07 smithi173 ceph-mon[20554]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:09.239 INFO:teuthology.orchestra.run.smithi106.stdout:34359738429 2024-08-30T23:25:09.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-30T23:25:09.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:09.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:09.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:09 smithi106 ceph-mon[16603]: osdmap e92: 12 total, 12 up, 12 in 2024-08-30T23:25:09.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:09 smithi106 ceph-mon[16603]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:09 smithi136 ceph-mon[20207]: osdmap e92: 12 total, 12 up, 12 in 2024-08-30T23:25:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:09 smithi136 ceph-mon[20207]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:09 smithi173 ceph-mon[20554]: osdmap e92: 12 total, 12 up, 12 in 2024-08-30T23:25:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:09 smithi173 ceph-mon[20554]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1183427168' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-30T23:25:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:11 smithi106 ceph-mon[16603]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1183427168' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-30T23:25:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:11 smithi136 ceph-mon[20207]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1183427168' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-30T23:25:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:11 smithi173 ceph-mon[20554]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:11.958 INFO:teuthology.orchestra.run.smithi106.stdout:55834574905 2024-08-30T23:25:11.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-30T23:25:12.644 INFO:teuthology.orchestra.run.smithi106.stdout:335007449101 2024-08-30T23:25:12.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-30T23:25:13.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:13 smithi106 ceph-mon[16603]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:13 smithi136 ceph-mon[20207]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:13 smithi173 ceph-mon[20554]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:14.109 INFO:teuthology.orchestra.run.smithi106.stdout:111669149743 2024-08-30T23:25:14.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-30T23:25:14.195 INFO:teuthology.orchestra.run.smithi106.stdout:193273528352 2024-08-30T23:25:14.861 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528351 got 193273528352 for osd.6 2024-08-30T23:25:14.861 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:15.331 INFO:teuthology.orchestra.run.smithi106.stdout:231928234010 2024-08-30T23:25:15.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-30T23:25:15.472 INFO:teuthology.orchestra.run.smithi106.stdout:77309411380 2024-08-30T23:25:15.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-30T23:25:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:15 smithi136 ceph-mon[20207]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:15 smithi173 ceph-mon[20554]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:15 smithi106 ceph-mon[16603]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:16.124 INFO:teuthology.orchestra.run.smithi106.stdout:266287972375 2024-08-30T23:25:16.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-30T23:25:16.789 INFO:teuthology.orchestra.run.smithi106.stdout:137438953513 2024-08-30T23:25:16.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-30T23:25:17.568 INFO:teuthology.orchestra.run.smithi106.stdout:369367187463 2024-08-30T23:25:17.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-30T23:25:17.624 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:17.626 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:17.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:17 smithi106 ceph-mon[16603]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:17 smithi136 ceph-mon[20207]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:17 smithi173 ceph-mon[20554]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:18.026 INFO:teuthology.orchestra.run.smithi106.stdout:300647710739 2024-08-30T23:25:18.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-30T23:25:18.263 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:18.263 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:18.512 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:18.514 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:18.514 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:18.691 INFO:teuthology.orchestra.run.smithi106.stdout:167503724581 2024-08-30T23:25:18.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/835555317' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-30T23:25:19.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/835555317' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-30T23:25:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/835555317' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-30T23:25:19.291 INFO:teuthology.orchestra.run.smithi106.stdout:34359738431 2024-08-30T23:25:19.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:19 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:19.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:19 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:19.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1224108284' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-30T23:25:19.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:19 smithi106 ceph-mon[16603]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:19.911 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738429 got 34359738431 for osd.0 2024-08-30T23:25:19.912 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1224108284' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-30T23:25:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:19 smithi136 ceph-mon[20207]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:20.063 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:19 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:19 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1224108284' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-30T23:25:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:19 smithi173 ceph-mon[20554]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:20.261 INFO:teuthology.orchestra.run.smithi106.stdout:111669149745 2024-08-30T23:25:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:20 smithi106 ceph-mon[16603]: osdmap e93: 12 total, 12 up, 12 in 2024-08-30T23:25:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/220423130' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-30T23:25:20.911 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:20 smithi136 ceph-mon[20207]: osdmap e93: 12 total, 12 up, 12 in 2024-08-30T23:25:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/220423130' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-30T23:25:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:20 smithi173 ceph-mon[20554]: osdmap e93: 12 total, 12 up, 12 in 2024-08-30T23:25:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/220423130' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-30T23:25:21.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724579 got 167503724581 for osd.5 2024-08-30T23:25:21.450 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:21.875 INFO:teuthology.orchestra.run.smithi106.stdout:335007449104 2024-08-30T23:25:22.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:21 smithi106 ceph-mon[16603]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:21 smithi136 ceph-mon[20207]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:21 smithi173 ceph-mon[20554]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:22.371 INFO:teuthology.orchestra.run.smithi106.stdout:137438953515 2024-08-30T23:25:22.581 INFO:teuthology.orchestra.run.smithi106.stdout:231928234012 2024-08-30T23:25:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4047036306' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-30T23:25:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3147105007' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-30T23:25:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3237281936' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-30T23:25:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4047036306' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-30T23:25:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3147105007' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-30T23:25:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3237281936' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-30T23:25:23.074 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4047036306' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-30T23:25:23.074 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3147105007' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-30T23:25:23.074 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3237281936' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-30T23:25:23.381 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449101 got 335007449104 for osd.10 2024-08-30T23:25:23.381 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:24.005 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:23 smithi106 ceph-mon[16603]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:24.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149743 got 111669149745 for osd.3 2024-08-30T23:25:24.033 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:23 smithi136 ceph-mon[20207]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:23 smithi173 ceph-mon[20554]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:24.447 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:24.646 INFO:teuthology.orchestra.run.smithi106.stdout:266287972378 2024-08-30T23:25:24.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:24 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:24.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:24 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:24.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4194349171' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-30T23:25:24.826 INFO:teuthology.orchestra.run.smithi106.stdout:55834574908 2024-08-30T23:25:24.996 INFO:teuthology.orchestra.run.smithi106.stdout:77309411383 2024-08-30T23:25:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:24 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:24 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:25.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4194349171' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-30T23:25:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:24 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:24 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4194349171' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-30T23:25:25.695 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953513 got 137438953515 for osd.4 2024-08-30T23:25:25.695 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:25.952 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:25 smithi106 ceph-mon[16603]: osdmap e94: 12 total, 12 up, 12 in 2024-08-30T23:25:25.952 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1073194724' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-30T23:25:25.952 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/537443659' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-30T23:25:25.952 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:25 smithi106 ceph-mon[16603]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:25 smithi136 ceph-mon[20207]: osdmap e94: 12 total, 12 up, 12 in 2024-08-30T23:25:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1073194724' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-30T23:25:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/537443659' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-30T23:25:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:25 smithi136 ceph-mon[20207]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:25 smithi173 ceph-mon[20554]: osdmap e94: 12 total, 12 up, 12 in 2024-08-30T23:25:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1073194724' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-30T23:25:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/537443659' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-30T23:25:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:25 smithi173 ceph-mon[20554]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:26.228 INFO:teuthology.orchestra.run.smithi106.stdout:369367187466 2024-08-30T23:25:26.500 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234010 got 231928234012 for osd.7 2024-08-30T23:25:26.500 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:26.920 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411380 got 77309411383 for osd.2 2024-08-30T23:25:26.921 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/710028489' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-30T23:25:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/710028489' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-30T23:25:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/710028489' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-30T23:25:27.610 INFO:teuthology.orchestra.run.smithi106.stdout:300647710741 2024-08-30T23:25:27.832 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574905 got 55834574908 for osd.1 2024-08-30T23:25:27.832 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:27 smithi173 ceph-mon[20554]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/354334739' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-30T23:25:28.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:27 smithi106 ceph-mon[16603]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:28.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/354334739' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-30T23:25:28.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:27 smithi136 ceph-mon[20207]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:28.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/354334739' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-30T23:25:28.523 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710739 got 300647710741 for osd.9 2024-08-30T23:25:28.523 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:29.164 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187463 got 369367187466 for osd.11 2024-08-30T23:25:29.164 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:29 smithi136 ceph-mon[20207]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:29 smithi173 ceph-mon[20554]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:29 smithi106 ceph-mon[16603]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:29.909 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972375 got 266287972378 for osd.8 2024-08-30T23:25:29.910 DEBUG:teuthology.parallel:result is None 2024-08-30T23:25:29.910 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-30T23:25:29.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:25:30.159 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:30.809 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:30.810 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:25:31.536 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-08-30T23:25:29.320790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338324,"kb_used_data":9828,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603500,"statfs":{"total":1151940427776,"available":1151593984000,"internally_reserved":0,"allocated":10063872,"data_stored":5948737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.572494"},"pg_stats":[{"pgid":"1.0","version":"23'92","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-30T23:25:25.501582+0000","last_change":"2024-08-30T23:24:49.323194+0000","last_active":"2024-08-30T23:25:25.501582+0000","last_peered":"2024-08-30T23:25:25.501582+0000","last_clean":"2024-08-30T23:25:25.501582+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:25:25.501582+0000","last_undegraded":"2024-08-30T23:25:25.501582+0000","last_fullsized":"2024-08-30T23:25:25.501582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:24:49.323126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:24:49.323126+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:02.593864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187467,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449105,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":696320,"data_stored":357965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":696320,"data_stored":357965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":696320,"data_stored":357965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31880,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:25:31.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:25:31.779 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:31 smithi136 ceph-mon[20207]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:31 smithi136 ceph-mon[20207]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:31 smithi173 ceph-mon[20554]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:31 smithi173 ceph-mon[20554]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:31 smithi106 ceph-mon[16603]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:31 smithi106 ceph-mon[16603]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:32.400 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:32.400 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:25:32.969 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-08-30T23:25:31.321196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338360,"kb_used_data":9864,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603464,"statfs":{"total":1151940427776,"available":1151593947136,"internally_reserved":0,"allocated":10100736,"data_stored":5970868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002278"},"pg_stats":[{"pgid":"1.0","version":"23'92","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-30T23:25:25.501582+0000","last_change":"2024-08-30T23:24:49.323194+0000","last_active":"2024-08-30T23:25:25.501582+0000","last_peered":"2024-08-30T23:25:25.501582+0000","last_clean":"2024-08-30T23:25:25.501582+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:25:25.501582+0000","last_undegraded":"2024-08-30T23:25:25.501582+0000","last_fullsized":"2024-08-30T23:25:25.501582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:24:49.323126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:24:49.323126+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:02.593864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187468,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449106,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710742,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31880,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:25:32.969 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-30T23:25:32.969 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-30T23:25:32.970 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-30T23:25:32.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph health --format=json 2024-08-30T23:25:33.210 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:33 smithi106 ceph-mon[16603]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:33 smithi106 ceph-mon[16603]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:33 smithi136 ceph-mon[20207]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:33 smithi136 ceph-mon[20207]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:33 smithi173 ceph-mon[20554]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:33 smithi173 ceph-mon[20554]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:33.926 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:33.927 INFO:teuthology.orchestra.run.smithi106.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-30T23:25:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/233383049' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-30T23:25:34.638 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-30T23:25:34.638 INFO:tasks.cephadm:Setup complete, yielding 2024-08-30T23:25:34.638 INFO:teuthology.run_tasks:Running task thrashosds... 2024-08-30T23:25:34.650 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-30T23:25:34.650 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath1-testing-2024-08-30-1604'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '24432b660539783e58f890fcff372342c034f6ec'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '24432b660539783e58f890fcff372342c034f6ec'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-bharath1-testing-2024-08-30-1604', 'sha1': '24432b660539783e58f890fcff372342c034f6ec'}} 2024-08-30T23:25:34.651 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-08-30T23:25:34.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-08-30T23:25:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/233383049' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-30T23:25:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/233383049' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-30T23:25:34.894 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:35.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:35 smithi106 ceph-mon[16603]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:35.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:35 smithi136 ceph-mon[20207]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:35.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:35 smithi173 ceph-mon[20554]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:36.300 INFO:tasks.thrashosds:Beginning thrashosds... 2024-08-30T23:25:36.301 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-30T23:25:36.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:25:36.541 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1077762515' entity='client.admin' 2024-08-30T23:25:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:25:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:25:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:25:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:25:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1077762515' entity='client.admin' 2024-08-30T23:25:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:25:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:25:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:25:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:25:36.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1077762515' entity='client.admin' 2024-08-30T23:25:36.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:25:36.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:25:36.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:25:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:25:37.147 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:37.148 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:25:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:37 smithi136 ceph-mon[20207]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:37 smithi136 ceph-mon[20207]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:37.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:37 smithi173 ceph-mon[20554]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:37.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:37 smithi173 ceph-mon[20554]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:37 smithi106 ceph-mon[16603]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:37 smithi106 ceph-mon[16603]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:37.891 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-08-30T23:25:35.322106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338360,"kb_used_data":9864,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603464,"statfs":{"total":1151940427776,"available":1151593947136,"internally_reserved":0,"allocated":10100736,"data_stored":5970868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.553718"},"pg_stats":[{"pgid":"1.0","version":"23'92","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-30T23:25:25.501582+0000","last_change":"2024-08-30T23:24:49.323194+0000","last_active":"2024-08-30T23:25:25.501582+0000","last_peered":"2024-08-30T23:25:25.501582+0000","last_clean":"2024-08-30T23:25:25.501582+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:25:25.501582+0000","last_undegraded":"2024-08-30T23:25:25.501582+0000","last_fullsized":"2024-08-30T23:25:25.501582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:24:49.323126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:24:49.323126+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:02.593864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187468,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449106,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31880,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:25:37.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:25:38.131 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:38.789 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:25:38.789 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:25:39.461 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-08-30T23:25:38.539174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338360,"kb_used_data":9864,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603464,"statfs":{"total":1151940427776,"available":1151593947136,"internally_reserved":0,"allocated":10100736,"data_stored":5970868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":336115001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.218761"},"pg_stats":[{"pgid":"1.0","version":"23'92","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-08-30T23:25:25.501582+0000","last_change":"2024-08-30T23:24:49.323194+0000","last_active":"2024-08-30T23:25:25.501582+0000","last_peered":"2024-08-30T23:25:25.501582+0000","last_clean":"2024-08-30T23:25:25.501582+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:25:25.501582+0000","last_undegraded":"2024-08-30T23:25:25.501582+0000","last_fullsized":"2024-08-30T23:25:25.501582+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:24:49.323126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:24:49.323126+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:02.593864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187469,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449107,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710743,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972381,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1241088,"data_stored":894270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31880,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":365342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:25:39.461 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-30T23:25:39.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump 2024-08-30T23:25:39.707 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:39 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:39 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:39 smithi136 ceph-mon[20207]: osdmap e95: 12 total, 12 up, 12 in 2024-08-30T23:25:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:39 smithi136 ceph-mon[20207]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:39 smithi136 ceph-mon[20207]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:39 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:39 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:39 smithi106 ceph-mon[16603]: osdmap e95: 12 total, 12 up, 12 in 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:39 smithi106 ceph-mon[16603]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:39 smithi106 ceph-mon[16603]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:39 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:39 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:39 smithi173 ceph-mon[20554]: osdmap e95: 12 total, 12 up, 12 in 2024-08-30T23:25:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:39 smithi173 ceph-mon[20554]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:25:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:39 smithi173 ceph-mon[20554]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:40.301 INFO:teuthology.orchestra.run.smithi106.stdout:epoch 95 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:fsid eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:created 2024-08-30T23:18:02.086618+0000 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:modified 2024-08-30T23:25:38.529388+0000 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:crush_version 26 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:full_ratio 0.95 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:backfillfull_ratio 0.9 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:nearfull_ratio 0.85 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:require_min_compat_client luminous 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:min_compat_client jewel 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:require_osd_release squid 2024-08-30T23:25:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:stretch_mode_enabled false 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:max_osd 12 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 up in weight 1 up_from 8 up_thru 54 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896] [v2:172.21.15.106:6804/3938533896,v1:172.21.15.106:6805/3938533896] exists,up 48096b59-4ae9-43da-a0c2-a689ad9fe2e9 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1 up in weight 1 up_from 13 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252] [v2:172.21.15.106:6812/1861847252,v1:172.21.15.106:6813/1861847252] exists,up 0b7f3c0d-489c-48b4-91ca-fa8da6baef53 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052] [v2:172.21.15.106:6820/2791969052,v1:172.21.15.106:6821/2791969052] exists,up b2453c8c-444c-4a5c-9c37-36f6f0213219 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435] [v2:172.21.15.106:6828/2233735435,v1:172.21.15.106:6829/2233735435] exists,up d1778ea7-0e45-4646-8fd2-b0f7c9e3086f 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341] [v2:172.21.15.136:6802/2130223341,v1:172.21.15.136:6803/2130223341] exists,up 0335aeb0-87b1-4cff-9351-c489ad188729 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833] [v2:172.21.15.136:6810/2282248833,v1:172.21.15.136:6811/2282248833] exists,up dc2279d3-fcec-4934-8a75-9befec1bbbe3 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865] [v2:172.21.15.136:6818/2133537865,v1:172.21.15.136:6819/2133537865] exists,up dddb0cd9-8f57-4836-96fd-f66925b36c84 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7 up in weight 1 up_from 54 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513] [v2:172.21.15.136:6826/1537958513,v1:172.21.15.136:6827/1537958513] exists,up b9894fb4-6f34-47cd-81cd-9e7d9758360b 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876] [v2:172.21.15.173:6802/526702876,v1:172.21.15.173:6803/526702876] exists,up 046e019e-1e33-403c-9577-757c7d546f83 2024-08-30T23:25:40.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222] [v2:172.21.15.173:6810/478092222,v1:172.21.15.173:6811/478092222] exists,up ea9ca538-e1da-4687-8b4f-542c0fc3737c 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10 up in weight 1 up_from 78 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808] [v2:172.21.15.173:6818/225763808,v1:172.21.15.173:6819/225763808] exists,up 085c32cd-dde4-4a54-bb0d-9182075040c2 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084] [v2:172.21.15.173:6826/2219423084,v1:172.21.15.173:6827/2219423084] exists,up 6bd9ce25-ce61-4998-af79-26c4ebe2ce3b 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6801/1186392683 expires 2024-08-31T23:18:51.239661+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/3708473409 expires 2024-08-31T23:18:51.239661+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/1031615522 expires 2024-08-31T23:18:51.239661+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/3817969167 expires 2024-08-31T23:18:51.239661+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/3688603346 expires 2024-08-31T23:18:27.701240+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6801/1549137855 expires 2024-08-31T23:18:27.701240+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6800/1186392683 expires 2024-08-31T23:18:51.239661+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/3790397394 expires 2024-08-31T23:18:27.701240+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:6800/1549137855 expires 2024-08-31T23:18:27.701240+0000 2024-08-30T23:25:40.304 INFO:teuthology.orchestra.run.smithi106.stdout:blocklist 172.21.15.106:0/1787333807 expires 2024-08-31T23:18:27.701240+0000 2024-08-30T23:25:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/599426179' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-30T23:25:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/599426179' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-30T23:25:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/599426179' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-30T23:25:41.034 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 54 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6802/3938533896,v1:172.21.15.106:6803/3938533896] [v2:172.21.15.106:6804/3938533896,v1:172.21.15.106:6805/3938533896] exists,up 48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'osd.1 up in weight 1 up_from 13 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6810/1861847252,v1:172.21.15.106:6811/1861847252] [v2:172.21.15.106:6812/1861847252,v1:172.21.15.106:6813/1861847252] exists,up 0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6818/2791969052,v1:172.21.15.106:6819/2791969052] [v2:172.21.15.106:6820/2791969052,v1:172.21.15.106:6821/2791969052] exists,up b2453c8c-444c-4a5c-9c37-36f6f0213219', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6826/2233735435,v1:172.21.15.106:6827/2233735435] [v2:172.21.15.106:6828/2233735435,v1:172.21.15.106:6829/2233735435] exists,up d1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/2130223341,v1:172.21.15.136:6801/2130223341] [v2:172.21.15.136:6802/2130223341,v1:172.21.15.136:6803/2130223341] exists,up 0335aeb0-87b1-4cff-9351-c489ad188729', 'osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6808/2282248833,v1:172.21.15.136:6809/2282248833] [v2:172.21.15.136:6810/2282248833,v1:172.21.15.136:6811/2282248833] exists,up dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/2133537865,v1:172.21.15.136:6817/2133537865] [v2:172.21.15.136:6818/2133537865,v1:172.21.15.136:6819/2133537865] exists,up dddb0cd9-8f57-4836-96fd-f66925b36c84', 'osd.7 up in weight 1 up_from 54 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/1537958513,v1:172.21.15.136:6825/1537958513] [v2:172.21.15.136:6826/1537958513,v1:172.21.15.136:6827/1537958513] exists,up b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'osd.8 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/526702876,v1:172.21.15.173:6801/526702876] [v2:172.21.15.173:6802/526702876,v1:172.21.15.173:6803/526702876] exists,up 046e019e-1e33-403c-9577-757c7d546f83', 'osd.9 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/478092222,v1:172.21.15.173:6809/478092222] [v2:172.21.15.173:6810/478092222,v1:172.21.15.173:6811/478092222] exists,up ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'osd.10 up in weight 1 up_from 78 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/225763808,v1:172.21.15.173:6817/225763808] [v2:172.21.15.173:6818/225763808,v1:172.21.15.173:6819/225763808] exists,up 085c32cd-dde4-4a54-bb0d-9182075040c2', 'osd.11 up in weight 1 up_from 86 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/2219423084,v1:172.21.15.173:6825/2219423084] [v2:172.21.15.173:6826/2219423084,v1:172.21.15.173:6827/2219423084] exists,up 6bd9ce25-ce61-4998-af79-26c4ebe2ce3b'] 2024-08-30T23:25:41.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon mon.a config show 2024-08-30T23:25:41.277 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:41.649 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:41 smithi106 ceph-mon[16603]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "name": "mon.a", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster": "ceph", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "admin_socket_mode": "", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_client_required": "cephx, none", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_cluster_required": "cephx", 2024-08-30T23:25:41.743 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_debug": "true", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_service_required": "cephx", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_service_ticket_ttl": "240.000000", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "auth_supported": "", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_aio": "true", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_aio_max_queue_depth": "1024", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_aio_poll_ms": "250", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_aio_reap_max": "16", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_async_discard_threads": "0", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_block_size": "4096", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_debug_aio": "false", 2024-08-30T23:25:41.744 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_debug_inflight_ios": "false", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_enable_discard": "false", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_flock_retry": "3", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_flock_retry_interval": "0.100000", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_inject_crash": "0", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_inject_crash_flush_delay": "2", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_ioring": "false", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_ioring_hipri": "false", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_ioring_sqthread_poll": "false", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-08-30T23:25:41.745 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_read_buffer_alignment": "4096", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bdev_type": "", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_alloc_size": "1048576", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_allocator": "hybrid", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_buffered_io": "true", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_check_for_zeros": "false", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_check_volume_selector_often": "false", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_compact_log_sync": "false", 2024-08-30T23:25:41.746 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_log_compact_min_size": "16777216", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_log_replay_check_allocations": "true", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_max_log_runway": "4194304", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_max_prefetch": "1048576", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_min_flush_size": "524288", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_min_log_runway": "1048576", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_replay_recovery": "false", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_shared_alloc_size": "65536", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_sync_write": "false", 2024-08-30T23:25:41.747 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_allocation_from_file": "true", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_allocator": "hybrid", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_async_db_compaction": "true", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-08-30T23:25:41.748 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bdev_label_multi": "true", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bdev_label_require_all": "true", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blobid_prealloc": "10240", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_create": "true", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_db_create": "false", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_db_path": "", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_db_size": "0", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_path": "", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_preallocate_file": "false", 2024-08-30T23:25:41.749 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_size": "107374182400", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_wal_create": "false", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_wal_path": "", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_block_wal_size": "100663296", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bluefs": "true", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bluefs_env_mirror": "false", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-08-30T23:25:41.750 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_autotune": "true", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_size": "0", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_type": "2q", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_clone_cow": "true", 2024-08-30T23:25:41.751 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_algorithm": "snappy", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_max_blob_size": "0", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_min_blob_size": "0", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_mode": "none", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_csum_type": "crc32c", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_enforce_settings": "default", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_freelist": "false", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_fsck_abort": "false", 2024-08-30T23:25:41.752 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_inject_read_err": "false", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_legacy_omap": "false", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_small_allocations": "0", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-08-30T23:25:41.753 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_default_buffered_read": "true", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_default_buffered_write": "false", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_deferred_batch_ops": "0", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_elastic_shared_blobs": "true", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fail_eio": "false", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-08-30T23:25:41.754 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_on_mkfs": "true", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_on_mount": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_on_umount": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-08-30T23:25:41.755 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_ignore_data_csum": "false", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_kvbackend": "rocksdb", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_log_op_age": "5.000000", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_max_alloc_size": "0", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_max_blob_size": "0", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_max_defer_interval": "3.000000", 2024-08-30T23:25:41.756 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_max_deferred_txc": "32", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_min_alloc_size": "0", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_nid_prealloc": "1024", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_prefer_deferred_size": "0", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-08-30T23:25:41.757 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_qfsck_on_mount": "true", 2024-08-30T23:25:41.758 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_retry_disk_reads": "3", 2024-08-30T23:25:41.758 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_rocksdb_cf": "true", 2024-08-30T23:25:41.758 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-08-30T23:25:41.758 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_rocksdb_options_annex": "", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_spdk_coremask": "0x1", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_spdk_io_sleep": "5", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_spdk_max_io_completion": "0", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_spdk_mem": "512", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_sync_submit_transaction": "false", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_throttle_bytes": "67108864", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_throttle_cost_per_io": "0", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-08-30T23:25:41.759 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_tracing": "false", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_volume_selection_reserved": "0", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_write_v2": "false", 2024-08-30T23:25:41.760 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_write_v2_random": "false", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_zero_block_detection": "false", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_action_update_interval": "2", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_mount_timeout": "10", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-08-30T23:25:41.761 INFO:teuthology.orchestra.run.smithi106.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_cluster_require_signatures": "false", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_cluster_require_version": "2", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_require_signatures": "false", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_require_version": "2", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_service_require_signatures": "false", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_service_require_version": "2", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "cephx_sign_messages": "true", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "chdir": "", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "client_acl_type": "", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "client_asio_thread_count": "2", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "client_cache_mid": "0.750000", 2024-08-30T23:25:41.762 INFO:teuthology.orchestra.run.smithi106.stdout: "client_cache_size": "16384", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_caps_release_delay": "5", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_check_pool_perm": "true", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_collect_and_send_global_metrics": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_debug_force_sync_read": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_debug_getattr_caps": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_debug_inject_features": "", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_debug_inject_tick_delay": "0", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_die_on_failed_remount": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_dirsize_rbytes": "true", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_force_lazyio": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_fs": "", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_inject_fixed_oldest_tid": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_inject_release_failure": "false", 2024-08-30T23:25:41.763 INFO:teuthology.orchestra.run.smithi106.stdout: "client_max_inline_size": "4096", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_max_retries_on_remount_failure": "5", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_mds_namespace": "", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_metadata": "", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_mount_gid": "-1", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_mount_timeout": "300", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_mount_uid": "-1", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_mountpoint": "/", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_notify_timeout": "10", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_oc": "true", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_oc_max_dirty": "104857600", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_oc_max_dirty_age": "5.000000", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_oc_max_objects": "1000", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_oc_size": "209715200", 2024-08-30T23:25:41.764 INFO:teuthology.orchestra.run.smithi106.stdout: "client_oc_target_dirty": "8388608", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_permissions": "true", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_quota": "true", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_quota_df": "true", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_readahead_max_bytes": "0", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_readahead_max_periods": "4", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_readahead_min": "131072", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_reconnect_stale": "false", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_shutdown_timeout": "30", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_snapdir": ".snap", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_tick_interval": "1", 2024-08-30T23:25:41.765 INFO:teuthology.orchestra.run.smithi106.stdout: "client_trace": "", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "client_try_dentry_invalidate": "false", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "client_use_faked_inos": "false", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "client_use_random_mds": "false", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_graylog": "false", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_graylog_port": "12201", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_monitors": "default=true", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_syslog": "false", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "clog_to_syslog_level": "info", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "-", 2024-08-30T23:25:41.766 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_network": "", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_network_interface": "", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "compressor_zlib_isal": "false", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "compressor_zlib_level": "5", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "compressor_zlib_winsize": "-15", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "compressor_zstd_level": "1", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_alien_op_num_threads": "6", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_alien_thread_cpu_cores": "", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_osd_obc_lru_size": "512", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_osd_stat_interval": "0", 2024-08-30T23:25:41.767 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_seastar_cpu_cores": "", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "crimson_seastar_num_threads": "0", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_location": "", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_location_hook": "", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_location_hook_timeout": "10", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "daemonize": "false", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "daos_pool": "tank", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "dbstore_db_name_prefix": "dbstore", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_alienstore": "0/5", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_allow_any_pool_priority": "false", 2024-08-30T23:25:41.768 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_asok": "1/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_asok_assert_abort": "false", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_asserts_on_shutdown": "true", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_auth": "1/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_bdev": "1/3", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_bluefs": "1/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_bluestore": "1/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_buffer": "0/1", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_ceph_dedup": "0/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_ceph_exporter": "1/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_cephfs_mirror": "0/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_cephsqlite": "0/5", 2024-08-30T23:25:41.769 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_client": "0/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_compressor": "1/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_context": "0/1", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_crimson_interrupt": "0/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_crush": "1/1", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_crypto": "1/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_cyanstore": "0/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_deliberately_leak_memory": "false", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_disable_randomized_ping": "false", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_dpdk": "1/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_eventtrace": "1/5", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_filer": "0/1", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_filestore": "1/3", 2024-08-30T23:25:41.770 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_finisher": "1/1", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_fuse": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_heartbeat_testing_span": "0", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_heartbeatmap": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_immutable_obj_cache": "0/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_javaclient": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_journal": "1/3", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_journaler": "0/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_kstore": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_lockdep": "0/1", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mclock": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds_balancer": "1/5", 2024-08-30T23:25:41.771 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds_locker": "1/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds_log": "1/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds_log_expire": "1/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds_migrator": "1/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mds_quiesce": "3/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_memstore": "1/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mgr": "2/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mgrc": "1/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_mon": "20/20", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_monc": "0/10", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_ms": "1/1", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_none": "0/5", 2024-08-30T23:25:41.772 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_objclass": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_objectcacher": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_objecter": "0/1", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_optracker": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_osd": "1/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_paxos": "20/20", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_perfcounter": "1/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_prioritycache": "1/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rados": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rbd": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rbd_mirror": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rbd_pwl": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rbd_replay": "0/5", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_refs": "0/0", 2024-08-30T23:25:41.773 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_reserver": "1/1", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_access": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_datacache": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_dbstore": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_flight": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_lifecycle": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_notification": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rgw_sync": "1/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_rocksdb": "4/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore": "0/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_backref": "0/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_cache": "0/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_cleaner": "0/5", 2024-08-30T23:25:41.774 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_device": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_epm": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_journal": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_lba": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_odata": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_omap": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_onode": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_t": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_seastore_tm": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_striper": "0/1", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_test": "0/5", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_throttle": "1/1", 2024-08-30T23:25:41.775 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_timer": "0/1", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_tp": "0/5", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "debug_trace": "1/5", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "device_failure_prediction_mode": "none", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "err_to_graylog": "false", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "err_to_journald": "false", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "err_to_stderr": "true", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "err_to_syslog": "false", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event_tracing": "false", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_addr": "0.0.0.0", 2024-08-30T23:25:41.776 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_cert_file": "", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_http_port": "9926", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_key_file": "", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_prio_limit": "5", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_sort_metrics": "true", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "exporter_stats_period": "5", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "fake_statfs_for_testing": "0", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "fatal_signal_handlers": "true", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "filer_max_purge_ops": "10", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "filer_max_truncate_ops": "128", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_apply_finisher_threads": "1", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_blackhole": "false", 2024-08-30T23:25:41.777 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_btrfs_clone_range": "true", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_btrfs_snap": "true", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_caller_concurrency": "10", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_collect_device_partition_information": "true", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_commit_timeout": "600.000000", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_debug_inject_read_err": "false", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_debug_omap_check": "false", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_debug_verify_split": "false", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_dump_file": "", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fadvise": "true", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fail_eio": "true", 2024-08-30T23:25:41.778 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fd_cache_shards": "16", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fd_cache_size": "128", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fiemap": "false", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fiemap_threshold": "4096", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_index_retry_probability": "0.000000", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_inject_stall": "0", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_journal_parallel": "false", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_journal_trailing": "false", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_journal_writeahead": "false", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_kill_at": "0", 2024-08-30T23:25:41.779 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattr_size": "0", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattrs": "0", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattrs_other": "2", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_sync_interval": "5.000000", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_xattr_value_size": "0", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-08-30T23:25:41.780 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_merge_threshold": "-10", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_min_sync_interval": "0.010000", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_odsync_write": "false", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_omap_backend": "rocksdb", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_omap_backend_path": "", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_omap_header_cache_size": "1024", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_ondisk_finisher_threads": "1", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_op_thread_timeout": "60", 2024-08-30T23:25:41.781 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_op_threads": "2", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_punch_hole": "false", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_max_bytes": "104857600", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-08-30T23:25:41.782 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_queue_max_ops": "50", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_seek_data_hole": "false", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_sloppy_crc": "false", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_splice": "false", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_split_multiple": "2", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_split_rand_factor": "20", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_update_to": "1000", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-08-30T23:25:41.783 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_enable": "true", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_xfs_extsize": "false", 2024-08-30T23:25:41.784 INFO:teuthology.orchestra.run.smithi106.stdout: "filestore_zfs_snap": "false", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fio_dir": "/tmp/fio", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_allow_other": "true", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_atomic_o_trunc": "true", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_big_writes": "true", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_debug": "false", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_default_permissions": "false", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_disable_pagecache": "false", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_max_write": "0", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_multithreaded": "true", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_require_active_mds": "true", 2024-08-30T23:25:41.785 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_set_user_groups": "true", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_splice_move": "true", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_splice_read": "true", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_splice_write": "true", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_syncfs_on_mksnap": "true", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "fuse_use_invalidate_cb": "true", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "gss_target_name": "ceph", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_file": "", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_inject_failure": "0", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_interval": "5", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "host": "smithi106", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-08-30T23:25:41.786 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_max_size": "1073741824", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "immutable_object_cache_watermark": "0.900000", 2024-08-30T23:25:41.787 INFO:teuthology.orchestra.run.smithi106.stdout: "inject_early_sigterm": "false", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "jaeger_agent_port": "6799", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "jaeger_tracing_enable": "false", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_aio": "true", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_align_min_size": "65536", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_block_align": "true", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_block_size": "4096", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_dio": "true", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_discard": "false", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_force_aio": "false", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_ignore_corruption": "false", 2024-08-30T23:25:41.788 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_max_write_bytes": "10485760", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_max_write_entries": "100", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_replay_from": "0", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_throttle_high_multiple": "0.000000", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_throttle_max_multiple": "0.000000", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_write_header_frequency": "0", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journal_zero_on_create": "false", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journaler_prefetch_periods": "10", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journaler_prezero_periods": "5", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "journaler_write_head_interval": "15", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "key": "", 2024-08-30T23:25:41.789 INFO:teuthology.orchestra.run.smithi106.stdout: "keyfile": "", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_backend": "rocksdb", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_default_stripe_size": "65536", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_fsck_on_mount": "false", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_fsck_on_mount_deep": "true", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_max_bytes": "67108864", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_max_ops": "512", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_nid_prealloc": "1024", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_onode_map_size": "1024", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_sync_submit_transaction": "false", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "kstore_sync_transaction": "false", 2024-08-30T23:25:41.790 INFO:teuthology.orchestra.run.smithi106.stdout: "librados_thread_count": "2", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "lockdep": "false", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "lockdep_force_backtrace": "false", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_coarse_timestamps": "true", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_flush_on_exit": "false", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_graylog_host": "127.0.0.1", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_graylog_port": "12201", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_max_new": "1000", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_max_recent": "10000", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_stderr_prefix": "", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_stop_at_utilization": "0.970000", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_file": "true", 2024-08-30T23:25:41.791 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_graylog": "false", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_journald": "false", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_stderr": "false", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "log_to_syslog": "false", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "max_rotating_auth_attempts": "10", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_action_on_write_error": "1", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_alternate_name_max": "8192", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_asio_thread_count": "2", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_export_pin": "true", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_fragment_dirs": "true", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-08-30T23:25:41.792 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_fragment_interval": "5", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_fragment_size_max": "100000", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_idle_threshold": "0.000000", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_interval": "10", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_max": "-1", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_max_until": "-1", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_merge_size": "50", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_midchunk": "0.300000", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_min_rebalance": "0.100000", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_min_start": "0.200000", 2024-08-30T23:25:41.793 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_minchunk": "0.001000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_mode": "0", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_need_max": "1.200000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_need_min": "0.800000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_overload_epochs": "2", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_sample_interval": "3.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_split_bits": "3", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_split_rd": "25000.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_split_size": "10000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_split_wr": "10000.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_target_decay": "10.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_beacon_grace": "15.000000", 2024-08-30T23:25:41.794 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_beacon_interval": "4.000000", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_beacon_mon_down_grace": "60", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_memory_limit": "4294967296", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_mid": "0.700000", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_quiesce_delay": "0", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_quiesce_sleep": "200", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_quiesce_splitauth": "true", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_quiesce_threshold": "524288", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_release_free_interval": "10", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_reservation": "0.050000", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_trim_interval": "1", 2024-08-30T23:25:41.795 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache_trim_threshold": "262144", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_client_delegate_inos_pct": "50", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_client_prealloc_inos": "1000", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_connect_bootstrapping": "false", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_damage_table_max_entries": "10000", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_debug_auth_pins": "false", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_debug_frag": "false", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_debug_scatterstat": "false", 2024-08-30T23:25:41.796 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_debug_subtrees": "false", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_decay_halflife": "5.000000", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_default_dir_hash": "2", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_defer_session_stale": "true", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_deny_all_reconnect": "false", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dir_keys_per_op": "16384", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dir_max_commit_size": "10", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dir_max_entries": "0", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dir_prefetch": "true", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dirstat_min_interval": "1.000000", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dump_cache_after_rejoin": "false", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dump_cache_on_map": "false", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dump_cache_threshold_file": "0", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_early_reply": "true", 2024-08-30T23:25:41.797 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_enable_op_tracker": "true", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_enforce_unique_name": "true", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_export_ephemeral_distributed": "true", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_export_ephemeral_random": "true", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_forward_all_requests_to_auth": "false", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_health_cache_threshold": "1.500000", 2024-08-30T23:25:41.798 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_health_summarize_threshold": "10", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_heartbeat_grace": "15.000000", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_heartbeat_reset_grace": "1000", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_inject_health_dummy": "false", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_inject_migrator_session_race": "false", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_join_fs": "", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_journal_format": "1", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_create_at": "0", 2024-08-30T23:25:41.799 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_dirfrag_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_export_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_import_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_journal_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_journal_expire_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_journal_replay_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_link_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_mdstable_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_openc_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_rename_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_kill_shutdown_at": "0", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_event_large_threshold": "524288", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_events_per_segment": "1024", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_major_segment_event_ratio": "12", 2024-08-30T23:25:41.800 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_max_events": "-1", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_max_segments": "128", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_pause": "false", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_segment_size": "0", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_skip_corrupt_events": "false", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_skip_unbounded_events": "false", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_trim_threshold": "128", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log_warn_factor": "2.000000", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_caps_per_client": "1048576", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_completed_flushes": "100000", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_completed_requests": "100000", 2024-08-30T23:25:41.801 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_export_size": "20971520", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_file_recover": "32", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_purge_files": "64", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_purge_ops": "8192", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_max_snaps_per_dir": "100", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_metrics_update_interval": "2", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_min_caps_per_client": "100", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_min_caps_working_set": "10000", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_numa_node": "-1", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_op_complaint_time": "30.000000", 2024-08-30T23:25:41.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_op_history_duration": "600", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_op_history_size": "20", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_op_history_slow_op_size": "20", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_op_history_slow_op_threshold": "10", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_op_log_threshold": "5", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_ping_grace": "15", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_ping_interval": "5", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_recall_max_caps": "30000", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_recall_max_decay_threshold": "131072", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_recall_warning_threshold": "262144", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_reconnect_timeout": "45.000000", 2024-08-30T23:25:41.803 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_replay_interval": "1.000000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_root_ino_gid": "0", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_root_ino_uid": "0", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_blocklist_on_evict": "true", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_blocklist_on_timeout": "true", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-08-30T23:25:41.804 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_session_metadata_threshold": "16777216", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_shutdown_check": "0", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_skip_ino": "0", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sleep_rank_change": "0.000000", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_snap_max_uid": "4294967294", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_snap_min_uid": "0", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_snap_rstat": "false", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_standby_replay_damaged": "false", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_symlink_recovery": "true", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_task_status_update_interval": "2.000000", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_thrash_exports": "0", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_thrash_fragments": "0", 2024-08-30T23:25:41.805 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_tick_interval": "5.000000", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_valgrind_exit": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_verify_backtrace": "1", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_verify_scatter": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_wipe_ino_prealloc": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_wipe_sessions": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool_debug": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "memstore_debug_omit_block_device_write": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "memstore_device_bytes": "1073741824", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "memstore_page_set": "false", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "memstore_page_size": "65536", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_client_bytes": "134217728", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_client_messages": "512", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-08-30T23:25:41.806 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_connect_retry_interval": "1.000000", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_disabled_modules": "", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_enable_op_tracker": "true", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_max_pg_creating": "1024", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_max_pg_num_change": "128", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_mds_bytes": "134217728", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_mds_messages": "128", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_mon_bytes": "134217728", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_mon_messages": "128", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_num_op_tracker_shard": "32", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_op_complaint_time": "30.000000", 2024-08-30T23:25:41.807 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_op_history_duration": "600", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_op_history_size": "20", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_op_history_slow_op_size": "20", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_op_log_threshold": "5", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_osd_bytes": "536870912", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_osd_messages": "8192", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_pool": "true", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_service_beacon_grace": "60.000000", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_standby_modules": "true", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_stats_period": "5", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_stats_threshold": "5", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_tick_period": "2", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-08-30T23:25:41.808 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_accept_timeout_factor": "2.000000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_allow_pool_delete": "true", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_allow_pool_size_one": "false", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_auth_validate_all_caps": "true", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_bytes": "104857600", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_directed_command_retry": "5", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_hunt_interval": "3.000000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_hunt_parallel": "3", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_log_interval": "1.000000", 2024-08-30T23:25:41.809 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_ping_interval": "10.000000", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_ping_timeout": "30.000000", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_client_target_rank": "-1", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_clock_drift_allowed": "1.000000", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_level": "debug", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_file": "true", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_graylog": "false", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-08-30T23:25:41.810 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_journald": "true", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_stderr": "false", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_compact_on_bootstrap": "false", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_compact_on_start": "false", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_compact_on_trim": "true", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_con_tracker_persist_interval": "10", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_con_tracker_score_halflife": "43200", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_config_key_max_entry_size": "65536", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_cpu_threads": "4", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_crush_min_required_version": "hammer", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_daemon_bytes": "419430400", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-08-30T23:25:41.811 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_data_avail_crit": "5", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_data_avail_warn": "5", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_data_size_warn": "16106127360", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_block_osdmap_trim": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_dump_json": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_dump_transactions": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_extra_checks": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_no_require_reef": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_no_require_squid": "false", 2024-08-30T23:25:41.812 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_delta_reset_interval": "10.000000", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_dns_srv_name": "ceph-mon", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_down_mkfs_grace": "60", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_down_uptime_grace": "60", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_election_default_strategy": "3", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_election_timeout": "5.000000", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_elector_ping_divisor": "2", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_elector_ping_timeout": "2.000000", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_enable_op_tracker": "true", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_fake_pool_delete": "false", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_force_quorum_join": "false", 2024-08-30T23:25:41.813 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_fsmap_prune_threshold": "300", 2024-08-30T23:25:41.816 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_globalid_prealloc": "10000", 2024-08-30T23:25:41.816 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_health_detail_to_clog": "true", 2024-08-30T23:25:41.816 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_health_log_update_period": "5", 2024-08-30T23:25:41.816 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_health_max_detail": "50", 2024-08-30T23:25:41.816 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_health_to_clog": "true", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_health_to_clog_interval": "600", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_host": "[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_host_override": "", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_initial_members": "", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_keyvaluedb": "rocksdb", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_lease": "5.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-08-30T23:25:41.817 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_log_full_interval": "50", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_log_max": "10000", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_log_max_summary": "50", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_log_entries_per_event": "4096", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_log_epochs": "500", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_mdsmap_epochs": "500", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_mgrmap_epochs": "500", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_nvmeof_epochs": "500", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_osd": "10000", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_pg_per_osd": "10000", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_pool_pg_num": "65536", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-08-30T23:25:41.818 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mds_force_trim_to": "0", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mds_skip_sanity": "false", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_memory_autotune": "true", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_memory_target": "2147483648", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mgr_beacon_grace": "30", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mgr_digest_period": "5", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mgr_inactive_grace": "60", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mgr_mkfs_grace": "240", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_min_osdmap_epochs": "50", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-08-30T23:25:41.819 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_op_complaint_time": "30", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_op_history_duration": "600", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_op_history_size": "20", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_op_history_slow_op_size": "20", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_op_history_slow_op_threshold": "10", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_op_log_threshold": "5", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_auto_mark_in": "false", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_auto_mark_new_in": "true", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-08-30T23:25:41.820 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_cache_size": "500", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_cache_size_min": "134217728", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_crush_smoke_test": "true", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_destroyed_out_interval": "600", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_down_out_interval": "600", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_force_trim_to": "0", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_full_ratio": "0.950000", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_laggy_halflife": "3600", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_laggy_max_interval": "300", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_laggy_weight": "0.300000", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-08-30T23:25:41.821 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_max_initial_pgs": "1024", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_min_down_reporters": "2", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_prime_pg_temp": "true", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_report_timeout": "900", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-08-30T23:25:41.822 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_warn_num_repaired": "10", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osd_warn_op_age": "32.000000", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osdmap_full_prune_interval": "2", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osdmap_full_prune_min": "15", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pg_stuck_threshold": "60", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-08-30T23:25:41.824 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pg_warn_min_objects": "10000", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pg_warn_min_per_osd": "0", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pool_quota_crit_threshold": "0", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_pool_quota_warn_threshold": "0", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_probe_timeout": "2.000000", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_reweight_max_change": "0.050000", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_reweight_max_osds": "4", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-08-30T23:25:41.825 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_scrub_interval": "300", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_scrub_max_keys": "100", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_scrub_timeout": "300", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_session_timeout": "300", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_smart_report_timeout": "5", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_stat_smooth_intervals": "6", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_stretch_pool_min_size": "2", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_stretch_pool_size": "4", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_subscribe_interval": "86400.000000", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_sync_debug": "false", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_sync_max_payload_keys": "2000", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_sync_max_payload_size": "1048576", 2024-08-30T23:25:41.826 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_sync_provider_kill_at": "0", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_sync_requester_kill_at": "0", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_sync_timeout": "60.000000", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_target_pg_per_osd": "100", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_tick_interval": "5", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_timecheck_interval": "300.000000", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_use_min_delay_socket": "false", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_older_version_delay": "604800", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_filestore_osds": "true", 2024-08-30T23:25:41.827 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_misplaced": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_older_version": "true", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_pool_no_app": "true", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_on_too_few_osds": "false", 2024-08-30T23:25:41.828 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "monmap": "", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "motr_tracing_enabled": "false", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_op_threads": "3", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_buffer_size": "131072", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_cm": "false", 2024-08-30T23:25:41.829 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_device_name": "", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_dscp": "96", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_gid_idx": "0", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_local_gid": "", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_polling_us": "1000", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_port_num": "1", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_roce_ver": "1", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_send_buffers": "1024", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_sl": "3", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_support_srq": "true", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_rdma_type": "ib", 2024-08-30T23:25:41.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_async_reap_threshold": "5", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_before_connect": "false", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_ipv4": "true", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_ipv6": "false", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_msgr1": "true", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_msgr2": "true", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_port_max": "7568", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_port_min": "6800", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_prefer_ipv4": "false", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_retry_count": "3", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_bind_retry_delay": "5", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_blackhole_client": "false", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_blackhole_mds": "false", 2024-08-30T23:25:41.831 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_blackhole_mgr": "false", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_blackhole_mon": "false", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_blackhole_osd": "false", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_client_mode": "crc secure", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_cluster_mode": "crc secure", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_cluster_type": "", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_compress_secure": "false", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_connection_idle_timeout": "900", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_connection_ready_timeout": "10", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_crc_data": "true", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_crc_header": "true", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_die_on_bad_msg": "false", 2024-08-30T23:25:41.832 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_die_on_bug": "true", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_die_on_old_message": "true", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_die_on_skipped_message": "false", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_die_on_unhandled_msg": "false", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_coremask": "0xF", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_devs_allowlist": "", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_enable_tso": "true", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_hugepages": "", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_hw_flow_control": "true", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-08-30T23:25:41.833 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_lro": "true", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_memory_channel": "4", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_pmd": "", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_port_id": "0", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dump_corrupt_message_level": "1", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_dump_on_send": "false", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_initial_backoff": "0.200000", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_inject_delay_max": "1.000000", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_inject_delay_probability": "0.005000", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_inject_delay_type": "osd", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_inject_internal_delays": "0.002000", 2024-08-30T23:25:41.834 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_inject_network_congestion": "0", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_inject_socket_failures": "2500", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_learn_addr_from_peer": "true", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_max_accept_failures": "4", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_max_backoff": "15.000000", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_mon_client_mode": "secure crc", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_mon_cluster_mode": "secure crc", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_mon_service_mode": "secure crc", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_osd_compress_min_size": "1024", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_osd_compress_mode": "none", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_osd_compression_algorithm": "snappy", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_pq_min_cost": "65536", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_public_type": "", 2024-08-30T23:25:41.835 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_service_mode": "crc secure", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_tcp_listen_backlog": "512", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_tcp_nodelay": "true", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_tcp_rcvbuf": "0", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_time_events_min_wait_interval": "1000", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "ms_type": "async+posix", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "no_config_file": "false", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "nvmeof_mon_client_tick_period": "2", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_completion_locks_per_session": "32", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_debug_inject_relock_delay": "false", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_inflight_op_bytes": "104857600", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_inflight_ops": "1024", 2024-08-30T23:25:41.836 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_inject_no_watch_ping": "false", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_tick_interval": "5.000000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter_timeout": "10.000000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "objectstore_blackhole": "false", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "openssl_engine_opts": "", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_delay_time": "5.000000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_hist_halflife": "1000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_max_low_ops": "2", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_max_ops": "4", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_quantize_effort": "0.100000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_agent_slop": "0.020000", 2024-08-30T23:25:41.837 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_allow_recovery_below_min_size": "true", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_asio_thread_count": "2", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_async_recovery_min_cost": "100", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_auto_mark_unfound_lost": "false", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_backfill_retry_interval": "30.000000", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_backfill_scan_max": "512", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_backfill_scan_min": "64", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_backoff_on_degraded": "false", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_backoff_on_peering": "false", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_backoff_on_unfound": "true", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_beacon_report_interval": "300", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bench_duration": "30", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-08-30T23:25:41.838 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bench_max_block_size": "67108864", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bench_small_size_max_iops": "100", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_blkin_trace_all": "false", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_blocked_scrub_grace_period": "120", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_check_for_log_corruption": "false", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_class_update_on_start": "true", 2024-08-30T23:25:41.839 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_client_message_cap": "256", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_client_message_size_cap": "524288000", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_client_op_priority": "63", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_client_watch_timeout": "30", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_command_max_records": "256", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_compact_on_start": "false", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_crush_chooseleaf_type": "1", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_crush_initial_weight": "-1.000000", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_crush_update_on_start": "true", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_crush_update_weight_set": "true", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-08-30T23:25:41.840 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_drop_ping_duration": "0", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_feed_pullee": "-1", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_misdirected_ops": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_no_acting_change": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_no_purge_strays": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_op_order": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_pg_log_writeout": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_pretend_recovery_active": "false", 2024-08-30T23:25:41.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_shutdown": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_trim_objects": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_verify_cached_snaps": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_verify_missing_on_start": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_verify_snaps": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_keys": "1024", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-08-30T23:25:41.842 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_stride": "524288", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_default_data_pool_replay_window": "45", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_default_notify_timeout": "30", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_delete_sleep": "0.000000", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_discard_disconnected_ops": "true", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_ec_partial_reads": "true", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_enable_op_tracker": "true", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_extblkdev_plugins": "vdo", 2024-08-30T23:25:41.843 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_fast_info": "true", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_fast_shutdown": "true", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_fast_shutdown_timeout": "15", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_function_tracing": "false", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_grace": "20", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_interval": "6", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-08-30T23:25:41.844 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_min_peers": "10", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_min_size": "2000", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_stale": "600", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_hit_set_max_size": "100000", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_hit_set_min_size": "1000", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_journal_flush_on_shutdown": "true", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_journal_size": "5120", 2024-08-30T23:25:41.845 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_kill_backfill_at": "0", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_loop_before_reset_tphandle": "64", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_map_cache_size": "50", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_map_dedup": "true", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_map_message_max": "40", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_map_message_max_bytes": "10485760", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_map_share_max_epochs": "40", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_attr_name_len": "100", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_attr_size": "0", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_backfills": "1", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_markdown_count": "5", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_markdown_period": "600", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_object_name_len": "2048", 2024-08-30T23:25:41.846 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_object_namespace_len": "256", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_object_size": "134217728", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_omap_entries_per_request": "1024", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_pg_blocked_by": "16", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_pg_log_entries": "10000", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_pgls": "1024", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_push_cost": "8388608", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_push_objects": "10", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_scrubs": "3", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_trimming_pgs": "2", 2024-08-30T23:25:41.847 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_write_op_reply_len": "64", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_max_write_size": "90", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_override_recovery_settings": "false", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_profile": "balanced", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-08-30T23:25:41.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mclock_skip_benchmark": "false", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_base": "805306368", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_cache_min": "134217728", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-08-30T23:25:41.849 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_target": "4294967296", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_target_autotune": "false", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_min_pg_log_entries": "250", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_min_recovery_priority": "0", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mon_heartbeat_interval": "30", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mon_report_interval": "5", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mon_report_max_in_flight": "2", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_num_cache_shards": "32", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_num_op_tracker_shard": "32", 2024-08-30T23:25:41.850 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_numa_auto_affinity": "true", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_numa_node": "-1", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_numa_prefer_iface": "true", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_objecter_finishers": "1", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_objectstore": "bluestore", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_objectstore_fuse": "false", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_objectstore_ideal_list_max": "64", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_objectstore_tracing": "false", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_complaint_time": "30.000000", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_history_duration": "600", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_history_size": "20", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_history_slow_op_size": "20", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-08-30T23:25:41.851 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_log_threshold": "5", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_num_shards": "0", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_num_shards_hdd": "5", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_num_shards_ssd": "8", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_num_threads_per_shard": "0", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_pq_min_cost": "65536", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_queue": "mclock_scheduler", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_queue_cut_off": "high", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_thread_suicide_timeout": "150", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_op_thread_timeout": "15", 2024-08-30T23:25:41.852 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_open_classes_on_start": "true", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_os_flags": "0", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_peering_op_priority": "255", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_delete_cost": "1048576", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_delete_priority": "5", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_log_dups_tracked": "3000", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_log_trim_max": "10000", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_log_trim_min": "100", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_object_context_cache_count": "64", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-08-30T23:25:41.853 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_crimson": "false", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_crush_rule": "-1", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_ec_fast_read": "false", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_flag_bulk": "false", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-08-30T23:25:41.854 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_flag_nodelete": "false", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_flags": "0", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_min_size": "2", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_pg_num": "32", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_pgp_num": "0", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_read_ratio": "70", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_size": "3", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_default_type": "replicated", 2024-08-30T23:25:41.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pool_use_gmt_hitset": "true", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_push_per_object_cost": "1000", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_read_ec_check_for_errors": "false", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recover_clone_overlap": "true", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recover_clone_overlap_limit": "10", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_cost": "20971520", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_delay_start": "0.000000", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_max_active": "0", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_max_active_hdd": "3", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_max_active_ssd": "10", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_max_chunk": "8388608", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-08-30T23:25:41.856 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_max_single_start": "1", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_op_priority": "3", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_op_warn_multiple": "16", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_priority": "5", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_retry_interval": "30.000000", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_sleep": "0.000000", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_repair_during_recovery": "false", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_requested_scrub_priority": "120", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_rollback_to_cluster_snap": "", 2024-08-30T23:25:41.857 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_auto_repair": "false", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_begin_hour": "0", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_begin_week_day": "0", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_chunk_max": "25", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_chunk_min": "5", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_cost": "52428800", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_during_recovery": "false", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_end_hour": "0", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_end_week_day": "0", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_event_cost": "4096", 2024-08-30T23:25:41.858 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_invalid_stats": "true", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_load_threshold": "0.500000", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_max_interval": "604800.000000", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_max_preemptions": "5", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_min_interval": "86400.000000", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_priority": "5", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_scrub_sleep": "0.000000", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_shutdown_pgref_assert": "false", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_skip_data_digest": "false", 2024-08-30T23:25:41.859 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_smart_report_timeout": "5", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_snap_trim_cost": "1048576", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_snap_trim_priority": "5", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_snap_trim_sleep": "0.000000", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_stats_update_period_scrubbing": "15", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_target_transaction_size": "30", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-08-30T23:25:41.860 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_default_cache_mode": "writeback", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_tracing": "false", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_use_stale_snap": "false", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "osdc_blkin_trace_all": "false", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_kill_at": "0", 2024-08-30T23:25:41.861 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_max_join_drift": "10", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_min": "500", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_min_wait": "0.050000", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_propose_interval": "1.000000", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_service_trim_max": "500", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_service_trim_max_multiplier": "20", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_service_trim_min": "10", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_stash_full_interval": "25", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_trim_max": "500", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "paxos_trim_min": "250", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "perf": "true", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "pid_file": "", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-08-30T23:25:41.862 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "-", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrv": "", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "public_bind_addr": "-", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "public_network": "", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "public_network_interface": "", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "qat_compressor_busy_polling": "false", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "qat_compressor_enabled": "false", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "qat_compressor_session_max_number": "256", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rados_mon_op_timeout": "0", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rados_osd_op_timeout": "0", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rados_tracing": "false", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_atime_update_interval": "60", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_balance_parent_reads": "false", 2024-08-30T23:25:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_balance_snap_reads": "false", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_blkin_trace_all": "false", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_blocklist_expire_seconds": "0", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_blocklist_on_break_lock": "true", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache": "true", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_block_writes_upfront": "false", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_max_dirty": "25165824", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_max_dirty_object": "0", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_policy": "writearound", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_size": "33554432", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_target_dirty": "16777216", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-08-30T23:25:41.864 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_clone_copy_on_read": "false", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_compression_hint": "none", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_concurrent_management_ops": "10", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_clone_format": "auto", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_data_pool": "", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_features": "61", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_format": "2", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_map_options": "", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_order": "22", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_pool": "rbd", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_stripe_count": "0", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_default_stripe_unit": "0", 2024-08-30T23:25:41.865 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_disable_zero_copy_writes": "true", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_discard_granularity_bytes": "65536", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_enable_alloc_hint": "true", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_io_scheduler": "simple", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_commit_age": "5.000000", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_object_flush_interval": "0", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-08-30T23:25:41.866 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_order": "24", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_pool": "", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_journal_splay_width": "4", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_localize_parent_reads": "false", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_localize_snap_reads": "false", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_image_policy_type": "simple", 2024-08-30T23:25:41.867 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_autotune": "true", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_base": "805306368", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-08-30T23:25:41.868 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_memory_target": "4294967296", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirroring_delete_delay": "0", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirroring_replay_delay": "0", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_move_to_trash_on_remove": "false", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_mtime_update_interval": "60", 2024-08-30T23:25:41.869 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_non_blocking_aio": "true", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_op_thread_timeout": "60", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_op_threads": "1", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_parent_cache_enabled": "false", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_persistent_cache_mode": "disabled", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_persistent_cache_path": "/tmp", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_persistent_cache_size": "1073741824", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_plugins": "", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_bps_burst": "0", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_bps_limit": "0", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_exclude_ops": "0", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_iops_burst": "0", 2024-08-30T23:25:41.870 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_iops_limit": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_read_bps_burst": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_read_bps_limit": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_read_iops_burst": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_read_iops_limit": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_schedule_tick_min": "50", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_write_bps_burst": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_write_bps_limit": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_write_iops_burst": "0", 2024-08-30T23:25:41.871 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_qos_write_iops_limit": "0", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_quiesce_notification_attempts": "10", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_read_from_replica_policy": "default", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_readahead_max_bytes": "524288", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_readahead_trigger_requests": "10", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_request_timed_out_seconds": "30", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_skip_partial_discard": "true", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_tracing": "false", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_validate_names": "true", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "rbd_validate_pool": "true", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "restapi_base_url": "", 2024-08-30T23:25:41.872 INFO:teuthology.orchestra.run.smithi106.stdout: "restapi_log_level": "", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_account_default_quota_max_size": "-1", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_acl_grants_max_num": "100", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_admin_entry": "admin", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_asio_assert_yielding": "false", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_backend_store": "rados", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_barbican_url": "", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_beast_enable_async": "true", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_counters_cache": "false", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-08-30T23:25:41.873 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_index_max_aio": "128", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_quota_ttl": "600", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_cache_enabled": "true", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_cache_expiry_interval": "900", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_cache_lru_size": "25000", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_config_store": "rados", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_content_length_compat": "false", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_copy_obj_progress": "true", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_cors_rules_max_num": "100", 2024-08-30T23:25:41.874 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_cross_domain_policy": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_default_encryption_key": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_addr": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_ca_path": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_client_cert": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_client_key": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_password": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_kmip_username": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_require_ssl": "true", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-08-30T23:25:41.875 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_suppress_logs": "true", 2024-08-30T23:25:41.876 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_addr": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_auth": "token", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_namespace": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_prefix": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_token_file": "", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_curl_buffersize": "524288", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_curl_low_speed_limit": "1024", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_curl_low_speed_time": "300", 2024-08-30T23:25:41.877 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_curl_tcp_keepalive": "0", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_l1_fadvise": "4", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-08-30T23:25:41.878 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_l1_fadvise": "4", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_log_changes_size": "1000", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_log_num_shards": "128", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_log_window": "30", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_notify_interval_msec": "0", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_sync_poll_interval": "20", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_data_sync_spawn_window": "20", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-08-30T23:25:41.879 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_default_data_log_backing": "fifo", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_default_region_info_oid": "default.region", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_defer_to_bucket_acls": "", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_disable_s3select": "false", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-08-30T23:25:41.880 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_data_res": "500.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dns_name": "", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dns_s3website_name": "", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dynamic_resharding": "true", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-08-30T23:25:41.881 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_gc_threads": "true", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_lc_threads": "true", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_mdsearch": "true", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_ops_log": "false", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_quota_threads": "true", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_static_website": "false", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enable_usage_log": "false", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_enforce_swift_acls": "true", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_exit_timeout_secs": "120", 2024-08-30T23:25:41.882 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_expose_bucket": "false", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_extended_http_attrs": "", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_filter": "none", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_frontends": "beast port=7480", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_max_concurrent_io": "10", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_max_deferred": "50", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_max_objs": "32", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_max_queue_size": "134213632", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_max_trim_chunk": "16", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_obj_min_wait": "7200", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_processor_max_time": "3600", 2024-08-30T23:25:41.883 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_gc_processor_period": "3600", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_get_obj_window_size": "16777216", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_healthcheck_disabling_path": "", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ignore_get_invalid_range": "false", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_init_timeout": "300", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_kafka_connection_idle": "300", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_kafka_message_timeout": "5000", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_kafka_sleep_timeout": "10", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-08-30T23:25:41.884 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_domain": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_password": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_password_path": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_project": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_tenant": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_token": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_token_path": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_admin_user": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_barbican_domain": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_barbican_password": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_barbican_project": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_barbican_tenant": "", 2024-08-30T23:25:41.885 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_barbican_user": "", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_implicit_tenants": "false", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_service_token_enabled": "false", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_token_cache_size": "10000", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_url": "", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_keystone_verify_ssl": "true", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_debug_interval": "-1", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_lock_max_time": "90", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_max_objs": "32", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_max_rules": "1000", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_max_worker": "3", 2024-08-30T23:25:41.886 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_max_wp_worker": "3", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lc_thread_delay": "0", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ldap_dnattr": "uid", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ldap_searchfilter": "", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ldap_uri": "", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lfuda_sync_frequency": "60", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_log_http_headers": "", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_log_nonexistent_bucket": "false", 2024-08-30T23:25:41.887 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_log_object_name_utc": "false", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_attr_name_len": "0", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_attr_size": "0", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_attrs_num_in_req": "0", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_chunk_size": "4194304", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_concurrent_requests": "1024", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_dynamic_shards": "1999", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_listing_results": "1000", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_notify_retries": "10", 2024-08-30T23:25:41.888 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_objs_per_shard": "100000", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_put_param_size": "1048576", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_put_size": "5368709120", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_max_slo_entries": "1000", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_md_log_max_shards": "64", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_md_notify_interval_msec": "200", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_meta_sync_poll_interval": "20", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_meta_sync_spawn_window": "20", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_mp_lock_max_time": "600", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_multipart_min_part_size": "5242880", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-08-30T23:25:41.889 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_fhcache_size": "2017", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_lru_lanes": "5", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_max_gc": "300", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_run_gc_threads": "false", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_run_lc_threads": "false", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_run_quota_threads": "false", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_run_sync_thread": "false", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_num_async_rados_threads": "32", 2024-08-30T23:25:41.890 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_num_control_oids": "8", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_numa_node": "-1", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_obj_stripe_size": "4194304", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_objexp_chunk_size": "100", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_objexp_gc_interval": "600", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_objexp_hints_num_shards": "127", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_op_thread_timeout": "600", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_op_tracing": "false", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_opa_token": "", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_opa_url": "", 2024-08-30T23:25:41.891 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_opa_verify_ssl": "true", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ops_log_rados": "false", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_ops_log_socket_path": "", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_period_push_interval": "2.000000", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_period_push_interval_max": "30.000000", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_period_root_pool": ".rgw.root", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-08-30T23:25:41.892 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_posix_cache_lanes": "3", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_posix_cache_max_buckets": "100", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_posix_cache_partitions": "3", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_print_continue": "true", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_print_prohibited_content_length": "false", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_rados_tracing": "false", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_realm": "", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_realm_id": "", 2024-08-30T23:25:41.893 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_region": "", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_region_root_pool": ".rgw.root", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_relaxed_region_enforcement": "false", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_request_uri": "", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_reshard_batch_size": "64", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_reshard_debug_interval": "-1", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_reshard_max_aio": "128", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_reshard_num_logs": "16", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_reshard_thread_interval": "600", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_resolve_cname": "false", 2024-08-30T23:25:41.894 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_rest_getusage_op_compat": "false", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_run_sync_thread": "true", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_auth_use_keystone": "false", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_auth_use_ldap": "false", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_auth_use_rados": "true", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_auth_use_sts": "false", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_s3_success_create_obj_status": "0", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_scheduler_type": "throttler", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_script_uri": "", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_service_provider_name": "", 2024-08-30T23:25:41.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_client_id": "", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_client_secret": "", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_entry": "sts", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_key": "sts", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_max_session_duration": "43200", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_min_session_duration": "900", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sts_token_introspection_url": "", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_account_in_url": "false", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_auth_entry": "auth", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_auth_url": "", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_custom_header": "", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_enforce_content_length": "false", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_need_stats": "true", 2024-08-30T23:25:41.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_tenant_name": "", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_token_expiration": "86400", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_url": "", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_url_prefix": "swift", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_swift_versioning_enabled": "false", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_lease_period": "120", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_log_trim_interval": "1200", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-08-30T23:25:41.897 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_obj_etag_verify": "false", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_trace_history_size": "4096", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_thread_pool_size": "512", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_topic_persistency_max_retries": "0", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_topic_require_publish_policy": "false", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_comment": "", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_createby": "", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_encoding": "", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_flag": "false", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_max_size": "5368709120", 2024-08-30T23:25:41.898 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_origin": "", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_sha_unit": "524288", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_torrent_tracker": "", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_trust_forwarded_https": "false", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_usage_log_tick_interval": "30", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_usage_max_shards": "32", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_usage_max_user_shards": "1", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_use_opa_authz": "false", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_counters_cache": "false", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_counters_cache_size": "10000", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_default_quota_max_size": "-1", 2024-08-30T23:25:41.899 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_max_buckets": "1000", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_policies_max_num": "100", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_quota_sync_interval": "86400", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_user_unique_email": "true", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_verify_ssl": "true", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_website_routing_rules_max_num": "50", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_zone": "", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_zone_id": "", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_zonegroup": "", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_zonegroup_id": "", 2024-08-30T23:25:41.900 INFO:teuthology.orchestra.run.smithi106.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rgwlc_auto_session_clear": "true", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rgwlc_skip_bucket_step": "false", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_block_size": "4096", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cache_shard_bits": "4", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cache_size": "536870912", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cache_type": "binned_lru", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-08-30T23:25:41.901 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_collect_compaction_stats": "false", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_collect_extended_stats": "false", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_collect_memory_stats": "false", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_index_type": "binary_search", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_log_to_ceph_log": "true", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_metadata_block_size": "4096", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_partition_filters": "false", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_perf": "false", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "rotating_keys_renewal_timeout": "10", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "run_dir": "/var/run/ceph", 2024-08-30T23:25:41.902 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_block_create": "true", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_cache_lru_size": "67108864", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_cbjournal_size": "5368709120", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_data_delta_based_overwrite": "0", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_default_max_object_size": "16777216", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_device_size": "53687091200", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_full_integrity_check": "false", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_journal_batch_capacity": "16", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_journal_iodepth_limit": "5", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_main_device_type": "SSD", 2024-08-30T23:25:41.903 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_max_concurrent_transactions": "8", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_max_data_allocation_size": "32768", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "seastore_segment_size": "67108864", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "set_keepcaps": "false", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "setgroup": "ceph", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "setuser": "ceph", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "setuser_match_path": "", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_misplaced_ratio": "0.050000", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "thp": "false", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "threadpool_default_timeout": "60", 2024-08-30T23:25:41.904 INFO:teuthology.orchestra.run.smithi106.stdout: "threadpool_empty_queue_max_wait": "2", 2024-08-30T23:25:41.905 INFO:teuthology.orchestra.run.smithi106.stdout: "throttler_perf_counter": "true", 2024-08-30T23:25:41.905 INFO:teuthology.orchestra.run.smithi106.stdout: "tmp_dir": "/tmp", 2024-08-30T23:25:41.905 INFO:teuthology.orchestra.run.smithi106.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-08-30T23:25:41.905 INFO:teuthology.orchestra.run.smithi106.stdout: "uadk_compressor_enabled": "false", 2024-08-30T23:25:41.905 INFO:teuthology.orchestra.run.smithi106.stdout: "uadk_wd_sync_ctx_num": "2" 2024-08-30T23:25:41.905 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:25:41.905 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:41 smithi173 ceph-mon[20554]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:41 smithi136 ceph-mon[20207]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:42.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-08-30T23:25:43.177 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:43.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:43 smithi106 ceph-mon[16603]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:43.671 INFO:teuthology.orchestra.run.smithi106.stdout:mon.a: {} 2024-08-30T23:25:43.671 INFO:teuthology.orchestra.run.smithi106.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T23:25:43.685 INFO:teuthology.orchestra.run.smithi106.stdout:mon.c: {} 2024-08-30T23:25:43.685 INFO:teuthology.orchestra.run.smithi106.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T23:25:43.698 INFO:teuthology.orchestra.run.smithi106.stdout:mon.b: {} 2024-08-30T23:25:43.699 INFO:teuthology.orchestra.run.smithi106.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-30T23:25:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:43 smithi136 ceph-mon[20207]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:43 smithi173 ceph-mon[20554]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:44.461 INFO:teuthology.run_tasks:Running task full_sequential... 2024-08-30T23:25:44.487 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:25:44.508 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:25:44.508 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-08-30T23:25:44.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-08-30T23:25:44.509 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-08-30T23:25:44.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:25:44.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-08-30T23:25:44.509 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-08-30T23:25:44.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:44.510 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-08-30T23:25:44.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:25:44.510 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-08-30T23:25:44.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:25:44.511 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-08-30T23:25:44.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:25:44.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:44.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:44.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:44.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:44.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1478430191' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:25:44.801 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:44.808 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:44.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:44.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:44.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1478430191' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:25:44.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1478430191' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-30T23:25:44.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:44.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:44.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:45.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:45.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:45.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:45.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:45.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:45.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:45.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:45.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:45.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:45 smithi106 ceph-mon[16603]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:45.499 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:25:45.499 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:25:45.499 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:25:45.500 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:25:45.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.570 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:45.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:45.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:45.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:45.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:45 smithi136 ceph-mon[20207]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:45.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:45.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:45.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:45 smithi173 ceph-mon[20554]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:45.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:45.938 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:45.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:45.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:45.950 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:25:45.950 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:25:45.974 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:25:45.974 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:25:45.994 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:25:45.994 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:25:46.014 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:25:46.014 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:25:46.035 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:25:46.036 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:25:46.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:46.057 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:25:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:25:46.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:46.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:46.078 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:25:46.078 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:25:46.100 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:25:46.100 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:25:46.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:25:46.121 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:25:46.142 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:25:46.143 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:25:46.164 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:25:46.164 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:25:46.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:46.185 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:25:46.186 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:25:46.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:46.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:25:46.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:25:46.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:25:46.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:46.486 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:46.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:46.601 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:46.612 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:46.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:46.722 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3420891018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:25:46.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3420891018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:25:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3420891018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:25:46.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:46.865 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:46.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:46.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:46.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:47.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:47.116 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:47.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:47.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:47.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:47.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:47.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:47.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:47.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:47.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:47 smithi106 ceph-mon[16603]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:47.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:47.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:47.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:47.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:47 smithi136 ceph-mon[20207]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:47 smithi173 ceph-mon[20554]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:47.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:47.871 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:47.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:25:47.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:47.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:25:48.010 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:48.011 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:48.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:48.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:48.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:48.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:48.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:48.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:48.429 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T23:25:48.430 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:25:48.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2621094072' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:25:48.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3618020165' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:25:48.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:48.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:48.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:48.628 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:48.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:48.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2621094072' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:25:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3618020165' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:25:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2621094072' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:25:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3618020165' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:25:48.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:48.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:48.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:48.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:49.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:49.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:49.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:49.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:49.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:49.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:25:49.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:49.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:49.437 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_0' created 2024-08-30T23:25:49.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:49.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2621094072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3618020165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: osdmap e96: 12 total, 12 up, 12 in 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/757466877' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-30T23:25:49.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:49 smithi106 ceph-mon[16603]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:49.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:49.643 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:49.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-08-30T23:25:49.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:49.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:25:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2621094072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:25:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3618020165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:25:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: osdmap e96: 12 total, 12 up, 12 in 2024-08-30T23:25:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/757466877' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-30T23:25:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:49 smithi136 ceph-mon[20207]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:25:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2621094072' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:25:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3618020165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:25:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: osdmap e96: 12 total, 12 up, 12 in 2024-08-30T23:25:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/757466877' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-30T23:25:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:49 smithi173 ceph-mon[20554]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:49.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:49.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:49.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:49.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:25:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:50.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:25:50.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:50.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:50.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:25:50.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:25:50.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:50.394 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:25:50.394 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:25:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:50.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:50.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:25:50.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:25:50.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/757466877' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-30T23:25:50.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:50 smithi106 ceph-mon[16603]: osdmap e97: 12 total, 12 up, 12 in 2024-08-30T23:25:50.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:25:50.661 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:25:50.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:50.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:50.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/757466877' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-30T23:25:50.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:50 smithi136 ceph-mon[20207]: osdmap e97: 12 total, 12 up, 12 in 2024-08-30T23:25:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/757466877' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-30T23:25:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:50 smithi173 ceph-mon[20554]: osdmap e97: 12 total, 12 up, 12 in 2024-08-30T23:25:50.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:50.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:50.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:51.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:51.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-08-30T23:25:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:51.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:51.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:25:51.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:51.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:51.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:51.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:51.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:51.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:51.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:51.551 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:51.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:25:51.582 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:51.612 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:51.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:51.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:51.711 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:51.711 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:51.711 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:51 smithi106 ceph-mon[16603]: osdmap e98: 12 total, 12 up, 12 in 2024-08-30T23:25:51.711 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:51 smithi106 ceph-mon[16603]: pgmap v298: 17 pgs: 1 creating+peering, 10 unknown, 6 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:51.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:51.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:51 smithi136 ceph-mon[20207]: osdmap e98: 12 total, 12 up, 12 in 2024-08-30T23:25:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:51 smithi136 ceph-mon[20207]: pgmap v298: 17 pgs: 1 creating+peering, 10 unknown, 6 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.805 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:51.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:51 smithi173 ceph-mon[20554]: osdmap e98: 12 total, 12 up, 12 in 2024-08-30T23:25:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:51 smithi173 ceph-mon[20554]: pgmap v298: 17 pgs: 1 creating+peering, 10 unknown, 6 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:51.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:25:51.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:51.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:25:52.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:52.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.061 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:52.122 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:52.124 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:52.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:52.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:52.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:52.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:52.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:52.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:52.481 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T23:25:52.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:52 smithi106 ceph-mon[16603]: osdmap e99: 12 total, 12 up, 12 in 2024-08-30T23:25:52.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:52 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:25:52.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2481051869' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:25:52.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:52.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:52.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:52.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:52.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:25:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:52 smithi136 ceph-mon[20207]: osdmap e99: 12 total, 12 up, 12 in 2024-08-30T23:25:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:52 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:25:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2481051869' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:25:52.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:25:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:52 smithi173 ceph-mon[20554]: osdmap e99: 12 total, 12 up, 12 in 2024-08-30T23:25:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:52 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:25:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2481051869' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:25:52.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:52.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:52.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:53.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:53.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:53.139 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:25:53.139 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:25:53.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:53.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:53.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:53.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:53.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:53.447 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:53.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:53.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:53.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:53.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:25:53.698 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:25:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2481051869' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:25:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:53 smithi136 ceph-mon[20207]: osdmap e100: 12 total, 12 up, 12 in 2024-08-30T23:25:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:53 smithi136 ceph-mon[20207]: pgmap v301: 17 pgs: 1 creating+peering, 2 unknown, 14 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:53.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:53.821 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2481051869' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:25:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:53 smithi173 ceph-mon[20554]: osdmap e100: 12 total, 12 up, 12 in 2024-08-30T23:25:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:53 smithi173 ceph-mon[20554]: pgmap v301: 17 pgs: 1 creating+peering, 2 unknown, 14 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:53.826 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2481051869' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:25:53.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:53 smithi106 ceph-mon[16603]: osdmap e100: 12 total, 12 up, 12 in 2024-08-30T23:25:53.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:53 smithi106 ceph-mon[16603]: pgmap v301: 17 pgs: 1 creating+peering, 2 unknown, 14 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:53.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:25:53.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:53.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:53.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:54.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:54.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:54.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:54.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:54.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:54.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:54.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:54.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:54.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:54.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:54.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:54.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:54.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:54 smithi136 ceph-mon[20207]: osdmap e101: 12 total, 12 up, 12 in 2024-08-30T23:25:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1603779054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:25:54.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:54 smithi173 ceph-mon[20554]: osdmap e101: 12 total, 12 up, 12 in 2024-08-30T23:25:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1603779054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:25:54.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:54 smithi106 ceph-mon[16603]: osdmap e101: 12 total, 12 up, 12 in 2024-08-30T23:25:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1603779054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:25:54.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-08-30T23:25:54.932 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:54.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:54.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:54.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:55.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:55.118 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:55.209 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:25:55.209 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:25:55.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:55.230 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:25:55.230 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:25:55.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:55.253 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:25:55.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:25:55.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:25:55.274 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:25:55.295 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:25:55.296 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:25:55.317 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:25:55.317 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:25:55.340 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:25:55.340 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:25:55.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:25:55.361 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:25:55.362 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:25:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:55.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:25:55.383 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:25:55.383 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:25:55.404 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:25:55.404 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:25:55.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:25:55.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:25:55.452 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:55.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:55.487 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:55.498 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:25:55.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:55.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:55.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:55.649 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:25:55.649 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:25:55.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3142649869' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:25:55.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:55 smithi106 ceph-mon[16603]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:55.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:55.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3142649869' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:25:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:55 smithi136 ceph-mon[20207]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3142649869' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:25:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:55 smithi173 ceph-mon[20554]: pgmap v303: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:55.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:55.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:55.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:55.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:56.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:56.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:56.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:56.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.266 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:56.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:56.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:56.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:56.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:56.499 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-08-30T23:25:56.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:56 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:25:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3142649869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:25:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:56 smithi106 ceph-mon[16603]: osdmap e102: 12 total, 12 up, 12 in 2024-08-30T23:25:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2704778794' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:25:56.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:56.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:56.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:56.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:56 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:25:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3142649869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:25:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:56 smithi136 ceph-mon[20207]: osdmap e102: 12 total, 12 up, 12 in 2024-08-30T23:25:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2704778794' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:25:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:56 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:25:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3142649869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:25:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:56 smithi173 ceph-mon[20554]: osdmap e102: 12 total, 12 up, 12 in 2024-08-30T23:25:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2704778794' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:25:56.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:56.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:56.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:57.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:57.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:57.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:57.260 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:25:57.260 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:25:57.260 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:25:57.260 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:25:57.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:25:57.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:25:57.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:57.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:57.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:57.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:57.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:57.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:57.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:57.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2704778794' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:25:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:57 smithi136 ceph-mon[20207]: osdmap e103: 12 total, 12 up, 12 in 2024-08-30T23:25:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:57 smithi136 ceph-mon[20207]: pgmap v306: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2704778794' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:25:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:57 smithi173 ceph-mon[20554]: osdmap e103: 12 total, 12 up, 12 in 2024-08-30T23:25:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:57 smithi173 ceph-mon[20554]: pgmap v306: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2704778794' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:25:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:57 smithi106 ceph-mon[16603]: osdmap e103: 12 total, 12 up, 12 in 2024-08-30T23:25:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:57 smithi106 ceph-mon[16603]: pgmap v306: 17 pgs: 17 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:25:57.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:57.915 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:57.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:58.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:25:58.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:25:58.069 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-08-30T23:25:58.087 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:25:58.087 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:25:58.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:58.161 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:25:58.161 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:25:58.161 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_47195 2024-08-30T23:25:58.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:25:58.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:25:58.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:58.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:25:58.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:25:58.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:25:58.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:25:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:58.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:25:58.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:25:58.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:58.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:58.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:58.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:25:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:58.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:25:58.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:58 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:25:58.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:58 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:25:58.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:58 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:25:58.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:58 smithi106 ceph-mon[16603]: osdmap e104: 12 total, 12 up, 12 in 2024-08-30T23:25:58.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:58 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:25:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:58 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:25:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:58 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:25:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:58 smithi136 ceph-mon[20207]: osdmap e104: 12 total, 12 up, 12 in 2024-08-30T23:25:58.810 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:58 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:25:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:58 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:25:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:58 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:25:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:58 smithi173 ceph-mon[20554]: osdmap e104: 12 total, 12 up, 12 in 2024-08-30T23:25:58.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:25:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:58.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:58.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:25:58.987 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:58.987 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:25:59.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:59.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:59.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1666 1650 103.119 103.125 0.00514065 0.00956967 2024-08-30T23:25:59.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:59.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:25:59.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:59.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:59.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:25:59.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:25:59.510 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T23:25:59.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:25:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:59.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:25:59.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:25:59.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:25:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:25:59.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/634233215' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:25:59.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:25:59 smithi106 ceph-mon[16603]: pgmap v308: 17 pgs: 17 active+clean; 12 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s wr, 31 op/s 2024-08-30T23:25:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/634233215' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:25:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:25:59 smithi136 ceph-mon[20207]: pgmap v308: 17 pgs: 17 active+clean; 12 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s wr, 31 op/s 2024-08-30T23:25:59.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:25:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/634233215' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:25:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:25:59 smithi173 ceph-mon[20554]: pgmap v308: 17 pgs: 17 active+clean; 12 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s wr, 31 op/s 2024-08-30T23:25:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:25:59.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:25:59.842 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:25:59.842 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:25:59.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:25:59.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:25:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:25:59.950 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:25:59.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:00.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.074 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:00.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:00.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3042 3026 94.5552 86 0.0343121 0.0104445 2024-08-30T23:26:00.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:00.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:00.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:00.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:00.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:00.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:00.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:00.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:00.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:00.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/634233215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:26:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:00 smithi136 ceph-mon[20207]: osdmap e105: 12 total, 12 up, 12 in 2024-08-30T23:26:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/57693645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:00 smithi136 ceph-mon[20207]: osdmap e106: 12 total, 12 up, 12 in 2024-08-30T23:26:00.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/634233215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:26:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:00 smithi173 ceph-mon[20554]: osdmap e105: 12 total, 12 up, 12 in 2024-08-30T23:26:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/57693645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:00 smithi173 ceph-mon[20554]: osdmap e106: 12 total, 12 up, 12 in 2024-08-30T23:26:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.825 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:00.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/634233215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:26:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:00 smithi106 ceph-mon[16603]: osdmap e105: 12 total, 12 up, 12 in 2024-08-30T23:26:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/57693645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:00 smithi106 ceph-mon[16603]: osdmap e106: 12 total, 12 up, 12 in 2024-08-30T23:26:00.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-08-30T23:26:00.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:00.949 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:00.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:01.039 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:01.039 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:01.039 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:01.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:01.041 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:01.041 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:01.041 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:01.041 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:01.041 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:01.042 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:01.042 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:01.042 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:01.042 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:01.043 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:01.043 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:01.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:01.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:01.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 4000 3984 82.9932 59.875 0.00110018 0.0119685 2024-08-30T23:26:01.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:01.200 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:01.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:01.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:01.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:01.454 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:01.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.578 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:01.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:01.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:01 smithi106 ceph-mon[16603]: pgmap v311: 17 pgs: 17 active+clean; 93 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 307 op/s 2024-08-30T23:26:01.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:01.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:01 smithi136 ceph-mon[20207]: pgmap v311: 17 pgs: 17 active+clean; 93 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 307 op/s 2024-08-30T23:26:01.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:01 smithi173 ceph-mon[20554]: pgmap v311: 17 pgs: 17 active+clean; 93 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 307 op/s 2024-08-30T23:26:01.832 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:01.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:01.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:01.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:02.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:02.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:02.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:26:02.109 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:02.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 5196 5180 80.9308 74.75 0.0141428 0.0122034 2024-08-30T23:26:02.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:02.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:02.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:02.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:02.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:02.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:02.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:02.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:02.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:02.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:02 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:02.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:02 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:02 smithi136 ceph-mon[20207]: osdmap e107: 12 total, 12 up, 12 in 2024-08-30T23:26:02.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:02 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:02 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:02 smithi173 ceph-mon[20554]: osdmap e107: 12 total, 12 up, 12 in 2024-08-30T23:26:02.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:02 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:02 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:02 smithi106 ceph-mon[16603]: osdmap e107: 12 total, 12 up, 12 in 2024-08-30T23:26:02.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:02.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:02.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:03.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:03.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:03.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 6324 6308 78.8422 70.5 0.00153751 0.0125602 2024-08-30T23:26:03.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:03.211 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:03.222 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:03.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:03.265 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:03.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:03.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:03.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:03.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:03.524 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:26:03.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:03.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:03 smithi106 ceph-mon[16603]: pgmap v313: 17 pgs: 17 active+clean; 264 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 724 op/s 2024-08-30T23:26:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/307142456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:03.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:26:03.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:03.740 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:03 smithi136 ceph-mon[20207]: pgmap v313: 17 pgs: 17 active+clean; 264 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 724 op/s 2024-08-30T23:26:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/307142456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:03 smithi173 ceph-mon[20554]: pgmap v313: 17 pgs: 17 active+clean; 264 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 724 op/s 2024-08-30T23:26:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/307142456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:03.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:03.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:03.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:03.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:03.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:04.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:04.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:04.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 8111 8095 84.3147 111.688 0.00394511 0.0118404 2024-08-30T23:26:04.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:04.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:04.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:04.363 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:04.373 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:04.373 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:04.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:04.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:04.529 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T23:26:04.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:04.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:04.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:04.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/307142456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:26:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: osdmap e108: 12 total, 12 up, 12 in 2024-08-30T23:26:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3404669592' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:26:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:26:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:04 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:26:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/307142456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:26:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: osdmap e108: 12 total, 12 up, 12 in 2024-08-30T23:26:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3404669592' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:26:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:26:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:04 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:26:04.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/307142456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: osdmap e108: 12 total, 12 up, 12 in 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3404669592' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:26:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:04 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:26:04.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.863 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:04.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:04.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:04.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:05.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:05.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:05.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 9113 9097 81.2156 62.625 0.00254977 0.0122136 2024-08-30T23:26:05.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:05.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:05.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:05.369 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:05.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:05.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:05.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:05.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:05.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-08-30T23:26:05.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:05.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3404669592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:26:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:05 smithi136 ceph-mon[20207]: osdmap e109: 12 total, 12 up, 12 in 2024-08-30T23:26:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:05 smithi136 ceph-mon[20207]: pgmap v316: 17 pgs: 17 active+clean; 340 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-30T23:26:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3404669592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:26:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:05 smithi173 ceph-mon[20554]: osdmap e109: 12 total, 12 up, 12 in 2024-08-30T23:26:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:05 smithi173 ceph-mon[20554]: pgmap v316: 17 pgs: 17 active+clean; 340 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-30T23:26:05.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3404669592' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:26:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:05 smithi106 ceph-mon[16603]: osdmap e109: 12 total, 12 up, 12 in 2024-08-30T23:26:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:05 smithi106 ceph-mon[16603]: pgmap v316: 17 pgs: 17 active+clean; 340 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-30T23:26:05.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:05.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:05.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:05.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:06.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:06.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:06.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:26:06.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:06.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 10311 10295 80.4223 74.875 0.00150808 0.0124192 2024-08-30T23:26:06.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:06.254 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:06.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:06.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:06.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:06.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:06.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:06.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:06.730 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:06.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:06.749 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:06.763 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:06.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:06.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:06.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:06.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:07.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:07.035 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:07.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:07.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:07.162 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 11866 11850 82.2842 97.1875 0.00243705 0.0121377 2024-08-30T23:26:07.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:07.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:07.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:07.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:07.429 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:07 smithi106 ceph-mon[16603]: pgmap v317: 17 pgs: 17 active+clean; 541 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-30T23:26:07.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:07.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.540 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:07.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:07.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:07.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:07 smithi136 ceph-mon[20207]: pgmap v317: 17 pgs: 17 active+clean; 541 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-30T23:26:07.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:07.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:07 smithi173 ceph-mon[20554]: pgmap v317: 17 pgs: 17 active+clean; 541 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-30T23:26:07.879 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:26:07.879 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:26:07.880 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:26:07.880 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:26:07.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:07.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:07.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:08.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:08.046 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:08.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:08.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 13323 13307 83.1613 91.0625 0.00625658 0.0120066 2024-08-30T23:26:08.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:08.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:08.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:08.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:08.422 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:08.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:08.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:08.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:26:08.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:08.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:26:08.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:08.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4252919197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:08.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:08.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:08.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:08.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:08.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:08.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:08.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:08.737 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:08.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4252919197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:08.806 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4252919197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:08.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:08.930 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:08.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:09.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:09.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:09.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:09.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 14432 14416 81.9018 69.3125 0.00435141 0.0121979 2024-08-30T23:26:09.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:09.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:09.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:09.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:09.385 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:09.387 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:09.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:09.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:09.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:09.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:09.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:09.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:09.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:09 smithi106 ceph-mon[16603]: pgmap v318: 17 pgs: 17 active+clean; 721 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-08-30T23:26:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:09 smithi136 ceph-mon[20207]: pgmap v318: 17 pgs: 17 active+clean; 721 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-08-30T23:26:09.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:09 smithi173 ceph-mon[20554]: pgmap v318: 17 pgs: 17 active+clean; 721 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-08-30T23:26:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:09.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:09.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:09.952 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:09.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:10.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:10.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:10.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 16199 16183 84.2791 110.438 0.00270466 0.0118201 2024-08-30T23:26:10.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:10.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:10.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:10.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:10.418 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T23:26:10.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:10.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:10.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:10.587 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1560893629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:26:10.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:10.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:10.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:10.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:10.764 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:26:10.764 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:26:10.764 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:26:10.764 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:26:10.764 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:26:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:10.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1560893629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:26:10.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1560893629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:26:10.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:10.888 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:10.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:10.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:11.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:11.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:11.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:11.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 17235 17219 82.7765 64.75 0.195673 0.0120574 2024-08-30T23:26:11.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:11.262 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:11.364 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:11.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.389 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:11.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:11.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:26:11.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:11.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:11.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:11.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:11.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1560893629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:26:11.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:11 smithi106 ceph-mon[16603]: osdmap e110: 12 total, 12 up, 12 in 2024-08-30T23:26:11.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:11 smithi106 ceph-mon[16603]: osdmap e111: 12 total, 12 up, 12 in 2024-08-30T23:26:11.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:11 smithi106 ceph-mon[16603]: pgmap v321: 17 pgs: 17 active+clean; 888 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-08-30T23:26:11.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:11.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1560893629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:26:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:11 smithi136 ceph-mon[20207]: osdmap e110: 12 total, 12 up, 12 in 2024-08-30T23:26:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:11 smithi136 ceph-mon[20207]: osdmap e111: 12 total, 12 up, 12 in 2024-08-30T23:26:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:11 smithi136 ceph-mon[20207]: pgmap v321: 17 pgs: 17 active+clean; 888 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-08-30T23:26:11.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:26:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1560893629' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:26:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:11 smithi173 ceph-mon[20554]: osdmap e110: 12 total, 12 up, 12 in 2024-08-30T23:26:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:11 smithi173 ceph-mon[20554]: osdmap e111: 12 total, 12 up, 12 in 2024-08-30T23:26:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:11 smithi173 ceph-mon[20554]: pgmap v321: 17 pgs: 17 active+clean; 888 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-08-30T23:26:11.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:11.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:11.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:11.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:12.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:12.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:12.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:12.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:12.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 18603 18587 82.9705 85.5 0.0038434 0.0120333 2024-08-30T23:26:12.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-08-30T23:26:12.212 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:12.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:12.260 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:12.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:12.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:12.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:12.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:12.499 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:26:12.525 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:12.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:12.649 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:12.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:12.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2911129186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2911129186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2911129186' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:12.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:12.908 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:12.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:13.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:13.019 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.14157.0:80 1.0 e23/20 1:30306672:devicehealth::main.db.0000000000000000:head v 23'58, mlcod=23'53)", 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 299.309227588, 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94186119800000001, 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.020 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5799999999999999e-07 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708893+0000", 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1081999999999998e-05 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650093+0000", 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94119932299999998 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650101+0000", 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7900000000000005e-06 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650169+0000", 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650197+0000", 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.72e-05 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650681+0000", 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00048408700000000002 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650723+0000", 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:26:13.024 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:842 2.7 e104/97 2:ebe581cf:::benchmark_data_smithi173_47195_object841:head v 104'46, mlcod=104'43)", 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 14.438566368, 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15218538700000001, 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.025 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5400000000000002e-07 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579637+0000", 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011383900000000001 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731295+0000", 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15165820299999999 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731300+0000", 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.023e-06 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731346+0000", 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5766999999999997e-05 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731360+0000", 2024-08-30T23:26:13.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3959e-05 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731680+0000", 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032010099999999999 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731708+0000", 2024-08-30T23:26:13.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8041000000000001e-05 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:845 2.1 2:8578137a:::benchmark_data_smithi173_47195_object844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.582722+0000", 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 14.435366871999999, 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15184962699999999, 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 845 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582722+0000", 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582722+0000", 2024-08-30T23:26:13.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582724+0000", 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4449999999999999e-06 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582767+0000", 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3519000000000001e-05 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582778+0000", 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1212e-05 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582784+0000", 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1789999999999999e-06 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582838+0000", 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4012999999999999e-05 2024-08-30T23:26:13.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582981+0000", 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000143783 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583134+0000", 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015213499999999999 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583522+0000", 2024-08-30T23:26:13.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038870099999999998 2024-08-30T23:26:13.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588769+0000", 2024-08-30T23:26:13.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052468990000000002 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734517+0000", 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.145747561 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734549+0000", 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2187000000000002e-05 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734572+0000", 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2993000000000002e-05 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:845 2.1 e104/97 ondisk, result = 0)", 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 14.434281291, 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15076826500000001, 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7e-07 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583811+0000", 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4650000000000001e-06 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734462+0000", 2024-08-30T23:26:13.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.150650906 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734470+0000", 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6100000000000006e-06 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734509+0000", 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8634999999999999e-05 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734514+0000", 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6070000000000001e-06 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734576+0000", 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2571999999999997e-05 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.042 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:849 2.e 2:7e7dd803:::benchmark_data_smithi173_47195_object848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.587830+0000", 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 14.430258905000001, 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14671935799999999, 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.043 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 849 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587830+0000", 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587830+0000", 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587832+0000", 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.319e-06 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587879+0000", 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7054e-05 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587890+0000", 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1506e-05 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587895+0000", 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1379999999999998e-06 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587941+0000", 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5983999999999999e-05 2024-08-30T23:26:13.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588022+0000", 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1144999999999996e-05 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588105+0000", 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2663999999999993e-05 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588517+0000", 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00041148399999999999 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588785+0000", 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026829000000000002 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734469+0000", 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14568394900000001 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734516+0000", 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7409e-05 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734550+0000", 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3416e-05 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.049 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:849 2.e e104/97 ondisk, result = 0)", 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.588669+0000", 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 14.429420179999999, 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.145886188, 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588669+0000", 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588669+0000", 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588670+0000", 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0999999999999998e-07 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588673+0000", 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3110000000000001e-06 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734403+0000", 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14573025000000001 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734409+0000", 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7270000000000004e-06 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734457+0000", 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8390999999999998e-05 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734464+0000", 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.815e-06 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734555+0000", 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1083999999999997e-05 2024-08-30T23:26:13.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14244 2.3 e109/97 2:c35428fa:::benchmark_data_smithi173_47195_object14243:head v 109'957, mlcod=109'954)", 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 4.1343085459999998, 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213971894, 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883782+0000", 2024-08-30T23:26:13.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999999e-07 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883844+0000", 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2580000000000001e-05 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096781+0000", 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212936878 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096785+0000", 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0389999999999998e-06 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096831+0000", 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6261999999999999e-05 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096844+0000", 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2883e-05 2024-08-30T23:26:13.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097712+0000", 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086761599999999998 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097753+0000", 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0729999999999998e-05 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14266 2.e 2:74094085:::benchmark_data_smithi173_47195_object14265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.947539+0000", 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 4.0705502300000003, 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14818767099999999, 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14266 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.059 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947539+0000", 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947539+0000", 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947540+0000", 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669999999999999e-06 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947586+0000", 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5439000000000003e-05 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947598+0000", 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041999999999999e-05 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947603+0000", 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5740000000000003e-06 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947644+0000", 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0979999999999997e-05 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947729+0000", 2024-08-30T23:26:13.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4864999999999995e-05 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947811+0000", 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1884000000000004e-05 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948332+0000", 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052075799999999996 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095442+0000", 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.147109929 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095602+0000", 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016049599999999999 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095659+0000", 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6586999999999999e-05 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095727+0000", 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7849999999999996e-05 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:14266 2.e e109/97 ondisk, result = 0)", 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.948563+0000", 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 4.0695262019999996, 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.146887187, 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.065 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948563+0000", 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948563+0000", 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948564+0000", 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948568+0000", 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2089999999999999e-06 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095378+0000", 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14680979399999999 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095385+0000", 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-06 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095428+0000", 2024-08-30T23:26:13.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3037000000000001e-05 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095436+0000", 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0989999999999999e-06 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095450+0000", 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4430000000000001e-05 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.069 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16208 2.1 2:8d8266cc:::benchmark_data_smithi173_47195_object16207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.197949+0000", 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.8201403689999998, 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12480836100000001, 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.070 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16208 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.197949+0000", 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.197949+0000", 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.197950+0000", 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.197972+0000", 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2132000000000001e-05 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.197983+0000", 2024-08-30T23:26:13.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.115e-05 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.197988+0000", 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4079999999999997e-06 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198034+0000", 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5840999999999997e-05 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198111+0000", 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6684e-05 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198196+0000", 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4919000000000005e-05 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198914+0000", 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071819500000000003 2024-08-30T23:26:13.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.315301+0000", 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.116386697 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322661+0000", 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0073602670000000002 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322723+0000", 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1909999999999995e-05 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322757+0000", 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4377e-05 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:16208 2.1 e109/97 ondisk, result = 0)", 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.199166+0000", 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.8189233109999998, 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123596834, 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.199166+0000", 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.199166+0000", 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.199167+0000", 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9400000000000005e-07 2024-08-30T23:26:13.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.199170+0000", 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.703e-06 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322568+0000", 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123397648 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322574+0000", 2024-08-30T23:26:13.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8189999999999997e-06 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322638+0000", 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4393999999999994e-05 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322646+0000", 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5240000000000003e-06 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.322763+0000", 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117052 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16941 2.e 2:74b71233:::benchmark_data_smithi173_47195_object16940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.690508+0000", 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.3275810699999999, 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16175384500000001, 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.081 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:13.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16941 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690508+0000", 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690508+0000", 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690509+0000", 2024-08-30T23:26:13.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2699999999999999e-07 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690594+0000", 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5748000000000006e-05 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690603+0000", 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0129999999999999e-06 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690610+0000", 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7309999999999996e-06 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690652+0000", 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1508000000000003e-05 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690725+0000", 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2838999999999998e-05 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690805+0000", 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0353999999999997e-05 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691361+0000", 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00055562600000000004 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691657+0000", 2024-08-30T23:26:13.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029633900000000001 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852197+0000", 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16054035899999999 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852231+0000", 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3791000000000002e-05 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852262+0000", 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0910000000000001e-05 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:16941 2.e e111/97 ondisk, result = 0)", 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.3265056159999999, 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16068374799999999, 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.089 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0999999999999998e-07 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691587+0000", 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0079999999999998e-06 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852149+0000", 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16056198199999999 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852154+0000", 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4329999999999997e-06 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852188+0000", 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4730000000000001e-05 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852194+0000", 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6400000000000002e-06 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852267+0000", 2024-08-30T23:26:13.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3344999999999997e-05 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:17190 2.e 2:7818eb93:::benchmark_data_smithi173_47195_object17189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.986732+0000", 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.0313569710000001, 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.125581254, 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.094 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 17190 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986732+0000", 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986732+0000", 2024-08-30T23:26:13.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986734+0000", 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986852+0000", 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011817800000000001 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986877+0000", 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4593e-05 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986886+0000", 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0569999999999999e-06 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.986952+0000", 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6550999999999994e-05 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987081+0000", 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012886199999999999 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987212+0000", 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013061 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987999+0000", 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078744500000000001 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112194+0000", 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.124194478 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112254+0000", 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9981000000000001e-05 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112287+0000", 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2768999999999997e-05 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112313+0000", 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6957999999999998e-05 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:17190 2.e e111/97 ondisk, result = 0)", 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.988209+0000", 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.029880387, 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.124109, 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988209+0000", 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988209+0000", 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988210+0000", 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5300000000000002e-07 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988213+0000", 2024-08-30T23:26:13.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5729999999999999e-06 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112205+0000", 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123991187 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112211+0000", 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.584e-06 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112244+0000", 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2635000000000002e-05 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112250+0000", 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8030000000000002e-06 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112318+0000", 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8264999999999998e-05 2024-08-30T23:26:13.104 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:17193 2.1 2:840d9093:::benchmark_data_smithi173_47195_object17192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.988610+0000", 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.0294796430000002, 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12518699999999999, 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 17193 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988610+0000", 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988610+0000", 2024-08-30T23:26:13.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988610+0000", 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9699999999999995e-07 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988677+0000", 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6636000000000004e-05 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988689+0000", 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1987000000000001e-05 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988696+0000", 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9550000000000003e-06 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988739+0000", 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2815999999999998e-05 2024-08-30T23:26:13.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988818+0000", 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9398000000000001e-05 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.988913+0000", 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5100999999999996e-05 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989526+0000", 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000612439 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989869+0000", 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000343593 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113739+0000", 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123869304 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113770+0000", 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1545999999999999e-05 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113797+0000", 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6427999999999998e-05 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:18967 2.1 2:85fbebba:::benchmark_data_smithi173_47195_object18966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:13.111 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:12.391277+0000", 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.62681221499999995, 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.154535755, 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 18967 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391277+0000", 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391277+0000", 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391278+0000", 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1399999999999996e-07 2024-08-30T23:26:13.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391333+0000", 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5627999999999997e-05 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391342+0000", 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4950000000000008e-06 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391346+0000", 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4070000000000003e-06 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391381+0000", 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4897999999999997e-05 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391461+0000", 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9752999999999993e-05 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391552+0000", 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1025000000000003e-05 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.392171+0000", 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00061906500000000002 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.392714+0000", 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054273799999999997 2024-08-30T23:26:13.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545745+0000", 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15303161300000001 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545784+0000", 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9014999999999998e-05 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545813+0000", 2024-08-30T23:26:13.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8404000000000001e-05 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:18967 2.1 e111/97 ondisk, result = 0)", 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:12.449695+0000", 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.56839408499999999, 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.096123431999999995, 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.118 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.449695+0000", 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.449695+0000", 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.449697+0000", 2024-08-30T23:26:13.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7230000000000001e-06 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.449703+0000", 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4409999999999998e-06 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545685+0000", 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.095981503999999995 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545689+0000", 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.605e-06 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545732+0000", 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2623000000000002e-05 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545739+0000", 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.481e-06 2024-08-30T23:26:13.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545819+0000", 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9054999999999994e-05 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:19001 2.e 2:71917253:::benchmark_data_smithi173_47195_object19000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:12.451614+0000", 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.56647560100000005, 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13990867300000001, 2024-08-30T23:26:13.122 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 19001 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451614+0000", 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451614+0000", 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451616+0000", 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9300000000000002e-06 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451657+0000", 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1888000000000003e-05 2024-08-30T23:26:13.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451669+0000", 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1647e-05 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451675+0000", 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5709999999999997e-06 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451723+0000", 2024-08-30T23:26:13.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8807999999999998e-05 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451811+0000", 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7999999999999998e-05 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.451917+0000", 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105956 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.452682+0000", 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00076458400000000001 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.453190+0000", 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00050815699999999997 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591454+0000", 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.138263845 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591495+0000", 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0561e-05 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591522+0000", 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7725999999999998e-05 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:19001 2.e e111/97 ondisk, result = 0)", 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:12.452898+0000", 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.56519087300000004, 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13862993200000001, 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.452898+0000", 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.452898+0000", 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.452899+0000", 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7499999999999999e-07 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.452903+0000", 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3889999999999998e-06 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591409+0000", 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13850517000000001 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591412+0000", 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.039e-06 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591445+0000", 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3005e-05 2024-08-30T23:26:13.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591450+0000", 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1719999999999998e-06 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.591528+0000", 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8381999999999996e-05 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:13.132 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:13.133 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:26:13.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 19996 19980 83.2429 87.0625 0.003516 0.012002 2024-08-30T23:26:13.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:13.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:13.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:13.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:13.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:13.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:13.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:13.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:13.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:13.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:26:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:26:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2911129186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: osdmap e112: 12 total, 12 up, 12 in 2024-08-30T23:26:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:13 smithi136 ceph-mon[20207]: pgmap v323: 17 pgs: 17 active+clean; 1.1 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-08-30T23:26:13.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:26:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:26:13.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2911129186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:13.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: osdmap e112: 12 total, 12 up, 12 in 2024-08-30T23:26:13.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:13 smithi173 ceph-mon[20554]: pgmap v323: 17 pgs: 17 active+clean; 1.1 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-08-30T23:26:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2911129186' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: osdmap e112: 12 total, 12 up, 12 in 2024-08-30T23:26:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:13 smithi106 ceph-mon[16603]: pgmap v323: 17 pgs: 17 active+clean; 1.1 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-08-30T23:26:13.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:13.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:13.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:14.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:14.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:14.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 21566 21550 84.1725 98.125 0.00641382 0.0118714 2024-08-30T23:26:14.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:14.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:14.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:14.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:14.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:14.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:14.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:14.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:14.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:14.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:14.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:26:14.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:14.851 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:14.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:14.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:14.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:15.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:15.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:15.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 22830 22814 83.8678 79 0.001739 0.0118983 2024-08-30T23:26:15.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:15.225 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:15.235 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:15.236 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:15.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:15.258 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:15.272 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:15.282 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:15.282 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:15.306 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:15.306 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:15.311 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:15.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:15.332 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:15.332 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:15.351 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:15.368 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:15.368 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:15.393 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:15.393 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:15.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:15.418 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:15.440 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:15.440 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:15.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:15.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:15.460 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:15.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:15.482 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:15.482 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:15.511 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:15.511 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:15.512 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3466062983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:15.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3466062983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:15.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3466062983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:15.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:15.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:15.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:15.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:15.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:15.853 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:15.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:15.984 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:15.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:16.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:16.113 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:16.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 23948 23932 83.0902 69.875 0.00402403 0.0120272 2024-08-30T23:26:16.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:16.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:16.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:26:16.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:16.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:16.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:16.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:16.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:16 smithi136 ceph-mon[20207]: pgmap v324: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-30T23:26:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:16 smithi136 ceph-mon[20207]: osdmap e113: 12 total, 12 up, 12 in 2024-08-30T23:26:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:16 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:16 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:16.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:16 smithi173 ceph-mon[20554]: pgmap v324: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-30T23:26:16.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:16 smithi173 ceph-mon[20554]: osdmap e113: 12 total, 12 up, 12 in 2024-08-30T23:26:16.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:16 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:16.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:16 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:16.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:16 smithi106 ceph-mon[16603]: pgmap v324: 17 pgs: 17 active+clean; 1.2 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-30T23:26:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:16 smithi106 ceph-mon[16603]: osdmap e113: 12 total, 12 up, 12 in 2024-08-30T23:26:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:16 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:16 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:16.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:16.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:16.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:16.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:16.882 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:16.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:16.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:16.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:17.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:17.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:17.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:17.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 25402 25386 83.4995 90.875 0.00465375 0.0119697 2024-08-30T23:26:17.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:17.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:17.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:17.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:17.468 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:17.468 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:17.488 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T23:26:17.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:17.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:17.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:17.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:17.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:17.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:17 smithi136 ceph-mon[20207]: osdmap e114: 12 total, 12 up, 12 in 2024-08-30T23:26:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/573445427' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:26:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:17 smithi136 ceph-mon[20207]: pgmap v327: 17 pgs: 17 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-30T23:26:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:17 smithi173 ceph-mon[20554]: osdmap e114: 12 total, 12 up, 12 in 2024-08-30T23:26:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/573445427' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:26:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:17 smithi173 ceph-mon[20554]: pgmap v327: 17 pgs: 17 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-30T23:26:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:17 smithi106 ceph-mon[16603]: osdmap e114: 12 total, 12 up, 12 in 2024-08-30T23:26:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/573445427' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:26:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:17 smithi106 ceph-mon[16603]: pgmap v327: 17 pgs: 17 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-30T23:26:17.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:17.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:17.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:17.926 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:17.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:18.014 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:18.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:18.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:18.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:26:18.164840+0000 min lat: 0.00101743 max lat: 0.271224 avg lat: 0.0120365 2024-08-30T23:26:18.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:26:18.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 26587 26571 83.0274 74.0625 0.00973404 0.0120365 2024-08-30T23:26:18.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:18.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:18.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:18.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:18.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:18.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/573445427' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:26:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:18 smithi106 ceph-mon[16603]: osdmap e115: 12 total, 12 up, 12 in 2024-08-30T23:26:18.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:18.642 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:18.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:18.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:18.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/573445427' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:26:18.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:18 smithi136 ceph-mon[20207]: osdmap e115: 12 total, 12 up, 12 in 2024-08-30T23:26:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/573445427' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:26:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:18 smithi173 ceph-mon[20554]: osdmap e115: 12 total, 12 up, 12 in 2024-08-30T23:26:18.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:26:18.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:18.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:18.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:18.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:26:19.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:19.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:19.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:19.156 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:19.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 28131 28115 83.6685 96.5 0.0381296 0.0119382 2024-08-30T23:26:19.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:19.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:19.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:19.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:19.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:19.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:19.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-08-30T23:26:19.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:19.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.692 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:19.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:19 smithi106 ceph-mon[16603]: pgmap v329: 17 pgs: 17 active+clean; 1.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:26:19.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:26:19.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:26:19.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:26:19.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:26:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:19 smithi136 ceph-mon[20207]: pgmap v329: 17 pgs: 17 active+clean; 1.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:26:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:26:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:26:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:26:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:26:19.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:19.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:19 smithi173 ceph-mon[20554]: pgmap v329: 17 pgs: 17 active+clean; 1.5 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:26:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:26:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:26:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:26:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:26:19.851 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:19.873 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:19.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:19.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:19.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:20.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:20.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:20.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 29492 29476 83.7317 85.0625 0.019572 0.0119288 2024-08-30T23:26:20.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:20.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:20.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:20.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:20.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:20.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:20.509 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:26:20.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:20.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:20.677 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:20.677 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:20.677 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:20.677 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:20.678 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:20.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:20.699 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:20.699 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:20.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:20.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:20.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:20.761 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:20.761 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2894511742' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:20.762 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2847410833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:26:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:26:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:26:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2894511742' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2847410833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:20.791 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:20.792 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:20.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:26:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:26:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:26:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2894511742' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2847410833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:20.824 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:20.825 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:20.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:20.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:20.853 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:20.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:20.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:20.967 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:21.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:21.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:21.163 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 31175 31159 84.6642 105.188 0.0403115 0.0117953 2024-08-30T23:26:21.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:21.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:21.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:21.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:21.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:21.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:21.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:21.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:21.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:21.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:21 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2894511742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:26:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:21 smithi136 ceph-mon[20207]: osdmap e116: 12 total, 12 up, 12 in 2024-08-30T23:26:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:21 smithi136 ceph-mon[20207]: pgmap v331: 17 pgs: 17 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:26:21.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:21 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2894511742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:26:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:21 smithi173 ceph-mon[20554]: osdmap e116: 12 total, 12 up, 12 in 2024-08-30T23:26:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:21 smithi173 ceph-mon[20554]: pgmap v331: 17 pgs: 17 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:26:21.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:21 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2894511742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:26:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:21 smithi106 ceph-mon[16603]: osdmap e116: 12 total, 12 up, 12 in 2024-08-30T23:26:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:21 smithi106 ceph-mon[16603]: pgmap v331: 17 pgs: 17 active+clean; 1.7 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:26:21.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:21.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:21.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:22.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:22.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:22.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 32774 32758 85.3003 99.9375 0.0136281 0.0117172 2024-08-30T23:26:22.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:22.228 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:22.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:22.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:22.417 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:26:22.417 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:26:22.417 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:26:22.417 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:26:22.417 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:26:22.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:22.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:22.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:22.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:22.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:22.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:22.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:22.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:22.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:22.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:22.980 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:23.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:23.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:23.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 34062 34046 85.108 80.5 0.0124436 0.0117341 2024-08-30T23:26:23.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:23.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:23.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:23.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:23.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:23.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:23.611 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:23.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:23.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:23.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:23 smithi106 ceph-mon[16603]: pgmap v332: 17 pgs: 17 active+clean; 1.9 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-30T23:26:23.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:23 smithi136 ceph-mon[20207]: pgmap v332: 17 pgs: 17 active+clean; 1.9 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-30T23:26:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:23 smithi173 ceph-mon[20554]: pgmap v332: 17 pgs: 17 active+clean; 1.9 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-08-30T23:26:23.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:26:23.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:23.883 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:23.901 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:23.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:23.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:26:24.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:24.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:24.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:24.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:24.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 15 35402 35387 85.058 83.8125 0.0158287 0.0117507 2024-08-30T23:26:24.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:24.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:24.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:24.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:24.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:24.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:24.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:24.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:24.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:24.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:24.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:24.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:24.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:25.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:25.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:25.129 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:26:25.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:25.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 36492 36476 84.4283 68.0625 0.072304 0.0117902 2024-08-30T23:26:25.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:25.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:25.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:25.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:25.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:25.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:25.565 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:25.571 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:25.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:25.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:25.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:25.701 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:25.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:25.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:25.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:25.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:25.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:25.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:26.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:26.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:26.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 37818 37802 84.3726 82.875 0.0028423 0.011844 2024-08-30T23:26:26.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:26.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:26.282 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.6 to 1 (10000) 2024-08-30T23:26:26.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:26.344 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:26.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:26.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:26 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:26 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:26 smithi136 ceph-mon[20207]: osdmap e117: 12 total, 12 up, 12 in 2024-08-30T23:26:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:26 smithi136 ceph-mon[20207]: pgmap v334: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-30T23:26:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1452229885' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:26:26.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:26 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:26 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:26 smithi173 ceph-mon[20554]: osdmap e117: 12 total, 12 up, 12 in 2024-08-30T23:26:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:26 smithi173 ceph-mon[20554]: pgmap v334: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-30T23:26:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1452229885' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:26:26.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:26 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:26 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:26 smithi106 ceph-mon[16603]: osdmap e117: 12 total, 12 up, 12 in 2024-08-30T23:26:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:26 smithi106 ceph-mon[16603]: pgmap v334: 17 pgs: 17 active+clean; 2.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-30T23:26:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1452229885' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:26:26.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:26.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:26.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:26.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:26.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:26.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:26.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:27.085 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:1677 2.a e104/97 2:5e291dcb:::benchmark_data_smithi173_47195_object1676:head v 104'105, mlcod=104'102)", 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:59.172537+0000", 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 27.911591233999999, 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11230569, 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.172537+0000", 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.172537+0000", 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.172538+0000", 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.443e-06 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.172571+0000", 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2171000000000001e-05 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.284419+0000", 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.111848291 2024-08-30T23:26:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.284425+0000", 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7000000000000002e-06 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.284487+0000", 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1175999999999998e-05 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.284502+0000", 2024-08-30T23:26:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4963000000000001e-05 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.284815+0000", 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031293299999999999 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.284843+0000", 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8013000000000001e-05 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:1963 2.a e104/97 2:51e41735:::benchmark_data_smithi173_47195_object1962:head v 104'115, mlcod=104'112)", 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:59.346999+0000", 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 27.737128843000001, 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.098819089999999998, 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.346999+0000", 2024-08-30T23:26:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.346999+0000", 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.347000+0000", 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.99e-07 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.347053+0000", 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3396000000000002e-05 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.445462+0000", 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.098409057999999994 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.445467+0000", 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6949999999999999e-06 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.445507+0000", 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9934000000000001e-05 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.445522+0000", 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4571e-05 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.445791+0000", 2024-08-30T23:26:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026916899999999998 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:59.445818+0000", 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7568000000000001e-05 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14265 2.9 e109/97 2:94889bd8:::benchmark_data_smithi173_47195_object14264:head v 109'931, mlcod=109'928)", 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.946658+0000", 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 18.137470118, 2024-08-30T23:26:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.096936416999999997, 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.946658+0000", 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.946658+0000", 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.946659+0000", 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5599999999999995e-07 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.946676+0000", 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7723999999999998e-05 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.042779+0000", 2024-08-30T23:26:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.096103055000000007 2024-08-30T23:26:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.042788+0000", 2024-08-30T23:26:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7320000000000004e-06 2024-08-30T23:26:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.042857+0000", 2024-08-30T23:26:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9048999999999996e-05 2024-08-30T23:26:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.042883+0000", 2024-08-30T23:26:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5587000000000001e-05 2024-08-30T23:26:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.043553+0000", 2024-08-30T23:26:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00067055599999999999 2024-08-30T23:26:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.043594+0000", 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1158000000000001e-05 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14266 2.e e109/97 2:74094085:::benchmark_data_smithi173_47195_object14265:head v 109'831, mlcod=109'828)", 2024-08-30T23:26:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.947874+0000", 2024-08-30T23:26:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 18.136253630999999, 2024-08-30T23:26:27.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14747749700000001, 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947874+0000", 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947874+0000", 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947875+0000", 2024-08-30T23:26:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3799999999999996e-07 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947894+0000", 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8499e-05 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094480+0000", 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14658631 2024-08-30T23:26:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094487+0000", 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9260000000000003e-06 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094560+0000", 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3230000000000002e-05 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094579+0000", 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.823e-05 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095312+0000", 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073342100000000005 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095352+0000", 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0043000000000003e-05 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16161 2.6 e109/97 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head v 109'1027, mlcod=109'1024)", 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.984217864000001, 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212155128, 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099911+0000", 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5499999999999997e-07 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099933+0000", 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1793999999999998e-05 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311098+0000", 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21116489199999999 2024-08-30T23:26:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311105+0000", 2024-08-30T23:26:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6920000000000002e-06 2024-08-30T23:26:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311160+0000", 2024-08-30T23:26:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4716999999999998e-05 2024-08-30T23:26:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311179+0000", 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8814e-05 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312023+0000", 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00084434600000000005 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312065+0000", 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2117999999999999e-05 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16208 2.1 e109/97 2:8d8266cc:::benchmark_data_smithi173_47195_object16207:head v 109'996, mlcod=109'993)", 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.198301+0000", 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.885827423999999, 2024-08-30T23:26:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.116810463, 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198301+0000", 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198301+0000", 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198301+0000", 2024-08-30T23:26:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2600000000000002e-07 2024-08-30T23:26:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.198331+0000", 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9079999999999999e-05 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313990+0000", 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.115659331 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313997+0000", 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0650000000000001e-06 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.314266+0000", 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026887699999999999 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.314285+0000", 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8766e-05 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.315067+0000", 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078285200000000001 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.315111+0000", 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3766000000000001e-05 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16920 2.9 e111/97 2:9fc83a1f:::benchmark_data_smithi173_47195_object16919:head v 111'1097, mlcod=111'1093)", 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.677577+0000", 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.406551046000001, 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081870798999999994, 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.677577+0000", 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.677577+0000", 2024-08-30T23:26:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.677578+0000", 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.677600+0000", 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2500000000000001e-05 2024-08-30T23:26:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.758525+0000", 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080924251000000003 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.758529+0000", 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.702e-06 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.758582+0000", 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2858000000000001e-05 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.758596+0000", 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3532e-05 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.759418+0000", 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082260999999999999 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.759448+0000", 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9607999999999998e-05 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16925 2.a e111/97 2:5a33a07d:::benchmark_data_smithi173_47195_object16924:head v 111'1026, mlcod=111'1022)", 2024-08-30T23:26:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.682356+0000", 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.401771664999998, 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.164783656, 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682356+0000", 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682356+0000", 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682357+0000", 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5899999999999995e-07 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682411+0000", 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3603999999999997e-05 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846437+0000", 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16402667400000001 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846442+0000", 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5390000000000001e-06 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846515+0000", 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3258999999999998e-05 2024-08-30T23:26:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846529+0000", 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4073999999999999e-05 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.847107+0000", 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057730800000000001 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.847140+0000", 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3439000000000003e-05 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16929 2.9 e111/97 2:9c48c47c:::benchmark_data_smithi173_47195_object16928:head v 111'1098, mlcod=111'1094)", 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.684495+0000", 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.39963354, 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.076466811999999995, 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.684495+0000", 2024-08-30T23:26:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.684495+0000", 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.684495+0000", 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5600000000000005e-07 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.684516+0000", 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0449999999999999e-05 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.760319+0000", 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.075803628999999997 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.760324+0000", 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9300000000000002e-06 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.760370+0000", 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5510000000000003e-05 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.760384+0000", 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4596999999999999e-05 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.760922+0000", 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00053797000000000003 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.760961+0000", 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9069999999999997e-05 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16947 2.9 e111/97 2:90f25e0b:::benchmark_data_smithi173_47195_object16946:head v 111'1102, mlcod=111'1099)", 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.765884+0000", 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.318244604, 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.073251830000000004, 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.765884+0000", 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.765884+0000", 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.765884+0000", 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6599999999999996e-07 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.765902+0000", 2024-08-30T23:26:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7873e-05 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.838301+0000", 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.072399176999999995 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.838314+0000", 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2632999999999999e-05 2024-08-30T23:26:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.838455+0000", 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000141533 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.838506+0000", 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0667000000000001e-05 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.839094+0000", 2024-08-30T23:26:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000588091 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.839135+0000", 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1189999999999997e-05 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17140 2.a e111/97 2:577f215e:::benchmark_data_smithi173_47195_object17139:head v 111'1039, mlcod=111'1035)", 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.967877+0000", 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.116251174999999, 2024-08-30T23:26:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.171820526, 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967877+0000", 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967877+0000", 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967878+0000", 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967917+0000", 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8785999999999998e-05 2024-08-30T23:26:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.138852+0000", 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17093591899999999 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.138890+0000", 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7098999999999998e-05 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.138989+0000", 2024-08-30T23:26:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9411000000000001e-05 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.139028+0000", 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9093e-05 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.139671+0000", 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064274799999999995 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.139697+0000", 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6665e-05 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17190 2.e e111/97 2:7818eb93:::benchmark_data_smithi173_47195_object17189:head v 111'1012, mlcod=111'1009)", 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.987305+0000", 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.096823356000002, 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12476780899999999, 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987305+0000", 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987305+0000", 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987306+0000", 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987334+0000", 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8742999999999999e-05 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111418+0000", 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12408384 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111423+0000", 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2320000000000003e-06 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111465+0000", 2024-08-30T23:26:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2301e-05 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111480+0000", 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5293000000000001e-05 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112044+0000", 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056360100000000005 2024-08-30T23:26:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112073+0000", 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8829000000000001e-05 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17191 2.6 e111/97 2:612ab7c0:::benchmark_data_smithi173_47195_object17190:head v 111'1091, mlcod=111'1088)", 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.987424+0000", 2024-08-30T23:26:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.096704330000001, 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123209083, 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987424+0000", 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987424+0000", 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987425+0000", 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4900000000000005e-07 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987478+0000", 2024-08-30T23:26:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3032000000000003e-05 2024-08-30T23:26:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109887+0000", 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.122409688 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109895+0000", 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8129999999999996e-06 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109951+0000", 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5695999999999997e-05 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109966+0000", 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5500000000000001e-05 2024-08-30T23:26:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.110603+0000", 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000637139 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.110633+0000", 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9465999999999999e-05 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17193 2.1 e111/97 2:840d9093:::benchmark_data_smithi173_47195_object17192:head v 111'1058, mlcod=111'1056)", 2024-08-30T23:26:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.989003+0000", 2024-08-30T23:26:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 16.095124946999999, 2024-08-30T23:26:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.124622043, 2024-08-30T23:26:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989003+0000", 2024-08-30T23:26:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989003+0000", 2024-08-30T23:26:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989004+0000", 2024-08-30T23:26:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:26:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989026+0000", 2024-08-30T23:26:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2013000000000001e-05 2024-08-30T23:26:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112885+0000", 2024-08-30T23:26:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123858969 2024-08-30T23:26:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112890+0000", 2024-08-30T23:26:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999998e-06 2024-08-30T23:26:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112950+0000", 2024-08-30T23:26:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9723999999999999e-05 2024-08-30T23:26:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112964+0000", 2024-08-30T23:26:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4404e-05 2024-08-30T23:26:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113594+0000", 2024-08-30T23:26:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00062978700000000003 2024-08-30T23:26:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113625+0000", 2024-08-30T23:26:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1294000000000002e-05 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17733 2.6 e111/97 2:6e93df89:::benchmark_data_smithi173_47195_object17732:head v 111'1123, mlcod=111'1120)", 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:11.509128+0000", 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 15.574999832, 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081056240000000002, 2024-08-30T23:26:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.509128+0000", 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.509128+0000", 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.509129+0000", 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.71e-07 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.509150+0000", 2024-08-30T23:26:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0557999999999999e-05 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.589430+0000", 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080280347000000002 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.589435+0000", 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1869999999999996e-06 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.589487+0000", 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1817000000000002e-05 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.589502+0000", 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4813e-05 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.590154+0000", 2024-08-30T23:26:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065233800000000003 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.590185+0000", 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0309000000000001e-05 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17761 2.6 e111/97 2:658ece0f:::benchmark_data_smithi173_47195_object17760:head v 111'1125, mlcod=111'1123)", 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:11.621706+0000", 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 15.462421856000001, 2024-08-30T23:26:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.092034106000000004, 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.621706+0000", 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.621706+0000", 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.621708+0000", 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3909999999999999e-06 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.621738+0000", 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0626000000000001e-05 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.712399+0000", 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.090661179999999994 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.712407+0000", 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9240000000000007e-06 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.712470+0000", 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2883000000000001e-05 2024-08-30T23:26:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.712490+0000", 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9510000000000001e-05 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.713695+0000", 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012049540000000001 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.713740+0000", 2024-08-30T23:26:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5637999999999999e-05 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:20477 2.d e112/97 2:b4d06087:::benchmark_data_smithi173_47195_object20476:head v 112'1322, mlcod=112'1318)", 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:13.498350+0000", 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.585778272000001, 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.081374098000000006, 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.498350+0000", 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.498350+0000", 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.498351+0000", 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1100000000000005e-07 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.498458+0000", 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107285 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.578617+0000", 2024-08-30T23:26:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080159490999999999 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.578626+0000", 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6379999999999993e-06 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.578699+0000", 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3039000000000003e-05 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.578717+0000", 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8241999999999999e-05 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.579680+0000", 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00096232000000000002 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:13.579724+0000", 2024-08-30T23:26:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4271999999999999e-05 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:31255 2.a e116/97 2:50500f73:::benchmark_data_smithi173_47195_object31254:head v 116'1903, mlcod=116'1900)", 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:21.249115+0000", 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 5.8350130399999998, 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099058610000000005, 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.249115+0000", 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.249115+0000", 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.249116+0000", 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.85e-07 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.249167+0000", 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0930999999999997e-05 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.346996+0000", 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.097829357000000006 2024-08-30T23:26:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.347003+0000", 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3849999999999999e-06 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.347060+0000", 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7321000000000002e-05 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.347080+0000", 2024-08-30T23:26:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0282999999999999e-05 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.348131+0000", 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010508 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:21.348174+0000", 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2747999999999998e-05 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:35944 2.e e116/97 2:7d87a663:::benchmark_data_smithi173_47195_object35943:head v 116'2197, mlcod=116'2194)", 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:24.525700+0000", 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.5584278519999999, 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.068721705999999994, 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.525700+0000", 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.525700+0000", 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.525701+0000", 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2200000000000004e-07 2024-08-30T23:26:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.525719+0000", 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7674999999999999e-05 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.593345+0000", 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.067626495999999994 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.593350+0000", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6970000000000001e-06 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.593400+0000", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9835e-05 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.593415+0000", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5352000000000001e-05 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.594389+0000", 2024-08-30T23:26:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00097422000000000003 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:24.594422+0000", 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2808999999999997e-05 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:36483 2.d e116/97 2:b8c0a5cd:::benchmark_data_smithi173_47195_object36482:head v 116'2353, mlcod=116'2350)", 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:25.049050+0000", 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.035078124, 2024-08-30T23:26:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16564358600000001, 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049050+0000", 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049050+0000", 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049051+0000", 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3e-07 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049144+0000", 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3708999999999993e-05 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213246+0000", 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.164101892 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213253+0000", 2024-08-30T23:26:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2910000000000004e-06 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213314+0000", 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1643999999999995e-05 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213339+0000", 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4748000000000001e-05 2024-08-30T23:26:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.214659+0000", 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001319656 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.214694+0000", 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4916000000000002e-05 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:26:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:26:27.206 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:26:27.206 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 39425 39409 84.9263 100.438 0.00619482 0.0117674 2024-08-30T23:26:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:27.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:27.298 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:26:27.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:27.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:27.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:27.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:27.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1452229885' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:26:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:27 smithi173 ceph-mon[20554]: osdmap e118: 12 total, 12 up, 12 in 2024-08-30T23:26:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/244660111' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:27.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1452229885' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:26:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:27 smithi106 ceph-mon[16603]: osdmap e118: 12 total, 12 up, 12 in 2024-08-30T23:26:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/244660111' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:27.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1452229885' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:26:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:27 smithi136 ceph-mon[20207]: osdmap e118: 12 total, 12 up, 12 in 2024-08-30T23:26:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/244660111' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:27.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:27.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:27.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:27.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:27.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:28.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:28.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:28.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 15 40700 40685 84.7535 79.75 0.00441013 0.0117926 2024-08-30T23:26:28.186 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:28.218 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:28.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:28.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:28.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:26:28.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:28.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:28.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:28.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:28.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:28 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:28.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/244660111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:28.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:28 smithi173 ceph-mon[20554]: osdmap e119: 12 total, 12 up, 12 in 2024-08-30T23:26:28.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:28 smithi173 ceph-mon[20554]: pgmap v337: 17 pgs: 17 active+clean; 2.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-30T23:26:28.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:28.606 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:28 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/244660111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:28 smithi106 ceph-mon[16603]: osdmap e119: 12 total, 12 up, 12 in 2024-08-30T23:26:28.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:28 smithi106 ceph-mon[16603]: pgmap v337: 17 pgs: 17 active+clean; 2.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-30T23:26:28.649 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:28.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:28.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:28.773 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:28 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:28.773 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/244660111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:28.773 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:28 smithi136 ceph-mon[20207]: osdmap e119: 12 total, 12 up, 12 in 2024-08-30T23:26:28.774 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:28 smithi136 ceph-mon[20207]: pgmap v337: 17 pgs: 17 active+clean; 2.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-08-30T23:26:28.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:28.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:28.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:28.978 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:28.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:29.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:29.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:29.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-08-30T23:26:29.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 42077 42061 84.7935 86 0.0224923 0.0117862 2024-08-30T23:26:29.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:29.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:29.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:29.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:29.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:29.482 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:29 smithi136 ceph-mon[20207]: pgmap v338: 17 pgs: 17 active+clean; 2.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-08-30T23:26:29.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:29.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:29.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:29.720 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:29.744 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:29 smithi173 ceph-mon[20554]: pgmap v338: 17 pgs: 17 active+clean; 2.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-08-30T23:26:29.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:29.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:29 smithi106 ceph-mon[16603]: pgmap v338: 17 pgs: 17 active+clean; 2.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-08-30T23:26:29.867 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:29.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:29.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:29.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:30.024 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:26:30.024 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:26:30.025 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:30.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:30.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:30.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 43354 43338 84.6376 79.8125 0.00570479 0.0118063 2024-08-30T23:26:30.210 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:30.210 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:30.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:30.237 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:30.238 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:30.250 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:30.263 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:30.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:30.292 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:30.292 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:30.297 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:30.322 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:30.322 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:30.345 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:30.346 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:30.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:30.366 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:30.367 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:30.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:30.387 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:30.388 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:30.410 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:30.410 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:30.434 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:30.435 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:30.461 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:30.462 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:30.462 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/373194720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:30.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:30.492 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:30.493 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:30.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:30.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:30.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:30.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:30.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:30.756 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/373194720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:30.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:30.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:26:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/373194720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:30.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:30.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:30.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:30.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:31.020 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:31.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:31.152 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:31.154 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:31.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 44670 44654 84.5651 82.25 0.00309847 0.0118172 2024-08-30T23:26:31.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:31.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:31.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:31.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:31.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:31.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:31 smithi136 ceph-mon[20207]: pgmap v339: 17 pgs: 17 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-30T23:26:31.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:31.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:31.716 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:31.716 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:31.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:31.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:31 smithi173 ceph-mon[20554]: pgmap v339: 17 pgs: 17 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-30T23:26:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:31 smithi106 ceph-mon[16603]: pgmap v339: 17 pgs: 17 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-30T23:26:31.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:31.933 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:31.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:31.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:26:32.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:32.067 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:32.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 46412 46396 85.2798 108.875 0.00246955 0.0117204 2024-08-30T23:26:32.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:32.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:32.293 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:32.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:32.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:32.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:32.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:32.523 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T23:26:32.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:32.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:32.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:32.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:32.777 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3281985175' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:26:32.777 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:32 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:26:32.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:32.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:26:32.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3281985175' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:26:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:32 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:26:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3281985175' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:26:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:32 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:26:32.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:32.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:32.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:33.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:33.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:33.107 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:33.164 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 47435 47419 84.67 63.9375 0.00259309 0.0118035 2024-08-30T23:26:33.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:33.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:33.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-08-30T23:26:33.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:33.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:26:33.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:33.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:33.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:33.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:26:33.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:33 smithi106 ceph-mon[16603]: osdmap e120: 12 total, 12 up, 12 in 2024-08-30T23:26:33.529 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:33 smithi106 ceph-mon[16603]: pgmap v341: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-30T23:26:33.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:33.601 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:33.644 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:33.644 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:33.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:33.707 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.742 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:33.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:33.769 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:26:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:33 smithi136 ceph-mon[20207]: osdmap e120: 12 total, 12 up, 12 in 2024-08-30T23:26:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:33 smithi136 ceph-mon[20207]: pgmap v341: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-30T23:26:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:26:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:33 smithi173 ceph-mon[20554]: osdmap e120: 12 total, 12 up, 12 in 2024-08-30T23:26:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:33 smithi173 ceph-mon[20554]: pgmap v341: 17 pgs: 17 active+clean; 2.7 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-30T23:26:33.843 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:33.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:33.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:33.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:33.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:34.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:34.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:34.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 48639 48623 84.4081 75.25 0.00236359 0.0118401 2024-08-30T23:26:34.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:34.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.249 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:34.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:26:34.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:34.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:34.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:34.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:34.558 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.8 to 1 (10000) 2024-08-30T23:26:34.562 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:26:34.593 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:34.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:34.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:34.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:34.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:34.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: osdmap e121: 12 total, 12 up, 12 in 2024-08-30T23:26:34.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:26:34.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:26:34.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1204727280' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:34.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1335692122' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:26:34.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:34.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: osdmap e121: 12 total, 12 up, 12 in 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1204727280' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:34.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1335692122' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:26:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: osdmap e121: 12 total, 12 up, 12 in 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1204727280' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:26:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1335692122' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:26:34.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:34.890 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:34.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:34.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:35.013 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:35.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:35.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:35.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 49873 49857 84.2111 77.125 0.00996583 0.0118695 2024-08-30T23:26:35.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:35.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:35.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:35.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:35.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:35.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:35.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:35.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:26:35.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1335692122' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:26:35.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: osdmap e122: 12 total, 12 up, 12 in 2024-08-30T23:26:35.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/345444497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:35.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: pgmap v344: 17 pgs: 17 active+clean; 2.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-30T23:26:35.551 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:35 smithi136 ceph-mon[20207]: osdmap e123: 12 total, 12 up, 12 in 2024-08-30T23:26:35.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:35.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:35.677 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:35.677 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:26:35.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50262 2.2 2:4be7cc22:::benchmark_data_smithi173_47195_object50261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.657520+0000", 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.019769689999999999, 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019803838000000001, 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50262 2024-08-30T23:26:35.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657520+0000", 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657520+0000", 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:35.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657522+0000", 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1149999999999999e-06 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657561+0000", 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8948e-05 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657572+0000", 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1885999999999999e-05 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657579+0000", 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7780000000000002e-06 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657621+0000", 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1854999999999998e-05 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657709+0000", 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7539000000000003e-05 2024-08-30T23:26:35.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.657855+0000", 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014589599999999999 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.659647+0000", 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017920309999999999 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.666303+0000", 2024-08-30T23:26:35.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006656861 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50271 2.2 2:4d028552:::benchmark_data_smithi173_47195_object50270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.664263+0000", 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013026727, 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013145288, 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50271 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664263+0000", 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664263+0000", 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:35.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664265+0000", 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1799999999999999e-06 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664324+0000", 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8891000000000002e-05 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664335+0000", 2024-08-30T23:26:35.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277000000000001e-05 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664340+0000", 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4319999999999998e-06 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664378+0000", 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7707999999999999e-05 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:35.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664454+0000", 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6154000000000003e-05 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.664533+0000", 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8807000000000006e-05 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:35.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.666073+0000", 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001540514 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-08-30T23:26:35.688 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:35.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:35.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:26:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1335692122' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:26:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: osdmap e122: 12 total, 12 up, 12 in 2024-08-30T23:26:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/345444497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: pgmap v344: 17 pgs: 17 active+clean; 2.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-30T23:26:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:35 smithi173 ceph-mon[20554]: osdmap e123: 12 total, 12 up, 12 in 2024-08-30T23:26:35.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:35.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:35.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:35.828 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:35.828 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:35.828 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:35.828 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:35.828 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:35.829 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:35.829 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:35.829 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:35.829 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:35.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:35.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:35.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:35.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:35.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:35.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:35.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:35.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:35.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:35.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:35.832 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:35.832 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:35.832 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:26:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1335692122' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:26:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: osdmap e122: 12 total, 12 up, 12 in 2024-08-30T23:26:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/345444497' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: pgmap v344: 17 pgs: 17 active+clean; 2.8 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-08-30T23:26:35.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:35 smithi106 ceph-mon[16603]: osdmap e123: 12 total, 12 up, 12 in 2024-08-30T23:26:35.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:35.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:35.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:36.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:36.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:36.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 15 51081 51066 83.9834 75.5625 0.00506715 0.0118992 2024-08-30T23:26:36.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:36.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:36.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:36.335 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:36.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:26:36.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:36.464 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:36.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:36.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:36.676 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:36.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:36.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:36.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:36.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:36.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:36.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:36.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:37.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:37.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:37.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 52769 52753 84.5333 105.438 0.00425582 0.0118245 2024-08-30T23:26:37.195 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:37.195 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:26:37.195 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:26:37.195 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:26:37.195 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:37.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:37.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:37.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:37.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:37.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:37.494 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:37.521 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:37 smithi106 ceph-mon[16603]: pgmap v346: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-30T23:26:37.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:37.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:37.725 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:37.749 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:37 smithi136 ceph-mon[20207]: pgmap v346: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-30T23:26:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:37 smithi173 ceph-mon[20554]: pgmap v346: 17 pgs: 17 active+clean; 3.0 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-30T23:26:37.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:37.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:37.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:37.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:38.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:38.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-08-30T23:26:38.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:38.145 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:38.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:26:38.166378+0000 min lat: 0.00101743 max lat: 0.311492 avg lat: 0.0118461 2024-08-30T23:26:38.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:26:38.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 54019 54003 84.3729 78.125 0.00743949 0.0118461 2024-08-30T23:26:38.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:38.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:38.397 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:38.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:38.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:38.525 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:38.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:38.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:38.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:26:38.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:38.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:38.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:26:38.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:38.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:39.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:39.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:39.074 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:39.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:39.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:39.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 55138 55122 84.0206 69.9375 0.0190471 0.0118964 2024-08-30T23:26:39.180 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:39.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:39.278 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.9 to 1 (10000) 2024-08-30T23:26:39.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:39.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:39.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:39.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1448260593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:26:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:39 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:26:39.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:39.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:26:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1448260593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:26:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:39 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:26:39.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1448260593' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:26:39.587 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:39 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:26:39.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:39.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:39.700 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:26:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:26:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:26:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:838 2.3 2:c3b7287d:::benchmark_data_smithi173_47195_object837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.573902+0000", 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.122900469999998, 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14770838999999999, 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 838 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.573902+0000", 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.573902+0000", 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.573903+0000", 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-30T23:26:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.573968+0000", 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5040000000000001e-05 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.573978+0000", 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.573983+0000", 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9429999999999999e-06 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574018+0000", 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4906000000000001e-05 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574082+0000", 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3968999999999997e-05 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574167+0000", 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5205999999999997e-05 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574467+0000", 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029992899999999999 2024-08-30T23:26:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574905+0000", 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00043739700000000001 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721537+0000", 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14663230699999999 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721576+0000", 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8565000000000001e-05 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721610+0000", 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4749000000000002e-05 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:838 2.3 e104/97 ondisk, result = 0)", 2024-08-30T23:26:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.574607+0000", 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.122195574999999, 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14701007599999999, 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574607+0000", 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574607+0000", 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574607+0000", 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9900000000000001e-07 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.574610+0000", 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.066e-06 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721479+0000", 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14686822799999999 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721485+0000", 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7259999999999996e-06 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721524+0000", 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9152000000000001e-05 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721533+0000", 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1550000000000007e-06 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721617+0000", 2024-08-30T23:26:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4250000000000001e-05 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:841 2.3 2:ca03e5d0:::benchmark_data_smithi173_47195_object840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.578170+0000", 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.118632707000003, 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14738305800000001, 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 841 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578170+0000", 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578170+0000", 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578170+0000", 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578218+0000", 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7339000000000002e-05 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578230+0000", 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2193e-05 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578235+0000", 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1309999999999997e-06 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578273+0000", 2024-08-30T23:26:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7564000000000002e-05 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578341+0000", 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7957e-05 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578422+0000", 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0910999999999998e-05 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578717+0000", 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029556199999999997 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579104+0000", 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038697999999999998 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725501+0000", 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14639670499999999 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725530+0000", 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9020999999999999e-05 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725553+0000", 2024-08-30T23:26:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2904e-05 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:841 2.3 e104/97 ondisk, result = 0)", 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.578928+0000", 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.117874385, 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14662830600000001, 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578928+0000", 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578928+0000", 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578928+0000", 2024-08-30T23:26:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9200000000000001e-07 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.578932+0000", 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1200000000000002e-06 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725461+0000", 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.146529505 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725465+0000", 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3649999999999997e-06 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725494+0000", 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8345999999999999e-05 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725498+0000", 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0980000000000004e-06 2024-08-30T23:26:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.725556+0000", 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.838e-05 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:14219 2.2 2:4d4dfd1e:::benchmark_data_smithi173_47195_object14218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.870340+0000", 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.826462199000002, 2024-08-30T23:26:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166500233, 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14219 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870340+0000", 2024-08-30T23:26:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870340+0000", 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870341+0000", 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.124e-06 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870407+0000", 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5265999999999995e-05 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870418+0000", 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1408e-05 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870425+0000", 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0489999999999998e-06 2024-08-30T23:26:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870469+0000", 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4311999999999999e-05 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870548+0000", 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8557e-05 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.870628+0000", 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9876000000000006e-05 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.871190+0000", 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00056271900000000004 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.876375+0000", 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051842879999999996 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036734+0000", 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.160359633 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036795+0000", 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0333e-05 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036840+0000", 2024-08-30T23:26:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5667999999999997e-05 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:14219 2.2 e109/97 ondisk, result = 0)", 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.871403+0000", 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.825399730000001, 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16544442000000001, 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.871403+0000", 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.871403+0000", 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.871404+0000", 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.006e-06 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.871407+0000", 2024-08-30T23:26:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4649999999999999e-06 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036672+0000", 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.165264779 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036677+0000", 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2700000000000004e-06 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036722+0000", 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4827000000000003e-05 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036729+0000", 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0219999999999999e-06 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.036847+0000", 2024-08-30T23:26:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118051 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:14227 2.2 2:4af79d78:::benchmark_data_smithi173_47195_object14226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.821764681000001, 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24349405099999999, 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14227 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875039+0000", 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046e-06 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875091+0000", 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2556999999999998e-05 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875103+0000", 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1331e-05 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875108+0000", 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1549999999999998e-06 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875152+0000", 2024-08-30T23:26:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4161e-05 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875235+0000", 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2814999999999999e-05 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875316+0000", 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699000000000005e-05 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875841+0000", 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052407300000000001 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.037068+0000", 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.161227964 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118454+0000", 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.081385499 2024-08-30T23:26:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118498+0000", 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3513000000000002e-05 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118532+0000", 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4238e-05 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:14227 2.2 e109/97 ondisk, result = 0)", 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.82080809, 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.242546031, 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:26:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875995+0000", 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0100000000000004e-07 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875999+0000", 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.177e-06 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.117926+0000", 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.241927006 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.117934+0000", 2024-08-30T23:26:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5290000000000003e-06 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118441+0000", 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050753599999999995 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118448+0000", 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6619999999999999e-06 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118540+0000", 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2419999999999999e-05 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:14244 2.3 2:c35428fa:::benchmark_data_smithi173_47195_object14243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.883481+0000", 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.813321081000002, 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21455508000000001, 2024-08-30T23:26:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14244 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883481+0000", 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883481+0000", 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883482+0000", 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.09e-07 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883510+0000", 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8248000000000001e-05 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883521+0000", 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0745e-05 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883544+0000", 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2484999999999999e-05 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883586+0000", 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2521999999999997e-05 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883664+0000", 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7589999999999994e-05 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.883746+0000", 2024-08-30T23:26:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2091999999999999e-05 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.884214+0000", 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00046864200000000002 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.884512+0000", 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029721700000000001 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.097927+0000", 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.213415141 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.097987+0000", 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0390999999999998e-05 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.098036+0000", 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9197999999999999e-05 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:16921 2.2 2:4bb89a5b:::benchmark_data_smithi173_47195_object16920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.678480+0000", 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.018322620999999, 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22023874399999999, 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 16921 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678480+0000", 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678480+0000", 2024-08-30T23:26:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678481+0000", 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.125e-06 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678534+0000", 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2828000000000003e-05 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678545+0000", 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0908e-05 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678550+0000", 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1259999999999997e-06 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678592+0000", 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2175999999999997e-05 2024-08-30T23:26:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678667+0000", 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4757000000000003e-05 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678743+0000", 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5879999999999999e-05 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.679411+0000", 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066885499999999999 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.679624+0000", 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021280499999999999 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.898640+0000", 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.219015931 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.898685+0000", 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4543999999999999e-05 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.898718+0000", 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3809000000000001e-05 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:17189 2.2 2:4470000c:::benchmark_data_smithi173_47195_object17188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.986428+0000", 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.710374516999998, 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.157988244, 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17189 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986428+0000", 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986428+0000", 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986433+0000", 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7620000000000003e-06 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986478+0000", 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5117999999999997e-05 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986493+0000", 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5551999999999999e-05 2024-08-30T23:26:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986501+0000", 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8280000000000003e-06 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986543+0000", 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1919999999999998e-05 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986628+0000", 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5226e-05 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.986725+0000", 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6953000000000005e-05 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.987441+0000", 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071547799999999997 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.987733+0000", 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029226799999999999 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144348+0000", 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15661544599999999 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144381+0000", 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2839999999999997e-05 2024-08-30T23:26:39.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144416+0000", 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4853e-05 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:17197 2.2 2:414a6f6f:::benchmark_data_smithi173_47195_object17196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.996199+0000", 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.700603474000001, 2024-08-30T23:26:39.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15463956800000001, 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17197 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996199+0000", 2024-08-30T23:26:39.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996199+0000", 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996200+0000", 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4559999999999999e-06 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996240+0000", 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9892999999999999e-05 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996257+0000", 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6526000000000001e-05 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996263+0000", 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9309999999999996e-06 2024-08-30T23:26:39.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996309+0000", 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6079999999999999e-05 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996403+0000", 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4327000000000007e-05 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.996492+0000", 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8925000000000007e-05 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.997284+0000", 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00079233400000000001 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.997476+0000", 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019120000000000001 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.150764+0000", 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15328810800000001 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.150796+0000", 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2634e-05 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.150838+0000", 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2154000000000003e-05 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:17189 2.2 e111/97 ondisk, result = 0)", 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.997607+0000", 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.699195588999999, 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.146815166, 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.774 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.997607+0000", 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.997607+0000", 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.997607+0000", 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7999999999999995e-07 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.997614+0000", 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.613e-06 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144296+0000", 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.146682226 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144303+0000", 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5509999999999996e-06 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144340+0000", 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6947999999999999e-05 2024-08-30T23:26:39.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144345+0000", 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9239999999999998e-06 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.144422+0000", 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7224000000000005e-05 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:17737 2.2 2:4fd7dd2a:::benchmark_data_smithi173_47195_object17736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.166552852999999, 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22122945399999999, 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17737 2024-08-30T23:26:39.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5199999999999995e-07 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530297+0000", 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6873999999999999e-05 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530308+0000", 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1147e-05 2024-08-30T23:26:39.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530315+0000", 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530355+0000", 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0649999999999999e-05 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530438+0000", 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3207999999999993e-05 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530520+0000", 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1971999999999994e-05 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531149+0000", 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000628139 2024-08-30T23:26:39.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531405+0000", 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000256142 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751373+0000", 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.219967995 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751433+0000", 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0591000000000003e-05 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751479+0000", 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5661999999999998e-05 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.785 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:17737 2.2 e111/97 ondisk, result = 0)", 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:11.531482+0000", 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.165320237, 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22000273300000001, 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531482+0000", 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531482+0000", 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531483+0000", 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6800000000000001e-07 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531486+0000", 2024-08-30T23:26:39.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0520000000000002e-06 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751313+0000", 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.219827576 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751319+0000", 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2750000000000003e-06 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751358+0000", 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9786999999999997e-05 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751367+0000", 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5939999999999994e-06 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751485+0000", 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011798099999999999 2024-08-30T23:26:39.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:36436 2.2 2:4f13755f:::benchmark_data_smithi173_47195_object36435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.713635694000001, 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23381564699999999, 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36436 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983169+0000", 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9470000000000002e-06 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983211+0000", 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1959000000000003e-05 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983229+0000", 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8508e-05 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983238+0000", 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9549999999999998e-06 2024-08-30T23:26:39.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983294+0000", 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5605e-05 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983377+0000", 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3768000000000002e-05 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983458+0000", 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0272000000000007e-05 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.984672+0000", 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214449 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.985077+0000", 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040527100000000002 2024-08-30T23:26:39.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216885+0000", 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.231807244 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216945+0000", 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0545999999999999e-05 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216982+0000", 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7122999999999997e-05 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:36447 2.2 2:42c8382e:::benchmark_data_smithi173_47195_object36446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.708564294, 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31128452299999998, 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36447 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988239+0000", 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4e-07 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988315+0000", 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5585000000000002e-05 2024-08-30T23:26:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988325+0000", 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0549e-05 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988330+0000", 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9729999999999997e-06 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988371+0000", 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1201000000000001e-05 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988453+0000", 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1965000000000006e-05 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988536+0000", 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3088000000000001e-05 2024-08-30T23:26:39.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.989696+0000", 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001159979 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.989973+0000", 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027723299999999999 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299419+0000", 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30944559100000002 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299477+0000", 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7568000000000002e-05 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299523+0000", 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5951000000000001e-05 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50013 2.2 2:45c153d4:::benchmark_data_smithi173_47195_object50012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.313453+0000", 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.3833494359999996, 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15463796399999999, 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50013 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313453+0000", 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313453+0000", 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313459+0000", 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6699999999999999e-06 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313502+0000", 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3618999999999997e-05 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313527+0000", 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4507000000000001e-05 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313540+0000", 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3022e-05 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313668+0000", 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012847100000000001 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313845+0000", 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000176887 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.313938+0000", 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3017999999999996e-05 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.315192+0000", 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012536839999999999 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.466718+0000", 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15152621999999999 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.468025+0000", 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001307009 2024-08-30T23:26:39.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.468059+0000", 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3609000000000003e-05 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.468091+0000", 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2248e-05 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:54550 2.3 2:c43076a9:::benchmark_data_smithi173_47195_object54549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:38.691650+0000", 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.0051519259999999, 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13912596199999999, 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54550 2024-08-30T23:26:39.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691650+0000", 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691650+0000", 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691651+0000", 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.04e-06 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691693+0000", 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1551999999999998e-05 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691708+0000", 2024-08-30T23:26:39.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4610000000000001e-05 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691716+0000", 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7660000000000004e-06 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691786+0000", 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9219000000000003e-05 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691876+0000", 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0754000000000006e-05 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.691960+0000", 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3465000000000001e-05 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.693495+0000", 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015350330000000001 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.693863+0000", 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036815099999999999 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830670+0000", 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13680719099999999 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830730+0000", 2024-08-30T23:26:39.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0173999999999997e-05 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830776+0000", 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6007000000000002e-05 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:54550 2.3 e123/97 ondisk, result = 0)", 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:38.693654+0000", 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.00314824, 2024-08-30T23:26:39.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.137127834, 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.693654+0000", 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.693654+0000", 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.693655+0000", 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.693657+0000", 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6539999999999999e-06 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830617+0000", 2024-08-30T23:26:39.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.136959942 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830622+0000", 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7210000000000002e-06 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830659+0000", 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6887000000000001e-05 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:39.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830665+0000", 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:38.830782+0000", 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116694 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:39.822 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:39.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:39.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:39.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:39.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:40.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-08-30T23:26:40.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:40.105 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:40.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 15 56377 56362 83.8652 77.5 0.01241 0.0119191 2024-08-30T23:26:40.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:40.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:40.284 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:40.302 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:40.303 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:40.340 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:40.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:40.371 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:40.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:40.402 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:40.403 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:40.431 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:40.431 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:40.457 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:40.458 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:40.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:40.481 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:40.482 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:40.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:40.506 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:40.506 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:40.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:40.532 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:40.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:40 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:26:40.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:40 smithi106 ceph-mon[16603]: osdmap e124: 12 total, 12 up, 12 in 2024-08-30T23:26:40.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:40 smithi106 ceph-mon[16603]: pgmap v348: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:26:40.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1261662869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:40.556 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:40 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:26:40.556 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:40 smithi136 ceph-mon[20207]: osdmap e124: 12 total, 12 up, 12 in 2024-08-30T23:26:40.556 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:40 smithi136 ceph-mon[20207]: pgmap v348: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:26:40.556 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1261662869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:40.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:26:40.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:40.564 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:40.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:40 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:26:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:40 smithi173 ceph-mon[20554]: osdmap e124: 12 total, 12 up, 12 in 2024-08-30T23:26:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:40 smithi173 ceph-mon[20554]: pgmap v348: 17 pgs: 17 active+clean; 3.2 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:26:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1261662869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:40.594 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:40.594 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:40.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:40.629 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:40.629 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:40.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:40.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:40.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:40.845 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:40.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:40.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:40.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:41.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:41.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:41.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:41.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.147 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:41.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:41.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 57494 57478 83.5369 69.75 0.0544837 0.0119597 2024-08-30T23:26:41.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:41.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:41.344 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:41.344 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:41.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:41.394 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:41.461 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:41.493 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:26:41.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:41.523 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:41.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:41.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:41.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:41.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:41 smithi136 ceph-mon[20207]: osdmap e125: 12 total, 12 up, 12 in 2024-08-30T23:26:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3786471531' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:41 smithi136 ceph-mon[20207]: pgmap v350: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.42k op/s 2024-08-30T23:26:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:41 smithi173 ceph-mon[20554]: osdmap e125: 12 total, 12 up, 12 in 2024-08-30T23:26:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3786471531' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:41 smithi173 ceph-mon[20554]: pgmap v350: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.42k op/s 2024-08-30T23:26:41.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:41 smithi106 ceph-mon[16603]: osdmap e125: 12 total, 12 up, 12 in 2024-08-30T23:26:41.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3786471531' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:41.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:41 smithi106 ceph-mon[16603]: pgmap v350: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.42k op/s 2024-08-30T23:26:41.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:41.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:41.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:42.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:42.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:42.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:26:42.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:42.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:42.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 58496 58480 83.0615 62.625 0.0154071 0.0120296 2024-08-30T23:26:42.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:42.290 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:42.372 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:42.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:42.418 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:42.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:42.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:42.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.670 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:42.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:42.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:42.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:42 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:26:42.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:42 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:26:42.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3786471531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:42.772 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:42 smithi136 ceph-mon[20207]: osdmap e126: 12 total, 12 up, 12 in 2024-08-30T23:26:42.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:42 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:26:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:42 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:26:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3786471531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:42 smithi173 ceph-mon[20554]: osdmap e126: 12 total, 12 up, 12 in 2024-08-30T23:26:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:42 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:26:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:42 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:26:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3786471531' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:42 smithi106 ceph-mon[16603]: osdmap e126: 12 total, 12 up, 12 in 2024-08-30T23:26:42.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:42.922 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:42.922 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:42.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:42.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:43.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:43.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:43.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:43.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 59545 59529 82.6725 65.5625 0.00969905 0.0120858 2024-08-30T23:26:43.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:43.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:43.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:43.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:43.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:43.504 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.10 to 1 (10000) 2024-08-30T23:26:43.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:43.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2332422138' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:26:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:43 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:26:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:43 smithi106 ceph-mon[16603]: pgmap v352: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-08-30T23:26:43.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:43.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:26:43.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2332422138' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:26:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:43 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:26:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:43 smithi136 ceph-mon[20207]: pgmap v352: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-08-30T23:26:43.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:43.821 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2332422138' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:26:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:43 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:26:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:43 smithi173 ceph-mon[20554]: pgmap v352: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-08-30T23:26:43.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:26:43.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:43.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:43.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:43.984 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:44.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:44.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:44.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 61044 61028 82.9118 93.6875 0.00565106 0.0120557 2024-08-30T23:26:44.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:44.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:44.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-08-30T23:26:44.304 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:44.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:44.340 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:44.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:44.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:44.507 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:44.507 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:44.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:44.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:44.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:44.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:44.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:44 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:26:44.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:44 smithi106 ceph-mon[16603]: osdmap e127: 12 total, 12 up, 12 in 2024-08-30T23:26:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:44 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:26:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:44 smithi136 ceph-mon[20207]: osdmap e127: 12 total, 12 up, 12 in 2024-08-30T23:26:44.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:44 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:26:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:44 smithi173 ceph-mon[20554]: osdmap e127: 12 total, 12 up, 12 in 2024-08-30T23:26:44.854 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:44.889 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:44.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:44.890 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:44.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:44.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:44.950 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:44.950 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:44.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:44.978 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:44.978 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:44.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:44.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:45.006 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:45.006 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:45.036 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:45.037 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:45.063 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:45.063 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:45.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:45.088 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:45.088 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.112 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:45.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:45.125 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:45.125 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:45.153 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:45.153 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:45.165 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 62307 62291 82.8271 78.9375 0.00268211 0.0120645 2024-08-30T23:26:45.183 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:45.183 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:45.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:26:45.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:45.214 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:45.214 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:45.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:45.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.369 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:45.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:45.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:26:45.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:45.497 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:45.499 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:45.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3245569851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:45 smithi106 ceph-mon[16603]: pgmap v354: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:26:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:45.626 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:45.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:45.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:45.759 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3245569851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:45.759 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:45 smithi136 ceph-mon[20207]: pgmap v354: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:26:45.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3245569851' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:45.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:45 smithi173 ceph-mon[20554]: pgmap v354: 17 pgs: 17 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:26:45.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:45.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:45.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:45.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:46.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:46.003 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:26:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:26:46.004 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:46.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:46.124 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:46.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 62983 62967 81.9817 42.25 0.127931 0.012185 2024-08-30T23:26:46.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:46.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:46.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:46.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:46.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:46.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.509 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:46.522 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.11 to 1 (10000) 2024-08-30T23:26:46.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:46.645 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:46.709 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1953525293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:26:46.710 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:46 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:26:46.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:46.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:26:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:46.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1953525293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:26:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:46 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:26:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1953525293' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:26:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:46 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:26:46.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:46.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:46.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:47.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:47.024 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:47.053 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:47.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:47.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:47.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 15 64155 64140 81.8046 73.3125 0.00594457 0.0122181 2024-08-30T23:26:47.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:47.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:47.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:47.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:47.411 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:47.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:47.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:47.554 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:47.554 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:26:47.554 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:26:47.554 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:26:47.554 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:47.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:47.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:47.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:47 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:26:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:47 smithi136 ceph-mon[20207]: osdmap e128: 12 total, 12 up, 12 in 2024-08-30T23:26:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:47 smithi136 ceph-mon[20207]: pgmap v356: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-30T23:26:47.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:47 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:26:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:47 smithi173 ceph-mon[20554]: osdmap e128: 12 total, 12 up, 12 in 2024-08-30T23:26:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:47 smithi173 ceph-mon[20554]: pgmap v356: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-30T23:26:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:47 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:26:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:47 smithi106 ceph-mon[16603]: osdmap e128: 12 total, 12 up, 12 in 2024-08-30T23:26:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:47 smithi106 ceph-mon[16603]: pgmap v356: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-30T23:26:47.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:47.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:47.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:47.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:48.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:48.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:48.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:48.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 65547 65531 81.9071 86.9375 0.0232817 0.0122035 2024-08-30T23:26:48.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:48.246 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T23:26:48.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:26:48.287 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:48.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:48.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:26:48.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:48.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:48.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:48.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:48.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:48.579 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:26:48.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:26:48.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:48.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:48.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:48.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:48.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:48.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:48.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:48.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:48.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:49.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:49.077 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:693 2.8 2:1191ab72:::benchmark_data_smithi173_47195_object692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:49.129 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.392302+0000", 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.734682057999997, 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.139937379, 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 693 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392302+0000", 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392302+0000", 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392306+0000", 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7859999999999998e-06 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392325+0000", 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392337+0000", 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1868999999999999e-05 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392344+0000", 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3550000000000001e-06 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392385+0000", 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1292999999999999e-05 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392460+0000", 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5288000000000002e-05 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392543+0000", 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2918999999999997e-05 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.392866+0000", 2024-08-30T23:26:49.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032305399999999997 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.393274+0000", 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040788 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532166+0000", 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13889214499999999 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532211+0000", 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4577999999999999e-05 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532240+0000", 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8799999999999999e-05 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.137 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:693 2.8 e104/97 ondisk, result = 0)", 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.393241+0000", 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.733743570999998, 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.139004247, 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.393241+0000", 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.393241+0000", 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.393241+0000", 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2099999999999997e-07 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.393244+0000", 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.157e-06 2024-08-30T23:26:49.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532107+0000", 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13886296000000001 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532112+0000", 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.42e-06 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532154+0000", 2024-08-30T23:26:49.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2358999999999999e-05 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532163+0000", 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.602e-06 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.532245+0000", 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2227999999999999e-05 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:700 2.8 2:1a916375:::benchmark_data_smithi173_47195_object699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.396856+0000", 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.730128028000003, 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13720406900000001, 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.142 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 700 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396856+0000", 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396856+0000", 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396857+0000", 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0999999999999997e-07 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396883+0000", 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5777000000000001e-05 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396896+0000", 2024-08-30T23:26:49.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2693e-05 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396901+0000", 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9799999999999998e-06 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.396938+0000", 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6974999999999998e-05 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397018+0000", 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0584e-05 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397096+0000", 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7943999999999998e-05 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397447+0000", 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000351233 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397947+0000", 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00049945100000000004 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.533964+0000", 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13601724700000001 2024-08-30T23:26:49.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.534023+0000", 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8811000000000003e-05 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.534060+0000", 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7564000000000002e-05 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:700 2.8 e104/97 ondisk, result = 0)", 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.397726+0000", 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.729258778000002, 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13633935999999999, 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.149 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397726+0000", 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397726+0000", 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397726+0000", 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7400000000000003e-07 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.397733+0000", 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3160000000000002e-06 2024-08-30T23:26:49.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.533907+0000", 2024-08-30T23:26:49.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13617442399999999 2024-08-30T23:26:49.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.533911+0000", 2024-08-30T23:26:49.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.42e-06 2024-08-30T23:26:49.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.533951+0000", 2024-08-30T23:26:49.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9601000000000003e-05 2024-08-30T23:26:49.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.533958+0000", 2024-08-30T23:26:49.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4379999999999998e-06 2024-08-30T23:26:49.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.534065+0000", 2024-08-30T23:26:49.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106487 2024-08-30T23:26:49.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.157 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.158 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:827 2.8 2:139d510f:::benchmark_data_smithi173_47195_object826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:49.158 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.564210+0000", 2024-08-30T23:26:49.158 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.562774519000001, 2024-08-30T23:26:49.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15882453499999999, 2024-08-30T23:26:49.158 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.158 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.159 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.159 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 827 2024-08-30T23:26:49.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.160 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564210+0000", 2024-08-30T23:26:49.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564210+0000", 2024-08-30T23:26:49.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564210+0000", 2024-08-30T23:26:49.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7999999999999995e-07 2024-08-30T23:26:49.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564282+0000", 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1501999999999994e-05 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564290+0000", 2024-08-30T23:26:49.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4440000000000008e-06 2024-08-30T23:26:49.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564314+0000", 2024-08-30T23:26:49.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3728999999999999e-05 2024-08-30T23:26:49.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564345+0000", 2024-08-30T23:26:49.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1378999999999999e-05 2024-08-30T23:26:49.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564412+0000", 2024-08-30T23:26:49.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6870000000000002e-05 2024-08-30T23:26:49.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564489+0000", 2024-08-30T23:26:49.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6408000000000004e-05 2024-08-30T23:26:49.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.564798+0000", 2024-08-30T23:26:49.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000309573 2024-08-30T23:26:49.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.565125+0000", 2024-08-30T23:26:49.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032646200000000002 2024-08-30T23:26:49.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.722925+0000", 2024-08-30T23:26:49.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15780018000000001 2024-08-30T23:26:49.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.722983+0000", 2024-08-30T23:26:49.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7694e-05 2024-08-30T23:26:49.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.723034+0000", 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1714e-05 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:827 2.8 e104/97 ondisk, result = 0)", 2024-08-30T23:26:49.173 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.565089+0000", 2024-08-30T23:26:49.173 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.561895395999997, 2024-08-30T23:26:49.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15795288199999999, 2024-08-30T23:26:49.173 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.173 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.173 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:49.175 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 66524 66508 81.4983 61.0625 0.0032585 0.0122627 2024-08-30T23:26:49.176 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.565089+0000", 2024-08-30T23:26:49.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.565089+0000", 2024-08-30T23:26:49.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.565089+0000", 2024-08-30T23:26:49.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6100000000000001e-07 2024-08-30T23:26:49.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.565092+0000", 2024-08-30T23:26:49.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.649e-06 2024-08-30T23:26:49.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.722840+0000", 2024-08-30T23:26:49.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15774823399999999 2024-08-30T23:26:49.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.722846+0000", 2024-08-30T23:26:49.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5489999999999997e-06 2024-08-30T23:26:49.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.722911+0000", 2024-08-30T23:26:49.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4992000000000004e-05 2024-08-30T23:26:49.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.722919+0000", 2024-08-30T23:26:49.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8690000000000004e-06 2024-08-30T23:26:49.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.723042+0000", 2024-08-30T23:26:49.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012312799999999999 2024-08-30T23:26:49.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.185 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.185 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:832 2.8 2:1abf3619:::benchmark_data_smithi173_47195_object831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:26:49.185 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.570171+0000", 2024-08-30T23:26:49.186 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.556812956000002, 2024-08-30T23:26:49.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16704251000000001, 2024-08-30T23:26:49.186 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.186 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.186 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.187 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.187 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.187 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 832 2024-08-30T23:26:49.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.187 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570171+0000", 2024-08-30T23:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570171+0000", 2024-08-30T23:26:49.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570172+0000", 2024-08-30T23:26:49.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0299999999999999e-07 2024-08-30T23:26:49.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570199+0000", 2024-08-30T23:26:49.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6964000000000001e-05 2024-08-30T23:26:49.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570214+0000", 2024-08-30T23:26:49.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5019e-05 2024-08-30T23:26:49.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570223+0000", 2024-08-30T23:26:49.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9099999999999994e-06 2024-08-30T23:26:49.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570276+0000", 2024-08-30T23:26:49.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2710000000000002e-05 2024-08-30T23:26:49.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570351+0000", 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4937000000000004e-05 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570428+0000", 2024-08-30T23:26:49.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7043000000000001e-05 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.570740+0000", 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000312059 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.571178+0000", 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000438856 2024-08-30T23:26:49.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737142+0000", 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.165963262 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737176+0000", 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4236999999999998e-05 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737214+0000", 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7910000000000001e-05 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:832 2.8 e104/97 ondisk, result = 0)", 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.571161+0000", 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.555822947000003, 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166057221, 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.197 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.571161+0000", 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.571161+0000", 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.571162+0000", 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.571165+0000", 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1599999999999998e-06 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737094+0000", 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16592834200000001 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737099+0000", 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5570000000000003e-06 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737132+0000", 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3238999999999998e-05 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737138+0000", 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1299999999999998e-06 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.737219+0000", 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0179999999999995e-05 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:22831 2.8 2:1e1fb9f3:::benchmark_data_smithi173_47195_object22830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:15.164952+0000", 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.962032008999998, 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14759620700000001, 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.201 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22831 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.164952+0000", 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.164952+0000", 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.164953+0000", 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165018+0000", 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-05 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165027+0000", 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165034+0000", 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5200000000000003e-06 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165072+0000", 2024-08-30T23:26:49.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8599000000000001e-05 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165146+0000", 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3752999999999997e-05 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165225+0000", 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9327999999999997e-05 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.165763+0000", 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053782000000000005 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.166199+0000", 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00043563400000000001 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.312492+0000", 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14629289300000001 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.312523+0000", 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1331000000000002e-05 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.312549+0000", 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5508999999999999e-05 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:22842 2.8 2:14312ee4:::benchmark_data_smithi173_47195_object22841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:15.182568+0000", 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.94441621, 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.146358867, 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.208 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22842 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182568+0000", 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182568+0000", 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182574+0000", 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1829999999999999e-06 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182620+0000", 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5479000000000003e-05 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182649+0000", 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9203000000000001e-05 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182673+0000", 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3546000000000001e-05 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182826+0000", 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015389000000000001 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.182918+0000", 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1732999999999997e-05 2024-08-30T23:26:49.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.183010+0000", 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1576999999999994e-05 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.183683+0000", 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00067326699999999996 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.184124+0000", 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044117000000000001 2024-08-30T23:26:49.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.328865+0000", 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14474071299999999 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.328900+0000", 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4647999999999998e-05 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.328927+0000", 2024-08-30T23:26:49.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7458e-05 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:37682 2.9 e117/97 2:9f18148c:::benchmark_data_smithi173_47195_object37681:head v 117'2404, mlcod=117'2401)", 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:26.022068+0000", 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.104915962, 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.107939677, 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.022068+0000", 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.022068+0000", 2024-08-30T23:26:49.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.022069+0000", 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0200000000000001e-07 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.022106+0000", 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7209000000000002e-05 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.128942+0000", 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10683580299999999 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.128947+0000", 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5909999999999998e-06 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.128985+0000", 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8476000000000002e-05 2024-08-30T23:26:49.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.129005+0000", 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0126e-05 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.129960+0000", 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095476000000000003 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:26.130008+0000", 2024-08-30T23:26:49.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.791e-05 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:43074 2.8 2:1e2a85c5:::benchmark_data_smithi173_47195_object43073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:29.835681+0000", 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.291303758000002, 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14578860699999999, 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 43074 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835681+0000", 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835681+0000", 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835682+0000", 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6190000000000001e-06 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835722+0000", 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0077999999999998e-05 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835737+0000", 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5203e-05 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835744+0000", 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8079999999999999e-06 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835801+0000", 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6465999999999998e-05 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.835933+0000", 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013209699999999999 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836059+0000", 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012640399999999999 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.838209+0000", 2024-08-30T23:26:49.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002149593 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.838218+0000", 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6750000000000008e-06 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.981368+0000", 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14315051100000001 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.981425+0000", 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6786999999999997e-05 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.981469+0000", 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4366000000000002e-05 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:43075 2.8 2:1bed56d6:::benchmark_data_smithi173_47195_object43074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:29.836655+0000", 2024-08-30T23:26:49.226 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.290329277000001, 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14743125100000001, 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 43075 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836655+0000", 2024-08-30T23:26:49.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836655+0000", 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836656+0000", 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4000000000000001e-07 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836717+0000", 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1223999999999995e-05 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836728+0000", 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0540000000000001e-05 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.836733+0000", 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0819999999999998e-06 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837268+0000", 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053528099999999997 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837369+0000", 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101067 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837460+0000", 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1472999999999996e-05 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.839106+0000", 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001645131 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.839140+0000", 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4057000000000003e-05 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.983967+0000", 2024-08-30T23:26:49.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14482743100000001 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.984040+0000", 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3022000000000007e-05 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.984086+0000", 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6202999999999998e-05 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:43076 2.8 2:176edc68:::benchmark_data_smithi173_47195_object43075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:29.837516+0000", 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.289468420999999, 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15071369400000001, 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 43076 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837516+0000", 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837516+0000", 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837517+0000", 2024-08-30T23:26:49.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3099999999999997e-07 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837593+0000", 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6907000000000001e-05 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837602+0000", 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.884e-06 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.837611+0000", 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4500000000000004e-06 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.838242+0000", 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00063125500000000005 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.838329+0000", 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6996000000000005e-05 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.838405+0000", 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6020999999999996e-05 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.839115+0000", 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00070989800000000004 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.839620+0000", 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050513499999999998 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.988158+0000", 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14853777900000001 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.988190+0000", 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2449000000000001e-05 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:29.988230+0000", 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9289000000000003e-05 2024-08-30T23:26:49.238 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:49992 2.0 2:0e70673a:::benchmark_data_smithi173_47195_object49991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.872087774000001, 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20138367700000001, 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49992 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8400000000000004e-07 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254943+0000", 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5250999999999998e-05 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254950+0000", 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9349999999999994e-06 2024-08-30T23:26:49.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254955+0000", 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2400000000000001e-06 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255011+0000", 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6350000000000001e-05 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255085+0000", 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3522000000000005e-05 2024-08-30T23:26:49.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255182+0000", 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7161e-05 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256087+0000", 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090517600000000005 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256836+0000", 2024-08-30T23:26:49.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074912999999999998 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456148+0000", 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19931177 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456236+0000", 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8363999999999996e-05 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456280+0000", 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4094000000000002e-05 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:49992 2.0 e122/97 ondisk, result = 0)", 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.256898+0000", 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.870086264999999, 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19938884900000001, 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:49.245 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256898+0000", 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256898+0000", 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256899+0000", 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2e-07 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256901+0000", 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4930000000000001e-06 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456045+0000", 2024-08-30T23:26:49.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19914447900000001 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456054+0000", 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.952e-06 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456132+0000", 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7441999999999995e-05 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456144+0000", 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1806e-05 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456287+0000", 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014325699999999999 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50008 2.0 2:0021b868:::benchmark_data_smithi173_47195_object50007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:26:49.249 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.282842+0000", 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.844142842, 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181126699, 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50008 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.282842+0000", 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.282842+0000", 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.282843+0000", 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.578e-06 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.282897+0000", 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3887000000000001e-05 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.282910+0000", 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2938999999999999e-05 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.282916+0000", 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6910000000000002e-06 2024-08-30T23:26:49.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.283003+0000", 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7268999999999994e-05 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.283086+0000", 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3079999999999997e-05 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.283163+0000", 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6748000000000004e-05 2024-08-30T23:26:49.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.284585+0000", 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00142205 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.284743+0000", 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015819099999999999 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463902+0000", 2024-08-30T23:26:49.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179159173 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463937+0000", 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4496000000000003e-05 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463968+0000", 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1597000000000003e-05 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:50008 2.0 e122/97 ondisk, result = 0)", 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.284772+0000", 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.842211896, 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17920064199999999, 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.284772+0000", 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.284772+0000", 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.284773+0000", 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.58e-07 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.284775+0000", 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0779999999999998e-06 2024-08-30T23:26:49.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463862+0000", 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17908739000000001 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463865+0000", 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1829999999999998e-06 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463896+0000", 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463899+0000", 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.083e-06 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.463973+0000", 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4022000000000004e-05 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:56070 2.0 2:06759afd:::benchmark_data_smithi173_47195_object56069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:39.790591+0000", 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.3363928870000006, 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200068458, 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 56070 2024-08-30T23:26:49.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790591+0000", 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790591+0000", 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790592+0000", 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790618+0000", 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5905999999999999e-05 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790631+0000", 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2559e-05 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790636+0000", 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5570000000000003e-06 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790680+0000", 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3238000000000002e-05 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790776+0000", 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6687000000000004e-05 2024-08-30T23:26:49.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790865+0000", 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8517000000000006e-05 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.792279+0000", 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00141439 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.792724+0000", 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044493199999999999 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.990580+0000", 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19785618899999999 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.990617+0000", 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6553000000000001e-05 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.990660+0000", 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.295e-05 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:65961 2.9 e128/97 2:9c9d097b:::benchmark_data_smithi173_47195_object65960:head v 128'4210, mlcod=128'4205)", 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:48.607337+0000", 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.51964736600000005, 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10942094400000001, 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:26:49.266 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.607337+0000", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.607337+0000", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.607338+0000", 2024-08-30T23:26:49.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1900000000000003e-07 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.607388+0000", 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.083e-05 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.715119+0000", 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10773099899999999 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:26:49.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.715124+0000", 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7729999999999999e-06 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.715168+0000", 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3436999999999998e-05 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.715182+0000", 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4473e-05 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.716728+0000", 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001546115 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:48.716758+0000", 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9798000000000002e-05 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:26:49.270 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:26:49.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:49.271 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:26:49.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/414230052' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:49.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:49 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/414230052' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:49 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:49.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:49.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:49.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:49.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:49.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:49.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/414230052' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:49.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:49 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:26:49.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:49.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:49.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:49.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:49.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:49.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:49.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:50.006 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:26:50.007 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:26:50.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:50.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:50.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 67657 67641 81.2927 70.8125 0.00251957 0.0122891 2024-08-30T23:26:50.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:50.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:50.222 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:50.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:26:50.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:50.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:50.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:50 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:50.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:50 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:26:50.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:50 smithi106 ceph-mon[16603]: osdmap e129: 12 total, 12 up, 12 in 2024-08-30T23:26:50.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:50 smithi106 ceph-mon[16603]: pgmap v358: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-30T23:26:50.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:50.454 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:50.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:50 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:50 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:26:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:50 smithi136 ceph-mon[20207]: osdmap e129: 12 total, 12 up, 12 in 2024-08-30T23:26:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:50 smithi136 ceph-mon[20207]: pgmap v358: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-30T23:26:50.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:50.564 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:26:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:50 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:50 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:26:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:50 smithi173 ceph-mon[20554]: osdmap e129: 12 total, 12 up, 12 in 2024-08-30T23:26:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:50 smithi173 ceph-mon[20554]: pgmap v358: 17 pgs: 17 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-30T23:26:50.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:50.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:50.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:50.717 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-08-30T23:26:49.351206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4197253664,"num_objects":64038,"num_object_clones":0,"num_object_copies":192114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64038,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":64159,"num_write_kb":4100454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64128,"ondisk_log_size":64128,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12759728,"kb_used_data":12200916,"kb_used_omap":18,"kb_used_meta":558573,"kb_avail":1112182096,"statfs":{"total":1151940427776,"available":1138874466304,"internally_reserved":0,"allocated":12493737984,"data_stored":12487776956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":571979065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":425525248,"num_objects":6493,"num_object_clones":0,"num_object_copies":19479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6493,"num_write_kb":415552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001146"},"pg_stats":[{"pgid":"2.f","version":"127'3926","reported_seq":3997,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-08-30T23:26:45.621917+0000","last_change":"2024-08-30T23:25:50.451288+0000","last_active":"2024-08-30T23:26:45.621917+0000","last_peered":"2024-08-30T23:26:45.621917+0000","last_clean":"2024-08-30T23:26:45.621917+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:26:45.621917+0000","last_undegraded":"2024-08-30T23:26:45.621917+0000","last_fullsized":"2024-08-30T23:26:45.621917+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:57.560523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257294336,"num_objects":3926,"num_object_clones":0,"num_object_copies":11778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":251264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"128'4064","reported_seq":4137,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.243477+0000","last_change":"2024-08-30T23:25:50.474312+0000","last_active":"2024-08-30T23:26:48.243477+0000","last_peered":"2024-08-30T23:26:48.243477+0000","last_clean":"2024-08-30T23:26:48.243477+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:26:48.243477+0000","last_undegraded":"2024-08-30T23:26:48.243477+0000","last_fullsized":"2024-08-30T23:26:48.243477+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:18.608667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":260096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"127'4004","reported_seq":4074,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-08-30T23:26:45.009901+0000","last_change":"2024-08-30T23:25:50.474455+0000","last_active":"2024-08-30T23:26:45.009901+0000","last_peered":"2024-08-30T23:26:45.009901+0000","last_clean":"2024-08-30T23:26:45.009901+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:26:45.009901+0000","last_undegraded":"2024-08-30T23:26:45.009901+0000","last_fullsized":"2024-08-30T23:26:45.009901+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:16.362364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262406144,"num_objects":4004,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4004,"num_write_kb":256256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"127'3888","reported_seq":3958,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-08-30T23:26:45.890249+0000","last_change":"2024-08-30T23:25:50.532163+0000","last_active":"2024-08-30T23:26:45.890249+0000","last_peered":"2024-08-30T23:26:45.890249+0000","last_clean":"2024-08-30T23:26:45.890249+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:26:45.890249+0000","last_undegraded":"2024-08-30T23:26:45.890249+0000","last_fullsized":"2024-08-30T23:26:45.890249+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:51.982184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254803968,"num_objects":3888,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":248832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":206,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:46.521830+0000","last_change":"2024-08-30T23:26:04.139978+0000","last_active":"2024-08-30T23:26:46.521830+0000","last_peered":"2024-08-30T23:26:46.521830+0000","last_clean":"2024-08-30T23:26:46.521830+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:26:46.521830+0000","last_undegraded":"2024-08-30T23:26:46.521830+0000","last_fullsized":"2024-08-30T23:26:46.521830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:26:04.139930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:26:04.139930+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:21.353939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"128'4193","reported_seq":4265,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.562879+0000","last_change":"2024-08-30T23:25:50.452633+0000","last_active":"2024-08-30T23:26:48.562879+0000","last_peered":"2024-08-30T23:26:48.562879+0000","last_clean":"2024-08-30T23:26:48.562879+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:26:48.562879+0000","last_undegraded":"2024-08-30T23:26:48.562879+0000","last_fullsized":"2024-08-30T23:26:48.562879+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:56.515696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":268352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"128'3977","reported_seq":4050,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.267593+0000","last_change":"2024-08-30T23:25:50.451520+0000","last_active":"2024-08-30T23:26:47.267593+0000","last_peered":"2024-08-30T23:26:47.267593+0000","last_clean":"2024-08-30T23:26:47.267593+0000","last_became_active":"2024-08-30T23:25:50.451193+0000","last_became_peered":"2024-08-30T23:25:50.451193+0000","last_unstale":"2024-08-30T23:26:47.267593+0000","last_undegraded":"2024-08-30T23:26:47.267593+0000","last_fullsized":"2024-08-30T23:26:47.267593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:55.292492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260636672,"num_objects":3977,"num_object_clones":0,"num_object_copies":11931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":254528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"128'4128","reported_seq":4201,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.286262+0000","last_change":"2024-08-30T23:25:50.531742+0000","last_active":"2024-08-30T23:26:48.286262+0000","last_peered":"2024-08-30T23:26:48.286262+0000","last_clean":"2024-08-30T23:26:48.286262+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:26:48.286262+0000","last_undegraded":"2024-08-30T23:26:48.286262+0000","last_fullsized":"2024-08-30T23:26:48.286262+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:53.584554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270532608,"num_objects":4128,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":264192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"128'4065","reported_seq":4138,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.519469+0000","last_change":"2024-08-30T23:25:50.451424+0000","last_active":"2024-08-30T23:26:48.519469+0000","last_peered":"2024-08-30T23:26:48.519469+0000","last_clean":"2024-08-30T23:26:48.519469+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:26:48.519469+0000","last_undegraded":"2024-08-30T23:26:48.519469+0000","last_fullsized":"2024-08-30T23:26:48.519469+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:19.191011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":260160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"128'4026","reported_seq":4098,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.357493+0000","last_change":"2024-08-30T23:25:50.532259+0000","last_active":"2024-08-30T23:26:47.357493+0000","last_peered":"2024-08-30T23:26:47.357493+0000","last_clean":"2024-08-30T23:26:47.357493+0000","last_became_active":"2024-08-30T23:25:50.531665+0000","last_became_peered":"2024-08-30T23:25:50.531665+0000","last_unstale":"2024-08-30T23:26:47.357493+0000","last_undegraded":"2024-08-30T23:26:47.357493+0000","last_fullsized":"2024-08-30T23:26:47.357493+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:05.834714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4026,"num_write_kb":257664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"127'3906","reported_seq":3975,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-08-30T23:26:45.616291+0000","last_change":"2024-08-30T23:25:50.532054+0000","last_active":"2024-08-30T23:26:45.616291+0000","last_peered":"2024-08-30T23:26:45.616291+0000","last_clean":"2024-08-30T23:26:45.616291+0000","last_became_active":"2024-08-30T23:25:50.531670+0000","last_became_peered":"2024-08-30T23:25:50.531670+0000","last_unstale":"2024-08-30T23:26:45.616291+0000","last_undegraded":"2024-08-30T23:26:45.616291+0000","last_fullsized":"2024-08-30T23:26:45.616291+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3906,"log_dups_size":0,"ondisk_log_size":3906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:06.951678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255983616,"num_objects":3906,"num_object_clones":0,"num_object_copies":11718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3906,"num_write_kb":249984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"128'4032","reported_seq":4103,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.321638+0000","last_change":"2024-08-30T23:25:50.474645+0000","last_active":"2024-08-30T23:26:47.321638+0000","last_peered":"2024-08-30T23:26:47.321638+0000","last_clean":"2024-08-30T23:26:47.321638+0000","last_became_active":"2024-08-30T23:25:50.474315+0000","last_became_peered":"2024-08-30T23:25:50.474315+0000","last_unstale":"2024-08-30T23:26:47.321638+0000","last_undegraded":"2024-08-30T23:26:47.321638+0000","last_fullsized":"2024-08-30T23:26:47.321638+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:05.978771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"128'3970","reported_seq":4042,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:46.792222+0000","last_change":"2024-08-30T23:25:50.451369+0000","last_active":"2024-08-30T23:26:46.792222+0000","last_peered":"2024-08-30T23:26:46.792222+0000","last_clean":"2024-08-30T23:26:46.792222+0000","last_became_active":"2024-08-30T23:25:50.451015+0000","last_became_peered":"2024-08-30T23:25:50.451015+0000","last_unstale":"2024-08-30T23:26:46.792222+0000","last_undegraded":"2024-08-30T23:26:46.792222+0000","last_fullsized":"2024-08-30T23:26:46.792222+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:11.902839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260177920,"num_objects":3970,"num_object_clones":0,"num_object_copies":11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":254080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"128'4026","reported_seq":4099,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.287821+0000","last_change":"2024-08-30T23:25:50.531975+0000","last_active":"2024-08-30T23:26:47.287821+0000","last_peered":"2024-08-30T23:26:47.287821+0000","last_clean":"2024-08-30T23:26:47.287821+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:26:47.287821+0000","last_undegraded":"2024-08-30T23:26:47.287821+0000","last_fullsized":"2024-08-30T23:26:47.287821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:52.802573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4026,"num_write_kb":257664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"128'4090","reported_seq":4161,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.348673+0000","last_change":"2024-08-30T23:25:50.474563+0000","last_active":"2024-08-30T23:26:47.348673+0000","last_peered":"2024-08-30T23:26:47.348673+0000","last_clean":"2024-08-30T23:26:47.348673+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:26:47.348673+0000","last_undegraded":"2024-08-30T23:26:47.348673+0000","last_fullsized":"2024-08-30T23:26:47.348673+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:17.998479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268042240,"num_objects":4090,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":261760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"127'3849","reported_seq":3918,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-08-30T23:26:45.870050+0000","last_change":"2024-08-30T23:25:50.474244+0000","last_active":"2024-08-30T23:26:45.870050+0000","last_peered":"2024-08-30T23:26:45.870050+0000","last_clean":"2024-08-30T23:26:45.870050+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:26:45.870050+0000","last_undegraded":"2024-08-30T23:26:45.870050+0000","last_fullsized":"2024-08-30T23:26:45.870050+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:57.627855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252248064,"num_objects":3849,"num_object_clones":0,"num_object_copies":11547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3849,"num_write_kb":246336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"127'3892","reported_seq":3962,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-08-30T23:26:45.599574+0000","last_change":"2024-08-30T23:25:50.451211+0000","last_active":"2024-08-30T23:26:45.599574+0000","last_peered":"2024-08-30T23:26:45.599574+0000","last_clean":"2024-08-30T23:26:45.599574+0000","last_became_active":"2024-08-30T23:25:50.450857+0000","last_became_peered":"2024-08-30T23:25:50.450857+0000","last_unstale":"2024-08-30T23:26:45.599574+0000","last_undegraded":"2024-08-30T23:26:45.599574+0000","last_fullsized":"2024-08-30T23:26:45.599574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3892,"log_dups_size":0,"ondisk_log_size":3892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:59.474362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255066112,"num_objects":3892,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":249088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4196663296,"num_objects":64036,"num_object_clones":0,"num_object_copies":192108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64036,"num_write_kb":4098304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12592218112,"data_stored":12592218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64036,"ondisk_log_size":64036,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187483,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790552,"kb_used_data":746888,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":92954600,"statfs":{"total":95995035648,"available":95185510400,"internally_reserved":0,"allocated":764813312,"data_stored":764315789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":44693956},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449121,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793616,"kb_used_data":754112,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92951536,"statfs":{"total":95995035648,"available":95182372864,"internally_reserved":0,"allocated":772210688,"data_stored":771716737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529172,"kb_used_data":1471616,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92215980,"statfs":{"total":95995035648,"available":94429163520,"internally_reserved":0,"allocated":1506934784,"data_stored":1506440833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":58915264},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972395,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053712,"kb_used_data":996160,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92691440,"statfs":{"total":95995035648,"available":94916034560,"internally_reserved":0,"allocated":1020067840,"data_stored":1019573889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234030,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329624,"kb_used_data":1272072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92415528,"statfs":{"total":95995035648,"available":94633500672,"internally_reserved":0,"allocated":1302601728,"data_stored":1302104225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":527512,"kb_used_data":500488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93217640,"statfs":{"total":95995035648,"available":95454863360,"internally_reserved":0,"allocated":512499712,"data_stored":512002209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550176,"kb_used_data":523148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93194976,"statfs":{"total":95995035648,"available":95431655424,"internally_reserved":0,"allocated":535703552,"data_stored":535205013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779604,"kb_used_data":740096,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92965548,"statfs":{"total":95995035648,"available":95196721152,"internally_reserved":0,"allocated":757858304,"data_stored":757364353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625056,"kb_used_data":1567500,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92120096,"statfs":{"total":95995035648,"available":94330978304,"internally_reserved":0,"allocated":1605120000,"data_stored":1604621461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1092768,"kb_used_data":1047692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92652384,"statfs":{"total":95995035648,"available":94876041216,"internally_reserved":0,"allocated":1072836608,"data_stored":1072338069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339104,"kb_used_data":1277388,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":92406048,"statfs":{"total":95995035648,"available":94623793152,"internally_reserved":0,"allocated":1308045312,"data_stored":1307546773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348832,"kb_used_data":1303756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92396320,"statfs":{"total":95995035648,"available":94613831680,"internally_reserved":0,"allocated":1335046144,"data_stored":1334547605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1339949056,"data_stored":1339949056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1317535744,"data_stored":1317535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1079705600,"data_stored":1079705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1622867968,"data_stored":1622867968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":762511360,"data_stored":762511360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":541196288,"data_stored":541196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":518651904,"data_stored":518651904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1322647552,"data_stored":1322647552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1025310720,"data_stored":1025310720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1527250944,"data_stored":1527250944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":771424256,"data_stored":771424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":763166720,"data_stored":763166720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:26:50.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:26:50.812 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:50.843 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:50.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:50.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:51.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:51.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:51.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.143 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:51.146 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:26:51.146 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:26:51.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:68785 2.b 2:deeef2af:::benchmark_data_smithi173_47195_object68784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:51.134388+0000", 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.011807082, 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011834333000000001, 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 68785 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:26:51.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134388+0000", 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134388+0000", 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134389+0000", 2024-08-30T23:26:51.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3820000000000001e-06 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134412+0000", 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2113999999999999e-05 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134426+0000", 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4191e-05 2024-08-30T23:26:51.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134431+0000", 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2610000000000001e-06 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134494+0000", 2024-08-30T23:26:51.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2520000000000004e-05 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134639+0000", 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000145869 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.134792+0000", 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015207400000000001 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:51.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.137721+0000", 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029299730000000002 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.137868+0000", 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014625099999999999 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:68786 2.5 2:a0fa1065:::benchmark_data_smithi173_47195_object68785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:26:51.152 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:51.135383+0000", 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.010812145, 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010907339, 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 68786 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:26:51.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135383+0000", 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135383+0000", 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135384+0000", 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.068e-06 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135409+0000", 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5165999999999999e-05 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135425+0000", 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5330000000000001e-05 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135433+0000", 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.994e-06 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135492+0000", 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9599999999999999e-05 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135738+0000", 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00024628399999999999 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.135908+0000", 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016971900000000001 2024-08-30T23:26:51.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.138131+0000", 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022224010000000002 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:51.138917+0000", 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078649199999999996 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:26:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 2 2024-08-30T23:26:51.158 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:26:51.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 68835 68819 81.1479 73.625 0.00355006 0.0123184 2024-08-30T23:26:51.235 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:51.235 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:51.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:51.250 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:51.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:51.280 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:51.280 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:51.317 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:51 smithi173 ceph-mon[20554]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:26:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3091175312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:51.354 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:51.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:51.386 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:51.387 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:51.402 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:51.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:51.418 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:51.447 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:51.447 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:51.478 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:51.479 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:51.479 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:51 smithi106 ceph-mon[16603]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:26:51.479 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3091175312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:51.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:51.515 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:51.515 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:51.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:51 smithi136 ceph-mon[20207]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:26:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3091175312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:51.544 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:51.544 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:51.567 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:51.567 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:51.591 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:51.591 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:51.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:51.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:51.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:51.794 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:51.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:51.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:51.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:51.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:26:52.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:52.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:26:52.149 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:26:52.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:52.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 70032 70016 81.0305 74.8125 0.00528369 0.0123338 2024-08-30T23:26:52.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:52.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:52.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:52.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:52 smithi106 ceph-mon[16603]: pgmap v359: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:26:52.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:52.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:52.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:52.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:52 smithi136 ceph-mon[20207]: pgmap v359: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:26:52.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:52.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:52 smithi173 ceph-mon[20554]: pgmap v359: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:26:52.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:52.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:52.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:52.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:52.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:26:52.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:52.944 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:52.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:53.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.077 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:53.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:53.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 71434 71418 81.1502 87.625 0.00449559 0.0123151 2024-08-30T23:26:53.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:53.204 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:53.206 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:26:53.306 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:53.333 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:53.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:53.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:53.466 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:53 smithi173 ceph-mon[20554]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:26:53.518 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:53 smithi106 ceph-mon[16603]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:26:53.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:53 smithi136 ceph-mon[20207]: from='client.24622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:26:53.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:53.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:53.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:53.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:53.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:53.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:53.864 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:53.884 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-08-30T23:26:51.351562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4331668000,"num_objects":66089,"num_object_clones":0,"num_object_copies":198267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66089,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":66210,"num_write_kb":4231718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66179,"ondisk_log_size":66179,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13123552,"kb_used_data":12618816,"kb_used_omap":18,"kb_used_meta":504493,"kb_avail":1111818272,"statfs":{"total":1151940427776,"available":1138501910528,"internally_reserved":0,"allocated":12921667584,"data_stored":12915665232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":516601145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,1,0,4,1,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":559939584,"num_objects":8544,"num_object_clones":0,"num_object_copies":25632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8544,"num_write_kb":546816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.849654"},"pg_stats":[{"pgid":"2.f","version":"129'4252","reported_seq":4327,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T23:26:50.608352+0000","last_change":"2024-08-30T23:25:50.451288+0000","last_active":"2024-08-30T23:26:50.608352+0000","last_peered":"2024-08-30T23:26:50.608352+0000","last_clean":"2024-08-30T23:26:50.608352+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:26:50.608352+0000","last_undegraded":"2024-08-30T23:26:50.608352+0000","last_fullsized":"2024-08-30T23:26:50.608352+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:57.560523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278659072,"num_objects":4252,"num_object_clones":0,"num_object_copies":12756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4252,"num_write_kb":272128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"128'4064","reported_seq":4137,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.243477+0000","last_change":"2024-08-30T23:25:50.474312+0000","last_active":"2024-08-30T23:26:48.243477+0000","last_peered":"2024-08-30T23:26:48.243477+0000","last_clean":"2024-08-30T23:26:48.243477+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:26:48.243477+0000","last_undegraded":"2024-08-30T23:26:48.243477+0000","last_fullsized":"2024-08-30T23:26:48.243477+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:18.608667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":260096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"129'4316","reported_seq":4389,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T23:26:50.008558+0000","last_change":"2024-08-30T23:25:50.474455+0000","last_active":"2024-08-30T23:26:50.008558+0000","last_peered":"2024-08-30T23:26:50.008558+0000","last_clean":"2024-08-30T23:26:50.008558+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:26:50.008558+0000","last_undegraded":"2024-08-30T23:26:50.008558+0000","last_fullsized":"2024-08-30T23:26:50.008558+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:16.362364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282853376,"num_objects":4316,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":276224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"129'4264","reported_seq":4337,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T23:26:51.030950+0000","last_change":"2024-08-30T23:25:50.532163+0000","last_active":"2024-08-30T23:26:51.030950+0000","last_peered":"2024-08-30T23:26:51.030950+0000","last_clean":"2024-08-30T23:26:51.030950+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:26:51.030950+0000","last_undegraded":"2024-08-30T23:26:51.030950+0000","last_fullsized":"2024-08-30T23:26:51.030950+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:51.982184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279445504,"num_objects":4264,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4264,"num_write_kb":272896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":206,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:46.521830+0000","last_change":"2024-08-30T23:26:04.139978+0000","last_active":"2024-08-30T23:26:46.521830+0000","last_peered":"2024-08-30T23:26:46.521830+0000","last_clean":"2024-08-30T23:26:46.521830+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:26:46.521830+0000","last_undegraded":"2024-08-30T23:26:46.521830+0000","last_fullsized":"2024-08-30T23:26:46.521830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:26:04.139930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:26:04.139930+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:21.353939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"128'4193","reported_seq":4265,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.562879+0000","last_change":"2024-08-30T23:25:50.452633+0000","last_active":"2024-08-30T23:26:48.562879+0000","last_peered":"2024-08-30T23:26:48.562879+0000","last_clean":"2024-08-30T23:26:48.562879+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:26:48.562879+0000","last_undegraded":"2024-08-30T23:26:48.562879+0000","last_fullsized":"2024-08-30T23:26:48.562879+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:56.515696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":268352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"128'3977","reported_seq":4050,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.267593+0000","last_change":"2024-08-30T23:25:50.451520+0000","last_active":"2024-08-30T23:26:47.267593+0000","last_peered":"2024-08-30T23:26:47.267593+0000","last_clean":"2024-08-30T23:26:47.267593+0000","last_became_active":"2024-08-30T23:25:50.451193+0000","last_became_peered":"2024-08-30T23:25:50.451193+0000","last_unstale":"2024-08-30T23:26:47.267593+0000","last_undegraded":"2024-08-30T23:26:47.267593+0000","last_fullsized":"2024-08-30T23:26:47.267593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:55.292492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260636672,"num_objects":3977,"num_object_clones":0,"num_object_copies":11931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3977,"num_write_kb":254528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"128'4128","reported_seq":4201,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.286262+0000","last_change":"2024-08-30T23:25:50.531742+0000","last_active":"2024-08-30T23:26:48.286262+0000","last_peered":"2024-08-30T23:26:48.286262+0000","last_clean":"2024-08-30T23:26:48.286262+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:26:48.286262+0000","last_undegraded":"2024-08-30T23:26:48.286262+0000","last_fullsized":"2024-08-30T23:26:48.286262+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:53.584554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270532608,"num_objects":4128,"num_object_clones":0,"num_object_copies":12384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":264192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"128'4065","reported_seq":4138,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:48.519469+0000","last_change":"2024-08-30T23:25:50.451424+0000","last_active":"2024-08-30T23:26:48.519469+0000","last_peered":"2024-08-30T23:26:48.519469+0000","last_clean":"2024-08-30T23:26:48.519469+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:26:48.519469+0000","last_undegraded":"2024-08-30T23:26:48.519469+0000","last_fullsized":"2024-08-30T23:26:48.519469+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:19.191011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":260160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"128'4026","reported_seq":4098,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.357493+0000","last_change":"2024-08-30T23:25:50.532259+0000","last_active":"2024-08-30T23:26:47.357493+0000","last_peered":"2024-08-30T23:26:47.357493+0000","last_clean":"2024-08-30T23:26:47.357493+0000","last_became_active":"2024-08-30T23:25:50.531665+0000","last_became_peered":"2024-08-30T23:25:50.531665+0000","last_unstale":"2024-08-30T23:26:47.357493+0000","last_undegraded":"2024-08-30T23:26:47.357493+0000","last_fullsized":"2024-08-30T23:26:47.357493+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:05.834714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4026,"num_write_kb":257664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"129'4258","reported_seq":4333,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T23:26:50.618640+0000","last_change":"2024-08-30T23:25:50.532054+0000","last_active":"2024-08-30T23:26:50.618640+0000","last_peered":"2024-08-30T23:26:50.618640+0000","last_clean":"2024-08-30T23:26:50.618640+0000","last_became_active":"2024-08-30T23:25:50.531670+0000","last_became_peered":"2024-08-30T23:25:50.531670+0000","last_unstale":"2024-08-30T23:26:50.618640+0000","last_undegraded":"2024-08-30T23:26:50.618640+0000","last_fullsized":"2024-08-30T23:26:50.618640+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:06.951678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279052288,"num_objects":4258,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4258,"num_write_kb":272512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"128'4032","reported_seq":4103,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.321638+0000","last_change":"2024-08-30T23:25:50.474645+0000","last_active":"2024-08-30T23:26:47.321638+0000","last_peered":"2024-08-30T23:26:47.321638+0000","last_clean":"2024-08-30T23:26:47.321638+0000","last_became_active":"2024-08-30T23:25:50.474315+0000","last_became_peered":"2024-08-30T23:25:50.474315+0000","last_unstale":"2024-08-30T23:26:47.321638+0000","last_undegraded":"2024-08-30T23:26:47.321638+0000","last_fullsized":"2024-08-30T23:26:47.321638+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:05.978771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"128'3970","reported_seq":4042,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:46.792222+0000","last_change":"2024-08-30T23:25:50.451369+0000","last_active":"2024-08-30T23:26:46.792222+0000","last_peered":"2024-08-30T23:26:46.792222+0000","last_clean":"2024-08-30T23:26:46.792222+0000","last_became_active":"2024-08-30T23:25:50.451015+0000","last_became_peered":"2024-08-30T23:25:50.451015+0000","last_unstale":"2024-08-30T23:26:46.792222+0000","last_undegraded":"2024-08-30T23:26:46.792222+0000","last_fullsized":"2024-08-30T23:26:46.792222+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:11.902839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260177920,"num_objects":3970,"num_object_clones":0,"num_object_copies":11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3970,"num_write_kb":254080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"128'4026","reported_seq":4099,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.287821+0000","last_change":"2024-08-30T23:25:50.531975+0000","last_active":"2024-08-30T23:26:47.287821+0000","last_peered":"2024-08-30T23:26:47.287821+0000","last_clean":"2024-08-30T23:26:47.287821+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:26:47.287821+0000","last_undegraded":"2024-08-30T23:26:47.287821+0000","last_fullsized":"2024-08-30T23:26:47.287821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:52.802573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263847936,"num_objects":4026,"num_object_clones":0,"num_object_copies":12078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4026,"num_write_kb":257664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"128'4090","reported_seq":4161,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-08-30T23:26:47.348673+0000","last_change":"2024-08-30T23:25:50.474563+0000","last_active":"2024-08-30T23:26:47.348673+0000","last_peered":"2024-08-30T23:26:47.348673+0000","last_clean":"2024-08-30T23:26:47.348673+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:26:47.348673+0000","last_undegraded":"2024-08-30T23:26:47.348673+0000","last_fullsized":"2024-08-30T23:26:47.348673+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:27:17.998479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268042240,"num_objects":4090,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4090,"num_write_kb":261760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"129'4201","reported_seq":4274,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T23:26:51.035827+0000","last_change":"2024-08-30T23:25:50.474244+0000","last_active":"2024-08-30T23:26:51.035827+0000","last_peered":"2024-08-30T23:26:51.035827+0000","last_clean":"2024-08-30T23:26:51.035827+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:26:51.035827+0000","last_undegraded":"2024-08-30T23:26:51.035827+0000","last_fullsized":"2024-08-30T23:26:51.035827+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:57.627855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275316736,"num_objects":4201,"num_object_clones":0,"num_object_copies":12603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4201,"num_write_kb":268864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"129'4225","reported_seq":4299,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-08-30T23:26:50.604546+0000","last_change":"2024-08-30T23:25:50.451211+0000","last_active":"2024-08-30T23:26:50.604546+0000","last_peered":"2024-08-30T23:26:50.604546+0000","last_clean":"2024-08-30T23:26:50.604546+0000","last_became_active":"2024-08-30T23:25:50.450857+0000","last_became_peered":"2024-08-30T23:25:50.450857+0000","last_unstale":"2024-08-30T23:26:50.604546+0000","last_undegraded":"2024-08-30T23:26:50.604546+0000","last_fullsized":"2024-08-30T23:26:50.604546+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":4225,"log_dups_size":0,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:26:59.474362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276889600,"num_objects":4225,"num_object_clones":0,"num_object_copies":12675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":270400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4331077632,"num_objects":66087,"num_object_clones":0,"num_object_copies":198261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66087,"num_write_kb":4229568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13020364800,"data_stored":13020364800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66087,"ondisk_log_size":66087,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187484,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846708,"kb_used_data":815520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92898444,"statfs":{"total":95995035648,"available":95128006656,"internally_reserved":0,"allocated":835092480,"data_stored":834585773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449122,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852904,"kb_used_data":825880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92892248,"statfs":{"total":95995035648,"available":95121661952,"internally_reserved":0,"allocated":845701120,"data_stored":845197985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1650412,"kb_used_data":1601176,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92094740,"statfs":{"total":95995035648,"available":94305013760,"internally_reserved":0,"allocated":1639604224,"data_stored":1639101089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972396,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128044,"kb_used_data":1078808,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92617108,"statfs":{"total":95995035648,"available":94839918592,"internally_reserved":0,"allocated":1104699392,"data_stored":1104196257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234030,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329624,"kb_used_data":1272072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92415528,"statfs":{"total":95995035648,"available":94633500672,"internally_reserved":0,"allocated":1302601728,"data_stored":1302104225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":527512,"kb_used_data":500488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93217640,"statfs":{"total":95995035648,"available":95454863360,"internally_reserved":0,"allocated":512499712,"data_stored":512002209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":550176,"kb_used_data":523148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93194976,"statfs":{"total":95995035648,"available":95431655424,"internally_reserved":0,"allocated":535703552,"data_stored":535205013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953533,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":832412,"kb_used_data":805388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92912740,"statfs":{"total":95995035648,"available":95142645760,"internally_reserved":0,"allocated":824717312,"data_stored":824218773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625056,"kb_used_data":1567500,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92120096,"statfs":{"total":95995035648,"available":94330978304,"internally_reserved":0,"allocated":1605120000,"data_stored":1604621461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1092768,"kb_used_data":1047692,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92652384,"statfs":{"total":95995035648,"available":94876041216,"internally_reserved":0,"allocated":1072836608,"data_stored":1072338069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1339104,"kb_used_data":1277388,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":92406048,"statfs":{"total":95995035648,"available":94623793152,"internally_reserved":0,"allocated":1308045312,"data_stored":1307546773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1348832,"kb_used_data":1303756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92396320,"statfs":{"total":95995035648,"available":94613831680,"internally_reserved":0,"allocated":1335046144,"data_stored":1334547605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1339949056,"data_stored":1339949056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1317535744,"data_stored":1317535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1079705600,"data_stored":1079705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1622867968,"data_stored":1622867968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":828637184,"data_stored":828637184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":541196288,"data_stored":541196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":518651904,"data_stored":518651904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1322647552,"data_stored":1322647552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1113980928,"data_stored":1113980928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1654063104,"data_stored":1654063104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":846921728,"data_stored":846921728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":834207744,"data_stored":834207744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:26:53.885 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-30T23:26:53.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:26:53.886 INFO:tasks.thrashosds.thrasher:Reweighting osd 8 to 0.6486270766708919 2024-08-30T23:26:53.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 0.6486270766708919 2024-08-30T23:26:53.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:53.980 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:53.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:54.013 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:54.013 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:54.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.116 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:54.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:54.166 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 72698 72682 81.1117 79 0.0115671 0.0123239 2024-08-30T23:26:54.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:54.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:54.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:54.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:54.425 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:54.426 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:54.426 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:54 smithi106 ceph-mon[16603]: osdmap e130: 12 total, 12 up, 12 in 2024-08-30T23:26:54.426 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:54 smithi106 ceph-mon[16603]: pgmap v361: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-08-30T23:26:54.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:54.483 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:54.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:54 smithi136 ceph-mon[20207]: osdmap e130: 12 total, 12 up, 12 in 2024-08-30T23:26:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:54 smithi136 ceph-mon[20207]: pgmap v361: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-08-30T23:26:54.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:54.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:54 smithi173 ceph-mon[20554]: osdmap e130: 12 total, 12 up, 12 in 2024-08-30T23:26:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:54 smithi173 ceph-mon[20554]: pgmap v361: 17 pgs: 17 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-08-30T23:26:54.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:54.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:54.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:54.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:26:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:54.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:54.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:54.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:54.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:54.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:55.015 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:55.047 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:26:55.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:55.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:55.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 73963 73947 81.0757 79.0625 0.0140069 0.0123292 2024-08-30T23:26:55.185 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:26:55.235 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:26:55.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:55.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:55.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:55.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:55.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/652045877' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:55.479 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:55 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:55.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:55.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/652045877' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:55 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/652045877' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:55 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:26:55.621 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:26:55.621 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:26:55.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:55.653 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:55.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:55.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:55.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:55.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:55.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:55.994 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:56.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:56.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:56.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:56.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 75214 75198 81.0253 78.1875 0.00400683 0.0123364 2024-08-30T23:26:56.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:56.232 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.8 to 0.648627 (a60c) 2024-08-30T23:26:56.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:56.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:56.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:56.310 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:56 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:56.310 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:56.311 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:56 smithi173 ceph-mon[20554]: osdmap e131: 12 total, 12 up, 12 in 2024-08-30T23:26:56.311 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:56 smithi173 ceph-mon[20554]: pgmap v363: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-30T23:26:56.311 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3511216060' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919} : dispatch 2024-08-30T23:26:56.311 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919} : dispatch 2024-08-30T23:26:56.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:26:56.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:56.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:56.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.541 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:56.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:56.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:56 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:56.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:56.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:56 smithi136 ceph-mon[20207]: osdmap e131: 12 total, 12 up, 12 in 2024-08-30T23:26:56.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:56 smithi136 ceph-mon[20207]: pgmap v363: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-30T23:26:56.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3511216060' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919} : dispatch 2024-08-30T23:26:56.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919} : dispatch 2024-08-30T23:26:56.591 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:26:56.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:56 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:26:56.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:26:56.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:56 smithi106 ceph-mon[16603]: osdmap e131: 12 total, 12 up, 12 in 2024-08-30T23:26:56.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:56 smithi106 ceph-mon[16603]: pgmap v363: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-30T23:26:56.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3511216060' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919} : dispatch 2024-08-30T23:26:56.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919} : dispatch 2024-08-30T23:26:56.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:56.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:56.706 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:26:56.707 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:26:56.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:26:56.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:26:56.748 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:26:56.748 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:26:56.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:56.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:26:56.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:26:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:56.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:56.789 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:26:56.789 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:26:56.810 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:26:56.811 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:26:56.834 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:26:56.834 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:26:56.857 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:26:56.857 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:26:56.878 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:26:56.879 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:26:56.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:56.899 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:26:56.900 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:26:56.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:56.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:56.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:26:56.921 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:26:56.942 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:26:56.942 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:26:57.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:57.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:57.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:57.160 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:57.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 75430 75414 79.8808 13.5 0.0784431 0.0123771 2024-08-30T23:26:57.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:26:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:57.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:26:57.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.408 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:57.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:57.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:57 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919}]': finished 2024-08-30T23:26:57.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:57 smithi106 ceph-mon[16603]: osdmap e132: 12 total, 12 up, 12 in 2024-08-30T23:26:57.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3179588671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:57.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:57.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:57 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919}]': finished 2024-08-30T23:26:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:57 smithi136 ceph-mon[20207]: osdmap e132: 12 total, 12 up, 12 in 2024-08-30T23:26:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3179588671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:57.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:57 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.6486270766708919}]': finished 2024-08-30T23:26:57.553 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:57 smithi173 ceph-mon[20554]: osdmap e132: 12 total, 12 up, 12 in 2024-08-30T23:26:57.554 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3179588671' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:26:57.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:57.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:26:57.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:57.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:57.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:57.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:57.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:57.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:26:58.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:58.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:58.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:58.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:58.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:26:58.168361+0000 min lat: 0.00101743 max lat: 0.874328 avg lat: 0.0123932 2024-08-30T23:26:58.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:26:58.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 75451 75435 78.5713 1.3125 0.299622 0.0123932 2024-08-30T23:26:58.207 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:26:58.207 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:26:58.207 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:26:58.207 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:26:58.217 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:26:58.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:26:58.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:26:58.381 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:26:58.406 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:26:58.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:58.538 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:58 smithi136 ceph-mon[20207]: osdmap e133: 12 total, 12 up, 12 in 2024-08-30T23:26:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:58 smithi136 ceph-mon[20207]: pgmap v366: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.14k op/s 2024-08-30T23:26:58.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:58 smithi173 ceph-mon[20554]: osdmap e133: 12 total, 12 up, 12 in 2024-08-30T23:26:58.557 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:58 smithi173 ceph-mon[20554]: pgmap v366: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.14k op/s 2024-08-30T23:26:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:58 smithi106 ceph-mon[16603]: osdmap e133: 12 total, 12 up, 12 in 2024-08-30T23:26:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:58 smithi106 ceph-mon[16603]: pgmap v366: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.14k op/s 2024-08-30T23:26:58.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:26:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:58.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:26:58.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:58.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:58.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:26:58.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:26:58.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:58.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:26:59.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:26:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:59.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:26:59.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:26:59.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 76260 76244 78.112 50.5625 0.00674996 0.0127967 2024-08-30T23:26:59.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:26:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:26:59.206 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:26:59.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:59.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:59.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:26:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:59.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:26:59.520 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:59 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:26:59.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:59 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:26:59.521 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:26:59 smithi173 ceph-mon[20554]: osdmap e134: 12 total, 12 up, 12 in 2024-08-30T23:26:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:59 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:26:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:59 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:26:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:26:59 smithi136 ceph-mon[20207]: osdmap e134: 12 total, 12 up, 12 in 2024-08-30T23:26:59.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:26:59.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:26:59.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:59 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:26:59.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:59 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:26:59.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:26:59 smithi106 ceph-mon[16603]: osdmap e134: 12 total, 12 up, 12 in 2024-08-30T23:26:59.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:26:59.708 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:26:59.743 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:26:59.743 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:26:59.743 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:26:59.744 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:26:59.744 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:26:59.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:26:59.844 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:26:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:26:59.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:26:59.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:26:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:26:59.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:00.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:00.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:00.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:27:00.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 77086 77070 77.6848 51.625 0.0264686 0.0128665 2024-08-30T23:27:00.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.240 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:00.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:00.261 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:27:00.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:00.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:00.388 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:00 smithi173 ceph-mon[20554]: pgmap v368: 17 pgs: 1 remapped+peering, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-30T23:27:00.388 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2440127143' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:27:00.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:27:00.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:00.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:00 smithi136 ceph-mon[20207]: pgmap v368: 17 pgs: 1 remapped+peering, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-30T23:27:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2440127143' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:27:00.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:00.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:00 smithi106 ceph-mon[16603]: pgmap v368: 17 pgs: 1 remapped+peering, 16 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-30T23:27:00.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2440127143' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:27:00.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:00.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:00.737 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:00.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:00.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:00.890 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:00.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:00.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:00.996 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:01.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:01.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:01.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:01.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 15 78070 78055 77.4288 61.5625 0.00308152 0.0129083 2024-08-30T23:27:01.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:01.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:01.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:01.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2440127143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: osdmap e135: 12 total, 12 up, 12 in 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:01.450 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:01 smithi173 ceph-mon[20554]: 2.f scrub starts 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:839 2.d 2:b8994a18:::benchmark_data_smithi173_47195_object838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:25:58.575687+0000", 2024-08-30T23:27:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 62.873622550999997, 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.146041321, 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 839 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575687+0000", 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575687+0000", 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575689+0000", 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.468e-06 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575708+0000", 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9357000000000001e-05 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575722+0000", 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3560000000000001e-05 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575729+0000", 2024-08-30T23:27:01.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3629999999999996e-06 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575782+0000", 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2994000000000001e-05 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.575899+0000", 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000116731 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.576043+0000", 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014452599999999999 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.576428+0000", 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038457500000000001 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.576791+0000", 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036347799999999998 2024-08-30T23:27:01.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.721656+0000", 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14486436 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.721690+0000", 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4329000000000003e-05 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.721729+0000", 2024-08-30T23:27:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.858e-05 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:16926 2.d 2:b54c02df:::benchmark_data_smithi173_47195_object16925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:10.682533+0000", 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 50.766776948, 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16671683100000001, 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 16926 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682533+0000", 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682533+0000", 2024-08-30T23:27:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682534+0000", 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1850000000000001e-06 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682552+0000", 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7297000000000001e-05 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682561+0000", 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1730000000000001e-06 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682565+0000", 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4800000000000003e-06 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682605+0000", 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9715999999999997e-05 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682690+0000", 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5047000000000001e-05 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.682771+0000", 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1092000000000002e-05 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.683462+0000", 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069107599999999999 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.683760+0000", 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000297757 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.849176+0000", 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16541575 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.849210+0000", 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4152000000000001e-05 2024-08-30T23:27:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.849250+0000", 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0105999999999999e-05 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:17182 2.d 2:b341af9b:::benchmark_data_smithi173_47195_object17181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:10.983405+0000", 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 50.465905225999997, 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13862163899999999, 2024-08-30T23:27:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 17182 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983405+0000", 2024-08-30T23:27:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983405+0000", 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983406+0000", 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999991e-07 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983427+0000", 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1373e-05 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983436+0000", 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6209999999999993e-06 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983441+0000", 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8239999999999999e-06 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983488+0000", 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6956999999999998e-05 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983603+0000", 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115698 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.983731+0000", 2024-08-30T23:27:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012762299999999999 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.984529+0000", 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00079849599999999999 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.984912+0000", 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00038243400000000002 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.121910+0000", 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13699789800000001 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.121977+0000", 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6970000000000004e-05 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.122026+0000", 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9771000000000002e-05 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:17196 2.d 2:b96cb6c6:::benchmark_data_smithi173_47195_object17195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:10.990440+0000", 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 50.45886986, 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13517913300000001, 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 17196 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990440+0000", 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990440+0000", 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990441+0000", 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990462+0000", 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0747999999999999e-05 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990471+0000", 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7809999999999994e-06 2024-08-30T23:27:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990476+0000", 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9110000000000001e-06 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990519+0000", 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3516999999999997e-05 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990598+0000", 2024-08-30T23:27:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9120000000000001e-05 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.990684+0000", 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6026999999999995e-05 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.991413+0000", 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00072845400000000002 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.991615+0000", 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000201733 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.125559+0000", 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13394474000000001 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.125592+0000", 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3098000000000001e-05 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.125619+0000", 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6838e-05 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:17740 2.f e111/97 2:f20a1960:::benchmark_data_smithi173_47195_object17739:head v 111'1101, mlcod=111'1098)", 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.576403+0000", 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.872906551, 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13539435, 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.576403+0000", 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.576403+0000", 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.576404+0000", 2024-08-30T23:27:01.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.85e-07 2024-08-30T23:27:01.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.576450+0000", 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5485000000000002e-05 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.710916+0000", 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13446655299999999 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.710922+0000", 2024-08-30T23:27:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6389999999999997e-06 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.710985+0000", 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2582999999999994e-05 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.711003+0000", 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8342000000000002e-05 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.711754+0000", 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075084000000000001 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.711798+0000", 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4023000000000002e-05 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:22590 2.d 2:b2e5369c:::benchmark_data_smithi173_47195_object22589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:27:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:14.839341+0000", 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.609968821999999, 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16376903600000001, 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22590 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839341+0000", 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839341+0000", 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839342+0000", 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5000000000000002e-07 2024-08-30T23:27:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839367+0000", 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4729999999999999e-05 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839376+0000", 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2220000000000008e-06 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839380+0000", 2024-08-30T23:27:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6859999999999997e-06 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839425+0000", 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4885000000000001e-05 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839515+0000", 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9942e-05 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.839590+0000", 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4375000000000005e-05 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.840168+0000", 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057875400000000001 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.840697+0000", 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052875800000000005 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.003029+0000", 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16233224299999999 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.003066+0000", 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6259e-05 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.003110+0000", 2024-08-30T23:27:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4532e-05 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:22590 2.d e112/97 ondisk, result = 0)", 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:14.840531+0000", 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.608779095999999, 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16258529599999999, 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.840531+0000", 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.840531+0000", 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.840531+0000", 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4300000000000003e-07 2024-08-30T23:27:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.840534+0000", 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6989999999999999e-06 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.002978+0000", 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.162444159 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.002983+0000", 2024-08-30T23:27:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4719999999999997e-06 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.003020+0000", 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6735e-05 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.003025+0000", 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9959999999999999e-06 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.003116+0000", 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0692000000000005e-05 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:22622 2.d 2:ba9bb854:::benchmark_data_smithi173_47195_object22621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:14.869257+0000", 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.580053210000003, 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13618219000000001, 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22622 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869257+0000", 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869257+0000", 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869258+0000", 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-06 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869278+0000", 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9565999999999999e-05 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869290+0000", 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1966e-05 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869298+0000", 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9510000000000006e-06 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869347+0000", 2024-08-30T23:27:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9236000000000001e-05 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869428+0000", 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1348999999999997e-05 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.869515+0000", 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6349999999999998e-05 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.870214+0000", 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069931300000000002 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.870665+0000", 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045070400000000001 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005383+0000", 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13471789000000001 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005413+0000", 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0772999999999998e-05 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005439+0000", 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5599e-05 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:22622 2.d e112/97 ondisk, result = 0)", 2024-08-30T23:27:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:14.870856+0000", 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.578454129999997, 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.134586925, 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.870856+0000", 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.870856+0000", 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.870856+0000", 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2399999999999998e-07 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:14.870859+0000", 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.717e-06 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005326+0000", 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.134467065 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005332+0000", 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0780000000000001e-06 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005373+0000", 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0781000000000001e-05 2024-08-30T23:27:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005379+0000", 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.677e-06 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.005443+0000", 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4083000000000003e-05 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:22830 2.d 2:b58fe0ee:::benchmark_data_smithi173_47195_object22829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:15.163437+0000", 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.285873203000001, 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13433334199999999, 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22830 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163437+0000", 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163437+0000", 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163438+0000", 2024-08-30T23:27:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1999999999999999e-06 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163456+0000", 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8321e-05 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163467+0000", 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0864e-05 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163472+0000", 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9559999999999996e-06 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163517+0000", 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4697000000000003e-05 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163599+0000", 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2033000000000006e-05 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.163686+0000", 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6920000000000001e-05 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.164386+0000", 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069980900000000002 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.164828+0000", 2024-08-30T23:27:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044194599999999997 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.297559+0000", 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13273136899999999 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.297705+0000", 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014606800000000001 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.297770+0000", 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5159000000000004e-05 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:22838 2.e e112/97 2:791a2867:::benchmark_data_smithi173_47195_object22837:head v 112'1364, mlcod=112'1361)", 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:15.176231+0000", 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.273078978000001, 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.159852679, 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.176231+0000", 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.176231+0000", 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.176232+0000", 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.176279+0000", 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6860999999999997e-05 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.335321+0000", 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15904264600000001 2024-08-30T23:27:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.335326+0000", 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3540000000000002e-06 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.335414+0000", 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8702999999999994e-05 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.335429+0000", 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4996e-05 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.336055+0000", 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062551900000000005 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.336084+0000", 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8869000000000001e-05 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:36483 2.d 2:b8c0a5cd:::benchmark_data_smithi173_47195_object36482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:25.048741+0000", 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 36.400569281999999, 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.166601572, 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36483 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048741+0000", 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048741+0000", 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048742+0000", 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2699999999999999e-06 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048762+0000", 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9737000000000001e-05 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048773+0000", 2024-08-30T23:27:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0975000000000001e-05 2024-08-30T23:27:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048778+0000", 2024-08-30T23:27:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:27:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048826+0000", 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7635e-05 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.048914+0000", 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8039000000000002e-05 2024-08-30T23:27:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.049016+0000", 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102191 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.050011+0000", 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0009951789999999999 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.050270+0000", 2024-08-30T23:27:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025850200000000003 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.215015+0000", 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16474538 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.215281+0000", 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026580499999999998 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.215342+0000", 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1488999999999994e-05 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:56069 2.d 2:b616bca3:::benchmark_data_smithi173_47195_object56068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.788369+0000", 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.660941487999999, 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169062514, 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.523 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:01.524 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:01.524 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:01.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:01.525 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 56069 2024-08-30T23:27:01.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788369+0000", 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788369+0000", 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788370+0000", 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5209999999999999e-06 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788391+0000", 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0863999999999999e-05 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788403+0000", 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.204e-05 2024-08-30T23:27:01.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788410+0000", 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0119999999999999e-06 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788457+0000", 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7209999999999997e-05 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788545+0000", 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7726000000000007e-05 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.788633+0000", 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7688999999999993e-05 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.789977+0000", 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013448539999999999 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.790274+0000", 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029662800000000001 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957365+0000", 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16709111500000001 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957404+0000", 2024-08-30T23:27:01.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9115000000000001e-05 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957431+0000", 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6740000000000001e-05 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:56069 2.d e124/97 ondisk, result = 0)", 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.790276+0000", 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 21.659033545, 2024-08-30T23:27:01.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16716029399999999, 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.790276+0000", 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.790276+0000", 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.790277+0000", 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3600000000000003e-07 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.790280+0000", 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.875e-06 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957275+0000", 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16699530400000001 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957280+0000", 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2549999999999999e-06 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957356+0000", 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6516999999999999e-05 2024-08-30T23:27:01.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957362+0000", 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7980000000000002e-06 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.957437+0000", 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4908999999999997e-05 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62896 2.a e127/97 2:5bc7d7b5:::benchmark_data_smithi173_47195_object62895:head v 127'3849, mlcod=127'3846)", 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.537068073, 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.271516058, 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912243+0000", 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8800000000000002e-07 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912264+0000", 2024-08-30T23:27:01.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1222000000000001e-05 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182196+0000", 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26993194300000001 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182200+0000", 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4549999999999997e-06 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182246+0000", 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5404000000000001e-05 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182263+0000", 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7167999999999999e-05 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183724+0000", 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014614000000000001 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183758+0000", 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3677999999999998e-05 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67113 2.e e129/97 2:7c5fddf0:::benchmark_data_smithi173_47195_object67112:head v 129'4149, mlcod=129'4145)", 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.83731059, 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.177302496, 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.540 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.612000+0000", 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5600000000000004e-07 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.612070+0000", 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9492000000000005e-05 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787357+0000", 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17528732899999999 2024-08-30T23:27:01.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787369+0000", 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2272999999999999e-05 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787501+0000", 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013128700000000001 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787559+0000", 2024-08-30T23:27:01.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8606000000000001e-05 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.789267+0000", 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017075619999999999 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.789302+0000", 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4990999999999998e-05 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67116 2.e e129/97 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head v 129'4150, mlcod=129'4146)", 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.834702655999999, 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22312085200000001, 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614608+0000", 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614657+0000", 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8501000000000002e-05 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835863+0000", 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.221206134 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835869+0000", 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6359999999999997e-06 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835916+0000", 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6378999999999998e-05 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835929+0000", 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3583e-05 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.837698+0000", 2024-08-30T23:27:01.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001769112 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.837728+0000", 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9808999999999999e-05 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67586 2.e e129/97 2:726358de:::benchmark_data_smithi173_47195_object67585:head v 129'4182, mlcod=129'4178)", 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:27:01.549 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.347501672, 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.182122652, 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:27:01.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101809+0000", 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101870+0000", 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1013e-05 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281568+0000", 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17969818400000001 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281577+0000", 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6270000000000006e-06 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281653+0000", 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5925000000000002e-05 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281681+0000", 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8076999999999999e-05 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.283872+0000", 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021911510000000001 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.283931+0000", 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9083000000000003e-05 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.553 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67956 2.a e129/97 2:5f39d156:::benchmark_data_smithi173_47195_object67955:head v 129'4165, mlcod=129'4162)", 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:50.483517+0000", 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.965792843999999, 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.155099557, 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.483517+0000", 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.483517+0000", 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.483519+0000", 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.466e-06 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.483542+0000", 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3354999999999999e-05 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.636532+0000", 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.152989821 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.636539+0000", 2024-08-30T23:27:01.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4699999999999996e-06 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.636609+0000", 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9697999999999996e-05 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.636630+0000", 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0869999999999998e-05 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.638585+0000", 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019555219999999999 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.638617+0000", 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1355e-05 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:75419 2.0 e132/97 2:02abbf90:::benchmark_data_smithi173_47195_object75418:head v 132'4710, mlcod=132'4708)", 2024-08-30T23:27:01.558 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.5998415059999997, 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87299763699999999, 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849469+0000", 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9299999999999996e-07 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849566+0000", 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6235000000000003e-05 2024-08-30T23:27:01.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719661+0000", 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87009526500000001 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719680+0000", 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8785999999999999e-05 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719819+0000", 2024-08-30T23:27:01.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139491 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719862+0000", 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2382000000000002e-05 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.722427+0000", 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025652909999999999 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:27:01.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.722466+0000", 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9294e-05 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:01.563 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2440127143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: osdmap e135: 12 total, 12 up, 12 in 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:01.564 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:01 smithi136 ceph-mon[20207]: 2.f scrub starts 2024-08-30T23:27:01.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:01.565 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2440127143' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: osdmap e135: 12 total, 12 up, 12 in 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:01.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:01 smithi106 ceph-mon[16603]: 2.f scrub starts 2024-08-30T23:27:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:01.584 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:01.584 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:01.614 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:01.614 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:01.640 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:01.640 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:01.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:01.667 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:01.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:01.699 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:01.701 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:01.701 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:01.730 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:01.730 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:01.759 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:01.759 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:01.788 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:01.788 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:01.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:01.811 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:01.812 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:01.825 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:01.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:01.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:01.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:02.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:02.047 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:02.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:02.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:02.075 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:02.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:02.105 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:02.105 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:02.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 78955 78939 77.0822 55.25 0.00446471 0.0129665 2024-08-30T23:27:02.172 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:27:02.172 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:27:02.172 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:27:02.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd out 10 2024-08-30T23:27:02.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:02.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:02.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:02 smithi173 ceph-mon[20554]: osdmap e136: 12 total, 12 up, 12 in 2024-08-30T23:27:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:02 smithi173 ceph-mon[20554]: pgmap v371: 17 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 15 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 4531/228426 objects misplaced (1.984%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:27:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1245584415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:02.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:27:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:02.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:02 smithi106 ceph-mon[16603]: osdmap e136: 12 total, 12 up, 12 in 2024-08-30T23:27:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:02 smithi106 ceph-mon[16603]: pgmap v371: 17 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 15 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 4531/228426 objects misplaced (1.984%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:27:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1245584415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:02.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:02.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:02.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:02 smithi136 ceph-mon[20207]: osdmap e136: 12 total, 12 up, 12 in 2024-08-30T23:27:02.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:02 smithi136 ceph-mon[20207]: pgmap v371: 17 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 15 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 4531/228426 objects misplaced (1.984%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:27:02.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1245584415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:02.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:02.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:02.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:02.728 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.736 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:02.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:02.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:02.870 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:02.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:02.903 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:02.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:27:02.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:03.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:03.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:03.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:03.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 80078 80062 76.976 70.1875 0.00606472 0.012986 2024-08-30T23:27:03.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:03.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:03.283 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:03.283 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:03 smithi106 ceph-mon[16603]: 2.f scrub ok 2024-08-30T23:27:03.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.408 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:03.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:03.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:03.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:03.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:03 smithi136 ceph-mon[20207]: 2.f scrub ok 2024-08-30T23:27:03.573 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:03 smithi173 ceph-mon[20554]: 2.f scrub ok 2024-08-30T23:27:03.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:03.662 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:03.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:03.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:03.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:03.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:03.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:03.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:03 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:27:03.768+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e136 definitely_dead 0 2024-08-30T23:27:04.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:04.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:04.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:27:04.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:04.167 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 80768 80752 76.4631 43.125 0.0079765 0.0130727 2024-08-30T23:27:04.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:04.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:04.292 INFO:teuthology.orchestra.run.smithi106.stderr:marked out osd.10. 2024-08-30T23:27:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:04 smithi173 ceph-mon[20554]: pgmap v372: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s; 9022/235125 objects misplaced (3.837%); 3.6 MiB/s, 57 objects/s recovering 2024-08-30T23:27:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2961223200' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:27:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:04 smithi173 ceph-mon[20554]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:27:04.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:04.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:04.425 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:04.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:04.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:04.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:04.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:04 smithi106 ceph-mon[16603]: pgmap v372: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s; 9022/235125 objects misplaced (3.837%); 3.6 MiB/s, 57 objects/s recovering 2024-08-30T23:27:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2961223200' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:27:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:04 smithi106 ceph-mon[16603]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:27:04.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:04.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:04 smithi136 ceph-mon[20207]: pgmap v372: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 866 op/s; 9022/235125 objects misplaced (3.837%); 3.6 MiB/s, 57 objects/s recovering 2024-08-30T23:27:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2961223200' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:27:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:04 smithi136 ceph-mon[20207]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:27:04.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:04.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:04.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:27:04.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:04.932 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:04.932 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:04.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:04.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:05.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:05.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:05.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 81027 81011 75.5634 16.1875 0.00353064 0.0131132 2024-08-30T23:27:05.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:05.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:05.290 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:27:05.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:05.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:05.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:05.439 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:05.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:05.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2961223200' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:27:05.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:05 smithi173 ceph-mon[20554]: osdmap e137: 12 total, 12 up, 11 in 2024-08-30T23:27:05.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/327426200' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:05.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2961223200' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:27:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:05 smithi106 ceph-mon[16603]: osdmap e137: 12 total, 12 up, 11 in 2024-08-30T23:27:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/327426200' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:05.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:27:05.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:05.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2961223200' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:27:05.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:05 smithi136 ceph-mon[20207]: osdmap e137: 12 total, 12 up, 11 in 2024-08-30T23:27:05.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/327426200' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:05.705 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:05.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:05.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:05.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:05.971 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:05.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:05.979 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:06.059 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:06.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:06.118 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:06.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 81487 81471 74.8749 28.75 0.00401479 0.0133485 2024-08-30T23:27:06.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:06.254 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:06.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:06.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:06.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:06 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:27:06.396 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:06 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:27:06.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/327426200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:06.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:06 smithi173 ceph-mon[20554]: osdmap e138: 12 total, 12 up, 11 in 2024-08-30T23:27:06.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:06 smithi173 ceph-mon[20554]: pgmap v375: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 48 MiB/s wr, 767 op/s; 9022/237795 objects misplaced (3.794%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:27:06.397 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:06 smithi173 ceph-mon[20554]: osdmap e139: 12 total, 12 up, 11 in 2024-08-30T23:27:06.424 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:06 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:27:06.424 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:06 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:27:06.424 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/327426200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:06.425 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:06 smithi136 ceph-mon[20207]: osdmap e138: 12 total, 12 up, 11 in 2024-08-30T23:27:06.425 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:06 smithi136 ceph-mon[20207]: pgmap v375: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 48 MiB/s wr, 767 op/s; 9022/237795 objects misplaced (3.794%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:27:06.425 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:06 smithi136 ceph-mon[20207]: osdmap e139: 12 total, 12 up, 11 in 2024-08-30T23:27:06.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:06.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:06.529 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:06.529 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:06.531 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:06 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:27:06.531 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:06 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:27:06.531 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/327426200' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:06.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:06 smithi106 ceph-mon[16603]: osdmap e138: 12 total, 12 up, 11 in 2024-08-30T23:27:06.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:06 smithi106 ceph-mon[16603]: pgmap v375: 17 pgs: 2 active+recovering+undersized+remapped, 15 active+clean; 4.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 48 MiB/s wr, 767 op/s; 9022/237795 objects misplaced (3.794%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:27:06.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:06 smithi106 ceph-mon[16603]: osdmap e139: 12 total, 12 up, 11 in 2024-08-30T23:27:06.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:06.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:06.705 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:06.705 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:06.738 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:06.738 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:06.768 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:06.769 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:06.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:06.798 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:06.798 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:06.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:06.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:06.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:06.864 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:06.864 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:06.894 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:06.895 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:06.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:06.924 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:06.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:06.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:06.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:06.953 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:06.953 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:06.983 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:06.983 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:07.013 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:07.013 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:07.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:07.045 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:07.045 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:07.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:07.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 82231 82215 74.4636 46.5 0.0248897 0.0134227 2024-08-30T23:27:07.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:07.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:07.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:27:07.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:07.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:07.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:07.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1298318440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:07.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:07.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:07.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.463 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:07.517 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:07.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:07.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:07.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:07 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:07.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1298318440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:07.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:07.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:07.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1298318440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:07.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:07.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:07.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:07.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:07.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:08.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:27:08.080 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:82797 2.b 2:dd865a21:::benchmark_data_smithi173_47195_object82796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:08.082855+0000", 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0049110839999999996, 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049320220000000003, 2024-08-30T23:27:08.088 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82797 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082855+0000", 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082855+0000", 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082857+0000", 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8309999999999999e-06 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082883+0000", 2024-08-30T23:27:08.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5553000000000001e-05 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082896+0000", 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3176999999999999e-05 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082903+0000", 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.481e-06 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:08.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082959+0000", 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5951e-05 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.083123+0000", 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016360499999999999 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.083297+0000", 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000174434 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.085406+0000", 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021081680000000001 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.086577+0000", 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001171348 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.093 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:82790 2.b 2:d3fed628:::benchmark_data_smithi173_47195_object82789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:08.075217+0000", 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.012549097, 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012617499000000001, 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 82790 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.075217+0000", 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.075217+0000", 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.075219+0000", 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6449999999999999e-06 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.075243+0000", 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3533e-05 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.075255+0000", 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2642e-05 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.077568+0000", 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023123150000000001 2024-08-30T23:27:08.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.078991+0000", 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001423082 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.079175+0000", 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018392300000000001 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.079345+0000", 2024-08-30T23:27:08.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017075700000000001 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.081926+0000", 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025811639999999999 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:08.082253+0000", 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000326726 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:27:08.098 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:08.099 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:27:08.099 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:27:08.099 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 2 2024-08-30T23:27:08.099 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:27:08.113 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:08.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 82858 82842 73.9596 39.1875 0.0408877 0.0135141 2024-08-30T23:27:08.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:08.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:08.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:08.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:08.384 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:08.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:08.507 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:08 smithi173 ceph-mon[20554]: osdmap e140: 12 total, 12 up, 11 in 2024-08-30T23:27:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:08 smithi173 ceph-mon[20554]: pgmap v378: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 4.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 36 MiB/s wr, 580 op/s; 18732/242091 objects misplaced (7.738%); 6.5 MiB/s, 103 objects/s recovering 2024-08-30T23:27:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:08 smithi106 ceph-mon[16603]: osdmap e140: 12 total, 12 up, 11 in 2024-08-30T23:27:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:08 smithi106 ceph-mon[16603]: pgmap v378: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 4.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 36 MiB/s wr, 580 op/s; 18732/242091 objects misplaced (7.738%); 6.5 MiB/s, 103 objects/s recovering 2024-08-30T23:27:08.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:08.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:08.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:08.764 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:08 smithi136 ceph-mon[20207]: osdmap e140: 12 total, 12 up, 11 in 2024-08-30T23:27:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:08 smithi136 ceph-mon[20207]: pgmap v378: 17 pgs: 4 active+recovering+undersized+remapped, 13 active+clean; 4.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 36 MiB/s wr, 580 op/s; 18732/242091 objects misplaced (7.738%); 6.5 MiB/s, 103 objects/s recovering 2024-08-30T23:27:08.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:27:08.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:08.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:08.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:08.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:09.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:09.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:09.143 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:09.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:09.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 83636 83620 73.6028 48.625 0.0269143 0.0135782 2024-08-30T23:27:09.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:09.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.301 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:09.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:09.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:09.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:09.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:09.549 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:09 smithi173 ceph-mon[20554]: pgmap v379: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 47 MiB/s wr, 748 op/s; 23430/246492 objects misplaced (9.505%); 9.7 MiB/s, 155 objects/s recovering 2024-08-30T23:27:09.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:09 smithi106 ceph-mon[16603]: pgmap v379: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 47 MiB/s wr, 748 op/s; 23430/246492 objects misplaced (9.505%); 9.7 MiB/s, 155 objects/s recovering 2024-08-30T23:27:09.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:09.679 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:27:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:27:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:27:09.679 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:27:09.679 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:27:09.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:09.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:09 smithi136 ceph-mon[20207]: pgmap v379: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 47 MiB/s wr, 748 op/s; 23430/246492 objects misplaced (9.505%); 9.7 MiB/s, 155 objects/s recovering 2024-08-30T23:27:09.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:27:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.814 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:09.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:09.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:09.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:09.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:10.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:10.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:10.159 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:10.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:10.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 84373 84357 73.2202 46.0625 0.0343751 0.0136499 2024-08-30T23:27:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:10.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:10.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:10.297 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:27:10.297 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:27:10.297 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:27:10.297 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.329250 2024-08-30T23:27:10.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 1 0.3292504376112728 2024-08-30T23:27:10.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:10.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:27:10.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:10.463 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:27:10.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:10.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2969616448' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:27:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2969616448' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:27:10.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:10.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:10.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:10.714 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.751 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:10.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:10.764 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:10.764 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2969616448' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:27:10.800 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:10.801 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:10.833 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:10.833 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:10.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:10.875 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:10.875 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:10.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:10.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:10.908 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:10.908 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:10.944 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:10.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:10.969 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:10.969 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:10.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:10.997 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:11.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:11.023 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:11.023 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:11.051 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:11.051 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:11.079 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:11.079 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:11.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:11.110 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:11.110 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:11.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:11.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:11.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 15 85064 85049 72.8096 43.25 0.00716952 0.0137279 2024-08-30T23:27:11.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:11.233 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:11.233 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:11.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:11.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:11.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:11.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:11.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:11.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:11.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:11.645 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:11.715 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:11.715 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:11.715 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:11.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2969616448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:27:11.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: osdmap e141: 12 total, 12 up, 11 in 2024-08-30T23:27:11.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2036071153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:11.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:11 smithi106 ceph-mon[16603]: pgmap v381: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 42 MiB/s wr, 675 op/s; 23122/249951 objects misplaced (9.251%); 11 MiB/s, 182 objects/s recovering 2024-08-30T23:27:11.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:11.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2969616448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:27:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: osdmap e141: 12 total, 12 up, 11 in 2024-08-30T23:27:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2036071153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:11 smithi136 ceph-mon[20207]: pgmap v381: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 42 MiB/s wr, 675 op/s; 23122/249951 objects misplaced (9.251%); 11 MiB/s, 182 objects/s recovering 2024-08-30T23:27:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2969616448' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:27:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: osdmap e141: 12 total, 12 up, 11 in 2024-08-30T23:27:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2036071153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:11 smithi173 ceph-mon[20554]: pgmap v381: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 42 MiB/s wr, 675 op/s; 23122/249951 objects misplaced (9.251%); 11 MiB/s, 182 objects/s recovering 2024-08-30T23:27:11.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:11.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:11.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:11.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:27:12.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:12.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:12.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:12.154 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:12.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 85777 85761 72.427 44.5 0.0124739 0.0137987 2024-08-30T23:27:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.170 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:12.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:12.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:12.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:12.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.435 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:12.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:12.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:12.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:12.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:12.699 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:12.782 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:12.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:12.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:12.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:12.970 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:12.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:13.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.125 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:13.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:13.131 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.1 primary-affinity to 0.32925 (8215772) 2024-08-30T23:27:13.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 86571 86555 72.1229 49.625 0.00808371 0.0138553 2024-08-30T23:27:13.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:13.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:13.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:13.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/665251586' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728} : dispatch 2024-08-30T23:27:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:13 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728} : dispatch 2024-08-30T23:27:13.374 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:13.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:27:13.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:13.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:13.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:13.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/665251586' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728} : dispatch 2024-08-30T23:27:13.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:13 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728} : dispatch 2024-08-30T23:27:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/665251586' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728} : dispatch 2024-08-30T23:27:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:13 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728} : dispatch 2024-08-30T23:27:13.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:13.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:13.683 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:13.683 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:13.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:13.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:13.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:13.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:13.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:13.998 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:13.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:14.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:14.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:27:14.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:14.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 86694 86678 71.2751 7.6875 0.00378512 0.013943 2024-08-30T23:27:14.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:14.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:14.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:14 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728}]': finished 2024-08-30T23:27:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:14 smithi106 ceph-mon[16603]: osdmap e142: 12 total, 12 up, 11 in 2024-08-30T23:27:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:14 smithi106 ceph-mon[16603]: pgmap v383: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 48 MiB/s wr, 767 op/s; 22243/256011 objects misplaced (8.688%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:27:14.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:14.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:14.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:14.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:14 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728}]': finished 2024-08-30T23:27:14.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:14 smithi136 ceph-mon[20207]: osdmap e142: 12 total, 12 up, 11 in 2024-08-30T23:27:14.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:14 smithi136 ceph-mon[20207]: pgmap v383: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 48 MiB/s wr, 767 op/s; 22243/256011 objects misplaced (8.688%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:27:14.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:14 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.3292504376112728}]': finished 2024-08-30T23:27:14.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:14 smithi173 ceph-mon[20554]: osdmap e142: 12 total, 12 up, 11 in 2024-08-30T23:27:14.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:14 smithi173 ceph-mon[20554]: pgmap v383: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 48 MiB/s wr, 767 op/s; 22243/256011 objects misplaced (8.688%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:27:14.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:14.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:14.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:14.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:14.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:14.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:14.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:14.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:15.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:15.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:15.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 86754 86738 70.3981 3.75 0.0039661 0.0139802 2024-08-30T23:27:15.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:15.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:15.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:15.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:15.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:15.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:15.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:15 smithi136 ceph-mon[20207]: osdmap e143: 12 total, 12 up, 11 in 2024-08-30T23:27:15.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:15 smithi173 ceph-mon[20554]: osdmap e143: 12 total, 12 up, 11 in 2024-08-30T23:27:15.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:15.606 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:15 smithi106 ceph-mon[16603]: osdmap e143: 12 total, 12 up, 11 in 2024-08-30T23:27:15.615 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:15.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:15.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:15.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:15.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:15.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:15.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:16.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:16.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:16.139 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:16.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 87410 87394 70.0212 41 0.00855441 0.0142742 2024-08-30T23:27:16.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:16.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:16 smithi106 ceph-mon[16603]: osdmap e144: 12 total, 12 up, 11 in 2024-08-30T23:27:16.259 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:16 smithi106 ceph-mon[16603]: pgmap v386: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 45 MiB/s wr, 725 op/s; 22243/257715 objects misplaced (8.631%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:27:16.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:16.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:16.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:16.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:16.517 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:16 smithi136 ceph-mon[20207]: osdmap e144: 12 total, 12 up, 11 in 2024-08-30T23:27:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:16 smithi136 ceph-mon[20207]: pgmap v386: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 45 MiB/s wr, 725 op/s; 22243/257715 objects misplaced (8.631%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:27:16.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:16 smithi173 ceph-mon[20554]: osdmap e144: 12 total, 12 up, 11 in 2024-08-30T23:27:16.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:16 smithi173 ceph-mon[20554]: pgmap v386: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 45 MiB/s wr, 725 op/s; 22243/257715 objects misplaced (8.631%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:27:16.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:16.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:16.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:16.776 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:16.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:16.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:17.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:17.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:17.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.167 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:17.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:17.168 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 88029 88013 69.6245 38.6875 0.00845777 0.0143556 2024-08-30T23:27:17.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.309 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:17.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:17.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:17.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:17.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:17.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:17.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:17.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:17 smithi136 ceph-mon[20207]: pgmap v387: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 38 MiB/s wr, 606 op/s; 21293/260868 objects misplaced (8.162%); 19 MiB/s, 304 objects/s recovering 2024-08-30T23:27:17.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:17 smithi173 ceph-mon[20554]: pgmap v387: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 38 MiB/s wr, 606 op/s; 21293/260868 objects misplaced (8.162%); 19 MiB/s, 304 objects/s recovering 2024-08-30T23:27:17.824 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:17 smithi106 ceph-mon[16603]: pgmap v387: 17 pgs: 5 active+recovering+undersized+remapped, 12 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 38 MiB/s wr, 606 op/s; 21293/260868 objects misplaced (8.162%); 19 MiB/s, 304 objects/s recovering 2024-08-30T23:27:17.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:17.954 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:17.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:18.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:18.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:18.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:27:18.170056+0000 min lat: 0.00101743 max lat: 2.27268 avg lat: 0.0144059 2024-08-30T23:27:18.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:27:18.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 88832 88816 69.3815 50.1875 0.00419538 0.0144059 2024-08-30T23:27:18.180 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:18.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:18.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.335 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:18.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:18.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:18.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:18.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:18.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:18.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:18.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:18.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:18.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:18.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:18.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:18.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:19.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:19.113 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:19.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 89625 89609 69.1368 49.5625 0.00831551 0.014455 2024-08-30T23:27:19.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.245 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:19.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:19.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:19.378 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:19.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:19.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:19.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:19.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:19.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:19.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:19 smithi136 ceph-mon[20207]: pgmap v388: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 34 MiB/s wr, 546 op/s; 20796/264051 objects misplaced (7.876%); 17 MiB/s, 267 objects/s recovering 2024-08-30T23:27:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:19 smithi173 ceph-mon[20554]: pgmap v388: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 34 MiB/s wr, 546 op/s; 20796/264051 objects misplaced (7.876%); 17 MiB/s, 267 objects/s recovering 2024-08-30T23:27:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:19 smithi106 ceph-mon[16603]: pgmap v388: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 34 MiB/s wr, 546 op/s; 20796/264051 objects misplaced (7.876%); 17 MiB/s, 267 objects/s recovering 2024-08-30T23:27:19.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:19.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:19.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:19.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:20.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:20.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:20.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:20.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 90427 90411 68.9049 50.125 0.0213234 0.0145073 2024-08-30T23:27:20.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:20.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:20.381 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:20.402 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:20.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:20.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:20.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:20.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:20.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:20.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:20.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:20.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:20.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:20.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:21.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:21.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:21.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:21.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 91161 91145 68.6274 45.875 0.00853007 0.0145644 2024-08-30T23:27:21.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:21.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:21.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:21.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:27:21.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:21.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:21.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:21.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:21 smithi106 ceph-mon[16603]: pgmap v389: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 32 MiB/s wr, 516 op/s; 20454/268413 objects misplaced (7.620%); 14 MiB/s, 223 objects/s recovering 2024-08-30T23:27:21.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:21.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:21 smithi136 ceph-mon[20207]: pgmap v389: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 32 MiB/s wr, 516 op/s; 20454/268413 objects misplaced (7.620%); 14 MiB/s, 223 objects/s recovering 2024-08-30T23:27:21.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:21.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:21 smithi173 ceph-mon[20554]: pgmap v389: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 32 MiB/s wr, 516 op/s; 20454/268413 objects misplaced (7.620%); 14 MiB/s, 223 objects/s recovering 2024-08-30T23:27:21.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:21.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:21.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:22.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:22.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:22.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 92074 92058 68.4896 57.0625 0.0273377 0.0145942 2024-08-30T23:27:22.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:22.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:22.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:22.335 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:22.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:22.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:22.530 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:22.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:22.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:22.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:22.734 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:22.742 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:22.743 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:22.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:22.874 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:22.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:22.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:23.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:23.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:23.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:23.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:23.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 92868 92852 68.2676 49.625 0.00722844 0.0146406 2024-08-30T23:27:23.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:23.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:23.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:23.386 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:23.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:23.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:23.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:23.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:23.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:23.775 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:23.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:23.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:23.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:24.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:24.032 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:24.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:24.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:24.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 93731 93715 68.101 53.9375 0.0300691 0.0146777 2024-08-30T23:27:24.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:23 smithi136 ceph-mon[20207]: osdmap e145: 12 total, 12 up, 11 in 2024-08-30T23:27:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:23 smithi136 ceph-mon[20207]: pgmap v391: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 621 op/s; 19588/273042 objects misplaced (7.174%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:27:24.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:23 smithi173 ceph-mon[20554]: osdmap e145: 12 total, 12 up, 11 in 2024-08-30T23:27:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:23 smithi173 ceph-mon[20554]: pgmap v391: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 621 op/s; 19588/273042 objects misplaced (7.174%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:27:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:23 smithi106 ceph-mon[16603]: osdmap e145: 12 total, 12 up, 11 in 2024-08-30T23:27:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:23 smithi106 ceph-mon[16603]: pgmap v391: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 621 op/s; 19588/273042 objects misplaced (7.174%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:27:24.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:24.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:24.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:24.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:24.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:24.678 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:24.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:24.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:24.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:24.944 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:24.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:25.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:25.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:25.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 94492 94476 67.8648 47.5625 0.0107411 0.0147302 2024-08-30T23:27:25.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:25.206 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:25.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:25.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.336 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:25.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:25.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:25.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:25.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:25 smithi106 ceph-mon[16603]: pgmap v392: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 636 op/s; 19588/276801 objects misplaced (7.077%); 17 MiB/s, 265 objects/s recovering 2024-08-30T23:27:25.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:25.733 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:25 smithi136 ceph-mon[20207]: pgmap v392: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 636 op/s; 19588/276801 objects misplaced (7.077%); 17 MiB/s, 265 objects/s recovering 2024-08-30T23:27:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:25 smithi173 ceph-mon[20554]: pgmap v392: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 40 MiB/s wr, 636 op/s; 19588/276801 objects misplaced (7.077%); 17 MiB/s, 265 objects/s recovering 2024-08-30T23:27:25.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:25.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:25.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:25.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:25.999 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:26.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:26.133 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:26.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 95317 95301 67.6795 51.5625 0.0040654 0.0147696 2024-08-30T23:27:26.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:26.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:26.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:26.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:27:26.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:26.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:26.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:26.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:26.686 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:26.686 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:26.687 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:26.687 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:26.687 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:26.687 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:26.687 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:26.687 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:26.702 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:26.702 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:26.708 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3479426008' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:26.708 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:26 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:26.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:26.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3479426008' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:26 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3479426008' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:26 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:26.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:26.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:26.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:27.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:27.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:27.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:27.166 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:27.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 96181 96165 67.5258 54 0.00372718 0.0148028 2024-08-30T23:27:27.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:27.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:27.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:27.338 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:27.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:27.372 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:27.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:27.406 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:27.406 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:27.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:27.445 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:27.445 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:27.485 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:27.485 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:27.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:27.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:27.524 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:27.527 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:27:27.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:27:27.527 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T23:27:27.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:27:27.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:27.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:27.564 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:27.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:27 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:27.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:27 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:27.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:27 smithi106 ceph-mon[16603]: osdmap e146: 12 total, 12 up, 11 in 2024-08-30T23:27:27.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3035021472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:27.664 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:27 smithi106 ceph-mon[16603]: pgmap v394: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 49 MiB/s wr, 789 op/s; 18724/282993 objects misplaced (6.616%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:27:27.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:27.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:27.715 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:27:27.715 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:27:27.715 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:27:27.715 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:27:27.715 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:27:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:27 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:27 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:27 smithi136 ceph-mon[20207]: osdmap e146: 12 total, 12 up, 11 in 2024-08-30T23:27:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3035021472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:27 smithi136 ceph-mon[20207]: pgmap v394: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 49 MiB/s wr, 789 op/s; 18724/282993 objects misplaced (6.616%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:27:27.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:27.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:27 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:27 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:27 smithi173 ceph-mon[20554]: osdmap e146: 12 total, 12 up, 11 in 2024-08-30T23:27:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3035021472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:27 smithi173 ceph-mon[20554]: pgmap v394: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 49 MiB/s wr, 789 op/s; 18724/282993 objects misplaced (6.616%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:27:27.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:27.955 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:27.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:28.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:28.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:28.169 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 7 97005 96998 67.3539 52.0625 0.0394006 0.0148418 2024-08-30T23:27:28.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:28.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:28.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:28.332 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:28.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:28.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:28.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:28.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:27:28.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:28.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:28.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:28.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:28.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:28.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:28.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:28.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:29.000 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:29.038 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:29.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:29.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.8659 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 97005 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 66.7226 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 23.8421 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 111.688 2024-08-30T23:27:29.170 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 1.3125 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1067 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 381.474 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 1787 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 21 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0148718 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.04058 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.27268 2024-08-30T23:27:29.171 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00101743 2024-08-30T23:27:29.175 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:27:29.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:29.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:29.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:29.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:29.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:29.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:29.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:27:29.530 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:29 smithi106 ceph-mon[16603]: pgmap v395: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 18278/288237 objects misplaced (6.341%); 17 MiB/s, 271 objects/s recovering 2024-08-30T23:27:29.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:29.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:29.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:29.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:29 smithi136 ceph-mon[20207]: pgmap v395: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 18278/288237 objects misplaced (6.341%); 17 MiB/s, 271 objects/s recovering 2024-08-30T23:27:29.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:29 smithi173 ceph-mon[20554]: pgmap v395: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 52 MiB/s wr, 825 op/s; 18278/288237 objects misplaced (6.341%); 17 MiB/s, 271 objects/s recovering 2024-08-30T23:27:29.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:29.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.14157.0:52 1.0 e22/20 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'34, mlcod=22'29)", 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 377.90568729, 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33784646499999998, 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.890 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977574+0000", 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7100000000000002e-07 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977579+0000", 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1080000000000001e-06 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.314904+0000", 2024-08-30T23:27:29.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33732482499999999 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.314915+0000", 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1576000000000001e-05 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315011+0000", 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5877000000000004e-05 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315031+0000", 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9718e-05 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315389+0000", 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000358184 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315420+0000", 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0605999999999998e-05 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:847 2.5 e104/97 2:ad44ea55:::benchmark_data_smithi173_47195_object846:head v 104'50, mlcod=104'47)", 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.585764+0000", 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 91.297496271, 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13449736400000001, 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.585764+0000", 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.585764+0000", 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.585765+0000", 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.585871+0000", 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010618100000000001 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.719729+0000", 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.133857804 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.719740+0000", 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0995999999999999e-05 2024-08-30T23:27:29.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.719808+0000", 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8380000000000006e-05 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.719828+0000", 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9843e-05 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.720222+0000", 2024-08-30T23:27:29.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039444799999999999 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.720262+0000", 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9048999999999998e-05 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14225 2.5 e109/97 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head v 109'898, mlcod=109'894)", 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 81.009250238999996, 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.239755616, 2024-08-30T23:27:29.899 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874011+0000", 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874108+0000", 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6961999999999998e-05 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112976+0000", 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23886803600000001 2024-08-30T23:27:29.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112983+0000", 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.951e-06 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113055+0000", 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113079+0000", 2024-08-30T23:27:29.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4754000000000001e-05 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113724+0000", 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064475700000000003 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113766+0000", 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1664e-05 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16156 2.4 e109/97 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head v 109'956, mlcod=109'953)", 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 79.785169542999995, 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213651339, 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098092+0000", 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4300000000000002e-07 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098114+0000", 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2234000000000001e-05 2024-08-30T23:27:29.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310963+0000", 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21284856199999999 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310969+0000", 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311028+0000", 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8857999999999998e-05 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311051+0000", 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2538999999999999e-05 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311702+0000", 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065106500000000004 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311742+0000", 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0318000000000003e-05 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:36489 2.1 e116/97 2:8364212d:::benchmark_data_smithi173_47195_object36488:head v 116'2298, mlcod=116'2295)", 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:25.098793+0000", 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.784467655, 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13870718600000001, 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.908 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.098793+0000", 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.098793+0000", 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.098794+0000", 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5000000000000001e-07 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.098814+0000", 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0710999999999999e-05 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.235991+0000", 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13717654400000001 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.235999+0000", 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8820000000000001e-06 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.236068+0000", 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8897000000000001e-05 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.236088+0000", 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0630999999999999e-05 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.237444+0000", 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013556519999999999 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.237500+0000", 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6019e-05 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.912 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:43074 2.8 e119/97 2:1e2a85c5:::benchmark_data_smithi173_47195_object43073:head v 119'2739, mlcod=119'2734)", 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:29.836143+0000", 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.047117880999998, 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14505273699999999, 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.836143+0000", 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.836143+0000", 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.836143+0000", 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3300000000000001e-07 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.836216+0000", 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2268000000000007e-05 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.979959+0000", 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14374328 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.979964+0000", 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2220000000000001e-06 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.980010+0000", 2024-08-30T23:27:29.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6140999999999998e-05 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.980024+0000", 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.385e-05 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.981160+0000", 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001135783 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.981195+0000", 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5460000000000003e-05 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:43075 2.8 e119/97 2:1bed56d6:::benchmark_data_smithi173_47195_object43074:head v 119'2740, mlcod=119'2734)", 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:29.837505+0000", 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.045755728000003, 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.146269762, 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.917 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.837505+0000", 2024-08-30T23:27:29.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.837505+0000", 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.837505+0000", 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2799999999999996e-07 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.837573+0000", 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8077999999999994e-05 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.982707+0000", 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14513340199999999 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.982711+0000", 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.459e-06 2024-08-30T23:27:29.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.982753+0000", 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1922999999999998e-05 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.982767+0000", 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4218e-05 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.983743+0000", 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00097579399999999999 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.983774+0000", 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1359999999999998e-05 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:43076 2.8 e119/97 2:176edc68:::benchmark_data_smithi173_47195_object43075:head v 119'2741, mlcod=119'2735)", 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:29.838448+0000", 2024-08-30T23:27:29.922 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.044812297, 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.149552255, 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.838448+0000", 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.838448+0000", 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.838449+0000", 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5900000000000002e-07 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.838511+0000", 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2526000000000004e-05 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.987083+0000", 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14857192599999999 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.987088+0000", 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7430000000000002e-06 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.987130+0000", 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2154999999999999e-05 2024-08-30T23:27:29.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.987144+0000", 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3893999999999999e-05 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.987971+0000", 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082670699999999998 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:29.988000+0000", 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9845e-05 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.254047782000001, 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207325907, 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.927 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629214+0000", 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.294e-06 2024-08-30T23:27:29.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629241+0000", 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6767999999999999e-05 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834158+0000", 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20491721900000001 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834164+0000", 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.268e-06 2024-08-30T23:27:29.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834227+0000", 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2712000000000005e-05 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2120000000000002e-05 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836507+0000", 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022580769999999998 2024-08-30T23:27:29.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836539+0000", 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1449000000000003e-05 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75317 2.5 2:a1907f80:::benchmark_data_smithi173_47195_object75316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.275930+0000", 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.607330549, 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13203245699999999, 2024-08-30T23:27:29.931 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75317 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.275930+0000", 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.275930+0000", 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.275930+0000", 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4499999999999997e-07 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.275975+0000", 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5067000000000001e-05 2024-08-30T23:27:29.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.275987+0000", 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1644000000000001e-05 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.275993+0000", 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8930000000000002e-06 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276777+0000", 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078352399999999996 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276785+0000", 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6600000000000001e-06 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.393836+0000", 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11705083199999999 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.393975+0000", 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013859399999999999 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.394088+0000", 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113736 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.401852+0000", 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077635890000000004 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.401874+0000", 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2161e-05 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.406738+0000", 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048641439999999999 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.406775+0000", 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7079000000000002e-05 2024-08-30T23:27:29.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.407962+0000", 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011869890000000001 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75323 2.5 2:adea3b0b:::benchmark_data_smithi173_47195_object75322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276003+0000", 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.607257873999998, 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13300339, 2024-08-30T23:27:29.938 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75323 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276003+0000", 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276003+0000", 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276005+0000", 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0260000000000002e-06 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276040+0000", 2024-08-30T23:27:29.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5831999999999998e-05 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276049+0000", 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5059999999999995e-06 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276054+0000", 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4410000000000001e-06 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276812+0000", 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00075739000000000004 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276816+0000", 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8639999999999998e-06 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.395868+0000", 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.119052359 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.396031+0000", 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016283700000000001 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.396219+0000", 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000188245 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.401881+0000", 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0056624279999999997 2024-08-30T23:27:29.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.401921+0000", 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9795000000000001e-05 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.408011+0000", 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006089957 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.408047+0000", 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5422000000000001e-05 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409006+0000", 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00095928799999999998 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75336 2.5 2:a4e80a46:::benchmark_data_smithi173_47195_object75335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276064+0000", 2024-08-30T23:27:29.945 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.607196831, 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13408351399999999, 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75336 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276064+0000", 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276064+0000", 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276064+0000", 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9299999999999998e-07 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276096+0000", 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1547999999999997e-05 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276104+0000", 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1419999999999993e-06 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276109+0000", 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2490000000000001e-06 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276848+0000", 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073916499999999996 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276858+0000", 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0037000000000001e-05 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.401953+0000", 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12509482399999999 2024-08-30T23:27:29.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.402117+0000", 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000163967 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.402293+0000", 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000175921 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409078+0000", 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006784857 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409124+0000", 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5893000000000003e-05 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409190+0000", 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5928999999999998e-05 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409223+0000", 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3244999999999998e-05 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.410147+0000", 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00092424399999999996 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75337 2.5 2:ad99f83b:::benchmark_data_smithi173_47195_object75336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276117+0000", 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.607143643000001, 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13526665500000001, 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75337 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.953 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276117+0000", 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276117+0000", 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276117+0000", 2024-08-30T23:27:29.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5699999999999998e-07 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276149+0000", 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2098000000000003e-05 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276157+0000", 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0099999999999995e-06 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276163+0000", 2024-08-30T23:27:29.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2270000000000001e-06 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276895+0000", 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073197500000000005 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276899+0000", 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9350000000000002e-06 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.404696+0000", 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12779671100000001 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.404802+0000", 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106091 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.404953+0000", 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015023599999999999 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409086+0000", 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041331079999999999 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.409158+0000", 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2836000000000005e-05 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.410209+0000", 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010504869999999999 2024-08-30T23:27:29.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.410262+0000", 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2834000000000003e-05 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.411383+0000", 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011216500000000001 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.960 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75394 2.5 2:aad820e4:::benchmark_data_smithi173_47195_object75393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.960 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276173+0000", 2024-08-30T23:27:29.960 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.607087180000001, 2024-08-30T23:27:29.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:29.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.155578191, 2024-08-30T23:27:29.961 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.961 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75394 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276173+0000", 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276173+0000", 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276174+0000", 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3700000000000001e-07 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276204+0000", 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0025000000000001e-05 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276212+0000", 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1189999999999997e-06 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276218+0000", 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5150000000000003e-06 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276936+0000", 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071749499999999996 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276944+0000", 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4610000000000008e-06 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.412338+0000", 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.135393969 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.412443+0000", 2024-08-30T23:27:29.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105191 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.412548+0000", 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010412300000000001 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417196+0000", 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046483449999999999 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421080+0000", 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038841650000000002 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.430565+0000", 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0094850170000000001 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.430606+0000", 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0756000000000001e-05 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.431751+0000", 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001145573 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75409 2.5 2:aa6bfbea:::benchmark_data_smithi173_47195_object75408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276226+0000", 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.607034785, 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18889718999999999, 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.968 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75409 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276226+0000", 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276226+0000", 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276226+0000", 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.34e-07 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276260+0000", 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4029000000000002e-05 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276269+0000", 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6219999999999998e-06 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276274+0000", 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5929999999999996e-06 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277000+0000", 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00072588700000000004 2024-08-30T23:27:29.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277009+0000", 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3629999999999994e-06 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.414793+0000", 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13778432500000001 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.414939+0000", 2024-08-30T23:27:29.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014597500000000001 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.415098+0000", 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015895199999999999 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421122+0000", 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00602368 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421162+0000", 2024-08-30T23:27:29.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0744000000000002e-05 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.431988+0000", 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010825391 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.432031+0000", 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3302e-05 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.465123+0000", 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.033091892999999997 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75428 2.5 2:af5d866e:::benchmark_data_smithi173_47195_object75427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.606977806, 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.189894692, 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.975 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75428 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9100000000000004e-07 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276309+0000", 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6095e-05 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276317+0000", 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7379999999999994e-06 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276323+0000", 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4280000000000001e-06 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277033+0000", 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070927499999999999 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277036+0000", 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7500000000000001e-06 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417234+0000", 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.140198079 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417328+0000", 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3068999999999999e-05 2024-08-30T23:27:29.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417423+0000", 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5353999999999995e-05 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421154+0000", 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037309819999999999 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421166+0000", 2024-08-30T23:27:29.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2213e-05 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.465172+0000", 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.044005711000000003 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.465209+0000", 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6876999999999999e-05 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.466177+0000", 2024-08-30T23:27:29.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00096862999999999995 2024-08-30T23:27:29.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.982 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75451 2.5 2:ac73061a:::benchmark_data_smithi173_47195_object75450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276330+0000", 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.606930143, 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.190842071, 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75451 2024-08-30T23:27:29.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276330+0000", 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276330+0000", 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276331+0000", 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3599999999999999e-07 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276355+0000", 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.419e-05 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276363+0000", 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2349999999999999e-06 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276369+0000", 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3400000000000003e-06 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277061+0000", 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069180900000000004 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277070+0000", 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2220000000000002e-06 2024-08-30T23:27:29.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.419135+0000", 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14206534200000001 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.419271+0000", 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013651000000000001 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.419420+0000", 2024-08-30T23:27:29.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000148851 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421212+0000", 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017922249999999999 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.431930+0000", 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010717844000000001 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.466225+0000", 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034295063000000001 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.466261+0000", 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5478999999999997e-05 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.467172+0000", 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00091152500000000005 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.990 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:83177 2.c e140/97 2:3a9201b6:::benchmark_data_smithi173_47195_object83176:head v 140'5172, mlcod=140'5165)", 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:08.607729+0000", 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.275531661999999, 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.162053423, 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.607729+0000", 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.607729+0000", 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.607730+0000", 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8000000000000005e-07 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.609693+0000", 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019635529999999998 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.766787+0000", 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15709358600000001 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.766793+0000", 2024-08-30T23:27:29.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8000000000000001e-06 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.766849+0000", 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5720000000000002e-05 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.766875+0000", 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5837999999999999e-05 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.769736+0000", 2024-08-30T23:27:29.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028606 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.769782+0000", 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6545999999999998e-05 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:83225 2.c e140/97 2:37152ee2:::benchmark_data_smithi173_47195_object83224:head v 140'5173, mlcod=140'5169)", 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:08.665974+0000", 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 21.217286858000001, 2024-08-30T23:27:29.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12911102599999999, 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.665974+0000", 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.665974+0000", 2024-08-30T23:27:29.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.665974+0000", 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0299999999999998e-07 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.666028+0000", 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3860999999999998e-05 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:29.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.792100+0000", 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12607203 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.792107+0000", 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0079999999999996e-06 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.792163+0000", 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5823999999999999e-05 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:29.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.792186+0000", 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2997e-05 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.795041+0000", 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002854611 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:08.795085+0000", 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3891999999999999e-05 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:29.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.385494921, 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:27:30.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:27:30.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:30.003 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:27:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:30.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:30.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:30.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:30.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:30.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:30.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:30.418 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:30.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:30.541 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:30.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:30.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:30.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2644365398' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:27:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.798 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:30.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2644365398' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:27:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2644365398' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:27:30.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:30.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:30.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:30.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:27:31.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:31.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:31.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:31.200 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:31.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:31.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:31.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:31.458 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 146, 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:27:26.445925+0000", 2024-08-30T23:27:31.546 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:24:45.463540+0000", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:27:04.283754+0000", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5806080, 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "nodeep-scrub", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "noscrub", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:27:31.547 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 26, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 2, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "jewel", 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.548 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:27:31.549 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 1, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 1, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 1, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 1, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 1, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:27:31.550 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "23", 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:27:31.551 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:27:31.552 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:27:31.553 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:27:31.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 12, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 12, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 0.25, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3, 2024-08-30T23:27:31.555 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.9441031813621521, 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.93999999761581421, 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 2, 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_0", 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:25:48.724973+0000", 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:27:31.556 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:27:31.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "103", 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:27:31.558 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:27:31.559 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:27:31.560 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.561 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 2.2400000095367432, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 2.2400000095367432, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 0.92000001668930054, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.059999942779541, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.059999942779541, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.9441031813621521, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.93999999761581421, 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:27:31.562 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 143, 2024-08-30T23:27:31.563 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:27:31.568 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.569 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 13, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 142, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:27:31.570 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.573 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/1861847252", 2024-08-30T23:27:31.574 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/1861847252", 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/1861847252", 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/1861847252", 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.575 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 97, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:27:31.576 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.578 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.579 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:27:31.580 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.581 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 0, 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.582 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.583 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:27:31.585 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:27:31.586 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.587 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 0, 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.588 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:27:31.590 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:27:31.593 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 39, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 97, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.594 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.596 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.597 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.598 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2282248833", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2282248833", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2282248833", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2282248833", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 138, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.600 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.602 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.603 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:27:31.605 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 138, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.606 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.608 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:27:31.609 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.610 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.611 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.64862060546875, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:27:31.612 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 142, 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.613 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:27:31.614 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:27:31.615 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.616 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.617 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.618 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 97, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:27:31.621 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:27:31.622 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.623 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 0, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 0, 2024-08-30T23:27:31.624 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.625 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.626 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.627 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.628 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 133, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:27:31.630 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.631 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.632 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.634 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.635 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:25.229015+0000", 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.636 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:33.498575+0000", 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.637 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:27:00.496155+0000", 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:19.146741+0000", 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:59.925461+0000", 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.639 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:15.489673+0000", 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:27:06.535948+0000", 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:52.221552+0000", 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:27:31.641 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.642 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.642 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.642 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.642 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.642 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:27:10.204811+0000", 2024-08-30T23:27:31.642 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:12.366807+0000", 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.643 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 65536, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:26:57.894060+0000", 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:27:31.644 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:27:22.858838+0000", 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [], 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [ 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "2.0", 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: 7, 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: 9 2024-08-30T23:27:31.645 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "2.5", 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: 4 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "2.7", 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-30T23:27:31.646 INFO:teuthology.orchestra.run.smithi106.stdout: 2 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "2.b", 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: 11, 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: 7 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:27:31.647 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:27:31.648 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:27:31.649 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:27:31.650 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:27:31.650 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:27:31.650 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:27:31.650 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:31.650 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:27:31.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:31.681 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:31.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:31 smithi136 ceph-mon[20207]: pgmap v396: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 121 B/s rd, 47 MiB/s wr, 805 op/s; 17858/288966 objects misplaced (6.180%); 16 MiB/s, 257 objects/s recovering 2024-08-30T23:27:31.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:31 smithi173 ceph-mon[20554]: pgmap v396: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 121 B/s rd, 47 MiB/s wr, 805 op/s; 17858/288966 objects misplaced (6.180%); 16 MiB/s, 257 objects/s recovering 2024-08-30T23:27:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:31 smithi106 ceph-mon[16603]: pgmap v396: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 121 B/s rd, 47 MiB/s wr, 805 op/s; 17858/288966 objects misplaced (6.180%); 16 MiB/s, 257 objects/s recovering 2024-08-30T23:27:31.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:31.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:31.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:32.038 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:32.068 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:32.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:32.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:32.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:32.418 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:32.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:32.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:32.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:32.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:32.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:32 smithi136 ceph-mon[20207]: osdmap e147: 12 total, 12 up, 11 in 2024-08-30T23:27:32.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:32 smithi173 ceph-mon[20554]: osdmap e147: 12 total, 12 up, 11 in 2024-08-30T23:27:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:32.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:32 smithi106 ceph-mon[16603]: osdmap e147: 12 total, 12 up, 11 in 2024-08-30T23:27:32.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:32.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:32.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:33.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:33.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:33.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:33.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:33.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:33.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:33.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:33.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:33.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:33.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:33.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:33.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:33 smithi136 ceph-mon[20207]: pgmap v398: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 37 MiB/s wr, 790 op/s; 16890/286263 objects misplaced (5.900%); 21 MiB/s, 337 objects/s recovering 2024-08-30T23:27:33.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:33 smithi173 ceph-mon[20554]: pgmap v398: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 37 MiB/s wr, 790 op/s; 16890/286263 objects misplaced (5.900%); 21 MiB/s, 337 objects/s recovering 2024-08-30T23:27:33.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:33 smithi106 ceph-mon[16603]: pgmap v398: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 37 MiB/s wr, 790 op/s; 16890/286263 objects misplaced (5.900%); 21 MiB/s, 337 objects/s recovering 2024-08-30T23:27:33.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:33.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:33.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:34.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:34.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:34.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:34.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:34.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:34.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:34.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:34.482 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:34.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:34.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:34.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:34.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:34.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:34 smithi136 ceph-mon[20207]: osdmap e148: 12 total, 12 up, 11 in 2024-08-30T23:27:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:27:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:34 smithi173 ceph-mon[20554]: osdmap e148: 12 total, 12 up, 11 in 2024-08-30T23:27:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:27:34.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:34.853 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:34.853 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:34.853 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:34 smithi106 ceph-mon[16603]: osdmap e148: 12 total, 12 up, 11 in 2024-08-30T23:27:34.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:27:34.887 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:34.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:34.904 INFO:tasks.thrashosds.thrasher:j is {'epoch': 146, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:27:26.445925+0000', 'last_up_change': '2024-08-30T23:24:45.463540+0000', 'last_in_change': '2024-08-30T23:27:04.283754+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9441031813621521, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-08-30T23:25:48.724973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9441031813621521, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 1861847252}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1861847252}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 1861847252}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1861847252}]}, 'public_addr': '172.21.15.106:6811/1861847252', 'cluster_addr': '172.21.15.106:6813/1861847252', 'heartbeat_back_addr': '172.21.15.106:6817/1861847252', 'heartbeat_front_addr': '172.21.15.106:6815/1861847252', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2282248833}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2282248833}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2282248833}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2282248833}]}, 'public_addr': '172.21.15.136:6809/2282248833', 'cluster_addr': '172.21.15.136:6811/2282248833', 'heartbeat_back_addr': '172.21.15.136:6815/2282248833', 'heartbeat_front_addr': '172.21.15.136:6813/2282248833', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 0.64862060546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:25.229015+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:33.498575+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:27:00.496155+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:19.146741+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:59.925461+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:15.489673+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:27:06.535948+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:52.221552+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:27:10.204811+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:26:12.366807+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-08-30T23:26:57.894060+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:27:22.858838+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '2.0', 'osds': [7, 9]}, {'pgid': '2.5', 'osds': [0, 4]}, {'pgid': '2.7', 'osds': [6, 2]}, {'pgid': '2.b', 'osds': [11, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:27:34.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:27:34.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:35.006 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:35.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:35.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:35.154 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:35.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:35.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:35.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:35.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:35.494 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:35.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:27:35.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:35.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:35.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:35.661 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:35.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:27:35.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:27:35.758 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:35 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:27:35.758 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:35 smithi106 ceph-mon[16603]: pgmap v400: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 21 MiB/s wr, 679 op/s; 16890/282747 objects misplaced (5.974%); 14 MiB/s, 229 objects/s recovering 2024-08-30T23:27:35.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:35.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:27:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:27:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:35 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:27:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:35 smithi136 ceph-mon[20207]: pgmap v400: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 21 MiB/s wr, 679 op/s; 16890/282747 objects misplaced (5.974%); 14 MiB/s, 229 objects/s recovering 2024-08-30T23:27:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:27:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:27:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:35 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:27:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:35 smithi173 ceph-mon[20554]: pgmap v400: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 127 B/s rd, 21 MiB/s wr, 679 op/s; 16890/282747 objects misplaced (5.974%); 14 MiB/s, 229 objects/s recovering 2024-08-30T23:27:35.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:35.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:35.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:36.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:36.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:36.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:36.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:36.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:36.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:36.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:36.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:36.448 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:36.500 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:36.500 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:36.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:36.539 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:36.539 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:36.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:36.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:36.577 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:36.611 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:36.611 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:36.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:36.648 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:36.648 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:36.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:36.682 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:36.682 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:36.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:36.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:36.745 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:36.745 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:36.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:36.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:36.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:36.819 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:36.828 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:36.829 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:36.894 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:36.894 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:36.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:36.934 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:36.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:36.960 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:36.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:37.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:37.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:37.149 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:27:37.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:37.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:37.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:37.325 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:37.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/776147584' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:27:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/939826793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:37.374 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:27:37.375 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:27:37.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.486 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:37.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/776147584' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:27:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/939826793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/776147584' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:27:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/939826793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:37.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:37.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:37.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:37.743 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:37.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:37.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:37.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:37.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:37.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:38.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.131 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:38.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:38.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:38.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:38.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:38.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:38.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:38 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:38.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/776147584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:27:38.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:38 smithi106 ceph-mon[16603]: osdmap e149: 12 total, 12 up, 11 in 2024-08-30T23:27:38.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:38 smithi106 ceph-mon[16603]: pgmap v402: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.0 MiB/s wr, 746 op/s; 16039/276693 objects misplaced (5.797%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:27:38.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:38 smithi106 ceph-mon[16603]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:27:38.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:38.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:38 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/776147584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:27:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:38 smithi136 ceph-mon[20207]: osdmap e149: 12 total, 12 up, 11 in 2024-08-30T23:27:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:38 smithi136 ceph-mon[20207]: pgmap v402: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.0 MiB/s wr, 746 op/s; 16039/276693 objects misplaced (5.797%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:27:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:38 smithi136 ceph-mon[20207]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:27:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:38 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/776147584' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:27:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:38 smithi173 ceph-mon[20554]: osdmap e149: 12 total, 12 up, 11 in 2024-08-30T23:27:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:38 smithi173 ceph-mon[20554]: pgmap v402: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.0 MiB/s wr, 746 op/s; 16039/276693 objects misplaced (5.797%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:27:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:38 smithi173 ceph-mon[20554]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:27:38.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:38.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:38.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:38.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:38.805 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-08-30T23:27:37.360605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6044844600,"num_objects":92231,"num_object_clones":0,"num_object_copies":276693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16039,"num_objects_unfound":0,"num_objects_dirty":92231,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":101906,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8534,"num_bytes_recovered":560477792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101875,"ondisk_log_size":101875,"up":51,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18068032,"kb_used_data":17397236,"kb_used_omap":17,"kb_used_meta":670446,"kb_avail":1013128640,"statfs":{"total":1055945392128,"available":1037443727360,"internally_reserved":0,"allocated":17814769664,"data_stored":17811778421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":686537593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,2,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-268107776,"num_objects":-4091,"num_object_clones":0,"num_object_copies":-12273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1819,"num_objects_unfound":0,"num_objects_dirty":-4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":12416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1819,"num_bytes_recovered":119209984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001267"},"pg_stats":[{"pgid":"2.f","version":"148'6453","reported_seq":6621,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:35.591029+0000","last_change":"2024-08-30T23:27:02.383340+0000","last_active":"2024-08-30T23:27:35.591029+0000","last_peered":"2024-08-30T23:27:35.591029+0000","last_clean":"2024-08-30T23:27:35.591029+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:27:35.591029+0000","last_undegraded":"2024-08-30T23:27:35.591029+0000","last_fullsized":"2024-08-30T23:27:35.591029+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'4915","last_scrub_stamp":"2024-08-30T23:27:02.383046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:02.383046+0000","objects_scrubbed":4881,"log_size":6453,"log_dups_size":0,"ondisk_log_size":6453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:28:14.757359+0000","scrub_duration":1620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376897536,"num_objects":5751,"num_object_clones":0,"num_object_copies":17253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"147'6174","reported_seq":6285,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-08-30T23:27:33.354103+0000","last_change":"2024-08-30T23:25:50.474312+0000","last_active":"2024-08-30T23:27:33.354103+0000","last_peered":"2024-08-30T23:27:33.354103+0000","last_clean":"2024-08-30T23:27:33.354103+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:27:33.354103+0000","last_undegraded":"2024-08-30T23:27:33.354103+0000","last_fullsized":"2024-08-30T23:27:33.354103+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377749504,"num_objects":5764,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6174,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"148'6403","reported_seq":6515,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:34.993727+0000","last_change":"2024-08-30T23:25:50.474455+0000","last_active":"2024-08-30T23:27:34.993727+0000","last_peered":"2024-08-30T23:27:34.993727+0000","last_clean":"2024-08-30T23:27:34.993727+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:27:34.993727+0000","last_undegraded":"2024-08-30T23:27:34.993727+0000","last_fullsized":"2024-08-30T23:27:34.993727+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380895256,"num_objects":5813,"num_object_clones":0,"num_object_copies":17439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6403,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"148'6351","reported_seq":6463,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:35.984231+0000","last_change":"2024-08-30T23:25:50.532163+0000","last_active":"2024-08-30T23:27:35.984231+0000","last_peered":"2024-08-30T23:27:35.984231+0000","last_clean":"2024-08-30T23:27:35.984231+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:27:35.984231+0000","last_undegraded":"2024-08-30T23:27:35.984231+0000","last_fullsized":"2024-08-30T23:27:35.984231+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6351,"log_dups_size":0,"ondisk_log_size":6351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371523584,"num_objects":5669,"num_object_clones":0,"num_object_copies":17007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6351,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":249,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-08-30T23:27:37.163419+0000","last_change":"2024-08-30T23:26:04.139978+0000","last_active":"2024-08-30T23:27:37.163419+0000","last_peered":"2024-08-30T23:27:37.163419+0000","last_clean":"2024-08-30T23:27:37.163419+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:27:37.163419+0000","last_undegraded":"2024-08-30T23:27:37.163419+0000","last_fullsized":"2024-08-30T23:27:37.163419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:26:04.139930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:26:04.139930+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"148'6426","reported_seq":6539,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:33.532215+0000","last_change":"2024-08-30T23:25:50.452633+0000","last_active":"2024-08-30T23:27:33.532215+0000","last_peered":"2024-08-30T23:27:33.532215+0000","last_clean":"2024-08-30T23:27:33.532215+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:27:33.532215+0000","last_undegraded":"2024-08-30T23:27:33.532215+0000","last_fullsized":"2024-08-30T23:27:33.532215+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":17706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"149'6528","reported_seq":9813,"reported_epoch":149,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:27:37.306096+0000","last_change":"2024-08-30T23:27:05.648657+0000","last_active":"2024-08-30T23:27:37.306096+0000","last_peered":"2024-08-30T23:27:37.306096+0000","last_clean":"2024-08-30T23:27:04.298160+0000","last_became_active":"2024-08-30T23:27:05.622268+0000","last_became_peered":"2024-08-30T23:27:05.622268+0000","last_unstale":"2024-08-30T23:27:37.306096+0000","last_undegraded":"2024-08-30T23:27:37.306096+0000","last_fullsized":"2024-08-30T23:27:05.496904+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372244480,"num_objects":5680,"num_object_clones":0,"num_object_copies":17040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3485,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1583,"num_bytes_recovered":103743488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"7,9","objects":2195},{"shards":"7,9,10","objects":3485}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"147'6326","reported_seq":6437,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-08-30T23:27:33.316523+0000","last_change":"2024-08-30T23:25:50.531742+0000","last_active":"2024-08-30T23:27:33.316523+0000","last_peered":"2024-08-30T23:27:33.316523+0000","last_clean":"2024-08-30T23:27:33.316523+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:27:33.316523+0000","last_undegraded":"2024-08-30T23:27:33.316523+0000","last_fullsized":"2024-08-30T23:27:33.316523+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220608,"num_objects":5878,"num_object_clones":0,"num_object_copies":17634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6326,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"148'6324","reported_seq":6437,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:33.529041+0000","last_change":"2024-08-30T23:25:50.451424+0000","last_active":"2024-08-30T23:27:33.529041+0000","last_peered":"2024-08-30T23:27:33.529041+0000","last_clean":"2024-08-30T23:27:33.529041+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:27:33.529041+0000","last_undegraded":"2024-08-30T23:27:33.529041+0000","last_fullsized":"2024-08-30T23:27:33.529041+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384040960,"num_objects":5860,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"147'6239","reported_seq":8355,"reported_epoch":147,"state":"active+recovering","last_fresh":"2024-08-30T23:27:33.206334+0000","last_change":"2024-08-30T23:27:15.230107+0000","last_active":"2024-08-30T23:27:33.206334+0000","last_peered":"2024-08-30T23:27:33.206334+0000","last_clean":"2024-08-30T23:27:04.280847+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:27:33.206334+0000","last_undegraded":"2024-08-30T23:27:33.206334+0000","last_fullsized":"2024-08-30T23:27:33.206334+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6239,"log_dups_size":0,"ondisk_log_size":6239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382926848,"num_objects":5843,"num_object_clones":0,"num_object_copies":17529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4071,"num_objects_unfound":0,"num_objects_dirty":5843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6239,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":65536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":["0","1","10"],"object_location_counts":[{"shards":"0,1,2","objects":1772},{"shards":"0,1,10","objects":4071}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"147'6263","reported_seq":10218,"reported_epoch":147,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:27:33.213017+0000","last_change":"2024-08-30T23:26:58.430607+0000","last_active":"2024-08-30T23:27:33.213017+0000","last_peered":"2024-08-30T23:27:33.213017+0000","last_clean":"2024-08-30T23:26:56.218662+0000","last_became_active":"2024-08-30T23:26:58.383273+0000","last_became_peered":"2024-08-30T23:26:58.383273+0000","last_unstale":"2024-08-30T23:27:33.213017+0000","last_undegraded":"2024-08-30T23:27:33.213017+0000","last_fullsized":"2024-08-30T23:26:58.259133+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6263,"log_dups_size":0,"ondisk_log_size":6263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381485056,"num_objects":5821,"num_object_clones":0,"num_object_copies":17463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2787,"num_objects_unfound":0,"num_objects_dirty":5821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6263,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1921,"num_bytes_recovered":125894656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4],"avail_no_missing":["0","4","8"],"object_location_counts":[{"shards":"0,4","objects":3034},{"shards":"0,4,8","objects":2787}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"148'6414","reported_seq":6535,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:35.603982+0000","last_change":"2024-08-30T23:27:14.160376+0000","last_active":"2024-08-30T23:27:35.603982+0000","last_peered":"2024-08-30T23:27:35.603982+0000","last_clean":"2024-08-30T23:27:35.603982+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:27:35.603982+0000","last_undegraded":"2024-08-30T23:27:35.603982+0000","last_fullsized":"2024-08-30T23:27:35.603982+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6414,"log_dups_size":0,"ondisk_log_size":6414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377225216,"num_objects":5756,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6414,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"148'6454","reported_seq":9281,"reported_epoch":148,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:27:36.819542+0000","last_change":"2024-08-30T23:27:05.694228+0000","last_active":"2024-08-30T23:27:36.819542+0000","last_peered":"2024-08-30T23:27:36.819542+0000","last_clean":"2024-08-30T23:27:04.288783+0000","last_became_active":"2024-08-30T23:27:05.664826+0000","last_became_peered":"2024-08-30T23:27:05.664826+0000","last_unstale":"2024-08-30T23:27:36.819542+0000","last_undegraded":"2024-08-30T23:27:36.819542+0000","last_fullsized":"2024-08-30T23:27:05.499086+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6454,"log_dups_size":0,"ondisk_log_size":6454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372768768,"num_objects":5688,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3702,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6454,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1355,"num_bytes_recovered":88801280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2],"avail_no_missing":["6","2","10"],"object_location_counts":[{"shards":"2,6","objects":1986},{"shards":"2,6,10","objects":3702}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"149'6428","reported_seq":6543,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-08-30T23:27:37.253720+0000","last_change":"2024-08-30T23:25:50.531975+0000","last_active":"2024-08-30T23:27:37.253720+0000","last_peered":"2024-08-30T23:27:37.253720+0000","last_clean":"2024-08-30T23:27:37.253720+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:27:37.253720+0000","last_undegraded":"2024-08-30T23:27:37.253720+0000","last_fullsized":"2024-08-30T23:27:37.253720+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368312320,"num_objects":5620,"num_object_clones":0,"num_object_copies":16860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6428,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"147'6334","reported_seq":6444,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-08-30T23:27:32.364943+0000","last_change":"2024-08-30T23:25:50.474563+0000","last_active":"2024-08-30T23:27:32.364943+0000","last_peered":"2024-08-30T23:27:32.364943+0000","last_clean":"2024-08-30T23:27:32.364943+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:27:32.364943+0000","last_undegraded":"2024-08-30T23:27:32.364943+0000","last_fullsized":"2024-08-30T23:27:32.364943+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6334,"log_dups_size":0,"ondisk_log_size":6334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389939200,"num_objects":5950,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"148'6258","reported_seq":6370,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-08-30T23:27:36.049005+0000","last_change":"2024-08-30T23:25:50.474244+0000","last_active":"2024-08-30T23:27:36.049005+0000","last_peered":"2024-08-30T23:27:36.049005+0000","last_clean":"2024-08-30T23:27:36.049005+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:27:36.049005+0000","last_undegraded":"2024-08-30T23:27:36.049005+0000","last_fullsized":"2024-08-30T23:27:36.049005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6258,"log_dups_size":0,"ondisk_log_size":6258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364773376,"num_objects":5566,"num_object_clones":0,"num_object_copies":16698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6258,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"148'6408","reported_seq":11859,"reported_epoch":148,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:27:36.073573+0000","last_change":"2024-08-30T23:26:58.523686+0000","last_active":"2024-08-30T23:27:36.073573+0000","last_peered":"2024-08-30T23:27:36.073573+0000","last_clean":"2024-08-30T23:26:56.199514+0000","last_became_active":"2024-08-30T23:26:58.491083+0000","last_became_peered":"2024-08-30T23:26:58.491083+0000","last_unstale":"2024-08-30T23:27:36.073573+0000","last_undegraded":"2024-08-30T23:27:36.073573+0000","last_fullsized":"2024-08-30T23:26:58.258623+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371458048,"num_objects":5668,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1994,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6408,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2668,"num_bytes_recovered":174850048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,5],"acting":[11,7],"avail_no_missing":["11","7","8"],"object_location_counts":[{"shards":"7,8,11","objects":1994},{"shards":"7,11","objects":3674}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6044254232,"num_objects":92229,"num_object_clones":0,"num_object_copies":276687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16039,"num_objects_unfound":0,"num_objects_dirty":92229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101783,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8528,"num_bytes_recovered":558891008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18731577344,"data_stored":18731565128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101783,"ondisk_log_size":101783,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187493,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301020,"kb_used_data":1229576,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92444132,"statfs":{"total":95995035648,"available":94662791168,"internally_reserved":0,"allocated":1259085824,"data_stored":1258810813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449131,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710767,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273816,"kb_used_data":2206532,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91471336,"statfs":{"total":95995035648,"available":93666648064,"internally_reserved":0,"allocated":2259488768,"data_stored":2259213257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972405,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1390224,"kb_used_data":1336832,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92354928,"statfs":{"total":95995035648,"available":94571446272,"internally_reserved":0,"allocated":1368915968,"data_stored":1368644529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1879784,"kb_used_data":1816660,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91865368,"statfs":{"total":95995035648,"available":94070136832,"internally_reserved":0,"allocated":1860259840,"data_stored":1859980417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528382,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766620,"kb_used_data":735432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92978532,"statfs":{"total":95995035648,"available":95210016768,"internally_reserved":0,"allocated":753082368,"data_stored":752807377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1028744,"kb_used_data":983604,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92716408,"statfs":{"total":95995035648,"available":94941601792,"internally_reserved":0,"allocated":1007210496,"data_stored":1006943439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1174292,"kb_used_data":1116676,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92570860,"statfs":{"total":95995035648,"available":94792560640,"internally_reserved":0,"allocated":1143476224,"data_stored":1143200713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2446924,"kb_used_data":2375480,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91298228,"statfs":{"total":95995035648,"available":93489385472,"internally_reserved":0,"allocated":2432491520,"data_stored":2432220391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690820,"kb_used_data":1619380,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92054332,"statfs":{"total":95995035648,"available":94263635968,"internally_reserved":0,"allocated":1658245120,"data_stored":1657978063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2161096,"kb_used_data":2093812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91584056,"statfs":{"total":95995035648,"available":93782073344,"internally_reserved":0,"allocated":2144063488,"data_stored":2143796431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1954692,"kb_used_data":1883252,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91790460,"statfs":{"total":95995035648,"available":93993431040,"internally_reserved":0,"allocated":1928450048,"data_stored":1928182991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1911816192,"data_stored":1911816192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2131886080,"data_stored":2131886080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1649147904,"data_stored":1649147904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2419462144,"data_stored":2419458072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1140330496,"data_stored":1140326424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1004208128,"data_stored":1004208128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":748224512,"data_stored":748224512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1858338816,"data_stored":1858338816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1368195072,"data_stored":1368195072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2253262848,"data_stored":2253258776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":995885056,"data_stored":995885056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1250820096,"data_stored":1250820096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:27:38.807 INFO:tasks.thrashosds.thrasher:Setting 2.4 to [1, 10, 8] 2024-08-30T23:27:38.807 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.4', '1', '10', '8'] 2024-08-30T23:27:38.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 2.4 1 10 8 2024-08-30T23:27:38.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:38.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:38.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:39.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:39.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:39.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:39.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:39.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:39.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:39.301 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:39.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:39.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:39.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.558 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:39.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:39.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:39.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:39.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:39.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:39.850 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:39.850 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:39.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:39.942 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:39.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:40.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:40.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:40.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:40.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.193 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:40.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:40.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:40.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:27:40.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:40.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:40 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:40 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:40 smithi136 ceph-mon[20207]: osdmap e150: 12 total, 12 up, 11 in 2024-08-30T23:27:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:40 smithi136 ceph-mon[20207]: pgmap v404: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 15584/271866 objects misplaced (5.732%); 14 MiB/s, 217 objects/s recovering 2024-08-30T23:27:40.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:40 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:40.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:40 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:40.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:40 smithi106 ceph-mon[16603]: osdmap e150: 12 total, 12 up, 11 in 2024-08-30T23:27:40.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:40 smithi106 ceph-mon[16603]: pgmap v404: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 15584/271866 objects misplaced (5.732%); 14 MiB/s, 217 objects/s recovering 2024-08-30T23:27:40.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:40.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:40.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:40 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:40 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:40 smithi173 ceph-mon[20554]: osdmap e150: 12 total, 12 up, 11 in 2024-08-30T23:27:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:40 smithi173 ceph-mon[20554]: pgmap v404: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 15584/271866 objects misplaced (5.732%); 14 MiB/s, 217 objects/s recovering 2024-08-30T23:27:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:40.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:27:40.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.715 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:40.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:40.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:40.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:40.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:40.985 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:40.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:41.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:41.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:41.147 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:41.170 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:41.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:41.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:27:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:41.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:41.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:41.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:41.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:41.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:41.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:41.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:41.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:41.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:41.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:41.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:41.876 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:41.877 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:41.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:41.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:41.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:41.916 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:41.916 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:41.955 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:41.955 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:41.995 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:41.995 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:42.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:42.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:42.072 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:42.072 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:42.107 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:42.107 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:42.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:42.146 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:42.146 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:42.152 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:42.177 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:42.178 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:42.214 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:42.214 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:42.248 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:42.249 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:42.249 INFO:teuthology.orchestra.run.smithi106.stderr:set 2.4 pg_upmap mapping to [1,10,8] 2024-08-30T23:27:42.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:42.266 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:27:42.267 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:27:42.267 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:27:42.267 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:27:42.267 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:27:42.273 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:42.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:42.290 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:42.290 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:42.330 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:42.330 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:42.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:42 smithi106 ceph-mon[16603]: osdmap e151: 12 total, 12 up, 11 in 2024-08-30T23:27:42.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:42 smithi106 ceph-mon[16603]: pgmap v406: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 844 op/s; 15206/267534 objects misplaced (5.684%); 18 MiB/s, 280 objects/s recovering 2024-08-30T23:27:42.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1538618424' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]} : dispatch 2024-08-30T23:27:42.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:42 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]} : dispatch 2024-08-30T23:27:42.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:42 smithi106 ceph-mon[16603]: 2.6 scrub starts 2024-08-30T23:27:42.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3320702015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:42.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:42.408 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:42.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:42.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:42 smithi136 ceph-mon[20207]: osdmap e151: 12 total, 12 up, 11 in 2024-08-30T23:27:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:42 smithi136 ceph-mon[20207]: pgmap v406: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 844 op/s; 15206/267534 objects misplaced (5.684%); 18 MiB/s, 280 objects/s recovering 2024-08-30T23:27:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1538618424' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]} : dispatch 2024-08-30T23:27:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:42 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]} : dispatch 2024-08-30T23:27:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:42 smithi136 ceph-mon[20207]: 2.6 scrub starts 2024-08-30T23:27:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3320702015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:42 smithi173 ceph-mon[20554]: osdmap e151: 12 total, 12 up, 11 in 2024-08-30T23:27:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:42 smithi173 ceph-mon[20554]: pgmap v406: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 844 op/s; 15206/267534 objects misplaced (5.684%); 18 MiB/s, 280 objects/s recovering 2024-08-30T23:27:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1538618424' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]} : dispatch 2024-08-30T23:27:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:42 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]} : dispatch 2024-08-30T23:27:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:42 smithi173 ceph-mon[20554]: 2.6 scrub starts 2024-08-30T23:27:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3320702015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:42.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:42.668 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:42.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:42.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:42.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:42.928 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:42.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:43.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:43.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:43.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:43.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:43.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:43.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:43.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:43.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:43.504 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:43 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]}]': finished 2024-08-30T23:27:43.504 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:43 smithi106 ceph-mon[16603]: osdmap e152: 12 total, 12 up, 11 in 2024-08-30T23:27:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:43 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]}]': finished 2024-08-30T23:27:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:43 smithi136 ceph-mon[20207]: osdmap e152: 12 total, 12 up, 11 in 2024-08-30T23:27:43.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:43.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:43 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.4", "id": [1, 10, 8]}]': finished 2024-08-30T23:27:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:43 smithi173 ceph-mon[20554]: osdmap e152: 12 total, 12 up, 11 in 2024-08-30T23:27:43.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:43.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.695 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:43.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:43.836 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:43.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:43.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:43.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:44.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:44.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:44.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:27:44.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:44.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:44.265 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:27:44.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:44.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:44 smithi106 ceph-mon[16603]: 2.6 scrub ok 2024-08-30T23:27:44.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:44 smithi106 ceph-mon[16603]: pgmap v408: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 14295/263139 objects misplaced (5.432%); 18 MiB/s, 290 objects/s recovering 2024-08-30T23:27:44.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1539439269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:44.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:44 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:44.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:44.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:44.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:44 smithi136 ceph-mon[20207]: 2.6 scrub ok 2024-08-30T23:27:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:44 smithi136 ceph-mon[20207]: pgmap v408: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 14295/263139 objects misplaced (5.432%); 18 MiB/s, 290 objects/s recovering 2024-08-30T23:27:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1539439269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:44 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:44 smithi173 ceph-mon[20554]: 2.6 scrub ok 2024-08-30T23:27:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:44 smithi173 ceph-mon[20554]: pgmap v408: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 14295/263139 objects misplaced (5.432%); 18 MiB/s, 290 objects/s recovering 2024-08-30T23:27:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1539439269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:44 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:44.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:44.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:44.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:44.759 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:44.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:44.890 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:44.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:44.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:45.014 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:45.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:45.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:45.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:45.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:45.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:45.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:45.481 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:45.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:45 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:27:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:45 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:27:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:45 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:45 smithi136 ceph-mon[20207]: osdmap e153: 12 total, 12 up, 11 in 2024-08-30T23:27:45.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:45 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:27:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:45 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:27:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:45 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:45 smithi173 ceph-mon[20554]: osdmap e153: 12 total, 12 up, 11 in 2024-08-30T23:27:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:45 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:27:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:45 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:27:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:45 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:45 smithi106 ceph-mon[16603]: osdmap e153: 12 total, 12 up, 11 in 2024-08-30T23:27:45.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:45.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:45.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:45.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:45.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:45.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:45.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:46.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:46.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:46.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:46.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:46.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:46.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:46.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:46.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:46.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:46 smithi136 ceph-mon[20207]: pgmap v410: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 693 op/s; 14295/259389 objects misplaced (5.511%); 13 MiB/s, 214 objects/s recovering 2024-08-30T23:27:46.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:46 smithi173 ceph-mon[20554]: pgmap v410: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 693 op/s; 14295/259389 objects misplaced (5.511%); 13 MiB/s, 214 objects/s recovering 2024-08-30T23:27:46.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:46 smithi106 ceph-mon[16603]: pgmap v410: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.3 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 693 op/s; 14295/259389 objects misplaced (5.511%); 13 MiB/s, 214 objects/s recovering 2024-08-30T23:27:46.609 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:46.667 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.310593+0000", 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.74767900699999, 2024-08-30T23:27:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.562890844, 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 1 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310593+0000", 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310593+0000", 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310594+0000", 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.173e-06 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310601+0000", 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3779999999999998e-06 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310610+0000", 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8219999999999996e-06 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310618+0000", 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9780000000000005e-06 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310676+0000", 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8109000000000002e-05 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310682+0000", 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3890000000000002e-06 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.872201+0000", 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56151878899999996 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.872330+0000", 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012908099999999999 2024-08-30T23:27:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.872498+0000", 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016801199999999999 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873285+0000", 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078663699999999997 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873316+0000", 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1844000000000002e-05 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873378+0000", 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1765000000000002e-05 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873441+0000", 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2958000000000003e-05 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873484+0000", 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2908999999999999e-05 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.08968281400001, 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34705745500000001, 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 52 2024-08-30T23:27:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968590+0000", 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6499999999999999e-07 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968594+0000", 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0400000000000003e-06 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968602+0000", 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9850000000000006e-06 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968607+0000", 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2050000000000001e-06 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977321+0000", 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0087134099999999996 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977400+0000", 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8967000000000005e-05 2024-08-30T23:27:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977514+0000", 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114164 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977774+0000", 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026058500000000003 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.978024+0000", 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00024982299999999998 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315575+0000", 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33755099 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315615+0000", 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9705e-05 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315647+0000", 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2016e-05 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 392.35143766900001, 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94419764299999998, 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 80 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7400000000000003e-07 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706923+0000", 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8308999999999998e-05 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706932+0000", 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7409999999999998e-06 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706938+0000", 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.263e-06 2024-08-30T23:27:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708514+0000", 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015753620000000001 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708601+0000", 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6849999999999997e-05 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708731+0000", 2024-08-30T23:27:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013043399999999999 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.709427+0000", 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069585600000000001 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.709468+0000", 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1125999999999999e-05 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650912+0000", 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94144422800000005 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650975+0000", 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2707999999999997e-05 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.651032+0000", 2024-08-30T23:27:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7191999999999997e-05 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14229 2.6 2:681e88d6:::benchmark_data_smithi173_47195_object14228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.182074760000006, 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21536640000000001, 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14229 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876198+0000", 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6899999999999996e-07 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876298+0000", 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9622000000000003e-05 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876310+0000", 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1817999999999999e-05 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876317+0000", 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8009999999999998e-06 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876377+0000", 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0489999999999999e-05 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876507+0000", 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012945900000000001 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876637+0000", 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013032300000000001 2024-08-30T23:27:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878792+0000", 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002154689 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883495+0000", 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047034950000000002 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091459+0000", 2024-08-30T23:27:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207963857 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091526+0000", 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6610000000000001e-05 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091564+0000", 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8266999999999998e-05 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14233 2.6 2:682ac747:::benchmark_data_smithi173_47195_object14232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.180540360999998, 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.215809479, 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14233 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877733+0000", 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.215e-06 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877761+0000", 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7671999999999999e-05 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877776+0000", 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5352000000000001e-05 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877783+0000", 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2660000000000004e-06 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878221+0000", 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043807200000000002 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878304+0000", 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2769999999999995e-05 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878392+0000", 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7997999999999993e-05 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.879128+0000", 2024-08-30T23:27:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073602100000000001 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883772+0000", 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046439580000000001 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093486+0000", 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20971398699999999 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093518+0000", 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2047e-05 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093541+0000", 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3121000000000001e-05 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14240 2.6 2:6ab00b19:::benchmark_data_smithi173_47195_object14239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:27:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.176498421000005, 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21184964100000001, 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14240 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881775+0000", 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.536e-06 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881838+0000", 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2528999999999997e-05 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881849+0000", 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1606999999999999e-05 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881857+0000", 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5349999999999999e-06 2024-08-30T23:27:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881915+0000", 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8459999999999999e-05 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882042+0000", 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012617300000000001 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882187+0000", 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014551399999999999 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883462+0000", 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012752149999999999 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.898625+0000", 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015162491 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093570+0000", 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.194944849 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093601+0000", 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0817e-05 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093623+0000", 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2915e-05 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14242 2.6 2:6fdab316:::benchmark_data_smithi173_47195_object14241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.176134289999993, 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21156007900000001, 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14242 2024-08-30T23:27:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882139+0000", 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2900000000000002e-07 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882196+0000", 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7549000000000001e-05 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882204+0000", 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4189999999999996e-06 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882208+0000", 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4390000000000002e-06 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882770+0000", 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056187700000000001 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882879+0000", 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108856 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883022+0000", 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014315899999999999 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883584+0000", 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056226400000000004 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.938528+0000", 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054943433 2024-08-30T23:27:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093653+0000", 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.155125759 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093680+0000", 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6239000000000001e-05 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093698+0000", 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8255999999999999e-05 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16156 2.4 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 95.960699896999998, 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21552370500000001, 2024-08-30T23:27:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16156 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097573+0000", 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3699999999999999e-07 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097632+0000", 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8870999999999999e-05 2024-08-30T23:27:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097642+0000", 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0219e-05 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097649+0000", 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1269999999999997e-06 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097805+0000", 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015530900000000001 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097941+0000", 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000135847 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098034+0000", 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3246999999999997e-05 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098764+0000", 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000730133 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099110+0000", 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034588700000000002 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313035+0000", 2024-08-30T23:27:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21392467000000001 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313062+0000", 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7013999999999999e-05 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313096+0000", 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4443999999999998e-05 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16160 2.9 2:9059f27b:::benchmark_data_smithi173_47195_object16159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099236+0000", 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 95.959036517000001, 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.187105983, 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16160 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099236+0000", 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099236+0000", 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099237+0000", 2024-08-30T23:27:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6799999999999999e-07 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099301+0000", 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4830000000000001e-05 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099312+0000", 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.048e-05 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099317+0000", 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0939999999999998e-06 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099366+0000", 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9104000000000003e-05 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099480+0000", 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011429000000000001 2024-08-30T23:27:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099560+0000", 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9608000000000001e-05 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100253+0000", 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069285199999999999 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100566+0000", 2024-08-30T23:27:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031355599999999999 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.286236+0000", 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18566922 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.286295+0000", 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9500000000000003e-05 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.286342+0000", 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6581e-05 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16161 2.6 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 95.958918892, 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212960288, 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16161 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099354+0000", 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6199999999999997e-07 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099425+0000", 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0605999999999995e-05 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099432+0000", 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6850000000000001e-06 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099438+0000", 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7069999999999997e-06 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099491+0000", 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3038000000000003e-05 2024-08-30T23:27:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099620+0000", 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012866900000000001 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099790+0000", 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170266 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100492+0000", 2024-08-30T23:27:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070156400000000001 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100884+0000", 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039193399999999998 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312191+0000", 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211307562 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312253+0000", 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2150000000000006e-05 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312314+0000", 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0344999999999999e-05 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16911 2.4 2:26b3b856:::benchmark_data_smithi173_47195_object16910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:27:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 95.385887689, 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22617736999999999, 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16911 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672455+0000", 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9855000000000001e-05 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672467+0000", 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1528e-05 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672473+0000", 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5400000000000001e-06 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672516+0000", 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2645999999999998e-05 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672585+0000", 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8751999999999995e-05 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672675+0000", 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0543000000000004e-05 2024-08-30T23:27:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.673389+0000", 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071381299999999999 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.673738+0000", 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034890399999999998 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898465+0000", 2024-08-30T23:27:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22472745399999999 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898518+0000", 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2428e-05 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898562+0000", 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3989000000000002e-05 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16921 2.2 e111/97 2:4bb89a5b:::benchmark_data_smithi173_47195_object16920:head v 111'1004, mlcod=111'999)", 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 95.379492736000003, 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21942291, 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:27:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678780+0000", 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5699999999999998e-07 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678834+0000", 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3992000000000001e-05 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.896861+0000", 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218027111 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.896896+0000", 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5326e-05 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.897044+0000", 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001473 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.897089+0000", 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5806000000000002e-05 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898158+0000", 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010683559999999999 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898202+0000", 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4561999999999998e-05 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40525 2.9 2:93a0bad4:::benchmark_data_smithi173_47195_object40524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.150321676999994, 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21130669299999999, 2024-08-30T23:27:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40525 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:27:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907952+0000", 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8700000000000001e-06 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907991+0000", 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8658999999999997e-05 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908011+0000", 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9877999999999998e-05 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908020+0000", 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6500000000000002e-06 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908076+0000", 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6359e-05 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908193+0000", 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011733199999999999 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908326+0000", 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013302999999999999 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.909927+0000", 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016005909999999999 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.910234+0000", 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030682700000000002 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119175+0000", 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20894082 2024-08-30T23:27:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119206+0000", 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1652000000000001e-05 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119257+0000", 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1025e-05 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40537 2.9 2:97981a73:::benchmark_data_smithi173_47195_object40536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.144022746999994, 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207993344, 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40537 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914251+0000", 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.339e-06 2024-08-30T23:27:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914366+0000", 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011518699999999999 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914379+0000", 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3424999999999999e-05 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914388+0000", 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6070000000000008e-06 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914447+0000", 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8585999999999998e-05 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914600+0000", 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015365800000000001 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914763+0000", 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016261300000000001 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.915944+0000", 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011807250000000001 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.916216+0000", 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000272692 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122145+0000", 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205928427 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122190+0000", 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5785000000000003e-05 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122243+0000", 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2299999999999997e-05 2024-08-30T23:27:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40552 2.6 2:67bfbe41:::benchmark_data_smithi173_47195_object40551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.131131460999995, 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19393617399999999, 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40552 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:27:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927142+0000", 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8899999999999998e-07 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927204+0000", 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2008000000000007e-05 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927213+0000", 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8409999999999995e-06 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927219+0000", 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8690000000000001e-06 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927264+0000", 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4690000000000001e-05 2024-08-30T23:27:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927338+0000", 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3831000000000005e-05 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927468+0000", 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012972299999999999 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.928902+0000", 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014344970000000001 2024-08-30T23:27:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929243+0000", 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034044300000000001 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121004+0000", 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19176105600000001 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121035+0000", 2024-08-30T23:27:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1269999999999997e-05 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121077+0000", 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2057000000000001e-05 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40557 2.9 2:93889969:::benchmark_data_smithi173_47195_object40556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:27:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.929782+0000", 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.128490459999995, 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.193788611, 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40557 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929782+0000", 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929782+0000", 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929783+0000", 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8100000000000001e-07 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929894+0000", 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111513 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929906+0000", 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2286000000000001e-05 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929914+0000", 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7330000000000003e-06 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929964+0000", 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0111999999999997e-05 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.930039+0000", 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4639000000000001e-05 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.930119+0000", 2024-08-30T23:27:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9825000000000002e-05 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.931419+0000", 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001300615 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.931639+0000", 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000219175 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.123451+0000", 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19181198499999999 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.123513+0000", 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2476000000000003e-05 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.123570+0000", 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7370999999999997e-05 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40564 2.6 2:65129be5:::benchmark_data_smithi173_47195_object40563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.934399+0000", 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.123873364000005, 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.1867685, 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40564 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934399+0000", 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934399+0000", 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934400+0000", 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3689999999999999e-06 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934455+0000", 2024-08-30T23:27:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4789e-05 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934469+0000", 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3623e-05 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934477+0000", 2024-08-30T23:27:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3969999999999994e-06 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934539+0000", 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2427000000000004e-05 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934677+0000", 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013763600000000001 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934890+0000", 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00021267899999999999 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.936090+0000", 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012003199999999999 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.936389+0000", 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000299103 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121108+0000", 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.184719315 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121136+0000", 2024-08-30T23:27:46.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7980999999999999e-05 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121167+0000", 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0861000000000001e-05 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:62934 2.2 e127/97 2:479e8ab6:::benchmark_data_smithi173_47195_object62933:head v 127'3887, mlcod=127'3884)", 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.108101859000001, 2024-08-30T23:27:46.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20906233900000001, 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950171+0000", 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2500000000000005e-07 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950233+0000", 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2382e-05 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157402+0000", 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20716816099999999 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157407+0000", 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7479999999999999e-06 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157462+0000", 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4840999999999999e-05 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157485+0000", 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3244999999999999e-05 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.159190+0000", 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017050120000000001 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.159233+0000", 2024-08-30T23:27:46.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2225000000000003e-05 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67558 2.9 2:98b6d163:::benchmark_data_smithi173_47195_object67557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 55.982103938999998, 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214383824, 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.784 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67558 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076169+0000", 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2300000000000001e-07 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076247+0000", 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8326999999999997e-05 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076255+0000", 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2660000000000001e-06 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076261+0000", 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.784e-06 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077492+0000", 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001231027 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077607+0000", 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011488 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077733+0000", 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012587500000000001 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.079761+0000", 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020281829999999998 2024-08-30T23:27:46.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.080335+0000", 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057335700000000003 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290485+0000", 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21015015400000001 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290523+0000", 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8250000000000001e-05 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290552+0000", 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9098000000000002e-05 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67582 2.9 2:97773903:::benchmark_data_smithi173_47195_object67581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:27:46.790 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 55.961506563999997, 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19555588700000001, 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67582 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:27:46.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096767+0000", 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096823+0000", 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6773e-05 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096835+0000", 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1556000000000001e-05 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096842+0000", 2024-08-30T23:27:46.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0770000000000002e-06 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096892+0000", 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0256000000000001e-05 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096974+0000", 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1879999999999995e-05 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:27:46.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.097064+0000", 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9821999999999995e-05 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.098777+0000", 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017132149999999999 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.099120+0000", 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034311699999999999 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292262+0000", 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19314146600000001 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292296+0000", 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3748000000000003e-05 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292322+0000", 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5986000000000002e-05 2024-08-30T23:27:46.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.797 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.797 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:27:46.797 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:27:46.797 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:27:46.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:46.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:27:46.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:46.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:46.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:46.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:47.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:47.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:47.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:47.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:47.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:47.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:27:47.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:47.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:47.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:47.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:47 smithi173 ceph-mon[20554]: pgmap v411: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 13292/253344 objects misplaced (5.247%); 20 MiB/s, 312 objects/s recovering 2024-08-30T23:27:47.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:47.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:47 smithi106 ceph-mon[16603]: pgmap v411: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 13292/253344 objects misplaced (5.247%); 20 MiB/s, 312 objects/s recovering 2024-08-30T23:27:47.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:47.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:27:47.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:47.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:47 smithi136 ceph-mon[20207]: pgmap v411: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 771 op/s; 13292/253344 objects misplaced (5.247%); 20 MiB/s, 312 objects/s recovering 2024-08-30T23:27:47.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:47.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:47.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:47.867 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:47.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:47.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:47.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:48.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:48.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:48.198 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:48.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:48.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:48.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:48.445 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:27:48.445 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:27:48.445 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-08-30T23:27:48.445 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:27:48.445 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:27:48.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-08-30T23:27:48.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:48.515 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:48.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:48.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:48.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:48.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:48.759 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:27:48.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:48.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:48.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:48.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:48.988 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:27:48.988 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:27:49.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:49.027 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:27:49.027 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:27:49.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:49.064 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:49.067 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:27:49.067 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:27:49.111 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:27:49.111 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:27:49.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:49.149 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:27:49.150 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:27:49.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:49.190 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:27:49.190 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:27:49.228 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1146168488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:49.232 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:27:49.232 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:27:49.267 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:27:49.267 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:27:49.268 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1146168488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:49.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1146168488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:27:49.303 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:27:49.303 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:27:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:49.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:49.341 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:27:49.341 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:27:49.363 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:27:49.363 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:27:49.363 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:27:49.376 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:27:49.376 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:27:49.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:49.436 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:27:49.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:27:49.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:49.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:49.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:49.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:49.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:49.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:49.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:49.899 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:49.900 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:49.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:49.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:49.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:50.002 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:27:50.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:50.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:50.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:50.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:50.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4023489087' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:27:50.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:49 smithi136 ceph-mon[20207]: pgmap v412: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 12910/248349 objects misplaced (5.198%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:27:50.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:50.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4023489087' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:27:50.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:49 smithi173 ceph-mon[20554]: pgmap v412: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 12910/248349 objects misplaced (5.198%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:27:50.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4023489087' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:27:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:49 smithi106 ceph-mon[16603]: pgmap v412: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 799 op/s; 12910/248349 objects misplaced (5.198%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:27:50.431 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:50.466 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:50.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:50.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:50.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:50.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:50.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:50.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:50.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:50.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:50.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:51.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:50 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4023489087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:27:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:50 smithi106 ceph-mon[16603]: osdmap e154: 12 total, 12 up, 11 in 2024-08-30T23:27:51.117 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:51.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:51.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:50 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4023489087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:27:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:50 smithi136 ceph-mon[20207]: osdmap e154: 12 total, 12 up, 11 in 2024-08-30T23:27:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:50 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4023489087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:27:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:50 smithi173 ceph-mon[20554]: osdmap e154: 12 total, 12 up, 11 in 2024-08-30T23:27:51.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:51.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:51.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:51.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:51.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:51.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:51.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:51.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:51.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:51.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:51.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:51.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:52.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:52.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:52.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:52.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:52.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:52 smithi136 ceph-mon[20207]: pgmap v414: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 12482/243990 objects misplaced (5.116%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:27:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:52 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:27:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:52 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:27:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:52 smithi173 ceph-mon[20554]: pgmap v414: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 12482/243990 objects misplaced (5.116%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:27:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:52 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:27:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:52 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:27:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:52 smithi106 ceph-mon[16603]: pgmap v414: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 12482/243990 objects misplaced (5.116%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:27:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:52 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:27:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:52 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:27:52.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:52.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:52.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:52.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:52.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:52.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:52.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:52.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:52.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:27:52.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:52.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:27:52.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:52.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:52.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:53.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:53.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:53.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:53 smithi106 ceph-mon[16603]: osdmap e155: 12 total, 12 up, 11 in 2024-08-30T23:27:53.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:53.218 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:53.262 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:53 smithi136 ceph-mon[20207]: osdmap e155: 12 total, 12 up, 11 in 2024-08-30T23:27:53.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:53.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:53 smithi173 ceph-mon[20554]: osdmap e155: 12 total, 12 up, 11 in 2024-08-30T23:27:53.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:53.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:53.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:53.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:27:53.573 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.632 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:53.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:53.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:53.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:53.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:53.909 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:53.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:54.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:54.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:54.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:54.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:54.230 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:27:54.230 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:27:54.231 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:54 smithi106 ceph-mon[16603]: pgmap v416: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 832 op/s; 11568/239415 objects misplaced (4.832%); 21 MiB/s, 340 objects/s recovering 2024-08-30T23:27:54.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:54.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:54 smithi136 ceph-mon[20207]: pgmap v416: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 832 op/s; 11568/239415 objects misplaced (4.832%); 21 MiB/s, 340 objects/s recovering 2024-08-30T23:27:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:54 smithi173 ceph-mon[20554]: pgmap v416: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 832 op/s; 11568/239415 objects misplaced (4.832%); 21 MiB/s, 340 objects/s recovering 2024-08-30T23:27:54.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:54.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:54.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:54.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:54.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:54.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:54.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:54.801 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:54.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:54.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:54.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:54.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:27:54.994 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:27:55.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:27:55.056 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:27:55.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-08-30T23:27:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.082 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:55.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:55.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:55.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:55.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/315717267' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:55.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/315717267' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:55.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:55.468 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/315717267' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:27:55.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.606 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:55.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:55.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:55.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:55.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:55.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:55.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:55.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:55.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:56.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:56.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:56.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:56.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:56.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:56 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:56.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/315717267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:56.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:56 smithi173 ceph-mon[20554]: osdmap e156: 12 total, 12 up, 11 in 2024-08-30T23:27:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:56 smithi173 ceph-mon[20554]: pgmap v418: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.8 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 11568/235479 objects misplaced (4.913%); 14 MiB/s, 223 objects/s recovering 2024-08-30T23:27:56.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:56.355 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:56.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:56 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/315717267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:56 smithi106 ceph-mon[16603]: osdmap e156: 12 total, 12 up, 11 in 2024-08-30T23:27:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:56 smithi106 ceph-mon[16603]: pgmap v418: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.8 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 11568/235479 objects misplaced (4.913%); 14 MiB/s, 223 objects/s recovering 2024-08-30T23:27:56.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:56.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:56.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:56 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:27:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/315717267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:27:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:56 smithi136 ceph-mon[20207]: osdmap e156: 12 total, 12 up, 11 in 2024-08-30T23:27:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:56 smithi136 ceph-mon[20207]: pgmap v418: 17 pgs: 4 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.8 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 11568/235479 objects misplaced (4.913%); 14 MiB/s, 223 objects/s recovering 2024-08-30T23:27:56.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:56.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:56.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:56.730 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:56.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:56.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:56.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:56.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:56.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:57.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:57.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:57 smithi106 ceph-mon[16603]: osdmap e157: 12 total, 12 up, 11 in 2024-08-30T23:27:57.115 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:57.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:57.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:57.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:57 smithi173 ceph-mon[20554]: osdmap e157: 12 total, 12 up, 11 in 2024-08-30T23:27:57.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:57.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:57.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:57.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:57 smithi136 ceph-mon[20207]: osdmap e157: 12 total, 12 up, 11 in 2024-08-30T23:27:57.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:57.639 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:57.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:57.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.765 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:57.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:27:57.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:57.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:27:58.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:27:58.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:58.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:27:58.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:27:58.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:27:58.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:58.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:58 smithi173 ceph-mon[20554]: osdmap e158: 12 total, 12 up, 11 in 2024-08-30T23:27:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:58 smithi173 ceph-mon[20554]: pgmap v421: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 824 op/s; 10340/230004 objects misplaced (4.496%); 23 MiB/s, 370 objects/s recovering 2024-08-30T23:27:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:58 smithi106 ceph-mon[16603]: osdmap e158: 12 total, 12 up, 11 in 2024-08-30T23:27:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:58 smithi106 ceph-mon[16603]: pgmap v421: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 824 op/s; 10340/230004 objects misplaced (4.496%); 23 MiB/s, 370 objects/s recovering 2024-08-30T23:27:58.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:58.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:58.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:27:58.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:27:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:58 smithi136 ceph-mon[20207]: osdmap e158: 12 total, 12 up, 11 in 2024-08-30T23:27:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:58 smithi136 ceph-mon[20207]: pgmap v421: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 824 op/s; 10340/230004 objects misplaced (4.496%); 23 MiB/s, 370 objects/s recovering 2024-08-30T23:27:58.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:27:58.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:27:58.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:27:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:58.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:27:58.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:58.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:58.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:59.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:59.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:59.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:27:59.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:27:59.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.318 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:59.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:27:59.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:27:59.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:27:59.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:27:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:27:59.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:27:59.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:27:59.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:27:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:59 smithi136 ceph-mon[20207]: pgmap v422: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 9809/225483 objects misplaced (4.350%); 18 MiB/s, 293 objects/s recovering 2024-08-30T23:27:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:27:59 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-30T23:27:59.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:59 smithi106 ceph-mon[16603]: pgmap v422: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 9809/225483 objects misplaced (4.350%); 18 MiB/s, 293 objects/s recovering 2024-08-30T23:27:59.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:27:59 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-30T23:27:59.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:27:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:59 smithi173 ceph-mon[20554]: pgmap v422: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 4.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 9809/225483 objects misplaced (4.350%); 18 MiB/s, 293 objects/s recovering 2024-08-30T23:27:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:27:59 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-30T23:27:59.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:27:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:27:59.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:27:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:27:59.962 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:27:59.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:00.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:00.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:00.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:00.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:00.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:00.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:00.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:00.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:00.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:00.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:00.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:00.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:00.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:00.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:00.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.855 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:00.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:00.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:00.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:01.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:01.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:01.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:01.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:01.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:01.369 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:01.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:01.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.495 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:01.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:01.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:01.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:01.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:01 smithi136 ceph-mon[20207]: pgmap v423: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 9809/221775 objects misplaced (4.423%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:28:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:01 smithi173 ceph-mon[20554]: pgmap v423: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 9809/221775 objects misplaced (4.423%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:28:01.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:01.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:01 smithi106 ceph-mon[16603]: pgmap v423: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 9809/221775 objects misplaced (4.423%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:28:01.889 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:01.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:01.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:02.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:02.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:02.166 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:02.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:02.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:02.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:02.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:02.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:02.557 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:02.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:02.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:02.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:02.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:02.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:02.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:02.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:03.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:03.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.077 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:03.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:03.212 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:03.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:03.353 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:03.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:03.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: osdmap e159: 12 total, 12 up, 11 in 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: pgmap v425: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 760 op/s; 8624/217266 objects misplaced (3.969%); 25 MiB/s, 402 objects/s recovering 2024-08-30T23:28:03.510 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2496472552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:03.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:03.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:03.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:03.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: osdmap e159: 12 total, 12 up, 11 in 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: pgmap v425: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 760 op/s; 8624/217266 objects misplaced (3.969%); 25 MiB/s, 402 objects/s recovering 2024-08-30T23:28:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2496472552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: osdmap e159: 12 total, 12 up, 11 in 2024-08-30T23:28:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: pgmap v425: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 760 op/s; 8624/217266 objects misplaced (3.969%); 25 MiB/s, 402 objects/s recovering 2024-08-30T23:28:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2496472552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:03.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:03.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:03.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:03.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:03.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:04.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:04.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:04.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:04.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:04.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:04.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:04.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:04.400 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:04.400 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:04.400 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:04.400 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:04.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:04.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:04.403 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:04.403 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:04.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:04.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:04.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:04.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:04.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:04.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:04 smithi106 ceph-mon[16603]: osdmap e160: 12 total, 12 up, 11 in 2024-08-30T23:28:04.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:04.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:04 smithi136 ceph-mon[20207]: osdmap e160: 12 total, 12 up, 11 in 2024-08-30T23:28:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:04 smithi173 ceph-mon[20554]: osdmap e160: 12 total, 12 up, 11 in 2024-08-30T23:28:04.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:04.895 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:04.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:04.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:05.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:05.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:05.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:05.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:05.255 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:05.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:05.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:05.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:05.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:05.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:05.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:05.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:05.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:05 smithi106 ceph-mon[16603]: pgmap v427: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 8624/213600 objects misplaced (4.037%); 13 MiB/s, 214 objects/s recovering 2024-08-30T23:28:05.735 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:05 smithi173 ceph-mon[20554]: pgmap v427: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 8624/213600 objects misplaced (4.037%); 13 MiB/s, 214 objects/s recovering 2024-08-30T23:28:05.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:05 smithi136 ceph-mon[20207]: pgmap v427: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 8624/213600 objects misplaced (4.037%); 13 MiB/s, 214 objects/s recovering 2024-08-30T23:28:05.803 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:05.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:05.819 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.14157.0:80 1.0 e23/20 1:30306672:devicehealth::main.db.0000000000000000:head v 23'58, mlcod=23'53)", 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 412.15381367399999, 2024-08-30T23:28:05.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94186119800000001, 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5799999999999999e-07 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708893+0000", 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1081999999999998e-05 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650093+0000", 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94119932299999998 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650101+0000", 2024-08-30T23:28:05.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7900000000000005e-06 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650169+0000", 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650197+0000", 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.72e-05 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650681+0000", 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00048408700000000002 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650723+0000", 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:842 2.7 e104/97 2:ebe581cf:::benchmark_data_smithi173_47195_object841:head v 104'46, mlcod=104'43)", 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 127.283152454, 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15218538700000001, 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.875 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579523+0000", 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5400000000000002e-07 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.579637+0000", 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011383900000000001 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731295+0000", 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15165820299999999 2024-08-30T23:28:05.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731300+0000", 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.023e-06 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731346+0000", 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5766999999999997e-05 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731360+0000", 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3959e-05 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731680+0000", 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032010099999999999 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.731708+0000", 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8041000000000001e-05 2024-08-30T23:28:05.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:845 2.1 2:8578137a:::benchmark_data_smithi173_47195_object844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.582722+0000", 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 127.279952958, 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15184962699999999, 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.880 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 845 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582722+0000", 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582722+0000", 2024-08-30T23:28:05.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582724+0000", 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4449999999999999e-06 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582767+0000", 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3519000000000001e-05 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582778+0000", 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1212e-05 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582784+0000", 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1789999999999999e-06 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582838+0000", 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4012999999999999e-05 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.582981+0000", 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000143783 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583134+0000", 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015213499999999999 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583522+0000", 2024-08-30T23:28:05.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038870099999999998 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588769+0000", 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052468990000000002 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734517+0000", 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.145747561 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734549+0000", 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2187000000000002e-05 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734572+0000", 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2993000000000002e-05 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:845 2.1 e104/97 ondisk, result = 0)", 2024-08-30T23:28:05.886 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 127.278867377, 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15076826500000001, 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583808+0000", 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7e-07 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.583811+0000", 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4650000000000001e-06 2024-08-30T23:28:05.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734462+0000", 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.150650906 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734470+0000", 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6100000000000006e-06 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734509+0000", 2024-08-30T23:28:05.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8634999999999999e-05 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734514+0000", 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6070000000000001e-06 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734576+0000", 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2571999999999997e-05 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:849 2.e 2:7e7dd803:::benchmark_data_smithi173_47195_object848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:25:58.587830+0000", 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 127.27484499099999, 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14671935799999999, 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.891 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 849 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587830+0000", 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587830+0000", 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587832+0000", 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.319e-06 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587879+0000", 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7054e-05 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587890+0000", 2024-08-30T23:28:05.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1506e-05 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587895+0000", 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1379999999999998e-06 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.587941+0000", 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5983999999999999e-05 2024-08-30T23:28:05.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588022+0000", 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1144999999999996e-05 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588105+0000", 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2663999999999993e-05 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588517+0000", 2024-08-30T23:28:05.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00041148399999999999 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.588785+0000", 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026829000000000002 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734469+0000", 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14568394900000001 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734516+0000", 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7409e-05 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:25:58.734550+0000", 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3416e-05 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14244 2.3 e109/97 2:c35428fa:::benchmark_data_smithi173_47195_object14243:head v 109'957, mlcod=109'954)", 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:28:05.897 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 116.97889463200001, 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213971894, 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883782+0000", 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999999e-07 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883844+0000", 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2580000000000001e-05 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096781+0000", 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212936878 2024-08-30T23:28:05.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096785+0000", 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0389999999999998e-06 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096831+0000", 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6261999999999999e-05 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096844+0000", 2024-08-30T23:28:05.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2883e-05 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097712+0000", 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086761599999999998 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097753+0000", 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0729999999999998e-05 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14266 2.e 2:74094085:::benchmark_data_smithi173_47195_object14265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.947539+0000", 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 116.915136316, 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14818767099999999, 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14266 2024-08-30T23:28:05.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947539+0000", 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947539+0000", 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947540+0000", 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669999999999999e-06 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947586+0000", 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5439000000000003e-05 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947598+0000", 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041999999999999e-05 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947603+0000", 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5740000000000003e-06 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947644+0000", 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0979999999999997e-05 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947729+0000", 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4864999999999995e-05 2024-08-30T23:28:05.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947811+0000", 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1884000000000004e-05 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948332+0000", 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052075799999999996 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095442+0000", 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.147109929 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095602+0000", 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016049599999999999 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095659+0000", 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6586999999999999e-05 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095727+0000", 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7849999999999996e-05 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:14266 2.e e109/97 ondisk, result = 0)", 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.948563+0000", 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 116.914112288, 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.146887187, 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.909 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948563+0000", 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948563+0000", 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948564+0000", 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.948568+0000", 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2089999999999999e-06 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095378+0000", 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14680979399999999 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095385+0000", 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-06 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095428+0000", 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3037000000000001e-05 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095436+0000", 2024-08-30T23:28:05.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0989999999999999e-06 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095450+0000", 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4430000000000001e-05 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16941 2.e 2:74b71233:::benchmark_data_smithi173_47195_object16940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.690508+0000", 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 115.172167156, 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16175384500000001, 2024-08-30T23:28:05.913 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16941 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690508+0000", 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690508+0000", 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690509+0000", 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2699999999999999e-07 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690594+0000", 2024-08-30T23:28:05.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5748000000000006e-05 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690603+0000", 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0129999999999999e-06 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690610+0000", 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7309999999999996e-06 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690652+0000", 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1508000000000003e-05 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690725+0000", 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2838999999999998e-05 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.690805+0000", 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0353999999999997e-05 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691361+0000", 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00055562600000000004 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691657+0000", 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029633900000000001 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852197+0000", 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16054035899999999 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852231+0000", 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3791000000000002e-05 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852262+0000", 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0910000000000001e-05 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:16941 2.e e111/97 ondisk, result = 0)", 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 115.171091702, 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16068374799999999, 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691584+0000", 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0999999999999998e-07 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.691587+0000", 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0079999999999998e-06 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852149+0000", 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16056198199999999 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852154+0000", 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4329999999999997e-06 2024-08-30T23:28:05.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852188+0000", 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4730000000000001e-05 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852194+0000", 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6400000000000002e-06 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.852267+0000", 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3344999999999997e-05 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:18967 2.1 2:85fbebba:::benchmark_data_smithi173_47195_object18966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:12.391277+0000", 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 113.47139830099999, 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.154535755, 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.924 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 18967 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391277+0000", 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391277+0000", 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391278+0000", 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1399999999999996e-07 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391333+0000", 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5627999999999997e-05 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391342+0000", 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4950000000000008e-06 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391346+0000", 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4070000000000003e-06 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391381+0000", 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4897999999999997e-05 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391461+0000", 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9752999999999993e-05 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.391552+0000", 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1025000000000003e-05 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.392171+0000", 2024-08-30T23:28:05.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00061906500000000002 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.392714+0000", 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054273799999999997 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545745+0000", 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15303161300000001 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545784+0000", 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9014999999999998e-05 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:12.545813+0000", 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8404000000000001e-05 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:56060 2.e 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:28:05.930 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.083198617999997, 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21409088000000001, 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 56060 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779478+0000", 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4869999999999999e-06 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779513+0000", 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4384999999999997e-05 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779529+0000", 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6447999999999999e-05 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779537+0000", 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999993e-06 2024-08-30T23:28:05.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779580+0000", 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3254999999999998e-05 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779658+0000", 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7711000000000002e-05 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779738+0000", 2024-08-30T23:28:05.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9775000000000001e-05 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781534+0000", 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017964820000000001 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992319+0000", 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21078449599999999 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993464+0000", 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001145294 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993511+0000", 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6344000000000002e-05 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993568+0000", 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6993000000000001e-05 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.936 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:56060 2.e e124/97 ondisk, result = 0)", 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.080858649000007, 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051657200000001, 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:28:05.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9999999999999997e-07 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781821+0000", 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9480000000000001e-06 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990657+0000", 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20883629200000001 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990664+0000", 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.793e-06 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992305+0000", 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016408480000000001 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992313+0000", 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1100000000000003e-06 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992333+0000", 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9981000000000001e-05 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.940 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67113 2.e 2:7c5fddf0:::benchmark_data_smithi173_47195_object67112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:49.611321+0000", 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.251353815000002, 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17819559800000001, 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.941 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67113 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611321+0000", 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:05.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611321+0000", 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611323+0000", 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6360000000000001e-06 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611450+0000", 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012673 2024-08-30T23:28:05.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611461+0000", 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1608e-05 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611468+0000", 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9489999999999999e-06 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611557+0000", 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9098000000000007e-05 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611687+0000", 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012953199999999999 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611780+0000", 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3236e-05 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614050+0000", 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022700590000000001 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.616962+0000", 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029115999999999999 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.789442+0000", 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17247995099999999 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.789482+0000", 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0188999999999998e-05 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.789517+0000", 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5009999999999999e-05 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67116 2.e 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.250744831999995, 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22641219700000001, 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67116 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611931+0000", 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612007+0000", 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5883999999999994e-05 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612021+0000", 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4007e-05 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612028+0000", 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4099999999999996e-06 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614110+0000", 2024-08-30T23:28:05.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020827520000000002 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614270+0000", 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000159235 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614420+0000", 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015048199999999999 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617176+0000", 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027561619999999999 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617722+0000", 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054594799999999998 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838237+0000", 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22051536999999999 2024-08-30T23:28:05.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838285+0000", 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7808e-05 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838343+0000", 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7340000000000003e-05 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67586 2.e 2:726358de:::benchmark_data_smithi173_47195_object67585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.101382+0000", 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 75.761293695000006, 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18275402600000001, 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67586 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.956 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101382+0000", 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101382+0000", 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101383+0000", 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.017e-06 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101427+0000", 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4370999999999999e-05 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101438+0000", 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0652999999999999e-05 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101443+0000", 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.169e-06 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101484+0000", 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0775999999999997e-05 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101573+0000", 2024-08-30T23:28:05.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9001999999999999e-05 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.101659+0000", 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6217000000000005e-05 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.103502+0000", 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001843616 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.113763+0000", 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010260138 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.284066+0000", 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17030309099999999 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.284103+0000", 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7698999999999999e-05 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.284136+0000", 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2277000000000003e-05 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75429 2.1 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.904236815999994, 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24538217700000001, 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.962 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75429 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958440+0000", 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9410000000000001e-06 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958471+0000", 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0919e-05 2024-08-30T23:28:05.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958494+0000", 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3078000000000002e-05 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958504+0000", 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1190000000000003e-06 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958565+0000", 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1364000000000004e-05 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958669+0000", 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010414199999999999 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958772+0000", 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102979 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.960745+0000", 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019727600000000001 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.066102+0000", 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.105356845 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203744+0000", 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13764249000000001 2024-08-30T23:28:05.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203787+0000", 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.286e-05 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203821+0000", 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3680000000000003e-05 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:82358 2.e 2:7a134f4e:::benchmark_data_smithi173_47195_object82357:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:07.362603+0000", 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.500072048, 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16964259700000001, 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 82358 2024-08-30T23:28:05.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362603+0000", 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362603+0000", 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362605+0000", 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3510000000000001e-06 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362660+0000", 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5577e-05 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362670+0000", 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0172e-05 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362675+0000", 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8369999999999996e-06 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362744+0000", 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8620000000000004e-05 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.362897+0000", 2024-08-30T23:28:05.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000153521 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.363036+0000", 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013828500000000001 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.365943+0000", 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002907864 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.367934+0000", 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019907620000000001 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.532173+0000", 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16423858299999999 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.532214+0000", 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0775999999999997e-05 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:07.532246+0000", 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2249000000000002e-05 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head, version: 114'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head@114'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 20.970517913999998, 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19944226600000001, 2024-08-30T23:28:05.975 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892158+0000", 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892186+0000", 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7710000000000001e-05 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089119+0000", 2024-08-30T23:28:05.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19693307299999999 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089137+0000", 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7533e-05 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089255+0000", 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118318 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089275+0000", 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9828000000000001e-05 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091600+0000", 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023249030000000001 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.979 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:270ad58a:::benchmark_data_smithi173_47195_object25231:head, version: 114'1538, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:270ad58a:::benchmark_data_smithi173_47195_object25231:head@114'1538, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:46.058973+0000", 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 19.803702625, 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16009123, 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.058973+0000", 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.058973+0000", 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.058974+0000", 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2839999999999999e-06 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.059010+0000", 2024-08-30T23:28:05.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5754999999999999e-05 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.216379+0000", 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15736934899999999 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.216400+0000", 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0676e-05 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:05.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.216517+0000", 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011758 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.216532+0000", 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5116000000000001e-05 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.219064+0000", 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025314700000000001 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.984 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:05.984 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:05.984 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:28:06.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:06.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:06.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:06.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:06.274 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:06.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:06.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:06.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:06.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:06.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:06.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:06.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:06.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:06.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:06.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:06.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:07.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:07.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:28:07.065 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:07.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:28:07.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:07.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:07.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:28:07.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:07.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:07.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:07.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:07.582 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:07.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:07.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:07 smithi106 ceph-mon[16603]: pgmap v428: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 8031/207405 objects misplaced (3.872%); 14 MiB/s, 222 objects/s recovering 2024-08-30T23:28:07.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:07 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:28:07.669 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:07.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:07 smithi136 ceph-mon[20207]: pgmap v428: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 8031/207405 objects misplaced (3.872%); 14 MiB/s, 222 objects/s recovering 2024-08-30T23:28:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:07 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:28:07.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:07.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:07 smithi173 ceph-mon[20554]: pgmap v428: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 752 op/s; 8031/207405 objects misplaced (3.872%); 14 MiB/s, 222 objects/s recovering 2024-08-30T23:28:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:07 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:28:07.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:07.960 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:07.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:08.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:08.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:08.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:08.223 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:08.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:08.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:08.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:08.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:08.486 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:08.497 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:28:08.497 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:28:08.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.497 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:127463 2.6 e160/142 2:62109c34:::benchmark_data_smithi173_47195_object30455:head v 160'8034, mlcod=160'8030)", 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:08.400598+0000", 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0028594179999999999, 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028942460000000001, 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400598+0000", 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400598+0000", 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400599+0000", 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0279999999999999e-06 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400604+0000", 2024-08-30T23:28:08.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7060000000000003e-06 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400659+0000", 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5421999999999999e-05 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400666+0000", 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8399999999999997e-06 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400717+0000", 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1048999999999999e-05 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:08.400746+0000", 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8449000000000002e-05 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:08.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:08.502 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:28:08.502 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-08-30T23:28:08.502 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:28:08.508 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T23:28:08.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:08.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/85910163' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:28:08.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:08 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:28:08.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:08.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:08.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/85910163' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:28:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:08 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:28:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/85910163' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:28:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:08 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:28:08.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:08.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:08.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:08.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:09.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:09.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:09.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:09.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:09.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:09.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:09.390 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:09.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:09.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:09 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:28:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:09 smithi106 ceph-mon[16603]: osdmap e161: 12 total, 12 up, 11 in 2024-08-30T23:28:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4050995615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:09 smithi106 ceph-mon[16603]: pgmap v430: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 796 op/s; 7530/202350 objects misplaced (3.721%); 15 MiB/s, 232 objects/s recovering 2024-08-30T23:28:09.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:09.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:09.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:09.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:09 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:28:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:09 smithi136 ceph-mon[20207]: osdmap e161: 12 total, 12 up, 11 in 2024-08-30T23:28:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4050995615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:09 smithi136 ceph-mon[20207]: pgmap v430: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 796 op/s; 7530/202350 objects misplaced (3.721%); 15 MiB/s, 232 objects/s recovering 2024-08-30T23:28:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:09 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:28:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:09 smithi173 ceph-mon[20554]: osdmap e161: 12 total, 12 up, 11 in 2024-08-30T23:28:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4050995615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:09 smithi173 ceph-mon[20554]: pgmap v430: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 796 op/s; 7530/202350 objects misplaced (3.721%); 15 MiB/s, 232 objects/s recovering 2024-08-30T23:28:09.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:09.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:09.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:10.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:10.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:10.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:10.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:10.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:10.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:10.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:10.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:10.487 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:10.487 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:10.487 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:10.488 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:10.489 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:10.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:10.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:10.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:10.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:10.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:10.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:10.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:10.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:10.819 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:10.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:10.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:10.945 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:11.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:11.078 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:11.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:11.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:11.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:11.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:11.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:11.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:11.513 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:11.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:11.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:11.725 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:11.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:11 smithi136 ceph-mon[20207]: pgmap v431: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 7530/197994 objects misplaced (3.803%); 8.5 MiB/s, 136 objects/s recovering 2024-08-30T23:28:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:11 smithi173 ceph-mon[20554]: pgmap v431: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 7530/197994 objects misplaced (3.803%); 8.5 MiB/s, 136 objects/s recovering 2024-08-30T23:28:11.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:11 smithi106 ceph-mon[16603]: pgmap v431: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 7530/197994 objects misplaced (3.803%); 8.5 MiB/s, 136 objects/s recovering 2024-08-30T23:28:11.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:11.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:11.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:11.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:28:11.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:12.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:12.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:12.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:12.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:12.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:12.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:12.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:12.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:12.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:12.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-08-30T23:28:12.540 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:12.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:12.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:12.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:12.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:12.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:12 smithi106 ceph-mon[16603]: osdmap e162: 12 total, 12 up, 11 in 2024-08-30T23:28:12.757 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:12.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:12 smithi136 ceph-mon[20207]: osdmap e162: 12 total, 12 up, 11 in 2024-08-30T23:28:12.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:12 smithi173 ceph-mon[20554]: osdmap e162: 12 total, 12 up, 11 in 2024-08-30T23:28:12.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:12.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:12.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:13.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.063 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:13.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:13.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:13.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:13.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:13.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:13.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:13.468 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:13.472 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:13.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:28:13.515 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:13.515 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:13.549 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:28:13.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:13.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:13.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:13.605 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:13 smithi106 ceph-mon[16603]: pgmap v433: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 6314/193443 objects misplaced (3.264%); 18 MiB/s, 288 objects/s recovering 2024-08-30T23:28:13.605 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3694710602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:13.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:13.738 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:13 smithi173 ceph-mon[20554]: pgmap v433: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 6314/193443 objects misplaced (3.264%); 18 MiB/s, 288 objects/s recovering 2024-08-30T23:28:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3694710602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:13.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:13.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:13.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:13.997 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:13.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:13 smithi136 ceph-mon[20207]: pgmap v433: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 839 op/s; 6314/193443 objects misplaced (3.264%); 18 MiB/s, 288 objects/s recovering 2024-08-30T23:28:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3694710602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:14.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:14.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:14.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:14.258 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:14.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:14.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:14.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:14.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:14.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:14.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:28:14.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:14.660 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:14.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:14.801 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:14.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:14 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:14.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3694710602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:28:14.811 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:14 smithi106 ceph-mon[16603]: osdmap e163: 12 total, 12 up, 11 in 2024-08-30T23:28:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:14 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3694710602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:28:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:14 smithi173 ceph-mon[20554]: osdmap e163: 12 total, 12 up, 11 in 2024-08-30T23:28:14.903 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:14.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:14.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:14 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3694710602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:28:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:14 smithi136 ceph-mon[20207]: osdmap e163: 12 total, 12 up, 11 in 2024-08-30T23:28:15.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:15.079 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:15.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.219 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:15.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:15.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:15.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:15.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.487 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:15.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:15.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:15.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:15.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:15.717 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:15.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:15 smithi173 ceph-mon[20554]: osdmap e164: 12 total, 12 up, 11 in 2024-08-30T23:28:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:15 smithi173 ceph-mon[20554]: pgmap v436: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 6314/189639 objects misplaced (3.329%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:28:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:15 smithi106 ceph-mon[16603]: osdmap e164: 12 total, 12 up, 11 in 2024-08-30T23:28:15.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:15 smithi106 ceph-mon[16603]: pgmap v436: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 6314/189639 objects misplaced (3.329%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:28:15.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:15.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:15.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:15.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:15 smithi136 ceph-mon[20207]: osdmap e164: 12 total, 12 up, 11 in 2024-08-30T23:28:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:15 smithi136 ceph-mon[20207]: pgmap v436: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 6314/189639 objects misplaced (3.329%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:28:16.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:16.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:16.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:16.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:16.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:16.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:16.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:16.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:16.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:16.590 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T23:28:16.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:16.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:16.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:16 smithi106 ceph-mon[16603]: 2.f scrub starts 2024-08-30T23:28:16.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:16 smithi106 ceph-mon[16603]: 2.3 scrub starts 2024-08-30T23:28:16.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3145137202' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:28:16.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:16 smithi173 ceph-mon[20554]: 2.f scrub starts 2024-08-30T23:28:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:16 smithi173 ceph-mon[20554]: 2.3 scrub starts 2024-08-30T23:28:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3145137202' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:28:16.830 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:16.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:16.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:16.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:16 smithi136 ceph-mon[20207]: 2.f scrub starts 2024-08-30T23:28:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:16 smithi136 ceph-mon[20207]: 2.3 scrub starts 2024-08-30T23:28:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3145137202' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:28:17.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:17.094 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:17.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:17.236 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:17.277 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:17.277 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:17.278 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:17.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:17.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:28:17.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:17.380 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:17.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:17.520 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:17.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:17 smithi106 ceph-mon[16603]: 2.f scrub ok 2024-08-30T23:28:17.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3145137202' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:28:17.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:17 smithi106 ceph-mon[16603]: osdmap e165: 12 total, 12 up, 11 in 2024-08-30T23:28:17.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4112196405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:17.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:17 smithi106 ceph-mon[16603]: 2.3 scrub ok 2024-08-30T23:28:17.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:17 smithi106 ceph-mon[16603]: pgmap v438: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 5592/184017 objects misplaced (3.039%); 21 MiB/s, 331 objects/s recovering 2024-08-30T23:28:17.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:17.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:17.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:17.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:17.790 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:17.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:17.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:17.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:18.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:18.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:17 smithi136 ceph-mon[20207]: 2.f scrub ok 2024-08-30T23:28:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3145137202' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:28:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:17 smithi136 ceph-mon[20207]: osdmap e165: 12 total, 12 up, 11 in 2024-08-30T23:28:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4112196405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:17 smithi136 ceph-mon[20207]: 2.3 scrub ok 2024-08-30T23:28:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:17 smithi136 ceph-mon[20207]: pgmap v438: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 5592/184017 objects misplaced (3.039%); 21 MiB/s, 331 objects/s recovering 2024-08-30T23:28:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:17 smithi173 ceph-mon[20554]: 2.f scrub ok 2024-08-30T23:28:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3145137202' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:28:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:17 smithi173 ceph-mon[20554]: osdmap e165: 12 total, 12 up, 11 in 2024-08-30T23:28:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4112196405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:17 smithi173 ceph-mon[20554]: 2.3 scrub ok 2024-08-30T23:28:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:17 smithi173 ceph-mon[20554]: pgmap v438: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 797 op/s; 5592/184017 objects misplaced (3.039%); 21 MiB/s, 331 objects/s recovering 2024-08-30T23:28:18.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:18.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:18.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-08-30T23:28:18.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:18.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:18.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:18.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:18.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.556 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:18.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:18.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:18.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.682 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:18.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:18.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:18.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:18.957 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:18.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:19.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:19.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:19.179 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:19.180 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:19.186 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:19.230 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:28:19.230 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:28:19.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14266 2.e e109/97 2:74094085:::benchmark_data_smithi173_47195_object14265:head v 109'831, mlcod=109'828)", 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.947874+0000", 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 130.279365334, 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14747749700000001, 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.231 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947874+0000", 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947874+0000", 2024-08-30T23:28:19.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947875+0000", 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3799999999999996e-07 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.947894+0000", 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8499e-05 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094480+0000", 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14658631 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094487+0000", 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9260000000000003e-06 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094560+0000", 2024-08-30T23:28:19.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3230000000000002e-05 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.094579+0000", 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.823e-05 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095312+0000", 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073342100000000005 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.095352+0000", 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0043000000000003e-05 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16161 2.6 e109/97 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head v 109'1027, mlcod=109'1024)", 2024-08-30T23:28:19.237 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 129.127329567, 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212155128, 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:28:19.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099911+0000", 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5499999999999997e-07 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099933+0000", 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1793999999999998e-05 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311098+0000", 2024-08-30T23:28:19.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21116489199999999 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311105+0000", 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6920000000000002e-06 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311160+0000", 2024-08-30T23:28:19.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4716999999999998e-05 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311179+0000", 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8814e-05 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312023+0000", 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00084434600000000005 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312065+0000", 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2117999999999999e-05 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16925 2.a e111/97 2:5a33a07d:::benchmark_data_smithi173_47195_object16924:head v 111'1026, mlcod=111'1022)", 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.682356+0000", 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 128.544883368, 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.164783656, 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.243 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682356+0000", 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682356+0000", 2024-08-30T23:28:19.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682357+0000", 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5899999999999995e-07 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.682411+0000", 2024-08-30T23:28:19.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3603999999999997e-05 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846437+0000", 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16402667400000001 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846442+0000", 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5390000000000001e-06 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846515+0000", 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3258999999999998e-05 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.846529+0000", 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4073999999999999e-05 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.847107+0000", 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057730800000000001 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.847140+0000", 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3439000000000003e-05 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17140 2.a e111/97 2:577f215e:::benchmark_data_smithi173_47195_object17139:head v 111'1039, mlcod=111'1035)", 2024-08-30T23:28:19.248 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.967877+0000", 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 128.25936287799999, 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.171820526, 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967877+0000", 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967877+0000", 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967878+0000", 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.967917+0000", 2024-08-30T23:28:19.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8785999999999998e-05 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.138852+0000", 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17093591899999999 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.138890+0000", 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7098999999999998e-05 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.138989+0000", 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9411000000000001e-05 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.139028+0000", 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9093e-05 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.139671+0000", 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064274799999999995 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.139697+0000", 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6665e-05 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.253 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17190 2.e e111/97 2:7818eb93:::benchmark_data_smithi173_47195_object17189:head v 111'1012, mlcod=111'1009)", 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.987305+0000", 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 128.239935059, 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12476780899999999, 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987305+0000", 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987305+0000", 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987306+0000", 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987334+0000", 2024-08-30T23:28:19.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8742999999999999e-05 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111418+0000", 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12408384 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111423+0000", 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2320000000000003e-06 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111465+0000", 2024-08-30T23:28:19.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2301e-05 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.111480+0000", 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5293000000000001e-05 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112044+0000", 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056360100000000005 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112073+0000", 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8829000000000001e-05 2024-08-30T23:28:19.260 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17191 2.6 e111/97 2:612ab7c0:::benchmark_data_smithi173_47195_object17190:head v 111'1091, mlcod=111'1088)", 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.987424+0000", 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 128.23981603300001, 2024-08-30T23:28:19.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123209083, 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987424+0000", 2024-08-30T23:28:19.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987424+0000", 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987425+0000", 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4900000000000005e-07 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.987478+0000", 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3032000000000003e-05 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109887+0000", 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.122409688 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109895+0000", 2024-08-30T23:28:19.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8129999999999996e-06 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109951+0000", 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5695999999999997e-05 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.109966+0000", 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5500000000000001e-05 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.110603+0000", 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000637139 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.110633+0000", 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9465999999999999e-05 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:17193 2.1 e111/97 2:840d9093:::benchmark_data_smithi173_47195_object17192:head v 111'1058, mlcod=111'1056)", 2024-08-30T23:28:19.267 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.989003+0000", 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 128.23823665, 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.124622043, 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989003+0000", 2024-08-30T23:28:19.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989003+0000", 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989004+0000", 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.989026+0000", 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2013000000000001e-05 2024-08-30T23:28:19.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112885+0000", 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123858969 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112890+0000", 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999998e-06 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112950+0000", 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9723999999999999e-05 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.112964+0000", 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4404e-05 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113594+0000", 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00062978700000000003 2024-08-30T23:28:19.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:11.113625+0000", 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1294000000000002e-05 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:36483 2.d e116/97 2:b8c0a5cd:::benchmark_data_smithi173_47195_object36482:head v 116'2353, mlcod=116'2350)", 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:25.049050+0000", 2024-08-30T23:28:19.273 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 114.178189827, 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16564358600000001, 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049050+0000", 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049050+0000", 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049051+0000", 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3e-07 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.049144+0000", 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3708999999999993e-05 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213246+0000", 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.164101892 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213253+0000", 2024-08-30T23:28:19.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2910000000000004e-06 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213314+0000", 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1643999999999995e-05 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.213339+0000", 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4748000000000001e-05 2024-08-30T23:28:19.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.214659+0000", 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001319656 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:25.214694+0000", 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4916000000000002e-05 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.278 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:56060 2.e e124/97 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head v 124'3464, mlcod=124'3461)", 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 99.447410667, 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21259787399999999, 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779830+0000", 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-07 2024-08-30T23:28:19.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779851+0000", 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1341000000000001e-05 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988300+0000", 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208448405 2024-08-30T23:28:19.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988311+0000", 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1141000000000001e-05 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988383+0000", 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2733999999999999e-05 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988404+0000", 2024-08-30T23:28:19.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0509999999999998e-05 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992259+0000", 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003854704 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992427+0000", 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016835099999999999 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75397 2.1 e132/97 2:8c19b29d:::benchmark_data_smithi173_47195_object75396:head v 132'4751, mlcod=132'4748)", 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 82.618228638999994, 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22761530499999999, 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609012+0000", 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8199999999999998e-07 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609036+0000", 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4414999999999999e-05 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834165+0000", 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22512892100000001 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834172+0000", 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2589999999999997e-06 2024-08-30T23:28:19.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834226+0000", 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3919999999999999e-05 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.298e-05 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836584+0000", 2024-08-30T23:28:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023353750000000002 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836626+0000", 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2552999999999997e-05 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:28:19.288 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 82.598001628000006, 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.240592844, 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:28:19.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629239+0000", 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:28:19.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629261+0000", 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2293000000000002e-05 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867840+0000", 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23857915299999999 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867849+0000", 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4940000000000003e-06 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867913+0000", 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3818000000000005e-05 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867935+0000", 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2024000000000001e-05 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869794+0000", 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001859018 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869831+0000", 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7239e-05 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75429 2.1 e132/97 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head v 132'4755, mlcod=132'4752)", 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:28:19.294 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 82.268341614999997, 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24468055399999999, 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:28:19.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958899+0000", 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1939999999999999e-06 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958944+0000", 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4851999999999997e-05 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201529+0000", 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24258502100000001 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201540+0000", 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0744000000000001e-05 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201630+0000", 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0168999999999997e-05 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201650+0000", 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9477999999999999e-05 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203542+0000", 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00189236 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203579+0000", 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6736000000000002e-05 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.298 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:76360 2.d e134/97 2:b978bfc7:::benchmark_data_smithi173_47195_object76359:head v 134'4864, mlcod=134'4861)", 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:59.255093+0000", 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 79.972147233000001, 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.119381785, 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.299 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.255093+0000", 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.255093+0000", 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.255094+0000", 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7999999999999999e-06 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.255237+0000", 2024-08-30T23:28:19.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014259099999999999 2024-08-30T23:28:19.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:19.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:19.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.371968+0000", 2024-08-30T23:28:19.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11673093499999999 2024-08-30T23:28:19.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.371974+0000", 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4189999999999999e-06 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.372025+0000", 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0646999999999998e-05 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.372044+0000", 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9327999999999999e-05 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.374424+0000", 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023792800000000001 2024-08-30T23:28:19.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.374474+0000", 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0785000000000003e-05 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:76487 2.e e134/97 2:799d422a:::benchmark_data_smithi173_47195_object76486:head v 134'4728, mlcod=134'4724)", 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:59.455226+0000", 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 79.772013337000004, 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.121414939, 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.309 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.455226+0000", 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.455226+0000", 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.455227+0000", 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1800000000000005e-07 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.455251+0000", 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3859999999999999e-05 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.574203+0000", 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.118952103 2024-08-30T23:28:19.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.574211+0000", 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1189999999999997e-06 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.574277+0000", 2024-08-30T23:28:19.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5313000000000003e-05 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.574300+0000", 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3068999999999999e-05 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.576607+0000", 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023076120000000001 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:59.576641+0000", 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4044999999999997e-05 2024-08-30T23:28:19.315 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:97004 2.1 e146/97 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head v 146'6102, mlcod=146'6098)", 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.162125+0000", 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.065115057, 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.123852711, 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.162125+0000", 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.162125+0000", 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.162125+0000", 2024-08-30T23:28:19.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7000000000000001e-07 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.162143+0000", 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8088e-05 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.283190+0000", 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12104680800000001 2024-08-30T23:28:19.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.283198+0000", 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7800000000000001e-06 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.283251+0000", 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2840000000000002e-05 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.283269+0000", 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8247999999999999e-05 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.285933+0000", 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026644070000000001 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.285977+0000", 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4270000000000001e-05 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.320 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.321 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.321 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100665 2.6 e147/142 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head v 147'6316, mlcod=147'6313)", 2024-08-30T23:28:19.321 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:28:19.321 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 45.883047036999997, 2024-08-30T23:28:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214342806, 2024-08-30T23:28:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.322 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:28:19.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344194+0000", 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344200+0000", 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5200000000000003e-06 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555870+0000", 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21167006199999999 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555878+0000", 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8420000000000005e-06 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555957+0000", 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8789999999999996e-05 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555979+0000", 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2206999999999999e-05 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.558491+0000", 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025112960000000001 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.558536+0000", 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5039e-05 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.327 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100667 2.6 e147/142 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head v 147'6317, mlcod=147'6313)", 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 45.879161771, 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.273264648, 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348079+0000", 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1459999999999999e-06 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348086+0000", 2024-08-30T23:28:19.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6649999999999997e-06 2024-08-30T23:28:19.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615008+0000", 2024-08-30T23:28:19.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26692174899999999 2024-08-30T23:28:19.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615016+0000", 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0909999999999993e-06 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615082+0000", 2024-08-30T23:28:19.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6359000000000006e-05 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615102+0000", 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9945999999999998e-05 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621308+0000", 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006205828 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621343+0000", 2024-08-30T23:28:19.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4863999999999997e-05 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.7 153/138 [PushOp(2:ee43aa8a:::benchmark_data_smithi173_47195_object27129:head, version: 115'1693, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ee43aa8a:::benchmark_data_smithi173_47195_object27129:head@115'1693, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:45.922998+0000", 2024-08-30T23:28:19.334 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.304241621999999, 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12384184400000001, 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.922998+0000", 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.922998+0000", 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.922999+0000", 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0079999999999999e-06 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.923067+0000", 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7341999999999993e-05 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.044231+0000", 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12116470999999999 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.044248+0000", 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6300999999999999e-05 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.044343+0000", 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5836999999999997e-05 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.044356+0000", 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2503999999999999e-05 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.046840+0000", 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024841419999999999 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.339 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.7 153/138 [PushOp(2:e5d4c559:::benchmark_data_smithi173_47195_object27159:head, version: 115'1695, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:e5d4c559:::benchmark_data_smithi173_47195_object27159:head@115'1695, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:46.087340+0000", 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.139899667999998, 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15073983599999999, 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.087340+0000", 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.087340+0000", 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.087341+0000", 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8700000000000005e-07 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.087423+0000", 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2311999999999994e-05 2024-08-30T23:28:19.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.234992+0000", 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.147568844 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.235010+0000", 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.825e-05 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.235227+0000", 2024-08-30T23:28:19.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000217104 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.235255+0000", 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7858999999999999e-05 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:46.238080+0000", 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00282468 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.7 153/138 [PushOp(2:ef3e66f6:::benchmark_data_smithi173_47195_object27966:head, version: 115'1748, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:ef3e66f6:::benchmark_data_smithi173_47195_object27966:head@115'1748, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:47.931986+0000", 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 31.295253499000001, 2024-08-30T23:28:19.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13152343799999999, 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:47.931986+0000", 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:47.931986+0000", 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:47.931987+0000", 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6300000000000004e-07 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:47.932039+0000", 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1912e-05 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:48.060418+0000", 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.128378827 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:48.060436+0000", 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8097e-05 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:48.060749+0000", 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031284500000000002 2024-08-30T23:28:19.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:48.060762+0000", 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3295999999999999e-05 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:48.063510+0000", 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027476979999999998 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:28:19.349 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:28:19.349 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:28:19.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:19.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:19.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:19.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:19.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:19.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:19.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:19.756 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:19 smithi106 ceph-mon[16603]: pgmap v439: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 5088/179025 objects misplaced (2.842%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:28:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:19 smithi136 ceph-mon[20207]: pgmap v439: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 5088/179025 objects misplaced (2.842%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:28:19.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:19.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:19 smithi173 ceph-mon[20554]: pgmap v439: 17 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 1 active+recovering, 12 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 5088/179025 objects misplaced (2.842%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:28:19.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:19.958 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:19.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:20.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:20.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:20.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:20.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:20.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:20.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:20.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:20.476 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T23:28:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:20.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:20.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:20.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:20.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:28:20.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:20.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1109954380' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:28:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:20 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:28:20.798 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1109954380' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:28:20.798 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:20 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:28:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1109954380' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:28:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:20 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:28:20.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:20.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:20.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:28:20.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:20.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:20.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:21.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:21.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.124 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:21.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:21.209 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:21.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:21.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:21.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:21.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:21.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:21.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:21.593 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:21 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:28:21.593 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:21 smithi136 ceph-mon[20207]: osdmap e166: 12 total, 12 up, 11 in 2024-08-30T23:28:21.593 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:21 smithi136 ceph-mon[20207]: pgmap v441: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 5088/174741 objects misplaced (2.912%); 11 MiB/s, 180 objects/s recovering 2024-08-30T23:28:21.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:21.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:21.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:137807 2.d e166/97 2:b051fc20:::benchmark_data_smithi173_47195_object40799:head v 166'8730, mlcod=166'8727)", 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:21.736605+0000", 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.001004598, 2024-08-30T23:28:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001034281, 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736605+0000", 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736605+0000", 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736606+0000", 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.004e-06 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736612+0000", 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4530000000000001e-06 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736643+0000", 2024-08-30T23:28:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1667e-05 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736649+0000", 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8250000000000001e-06 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736703+0000", 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3584000000000001e-05 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:21.736723+0000", 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0574999999999999e-05 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:28:21.746 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:28:21.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:21.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:21 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:28:21.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:21 smithi106 ceph-mon[16603]: osdmap e166: 12 total, 12 up, 11 in 2024-08-30T23:28:21.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:21 smithi106 ceph-mon[16603]: pgmap v441: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 5088/174741 objects misplaced (2.912%); 11 MiB/s, 180 objects/s recovering 2024-08-30T23:28:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:21 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:28:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:21 smithi173 ceph-mon[20554]: osdmap e166: 12 total, 12 up, 11 in 2024-08-30T23:28:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:21 smithi173 ceph-mon[20554]: pgmap v441: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 5088/174741 objects misplaced (2.912%); 11 MiB/s, 180 objects/s recovering 2024-08-30T23:28:21.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:21.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:21.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:21.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-08-30T23:28:21.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:22.018 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:22.114 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:22.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:22.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:22.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:22.291 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:22.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:22.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:22.489 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:28:22.508 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:22 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:22.509 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:22 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:22.509 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2163555356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:22.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:22.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:28:22.549 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:22.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:22.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:22.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:22.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:22 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:22 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2163555356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:22.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:22 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:22 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2163555356' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:22.904 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:22.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:22.944 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:22.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:23.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:23.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:23.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:23.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:23.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:23.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:23.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:23.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:23.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:23.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:23.639 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:23 smithi136 ceph-mon[20207]: 2.2 scrub starts 2024-08-30T23:28:23.639 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:23 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:28:23.640 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2163555356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:23.640 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:23 smithi136 ceph-mon[20207]: osdmap e167: 12 total, 12 up, 11 in 2024-08-30T23:28:23.640 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:23 smithi136 ceph-mon[20207]: pgmap v443: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 3670/170355 objects misplaced (2.154%); 24 MiB/s, 389 objects/s recovering 2024-08-30T23:28:23.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:23.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:23.766 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:28:23.766 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:28:23.766 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:28:23.766 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:28:23.766 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:28:23.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:23 smithi106 ceph-mon[16603]: 2.2 scrub starts 2024-08-30T23:28:23.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:23 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:28:23.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2163555356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:23.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:23 smithi106 ceph-mon[16603]: osdmap e167: 12 total, 12 up, 11 in 2024-08-30T23:28:23.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:23 smithi106 ceph-mon[16603]: pgmap v443: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 3670/170355 objects misplaced (2.154%); 24 MiB/s, 389 objects/s recovering 2024-08-30T23:28:23.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:23 smithi173 ceph-mon[20554]: 2.2 scrub starts 2024-08-30T23:28:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:23 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:28:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2163555356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:23 smithi173 ceph-mon[20554]: osdmap e167: 12 total, 12 up, 11 in 2024-08-30T23:28:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:23 smithi173 ceph-mon[20554]: pgmap v443: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 773 op/s; 3670/170355 objects misplaced (2.154%); 24 MiB/s, 389 objects/s recovering 2024-08-30T23:28:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.854 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:23.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:23.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:23.964 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:23.964 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:23.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:23.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:24.004 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:24.004 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:24.045 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:24.045 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:24.082 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:24.082 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:24.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:24.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:24.117 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:24.118 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:24.153 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:24.153 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:24.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:24.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:24.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:24.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:24.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:24.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:24.256 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:24.256 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:24.293 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:24.293 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:24.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:24.338 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:24.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:24.365 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:24.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:24.377 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:24.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:24.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:28:24.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:24.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:24.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:24 smithi106 ceph-mon[16603]: 2.2 scrub ok 2024-08-30T23:28:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2970159371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:24 smithi106 ceph-mon[16603]: 2.8 scrub starts 2024-08-30T23:28:24.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:24.686 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:24.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:24.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:24 smithi136 ceph-mon[20207]: 2.2 scrub ok 2024-08-30T23:28:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2970159371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:24 smithi136 ceph-mon[20207]: 2.8 scrub starts 2024-08-30T23:28:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:24 smithi173 ceph-mon[20554]: 2.2 scrub ok 2024-08-30T23:28:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2970159371' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:24 smithi173 ceph-mon[20554]: 2.8 scrub starts 2024-08-30T23:28:24.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:24.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:24.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:24.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:25.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:25.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:25.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:25.164 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:25.164 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:25.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:25.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:25.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:25.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:25.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:25.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:25.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:25.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:25.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:25.745 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:25 smithi136 ceph-mon[20207]: osdmap e168: 12 total, 12 up, 11 in 2024-08-30T23:28:25.746 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:25 smithi136 ceph-mon[20207]: 2.8 scrub ok 2024-08-30T23:28:25.746 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:25 smithi136 ceph-mon[20207]: pgmap v445: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 3670/166857 objects misplaced (2.199%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:28:25.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:25.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:28:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:25 smithi173 ceph-mon[20554]: osdmap e168: 12 total, 12 up, 11 in 2024-08-30T23:28:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:25 smithi173 ceph-mon[20554]: 2.8 scrub ok 2024-08-30T23:28:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:25 smithi173 ceph-mon[20554]: pgmap v445: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 3670/166857 objects misplaced (2.199%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:28:25.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:25 smithi106 ceph-mon[16603]: osdmap e168: 12 total, 12 up, 11 in 2024-08-30T23:28:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:25 smithi106 ceph-mon[16603]: 2.8 scrub ok 2024-08-30T23:28:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:25 smithi106 ceph-mon[16603]: pgmap v445: 17 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 675 op/s; 3670/166857 objects misplaced (2.199%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:28:25.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:25.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:25.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.026 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:26.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:26.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:26.134 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:26.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:26.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:26.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:26.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:28:26.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:26.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:26.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:26.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:26 smithi136 ceph-mon[20207]: osdmap e169: 12 total, 12 up, 11 in 2024-08-30T23:28:26.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:26 smithi136 ceph-mon[20207]: osdmap e170: 12 total, 12 up, 11 in 2024-08-30T23:28:26.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:26 smithi136 ceph-mon[20207]: 2.c scrub starts 2024-08-30T23:28:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:26.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:26.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:26.657 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:26.657 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:26.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:26.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:26.779 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:26.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:26 smithi173 ceph-mon[20554]: osdmap e169: 12 total, 12 up, 11 in 2024-08-30T23:28:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:26 smithi173 ceph-mon[20554]: osdmap e170: 12 total, 12 up, 11 in 2024-08-30T23:28:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:26 smithi173 ceph-mon[20554]: 2.c scrub starts 2024-08-30T23:28:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:26 smithi106 ceph-mon[16603]: osdmap e169: 12 total, 12 up, 11 in 2024-08-30T23:28:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:26 smithi106 ceph-mon[16603]: osdmap e170: 12 total, 12 up, 11 in 2024-08-30T23:28:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:26 smithi106 ceph-mon[16603]: 2.c scrub starts 2024-08-30T23:28:26.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:26.929 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:26.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:27.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:27.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:27.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:27.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:27.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:27.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:27.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:28:27.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:27.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:27.517 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T23:28:27.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:27.576 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:27.641 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:27.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:27.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:27.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2544157319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:28:27.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:27 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:28:27.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:27 smithi136 ceph-mon[20207]: 2.c scrub ok 2024-08-30T23:28:27.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:27 smithi136 ceph-mon[20207]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 3317/162072 objects misplaced (2.047%); 12 MiB/s, 185 objects/s recovering 2024-08-30T23:28:27.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2544157319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:28:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:27 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:28:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:27 smithi173 ceph-mon[20554]: 2.c scrub ok 2024-08-30T23:28:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:27 smithi173 ceph-mon[20554]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 3317/162072 objects misplaced (2.047%); 12 MiB/s, 185 objects/s recovering 2024-08-30T23:28:27.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2544157319' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:28:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:27 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:28:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:27 smithi106 ceph-mon[16603]: 2.c scrub ok 2024-08-30T23:28:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:27 smithi106 ceph-mon[16603]: pgmap v448: 17 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering, 13 active+clean; 3.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 665 op/s; 3317/162072 objects misplaced (2.047%); 12 MiB/s, 185 objects/s recovering 2024-08-30T23:28:27.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:27.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:27.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:28.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:28:28.075 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:28.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:28.181 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:28.181 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:28.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:28.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:28.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-08-30T23:28:28.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:28.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:28.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:28.489 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.521 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:28.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:28.529 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:28:28.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:28.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:28.750 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:28.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:28.784 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:28 smithi136 ceph-mon[20207]: 2.d scrub starts 2024-08-30T23:28:28.785 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:28 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:28:28.785 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:28 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:28:28.785 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:28 smithi136 ceph-mon[20207]: osdmap e171: 12 total, 12 up, 11 in 2024-08-30T23:28:28.785 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/641272535' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:28.785 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:28 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:28 smithi173 ceph-mon[20554]: 2.d scrub starts 2024-08-30T23:28:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:28 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:28:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:28 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:28:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:28 smithi173 ceph-mon[20554]: osdmap e171: 12 total, 12 up, 11 in 2024-08-30T23:28:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/641272535' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:28 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:28.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:28:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:28 smithi106 ceph-mon[16603]: 2.d scrub starts 2024-08-30T23:28:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:28 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:28:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:28 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:28:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:28 smithi106 ceph-mon[16603]: osdmap e171: 12 total, 12 up, 11 in 2024-08-30T23:28:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/641272535' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:28 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:28.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:28.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:28.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:29.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:29.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.042 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:29.139 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:29.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:29.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:29.279 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:29.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:29.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:29.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:29.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:29.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:29.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:29.657 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:29.657 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:29.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:29.747 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:29.748 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:29.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:29 smithi106 ceph-mon[16603]: 2.d scrub ok 2024-08-30T23:28:29.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:29 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:29.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:29 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:28:29.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:29 smithi106 ceph-mon[16603]: osdmap e172: 12 total, 12 up, 11 in 2024-08-30T23:28:29.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:29 smithi106 ceph-mon[16603]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering, 14 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2989/157767 objects misplaced (1.895%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:28:29.754 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:29.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:29.783 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:29.783 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:29 smithi136 ceph-mon[20207]: 2.d scrub ok 2024-08-30T23:28:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:29 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:29 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:28:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:29 smithi136 ceph-mon[20207]: osdmap e172: 12 total, 12 up, 11 in 2024-08-30T23:28:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:29 smithi136 ceph-mon[20207]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering, 14 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2989/157767 objects misplaced (1.895%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:28:29.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:29.813 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:29.813 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:29 smithi173 ceph-mon[20554]: 2.d scrub ok 2024-08-30T23:28:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:29 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:29 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:28:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:29 smithi173 ceph-mon[20554]: osdmap e172: 12 total, 12 up, 11 in 2024-08-30T23:28:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:29 smithi173 ceph-mon[20554]: pgmap v451: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovering, 14 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2989/157767 objects misplaced (1.895%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:28:29.842 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:29.842 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:29.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:29.872 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:29.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:29.902 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:29.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:29.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:29.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:29.939 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:29.939 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:30.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:30.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:30.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:30.179 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:30.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:30.203 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:30.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:30.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:30.256 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:30.256 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:30.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:28:30.297 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:30.297 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:30.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:30.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:30.337 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:30.337 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:30.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:30.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:30.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:30.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:30.597 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:30.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:30.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:30.817 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2409011157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:30 smithi173 ceph-mon[20554]: osdmap e173: 12 total, 12 up, 11 in 2024-08-30T23:28:30.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2409011157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:30 smithi106 ceph-mon[16603]: osdmap e173: 12 total, 12 up, 11 in 2024-08-30T23:28:30.892 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2409011157' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:30.892 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:30 smithi136 ceph-mon[20207]: osdmap e173: 12 total, 12 up, 11 in 2024-08-30T23:28:30.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:30.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:30.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:31.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:31.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:31.163 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:28:31.163 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:144458 2.7 e173/138 ondisk, result = 0)", 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:31.162386+0000", 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00062063999999999999, 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064195300000000001, 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162386+0000", 2024-08-30T23:28:31.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162386+0000", 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162388+0000", 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.243e-06 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162396+0000", 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7970000000000006e-06 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162430+0000", 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3791999999999998e-05 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162438+0000", 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0530000000000007e-06 2024-08-30T23:28:31.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162488+0000", 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9617000000000003e-05 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162494+0000", 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0000000000000002e-06 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 173/138 [PushReplyOp(2:ef6861fb:::benchmark_data_smithi173_47195_object63581:head)])", 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:31.162550+0000", 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00045700299999999999, 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053393500000000003, 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162550+0000", 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162550+0000", 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162551+0000", 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2500000000000005e-07 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162558+0000", 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1629999999999999e-06 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162618+0000", 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0276999999999999e-05 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162626+0000", 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8099999999999998e-06 2024-08-30T23:28:31.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:144458 2.7 2:e8a5d7e8:::benchmark_data_smithi173_47195_object47450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:31.147898+0000", 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01510855, 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015243009, 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:31.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 144458 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.147898+0000", 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.147898+0000", 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.147900+0000", 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5489999999999999e-06 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.147905+0000", 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4790000000000004e-06 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.147920+0000", 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4174e-05 2024-08-30T23:28:31.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.147928+0000", 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6889999999999993e-06 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.148001+0000", 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3113000000000003e-05 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:31.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.148160+0000", 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000158326 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.148313+0000", 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015330199999999999 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.152280+0000", 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039665309999999997 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.153096+0000", 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081601600000000003 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162498+0000", 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094022689999999996 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.162540+0000", 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1758999999999998e-05 2024-08-30T23:28:31.176 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.177 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:31.177 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.177 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:31.177 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:28:31.177 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 3 2024-08-30T23:28:31.177 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:28:31.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:31.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:31.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:31.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:31.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:31.493 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:31.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:28:31.549 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T23:28:31.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2321868160' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:28:31.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:31 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:28:31.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:31 smithi106 ceph-mon[16603]: pgmap v453: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2989/153546 objects misplaced (1.947%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:28:31.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:31.632 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:31.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:31.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2321868160' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:28:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:31 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:28:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:31 smithi173 ceph-mon[20554]: pgmap v453: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2989/153546 objects misplaced (1.947%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:28:31.850 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2321868160' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:28:31.850 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:31 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:28:31.850 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:31 smithi136 ceph-mon[20207]: pgmap v453: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2989/153546 objects misplaced (1.947%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:28:31.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:31.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:31.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:31.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:28:31.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:32.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:32.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:32.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:32.196 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:32.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-08-30T23:28:32.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:32.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:32.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:32.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:32.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:32.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:32.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:32.610 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:32 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:28:32.610 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:32 smithi136 ceph-mon[20207]: osdmap e174: 12 total, 12 up, 11 in 2024-08-30T23:28:32.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:32.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:32.729 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:28:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:28:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:28:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:28:32.730 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:28:32.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:32 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:28:32.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:32 smithi106 ceph-mon[16603]: osdmap e174: 12 total, 12 up, 11 in 2024-08-30T23:28:32.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:32.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:32 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:28:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:32 smithi173 ceph-mon[20554]: osdmap e174: 12 total, 12 up, 11 in 2024-08-30T23:28:32.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:32.915 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:32.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:32.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:33.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:33.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:33.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:33.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:33.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:33.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:33.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:28:33.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:28:33.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:33.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:33.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:33.572 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:28:33.597 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:33.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:33.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:33.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/686601542' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:33.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:33.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:33 smithi106 ceph-mon[16603]: pgmap v455: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.0 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 2422/149484 objects misplaced (1.620%); 9.5 MiB/s, 152 objects/s recovering 2024-08-30T23:28:33.676 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:33.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:33.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/686601542' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:33 smithi136 ceph-mon[20207]: pgmap v455: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.0 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 2422/149484 objects misplaced (1.620%); 9.5 MiB/s, 152 objects/s recovering 2024-08-30T23:28:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/686601542' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:33 smithi173 ceph-mon[20554]: pgmap v455: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.0 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 680 op/s; 2422/149484 objects misplaced (1.620%); 9.5 MiB/s, 152 objects/s recovering 2024-08-30T23:28:33.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:33.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:33.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:33.990 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:33.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:34.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:34.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:34.207 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:842 2.7 2:ebe581cf:::benchmark_data_smithi173_47195_object841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.579198+0000", 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.626643008, 2024-08-30T23:28:34.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.152788497, 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 842 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.209 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579198+0000", 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579198+0000", 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579200+0000", 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7409999999999999e-06 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579255+0000", 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5479000000000002e-05 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579270+0000", 2024-08-30T23:28:34.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4589999999999999e-05 2024-08-30T23:28:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579276+0000", 2024-08-30T23:28:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2999999999999998e-06 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579326+0000", 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9539999999999997e-05 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579402+0000", 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6378000000000006e-05 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579487+0000", 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5186999999999996e-05 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.579829+0000", 2024-08-30T23:28:34.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000341847 2024-08-30T23:28:34.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.580240+0000", 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041074599999999997 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.731905+0000", 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15166563 2024-08-30T23:28:34.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.731939+0000", 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3967000000000002e-05 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.731986+0000", 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7092000000000002e-05 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1905 2.7 2:e3c3e2b6:::benchmark_data_smithi173_47195_object1904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.88572855999999, 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.214235494, 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1905 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320114+0000", 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8139999999999999e-06 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320140+0000", 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5607999999999999e-05 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320149+0000", 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1779999999999992e-06 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320156+0000", 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:28:34.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320214+0000", 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7816999999999999e-05 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320293+0000", 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9528000000000002e-05 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320375+0000", 2024-08-30T23:28:34.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1314000000000001e-05 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320722+0000", 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034726800000000002 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.321314+0000", 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059229900000000002 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534242+0000", 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21292789500000001 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534303+0000", 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0656000000000003e-05 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534348+0000", 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5039e-05 2024-08-30T23:28:34.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1915 2.7 2:e31a5d2a:::benchmark_data_smithi173_47195_object1914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.88354628600001, 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21216159100000001, 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1915 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3600000000000002e-07 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322318+0000", 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2801000000000001e-05 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322326+0000", 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3629999999999996e-06 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322330+0000", 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8069999999999999e-06 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322371+0000", 2024-08-30T23:28:34.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0200000000000001e-05 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322445+0000", 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4750000000000001e-05 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322526+0000", 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0927000000000006e-05 2024-08-30T23:28:34.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322865+0000", 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033871900000000002 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.323144+0000", 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027930899999999999 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534388+0000", 2024-08-30T23:28:34.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21124331399999999 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534422+0000", 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4087000000000001e-05 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534456+0000", 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4477999999999998e-05 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1922 2.7 2:ee8ad553:::benchmark_data_smithi173_47195_object1921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.87916334400001, 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207961537, 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.232 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1922 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:28:34.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326753+0000", 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4973000000000002e-05 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326764+0000", 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0787e-05 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326769+0000", 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0189999999999997e-06 2024-08-30T23:28:34.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326810+0000", 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.125e-05 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326889+0000", 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8110999999999998e-05 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326971+0000", 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2132999999999994e-05 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.327277+0000", 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030633899999999998 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.327759+0000", 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048190199999999998 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534555+0000", 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20679639799999999 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534604+0000", 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8588999999999998e-05 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534639+0000", 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5185000000000003e-05 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1930 2.7 2:ea8dc1bc:::benchmark_data_smithi173_47195_object1929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.87566850900001, 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20455900900000001, 2024-08-30T23:28:34.239 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1930 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:28:34.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:28:34.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330174+0000", 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1540000000000001e-06 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330209+0000", 2024-08-30T23:28:34.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5206000000000001e-05 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330220+0000", 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1443999999999999e-05 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330225+0000", 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2299999999999999e-06 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330272+0000", 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6168000000000002e-05 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330386+0000", 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114523 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330484+0000", 2024-08-30T23:28:34.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7739999999999996e-05 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330815+0000", 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000331202 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.331211+0000", 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039613199999999999 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534670+0000", 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203458423 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534698+0000", 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8432000000000002e-05 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534731+0000", 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3355000000000002e-05 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.247 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1967 2.7 2:e55fccac:::benchmark_data_smithi173_47195_object1966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.85754532799999, 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18683412999999999, 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.248 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1967 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348297+0000", 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.548e-06 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348327+0000", 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0312e-05 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348342+0000", 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4647000000000001e-05 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348365+0000", 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3098000000000001e-05 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348409+0000", 2024-08-30T23:28:34.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4261000000000002e-05 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348496+0000", 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6846000000000002e-05 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348576+0000", 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9411000000000003e-05 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348908+0000", 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033185099999999998 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.349356+0000", 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044820899999999998 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535032+0000", 2024-08-30T23:28:34.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18567658200000001 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535085+0000", 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2559000000000003e-05 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535130+0000", 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4805999999999998e-05 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1975 2.7 2:eeb7a2cc:::benchmark_data_smithi173_47195_object1974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.85259921599999, 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18200930000000001, 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1975 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:28:34.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353243+0000", 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5899999999999995e-07 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353322+0000", 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9729999999999997e-05 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353335+0000", 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2719e-05 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353340+0000", 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353381+0000", 2024-08-30T23:28:34.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0828000000000003e-05 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353459+0000", 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8151000000000004e-05 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353543+0000", 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3549999999999998e-05 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353838+0000", 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029477600000000001 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.354159+0000", 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000321007 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535175+0000", 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181016702 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535227+0000", 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1307000000000002e-05 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535251+0000", 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4453000000000001e-05 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1977 2.7 2:e9772656:::benchmark_data_smithi173_47195_object1976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.850649248, 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180200314, 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.262 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1977 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.09e-07 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355252+0000", 2024-08-30T23:28:34.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9171e-05 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355262+0000", 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0259e-05 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355267+0000", 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.938e-06 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355296+0000", 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9252e-05 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355364+0000", 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7589000000000006e-05 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355445+0000", 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1457000000000003e-05 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355716+0000", 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027039099999999998 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.356116+0000", 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039998099999999999 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535289+0000", 2024-08-30T23:28:34.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17917321 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535356+0000", 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7482000000000002e-05 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535392+0000", 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5775000000000002e-05 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:36436 2.2 e116/97 2:4f13755f:::benchmark_data_smithi173_47195_object36435:head v 116'2235, mlcod=116'2231)", 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 129.22227261899999, 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23314933600000001, 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983569+0000", 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2799999999999995e-07 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983591+0000", 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1899000000000002e-05 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215327+0000", 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23173639900000001 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215335+0000", 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.216e-06 2024-08-30T23:28:34.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215397+0000", 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2348999999999995e-05 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215421+0000", 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4114e-05 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216674+0000", 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012534379999999999 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216718+0000", 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3192999999999998e-05 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:36447 2.2 e116/97 2:42c8382e:::benchmark_data_smithi173_47195_object36446:head v 116'2236, mlcod=116'2232)", 2024-08-30T23:28:34.273 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 129.21722153600001, 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310671684, 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988620+0000", 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.099e-06 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988646+0000", 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5996999999999999e-05 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297448+0000", 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30880194900000002 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297456+0000", 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5059999999999998e-06 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297540+0000", 2024-08-30T23:28:34.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4086999999999996e-05 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297565+0000", 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5304e-05 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299247+0000", 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016817690000000001 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299291+0000", 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3973000000000001e-05 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:40558 2.7 2:ec8f91a5:::benchmark_data_smithi173_47195_object40557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 126.276013733, 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18168003199999999, 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.278 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 40558 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929828+0000", 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.23e-07 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929928+0000", 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100546 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929938+0000", 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8260000000000005e-06 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929945+0000", 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6669999999999999e-06 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930530+0000", 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058469100000000003 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930611+0000", 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1532000000000005e-05 2024-08-30T23:28:34.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930702+0000", 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0371000000000006e-05 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931513+0000", 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081120499999999996 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111320+0000", 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17980744600000001 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111443+0000", 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123242 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111479+0000", 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5734e-05 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111507+0000", 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8048999999999999e-05 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:40558 2.7 e119/97 ondisk, result = 0)", 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 126.27396627500001, 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17945143799999999, 2024-08-30T23:28:34.285 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931878+0000", 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5519999999999999e-06 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111265+0000", 2024-08-30T23:28:34.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179387094 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111270+0000", 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1100000000000002e-06 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111310+0000", 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0417999999999998e-05 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111316+0000", 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0809999999999999e-06 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111326+0000", 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7559999999999994e-06 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.290 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:40558 2.7 e119/97 ondisk, result = 0)", 2024-08-30T23:28:34.290 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:28:34.290 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 126.27393934, 2024-08-30T23:28:34.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179610452, 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931905+0000", 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7190000000000001e-06 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111399+0000", 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17949393599999999 2024-08-30T23:28:34.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111403+0000", 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3719999999999998e-06 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111434+0000", 2024-08-30T23:28:34.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1375999999999999e-05 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111440+0000", 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8209999999999998e-06 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111512+0000", 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1800999999999999e-05 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50002 2.7 2:e84aa78e:::benchmark_data_smithi173_47195_object50001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 118.937401698, 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.199583015, 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50002 2024-08-30T23:28:34.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268441+0000", 2024-08-30T23:28:34.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9139999999999998e-06 2024-08-30T23:28:34.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268476+0000", 2024-08-30T23:28:34.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4665000000000001e-05 2024-08-30T23:28:34.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268490+0000", 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4104000000000001e-05 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268498+0000", 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2619999999999998e-06 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268559+0000", 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0537e-05 2024-08-30T23:28:34.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268646+0000", 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7587e-05 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268736+0000", 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9363999999999993e-05 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.269952+0000", 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012166290000000001 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.270470+0000", 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051778800000000004 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.467915+0000", 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.197445024 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.467979+0000", 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.41e-05 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.468022+0000", 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3040999999999997e-05 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:50013 2.2 e122/97 2:45c153d4:::benchmark_data_smithi173_47195_object50012:head v 122'3079, mlcod=122'3076)", 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.314034+0000", 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 118.89180671, 2024-08-30T23:28:34.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15255661600000001, 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.314034+0000", 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.314034+0000", 2024-08-30T23:28:34.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.314035+0000", 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.062e-06 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.314057+0000", 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1619000000000001e-05 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.464873+0000", 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15081602799999999 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.464885+0000", 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.165e-05 2024-08-30T23:28:34.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.464957+0000", 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2089999999999996e-05 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.464980+0000", 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3604999999999998e-05 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.466555+0000", 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015750460000000001 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.466591+0000", 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5515999999999997e-05 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:56072 2.7 2:e7493b3b:::benchmark_data_smithi173_47195_object56071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:39.795290+0000", 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 114.410550852, 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17400230799999999, 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.312 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 56072 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795290+0000", 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795290+0000", 2024-08-30T23:28:34.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795296+0000", 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2960000000000004e-06 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795343+0000", 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6869000000000001e-05 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795374+0000", 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.04e-05 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795390+0000", 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6711999999999999e-05 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795482+0000", 2024-08-30T23:28:34.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1751999999999998e-05 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795571+0000", 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8468999999999996e-05 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795655+0000", 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4734000000000006e-05 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.797013+0000", 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013579130000000001 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.797466+0000", 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045263200000000001 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.969189+0000", 2024-08-30T23:28:34.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17172362799999999 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.969250+0000", 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0625000000000003e-05 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.969292+0000", 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2277999999999997e-05 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.318 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:67199 2.2 e129/97 2:462394bd:::benchmark_data_smithi173_47195_object67198:head v 129'4136, mlcod=129'4133)", 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:49.679938+0000", 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 104.52590321700001, 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14385173000000001, 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.679938+0000", 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.679938+0000", 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.679939+0000", 2024-08-30T23:28:34.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0330000000000001e-06 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.679961+0000", 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2768e-05 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.822239+0000", 2024-08-30T23:28:34.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14227788799999999 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.822244+0000", 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4440000000000001e-06 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.822293+0000", 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9425000000000002e-05 2024-08-30T23:28:34.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.822308+0000", 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5043e-05 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.823760+0000", 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014514899999999999 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:49.823789+0000", 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9638999999999999e-05 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 146/138 [PushReplyOp(2:ee294341:::benchmark_data_smithi173_47195_object15777:head)])", 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 66.014809291000006, 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.001463438, 2024-08-30T23:28:34.324 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191037+0000", 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2749999999999997e-06 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181108+0000", 2024-08-30T23:28:34.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99007171599999999 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181115+0000", 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.189300+0000", 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081848760000000007 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.192495+0000", 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031953250000000002 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 153/138 [PushReplyOp(2:eebfff31:::benchmark_data_smithi173_47195_object26886:head)])", 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 49.280810379000002, 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21501724999999999, 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.328 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.09e-07 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925033+0000", 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.419e-06 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136148+0000", 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.211115204 2024-08-30T23:28:34.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136156+0000", 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1880000000000002e-06 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136222+0000", 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5320999999999993e-05 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.140048+0000", 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003826009 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:126363 2.7 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:28:34.332 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.476604484999999, 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221802834, 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 126363 2024-08-30T23:28:34.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:28:34.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729238+0000", 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9360000000000002e-06 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729244+0000", 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8819999999999998e-06 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729258+0000", 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3994e-05 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729265+0000", 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0570000000000004e-06 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729318+0000", 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2410000000000001e-05 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729417+0000", 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9414999999999996e-05 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729508+0000", 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1209e-05 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.732169+0000", 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026607919999999999 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.737250+0000", 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050804370000000001 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949780+0000", 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21253043999999999 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949815+0000", 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5308000000000001e-05 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.951039+0000", 2024-08-30T23:28:34.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001223954 2024-08-30T23:28:34.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.341 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:28:34.341 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:28:34.341 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:28:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:34.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:34.355 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:34.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:34.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:34.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:34.573 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T23:28:34.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:34.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:34.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:34.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:34.836 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:34 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:34.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:34.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:34 smithi136 ceph-mon[20207]: osdmap e175: 12 total, 12 up, 11 in 2024-08-30T23:28:34.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2891192481' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:28:34.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:28:34.849 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:34 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:34 smithi106 ceph-mon[16603]: osdmap e175: 12 total, 12 up, 11 in 2024-08-30T23:28:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2891192481' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:28:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:28:34.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:34.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:34.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:34 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:34 smithi173 ceph-mon[20554]: osdmap e175: 12 total, 12 up, 11 in 2024-08-30T23:28:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2891192481' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:28:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:28:35.076 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:35.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:35.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:35.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:35.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:35.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:35.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:28:35.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:35.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:35.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:35.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:35.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:35.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:35.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:35.480 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:35.480 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:35.482 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:35.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-08-30T23:28:35.516 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:35.516 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:35.554 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:35.554 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:35.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:35.593 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:35.593 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:35.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:35.628 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:35.628 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:35.657 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:35.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:35.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:35.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:35.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:35.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:35.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:35.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:35.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:35 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:28:35.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:35 smithi106 ceph-mon[16603]: osdmap e176: 12 total, 12 up, 11 in 2024-08-30T23:28:35.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:28:35.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3855378370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:35.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:35 smithi106 ceph-mon[16603]: pgmap v458: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.0 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2422/145926 objects misplaced (1.660%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:28:35.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-30T23:28:35.798 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:35.799 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:35.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:35.885 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:35.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:35.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:35 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:28:35.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:35 smithi136 ceph-mon[20207]: osdmap e176: 12 total, 12 up, 11 in 2024-08-30T23:28:35.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:28:35.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3855378370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:35.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:35 smithi136 ceph-mon[20207]: pgmap v458: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.0 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2422/145926 objects misplaced (1.660%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:28:35.931 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-30T23:28:35.974 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:35 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:28:35.974 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:35 smithi173 ceph-mon[20554]: osdmap e176: 12 total, 12 up, 11 in 2024-08-30T23:28:35.974 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:28:35.974 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3855378370' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:35.974 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:35 smithi173 ceph-mon[20554]: pgmap v458: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 3.0 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 2422/145926 objects misplaced (1.660%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:28:35.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-08-30T23:28:35.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:36.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:36.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:36.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:36.196 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:36.197 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:36.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:36.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:36.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:28:36.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:36.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:36.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:36.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:36.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:36.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:36.689 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:36.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:36.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:36.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:28:36.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:28:36.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:36 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:36.858 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:28:36.858 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:28:36.858 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:36 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:36.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:28:36.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:36.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:36.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:37.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:37.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:28:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:28:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:36 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:37.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:37.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:37.189 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:37.255 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:37.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:37.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:37.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:37.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:37.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:37.560 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:37.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:37.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:37.702 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:37 smithi106 ceph-mon[16603]: pgmap v459: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.9 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 717 op/s; 2076/140637 objects misplaced (1.476%); 9.5 MiB/s, 152 objects/s recovering 2024-08-30T23:28:37.734 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:37.734 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:37.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:37.832 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:37.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:37.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:37.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:37 smithi136 ceph-mon[20207]: pgmap v459: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.9 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 717 op/s; 2076/140637 objects misplaced (1.476%); 9.5 MiB/s, 152 objects/s recovering 2024-08-30T23:28:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:37 smithi173 ceph-mon[20554]: pgmap v459: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.9 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 717 op/s; 2076/140637 objects misplaced (1.476%); 9.5 MiB/s, 152 objects/s recovering 2024-08-30T23:28:38.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:38.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:38.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:38.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:38.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:38.363 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:38.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:28:38.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:38.494 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:38.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:28:38.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:38.607 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.6 to 1 (10000) 2024-08-30T23:28:38.611 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:28:38.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:38.681 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:28:38.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:38.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:38.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:38.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1766726381' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:28:38.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/952448073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:38.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:38 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:38.892 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:38.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:38.947 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1766726381' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:28:38.947 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/952448073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:38.947 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:38 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:38.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:39.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1766726381' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:28:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/952448073' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:38 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:28:39.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:39.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:39.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:39.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:39.342 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:39.343 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:39.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:39.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:39.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:39.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:39.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-08-30T23:28:39.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:39.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:39.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:39.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:39 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1766726381' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:28:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:39 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:28:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:39 smithi106 ceph-mon[16603]: osdmap e177: 12 total, 12 up, 11 in 2024-08-30T23:28:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:39 smithi106 ceph-mon[16603]: 2.a scrub starts 2024-08-30T23:28:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:39 smithi106 ceph-mon[16603]: pgmap v461: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.8 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 1734/135954 objects misplaced (1.275%); 7.2 MiB/s, 114 objects/s recovering 2024-08-30T23:28:39.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:39.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:39.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:39.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:39 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:39.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1766726381' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:28:39.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:39 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:28:39.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:39 smithi136 ceph-mon[20207]: osdmap e177: 12 total, 12 up, 11 in 2024-08-30T23:28:39.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:39 smithi136 ceph-mon[20207]: 2.a scrub starts 2024-08-30T23:28:39.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:39 smithi136 ceph-mon[20207]: pgmap v461: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.8 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 1734/135954 objects misplaced (1.275%); 7.2 MiB/s, 114 objects/s recovering 2024-08-30T23:28:40.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:39 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1766726381' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:28:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:39 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:28:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:39 smithi173 ceph-mon[20554]: osdmap e177: 12 total, 12 up, 11 in 2024-08-30T23:28:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:39 smithi173 ceph-mon[20554]: 2.a scrub starts 2024-08-30T23:28:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:39 smithi173 ceph-mon[20554]: pgmap v461: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.8 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 751 op/s; 1734/135954 objects misplaced (1.275%); 7.2 MiB/s, 114 objects/s recovering 2024-08-30T23:28:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:40.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:40.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:28:40.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:40.231 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:40.249 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:40.291 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:40.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:40.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:40.452 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:40.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:40.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:40.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:40.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:40.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:40.738 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:40 smithi106 ceph-mon[16603]: 2.a scrub ok 2024-08-30T23:28:40.738 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:40 smithi106 ceph-mon[16603]: osdmap e178: 12 total, 12 up, 11 in 2024-08-30T23:28:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.776 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:40.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:40.858 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:40 smithi173 ceph-mon[20554]: 2.a scrub ok 2024-08-30T23:28:40.859 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:40 smithi173 ceph-mon[20554]: osdmap e178: 12 total, 12 up, 11 in 2024-08-30T23:28:40.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:40.913 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:40.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:40.917 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:40.917 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:40.958 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:40.958 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:40.999 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:40.999 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:41.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:41.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:41.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:40 smithi136 ceph-mon[20207]: 2.a scrub ok 2024-08-30T23:28:41.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:40 smithi136 ceph-mon[20207]: osdmap e178: 12 total, 12 up, 11 in 2024-08-30T23:28:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:41.049 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:41.049 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:41.085 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:41.088 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:41.118 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:41.118 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:41.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:41.154 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:41.154 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:41.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:41.190 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:41.190 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:41.235 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:41.235 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:41.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:41.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:41.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:41.318 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:41.323 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:41.323 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:41.358 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:41.358 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:41.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:41.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:41.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:41.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:41.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:41.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:41.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:41.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:41.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:41.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1123703284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:41.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:41 smithi173 ceph-mon[20554]: pgmap v463: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.7 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 1734/131697 objects misplaced (1.317%); 6.3 MiB/s, 101 objects/s recovering 2024-08-30T23:28:41.978 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:41.978 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:28:41.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:152374 2.f 2:ff5232bf:::benchmark_data_smithi173_47195_object55366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:41.973538+0000", 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0045856179999999996, 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046131100000000001, 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 152374 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.979 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973538+0000", 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973538+0000", 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:41.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973539+0000", 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9000000000000005e-07 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973543+0000", 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3660000000000002e-06 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973554+0000", 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1192000000000001e-05 2024-08-30T23:28:41.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973559+0000", 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7709999999999997e-06 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973600+0000", 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0834000000000002e-05 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:41.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973697+0000", 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6890000000000002e-05 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.973761+0000", 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4146999999999994e-05 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.977580+0000", 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038192569999999999 2024-08-30T23:28:41.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.977756+0000", 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017551099999999999 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:152378 2.6 2:63d57cad:::benchmark_data_smithi173_47195_object55370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:41.975710+0000", 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0024137949999999998, 2024-08-30T23:28:41.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024950939999999998, 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 152378 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975710+0000", 2024-08-30T23:28:41.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975710+0000", 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975711+0000", 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.054e-06 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975716+0000", 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9420000000000003e-06 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975729+0000", 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3389e-05 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975737+0000", 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0399999999999993e-06 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975795+0000", 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7887999999999999e-05 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.975902+0000", 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106758 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.976008+0000", 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106456 2024-08-30T23:28:41.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:152374 2.f e178/97 ondisk, result = 0)", 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:41.978103+0000", 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.0562000000000001e-05, 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013476300000000001, 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978103+0000", 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978103+0000", 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978104+0000", 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978112+0000", 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1899999999999995e-06 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978142+0000", 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0608000000000002e-05 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978149+0000", 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9659999999999999e-06 2024-08-30T23:28:41.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978192+0000", 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2419000000000002e-05 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:41.978199+0000", 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4739999999999999e-06 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.992 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:41.993 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:28:41.993 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-08-30T23:28:41.993 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:41.994 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:41.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:42.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1123703284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:42.038 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:41 smithi106 ceph-mon[16603]: pgmap v463: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.7 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 1734/131697 objects misplaced (1.317%); 6.3 MiB/s, 101 objects/s recovering 2024-08-30T23:28:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1123703284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:41 smithi136 ceph-mon[20207]: pgmap v463: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering, 15 active+clean; 2.7 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 731 op/s; 1734/131697 objects misplaced (1.317%); 6.3 MiB/s, 101 objects/s recovering 2024-08-30T23:28:42.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:42.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:42.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:28:42.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:42.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:42.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:42.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:42.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:42.540 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:42.543 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:42.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:42.644 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T23:28:42.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:42.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:42.793 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:42.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:28:42.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:42.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:42.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:43.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:43.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4003785666' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:28:43.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:42 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:28:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:42 smithi136 ceph-mon[20207]: 2.0 scrub starts 2024-08-30T23:28:43.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:43.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4003785666' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:28:43.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:42 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:28:43.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:42 smithi173 ceph-mon[20554]: 2.0 scrub starts 2024-08-30T23:28:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4003785666' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:28:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:42 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:28:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:42 smithi106 ceph-mon[16603]: 2.0 scrub starts 2024-08-30T23:28:43.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:43.150 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:43.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:43.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:43.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:43.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:43.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:43.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:43.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:43.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:43.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:43.674 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:43.674 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:28:43.674 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:28:43.674 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:28:43.674 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:43.693 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:43.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:43.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-08-30T23:28:43.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:43.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: osdmap e179: 12 total, 12 up, 11 in 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: 2.b scrub starts 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: 2.0 scrub ok 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:28:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:43 smithi173 ceph-mon[20554]: pgmap v465: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 2.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 728 op/s; 1072/127521 objects misplaced (0.841%); 10 MiB/s, 167 objects/s recovering 2024-08-30T23:28:43.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:43.961 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:43.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:44.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:28:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: osdmap e179: 12 total, 12 up, 11 in 2024-08-30T23:28:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: 2.b scrub starts 2024-08-30T23:28:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: 2.0 scrub ok 2024-08-30T23:28:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:28:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:28:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:43 smithi136 ceph-mon[20207]: pgmap v465: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 2.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 728 op/s; 1072/127521 objects misplaced (0.841%); 10 MiB/s, 167 objects/s recovering 2024-08-30T23:28:44.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:44.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:28:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: osdmap e179: 12 total, 12 up, 11 in 2024-08-30T23:28:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: 2.b scrub starts 2024-08-30T23:28:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: 2.0 scrub ok 2024-08-30T23:28:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:28:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:28:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:43 smithi106 ceph-mon[16603]: pgmap v465: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 2.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 728 op/s; 1072/127521 objects misplaced (0.841%); 10 MiB/s, 167 objects/s recovering 2024-08-30T23:28:44.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:44.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:44.222 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:44.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:44.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:44.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:28:44.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:28:44.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:44.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:44.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:44.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:44.674 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:28:44.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:44.718 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:44.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:44.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:44.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:44.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:44.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:44.992 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:44 smithi173 ceph-mon[20554]: 2.b scrub ok 2024-08-30T23:28:44.992 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/499898170' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:44.993 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:44 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:44.993 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:44 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:45.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:44 smithi136 ceph-mon[20207]: 2.b scrub ok 2024-08-30T23:28:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/499898170' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:44 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:44 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:45.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:44 smithi106 ceph-mon[16603]: 2.b scrub ok 2024-08-30T23:28:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/499898170' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:28:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:44 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:44 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:45.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:45.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:834 2.b 2:d73da4ae:::benchmark_data_smithi173_47195_object833:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:25:58.571456+0000", 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 166.68270788300001, 2024-08-30T23:28:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16821128799999999, 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 834 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571456+0000", 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571456+0000", 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571457+0000", 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.09e-07 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571475+0000", 2024-08-30T23:28:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7765999999999999e-05 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571486+0000", 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1024e-05 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571497+0000", 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0661e-05 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571546+0000", 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9353e-05 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571657+0000", 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110518 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.571738+0000", 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1771e-05 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.572112+0000", 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037404000000000002 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.572443+0000", 2024-08-30T23:28:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033022299999999999 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739598+0000", 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16715501799999999 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739631+0000", 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3188999999999997e-05 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739668+0000", 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6816000000000001e-05 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:834 2.b e104/97 ondisk, result = 0)", 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:25:58.572276+0000", 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 166.681887714, 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16739600499999999, 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:28:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.572276+0000", 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.572276+0000", 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.572277+0000", 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.572281+0000", 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9729999999999999e-06 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739550+0000", 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.167268798 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739553+0000", 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2729999999999998e-06 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739590+0000", 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6807999999999997e-05 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739594+0000", 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4920000000000004e-06 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:25:58.739672+0000", 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8016999999999995e-05 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:14225 2.5 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 156.38041966899999, 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24031856100000001, 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14225 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873763+0000", 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7887999999999999e-05 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873773+0000", 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8700000000000004e-06 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873780+0000", 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6850000000000004e-06 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873819+0000", 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9190999999999998e-05 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873906+0000", 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7612000000000001e-05 2024-08-30T23:28:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873999+0000", 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2088000000000003e-05 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.874429+0000", 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043006099999999997 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108141+0000", 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23371239399999999 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.113956+0000", 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058150340000000002 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114021+0000", 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500999999999998e-05 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114063+0000", 2024-08-30T23:28:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2512999999999998e-05 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:14225 2.5 e109/97 ondisk, result = 0)", 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 156.31940318599999, 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.173386651, 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:28:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934762+0000", 2024-08-30T23:28:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9700000000000005e-07 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934766+0000", 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5630000000000002e-06 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108075+0000", 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17330804799999999 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108080+0000", 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4049999999999999e-06 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108127+0000", 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7547999999999999e-05 2024-08-30T23:28:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108135+0000", 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8059999999999995e-06 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108148+0000", 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2384e-05 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:17728 2.5 2:a3d96d17:::benchmark_data_smithi173_47195_object17727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 153.75024087899999, 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20912092800000001, 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 17728 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503924+0000", 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.158e-06 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503947+0000", 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2192999999999999e-05 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503960+0000", 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3098999999999999e-05 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503975+0000", 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5167999999999999e-05 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504020+0000", 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5464999999999999e-05 2024-08-30T23:28:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504106+0000", 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5754000000000006e-05 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504188+0000", 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1489000000000005e-05 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504942+0000", 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075475800000000001 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.577391+0000", 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072448769999999996 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712945+0000", 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13555429899999999 2024-08-30T23:28:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713005+0000", 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9135000000000002e-05 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713044+0000", 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.964e-05 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:17728 2.5 e111/97 ondisk, result = 0)", 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 153.74898535599999, 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20787409000000001, 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505185+0000", 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712478+0000", 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20729294100000001 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712485+0000", 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712934+0000", 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044926100000000001 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712941+0000", 2024-08-30T23:28:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4999999999999996e-06 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713053+0000", 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011231900000000001 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:36469 2.5 2:a4a9a9b2:::benchmark_data_smithi173_47195_object36468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:25.001670+0000", 2024-08-30T23:28:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 140.252493797, 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.166538986, 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36469 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001670+0000", 2024-08-30T23:28:45.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001670+0000", 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001672+0000", 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1599999999999999e-06 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001695+0000", 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3853e-05 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001710+0000", 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4552000000000001e-05 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001717+0000", 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1609999999999997e-06 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001772+0000", 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5358000000000001e-05 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.001899+0000", 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012624500000000001 2024-08-30T23:28:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.002047+0000", 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000148317 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.003422+0000", 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001374861 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.084300+0000", 2024-08-30T23:28:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080878126999999994 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.168147+0000", 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083847018999999995 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.168181+0000", 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4187000000000003e-05 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.168209+0000", 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8146000000000001e-05 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:49996 2.5 2:aad518df:::benchmark_data_smithi173_47195_object49995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 129.996303692, 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20351808099999999, 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 49996 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:28:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257862+0000", 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.33e-06 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257920+0000", 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8489000000000002e-05 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257934+0000", 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3372e-05 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257941+0000", 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6459999999999993e-06 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257996+0000", 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5043999999999997e-05 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258082+0000", 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5890000000000006e-05 2024-08-30T23:28:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258176+0000", 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3366e-05 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259466+0000", 2024-08-30T23:28:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012900190000000001 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259518+0000", 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2787999999999997e-05 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461274+0000", 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201755238 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461333+0000", 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9394000000000001e-05 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461379+0000", 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5504999999999999e-05 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:56067 2.f 2:f6065497:::benchmark_data_smithi173_47195_object56066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 125.468367715, 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17632836900000001, 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 56067 2024-08-30T23:28:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785797+0000", 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785815+0000", 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8224e-05 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785825+0000", 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9869999999999994e-06 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785832+0000", 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.029e-06 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785866+0000", 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3729999999999997e-05 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785937+0000", 2024-08-30T23:28:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1027999999999998e-05 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786021+0000", 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3511999999999996e-05 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787483+0000", 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014626260000000001 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787561+0000", 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7565999999999996e-05 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962059+0000", 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17449777999999999 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962099+0000", 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9953000000000002e-05 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962125+0000", 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6278000000000001e-05 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56068 2.6 e124/97 2:62a84442:::benchmark_data_smithi173_47195_object56067:head v 124'3528, mlcod=124'3525)", 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 125.46777379300001, 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17324930699999999, 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786391+0000", 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3200000000000004e-07 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786446+0000", 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4661999999999999e-05 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958168+0000", 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171722186 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958175+0000", 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7009999999999999e-06 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958216+0000", 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1102000000000001e-05 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958230+0000", 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4688999999999999e-05 2024-08-30T23:28:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.959607+0000", 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013761559999999999 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.959640+0000", 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3079e-05 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.318 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.319 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:62555 2.5 2:a45a541d:::benchmark_data_smithi173_47195_object62554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-30T23:28:45.319 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:28:45.319 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 119.816278701, 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.175746766, 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62555 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.321 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437886+0000", 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5000000000000002e-07 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437908+0000", 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2069000000000002e-05 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437919+0000", 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.098e-05 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437927+0000", 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9519999999999994e-06 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438090+0000", 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016235300000000001 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438190+0000", 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010084199999999999 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438298+0000", 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107347 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.439985+0000", 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016870419999999999 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.440527+0000", 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054198600000000005 2024-08-30T23:28:45.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613517+0000", 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17299020600000001 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613579+0000", 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1791000000000005e-05 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613632+0000", 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3448000000000001e-05 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:96989 2.f 2:f906d66e:::benchmark_data_smithi173_47195_object96988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 77.111473089, 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84153836199999998, 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 96989 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142692+0000", 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142718+0000", 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5548999999999999e-05 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142729+0000", 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1126e-05 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142736+0000", 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1840000000000002e-06 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148635+0000", 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058993520000000001 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148767+0000", 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013146699999999999 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148916+0000", 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149083 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152326+0000", 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034105860000000002 2024-08-30T23:28:45.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152337+0000", 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1100999999999999e-05 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984109+0000", 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83177168400000001 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984180+0000", 2024-08-30T23:28:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0542000000000003e-05 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984229+0000", 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9808000000000002e-05 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:96989 2.f e146/97 ondisk, result = 0)", 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 77.102865730000005, 2024-08-30T23:28:45.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83293943299999995, 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151299+0000", 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5600000000000005e-07 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151308+0000", 2024-08-30T23:28:45.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984021+0000", 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83271241100000004 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984029+0000", 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9379999999999992e-06 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984091+0000", 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.211e-05 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984101+0000", 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0368e-05 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984238+0000", 2024-08-30T23:28:45.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013670199999999999 2024-08-30T23:28:45.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.338 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100665 2.6 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.910349242999999, 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21497229600000001, 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.339 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100665 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343816+0000", 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.494e-06 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343824+0000", 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2570000000000001e-06 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343841+0000", 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7107000000000001e-05 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343850+0000", 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4299999999999995e-06 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343921+0000", 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0264000000000004e-05 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344020+0000", 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9296999999999995e-05 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344100+0000", 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0465999999999998e-05 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.346504+0000", 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002403723 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347743+0000", 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001238549 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558717+0000", 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21097406299999999 2024-08-30T23:28:45.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558756+0000", 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9728000000000003e-05 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558787+0000", 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0917999999999998e-05 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100667 2.6 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 71.906382757000003, 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27402247099999999, 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.346 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100667 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347782+0000", 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347787+0000", 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3039999999999998e-06 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347798+0000", 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0834e-05 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347803+0000", 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.242e-06 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347840+0000", 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7630999999999999e-05 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347918+0000", 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7778e-05 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347989+0000", 2024-08-30T23:28:45.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0988999999999994e-05 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.349336+0000", 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013472670000000001 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.351224+0000", 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018879960000000001 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621737+0000", 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27051204000000001 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621775+0000", 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.875e-05 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621804+0000", 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8620999999999999e-05 2024-08-30T23:28:45.352 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:102301 2.f 2:fc0af6c0:::benchmark_data_smithi173_47195_object5293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:35.541666+0000", 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 69.712498437999997, 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169542736, 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.353 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102301 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541666+0000", 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541666+0000", 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541667+0000", 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.209e-06 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541672+0000", 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.617e-06 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541685+0000", 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3052e-05 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541691+0000", 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6200000000000001e-06 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.542436+0000", 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074515699999999998 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.542542+0000", 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106061 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.542612+0000", 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9160999999999997e-05 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.544890+0000", 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022781839999999999 2024-08-30T23:28:45.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.545054+0000", 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164196 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.711142+0000", 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16608751799999999 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.711184+0000", 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2862999999999999e-05 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.711208+0000", 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4097999999999999e-05 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:127040 2.6 2:6091191f:::benchmark_data_smithi173_47195_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:28:45.359 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.403017331999997, 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18230701299999999, 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127040 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851148+0000", 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851155+0000", 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8990000000000004e-06 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851175+0000", 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9711000000000002e-05 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851184+0000", 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6780000000000006e-06 2024-08-30T23:28:45.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851247+0000", 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2780000000000005e-05 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851360+0000", 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112946 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851485+0000", 2024-08-30T23:28:45.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012559 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.854952+0000", 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034665049999999999 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.862420+0000", 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0074685050000000003 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033372+0000", 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17095152199999999 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033407+0000", 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5567e-05 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033454+0000", 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681000000000002e-05 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:144898 2.6 2:692f2db4:::benchmark_data_smithi173_47195_object47890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 13.574996574, 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18135380100000001, 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.366 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144898 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679169+0000", 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679174+0000", 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.198e-06 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679187+0000", 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.31e-05 2024-08-30T23:28:45.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679194+0000", 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679236+0000", 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1828e-05 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679316+0000", 2024-08-30T23:28:45.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0026999999999998e-05 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679383+0000", 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6925e-05 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.682228+0000", 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028449149999999999 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.829406+0000", 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14717798100000001 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860431+0000", 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031025377 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860482+0000", 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0779000000000003e-05 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860521+0000", 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9307999999999997e-05 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:145440 2.f 2:fe697e68:::benchmark_data_smithi173_47195_object48432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:32.437723+0000", 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.816441229, 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16980590700000001, 2024-08-30T23:28:45.372 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 145440 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437723+0000", 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437723+0000", 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437724+0000", 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1850000000000001e-06 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437730+0000", 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8939999999999998e-06 2024-08-30T23:28:45.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437745+0000", 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4686e-05 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437752+0000", 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8040000000000004e-06 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437798+0000", 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6683e-05 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437878+0000", 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0160000000000005e-05 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437947+0000", 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8508000000000002e-05 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.442029+0000", 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040821110000000002 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.442043+0000", 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4347e-05 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.607466+0000", 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16542242900000001 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.607500+0000", 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4694999999999999e-05 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.607529+0000", 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8405e-05 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.378 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:147254 2.f 2:f940aba9:::benchmark_data_smithi173_47195_object50246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.134659141, 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17365630000000001, 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 147254 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119506+0000", 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3069999999999999e-06 2024-08-30T23:28:45.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119512+0000", 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5450000000000003e-06 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119525+0000", 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2979999999999999e-05 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119532+0000", 2024-08-30T23:28:45.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.99e-06 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119579+0000", 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7336999999999998e-05 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119668+0000", 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8710999999999998e-05 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119740+0000", 2024-08-30T23:28:45.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2224999999999994e-05 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.123093+0000", 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033526760000000002 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.123100+0000", 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5610000000000001e-06 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:28:45.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293060+0000", 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169959361 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293114+0000", 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3825000000000001e-05 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293161+0000", 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7781999999999998e-05 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.384 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.385 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:45.385 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:45.385 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.385 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:45.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:45.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:45.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:45.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:45.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:45.686 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.8 to 1 (10000) 2024-08-30T23:28:45.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:45.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:45.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:45.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:45.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:46.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:46.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:45 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:28:46.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/499898170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:46.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:45 smithi106 ceph-mon[16603]: osdmap e180: 12 total, 12 up, 11 in 2024-08-30T23:28:46.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1391533110' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:28:46.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:45 smithi106 ceph-mon[16603]: pgmap v467: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 2.5 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 1072/123879 objects misplaced (0.865%); 6.9 MiB/s, 110 objects/s recovering 2024-08-30T23:28:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:46.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:45 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:28:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/499898170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:45 smithi136 ceph-mon[20207]: osdmap e180: 12 total, 12 up, 11 in 2024-08-30T23:28:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1391533110' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:28:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:45 smithi136 ceph-mon[20207]: pgmap v467: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 2.5 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 1072/123879 objects misplaced (0.865%); 6.9 MiB/s, 110 objects/s recovering 2024-08-30T23:28:46.048 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:45 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:28:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/499898170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:28:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:45 smithi173 ceph-mon[20554]: osdmap e180: 12 total, 12 up, 11 in 2024-08-30T23:28:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1391533110' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:28:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:45 smithi173 ceph-mon[20554]: pgmap v467: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovering, 14 active+clean; 2.5 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 670 op/s; 1072/123879 objects misplaced (0.865%); 6.9 MiB/s, 110 objects/s recovering 2024-08-30T23:28:46.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:46.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:46.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:46.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:46.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:28:46.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:46.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.401 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:46.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:46.527 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:46.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-08-30T23:28:46.576 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:46.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.653 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:46.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:46.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:46.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1391533110' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:28:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:46 smithi106 ceph-mon[16603]: osdmap e181: 12 total, 12 up, 11 in 2024-08-30T23:28:46.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:46.903 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:46.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:47.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:47.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1391533110' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:28:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:46 smithi136 ceph-mon[20207]: osdmap e181: 12 total, 12 up, 11 in 2024-08-30T23:28:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1391533110' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:28:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:46 smithi173 ceph-mon[20554]: osdmap e181: 12 total, 12 up, 11 in 2024-08-30T23:28:47.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:47.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:47.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:47.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:47.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:47.287 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:47.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:47.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:47.444 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:47.445 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:28:47.445 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:28:47.445 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:47.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.531 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:47.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:47.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:47.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:47.694 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:47 smithi106 ceph-mon[16603]: osdmap e182: 12 total, 12 up, 11 in 2024-08-30T23:28:47.694 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1183638536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:47.694 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:47 smithi106 ceph-mon[16603]: pgmap v470: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 2.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 789/119691 objects misplaced (0.659%); 7.2 MiB/s, 115 objects/s recovering 2024-08-30T23:28:47.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:47.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:47.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:28:47.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:47.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:47.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:47.955 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:47 smithi106 ceph-mon[16603]: osdmap e183: 12 total, 12 up, 11 in 2024-08-30T23:28:47.975 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:47.976 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:48.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:48.019 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:48.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:48.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:47 smithi136 ceph-mon[20207]: osdmap e182: 12 total, 12 up, 11 in 2024-08-30T23:28:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1183638536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:47 smithi136 ceph-mon[20207]: pgmap v470: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 2.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 789/119691 objects misplaced (0.659%); 7.2 MiB/s, 115 objects/s recovering 2024-08-30T23:28:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:47 smithi136 ceph-mon[20207]: osdmap e183: 12 total, 12 up, 11 in 2024-08-30T23:28:48.064 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:48.064 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:47 smithi173 ceph-mon[20554]: osdmap e182: 12 total, 12 up, 11 in 2024-08-30T23:28:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1183638536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:47 smithi173 ceph-mon[20554]: pgmap v470: 17 pgs: 1 peering, 1 active+recovering, 15 active+clean; 2.4 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 789/119691 objects misplaced (0.659%); 7.2 MiB/s, 115 objects/s recovering 2024-08-30T23:28:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:47 smithi173 ceph-mon[20554]: osdmap e183: 12 total, 12 up, 11 in 2024-08-30T23:28:48.112 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:48.112 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:48.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:48.151 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:48.152 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:48.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:48.202 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:48.202 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:48.241 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:48.242 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:48.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:48.280 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:48.280 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:48.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:48.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:28:48.308 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:48.315 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:48.316 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:48.353 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:48.353 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:48.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:48.390 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:48.390 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:48.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:48.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:48.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:48.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:48.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:48.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:48.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:48.655 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:48.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:48.706 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.9 to 1 (10000) 2024-08-30T23:28:48.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.806 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:48.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:48 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-30T23:28:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:48 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:28:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4170211643' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:28:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:48 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:28:48.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:48.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:48.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:48.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:48 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-30T23:28:48.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:48 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:28:48.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4170211643' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:28:48.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:48 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:28:49.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:48 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-08-30T23:28:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:48 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:28:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4170211643' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:28:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:48 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:28:49.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:49.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.191 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:49.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:49.215 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:49.215 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:49.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:49.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:49.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:49.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:49.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:49.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:49.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:49.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:49.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:49.854 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:49.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:49.982 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:49.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:50.037 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:49 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:28:50.037 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:49 smithi173 ceph-mon[20554]: osdmap e184: 12 total, 12 up, 11 in 2024-08-30T23:28:50.037 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:28:50.037 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:28:50.037 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:28:50.037 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:28:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:49 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:28:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:49 smithi136 ceph-mon[20207]: osdmap e184: 12 total, 12 up, 11 in 2024-08-30T23:28:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:28:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:28:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:28:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:28:50.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:28:50.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:50.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:49 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:28:50.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:49 smithi106 ceph-mon[16603]: osdmap e184: 12 total, 12 up, 11 in 2024-08-30T23:28:50.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:28:50.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:28:50.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:28:50.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:28:50.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:50.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:50.241 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:50.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:50.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:50.391 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:50.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:50.517 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:50.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:50.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:50.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:50.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:50 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 16/115869 objects degraded (0.014%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:28:50.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:50.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:50.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:50.973 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:50.973 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:51.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:50 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 16/115869 objects degraded (0.014%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:28:51.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:50 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 16/115869 objects degraded (0.014%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:28:51.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:51.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:51.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:51.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:28:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:51.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:51.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:51.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:51.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:51.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:51.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:51.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:51.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:51.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:28:51.821 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:51.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:51.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:51.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:51 smithi136 ceph-mon[20207]: osdmap e185: 12 total, 12 up, 11 in 2024-08-30T23:28:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:51 smithi136 ceph-mon[20207]: pgmap v475: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering, 13 active+clean; 2.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 16/114000 objects degraded (0.014%); 5113/114000 objects misplaced (4.485%); 11 MiB/s, 172 objects/s recovering 2024-08-30T23:28:52.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:52.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:52.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:51 smithi173 ceph-mon[20554]: osdmap e185: 12 total, 12 up, 11 in 2024-08-30T23:28:52.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:51 smithi173 ceph-mon[20554]: pgmap v475: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering, 13 active+clean; 2.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 16/114000 objects degraded (0.014%); 5113/114000 objects misplaced (4.485%); 11 MiB/s, 172 objects/s recovering 2024-08-30T23:28:52.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:52.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:51 smithi106 ceph-mon[16603]: osdmap e185: 12 total, 12 up, 11 in 2024-08-30T23:28:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:51 smithi106 ceph-mon[16603]: pgmap v475: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 1 active+recovering, 13 active+clean; 2.3 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 16/114000 objects degraded (0.014%); 5113/114000 objects misplaced (4.485%); 11 MiB/s, 172 objects/s recovering 2024-08-30T23:28:52.140 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:52.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:52.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:52.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:52.332 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:52.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:52.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:52.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:52.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:52.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:52.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:52.757 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:52.757 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:52.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:52.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:52.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:52.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:52.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:52.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:52 smithi173 ceph-mon[20554]: 2.1 scrub starts 2024-08-30T23:28:53.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:53.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:52 smithi106 ceph-mon[16603]: 2.1 scrub starts 2024-08-30T23:28:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:53.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:53.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:53.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:52 smithi136 ceph-mon[20207]: 2.1 scrub starts 2024-08-30T23:28:53.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:53.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:53.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:53.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:53.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:28:53.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:53.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:53.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:53.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:53.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:53.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:53.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:53.941 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:53.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:54.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:54.095 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:53 smithi173 ceph-mon[20554]: 2.1 scrub ok 2024-08-30T23:28:54.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:53 smithi173 ceph-mon[20554]: pgmap v476: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 2.2 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 71/110196 objects degraded (0.064%); 4591/110196 objects misplaced (4.166%); 10 MiB/s, 165 objects/s recovering 2024-08-30T23:28:54.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:54.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:53 smithi106 ceph-mon[16603]: 2.1 scrub ok 2024-08-30T23:28:54.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:53 smithi106 ceph-mon[16603]: pgmap v476: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 2.2 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 71/110196 objects degraded (0.064%); 4591/110196 objects misplaced (4.166%); 10 MiB/s, 165 objects/s recovering 2024-08-30T23:28:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:53 smithi136 ceph-mon[20207]: 2.1 scrub ok 2024-08-30T23:28:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:53 smithi136 ceph-mon[20207]: pgmap v476: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 2.2 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 71/110196 objects degraded (0.064%); 4591/110196 objects misplaced (4.166%); 10 MiB/s, 165 objects/s recovering 2024-08-30T23:28:54.291 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:54.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:54.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:54.511 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:28:54.511 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:28:54.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:54.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:54.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:54.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:54.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-08-30T23:28:54.752 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:54.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:54.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:54.840 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:28:54.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:54.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:54.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:55.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:55.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: 2.7 scrub starts 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2903284527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:28:55.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:55.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: 2.7 scrub starts 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2903284527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:28:55.222 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:28:55.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: 2.7 scrub starts 2024-08-30T23:28:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2903284527' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:28:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:28:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:28:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:28:55.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:55.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:28:55.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:28:55.514 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:55.558 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:28:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:28:55.578 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:55.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:55.591 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:28:55.592 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:28:55.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:55.626 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:28:55.627 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:28:55.657 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:28:55.657 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:28:55.691 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:28:55.691 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:28:55.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:55.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:28:55.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:28:55.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:55.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:28:55.761 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:28:55.787 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:28:55.787 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:28:55.819 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:28:55.820 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:28:55.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:55.855 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:28:55.855 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:28:55.875 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:55.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:55.889 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:28:55.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:28:55.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:55 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:55.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2903284527' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:28:55.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:55 smithi106 ceph-mon[16603]: osdmap e186: 12 total, 12 up, 11 in 2024-08-30T23:28:55.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:55 smithi106 ceph-mon[16603]: pgmap v478: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 71/106473 objects degraded (0.067%); 4348/106473 objects misplaced (4.084%); 6.6 MiB/s, 106 objects/s recovering 2024-08-30T23:28:55.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3259609229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:55.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:55 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 71/110196 objects degraded (0.064%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:28:55.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:28:55.945 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:28:55.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:55.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:56.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:55 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:56.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2903284527' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:28:56.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:55 smithi173 ceph-mon[20554]: osdmap e186: 12 total, 12 up, 11 in 2024-08-30T23:28:56.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:55 smithi173 ceph-mon[20554]: pgmap v478: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 71/106473 objects degraded (0.067%); 4348/106473 objects misplaced (4.084%); 6.6 MiB/s, 106 objects/s recovering 2024-08-30T23:28:56.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3259609229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:56.018 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:55 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 71/110196 objects degraded (0.064%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:28:56.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:56.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:56.151 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:56.151 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:28:56.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:16911 2.4 e111/97 2:26b3b856:::benchmark_data_smithi173_47195_object16910:head v 111'1012, mlcod=111'1009)", 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 165.47729082399999, 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225497685, 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672856+0000", 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0200000000000003e-07 2024-08-30T23:28:56.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672918+0000", 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1801e-05 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897644+0000", 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22472613699999999 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897651+0000", 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.99e-06 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897700+0000", 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9373000000000003e-05 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897721+0000", 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1314e-05 2024-08-30T23:28:56.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.898318+0000", 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059680200000000001 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.898353+0000", 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4866000000000001e-05 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:19019 2.7 e111/97 2:e1c9e35c:::benchmark_data_smithi173_47195_object19018:head v 111'1208, mlcod=111'1204)", 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 163.678494476, 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.113963728, 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7700000000000004e-07 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471730+0000", 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7249000000000005e-05 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584625+0000", 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11289540200000001 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584630+0000", 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9470000000000002e-06 2024-08-30T23:28:56.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584749+0000", 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011939599999999999 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584777+0000", 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7206e-05 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.585586+0000", 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00080899100000000001 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.585616+0000", 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9960000000000001e-05 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.161 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:19160 2.0 e112/97 2:059fd018:::benchmark_data_smithi173_47195_object19159:head v 112'1222, mlcod=112'1219)", 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:12.681651+0000", 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 163.468495209, 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.069984315000000005, 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.681651+0000", 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.681651+0000", 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.681652+0000", 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4099999999999999e-07 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.681701+0000", 2024-08-30T23:28:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.888e-05 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.750906+0000", 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.069205263000000003 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.750915+0000", 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5000000000000005e-06 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.750976+0000", 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1051999999999998e-05 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.750992+0000", 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5801999999999999e-05 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.751598+0000", 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060627800000000003 2024-08-30T23:28:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.751635+0000", 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6999000000000003e-05 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:20668 2.0 e112/97 2:033ae7b0:::benchmark_data_smithi173_47195_object20667:head v 112'1312, mlcod=112'1308)", 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:13.671018+0000", 2024-08-30T23:28:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 162.47912784900001, 2024-08-30T23:28:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.088517390000000001, 2024-08-30T23:28:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671018+0000", 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671018+0000", 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671019+0000", 2024-08-30T23:28:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1900000000000002e-07 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671073+0000", 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4072e-05 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758556+0000", 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.087482480000000001 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758563+0000", 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4139999999999997e-06 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758635+0000", 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1639999999999998e-05 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758659+0000", 2024-08-30T23:28:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3864999999999999e-05 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.759476+0000", 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081773200000000005 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.759536+0000", 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9568000000000003e-05 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:22849 2.7 e112/97 2:ebcd612e:::benchmark_data_smithi173_47195_object22848:head v 112'1442, mlcod=112'1439)", 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:15.244998+0000", 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 160.90514794200001, 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117383665, 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.244998+0000", 2024-08-30T23:28:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.244998+0000", 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.244999+0000", 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.245059+0000", 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9746e-05 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361366+0000", 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.116306929 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361370+0000", 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6070000000000001e-06 2024-08-30T23:28:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361409+0000", 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8924000000000002e-05 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361423+0000", 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3698e-05 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.362348+0000", 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000925521 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.362382+0000", 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.362e-05 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:31164 2.4 e116/97 2:2f94b145:::benchmark_data_smithi173_47195_object31163:head v 116'1913, mlcod=116'1909)", 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:21.144978+0000", 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 155.00516866500001, 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080170878000000001, 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.144978+0000", 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.144978+0000", 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.144979+0000", 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2410000000000001e-06 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.145008+0000", 2024-08-30T23:28:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9189000000000001e-05 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224228+0000", 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.079219519000000002 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224233+0000", 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8250000000000001e-06 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224272+0000", 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8659999999999999e-05 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224286+0000", 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4358999999999999e-05 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.225109+0000", 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082302500000000002 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.225149+0000", 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9060000000000002e-05 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:37028 2.4 e117/97 2:29a37efb:::benchmark_data_smithi173_47195_object37027:head v 117'2296, mlcod=117'2293)", 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:25.640456+0000", 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 150.50968993199999, 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.075770119999999996, 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.640456+0000", 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.640456+0000", 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.640457+0000", 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5499999999999997e-07 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.640507+0000", 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9523e-05 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.715259+0000", 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.074751953999999995 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.715264+0000", 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0409999999999997e-06 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.715311+0000", 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7784000000000002e-05 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.715324+0000", 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2892e-05 2024-08-30T23:28:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.716200+0000", 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087576400000000001 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:25.716226+0000", 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6407e-05 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:39829 2.7 e119/97 2:ea0d6c5f:::benchmark_data_smithi173_47195_object39828:head v 119'2487, mlcod=119'2483)", 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:27.415758+0000", 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 148.734387811, 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083043754999999997, 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415758+0000", 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415758+0000", 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415759+0000", 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1200000000000002e-07 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415875+0000", 2024-08-30T23:28:56.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011565000000000001 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497633+0000", 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.081757904000000006 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497642+0000", 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8179999999999993e-06 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497694+0000", 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2194000000000002e-05 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497711+0000", 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7136e-05 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.498758+0000", 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010474569999999999 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.498802+0000", 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3884000000000002e-05 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:39861 2.4 e119/97 2:2f9ea911:::benchmark_data_smithi173_47195_object39860:head v 119'2492, mlcod=119'2489)", 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 148.71356969600001, 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.090130358999999993, 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.190 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7100000000000001e-07 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436632+0000", 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5006000000000002e-05 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525705+0000", 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.089072993000000003 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525710+0000", 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6310000000000002e-06 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525758+0000", 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8069999999999999e-05 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525773+0000", 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4843e-05 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.526680+0000", 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090692099999999999 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.526707+0000", 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7223999999999999e-05 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.194 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:43503 2.0 e119/97 2:0f4dae72:::benchmark_data_smithi173_47195_object43502:head v 119'2693, mlcod=119'2690)", 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:30.305617+0000", 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 145.84452912200001, 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104864759, 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305617+0000", 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305617+0000", 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305618+0000", 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.051e-06 2024-08-30T23:28:56.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305661+0000", 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3087999999999998e-05 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409257+0000", 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103595863 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409264+0000", 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4119999999999998e-06 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409321+0000", 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7048e-05 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409344+0000", 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3490999999999999e-05 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.410451+0000", 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011067150000000001 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.410482+0000", 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1090999999999998e-05 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.199 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:50002 2.7 e122/97 2:e84aa78e:::benchmark_data_smithi173_47195_object50001:head v 122'3082, mlcod=122'3079)", 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:35.268911+0000", 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 140.88123501300001, 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19888059999999999, 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268911+0000", 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268911+0000", 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268913+0000", 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.305e-06 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268950+0000", 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7697000000000002e-05 2024-08-30T23:28:56.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466526+0000", 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757559199999999 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466531+0000", 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3940000000000004e-06 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466584+0000", 2024-08-30T23:28:56.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349000000000003e-05 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466605+0000", 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1880000000000001e-05 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.467760+0000", 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011541419999999999 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.467792+0000", 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2240999999999998e-05 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:50964 2.4 e123/97 2:22c068cb:::benchmark_data_smithi173_47195_object50963:head v 123'3182, mlcod=123'3178)", 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:36.098133+0000", 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 140.05201303999999, 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072803411999999998, 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.204 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.098133+0000", 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.098133+0000", 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.098134+0000", 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.098167+0000", 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2295999999999997e-05 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.169088+0000", 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.070921234999999999 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.169094+0000", 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1120000000000001e-06 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.169150+0000", 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6180000000000001e-05 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.169174+0000", 2024-08-30T23:28:56.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4185999999999998e-05 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.170847+0000", 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001672783 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:36.170937+0000", 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9649999999999997e-05 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:54264 2.4 e123/97 2:260bcf68:::benchmark_data_smithi173_47195_object54263:head v 123'3380, mlcod=123'3376)", 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:38.314919+0000", 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.835227347, 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.098021554999999996, 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314919+0000", 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314919+0000", 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314920+0000", 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314981+0000", 2024-08-30T23:28:56.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1614999999999999e-05 2024-08-30T23:28:56.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411576+0000", 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096594261000000001 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411581+0000", 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5799999999999999e-06 2024-08-30T23:28:56.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411629+0000", 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.74e-05 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411643+0000", 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.473e-05 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.412895+0000", 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001251963 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.412941+0000", 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5126000000000001e-05 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56070 2.0 e124/97 2:06759afd:::benchmark_data_smithi173_47195_object56069:head v 124'3466, mlcod=124'3463)", 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.791057+0000", 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 136.35908895200001, 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19941077600000001, 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791057+0000", 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791057+0000", 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791059+0000", 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.167e-06 2024-08-30T23:28:56.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791090+0000", 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1420000000000001e-05 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988501+0000", 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.197411267 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988508+0000", 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0829999999999998e-06 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988600+0000", 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1765e-05 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988620+0000", 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9877999999999998e-05 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.990423+0000", 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018031080000000001 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.990468+0000", 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5087999999999999e-05 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.219 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56072 2.7 e124/97 2:e7493b3b:::benchmark_data_smithi173_47195_object56071:head v 124'3469, mlcod=124'3466)", 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.795821+0000", 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 136.35432503499999, 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17323951900000001, 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795821+0000", 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795821+0000", 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795822+0000", 2024-08-30T23:28:56.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7599999999999996e-07 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795863+0000", 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0525000000000002e-05 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967597+0000", 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17173391800000001 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967603+0000", 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.674e-06 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967659+0000", 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6178000000000003e-05 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967682+0000", 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2158e-05 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.969029+0000", 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013469790000000001 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.969061+0000", 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2211e-05 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62661 2.7 e127/97 2:e0aff4c8:::benchmark_data_smithi173_47195_object62660:head v 127'3889, mlcod=127'3886)", 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.608751+0000", 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.541395099, 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.085590695999999994, 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608751+0000", 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608751+0000", 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608752+0000", 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7000000000000003e-07 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608824+0000", 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692767+0000", 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083942873000000001 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692772+0000", 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4280000000000004e-06 2024-08-30T23:28:56.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692821+0000", 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8829000000000003e-05 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692835+0000", 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4347e-05 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.694308+0000", 2024-08-30T23:28:56.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00147241 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.694342+0000", 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4131999999999998e-05 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62809 2.7 e127/97 2:e4464b6d:::benchmark_data_smithi173_47195_object62808:head v 127'3894, mlcod=127'3891)", 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.781150+0000", 2024-08-30T23:28:56.229 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.368996735, 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080382227000000001, 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781150+0000", 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781150+0000", 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781151+0000", 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781213+0000", 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2241000000000003e-05 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.859966+0000", 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078752848 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.859975+0000", 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1060000000000006e-06 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.860026+0000", 2024-08-30T23:28:56.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0804999999999999e-05 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.860042+0000", 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6072000000000001e-05 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.861502+0000", 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014604100000000001 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.861532+0000", 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9564999999999999e-05 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62949 2.7 e127/97 2:e71ee7c9:::benchmark_data_smithi173_47195_object62948:head v 127'3908, mlcod=127'3905)", 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:46.017544+0000", 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.13260270200001, 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.070718389000000006, 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.234 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.017544+0000", 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.017544+0000", 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.017544+0000", 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.017580+0000", 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6022000000000002e-05 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086563+0000", 2024-08-30T23:28:56.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.068982851999999997 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086569+0000", 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4879999999999998e-06 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086644+0000", 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5458999999999997e-05 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086658+0000", 2024-08-30T23:28:56.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4392e-05 2024-08-30T23:28:56.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.088235+0000", 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001576098 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.088262+0000", 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7474000000000002e-05 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62959 2.7 e127/97 2:e1c982e1:::benchmark_data_smithi173_47195_object62958:head v 127'3909, mlcod=127'3906)", 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:46.086449+0000", 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 130.06369713800001, 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080663077, 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086449+0000", 2024-08-30T23:28:56.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086449+0000", 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086450+0000", 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086546+0000", 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6031000000000003e-05 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165445+0000", 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078899104999999997 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165452+0000", 2024-08-30T23:28:56.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4669999999999998e-06 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165514+0000", 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1694999999999998e-05 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165535+0000", 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1128000000000002e-05 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.167079+0000", 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015441529999999999 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.167112+0000", 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2777999999999997e-05 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:77053 2.0 e134/97 2:06a5d6c0:::benchmark_data_smithi173_47195_object77052:head v 134'4820, mlcod=134'4817)", 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:00.133250+0000", 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 116.016896528, 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.086050852999999997, 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133250+0000", 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133250+0000", 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133251+0000", 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2799999999999995e-07 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133280+0000", 2024-08-30T23:28:56.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8938999999999999e-05 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216909+0000", 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083629259999999997 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216916+0000", 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216976+0000", 2024-08-30T23:28:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9905999999999999e-05 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216998+0000", 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2308000000000001e-05 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.219257+0000", 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022584319999999999 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:28:56.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.219301+0000", 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3939000000000001e-05 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:28:56.249 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:56.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:56.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:56.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:28:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:55 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:28:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2903284527' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:28:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:55 smithi136 ceph-mon[20207]: osdmap e186: 12 total, 12 up, 11 in 2024-08-30T23:28:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:55 smithi136 ceph-mon[20207]: pgmap v478: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 71/106473 objects degraded (0.067%); 4348/106473 objects misplaced (4.084%); 6.6 MiB/s, 106 objects/s recovering 2024-08-30T23:28:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3259609229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:28:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:55 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 71/110196 objects degraded (0.064%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:28:56.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:56.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:56.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:56.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:56.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.673 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:56.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:56.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:56.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:56.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:56.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:56.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:57.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:57.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:57.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:57.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:57.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:28:57.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:28:57.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:28:57.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:28:57.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:28:57.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:57.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:57.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:28:57.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:28:57.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:57.860 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:57.863 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:57.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:57.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:28:57.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:57.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:58.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:28:58.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:58.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:28:58.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:58.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:58.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:58.386 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:58.394 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:58.394 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:28:58.394 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:28:58.394 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:58.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:58.443 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.11 to 1 (10000) 2024-08-30T23:28:58.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:28:58.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:28:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:58 smithi173 ceph-mon[20554]: pgmap v479: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.1 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 71/101760 objects degraded (0.070%); 4348/101760 objects misplaced (4.273%); 5.5 MiB/s, 88 objects/s recovering 2024-08-30T23:28:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:58 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:28:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:58 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:28:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/781859779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:28:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:28:58.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:28:58.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:58.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:28:58.695 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:58 smithi106 ceph-mon[16603]: pgmap v479: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.1 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 71/101760 objects degraded (0.070%); 4348/101760 objects misplaced (4.273%); 5.5 MiB/s, 88 objects/s recovering 2024-08-30T23:28:58.695 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:58 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:28:58.695 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:58 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:28:58.695 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/781859779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:28:58.695 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:28:58.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:28:58.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:58.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:58 smithi136 ceph-mon[20207]: pgmap v479: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.1 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 587 op/s; 71/101760 objects degraded (0.070%); 4348/101760 objects misplaced (4.273%); 5.5 MiB/s, 88 objects/s recovering 2024-08-30T23:28:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:58 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:28:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:58 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:28:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/781859779' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:28:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:28:58.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:58.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:58.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:59.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:59.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:59.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:28:59.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:59.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:59.297 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T23:28:59.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:28:59.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:28:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:59.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:28:59.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:59.457 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:28:59.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:59.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:59 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:28:59.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:28:59 smithi106 ceph-mon[16603]: osdmap e187: 12 total, 12 up, 11 in 2024-08-30T23:28:59.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:28:59.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:28:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:28:59.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:28:59.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:28:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:28:59.745 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:59 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:28:59.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:28:59 smithi173 ceph-mon[20554]: osdmap e187: 12 total, 12 up, 11 in 2024-08-30T23:28:59.760 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:59 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:28:59.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:28:59 smithi136 ceph-mon[20207]: osdmap e187: 12 total, 12 up, 11 in 2024-08-30T23:28:59.797 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:28:59.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:28:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:28:59.873 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:28:59.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:28:59.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:28:59.995 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:28:59.995 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:28:59.995 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:28:59.995 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:28:59.995 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:28:59.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:00.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:00.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:00.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:00.259 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:00.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:00.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:00.463 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:29:00.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:00.515 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:00.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:00.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:00.712 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:00 smithi106 ceph-mon[16603]: pgmap v481: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 733 op/s; 145/96402 objects degraded (0.150%); 3903/96402 objects misplaced (4.049%); 8.4 MiB/s, 135 objects/s recovering 2024-08-30T23:29:00.712 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/796308987' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:00.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:00.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:00 smithi136 ceph-mon[20207]: pgmap v481: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 733 op/s; 145/96402 objects degraded (0.150%); 3903/96402 objects misplaced (4.049%); 8.4 MiB/s, 135 objects/s recovering 2024-08-30T23:29:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/796308987' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:00 smithi173 ceph-mon[20554]: pgmap v481: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.0 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 733 op/s; 145/96402 objects degraded (0.150%); 3903/96402 objects misplaced (4.049%); 8.4 MiB/s, 135 objects/s recovering 2024-08-30T23:29:00.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/796308987' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:00.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:00.908 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:00.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:00.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:00.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:00.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:00.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:00.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:00.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:00.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:00.960 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:00.960 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:00.961 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:00.961 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:00.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:29:01.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:01.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:01.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:01.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:01.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:01.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:01.308 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:29:01.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:01.446 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:01.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:01.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:01 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/796308987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:01 smithi106 ceph-mon[16603]: osdmap e188: 12 total, 12 up, 11 in 2024-08-30T23:29:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1807345640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 145/96402 objects degraded (0.150%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:01.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:01.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:01.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:01 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:01.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/796308987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:01.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:01 smithi173 ceph-mon[20554]: osdmap e188: 12 total, 12 up, 11 in 2024-08-30T23:29:01.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1807345640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:01.748 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 145/96402 objects degraded (0.150%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:01 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/796308987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:01 smithi136 ceph-mon[20207]: osdmap e188: 12 total, 12 up, 11 in 2024-08-30T23:29:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1807345640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 145/96402 objects degraded (0.150%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:01.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:01.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:01.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:01.877 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:29:01.877 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:29:01.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:01.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:01.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:02.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:02.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:02.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:02.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:02.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:02.345 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-08-30T23:29:00.466078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2091713080,"num_objects":31911,"num_object_clones":0,"num_object_copies":95733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":145,"num_objects_misplaced":3903,"num_objects_unfound":0,"num_objects_dirty":31911,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":162226,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25392,"num_bytes_recovered":1664956000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160595,"ondisk_log_size":160595,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":7077672,"kb_used_data":6154148,"kb_used_omap":17,"kb_used_meta":923118,"kb_avail":1024119000,"statfs":{"total":1055945392128,"available":1048697856000,"internally_reserved":0,"allocated":6301847552,"data_stored":6297600362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":945273721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,3,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-315949056,"num_objects":-4821,"num_object_clones":0,"num_object_copies":-14463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":-688,"num_objects_unfound":0,"num_objects_dirty":-4821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":614,"num_bytes_recovered":39976960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.084730"},"pg_stats":[{"pgid":"2.f","version":"186'10094","reported_seq":10389,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:55.642408+0000","last_change":"2024-08-30T23:28:16.218035+0000","last_active":"2024-08-30T23:28:55.642408+0000","last_peered":"2024-08-30T23:28:55.642408+0000","last_clean":"2024-08-30T23:28:55.642408+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:28:55.642408+0000","last_undegraded":"2024-08-30T23:28:55.642408+0000","last_fullsized":"2024-08-30T23:28:55.642408+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'8409","last_scrub_stamp":"2024-08-30T23:28:16.217972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.217972+0000","objects_scrubbed":3813,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:45.741305+0000","scrub_duration":782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138280960,"num_objects":2110,"num_object_clones":0,"num_object_copies":6330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10094,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"187'10033","reported_seq":10225,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.455214+0000","last_change":"2024-08-30T23:25:50.474312+0000","last_active":"2024-08-30T23:28:58.455214+0000","last_peered":"2024-08-30T23:28:58.455214+0000","last_clean":"2024-08-30T23:28:58.455214+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:28:58.455214+0000","last_undegraded":"2024-08-30T23:28:58.455214+0000","last_fullsized":"2024-08-30T23:28:58.455214+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:02.709892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124846080,"num_objects":1905,"num_object_clones":0,"num_object_copies":5715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"187'10385","reported_seq":10619,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:29:00.052138+0000","last_change":"2024-08-30T23:28:27.803716+0000","last_active":"2024-08-30T23:29:00.052138+0000","last_peered":"2024-08-30T23:29:00.052138+0000","last_clean":"2024-08-30T23:29:00.052138+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:29:00.052138+0000","last_undegraded":"2024-08-30T23:29:00.052138+0000","last_fullsized":"2024-08-30T23:29:00.052138+0000","mapping_epoch":97,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'9007","last_scrub_stamp":"2024-08-30T23:28:27.803577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.803577+0000","objects_scrubbed":3238,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:55.835178+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119930904,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10385,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"186'9960","reported_seq":10191,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:56.068573+0000","last_change":"2024-08-30T23:28:27.113440+0000","last_active":"2024-08-30T23:28:56.068573+0000","last_peered":"2024-08-30T23:28:56.068573+0000","last_clean":"2024-08-30T23:28:56.068573+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:28:56.068573+0000","last_undegraded":"2024-08-30T23:28:56.068573+0000","last_fullsized":"2024-08-30T23:28:56.068573+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'8728","last_scrub_stamp":"2024-08-30T23:28:27.113155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.113155+0000","objects_scrubbed":3323,"log_size":9960,"log_dups_size":0,"ondisk_log_size":9960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:28.567110+0000","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135004160,"num_objects":2060,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9960,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":331,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:54.853813+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:28:54.853813+0000","last_peered":"2024-08-30T23:28:54.853813+0000","last_clean":"2024-08-30T23:28:54.853813+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:28:54.853813+0000","last_undegraded":"2024-08-30T23:28:54.853813+0000","last_fullsized":"2024-08-30T23:28:54.853813+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"187'10348","reported_seq":10585,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.491868+0000","last_change":"2024-08-30T23:28:16.855547+0000","last_active":"2024-08-30T23:28:58.491868+0000","last_peered":"2024-08-30T23:28:58.491868+0000","last_clean":"2024-08-30T23:28:58.491868+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:28:58.491868+0000","last_undegraded":"2024-08-30T23:28:58.491868+0000","last_fullsized":"2024-08-30T23:28:58.491868+0000","mapping_epoch":97,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'8559","last_scrub_stamp":"2024-08-30T23:28:16.855286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.855286+0000","objects_scrubbed":3791,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:23.340082+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129761280,"num_objects":1980,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10348,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"186'10112","reported_seq":20486,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:57.317461+0000","last_change":"2024-08-30T23:28:43.056625+0000","last_active":"2024-08-30T23:28:57.317461+0000","last_peered":"2024-08-30T23:28:57.317461+0000","last_clean":"2024-08-30T23:28:57.317461+0000","last_became_active":"2024-08-30T23:28:25.514631+0000","last_became_peered":"2024-08-30T23:28:25.514631+0000","last_unstale":"2024-08-30T23:28:57.317461+0000","last_undegraded":"2024-08-30T23:28:57.317461+0000","last_fullsized":"2024-08-30T23:28:57.317461+0000","mapping_epoch":168,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9573","last_scrub_stamp":"2024-08-30T23:28:43.056366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.056366+0000","objects_scrubbed":2652,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:58.660703+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10112,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332136448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"187'10230","reported_seq":10451,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.500994+0000","last_change":"2024-08-30T23:28:53.329917+0000","last_active":"2024-08-30T23:28:58.500994+0000","last_peered":"2024-08-30T23:28:58.500994+0000","last_clean":"2024-08-30T23:28:58.500994+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:28:58.500994+0000","last_undegraded":"2024-08-30T23:28:58.500994+0000","last_fullsized":"2024-08-30T23:28:58.500994+0000","mapping_epoch":97,"log_start":"106'200","ondisk_log_start":"106'200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9999","last_scrub_stamp":"2024-08-30T23:28:53.329808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:53.329808+0000","objects_scrubbed":2220,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:04.773357+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129368064,"num_objects":1974,"num_object_clones":0,"num_object_copies":5922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10230,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"187'10150","reported_seq":10387,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.564908+0000","last_change":"2024-08-30T23:28:23.087871+0000","last_active":"2024-08-30T23:28:58.564908+0000","last_peered":"2024-08-30T23:28:58.564908+0000","last_clean":"2024-08-30T23:28:58.564908+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:28:58.564908+0000","last_undegraded":"2024-08-30T23:28:58.564908+0000","last_fullsized":"2024-08-30T23:28:58.564908+0000","mapping_epoch":97,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'8647","last_scrub_stamp":"2024-08-30T23:28:23.087486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:23.087486+0000","objects_scrubbed":3550,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:51.858797+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133300224,"num_objects":2034,"num_object_clones":0,"num_object_copies":6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"186'10152","reported_seq":20492,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.221371+0000","last_change":"2024-08-30T23:28:51.930260+0000","last_active":"2024-08-30T23:28:58.221371+0000","last_peered":"2024-08-30T23:28:58.221371+0000","last_clean":"2024-08-30T23:28:58.221371+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:28:58.221371+0000","last_undegraded":"2024-08-30T23:28:58.221371+0000","last_fullsized":"2024-08-30T23:28:58.221371+0000","mapping_epoch":143,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126484480,"num_objects":1930,"num_object_clones":0,"num_object_copies":5790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5071,"num_bytes_recovered":332333056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"186'10136","reported_seq":20411,"reported_epoch":186,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:28:58.347730+0000","last_change":"2024-08-30T23:28:47.992578+0000","last_active":"2024-08-30T23:28:58.347730+0000","last_peered":"2024-08-30T23:28:58.347730+0000","last_clean":"2024-08-30T23:28:45.671248+0000","last_became_active":"2024-08-30T23:28:47.905585+0000","last_became_peered":"2024-08-30T23:28:47.905585+0000","last_unstale":"2024-08-30T23:28:58.347730+0000","last_undegraded":"2024-08-30T23:28:47.716401+0000","last_fullsized":"2024-08-30T23:28:47.688373+0000","mapping_epoch":182,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127664128,"num_objects":1948,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":145,"num_objects_misplaced":1948,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10136,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5037,"num_bytes_recovered":330039296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1948},{"shards":"0,4","objects":-145}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"186'10077","reported_seq":10336,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:55.647769+0000","last_change":"2024-08-30T23:27:43.200813+0000","last_active":"2024-08-30T23:28:55.647769+0000","last_peered":"2024-08-30T23:28:55.647769+0000","last_clean":"2024-08-30T23:28:55.647769+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:28:55.647769+0000","last_undegraded":"2024-08-30T23:28:55.647769+0000","last_fullsized":"2024-08-30T23:28:55.647769+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'6776","last_scrub_stamp":"2024-08-30T23:27:43.200650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:43.200650+0000","objects_scrubbed":5424,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137166848,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10077,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"186'10101","reported_seq":20433,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:56.819830+0000","last_change":"2024-08-30T23:28:54.859818+0000","last_active":"2024-08-30T23:28:56.819830+0000","last_peered":"2024-08-30T23:28:56.819830+0000","last_clean":"2024-08-30T23:28:56.819830+0000","last_became_active":"2024-08-30T23:28:47.690562+0000","last_became_peered":"2024-08-30T23:28:47.690562+0000","last_unstale":"2024-08-30T23:28:56.819830+0000","last_undegraded":"2024-08-30T23:28:56.819830+0000","last_fullsized":"2024-08-30T23:28:56.819830+0000","mapping_epoch":182,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":1249,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:28:59.840438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133758976,"num_objects":2041,"num_object_clones":0,"num_object_copies":6123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10101,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":331415552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"186'10079","reported_seq":10309,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:57.263897+0000","last_change":"2024-08-30T23:28:24.690367+0000","last_active":"2024-08-30T23:28:57.263897+0000","last_peered":"2024-08-30T23:28:57.263897+0000","last_clean":"2024-08-30T23:28:57.263897+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:28:57.263897+0000","last_undegraded":"2024-08-30T23:28:57.263897+0000","last_fullsized":"2024-08-30T23:28:57.263897+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'8760","last_scrub_stamp":"2024-08-30T23:28:24.690182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:24.690182+0000","objects_scrubbed":3297,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:26.848779+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129040384,"num_objects":1969,"num_object_clones":0,"num_object_copies":5907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10079,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"186'10288","reported_seq":10475,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:57.418680+0000","last_change":"2024-08-30T23:25:50.474563+0000","last_active":"2024-08-30T23:28:57.418680+0000","last_peered":"2024-08-30T23:28:57.418680+0000","last_clean":"2024-08-30T23:28:57.418680+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:28:57.418680+0000","last_undegraded":"2024-08-30T23:28:57.418680+0000","last_fullsized":"2024-08-30T23:28:57.418680+0000","mapping_epoch":97,"log_start":"105'200","ondisk_log_start":"105'200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:02.294622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130809856,"num_objects":1996,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"186'9836","reported_seq":10060,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-08-30T23:28:56.053381+0000","last_change":"2024-08-30T23:28:39.652116+0000","last_active":"2024-08-30T23:28:56.053381+0000","last_peered":"2024-08-30T23:28:56.053381+0000","last_clean":"2024-08-30T23:28:56.053381+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:28:56.053381+0000","last_undegraded":"2024-08-30T23:28:56.053381+0000","last_fullsized":"2024-08-30T23:28:56.053381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9191","last_scrub_stamp":"2024-08-30T23:28:39.651956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:39.651956+0000","objects_scrubbed":2655,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.058479+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130285568,"num_objects":1988,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9836,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"187'10122","reported_seq":20664,"reported_epoch":187,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:28:58.592568+0000","last_change":"2024-08-30T23:28:47.980377+0000","last_active":"2024-08-30T23:28:58.592568+0000","last_peered":"2024-08-30T23:28:58.592568+0000","last_clean":"2024-08-30T23:28:45.654798+0000","last_became_active":"2024-08-30T23:28:47.911739+0000","last_became_peered":"2024-08-30T23:28:47.911739+0000","last_unstale":"2024-08-30T23:28:58.592568+0000","last_undegraded":"2024-08-30T23:28:58.592568+0000","last_fullsized":"2024-08-30T23:28:58.592568+0000","mapping_epoch":182,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9517","last_scrub_stamp":"2024-08-30T23:28:43.578020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.578020+0000","objects_scrubbed":2573,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128057344,"num_objects":1954,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1955,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5152,"num_bytes_recovered":337379328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[5,11,7],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"5,7,11","objects":-1}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2091122712,"num_objects":31909,"num_object_clones":0,"num_object_copies":95727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":145,"num_objects_misplaced":3903,"num_objects_unfound":0,"num_objects_dirty":31909,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162103,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25386,"num_bytes_recovered":1663369216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7141404672,"data_stored":7141392456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160503,"ondisk_log_size":160503,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187509,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626068,"kb_used_data":532416,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93119084,"statfs":{"total":95995035648,"available":95353942016,"internally_reserved":0,"allocated":545193984,"data_stored":544807184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":95877572},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449147,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710784,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825308,"kb_used_data":726088,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92919844,"statfs":{"total":95995035648,"available":95149920256,"internally_reserved":0,"allocated":743514112,"data_stored":743122417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972421,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351500,"kb_used_data":275832,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93393652,"statfs":{"total":95995035648,"available":95635099648,"internally_reserved":0,"allocated":282451968,"data_stored":282068740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234056,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722640,"kb_used_data":651200,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93022512,"statfs":{"total":95995035648,"available":95255052288,"internally_reserved":0,"allocated":666828800,"data_stored":666442020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528398,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316692,"kb_used_data":271616,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93428460,"statfs":{"total":95995035648,"available":95670743040,"internally_reserved":0,"allocated":278134784,"data_stored":277748004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724626,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458312,"kb_used_data":386872,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93286840,"statfs":{"total":95995035648,"available":95525724160,"internally_reserved":0,"allocated":396156928,"data_stored":395773700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438492,"kb_used_data":367048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93306660,"statfs":{"total":95995035648,"available":95546019840,"internally_reserved":0,"allocated":375857152,"data_stored":375465457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982156,"kb_used_data":870460,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92762996,"statfs":{"total":95995035648,"available":94989307904,"internally_reserved":0,"allocated":891351040,"data_stored":890963740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736136,"kb_used_data":636920,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93009016,"statfs":{"total":95995035648,"available":95241232384,"internally_reserved":0,"allocated":652206080,"data_stored":651822852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905416,"kb_used_data":801912,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92839736,"statfs":{"total":95995035648,"available":95067889664,"internally_reserved":0,"allocated":821157888,"data_stored":820774660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714952,"kb_used_data":633784,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93030200,"statfs":{"total":95995035648,"available":95262924800,"internally_reserved":0,"allocated":648994816,"data_stored":648611588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":637861888,"data_stored":637861888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":817102848,"data_stored":817102848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":639041536,"data_stored":639041536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":887623680,"data_stored":887619608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":372969472,"data_stored":372965400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":391053312,"data_stored":391053312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":271384576,"data_stored":271384576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":662044672,"data_stored":662044672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":278790144,"data_stored":278790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":734466048,"data_stored":734461976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":907608064,"data_stored":907608064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":541458432,"data_stored":541458432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:29:02.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:29:02.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:02.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:02.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:02.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:02.606 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:02 smithi106 ceph-mon[16603]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:02 smithi106 ceph-mon[16603]: pgmap v483: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 145/93102 objects degraded (0.156%); 3903/93102 objects misplaced (4.192%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:29:02.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:02.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:02.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:02 smithi173 ceph-mon[20554]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:02.746 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:02 smithi173 ceph-mon[20554]: pgmap v483: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 145/93102 objects degraded (0.156%); 3903/93102 objects misplaced (4.192%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:29:02.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:02.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:02 smithi136 ceph-mon[20207]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:02.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:02 smithi136 ceph-mon[20207]: pgmap v483: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 6.7 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 145/93102 objects degraded (0.156%); 3903/93102 objects misplaced (4.192%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:29:02.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:02.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:02.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:02.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:03.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:03.066 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:03.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:29:03.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:03.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:03.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:03.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:03.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:03.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:29:03.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:03.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:03.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:03.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:03.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:03.737 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:03.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:03.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:03.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:03.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:03.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:03.972 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:03.973 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:04.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:04.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:04.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:04.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:04.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:04.275 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:04.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:04.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:04.456 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:04.486 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:29:04.486 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:29:04.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:04.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:04 smithi106 ceph-mon[16603]: pgmap v484: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 1.8 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 203/88929 objects degraded (0.228%); 3674/88929 objects misplaced (4.131%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:29:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:04 smithi106 ceph-mon[16603]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:04.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:04.670 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:04.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:04 smithi136 ceph-mon[20207]: pgmap v484: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 1.8 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 203/88929 objects degraded (0.228%); 3674/88929 objects misplaced (4.131%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:29:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:04 smithi136 ceph-mon[20207]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:04.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:04 smithi173 ceph-mon[20554]: pgmap v484: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 1.8 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 730 op/s; 203/88929 objects degraded (0.228%); 3674/88929 objects misplaced (4.131%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:29:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:04 smithi173 ceph-mon[20554]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:04.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:04.916 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-08-30T23:29:03.383495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1943077432,"num_objects":29643,"num_object_clones":0,"num_object_copies":88929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":203,"num_objects_misplaced":3674,"num_objects_unfound":0,"num_objects_dirty":29643,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":164494,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25563,"num_bytes_recovered":1676162656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6775976,"kb_used_data":5824744,"kb_used_omap":17,"kb_used_meta":950894,"kb_avail":1024420696,"statfs":{"total":1055945392128,"available":1049006792704,"internally_reserved":0,"allocated":5964537856,"data_stored":5960242031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":973716345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,6,4,4,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383254528,"num_objects":-5848,"num_object_clones":0,"num_object_copies":-17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":132,"num_objects_misplaced":-674,"num_objects_unfound":0,"num_objects_dirty":-5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5848,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":35520512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001679"},"pg_stats":[{"pgid":"2.f","version":"188'10300","reported_seq":10599,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:00.649671+0000","last_change":"2024-08-30T23:28:16.218035+0000","last_active":"2024-08-30T23:29:00.649671+0000","last_peered":"2024-08-30T23:29:00.649671+0000","last_clean":"2024-08-30T23:29:00.649671+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:29:00.649671+0000","last_undegraded":"2024-08-30T23:29:00.649671+0000","last_fullsized":"2024-08-30T23:29:00.649671+0000","mapping_epoch":97,"log_start":"105'200","ondisk_log_start":"105'200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'8409","last_scrub_stamp":"2024-08-30T23:28:16.217972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.217972+0000","objects_scrubbed":3813,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:45.741305+0000","scrub_duration":782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"187'10033","reported_seq":10225,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.455214+0000","last_change":"2024-08-30T23:25:50.474312+0000","last_active":"2024-08-30T23:28:58.455214+0000","last_peered":"2024-08-30T23:28:58.455214+0000","last_clean":"2024-08-30T23:28:58.455214+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:28:58.455214+0000","last_undegraded":"2024-08-30T23:28:58.455214+0000","last_fullsized":"2024-08-30T23:28:58.455214+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:02.709892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124846080,"num_objects":1905,"num_object_clones":0,"num_object_copies":5715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10033,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"187'10385","reported_seq":10619,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:29:00.052138+0000","last_change":"2024-08-30T23:28:27.803716+0000","last_active":"2024-08-30T23:29:00.052138+0000","last_peered":"2024-08-30T23:29:00.052138+0000","last_clean":"2024-08-30T23:29:00.052138+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:29:00.052138+0000","last_undegraded":"2024-08-30T23:29:00.052138+0000","last_fullsized":"2024-08-30T23:29:00.052138+0000","mapping_epoch":97,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'9007","last_scrub_stamp":"2024-08-30T23:28:27.803577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.803577+0000","objects_scrubbed":3238,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:55.835178+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119930904,"num_objects":1831,"num_object_clones":0,"num_object_copies":5493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10385,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"188'10201","reported_seq":10437,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:01.005217+0000","last_change":"2024-08-30T23:28:27.113440+0000","last_active":"2024-08-30T23:29:01.005217+0000","last_peered":"2024-08-30T23:29:01.005217+0000","last_clean":"2024-08-30T23:29:01.005217+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:29:01.005217+0000","last_undegraded":"2024-08-30T23:29:01.005217+0000","last_fullsized":"2024-08-30T23:29:01.005217+0000","mapping_epoch":97,"log_start":"106'200","ondisk_log_start":"106'200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'8728","last_scrub_stamp":"2024-08-30T23:28:27.113155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.113155+0000","objects_scrubbed":3323,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:28.567110+0000","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119209984,"num_objects":1819,"num_object_clones":0,"num_object_copies":5457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10201,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":335,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:00.470999+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:00.470999+0000","last_peered":"2024-08-30T23:29:00.470999+0000","last_clean":"2024-08-30T23:29:00.470999+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:00.470999+0000","last_undegraded":"2024-08-30T23:29:00.470999+0000","last_fullsized":"2024-08-30T23:29:00.470999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"187'10348","reported_seq":10585,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.491868+0000","last_change":"2024-08-30T23:28:16.855547+0000","last_active":"2024-08-30T23:28:58.491868+0000","last_peered":"2024-08-30T23:28:58.491868+0000","last_clean":"2024-08-30T23:28:58.491868+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:28:58.491868+0000","last_undegraded":"2024-08-30T23:28:58.491868+0000","last_fullsized":"2024-08-30T23:28:58.491868+0000","mapping_epoch":97,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'8559","last_scrub_stamp":"2024-08-30T23:28:16.855286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.855286+0000","objects_scrubbed":3791,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:23.340082+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129761280,"num_objects":1980,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10348,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"188'10364","reported_seq":20742,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:02.353787+0000","last_change":"2024-08-30T23:28:43.056625+0000","last_active":"2024-08-30T23:29:02.353787+0000","last_peered":"2024-08-30T23:29:02.353787+0000","last_clean":"2024-08-30T23:29:02.353787+0000","last_became_active":"2024-08-30T23:28:25.514631+0000","last_became_peered":"2024-08-30T23:28:25.514631+0000","last_unstale":"2024-08-30T23:29:02.353787+0000","last_undegraded":"2024-08-30T23:29:02.353787+0000","last_fullsized":"2024-08-30T23:29:02.353787+0000","mapping_epoch":168,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9573","last_scrub_stamp":"2024-08-30T23:28:43.056366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.056366+0000","objects_scrubbed":2652,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:58.660703+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120848384,"num_objects":1844,"num_object_clones":0,"num_object_copies":5532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332136448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"187'10230","reported_seq":10451,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.500994+0000","last_change":"2024-08-30T23:28:53.329917+0000","last_active":"2024-08-30T23:28:58.500994+0000","last_peered":"2024-08-30T23:28:58.500994+0000","last_clean":"2024-08-30T23:28:58.500994+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:28:58.500994+0000","last_undegraded":"2024-08-30T23:28:58.500994+0000","last_fullsized":"2024-08-30T23:28:58.500994+0000","mapping_epoch":97,"log_start":"106'200","ondisk_log_start":"106'200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9999","last_scrub_stamp":"2024-08-30T23:28:53.329808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:53.329808+0000","objects_scrubbed":2220,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:04.773357+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129368064,"num_objects":1974,"num_object_clones":0,"num_object_copies":5922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10230,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"187'10150","reported_seq":10387,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-08-30T23:28:58.564908+0000","last_change":"2024-08-30T23:28:23.087871+0000","last_active":"2024-08-30T23:28:58.564908+0000","last_peered":"2024-08-30T23:28:58.564908+0000","last_clean":"2024-08-30T23:28:58.564908+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:28:58.564908+0000","last_undegraded":"2024-08-30T23:28:58.564908+0000","last_fullsized":"2024-08-30T23:28:58.564908+0000","mapping_epoch":97,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'8647","last_scrub_stamp":"2024-08-30T23:28:23.087486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:23.087486+0000","objects_scrubbed":3550,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:51.858797+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133300224,"num_objects":2034,"num_object_clones":0,"num_object_copies":6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"188'10389","reported_seq":20733,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:03.274509+0000","last_change":"2024-08-30T23:28:51.930260+0000","last_active":"2024-08-30T23:29:03.274509+0000","last_peered":"2024-08-30T23:29:03.274509+0000","last_clean":"2024-08-30T23:29:03.274509+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:29:03.274509+0000","last_undegraded":"2024-08-30T23:29:03.274509+0000","last_fullsized":"2024-08-30T23:29:03.274509+0000","mapping_epoch":143,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110952448,"num_objects":1693,"num_object_clones":0,"num_object_copies":5079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5071,"num_bytes_recovered":332333056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"188'10365","reported_seq":20988,"reported_epoch":188,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:29:03.329777+0000","last_change":"2024-08-30T23:28:47.992578+0000","last_active":"2024-08-30T23:29:03.329777+0000","last_peered":"2024-08-30T23:29:03.329777+0000","last_clean":"2024-08-30T23:28:45.671248+0000","last_became_active":"2024-08-30T23:28:47.905585+0000","last_became_peered":"2024-08-30T23:28:47.905585+0000","last_unstale":"2024-08-30T23:29:03.329777+0000","last_undegraded":"2024-08-30T23:28:47.716401+0000","last_fullsized":"2024-08-30T23:28:47.688373+0000","mapping_epoch":182,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112656384,"num_objects":1719,"num_object_clones":0,"num_object_copies":5157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":203,"num_objects_misplaced":1719,"num_objects_unfound":0,"num_objects_dirty":1719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10365,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5208,"num_bytes_recovered":341245952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1719},{"shards":"0,4","objects":-203}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"188'10290","reported_seq":10552,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:00.625671+0000","last_change":"2024-08-30T23:27:43.200813+0000","last_active":"2024-08-30T23:29:00.625671+0000","last_peered":"2024-08-30T23:29:00.625671+0000","last_clean":"2024-08-30T23:29:00.625671+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:29:00.625671+0000","last_undegraded":"2024-08-30T23:29:00.625671+0000","last_fullsized":"2024-08-30T23:29:00.625671+0000","mapping_epoch":142,"log_start":"105'200","ondisk_log_start":"105'200","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'6776","last_scrub_stamp":"2024-08-30T23:27:43.200650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:43.200650+0000","objects_scrubbed":5424,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123207680,"num_objects":1880,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10290,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"188'10316","reported_seq":20651,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:01.818090+0000","last_change":"2024-08-30T23:28:54.859818+0000","last_active":"2024-08-30T23:29:01.818090+0000","last_peered":"2024-08-30T23:29:01.818090+0000","last_clean":"2024-08-30T23:29:01.818090+0000","last_became_active":"2024-08-30T23:28:47.690562+0000","last_became_peered":"2024-08-30T23:28:47.690562+0000","last_unstale":"2024-08-30T23:29:01.818090+0000","last_undegraded":"2024-08-30T23:29:01.818090+0000","last_fullsized":"2024-08-30T23:29:01.818090+0000","mapping_epoch":182,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":1249,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:05.079502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119668736,"num_objects":1826,"num_object_clones":0,"num_object_copies":5478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":331415552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"188'10303","reported_seq":10537,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:02.274711+0000","last_change":"2024-08-30T23:28:24.690367+0000","last_active":"2024-08-30T23:29:02.274711+0000","last_peered":"2024-08-30T23:29:02.274711+0000","last_clean":"2024-08-30T23:29:02.274711+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:29:02.274711+0000","last_undegraded":"2024-08-30T23:29:02.274711+0000","last_fullsized":"2024-08-30T23:29:02.274711+0000","mapping_epoch":97,"log_start":"107'300","ondisk_log_start":"107'300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'8760","last_scrub_stamp":"2024-08-30T23:28:24.690182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:24.690182+0000","objects_scrubbed":3297,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:26.848779+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114360320,"num_objects":1745,"num_object_clones":0,"num_object_copies":5235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"188'10522","reported_seq":10715,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:02.498966+0000","last_change":"2024-08-30T23:25:50.474563+0000","last_active":"2024-08-30T23:29:02.498966+0000","last_peered":"2024-08-30T23:29:02.498966+0000","last_clean":"2024-08-30T23:29:02.498966+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:29:02.498966+0000","last_undegraded":"2024-08-30T23:29:02.498966+0000","last_fullsized":"2024-08-30T23:29:02.498966+0000","mapping_epoch":97,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115474432,"num_objects":1762,"num_object_clones":0,"num_object_copies":5286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10522,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"188'10053","reported_seq":10281,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-08-30T23:29:01.050727+0000","last_change":"2024-08-30T23:28:39.652116+0000","last_active":"2024-08-30T23:29:01.050727+0000","last_peered":"2024-08-30T23:29:01.050727+0000","last_clean":"2024-08-30T23:29:01.050727+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:29:01.050727+0000","last_undegraded":"2024-08-30T23:29:01.050727+0000","last_fullsized":"2024-08-30T23:29:01.050727+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9191","last_scrub_stamp":"2024-08-30T23:28:39.651956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:39.651956+0000","objects_scrubbed":2655,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.058479+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116064256,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10053,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"187'10122","reported_seq":20664,"reported_epoch":187,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:28:58.592568+0000","last_change":"2024-08-30T23:28:47.980377+0000","last_active":"2024-08-30T23:28:58.592568+0000","last_peered":"2024-08-30T23:28:58.592568+0000","last_clean":"2024-08-30T23:28:45.654798+0000","last_became_active":"2024-08-30T23:28:47.911739+0000","last_became_peered":"2024-08-30T23:28:47.911739+0000","last_unstale":"2024-08-30T23:28:58.592568+0000","last_undegraded":"2024-08-30T23:28:58.592568+0000","last_fullsized":"2024-08-30T23:28:58.592568+0000","mapping_epoch":182,"log_start":"104'100","ondisk_log_start":"104'100","created":97,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9517","last_scrub_stamp":"2024-08-30T23:28:43.578020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.578020+0000","objects_scrubbed":2573,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128057344,"num_objects":1954,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1955,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10122,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5152,"num_bytes_recovered":337379328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[5,11,7],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"5,7,11","objects":-1}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1942487064,"num_objects":29641,"num_object_clones":0,"num_object_copies":88923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":203,"num_objects_misplaced":3674,"num_objects_unfound":0,"num_objects_dirty":29641,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164371,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25557,"num_bytes_recovered":1674575872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6773616640,"data_stored":6773604424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187510,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":572712,"kb_used_data":473496,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93172440,"statfs":{"total":95995035648,"available":95408578560,"internally_reserved":0,"allocated":484859904,"data_stored":484464256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":101579204},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710784,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":825308,"kb_used_data":726088,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92919844,"statfs":{"total":95995035648,"available":95149920256,"internally_reserved":0,"allocated":743514112,"data_stored":743122417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972422,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310932,"kb_used_data":247812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93434220,"statfs":{"total":95995035648,"available":95676641280,"internally_reserved":0,"allocated":253759488,"data_stored":253371865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234057,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":656172,"kb_used_data":580568,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93088980,"statfs":{"total":95995035648,"available":95323115520,"internally_reserved":0,"allocated":594501632,"data_stored":594106004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528399,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":294184,"kb_used_data":244888,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93450968,"statfs":{"total":95995035648,"available":95693791232,"internally_reserved":0,"allocated":250765312,"data_stored":250369684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724626,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458312,"kb_used_data":386872,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93286840,"statfs":{"total":95995035648,"available":95525724160,"internally_reserved":0,"allocated":396156928,"data_stored":395773700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438492,"kb_used_data":367048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93306660,"statfs":{"total":95995035648,"available":95546019840,"internally_reserved":0,"allocated":375857152,"data_stored":375465457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":982156,"kb_used_data":870460,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92762996,"statfs":{"total":95995035648,"available":94989307904,"internally_reserved":0,"allocated":891351040,"data_stored":890963740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":736136,"kb_used_data":636920,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93009016,"statfs":{"total":95995035648,"available":95241232384,"internally_reserved":0,"allocated":652206080,"data_stored":651822852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837600,"kb_used_data":730000,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92907552,"statfs":{"total":95995035648,"available":95137333248,"internally_reserved":0,"allocated":747520000,"data_stored":747127924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663972,"kb_used_data":560592,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93081180,"statfs":{"total":95995035648,"available":95315128320,"internally_reserved":0,"allocated":574046208,"data_stored":573654132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":561250304,"data_stored":561250304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":743636992,"data_stored":743636992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":639041536,"data_stored":639041536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":887623680,"data_stored":887619608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":372969472,"data_stored":372965400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":391053312,"data_stored":391053312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":244318208,"data_stored":244318208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":587661312,"data_stored":587661312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":251330560,"data_stored":251330560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":734466048,"data_stored":734461976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":878116864,"data_stored":878116864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":482148352,"data_stored":482148352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:29:04.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-08-30T23:29:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:04.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:04.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:04.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:29:05.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:05.056 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:05.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:05.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:05.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:05.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:05.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:05.444 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:05.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:05.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:05.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:05.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:05.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:05.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:05.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:05 smithi173 ceph-mon[20554]: osdmap e189: 12 total, 12 up, 11 in 2024-08-30T23:29:05.737 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:05 smithi173 ceph-mon[20554]: pgmap v486: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 203/85143 objects degraded (0.238%); 3422/85143 objects misplaced (4.019%); 5.8 MiB/s, 93 objects/s recovering 2024-08-30T23:29:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:05.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:05.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:05 smithi136 ceph-mon[20207]: osdmap e189: 12 total, 12 up, 11 in 2024-08-30T23:29:05.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:05 smithi136 ceph-mon[20207]: pgmap v486: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 203/85143 objects degraded (0.238%); 3422/85143 objects misplaced (4.019%); 5.8 MiB/s, 93 objects/s recovering 2024-08-30T23:29:05.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:05.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:05 smithi106 ceph-mon[16603]: osdmap e189: 12 total, 12 up, 11 in 2024-08-30T23:29:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:05 smithi106 ceph-mon[16603]: pgmap v486: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 203/85143 objects degraded (0.238%); 3422/85143 objects misplaced (4.019%); 5.8 MiB/s, 93 objects/s recovering 2024-08-30T23:29:05.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:29:05.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:05.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:05.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:06.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:06.089 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:06.146 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:06.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:06.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:06.242 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:06.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:06.363 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:06.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:06.487 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 203/85143 objects degraded (0.238%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:06.487 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:06 smithi106 ceph-mon[16603]: osdmap e190: 12 total, 12 up, 11 in 2024-08-30T23:29:06.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:06.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 203/85143 objects degraded (0.238%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:06.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:06 smithi136 ceph-mon[20207]: osdmap e190: 12 total, 12 up, 11 in 2024-08-30T23:29:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 203/85143 objects degraded (0.238%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:06 smithi173 ceph-mon[20554]: osdmap e190: 12 total, 12 up, 11 in 2024-08-30T23:29:06.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:06.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:06.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:06.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:06.760 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:06.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:06.895 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:06.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:06.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:07.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:07.061 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:07.061 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:07.103 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:07.103 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:07.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:07.143 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:07.143 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:07.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:07.179 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:07.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:07.201 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:07.214 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:07.215 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:07.251 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:07.251 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:07.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:07.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:07.286 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:07.287 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:07.323 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:07.324 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:07.335 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:07.336 INFO:teuthology.orchestra.run.smithi106.stdout:{"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 203/85143 objects degraded (0.238%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":562,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":190,"num_osds":12,"num_up_osds":12,"osd_up_since":1725060285,"num_in_osds":11,"osd_in_since":1725060424,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":28381,"data_bytes":1860371000,"bytes_used":6599557120,"bytes_avail":1049345835008,"bytes_total":1055945392128,"degraded_objects":203,"degraded_total":85143,"degraded_ratio":0.0023842241875433093,"misplaced_objects":3422,"misplaced_total":85143,"misplaced_ratio":0.040191207732872929,"recovering_objects_per_sec":93,"recovering_bytes_per_sec":6123431,"recovering_keys_per_sec":0,"num_objects_recovered":649,"num_bytes_recovered":42532864,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":698},"fsmap":{"epoch":1,"btime":"2024-08-30T23:18:02:086399+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.106:8443/"}},"servicemap":{"epoch":18,"modified":"2024-08-30T23:27:07.354623+0000","services":{}},"progress_events":{}} 2024-08-30T23:29:07.350 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:07.350 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:07.374 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:07.374 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:07.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:07.403 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:07.403 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:07.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:07.435 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:07.435 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:07.493 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:29:07.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:07.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:07.576 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1420684378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:07.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/635785182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:07.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/808767577' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:07.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:07 smithi106 ceph-mon[16603]: pgmap v488: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 203/80712 objects degraded (0.252%); 3422/80712 objects misplaced (4.240%); 3.8 MiB/s, 61 objects/s recovering 2024-08-30T23:29:07.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:07.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:07.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:07.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1420684378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/635785182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/808767577' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:07 smithi136 ceph-mon[20207]: pgmap v488: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 203/80712 objects degraded (0.252%); 3422/80712 objects misplaced (4.240%); 3.8 MiB/s, 61 objects/s recovering 2024-08-30T23:29:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1420684378' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/635785182' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/808767577' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:07 smithi173 ceph-mon[20554]: pgmap v488: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.6 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 203/80712 objects degraded (0.252%); 3422/80712 objects misplaced (4.240%); 3.8 MiB/s, 61 objects/s recovering 2024-08-30T23:29:07.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:07.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:07.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:08.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:08.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:08.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:08.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:08.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:08.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:08.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:08.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:08.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:08.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:08.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:08.662 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:08 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:08.662 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1420684378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:29:08.662 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:08 smithi106 ceph-mon[16603]: osdmap e191: 12 total, 12 up, 11 in 2024-08-30T23:29:08.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:08.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:08 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1420684378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:29:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:08 smithi136 ceph-mon[20207]: osdmap e191: 12 total, 12 up, 11 in 2024-08-30T23:29:08.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:08 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1420684378' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:29:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:08 smithi173 ceph-mon[20554]: osdmap e191: 12 total, 12 up, 11 in 2024-08-30T23:29:08.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:08.933 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:29:08.933 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:29:08.933 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:29:08.933 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:29:08.933 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:29:08.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:08.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:09.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:09.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:09.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:09.208 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:09.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:09.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:09.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:09.470 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:09.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.602 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:09.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:09.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:09.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:09.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:09.777 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:09.777 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:09 smithi106 ceph-mon[16603]: pgmap v490: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 216/75843 objects degraded (0.285%); 2968/75843 objects misplaced (3.913%); 7.2 MiB/s, 115 objects/s recovering 2024-08-30T23:29:09.781 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-30T23:29:09.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:29:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:09 smithi136 ceph-mon[20207]: pgmap v490: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 216/75843 objects degraded (0.285%); 2968/75843 objects misplaced (3.913%); 7.2 MiB/s, 115 objects/s recovering 2024-08-30T23:29:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:09 smithi173 ceph-mon[20554]: pgmap v490: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 216/75843 objects degraded (0.285%); 2968/75843 objects misplaced (3.913%); 7.2 MiB/s, 115 objects/s recovering 2024-08-30T23:29:09.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:09.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:09.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:09.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:09.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:10.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:10.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:10.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:10.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:10.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:10.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:10.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:10.489 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:10.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:10.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:10.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:29:10.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:10.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:10 smithi136 ceph-mon[20207]: osdmap e192: 12 total, 12 up, 11 in 2024-08-30T23:29:10.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:10 smithi173 ceph-mon[20554]: osdmap e192: 12 total, 12 up, 11 in 2024-08-30T23:29:10.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:10 smithi106 ceph-mon[16603]: osdmap e192: 12 total, 12 up, 11 in 2024-08-30T23:29:10.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:10.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:10.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:10.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:29:10.971 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:11.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:11.059 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:11.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:11.191 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:11.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:11.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:11.360 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:11.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:29:11.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:11.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:11.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:11.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:11.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:11.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:11 smithi136 ceph-mon[20207]: 2.9 scrub starts 2024-08-30T23:29:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 216/75843 objects degraded (0.285%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:11 smithi136 ceph-mon[20207]: 2.9 scrub ok 2024-08-30T23:29:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:11 smithi136 ceph-mon[20207]: pgmap v492: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 216/72729 objects degraded (0.297%); 2968/72729 objects misplaced (4.081%); 4.6 MiB/s, 73 objects/s recovering 2024-08-30T23:29:11.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:11.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:11 smithi106 ceph-mon[16603]: 2.9 scrub starts 2024-08-30T23:29:11.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 216/75843 objects degraded (0.285%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:11.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:11 smithi106 ceph-mon[16603]: 2.9 scrub ok 2024-08-30T23:29:11.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:11 smithi106 ceph-mon[16603]: pgmap v492: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 216/72729 objects degraded (0.297%); 2968/72729 objects misplaced (4.081%); 4.6 MiB/s, 73 objects/s recovering 2024-08-30T23:29:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:11 smithi173 ceph-mon[20554]: 2.9 scrub starts 2024-08-30T23:29:11.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 216/75843 objects degraded (0.285%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:11.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:11 smithi173 ceph-mon[20554]: 2.9 scrub ok 2024-08-30T23:29:11.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:11 smithi173 ceph-mon[20554]: pgmap v492: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 216/72729 objects degraded (0.297%); 2968/72729 objects misplaced (4.081%); 4.6 MiB/s, 73 objects/s recovering 2024-08-30T23:29:11.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:11.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:11.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:11.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:12.017 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:12.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:12.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:12.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:12.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:12.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:12.363 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:12.394 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:12.442 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:12.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:12.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:12.600 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:29:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:12 smithi106 ceph-mon[16603]: 2.7 scrub starts 2024-08-30T23:29:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:12 smithi106 ceph-mon[16603]: 2.7 scrub ok 2024-08-30T23:29:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:12 smithi106 ceph-mon[16603]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:12.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:12.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:12.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:12.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:12.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:12 smithi173 ceph-mon[20554]: 2.7 scrub starts 2024-08-30T23:29:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:12 smithi173 ceph-mon[20554]: 2.7 scrub ok 2024-08-30T23:29:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:12 smithi173 ceph-mon[20554]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:12.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:12.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:12.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:13.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:12 smithi136 ceph-mon[20207]: 2.7 scrub starts 2024-08-30T23:29:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:12 smithi136 ceph-mon[20207]: 2.7 scrub ok 2024-08-30T23:29:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:12 smithi136 ceph-mon[20207]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:13.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:13.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:13.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:13.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:13.302 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:13.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:13.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:13.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:13.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:13.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1443852292' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:13.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:13.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:13.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:29:13.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1443852292' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:13.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: osdmap e193: 12 total, 12 up, 11 in 2024-08-30T23:29:13.674 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: 2.e scrub starts 2024-08-30T23:29:13.674 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:13 smithi106 ceph-mon[16603]: pgmap v494: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 1.4 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 231/68760 objects degraded (0.336%); 2743/68760 objects misplaced (3.989%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:29:13.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:13.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:13.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:13.821 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1443852292' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:29:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1443852292' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: osdmap e193: 12 total, 12 up, 11 in 2024-08-30T23:29:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: 2.e scrub starts 2024-08-30T23:29:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:13 smithi173 ceph-mon[20554]: pgmap v494: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 1.4 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 231/68760 objects degraded (0.336%); 2743/68760 objects misplaced (3.989%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:29:13.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:13.947 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:13.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1443852292' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:29:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1443852292' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: osdmap e193: 12 total, 12 up, 11 in 2024-08-30T23:29:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: 2.e scrub starts 2024-08-30T23:29:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:13 smithi136 ceph-mon[20207]: pgmap v494: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 15 active+clean; 1.4 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 231/68760 objects degraded (0.336%); 2743/68760 objects misplaced (3.989%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:29:14.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:14.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:14.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:14.196 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:14.298 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:14.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:14.431 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:14.454 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-08-30T23:29:11.386897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1589183032,"num_objects":24243,"num_object_clones":0,"num_object_copies":72729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":216,"num_objects_misplaced":2968,"num_objects_unfound":0,"num_objects_dirty":24243,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":169894,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26256,"num_bytes_recovered":1721579104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160763,"ondisk_log_size":160763,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":5837620,"kb_used_data":4800416,"kb_used_omap":17,"kb_used_meta":1036910,"kb_avail":1025359052,"statfs":{"total":1055945392128,"available":1049967669248,"internally_reserved":0,"allocated":4915625984,"data_stored":4911144007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1061796729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-271187968,"num_objects":-4138,"num_object_clones":0,"num_object_copies":-12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":-454,"num_objects_unfound":0,"num_objects_dirty":-4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":28901376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002974"},"pg_stats":[{"pgid":"2.f","version":"192'10728","reported_seq":11035,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-08-30T23:29:10.642693+0000","last_change":"2024-08-30T23:28:16.218035+0000","last_active":"2024-08-30T23:29:10.642693+0000","last_peered":"2024-08-30T23:29:10.642693+0000","last_clean":"2024-08-30T23:29:10.642693+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:29:10.642693+0000","last_undegraded":"2024-08-30T23:29:10.642693+0000","last_fullsized":"2024-08-30T23:29:10.642693+0000","mapping_epoch":97,"log_start":"109'700","ondisk_log_start":"109'700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'8409","last_scrub_stamp":"2024-08-30T23:28:16.217972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.217972+0000","objects_scrubbed":3813,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:45.741305+0000","scrub_duration":782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96731136,"num_objects":1476,"num_object_clones":0,"num_object_copies":4428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"191'10437","reported_seq":10637,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-08-30T23:29:08.455577+0000","last_change":"2024-08-30T23:25:50.474312+0000","last_active":"2024-08-30T23:29:08.455577+0000","last_peered":"2024-08-30T23:29:08.455577+0000","last_clean":"2024-08-30T23:29:08.455577+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:29:08.455577+0000","last_undegraded":"2024-08-30T23:29:08.455577+0000","last_fullsized":"2024-08-30T23:29:08.455577+0000","mapping_epoch":97,"log_start":"107'400","ondisk_log_start":"107'400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":4503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"192'10808","reported_seq":11051,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-08-30T23:29:10.054154+0000","last_change":"2024-08-30T23:28:27.803716+0000","last_active":"2024-08-30T23:29:10.054154+0000","last_peered":"2024-08-30T23:29:10.054154+0000","last_clean":"2024-08-30T23:29:10.054154+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:29:10.054154+0000","last_undegraded":"2024-08-30T23:29:10.054154+0000","last_fullsized":"2024-08-30T23:29:10.054154+0000","mapping_epoch":97,"log_start":"109'800","ondisk_log_start":"109'800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'9007","last_scrub_stamp":"2024-08-30T23:28:27.803577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.803577+0000","objects_scrubbed":3238,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:55.835178+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209176,"num_objects":1408,"num_object_clones":0,"num_object_copies":4224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1408,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10808,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"192'10607","reported_seq":10851,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-08-30T23:29:11.085605+0000","last_change":"2024-08-30T23:28:27.113440+0000","last_active":"2024-08-30T23:29:11.085605+0000","last_peered":"2024-08-30T23:29:11.085605+0000","last_clean":"2024-08-30T23:29:11.085605+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:29:11.085605+0000","last_undegraded":"2024-08-30T23:29:11.085605+0000","last_fullsized":"2024-08-30T23:29:11.085605+0000","mapping_epoch":97,"log_start":"109'600","ondisk_log_start":"109'600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'8728","last_scrub_stamp":"2024-08-30T23:28:27.113155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.113155+0000","objects_scrubbed":3323,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:28.567110+0000","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92602368,"num_objects":1413,"num_object_clones":0,"num_object_copies":4239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":339,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-08-30T23:29:05.537523+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:05.537523+0000","last_peered":"2024-08-30T23:29:05.537523+0000","last_clean":"2024-08-30T23:29:05.537523+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:05.537523+0000","last_undegraded":"2024-08-30T23:29:05.537523+0000","last_fullsized":"2024-08-30T23:29:05.537523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"191'10764","reported_seq":11009,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-08-30T23:29:08.603539+0000","last_change":"2024-08-30T23:28:16.855547+0000","last_active":"2024-08-30T23:29:08.603539+0000","last_peered":"2024-08-30T23:29:08.603539+0000","last_clean":"2024-08-30T23:29:08.603539+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:29:08.603539+0000","last_undegraded":"2024-08-30T23:29:08.603539+0000","last_fullsized":"2024-08-30T23:29:08.603539+0000","mapping_epoch":97,"log_start":"109'700","ondisk_log_start":"109'700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'8559","last_scrub_stamp":"2024-08-30T23:28:16.855286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.855286+0000","objects_scrubbed":3791,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:23.340082+0000","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102498304,"num_objects":1564,"num_object_clones":0,"num_object_copies":4692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10764,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"190'10572","reported_seq":20954,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-08-30T23:29:07.293126+0000","last_change":"2024-08-30T23:28:43.056625+0000","last_active":"2024-08-30T23:29:07.293126+0000","last_peered":"2024-08-30T23:29:07.293126+0000","last_clean":"2024-08-30T23:29:07.293126+0000","last_became_active":"2024-08-30T23:28:25.514631+0000","last_became_peered":"2024-08-30T23:28:25.514631+0000","last_unstale":"2024-08-30T23:29:07.293126+0000","last_undegraded":"2024-08-30T23:29:07.293126+0000","last_fullsized":"2024-08-30T23:29:07.293126+0000","mapping_epoch":168,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9573","last_scrub_stamp":"2024-08-30T23:28:43.056366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.056366+0000","objects_scrubbed":2652,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:58.660703+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10572,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332136448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"191'10671","reported_seq":10901,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-08-30T23:29:08.533739+0000","last_change":"2024-08-30T23:28:53.329917+0000","last_active":"2024-08-30T23:29:08.533739+0000","last_peered":"2024-08-30T23:29:08.533739+0000","last_clean":"2024-08-30T23:29:08.533739+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:29:08.533739+0000","last_undegraded":"2024-08-30T23:29:08.533739+0000","last_fullsized":"2024-08-30T23:29:08.533739+0000","mapping_epoch":97,"log_start":"109'600","ondisk_log_start":"109'600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9999","last_scrub_stamp":"2024-08-30T23:28:53.329808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:53.329808+0000","objects_scrubbed":2220,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:04.773357+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100466688,"num_objects":1533,"num_object_clones":0,"num_object_copies":4599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10671,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"191'10574","reported_seq":10819,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-08-30T23:29:08.603474+0000","last_change":"2024-08-30T23:28:23.087871+0000","last_active":"2024-08-30T23:29:08.603474+0000","last_peered":"2024-08-30T23:29:08.603474+0000","last_clean":"2024-08-30T23:29:08.603474+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:29:08.603474+0000","last_undegraded":"2024-08-30T23:29:08.603474+0000","last_fullsized":"2024-08-30T23:29:08.603474+0000","mapping_epoch":97,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'8647","last_scrub_stamp":"2024-08-30T23:28:23.087486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:23.087486+0000","objects_scrubbed":3550,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:51.858797+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105512960,"num_objects":1610,"num_object_clones":0,"num_object_copies":4830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10574,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"191'10580","reported_seq":20930,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-08-30T23:29:08.342019+0000","last_change":"2024-08-30T23:28:51.930260+0000","last_active":"2024-08-30T23:29:08.342019+0000","last_peered":"2024-08-30T23:29:08.342019+0000","last_clean":"2024-08-30T23:29:08.342019+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:29:08.342019+0000","last_undegraded":"2024-08-30T23:29:08.342019+0000","last_fullsized":"2024-08-30T23:29:08.342019+0000","mapping_epoch":143,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98435072,"num_objects":1502,"num_object_clones":0,"num_object_copies":4506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10580,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5071,"num_bytes_recovered":332333056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"191'10557","reported_seq":21585,"reported_epoch":191,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:29:08.366840+0000","last_change":"2024-08-30T23:28:47.992578+0000","last_active":"2024-08-30T23:29:08.366840+0000","last_peered":"2024-08-30T23:29:08.366840+0000","last_clean":"2024-08-30T23:28:45.671248+0000","last_became_active":"2024-08-30T23:28:47.905585+0000","last_became_peered":"2024-08-30T23:28:47.905585+0000","last_unstale":"2024-08-30T23:29:08.366840+0000","last_undegraded":"2024-08-30T23:28:47.716401+0000","last_fullsized":"2024-08-30T23:28:47.688373+0000","mapping_epoch":182,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100073472,"num_objects":1527,"num_object_clones":0,"num_object_copies":4581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":216,"num_objects_misplaced":1507,"num_objects_unfound":0,"num_objects_dirty":1527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10557,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5407,"num_bytes_recovered":354287616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1507},{"shards":"0,4","objects":-196}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"192'10703","reported_seq":10975,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-08-30T23:29:10.621953+0000","last_change":"2024-08-30T23:27:43.200813+0000","last_active":"2024-08-30T23:29:10.621953+0000","last_peered":"2024-08-30T23:29:10.621953+0000","last_clean":"2024-08-30T23:29:10.621953+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:29:10.621953+0000","last_undegraded":"2024-08-30T23:29:10.621953+0000","last_fullsized":"2024-08-30T23:29:10.621953+0000","mapping_epoch":142,"log_start":"109'700","ondisk_log_start":"109'700","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'6776","last_scrub_stamp":"2024-08-30T23:27:43.200650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:43.200650+0000","objects_scrubbed":5424,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96141312,"num_objects":1467,"num_object_clones":0,"num_object_copies":4401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10703,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"190'10533","reported_seq":20872,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-08-30T23:29:06.641684+0000","last_change":"2024-08-30T23:28:54.859818+0000","last_active":"2024-08-30T23:29:06.641684+0000","last_peered":"2024-08-30T23:29:06.641684+0000","last_clean":"2024-08-30T23:29:06.641684+0000","last_became_active":"2024-08-30T23:28:47.690562+0000","last_became_peered":"2024-08-30T23:28:47.690562+0000","last_unstale":"2024-08-30T23:29:06.641684+0000","last_undegraded":"2024-08-30T23:29:06.641684+0000","last_fullsized":"2024-08-30T23:29:06.641684+0000","mapping_epoch":182,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":1249,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:10.170623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105447424,"num_objects":1609,"num_object_clones":0,"num_object_copies":4827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10533,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":331415552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"190'10516","reported_seq":10754,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-08-30T23:29:07.198618+0000","last_change":"2024-08-30T23:28:24.690367+0000","last_active":"2024-08-30T23:29:07.198618+0000","last_peered":"2024-08-30T23:29:07.198618+0000","last_clean":"2024-08-30T23:29:07.198618+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:29:07.198618+0000","last_undegraded":"2024-08-30T23:29:07.198618+0000","last_fullsized":"2024-08-30T23:29:07.198618+0000","mapping_epoch":97,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'8760","last_scrub_stamp":"2024-08-30T23:28:24.690182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:24.690182+0000","objects_scrubbed":3297,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:26.848779+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100401152,"num_objects":1532,"num_object_clones":0,"num_object_copies":4596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10516,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"190'10736","reported_seq":10935,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-08-30T23:29:07.510837+0000","last_change":"2024-08-30T23:25:50.474563+0000","last_active":"2024-08-30T23:29:07.510837+0000","last_peered":"2024-08-30T23:29:07.510837+0000","last_clean":"2024-08-30T23:29:07.510837+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:29:07.510837+0000","last_undegraded":"2024-08-30T23:29:07.510837+0000","last_fullsized":"2024-08-30T23:29:07.510837+0000","mapping_epoch":97,"log_start":"109'700","ondisk_log_start":"109'700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101449728,"num_objects":1548,"num_object_clones":0,"num_object_copies":4644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10736,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"192'10456","reported_seq":10692,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-08-30T23:29:11.041475+0000","last_change":"2024-08-30T23:28:39.652116+0000","last_active":"2024-08-30T23:29:11.041475+0000","last_peered":"2024-08-30T23:29:11.041475+0000","last_clean":"2024-08-30T23:29:11.041475+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:29:11.041475+0000","last_undegraded":"2024-08-30T23:29:11.041475+0000","last_fullsized":"2024-08-30T23:29:11.041475+0000","mapping_epoch":97,"log_start":"107'400","ondisk_log_start":"107'400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9191","last_scrub_stamp":"2024-08-30T23:28:39.651956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:39.651956+0000","objects_scrubbed":2655,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.058479+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89653248,"num_objects":1368,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10456,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"191'10529","reported_seq":22067,"reported_epoch":191,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:29:08.607561+0000","last_change":"2024-08-30T23:28:47.980377+0000","last_active":"2024-08-30T23:29:08.607561+0000","last_peered":"2024-08-30T23:29:08.607561+0000","last_clean":"2024-08-30T23:28:45.654798+0000","last_became_active":"2024-08-30T23:28:47.911739+0000","last_became_peered":"2024-08-30T23:28:47.911739+0000","last_unstale":"2024-08-30T23:29:08.607561+0000","last_undegraded":"2024-08-30T23:29:08.607561+0000","last_fullsized":"2024-08-30T23:29:08.607561+0000","mapping_epoch":182,"log_start":"108'500","ondisk_log_start":"108'500","created":97,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9517","last_scrub_stamp":"2024-08-30T23:28:43.578020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.578020+0000","objects_scrubbed":2573,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101384192,"num_objects":1547,"num_object_clones":0,"num_object_copies":4641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1461,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5646,"num_bytes_recovered":369754112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[5,11,7],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"5,7,11","objects":86}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1588592664,"num_objects":24241,"num_object_clones":0,"num_object_copies":72723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":216,"num_objects_misplaced":2968,"num_objects_unfound":0,"num_objects_dirty":24241,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169771,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26250,"num_bytes_recovered":1719992320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5665206272,"data_stored":5665194056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160671,"ondisk_log_size":160671,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187512,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472856,"kb_used_data":365320,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93272296,"statfs":{"total":95995035648,"available":95510831104,"internally_reserved":0,"allocated":374087680,"data_stored":373674320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449150,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710786,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671384,"kb_used_data":563844,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93073768,"statfs":{"total":95995035648,"available":95307538432,"internally_reserved":0,"allocated":577376256,"data_stored":576962396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972424,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":272340,"kb_used_data":200896,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93472812,"statfs":{"total":95995035648,"available":95716159488,"internally_reserved":0,"allocated":205717504,"data_stored":205307716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234058,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":628416,"kb_used_data":512560,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93116736,"statfs":{"total":95995035648,"available":95351537664,"internally_reserved":0,"allocated":524861440,"data_stored":524456956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":118618570},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528400,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":264840,"kb_used_data":215600,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93480312,"statfs":{"total":95995035648,"available":95723839488,"internally_reserved":0,"allocated":220774400,"data_stored":220369916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724628,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384648,"kb_used_data":309044,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93360504,"statfs":{"total":95995035648,"available":95601156096,"internally_reserved":0,"allocated":316461056,"data_stored":316055671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953562,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361228,"kb_used_data":285624,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93383924,"statfs":{"total":95995035648,"available":95625138176,"internally_reserved":0,"allocated":292478976,"data_stored":292069519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789960,"kb_used_data":682360,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92955192,"statfs":{"total":95995035648,"available":95186116608,"internally_reserved":0,"allocated":698736640,"data_stored":698327183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610120,"kb_used_data":502580,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93135032,"statfs":{"total":95995035648,"available":95370272768,"internally_reserved":0,"allocated":514641920,"data_stored":514236535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":775868,"kb_used_data":664168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92969284,"statfs":{"total":95995035648,"available":95200546816,"internally_reserved":0,"allocated":680108032,"data_stored":679707100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605960,"kb_used_data":498420,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93139192,"statfs":{"total":95995035648,"available":95374532608,"internally_reserved":0,"allocated":510382080,"data_stored":509976695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":497811456,"data_stored":497811456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":676528128,"data_stored":676528128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":501219328,"data_stored":501219328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":693178368,"data_stored":693174296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":290328576,"data_stored":290324504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":309329920,"data_stored":309329920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":215810048,"data_stored":215810048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":519962624,"data_stored":519962624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":203358208,"data_stored":203358208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":567152640,"data_stored":567148568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":819134464,"data_stored":819134464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":371392512,"data_stored":371392512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:29:14.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:14.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:14.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:14.661 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:14 smithi106 ceph-mon[16603]: 2.e scrub ok 2024-08-30T23:29:14.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:14.731 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:14.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:14.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:14.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:14.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:14.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:14.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:14 smithi136 ceph-mon[20207]: 2.e scrub ok 2024-08-30T23:29:15.067 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:15.067 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:14 smithi173 ceph-mon[20554]: 2.e scrub ok 2024-08-30T23:29:15.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:15.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:15.113 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:15.119 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:15.151 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:15.151 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:15.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:15.188 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:15.217 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:15.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:15.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:15.250 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:15.250 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:15.251 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:15.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:15.284 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:15.315 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:15.316 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:15.325 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.14157.0:52 1.0 e22/20 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'34, mlcod=22'29)", 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 482.93864654700002, 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33784646499999998, 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.326 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977574+0000", 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7100000000000002e-07 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977579+0000", 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1080000000000001e-06 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.314904+0000", 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33732482499999999 2024-08-30T23:29:15.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.314915+0000", 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1576000000000001e-05 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315011+0000", 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5877000000000004e-05 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315031+0000", 2024-08-30T23:29:15.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9718e-05 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315389+0000", 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000358184 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315420+0000", 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0605999999999998e-05 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.330 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14225 2.5 e109/97 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head v 109'898, mlcod=109'894)", 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.042209496, 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.239755616, 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874011+0000", 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874108+0000", 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6961999999999998e-05 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112976+0000", 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23886803600000001 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112983+0000", 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.951e-06 2024-08-30T23:29:15.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113055+0000", 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113079+0000", 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4754000000000001e-05 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113724+0000", 2024-08-30T23:29:15.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064475700000000003 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113766+0000", 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1664e-05 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16156 2.4 e109/97 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head v 109'956, mlcod=109'953)", 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 184.81812880000001, 2024-08-30T23:29:15.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213651339, 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:29:15.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098092+0000", 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4300000000000002e-07 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098114+0000", 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2234000000000001e-05 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310963+0000", 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21284856199999999 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310969+0000", 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311028+0000", 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8857999999999998e-05 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311051+0000", 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2538999999999999e-05 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311702+0000", 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065106500000000004 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311742+0000", 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0318000000000003e-05 2024-08-30T23:29:15.339 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 138.287007039, 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207325907, 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629214+0000", 2024-08-30T23:29:15.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.294e-06 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629241+0000", 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6767999999999999e-05 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834158+0000", 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20491721900000001 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834164+0000", 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.268e-06 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834227+0000", 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2712000000000005e-05 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2120000000000002e-05 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836507+0000", 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022580769999999998 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836539+0000", 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1449000000000003e-05 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75428 2.5 2:af5d866e:::benchmark_data_smithi173_47195_object75427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.63993706299999, 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.189894692, 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75428 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.345 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276283+0000", 2024-08-30T23:29:15.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9100000000000004e-07 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276309+0000", 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6095e-05 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276317+0000", 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7379999999999994e-06 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276323+0000", 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4280000000000001e-06 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277033+0000", 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070927499999999999 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277036+0000", 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7500000000000001e-06 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417234+0000", 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.140198079 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417328+0000", 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3068999999999999e-05 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.417423+0000", 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5353999999999995e-05 2024-08-30T23:29:15.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421154+0000", 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037309819999999999 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421166+0000", 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2213e-05 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.465172+0000", 2024-08-30T23:29:15.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.044005711000000003 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.465209+0000", 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6876999999999999e-05 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.466177+0000", 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00096862999999999995 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75451 2.5 2:ac73061a:::benchmark_data_smithi173_47195_object75450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:58.276330+0000", 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.63988939999999, 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.190842071, 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75451 2024-08-30T23:29:15.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276330+0000", 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276330+0000", 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276331+0000", 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3599999999999999e-07 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276355+0000", 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.419e-05 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276363+0000", 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2349999999999999e-06 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.276369+0000", 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3400000000000003e-06 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277061+0000", 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069180900000000004 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.277070+0000", 2024-08-30T23:29:15.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2220000000000002e-06 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.419135+0000", 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14206534200000001 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.419271+0000", 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013651000000000001 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.419420+0000", 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000148851 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.421212+0000", 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017922249999999999 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.431930+0000", 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010717844000000001 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.466225+0000", 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034295063000000001 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.466261+0000", 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5478999999999997e-05 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:58.467172+0000", 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00091152500000000005 2024-08-30T23:29:15.358 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 106.418454178, 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:29:15.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109556 2.5 2:a6b86250:::benchmark_data_smithi173_47195_object12548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.061871193000002, 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220264298, 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109556 2024-08-30T23:29:15.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854350+0000", 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8840000000000001e-06 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854356+0000", 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6590000000000004e-06 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854369+0000", 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.289e-05 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854377+0000", 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4649999999999994e-06 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854454+0000", 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6791000000000004e-05 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854600+0000", 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145864 2024-08-30T23:29:15.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854751+0000", 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015046499999999999 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.857557+0000", 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028060419999999999 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.873657+0000", 2024-08-30T23:29:15.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016100032 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073675+0000", 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200018474 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073721+0000", 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6072999999999997e-05 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.074613+0000", 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089165900000000005 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109564 2.4 2:242bfffa:::benchmark_data_smithi173_47195_object12556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.054427172999993, 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220047624, 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.369 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109564 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861794+0000", 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5889999999999999e-06 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861800+0000", 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861809+0000", 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999997e-06 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861815+0000", 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861910+0000", 2024-08-30T23:29:15.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5278000000000005e-05 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862048+0000", 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137911 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862173+0000", 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012423299999999999 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878997+0000", 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016824741000000001 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879048+0000", 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0890999999999998e-05 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080272+0000", 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20122369600000001 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080313+0000", 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0785999999999999e-05 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.081840+0000", 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015274679999999999 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.375 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109567 2.4 2:235a232e:::benchmark_data_smithi173_47195_object12559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.049372348000006, 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218097968, 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109567 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866849+0000", 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:29:15.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866855+0000", 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1199999999999999e-06 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866877+0000", 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1954e-05 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866885+0000", 2024-08-30T23:29:15.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9070000000000007e-06 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869749+0000", 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028635940000000001 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869826+0000", 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7583999999999994e-05 2024-08-30T23:29:15.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869907+0000", 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0408999999999995e-05 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879006+0000", 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090992710000000008 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883826+0000", 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048203719999999999 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083592+0000", 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.199765677 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083632+0000", 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0262000000000002e-05 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.084945+0000", 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013130590000000001 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109569 2.5 2:a8435ff8:::benchmark_data_smithi173_47195_object12561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.046729995000007, 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22083613699999999, 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.382 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109569 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869491+0000", 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4270000000000001e-06 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869498+0000", 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:29:15.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869510+0000", 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1630999999999999e-05 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869519+0000", 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869586+0000", 2024-08-30T23:29:15.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6699999999999995e-05 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869741+0000", 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155141 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869888+0000", 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014657099999999999 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879878+0000", 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099902989999999994 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879907+0000", 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8612999999999999e-05 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089168+0000", 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20926119900000001 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089267+0000", 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9097000000000003e-05 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090326+0000", 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010589670000000001 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109574 2.5 2:aaf91c06:::benchmark_data_smithi173_47195_object12566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.040164172000004, 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21517555899999999, 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109574 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.389 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876058+0000", 2024-08-30T23:29:15.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.046e-06 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876065+0000", 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876084+0000", 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8702000000000002e-05 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876094+0000", 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5850000000000006e-06 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876163+0000", 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9441000000000001e-05 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876332+0000", 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016932699999999999 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876473+0000", 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014066600000000001 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882795+0000", 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063221149999999997 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883230+0000", 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043490299999999999 2024-08-30T23:29:15.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090374+0000", 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20714431699999999 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090425+0000", 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0865000000000002e-05 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091231+0000", 2024-08-30T23:29:15.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00080588100000000005 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109576 2.5 2:ae1552f5:::benchmark_data_smithi173_47195_object12568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.037720356999998, 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21498098400000001, 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.395 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109576 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:29:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878501+0000", 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.751e-06 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878508+0000", 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8340000000000001e-06 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878521+0000", 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669e-05 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878529+0000", 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.055e-06 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880019+0000", 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001490686 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880144+0000", 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012418800000000001 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880266+0000", 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000122552 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882939+0000", 2024-08-30T23:29:15.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026733849999999999 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.917160+0000", 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034220360999999998 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092559+0000", 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.175398734 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092623+0000", 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4208999999999995e-05 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.093480+0000", 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00085756000000000005 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109577 2.4 2:26051de8:::benchmark_data_smithi173_47195_object12569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:29:15.401 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.036749630000003, 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20693977799999999, 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109577 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879471+0000", 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.077e-06 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879476+0000", 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8289999999999998e-06 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879488+0000", 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1883e-05 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879497+0000", 2024-08-30T23:29:15.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6570000000000003e-06 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879535+0000", 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8574e-05 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879631+0000", 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5890999999999994e-05 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879719+0000", 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8331999999999994e-05 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882451+0000", 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027319240000000002 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.893310+0000", 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01085908 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.085013+0000", 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19170245499999999 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.085071+0000", 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7884999999999999e-05 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.086410+0000", 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001339191 2024-08-30T23:29:15.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144821 2.4 2:2415a84c:::benchmark_data_smithi173_47195_object47813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.352836019999998, 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23107453999999999, 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144821 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563385+0000", 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.59e-06 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563391+0000", 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563404+0000", 2024-08-30T23:29:15.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2982e-05 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563411+0000", 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565303+0000", 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001891846 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565431+0000", 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012746400000000001 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565550+0000", 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972900000000001 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.573426+0000", 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078756799999999995 2024-08-30T23:29:15.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588851+0000", 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015424679 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793322+0000", 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20447127600000001 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793387+0000", 2024-08-30T23:29:15.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4586999999999997e-05 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.794458+0000", 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010715060000000001 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144836 2.4 2:2d7ce86c:::benchmark_data_smithi173_47195_object47828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.328102653999998, 2024-08-30T23:29:15.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23589892500000001, 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144836 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588118+0000", 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3829999999999999e-06 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588123+0000", 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588169+0000", 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199999999999998e-05 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588182+0000", 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2748999999999999e-05 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.592926+0000", 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047443779999999996 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595020+0000", 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002094243 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595429+0000", 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040842299999999999 2024-08-30T23:29:15.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603861+0000", 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0084324050000000005 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.604206+0000", 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034452400000000002 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.822951+0000", 2024-08-30T23:29:15.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21874496900000001 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.823010+0000", 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9601000000000001e-05 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824016+0000", 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010057499999999999 2024-08-30T23:29:15.420 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144863 2.4 2:2d698656:::benchmark_data_smithi173_47195_object47855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.624898+0000", 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.291321431999997, 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19982124700000001, 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.421 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144863 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.624898+0000", 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.624898+0000", 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.624901+0000", 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6910000000000002e-06 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.624911+0000", 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8409999999999995e-06 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.624939+0000", 2024-08-30T23:29:15.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8167999999999999e-05 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.624950+0000", 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1377e-05 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.625073+0000", 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000122204 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.625285+0000", 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00021283200000000001 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.625410+0000", 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000124492 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.630362+0000", 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049525719999999997 2024-08-30T23:29:15.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.632760+0000", 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023971019999999999 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824082+0000", 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19132211800000001 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824155+0000", 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3102000000000006e-05 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824720+0000", 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056474800000000001 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144870 2.4 2:22133d0e:::benchmark_data_smithi173_47195_object47862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.633904+0000", 2024-08-30T23:29:15.427 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.282316117000001, 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.191385112, 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144870 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.633904+0000", 2024-08-30T23:29:15.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.633904+0000", 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.633906+0000", 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.018e-06 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.633912+0000", 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5690000000000001e-06 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.633929+0000", 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6659000000000001e-05 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.633937+0000", 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4100000000000008e-06 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.634015+0000", 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7855999999999995e-05 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.634161+0000", 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014545899999999999 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.634295+0000", 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013486999999999999 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.638160+0000", 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003864503 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.647557+0000", 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0093972299999999995 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824773+0000", 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17721589500000001 2024-08-30T23:29:15.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824843+0000", 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9826000000000005e-05 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.825289+0000", 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00044581699999999999 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144871 2.4 2:2107111c:::benchmark_data_smithi173_47195_object47863:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.636689+0000", 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.279530244999997, 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18919949999999999, 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144871 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.636689+0000", 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.636689+0000", 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.636691+0000", 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.592e-06 2024-08-30T23:29:15.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.636699+0000", 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0569999999999993e-06 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.636710+0000", 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0434e-05 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.436 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.636715+0000", 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.908e-06 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.639009+0000", 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022934000000000001 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.639308+0000", 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029868400000000002 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.437 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.639627+0000", 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000319489 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.645049+0000", 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0054220550000000003 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.647912+0000", 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028628849999999999 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.438 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.825338+0000", 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17742579 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.825389+0000", 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1483999999999997e-05 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.825889+0000", 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049972199999999997 2024-08-30T23:29:15.439 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155204 2.4 2:2613b3a5:::benchmark_data_smithi173_47195_object58196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 29.139821778000002, 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21707409, 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:15.440 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155204 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776400+0000", 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776406+0000", 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1890000000000004e-06 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776421+0000", 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449999999999999e-05 2024-08-30T23:29:15.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776430+0000", 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0420000000000008e-06 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776499+0000", 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776643+0000", 2024-08-30T23:29:15.443 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000144721 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776820+0000", 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017706199999999999 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.783777+0000", 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006956322 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.444 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.786557+0000", 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027807270000000002 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992894+0000", 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20633659600000001 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992950+0000", 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5664000000000001e-05 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:15.445 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.993472+0000", 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052240299999999995 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:15.446 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:29:15.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:15.447 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:15.448 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:15.449 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:15.449 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:15.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:15.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:15.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:15.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:15.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:29:15.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:15.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/16613234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:15.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:15.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:15.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:15 smithi106 ceph-mon[16603]: pgmap v495: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 231/64641 objects degraded (0.357%); 2453/64641 objects misplaced (3.795%); 7.5 MiB/s, 119 objects/s recovering 2024-08-30T23:29:15.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 231/68760 objects degraded (0.336%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:15.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:15 smithi106 ceph-mon[16603]: osdmap e194: 12 total, 12 up, 11 in 2024-08-30T23:29:15.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:15.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:15.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:15.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:15.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/16613234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:15 smithi136 ceph-mon[20207]: pgmap v495: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 231/64641 objects degraded (0.357%); 2453/64641 objects misplaced (3.795%); 7.5 MiB/s, 119 objects/s recovering 2024-08-30T23:29:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 231/68760 objects degraded (0.336%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:15 smithi136 ceph-mon[20207]: osdmap e194: 12 total, 12 up, 11 in 2024-08-30T23:29:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/16613234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:15 smithi173 ceph-mon[20554]: pgmap v495: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 231/64641 objects degraded (0.357%); 2453/64641 objects misplaced (3.795%); 7.5 MiB/s, 119 objects/s recovering 2024-08-30T23:29:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 231/68760 objects degraded (0.336%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:15 smithi173 ceph-mon[20554]: osdmap e194: 12 total, 12 up, 11 in 2024-08-30T23:29:16.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:16.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:16.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:16.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:16.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:16.374 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:16.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:16.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:16.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:16.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:16.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:16.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:16.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:16.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:16.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:16.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:17.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:17.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:17.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.132 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:17.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:17.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:17.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:17.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:29:17.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:17.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:17.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:17.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:17.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:17.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:17.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:17.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:17.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:17.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:18.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:18.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:18.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:18.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:18.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:18.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:18.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:18.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:18.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:18.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:18.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:18.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:18.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:18.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:18 smithi136 ceph-mon[20207]: pgmap v497: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 231/60192 objects degraded (0.384%); 2453/60192 objects misplaced (4.075%); 4.0 MiB/s, 63 objects/s recovering 2024-08-30T23:29:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:18 smithi173 ceph-mon[20554]: pgmap v497: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 231/60192 objects degraded (0.384%); 2453/60192 objects misplaced (4.075%); 4.0 MiB/s, 63 objects/s recovering 2024-08-30T23:29:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:18 smithi106 ceph-mon[16603]: pgmap v497: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.2 GiB data, 4.8 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 231/60192 objects degraded (0.384%); 2453/60192 objects misplaced (4.075%); 4.0 MiB/s, 63 objects/s recovering 2024-08-30T23:29:18.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:18.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:18.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:19.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:19.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:19.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:19.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:19.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:19.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:19.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:19.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:19.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:19.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.581 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:19.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:19.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:19.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:19.812 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:19.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:19.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:19.974 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:19.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:20.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:20.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:20.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:20.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:20.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:20.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:20.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:20.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:20.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:20.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:20.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:20.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:20 smithi136 ceph-mon[20207]: pgmap v498: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 244/55353 objects degraded (0.441%); 1947/55353 objects misplaced (3.517%); 7.8 MiB/s, 124 objects/s recovering 2024-08-30T23:29:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:20 smithi173 ceph-mon[20554]: pgmap v498: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 244/55353 objects degraded (0.441%); 1947/55353 objects misplaced (3.517%); 7.8 MiB/s, 124 objects/s recovering 2024-08-30T23:29:20.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:20 smithi106 ceph-mon[16603]: pgmap v498: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 244/55353 objects degraded (0.441%); 1947/55353 objects misplaced (3.517%); 7.8 MiB/s, 124 objects/s recovering 2024-08-30T23:29:20.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:20.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:20.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:21.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:21.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:21.143 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:21.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:29:21.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:21.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:21.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:21.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:21.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:21.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:21.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:21.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:21.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:21.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 244/55353 objects degraded (0.441%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 244/55353 objects degraded (0.441%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 244/55353 objects degraded (0.441%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:21.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:21.917 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:21.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:22.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:22.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.046 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:22.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:22.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.178 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:22.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:22.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:22.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:22.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:22.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:22.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:22.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:22 smithi136 ceph-mon[20207]: pgmap v499: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 244/52236 objects degraded (0.467%); 1947/52236 objects misplaced (3.727%); 7.1 MiB/s, 112 objects/s recovering 2024-08-30T23:29:22.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:22.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:22 smithi173 ceph-mon[20554]: pgmap v499: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 244/52236 objects degraded (0.467%); 1947/52236 objects misplaced (3.727%); 7.1 MiB/s, 112 objects/s recovering 2024-08-30T23:29:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:22 smithi106 ceph-mon[16603]: pgmap v499: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.1 GiB data, 4.3 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 244/52236 objects degraded (0.467%); 1947/52236 objects misplaced (3.727%); 7.1 MiB/s, 112 objects/s recovering 2024-08-30T23:29:22.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:22.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:22.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:23.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:23.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:23.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:23.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:23.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:23.308 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:23.322 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:23.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:23.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:23.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:23.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:23.612 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:23.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:23.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:23.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:23.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:23.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:23.997 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:23.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:24.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:24.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:24.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:24.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:24.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:24.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:24.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:24.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:24.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:24.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:24.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:24.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:24.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:24.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:24.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:24.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:24 smithi106 ceph-mon[16603]: pgmap v500: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.0 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 244/49629 objects degraded (0.492%); 1947/49629 objects misplaced (3.923%); 4.9 MiB/s, 78 objects/s recovering 2024-08-30T23:29:24.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:24 smithi136 ceph-mon[20207]: pgmap v500: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.0 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 244/49629 objects degraded (0.492%); 1947/49629 objects misplaced (3.923%); 4.9 MiB/s, 78 objects/s recovering 2024-08-30T23:29:24.800 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:24.879 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:24 smithi173 ceph-mon[20554]: pgmap v500: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.0 GiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 244/49629 objects degraded (0.492%); 1947/49629 objects misplaced (3.923%); 4.9 MiB/s, 78 objects/s recovering 2024-08-30T23:29:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:24.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:24.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:25.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:25.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:25.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:25.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:25.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:25.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:25.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:25.442 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:25.442 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:25.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.524 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:25.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:25.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:25.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:25.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:25.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:25.810 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:25.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:25.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:25.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:26.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.030 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:26.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:26.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:26.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:26.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:26.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:26.291 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-08-30T23:29:25.392751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":962134584,"num_objects":14675,"num_object_clones":0,"num_object_copies":44025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":1431,"num_objects_unfound":0,"num_objects_dirty":14675,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":179462,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27765,"num_bytes_recovered":1820472928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160831,"ondisk_log_size":160831,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":3998008,"kb_used_data":2949768,"kb_used_omap":17,"kb_used_meta":1048046,"kb_avail":1027198664,"statfs":{"total":1055945392128,"available":1051851431936,"internally_reserved":0,"allocated":3020562432,"data_stored":3017189974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1073199993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450363392,"num_objects":-6872,"num_object_clones":0,"num_object_copies":-20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":-1022,"num_objects_unfound":0,"num_objects_dirty":-6872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1009,"num_bytes_recovered":66125824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004831"},"pg_stats":[{"pgid":"2.f","version":"194'11134","reported_seq":11443,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:20.638656+0000","last_change":"2024-08-30T23:28:16.218035+0000","last_active":"2024-08-30T23:29:20.638656+0000","last_peered":"2024-08-30T23:29:20.638656+0000","last_clean":"2024-08-30T23:29:20.638656+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:29:20.638656+0000","last_undegraded":"2024-08-30T23:29:20.638656+0000","last_fullsized":"2024-08-30T23:29:20.638656+0000","mapping_epoch":97,"log_start":"111'1100","ondisk_log_start":"111'1100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'8409","last_scrub_stamp":"2024-08-30T23:28:16.217972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.217972+0000","objects_scrubbed":3813,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:45.741305+0000","scrub_duration":782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70123520,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"194'11087","reported_seq":11316,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:23.544370+0000","last_change":"2024-08-30T23:29:13.368781+0000","last_active":"2024-08-30T23:29:23.544370+0000","last_peered":"2024-08-30T23:29:23.544370+0000","last_clean":"2024-08-30T23:29:23.544370+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:29:23.544370+0000","last_undegraded":"2024-08-30T23:29:23.544370+0000","last_fullsized":"2024-08-30T23:29:23.544370+0000","mapping_epoch":97,"log_start":"111'1000","ondisk_log_start":"111'1000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'10663","last_scrub_stamp":"2024-08-30T23:29:13.368419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:13.368419+0000","objects_scrubbed":1283,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:14.410101+0000","scrub_duration":729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"194'11441","reported_seq":11689,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:25.027491+0000","last_change":"2024-08-30T23:28:27.803716+0000","last_active":"2024-08-30T23:29:25.027491+0000","last_peered":"2024-08-30T23:29:25.027491+0000","last_clean":"2024-08-30T23:29:25.027491+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:29:25.027491+0000","last_undegraded":"2024-08-30T23:29:25.027491+0000","last_fullsized":"2024-08-30T23:29:25.027491+0000","mapping_epoch":97,"log_start":"112'1400","ondisk_log_start":"112'1400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'9007","last_scrub_stamp":"2024-08-30T23:28:27.803577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.803577+0000","objects_scrubbed":3238,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:55.835178+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50724888,"num_objects":775,"num_object_clones":0,"num_object_copies":2325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11441,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"194'11041","reported_seq":11289,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:21.088316+0000","last_change":"2024-08-30T23:28:27.113440+0000","last_active":"2024-08-30T23:29:21.088316+0000","last_peered":"2024-08-30T23:29:21.088316+0000","last_clean":"2024-08-30T23:29:21.088316+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:29:21.088316+0000","last_undegraded":"2024-08-30T23:29:21.088316+0000","last_fullsized":"2024-08-30T23:29:21.088316+0000","mapping_epoch":97,"log_start":"109'1000","ondisk_log_start":"109'1000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'8728","last_scrub_stamp":"2024-08-30T23:28:27.113155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.113155+0000","objects_scrubbed":3323,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:28.567110+0000","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64159744,"num_objects":979,"num_object_clones":0,"num_object_copies":2937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":348,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:15.541566+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:15.541566+0000","last_peered":"2024-08-30T23:29:15.541566+0000","last_clean":"2024-08-30T23:29:15.541566+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:15.541566+0000","last_undegraded":"2024-08-30T23:29:15.541566+0000","last_fullsized":"2024-08-30T23:29:15.541566+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"194'11453","reported_seq":11704,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:23.593587+0000","last_change":"2024-08-30T23:28:16.855547+0000","last_active":"2024-08-30T23:29:23.593587+0000","last_peered":"2024-08-30T23:29:23.593587+0000","last_clean":"2024-08-30T23:29:23.593587+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:29:23.593587+0000","last_undegraded":"2024-08-30T23:29:23.593587+0000","last_fullsized":"2024-08-30T23:29:23.593587+0000","mapping_epoch":97,"log_start":"112'1400","ondisk_log_start":"112'1400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'8559","last_scrub_stamp":"2024-08-30T23:28:16.855286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.855286+0000","objects_scrubbed":3791,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11453,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"194'11250","reported_seq":21639,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:22.301734+0000","last_change":"2024-08-30T23:28:43.056625+0000","last_active":"2024-08-30T23:29:22.301734+0000","last_peered":"2024-08-30T23:29:22.301734+0000","last_clean":"2024-08-30T23:29:22.301734+0000","last_became_active":"2024-08-30T23:28:25.514631+0000","last_became_peered":"2024-08-30T23:28:25.514631+0000","last_unstale":"2024-08-30T23:29:22.301734+0000","last_undegraded":"2024-08-30T23:29:22.301734+0000","last_fullsized":"2024-08-30T23:29:22.301734+0000","mapping_epoch":168,"log_start":"111'1200","ondisk_log_start":"111'1200","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9573","last_scrub_stamp":"2024-08-30T23:28:43.056366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.056366+0000","objects_scrubbed":2652,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:58.660703+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62783488,"num_objects":958,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332136448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"194'11323","reported_seq":11557,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:23.534775+0000","last_change":"2024-08-30T23:28:53.329917+0000","last_active":"2024-08-30T23:29:23.534775+0000","last_peered":"2024-08-30T23:29:23.534775+0000","last_clean":"2024-08-30T23:29:23.534775+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:29:23.534775+0000","last_undegraded":"2024-08-30T23:29:23.534775+0000","last_fullsized":"2024-08-30T23:29:23.534775+0000","mapping_epoch":97,"log_start":"112'1300","ondisk_log_start":"112'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9999","last_scrub_stamp":"2024-08-30T23:28:53.329808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:53.329808+0000","objects_scrubbed":2220,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:04.773357+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57737216,"num_objects":881,"num_object_clones":0,"num_object_copies":2643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11323,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"194'11285","reported_seq":11536,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:23.606785+0000","last_change":"2024-08-30T23:28:23.087871+0000","last_active":"2024-08-30T23:29:23.606785+0000","last_peered":"2024-08-30T23:29:23.606785+0000","last_clean":"2024-08-30T23:29:23.606785+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:29:23.606785+0000","last_undegraded":"2024-08-30T23:29:23.606785+0000","last_fullsized":"2024-08-30T23:29:23.606785+0000","mapping_epoch":97,"log_start":"112'1200","ondisk_log_start":"112'1200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'8647","last_scrub_stamp":"2024-08-30T23:28:23.087486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:23.087486+0000","objects_scrubbed":3550,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:51.858797+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58916864,"num_objects":899,"num_object_clones":0,"num_object_copies":2697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11285,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"194'11229","reported_seq":21584,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:23.375094+0000","last_change":"2024-08-30T23:28:51.930260+0000","last_active":"2024-08-30T23:29:23.375094+0000","last_peered":"2024-08-30T23:29:23.375094+0000","last_clean":"2024-08-30T23:29:23.375094+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:29:23.375094+0000","last_undegraded":"2024-08-30T23:29:23.375094+0000","last_fullsized":"2024-08-30T23:29:23.375094+0000","mapping_epoch":143,"log_start":"112'1200","ondisk_log_start":"112'1200","created":97,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55902208,"num_objects":853,"num_object_clones":0,"num_object_copies":2559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5071,"num_bytes_recovered":332333056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"194'11232","reported_seq":23540,"reported_epoch":194,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:29:23.373152+0000","last_change":"2024-08-30T23:28:47.992578+0000","last_active":"2024-08-30T23:29:23.373152+0000","last_peered":"2024-08-30T23:29:23.373152+0000","last_clean":"2024-08-30T23:28:45.671248+0000","last_became_active":"2024-08-30T23:28:47.905585+0000","last_became_peered":"2024-08-30T23:28:47.905585+0000","last_unstale":"2024-08-30T23:29:23.373152+0000","last_undegraded":"2024-08-30T23:28:47.716401+0000","last_fullsized":"2024-08-30T23:28:47.688373+0000","mapping_epoch":182,"log_start":"112'1200","ondisk_log_start":"112'1200","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55836672,"num_objects":852,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":841,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11232,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6045,"num_bytes_recovered":396099584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":841},{"shards":"0,4","objects":-233}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"194'11166","reported_seq":11442,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:20.654130+0000","last_change":"2024-08-30T23:27:43.200813+0000","last_active":"2024-08-30T23:29:20.654130+0000","last_peered":"2024-08-30T23:29:20.654130+0000","last_clean":"2024-08-30T23:29:20.654130+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:29:20.654130+0000","last_undegraded":"2024-08-30T23:29:20.654130+0000","last_fullsized":"2024-08-30T23:29:20.654130+0000","mapping_epoch":142,"log_start":"111'1100","ondisk_log_start":"111'1100","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'6776","last_scrub_stamp":"2024-08-30T23:27:43.200650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:43.200650+0000","objects_scrubbed":5424,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65798144,"num_objects":1004,"num_object_clones":0,"num_object_copies":3012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11166,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"194'11176","reported_seq":21547,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:21.811792+0000","last_change":"2024-08-30T23:29:11.641650+0000","last_active":"2024-08-30T23:29:21.811792+0000","last_peered":"2024-08-30T23:29:21.811792+0000","last_clean":"2024-08-30T23:29:21.811792+0000","last_became_active":"2024-08-30T23:28:47.690562+0000","last_became_peered":"2024-08-30T23:28:47.690562+0000","last_unstale":"2024-08-30T23:29:21.811792+0000","last_undegraded":"2024-08-30T23:29:21.811792+0000","last_fullsized":"2024-08-30T23:29:21.811792+0000","mapping_epoch":182,"log_start":"111'1100","ondisk_log_start":"111'1100","created":97,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10738","last_scrub_stamp":"2024-08-30T23:29:11.641462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:11.641462+0000","objects_scrubbed":1414,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:34.059229+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11176,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":331415552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"194'11108","reported_seq":11351,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:22.326495+0000","last_change":"2024-08-30T23:28:24.690367+0000","last_active":"2024-08-30T23:29:22.326495+0000","last_peered":"2024-08-30T23:29:22.326495+0000","last_clean":"2024-08-30T23:29:22.326495+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:29:22.326495+0000","last_undegraded":"2024-08-30T23:29:22.326495+0000","last_fullsized":"2024-08-30T23:29:22.326495+0000","mapping_epoch":97,"log_start":"111'1100","ondisk_log_start":"111'1100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'8760","last_scrub_stamp":"2024-08-30T23:28:24.690182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:24.690182+0000","objects_scrubbed":3297,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:26.848779+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11108,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"194'11348","reported_seq":11576,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:22.558142+0000","last_change":"2024-08-30T23:29:10.813505+0000","last_active":"2024-08-30T23:29:22.558142+0000","last_peered":"2024-08-30T23:29:22.558142+0000","last_clean":"2024-08-30T23:29:22.558142+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:29:22.558142+0000","last_undegraded":"2024-08-30T23:29:22.558142+0000","last_fullsized":"2024-08-30T23:29:22.558142+0000","mapping_epoch":97,"log_start":"112'1300","ondisk_log_start":"112'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10887","last_scrub_stamp":"2024-08-30T23:29:10.813301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:10.813301+0000","objects_scrubbed":1415,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:27.781500+0000","scrub_duration":591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61341696,"num_objects":936,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11348,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"194'10856","reported_seq":11095,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-08-30T23:29:21.093148+0000","last_change":"2024-08-30T23:28:39.652116+0000","last_active":"2024-08-30T23:29:21.093148+0000","last_peered":"2024-08-30T23:29:21.093148+0000","last_clean":"2024-08-30T23:29:21.093148+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:29:21.093148+0000","last_undegraded":"2024-08-30T23:29:21.093148+0000","last_fullsized":"2024-08-30T23:29:21.093148+0000","mapping_epoch":97,"log_start":"109'800","ondisk_log_start":"109'800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9191","last_scrub_stamp":"2024-08-30T23:28:39.651956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:39.651956+0000","objects_scrubbed":2655,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.058479+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438848,"num_objects":968,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10856,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"194'11210","reported_seq":24493,"reported_epoch":194,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:29:23.615431+0000","last_change":"2024-08-30T23:28:47.980377+0000","last_active":"2024-08-30T23:29:23.615431+0000","last_peered":"2024-08-30T23:29:23.615431+0000","last_clean":"2024-08-30T23:28:45.654798+0000","last_became_active":"2024-08-30T23:28:47.911739+0000","last_became_peered":"2024-08-30T23:28:47.911739+0000","last_unstale":"2024-08-30T23:29:23.615431+0000","last_undegraded":"2024-08-30T23:29:23.615431+0000","last_fullsized":"2024-08-30T23:29:23.615431+0000","mapping_epoch":182,"log_start":"112'1200","ondisk_log_start":"112'1200","created":97,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9517","last_scrub_stamp":"2024-08-30T23:28:43.578020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.578020+0000","objects_scrubbed":2573,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":590,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11210,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6517,"num_bytes_recovered":426835968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[5,11,7],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"5,7,11","objects":276}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":961544216,"num_objects":14673,"num_object_clones":0,"num_object_copies":44019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":244,"num_objects_misplaced":1431,"num_objects_unfound":0,"num_objects_dirty":14673,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179339,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27759,"num_bytes_recovered":1818886144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3691851776,"data_stored":3691839560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160739,"ondisk_log_size":160739,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187514,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367876,"kb_used_data":256180,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93377276,"statfs":{"total":95995035648,"available":95618330624,"internally_reserved":0,"allocated":262328320,"data_stored":262020242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710789,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":428544,"kb_used_data":321008,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93316608,"statfs":{"total":95995035648,"available":95556206592,"internally_reserved":0,"allocated":328712192,"data_stored":328403614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972426,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":221372,"kb_used_data":154092,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93523780,"statfs":{"total":95995035648,"available":95768350720,"internally_reserved":0,"allocated":157790208,"data_stored":157485702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234061,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":423812,"kb_used_data":312116,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93321340,"statfs":{"total":95995035648,"available":95561052160,"internally_reserved":0,"allocated":319606784,"data_stored":319298726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528403,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182148,"kb_used_data":128756,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93563004,"statfs":{"total":95995035648,"available":95808516096,"internally_reserved":0,"allocated":131846144,"data_stored":131538086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":244284,"kb_used_data":177004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93500868,"statfs":{"total":95995035648,"available":95744888832,"internally_reserved":0,"allocated":181252096,"data_stored":180947590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953565,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":229760,"kb_used_data":162480,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93515392,"statfs":{"total":95995035648,"available":95759761408,"internally_reserved":0,"allocated":166379520,"data_stored":166070942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":540800,"kb_used_data":397168,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93204352,"statfs":{"total":95995035648,"available":95441256448,"internally_reserved":0,"allocated":406700032,"data_stored":406391454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":400060,"kb_used_data":292524,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93345092,"statfs":{"total":95995035648,"available":95585374208,"internally_reserved":0,"allocated":299544576,"data_stored":299240070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564732,"kb_used_data":457196,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93180420,"statfs":{"total":95995035648,"available":95416750080,"internally_reserved":0,"allocated":468168704,"data_stored":467864198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":394620,"kb_used_data":291244,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93350532,"statfs":{"total":95995035648,"available":95590944768,"internally_reserved":0,"allocated":298233856,"data_stored":297929350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284688384,"data_stored":284688384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":459997184,"data_stored":459997184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284360704,"data_stored":284360704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400363520,"data_stored":400359448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":162402304,"data_stored":162398232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":173015040,"data_stored":173015040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":127270912,"data_stored":127270912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":309002240,"data_stored":309002240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":156041216,"data_stored":156041216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":315625472,"data_stored":315621400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":760152064,"data_stored":760152064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":258932736,"data_stored":258932736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:29:26.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-08-30T23:29:26.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:26.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:26.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:29:26.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:26.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:26 smithi106 ceph-mon[16603]: pgmap v501: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 918 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 244/44025 objects degraded (0.554%); 1431/44025 objects misplaced (3.250%); 6.3 MiB/s, 100 objects/s recovering 2024-08-30T23:29:26.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:26 smithi106 ceph-mon[16603]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:26.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 244/44025 objects degraded (0.554%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:26.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4113646342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:29:26.540 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:29:26.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:26.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:26.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:26 smithi136 ceph-mon[20207]: pgmap v501: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 918 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 244/44025 objects degraded (0.554%); 1431/44025 objects misplaced (3.250%); 6.3 MiB/s, 100 objects/s recovering 2024-08-30T23:29:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:26 smithi136 ceph-mon[20207]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 244/44025 objects degraded (0.554%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4113646342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:29:26.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:26.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:26 smithi173 ceph-mon[20554]: pgmap v501: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 918 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 686 op/s; 244/44025 objects degraded (0.554%); 1431/44025 objects misplaced (3.250%); 6.3 MiB/s, 100 objects/s recovering 2024-08-30T23:29:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:26 smithi173 ceph-mon[20554]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 244/44025 objects degraded (0.554%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4113646342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:29:26.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:26.962 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:26.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:27.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:27.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:27.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:27.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:27.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:27.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:27.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:27.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:27.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:27.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:27.639 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:27 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:27.639 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4113646342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:29:27.639 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:27 smithi106 ceph-mon[16603]: osdmap e195: 12 total, 12 up, 11 in 2024-08-30T23:29:27.639 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:27 smithi106 ceph-mon[16603]: pgmap v503: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 812 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 244/38979 objects degraded (0.626%); 1431/38979 objects misplaced (3.671%); 6.3 MiB/s, 100 objects/s recovering 2024-08-30T23:29:27.684 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:29:27.684 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:29:27.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:27.737 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:27 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4113646342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:29:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:27 smithi136 ceph-mon[20207]: osdmap e195: 12 total, 12 up, 11 in 2024-08-30T23:29:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:27 smithi136 ceph-mon[20207]: pgmap v503: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 812 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 244/38979 objects degraded (0.626%); 1431/38979 objects misplaced (3.671%); 6.3 MiB/s, 100 objects/s recovering 2024-08-30T23:29:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:27 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4113646342' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:29:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:27 smithi173 ceph-mon[20554]: osdmap e195: 12 total, 12 up, 11 in 2024-08-30T23:29:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:27 smithi173 ceph-mon[20554]: pgmap v503: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 812 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 706 op/s; 244/38979 objects degraded (0.626%); 1431/38979 objects misplaced (3.671%); 6.3 MiB/s, 100 objects/s recovering 2024-08-30T23:29:27.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:27.865 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:27.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:27.984 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:27.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:27.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:27.993 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:28.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:28.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:28.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:28.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:28.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:28.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:28.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:28.523 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:28.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:28.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:28.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:28.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:28.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:28.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:28.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:29.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:29.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:29.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2122026248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:29.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:29.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:29.221 INFO:teuthology.orchestra.run.smithi106.stdout:{"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 244/38979 objects degraded (0.626%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":584,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":195,"num_osds":12,"num_up_osds":12,"osd_up_since":1725060285,"num_in_osds":11,"osd_in_since":1725060424,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":12993,"data_bytes":851903032,"bytes_used":3884916736,"bytes_avail":1052060475392,"bytes_total":1055945392128,"degraded_objects":244,"degraded_total":38979,"degraded_ratio":0.0062597809076682318,"misplaced_objects":1431,"misplaced_total":38979,"misplaced_ratio":0.036712075733087046,"recovering_objects_per_sec":100,"recovering_bytes_per_sec":6611058,"recovering_keys_per_sec":0,"num_objects_recovered":1009,"num_bytes_recovered":66125824,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":706},"fsmap":{"epoch":1,"btime":"2024-08-30T23:18:02:086399+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.106:8443/"}},"servicemap":{"epoch":19,"modified":"2024-08-30T23:29:23.392381+0000","services":{"osd":{"daemons":{"summary":"","4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-30T23:29:29.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:29.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:29.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2122026248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2122026248' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:29.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:29.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:29.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:29:29.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:29.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:29.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.684 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:29.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:29.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:29.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:29.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:29.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:29.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:29.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:29.839 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:29.839 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:29.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:29.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:29.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:30.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:30.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:29:30.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:30.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2493183646' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:30.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:29 smithi106 ceph-mon[16603]: pgmap v504: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 694 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 735 op/s; 252/33279 objects degraded (0.757%); 906/33279 objects misplaced (2.722%); 6.5 MiB/s, 103 objects/s recovering 2024-08-30T23:29:30.087 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:30.088 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:30.127 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:30.127 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:30.140 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-30T23:29:30.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:29:30.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:30.168 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:30.168 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:30.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:30.204 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:30.204 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:30.241 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:30.242 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:30.260 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:30.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:30.284 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:30.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2493183646' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:30.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:29 smithi136 ceph-mon[20207]: pgmap v504: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 694 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 735 op/s; 252/33279 objects degraded (0.757%); 906/33279 objects misplaced (2.722%); 6.5 MiB/s, 103 objects/s recovering 2024-08-30T23:29:30.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:30.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:30.321 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:30.321 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2493183646' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:29 smithi173 ceph-mon[20554]: pgmap v504: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 694 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 735 op/s; 252/33279 objects degraded (0.757%); 906/33279 objects misplaced (2.722%); 6.5 MiB/s, 103 objects/s recovering 2024-08-30T23:29:30.366 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:30.366 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:30.405 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:30.406 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:30.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:30.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:30.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:30.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:30.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:30.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:30.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:30.836 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:30.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:30.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:30.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:31.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:31.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:30 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 252/33279 objects degraded (0.757%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:30 smithi106 ceph-mon[16603]: osdmap e196: 12 total, 12 up, 11 in 2024-08-30T23:29:31.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:31.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:31.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:30 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 252/33279 objects degraded (0.757%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:31.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:30 smithi136 ceph-mon[20207]: osdmap e196: 12 total, 12 up, 11 in 2024-08-30T23:29:31.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:30 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 252/33279 objects degraded (0.757%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:30 smithi173 ceph-mon[20554]: osdmap e196: 12 total, 12 up, 11 in 2024-08-30T23:29:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:31.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:31.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:31.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:31.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.614 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:31.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:31.680 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:29:31.680 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:29:31.680 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:29:31.680 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:29:31.680 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:29:31.687 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:31.704 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:31.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:31.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:31.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:31.873 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:31.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:31.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:31.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:32.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:32.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:32.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:32.216 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:31 smithi106 ceph-mon[16603]: pgmap v506: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 621 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 252/29808 objects degraded (0.845%); 906/29808 objects misplaced (3.039%); 8.1 MiB/s, 129 objects/s recovering 2024-08-30T23:29:32.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:32.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:32.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:31 smithi136 ceph-mon[20207]: pgmap v506: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 621 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 252/29808 objects degraded (0.845%); 906/29808 objects misplaced (3.039%); 8.1 MiB/s, 129 objects/s recovering 2024-08-30T23:29:32.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:31 smithi173 ceph-mon[20554]: pgmap v506: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 621 MiB data, 3.1 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 825 op/s; 252/29808 objects degraded (0.845%); 906/29808 objects misplaced (3.039%); 8.1 MiB/s, 129 objects/s recovering 2024-08-30T23:29:32.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:32.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:32.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:32.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:32.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:32.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:32.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:32.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:29:32.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:32.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:32.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:32.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:32.942 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:29:32.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.026 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:33.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2130563463' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:32 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:33.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:33.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:33.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:29:33.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:33.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:33.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2130563463' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:32 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:33.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:33.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:33.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2130563463' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:32 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:33.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.430 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:33.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:33.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:33.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:33.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:33.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:33.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:33.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:33.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:33.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:33.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:34.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:34.055 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:34.092 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:33 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:33 smithi106 ceph-mon[16603]: osdmap e197: 12 total, 12 up, 11 in 2024-08-30T23:29:34.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:33 smithi106 ceph-mon[16603]: pgmap v508: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 562 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 688 op/s; 252/26955 objects degraded (0.935%); 906/26955 objects misplaced (3.361%); 4.7 MiB/s, 75 objects/s recovering 2024-08-30T23:29:34.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:34.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:34.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:33 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:34.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:34.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:33 smithi136 ceph-mon[20207]: osdmap e197: 12 total, 12 up, 11 in 2024-08-30T23:29:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:33 smithi136 ceph-mon[20207]: pgmap v508: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 562 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 688 op/s; 252/26955 objects degraded (0.935%); 906/26955 objects misplaced (3.361%); 4.7 MiB/s, 75 objects/s recovering 2024-08-30T23:29:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:34.314 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:34.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:33 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:33 smithi173 ceph-mon[20554]: osdmap e197: 12 total, 12 up, 11 in 2024-08-30T23:29:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:33 smithi173 ceph-mon[20554]: pgmap v508: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 562 MiB data, 2.9 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 688 op/s; 252/26955 objects degraded (0.935%); 906/26955 objects misplaced (3.361%); 4.7 MiB/s, 75 objects/s recovering 2024-08-30T23:29:34.356 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:34.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:34.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:34.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:34.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:34.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.738 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:34.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:34.782 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:34.796 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:29:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:29:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.310593+0000", 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 503.48082535100002, 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.562890844, 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:29:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 1 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310593+0000", 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310593+0000", 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310594+0000", 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.173e-06 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310601+0000", 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3779999999999998e-06 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310610+0000", 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8219999999999996e-06 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310618+0000", 2024-08-30T23:29:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9780000000000005e-06 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310676+0000", 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8109000000000002e-05 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.310682+0000", 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3890000000000002e-06 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.872201+0000", 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56151878899999996 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.872330+0000", 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012908099999999999 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.872498+0000", 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016801199999999999 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873285+0000", 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078663699999999997 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873316+0000", 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1844000000000002e-05 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873378+0000", 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1765000000000002e-05 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873441+0000", 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2958000000000003e-05 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.873484+0000", 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2908999999999999e-05 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 502.82282915799999, 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34705745500000001, 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 52 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968590+0000", 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6499999999999999e-07 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968594+0000", 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0400000000000003e-06 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968602+0000", 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9850000000000006e-06 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968607+0000", 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2050000000000001e-06 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977321+0000", 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0087134099999999996 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977400+0000", 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8967000000000005e-05 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977514+0000", 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114164 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977774+0000", 2024-08-30T23:29:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026058500000000003 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.978024+0000", 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00024982299999999998 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315575+0000", 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33755099 2024-08-30T23:29:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315615+0000", 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9705e-05 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315647+0000", 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2016e-05 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 501.08458401299998, 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94419764299999998, 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 80 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7400000000000003e-07 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706923+0000", 2024-08-30T23:29:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8308999999999998e-05 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706932+0000", 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7409999999999998e-06 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706938+0000", 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.263e-06 2024-08-30T23:29:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708514+0000", 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015753620000000001 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708601+0000", 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6849999999999997e-05 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708731+0000", 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013043399999999999 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.709427+0000", 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069585600000000001 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.709468+0000", 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1125999999999999e-05 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650912+0000", 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94144422800000005 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650975+0000", 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2707999999999997e-05 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.651032+0000", 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7191999999999997e-05 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14229 2.6 2:681e88d6:::benchmark_data_smithi173_47195_object14228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 205.91522110400001, 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21536640000000001, 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14229 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:29:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876198+0000", 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6899999999999996e-07 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876298+0000", 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9622000000000003e-05 2024-08-30T23:29:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876310+0000", 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1817999999999999e-05 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876317+0000", 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8009999999999998e-06 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876377+0000", 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0489999999999999e-05 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876507+0000", 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012945900000000001 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876637+0000", 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013032300000000001 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878792+0000", 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002154689 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883495+0000", 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047034950000000002 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091459+0000", 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207963857 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091526+0000", 2024-08-30T23:29:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6610000000000001e-05 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091564+0000", 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8266999999999998e-05 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14233 2.6 2:682ac747:::benchmark_data_smithi173_47195_object14232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:29:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 205.913686705, 2024-08-30T23:29:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.215809479, 2024-08-30T23:29:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14233 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877733+0000", 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.215e-06 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877761+0000", 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7671999999999999e-05 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877776+0000", 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5352000000000001e-05 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877783+0000", 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2660000000000004e-06 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878221+0000", 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043807200000000002 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878304+0000", 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2769999999999995e-05 2024-08-30T23:29:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878392+0000", 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7997999999999993e-05 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.879128+0000", 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073602100000000001 2024-08-30T23:29:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883772+0000", 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046439580000000001 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093486+0000", 2024-08-30T23:29:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20971398699999999 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093518+0000", 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2047e-05 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093541+0000", 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3121000000000001e-05 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14240 2.6 2:6ab00b19:::benchmark_data_smithi173_47195_object14239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 205.909644765, 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21184964100000001, 2024-08-30T23:29:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14240 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881775+0000", 2024-08-30T23:29:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.536e-06 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881838+0000", 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2528999999999997e-05 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881849+0000", 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1606999999999999e-05 2024-08-30T23:29:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881857+0000", 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5349999999999999e-06 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881915+0000", 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8459999999999999e-05 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882042+0000", 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012617300000000001 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882187+0000", 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014551399999999999 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883462+0000", 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012752149999999999 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.898625+0000", 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015162491 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093570+0000", 2024-08-30T23:29:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.194944849 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093601+0000", 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0817e-05 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093623+0000", 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2915e-05 2024-08-30T23:29:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14242 2.6 2:6fdab316:::benchmark_data_smithi173_47195_object14241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 205.909280634, 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21156007900000001, 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14242 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882139+0000", 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2900000000000002e-07 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882196+0000", 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7549000000000001e-05 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882204+0000", 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4189999999999996e-06 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882208+0000", 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4390000000000002e-06 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882770+0000", 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056187700000000001 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882879+0000", 2024-08-30T23:29:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108856 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883022+0000", 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014315899999999999 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883584+0000", 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056226400000000004 2024-08-30T23:29:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.938528+0000", 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054943433 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093653+0000", 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.155125759 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093680+0000", 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6239000000000001e-05 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093698+0000", 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8255999999999999e-05 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16156 2.4 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 204.69384624099999, 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21552370500000001, 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16156 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097573+0000", 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3699999999999999e-07 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097632+0000", 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8870999999999999e-05 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097642+0000", 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0219e-05 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097649+0000", 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1269999999999997e-06 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097805+0000", 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015530900000000001 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097941+0000", 2024-08-30T23:29:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000135847 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098034+0000", 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3246999999999997e-05 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098764+0000", 2024-08-30T23:29:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000730133 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099110+0000", 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034588700000000002 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313035+0000", 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21392467000000001 2024-08-30T23:29:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313062+0000", 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7013999999999999e-05 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313096+0000", 2024-08-30T23:29:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4443999999999998e-05 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16160 2.9 2:9059f27b:::benchmark_data_smithi173_47195_object16159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099236+0000", 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 204.69218286099999, 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.187105983, 2024-08-30T23:29:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16160 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099236+0000", 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099236+0000", 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099237+0000", 2024-08-30T23:29:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6799999999999999e-07 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099301+0000", 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4830000000000001e-05 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099312+0000", 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.048e-05 2024-08-30T23:29:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099317+0000", 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0939999999999998e-06 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099366+0000", 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9104000000000003e-05 2024-08-30T23:29:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099480+0000", 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011429000000000001 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099560+0000", 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9608000000000001e-05 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100253+0000", 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069285199999999999 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100566+0000", 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031355599999999999 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.286236+0000", 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18566922 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.286295+0000", 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9500000000000003e-05 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.286342+0000", 2024-08-30T23:29:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6581e-05 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16161 2.6 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 204.69206523599999, 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212960288, 2024-08-30T23:29:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16161 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099354+0000", 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6199999999999997e-07 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099425+0000", 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0605999999999995e-05 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099432+0000", 2024-08-30T23:29:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6850000000000001e-06 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099438+0000", 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7069999999999997e-06 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099491+0000", 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3038000000000003e-05 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099620+0000", 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012866900000000001 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099790+0000", 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170266 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100492+0000", 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070156400000000001 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100884+0000", 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039193399999999998 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312191+0000", 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211307562 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312253+0000", 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2150000000000006e-05 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312314+0000", 2024-08-30T23:29:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0344999999999999e-05 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16911 2.4 2:26b3b856:::benchmark_data_smithi173_47195_object16910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 204.11903403299999, 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22617736999999999, 2024-08-30T23:29:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16911 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672455+0000", 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9855000000000001e-05 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672467+0000", 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1528e-05 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672473+0000", 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5400000000000001e-06 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672516+0000", 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2645999999999998e-05 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672585+0000", 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8751999999999995e-05 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:29:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672675+0000", 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0543000000000004e-05 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.673389+0000", 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071381299999999999 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.673738+0000", 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034890399999999998 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898465+0000", 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22472745399999999 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898518+0000", 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2428e-05 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898562+0000", 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3989000000000002e-05 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.887 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16921 2.2 e111/97 2:4bb89a5b:::benchmark_data_smithi173_47195_object16920:head v 111'1004, mlcod=111'999)", 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 204.11263908000001, 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21942291, 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.888 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678780+0000", 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5699999999999998e-07 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678834+0000", 2024-08-30T23:29:34.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3992000000000001e-05 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.896861+0000", 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218027111 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.896896+0000", 2024-08-30T23:29:34.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5326e-05 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.897044+0000", 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001473 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.897089+0000", 2024-08-30T23:29:34.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5806000000000002e-05 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898158+0000", 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010683559999999999 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898202+0000", 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4561999999999998e-05 2024-08-30T23:29:34.893 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40525 2.9 2:93a0bad4:::benchmark_data_smithi173_47195_object40524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.883468021, 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21130669299999999, 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.894 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40525 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907952+0000", 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8700000000000001e-06 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907991+0000", 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8658999999999997e-05 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908011+0000", 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9877999999999998e-05 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908020+0000", 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6500000000000002e-06 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908076+0000", 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6359e-05 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908193+0000", 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011733199999999999 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908326+0000", 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013302999999999999 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.909927+0000", 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016005909999999999 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.910234+0000", 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030682700000000002 2024-08-30T23:29:34.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119175+0000", 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20894082 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119206+0000", 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1652000000000001e-05 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119257+0000", 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1025e-05 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.902 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40537 2.9 2:97981a73:::benchmark_data_smithi173_47195_object40536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:29:34.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.87716909100001, 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207993344, 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40537 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914251+0000", 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.339e-06 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914366+0000", 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011518699999999999 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914379+0000", 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3424999999999999e-05 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914388+0000", 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6070000000000008e-06 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914447+0000", 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8585999999999998e-05 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914600+0000", 2024-08-30T23:29:34.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015365800000000001 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914763+0000", 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016261300000000001 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.915944+0000", 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011807250000000001 2024-08-30T23:29:34.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.916216+0000", 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000272692 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122145+0000", 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205928427 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122190+0000", 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5785000000000003e-05 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122243+0000", 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2299999999999997e-05 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40552 2.6 2:67bfbe41:::benchmark_data_smithi173_47195_object40551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.864277805, 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19393617399999999, 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.912 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40552 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927142+0000", 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8899999999999998e-07 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927204+0000", 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2008000000000007e-05 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927213+0000", 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8409999999999995e-06 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927219+0000", 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8690000000000001e-06 2024-08-30T23:29:34.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927264+0000", 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4690000000000001e-05 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927338+0000", 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3831000000000005e-05 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927468+0000", 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012972299999999999 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.928902+0000", 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014344970000000001 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929243+0000", 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034044300000000001 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121004+0000", 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19176105600000001 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121035+0000", 2024-08-30T23:29:34.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1269999999999997e-05 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121077+0000", 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2057000000000001e-05 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40557 2.9 2:93889969:::benchmark_data_smithi173_47195_object40556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:29:34.919 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.929782+0000", 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.861636804, 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.193788611, 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40557 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.920 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929782+0000", 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929782+0000", 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929783+0000", 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8100000000000001e-07 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929894+0000", 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111513 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929906+0000", 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2286000000000001e-05 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929914+0000", 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7330000000000003e-06 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929964+0000", 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0111999999999997e-05 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.930039+0000", 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4639000000000001e-05 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:29:34.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.930119+0000", 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9825000000000002e-05 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.931419+0000", 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001300615 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.931639+0000", 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000219175 2024-08-30T23:29:34.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.123451+0000", 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19181198499999999 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.123513+0000", 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2476000000000003e-05 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.123570+0000", 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7370999999999997e-05 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40564 2.6 2:65129be5:::benchmark_data_smithi173_47195_object40563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.934399+0000", 2024-08-30T23:29:34.927 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.857019708, 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.1867685, 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40564 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934399+0000", 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934399+0000", 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934400+0000", 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3689999999999999e-06 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934455+0000", 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4789e-05 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934469+0000", 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3623e-05 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934477+0000", 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3969999999999994e-06 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934539+0000", 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2427000000000004e-05 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934677+0000", 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013763600000000001 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.934890+0000", 2024-08-30T23:29:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00021267899999999999 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.936090+0000", 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012003199999999999 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.936389+0000", 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000299103 2024-08-30T23:29:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121108+0000", 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.184719315 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121136+0000", 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7980999999999999e-05 2024-08-30T23:29:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121167+0000", 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0861000000000001e-05 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:62934 2.2 e127/97 2:479e8ab6:::benchmark_data_smithi173_47195_object62933:head v 127'3887, mlcod=127'3884)", 2024-08-30T23:29:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 168.84124820299999, 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20906233900000001, 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950171+0000", 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2500000000000005e-07 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950233+0000", 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2382e-05 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157402+0000", 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20716816099999999 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157407+0000", 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7479999999999999e-06 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157462+0000", 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4840999999999999e-05 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157485+0000", 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3244999999999999e-05 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.159190+0000", 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017050120000000001 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.159233+0000", 2024-08-30T23:29:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2225000000000003e-05 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67558 2.9 2:98b6d163:::benchmark_data_smithi173_47195_object67557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.71525028299999, 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214383824, 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67558 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076169+0000", 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2300000000000001e-07 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076247+0000", 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8326999999999997e-05 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076255+0000", 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2660000000000001e-06 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076261+0000", 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.784e-06 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077492+0000", 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001231027 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077607+0000", 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011488 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077733+0000", 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012587500000000001 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.079761+0000", 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020281829999999998 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.080335+0000", 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057335700000000003 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290485+0000", 2024-08-30T23:29:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21015015400000001 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290523+0000", 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8250000000000001e-05 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290552+0000", 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9098000000000002e-05 2024-08-30T23:29:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67582 2.9 2:97773903:::benchmark_data_smithi173_47195_object67581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.69465290799999, 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19555588700000001, 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67582 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096767+0000", 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096823+0000", 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6773e-05 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096835+0000", 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1556000000000001e-05 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096842+0000", 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0770000000000002e-06 2024-08-30T23:29:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096892+0000", 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0256000000000001e-05 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096974+0000", 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1879999999999995e-05 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.097064+0000", 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9821999999999995e-05 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.098777+0000", 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017132149999999999 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.099120+0000", 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034311699999999999 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292262+0000", 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19314146600000001 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292296+0000", 2024-08-30T23:29:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3748000000000003e-05 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292322+0000", 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5986000000000002e-05 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:34.956 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:29:34.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:34.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:35.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:35.107 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:35.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:35.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:34 smithi173 ceph-mon[20554]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:35.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:35.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:34 smithi136 ceph-mon[20207]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:35.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:35.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:34 smithi106 ceph-mon[16603]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:35.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:35.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:35.465 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-08-30T23:29:33.394534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589234744,"num_objects":8985,"num_object_clones":0,"num_object_copies":26955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":906,"num_objects_unfound":0,"num_objects_dirty":8985,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":185152,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28282,"num_bytes_recovered":1854355040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161021,"ondisk_log_size":161021,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":3047164,"kb_used_data":1976636,"kb_used_omap":17,"kb_used_meta":1070254,"kb_avail":1028149508,"statfs":{"total":1055945392128,"available":1052825096192,"internally_reserved":0,"allocated":2024075264,"data_stored":2020636011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1095940985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-309198848,"num_objects":-4718,"num_object_clones":0,"num_object_copies":-14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":-525,"num_objects_unfound":0,"num_objects_dirty":-4718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":33882112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.852886"},"pg_stats":[{"pgid":"2.f","version":"196'11627","reported_seq":11942,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:30.627565+0000","last_change":"2024-08-30T23:28:16.218035+0000","last_active":"2024-08-30T23:29:30.627565+0000","last_peered":"2024-08-30T23:29:30.627565+0000","last_clean":"2024-08-30T23:29:30.627565+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:29:30.627565+0000","last_undegraded":"2024-08-30T23:29:30.627565+0000","last_fullsized":"2024-08-30T23:29:30.627565+0000","mapping_epoch":97,"log_start":"114'1600","ondisk_log_start":"114'1600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'8409","last_scrub_stamp":"2024-08-30T23:28:16.217972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.217972+0000","objects_scrubbed":3813,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:45.741305+0000","scrub_duration":782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11627,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"195'11346","reported_seq":11576,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-08-30T23:29:28.487280+0000","last_change":"2024-08-30T23:29:13.368781+0000","last_active":"2024-08-30T23:29:28.487280+0000","last_peered":"2024-08-30T23:29:28.487280+0000","last_clean":"2024-08-30T23:29:28.487280+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:29:28.487280+0000","last_undegraded":"2024-08-30T23:29:28.487280+0000","last_fullsized":"2024-08-30T23:29:28.487280+0000","mapping_epoch":97,"log_start":"112'1300","ondisk_log_start":"112'1300","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'10663","last_scrub_stamp":"2024-08-30T23:29:13.368419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:13.368419+0000","objects_scrubbed":1283,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:14.410101+0000","scrub_duration":729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797312,"num_objects":592,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"195'11684","reported_seq":11934,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-08-30T23:29:30.035650+0000","last_change":"2024-08-30T23:28:27.803716+0000","last_active":"2024-08-30T23:29:30.035650+0000","last_peered":"2024-08-30T23:29:30.035650+0000","last_clean":"2024-08-30T23:29:30.035650+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:29:30.035650+0000","last_undegraded":"2024-08-30T23:29:30.035650+0000","last_fullsized":"2024-08-30T23:29:30.035650+0000","mapping_epoch":97,"log_start":"114'1600","ondisk_log_start":"114'1600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'9007","last_scrub_stamp":"2024-08-30T23:28:27.803577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.803577+0000","objects_scrubbed":3238,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:55.835178+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799640,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11684,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"196'11518","reported_seq":11770,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:31.040725+0000","last_change":"2024-08-30T23:28:27.113440+0000","last_active":"2024-08-30T23:29:31.040725+0000","last_peered":"2024-08-30T23:29:31.040725+0000","last_clean":"2024-08-30T23:29:31.040725+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:29:31.040725+0000","last_undegraded":"2024-08-30T23:29:31.040725+0000","last_fullsized":"2024-08-30T23:29:31.040725+0000","mapping_epoch":97,"log_start":"113'1500","ondisk_log_start":"113'1500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'8728","last_scrub_stamp":"2024-08-30T23:28:27.113155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.113155+0000","objects_scrubbed":3323,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32899072,"num_objects":502,"num_object_clones":0,"num_object_copies":1506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":352,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:30.542254+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:30.542254+0000","last_peered":"2024-08-30T23:29:30.542254+0000","last_clean":"2024-08-30T23:29:30.542254+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:30.542254+0000","last_undegraded":"2024-08-30T23:29:30.542254+0000","last_fullsized":"2024-08-30T23:29:30.542254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"195'11692","reported_seq":11945,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-08-30T23:29:28.549678+0000","last_change":"2024-08-30T23:28:16.855547+0000","last_active":"2024-08-30T23:29:28.549678+0000","last_peered":"2024-08-30T23:29:28.549678+0000","last_clean":"2024-08-30T23:29:28.549678+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:29:28.549678+0000","last_undegraded":"2024-08-30T23:29:28.549678+0000","last_fullsized":"2024-08-30T23:29:28.549678+0000","mapping_epoch":97,"log_start":"113'1600","ondisk_log_start":"113'1600","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'8559","last_scrub_stamp":"2024-08-30T23:28:16.855286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.855286+0000","objects_scrubbed":3791,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11692,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"196'11729","reported_seq":22122,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:32.367529+0000","last_change":"2024-08-30T23:28:43.056625+0000","last_active":"2024-08-30T23:29:32.367529+0000","last_peered":"2024-08-30T23:29:32.367529+0000","last_clean":"2024-08-30T23:29:32.367529+0000","last_became_active":"2024-08-30T23:28:25.514631+0000","last_became_peered":"2024-08-30T23:28:25.514631+0000","last_unstale":"2024-08-30T23:29:32.367529+0000","last_undegraded":"2024-08-30T23:29:32.367529+0000","last_fullsized":"2024-08-30T23:29:32.367529+0000","mapping_epoch":168,"log_start":"115'1700","ondisk_log_start":"115'1700","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9573","last_scrub_stamp":"2024-08-30T23:28:43.056366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.056366+0000","objects_scrubbed":2652,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:58.660703+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11729,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332136448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"195'11581","reported_seq":11819,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-08-30T23:29:28.531772+0000","last_change":"2024-08-30T23:28:53.329917+0000","last_active":"2024-08-30T23:29:28.531772+0000","last_peered":"2024-08-30T23:29:28.531772+0000","last_clean":"2024-08-30T23:29:28.531772+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:29:28.531772+0000","last_undegraded":"2024-08-30T23:29:28.531772+0000","last_fullsized":"2024-08-30T23:29:28.531772+0000","mapping_epoch":97,"log_start":"113'1500","ondisk_log_start":"113'1500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9999","last_scrub_stamp":"2024-08-30T23:28:53.329808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:53.329808+0000","objects_scrubbed":2220,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:04.773357+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":623,"num_object_clones":0,"num_object_copies":1869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"195'11500","reported_seq":11752,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-08-30T23:29:28.576804+0000","last_change":"2024-08-30T23:28:23.087871+0000","last_active":"2024-08-30T23:29:28.576804+0000","last_peered":"2024-08-30T23:29:28.576804+0000","last_clean":"2024-08-30T23:29:28.576804+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:29:28.576804+0000","last_undegraded":"2024-08-30T23:29:28.576804+0000","last_fullsized":"2024-08-30T23:29:28.576804+0000","mapping_epoch":97,"log_start":"113'1400","ondisk_log_start":"113'1400","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'8647","last_scrub_stamp":"2024-08-30T23:28:23.087486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:23.087486+0000","objects_scrubbed":3550,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:51.858797+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44826624,"num_objects":684,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"195'11454","reported_seq":21813,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-08-30T23:29:28.400261+0000","last_change":"2024-08-30T23:28:51.930260+0000","last_active":"2024-08-30T23:29:28.400261+0000","last_peered":"2024-08-30T23:29:28.400261+0000","last_clean":"2024-08-30T23:29:28.400261+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:29:28.400261+0000","last_undegraded":"2024-08-30T23:29:28.400261+0000","last_fullsized":"2024-08-30T23:29:28.400261+0000","mapping_epoch":143,"log_start":"113'1400","ondisk_log_start":"113'1400","created":97,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5071,"num_bytes_recovered":332333056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"195'11469","reported_seq":24216,"reported_epoch":195,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:29:28.358502+0000","last_change":"2024-08-30T23:28:47.992578+0000","last_active":"2024-08-30T23:29:28.358502+0000","last_peered":"2024-08-30T23:29:28.358502+0000","last_clean":"2024-08-30T23:28:45.671248+0000","last_became_active":"2024-08-30T23:28:47.905585+0000","last_became_peered":"2024-08-30T23:28:47.905585+0000","last_unstale":"2024-08-30T23:29:28.358502+0000","last_undegraded":"2024-08-30T23:28:47.716401+0000","last_fullsized":"2024-08-30T23:28:47.688373+0000","mapping_epoch":182,"log_start":"112'1400","ondisk_log_start":"112'1400","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":615,"num_object_clones":0,"num_object_copies":1845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":615,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11469,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6263,"num_bytes_recovered":410386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":615},{"shards":"0,4","objects":-252}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"196'11640","reported_seq":11920,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:30.660454+0000","last_change":"2024-08-30T23:27:43.200813+0000","last_active":"2024-08-30T23:29:30.660454+0000","last_peered":"2024-08-30T23:29:30.660454+0000","last_clean":"2024-08-30T23:29:30.660454+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:29:30.660454+0000","last_undegraded":"2024-08-30T23:29:30.660454+0000","last_fullsized":"2024-08-30T23:29:30.660454+0000","mapping_epoch":142,"log_start":"114'1600","ondisk_log_start":"114'1600","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'6776","last_scrub_stamp":"2024-08-30T23:27:43.200650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:43.200650+0000","objects_scrubbed":5424,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"196'11660","reported_seq":22034,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:31.821234+0000","last_change":"2024-08-30T23:29:11.641650+0000","last_active":"2024-08-30T23:29:31.821234+0000","last_peered":"2024-08-30T23:29:31.821234+0000","last_clean":"2024-08-30T23:29:31.821234+0000","last_became_active":"2024-08-30T23:28:47.690562+0000","last_became_peered":"2024-08-30T23:28:47.690562+0000","last_unstale":"2024-08-30T23:29:31.821234+0000","last_undegraded":"2024-08-30T23:29:31.821234+0000","last_fullsized":"2024-08-30T23:29:31.821234+0000","mapping_epoch":182,"log_start":"114'1600","ondisk_log_start":"114'1600","created":97,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10738","last_scrub_stamp":"2024-08-30T23:29:11.641462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:11.641462+0000","objects_scrubbed":1414,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:34.059229+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31588352,"num_objects":482,"num_object_clones":0,"num_object_copies":1446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11660,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":331415552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"196'11583","reported_seq":11833,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:32.324680+0000","last_change":"2024-08-30T23:28:24.690367+0000","last_active":"2024-08-30T23:29:32.324680+0000","last_peered":"2024-08-30T23:29:32.324680+0000","last_clean":"2024-08-30T23:29:32.324680+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:29:32.324680+0000","last_undegraded":"2024-08-30T23:29:32.324680+0000","last_fullsized":"2024-08-30T23:29:32.324680+0000","mapping_epoch":97,"log_start":"113'1500","ondisk_log_start":"113'1500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'8760","last_scrub_stamp":"2024-08-30T23:28:24.690182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:24.690182+0000","objects_scrubbed":3297,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30474240,"num_objects":465,"num_object_clones":0,"num_object_copies":1395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11583,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"196'11809","reported_seq":12040,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:32.567720+0000","last_change":"2024-08-30T23:29:10.813505+0000","last_active":"2024-08-30T23:29:32.567720+0000","last_peered":"2024-08-30T23:29:32.567720+0000","last_clean":"2024-08-30T23:29:32.567720+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:29:32.567720+0000","last_undegraded":"2024-08-30T23:29:32.567720+0000","last_fullsized":"2024-08-30T23:29:32.567720+0000","mapping_epoch":97,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10887","last_scrub_stamp":"2024-08-30T23:29:10.813301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:10.813301+0000","objects_scrubbed":1415,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:27.781500+0000","scrub_duration":591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31129600,"num_objects":475,"num_object_clones":0,"num_object_copies":1425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11809,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"196'11298","reported_seq":11542,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-08-30T23:29:31.076555+0000","last_change":"2024-08-30T23:28:39.652116+0000","last_active":"2024-08-30T23:29:31.076555+0000","last_peered":"2024-08-30T23:29:31.076555+0000","last_clean":"2024-08-30T23:29:31.076555+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:29:31.076555+0000","last_undegraded":"2024-08-30T23:29:31.076555+0000","last_fullsized":"2024-08-30T23:29:31.076555+0000","mapping_epoch":97,"log_start":"112'1200","ondisk_log_start":"112'1200","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9191","last_scrub_stamp":"2024-08-30T23:28:39.651956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:39.651956+0000","objects_scrubbed":2655,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.058479+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11298,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"195'11439","reported_seq":25325,"reported_epoch":195,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:29:28.612454+0000","last_change":"2024-08-30T23:28:47.980377+0000","last_active":"2024-08-30T23:29:28.612454+0000","last_peered":"2024-08-30T23:29:28.612454+0000","last_clean":"2024-08-30T23:28:45.654798+0000","last_became_active":"2024-08-30T23:28:47.911739+0000","last_became_peered":"2024-08-30T23:28:47.911739+0000","last_unstale":"2024-08-30T23:29:28.612454+0000","last_undegraded":"2024-08-30T23:29:28.612454+0000","last_fullsized":"2024-08-30T23:29:28.612454+0000","mapping_epoch":182,"log_start":"113'1400","ondisk_log_start":"113'1400","created":97,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9517","last_scrub_stamp":"2024-08-30T23:28:43.578020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.578020+0000","objects_scrubbed":2573,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41746432,"num_objects":637,"num_object_clones":0,"num_object_copies":1911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":291,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6816,"num_bytes_recovered":446431232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[5,11,7],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"5,7,11","objects":346}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":588644376,"num_objects":8983,"num_object_clones":0,"num_object_copies":26949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":906,"num_objects_unfound":0,"num_objects_dirty":8983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185029,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28276,"num_bytes_recovered":1852768256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2637115392,"data_stored":2637103176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187516,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":242972,"kb_used_data":139532,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93502180,"statfs":{"total":95995035648,"available":95746232320,"internally_reserved":0,"allocated":142880768,"data_stored":142563784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449154,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710790,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":337804,"kb_used_data":226108,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93407348,"statfs":{"total":95995035648,"available":95649124352,"internally_reserved":0,"allocated":231534592,"data_stored":231221561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972428,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186184,"kb_used_data":105016,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93558968,"statfs":{"total":95995035648,"available":95804383232,"internally_reserved":0,"allocated":107536384,"data_stored":107227425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234063,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":268512,"kb_used_data":165132,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93476640,"statfs":{"total":95995035648,"available":95720079360,"internally_reserved":0,"allocated":169095168,"data_stored":168778204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528405,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127260,"kb_used_data":69708,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93617892,"statfs":{"total":95995035648,"available":95864721408,"internally_reserved":0,"allocated":71380992,"data_stored":71064028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724632,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201996,"kb_used_data":130552,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93543156,"statfs":{"total":95995035648,"available":95788191744,"internally_reserved":0,"allocated":133685248,"data_stored":133376289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186380,"kb_used_data":114940,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93558772,"statfs":{"total":95995035648,"available":95804182528,"internally_reserved":0,"allocated":117698560,"data_stored":117385529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437008,"kb_used_data":289212,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93308144,"statfs":{"total":95995035648,"available":95547539456,"internally_reserved":0,"allocated":296153088,"data_stored":295840057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325448,"kb_used_data":213752,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93419704,"statfs":{"total":95995035648,"available":95661776896,"internally_reserved":0,"allocated":218882048,"data_stored":218573089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":411672,"kb_used_data":308292,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93333480,"statfs":{"total":95995035648,"available":95573483520,"internally_reserved":0,"allocated":315691008,"data_stored":315377596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321928,"kb_used_data":214392,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93423224,"statfs":{"total":95995035648,"available":95665381376,"internally_reserved":0,"allocated":219537408,"data_stored":219228449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205324288,"data_stored":205324288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307888128,"data_stored":307888128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203554816,"data_stored":203554816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286658560,"data_stored":286654488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":114692096,"data_stored":114688024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":128188416,"data_stored":128188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65863680,"data_stored":65863680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":156565504,"data_stored":156565504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":106168320,"data_stored":106168320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":221777920,"data_stored":221773848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":701169664,"data_stored":701169664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":139264000,"data_stored":139264000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:29:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:35.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:35.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:35.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:35.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:35.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:35.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:29:35.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:35.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:35.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:35.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:36.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:36.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:35 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:35 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:35 smithi106 ceph-mon[16603]: pgmap v509: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 452 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 296/21690 objects degraded (1.365%); 425/21690 objects misplaced (1.959%); 7.5 MiB/s, 128 objects/s recovering 2024-08-30T23:29:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 252/26955 objects degraded (0.935%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:35 smithi106 ceph-mon[16603]: osdmap e198: 12 total, 12 up, 11 in 2024-08-30T23:29:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:36.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:36.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:36.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:36.266 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:36.266 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:36.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:35 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:36.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:35 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:36.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:35 smithi136 ceph-mon[20207]: pgmap v509: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 452 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 296/21690 objects degraded (1.365%); 425/21690 objects misplaced (1.959%); 7.5 MiB/s, 128 objects/s recovering 2024-08-30T23:29:36.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 252/26955 objects degraded (0.935%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:35 smithi136 ceph-mon[20207]: osdmap e198: 12 total, 12 up, 11 in 2024-08-30T23:29:36.300 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:36.300 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:35 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:35 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:35 smithi173 ceph-mon[20554]: pgmap v509: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 452 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 296/21690 objects degraded (1.365%); 425/21690 objects misplaced (1.959%); 7.5 MiB/s, 128 objects/s recovering 2024-08-30T23:29:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 252/26955 objects degraded (0.935%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:35 smithi173 ceph-mon[20554]: osdmap e198: 12 total, 12 up, 11 in 2024-08-30T23:29:36.330 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:36.330 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:36.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:36.363 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:36.363 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:36.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:36.390 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:36.390 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:36.421 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:36.454 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:36.454 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:36.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:36.485 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:36.485 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:36.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:36.515 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:36.516 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:36.544 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:36.545 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:36.573 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:36.573 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:36.603 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:36.603 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:36.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:36.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:36.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:29:36.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:36.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:36.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:36.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:36.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:36.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:36.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:37.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:37.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:37.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:37.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:37.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:37.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/499980888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:37.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/499980888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:37.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/499980888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:37.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:37.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:37.408 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:37.439 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:37.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:37.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:37.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:37.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:37.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:37.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:37.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:37.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:37.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:38.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:38.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:38.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:38.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:38.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:37 smithi106 ceph-mon[16603]: pgmap v511: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 360 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 296/17238 objects degraded (1.717%); 425/17238 objects misplaced (2.465%); 4.0 MiB/s, 74 objects/s recovering 2024-08-30T23:29:38.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:38.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:38.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:37 smithi136 ceph-mon[20207]: pgmap v511: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 360 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 296/17238 objects degraded (1.717%); 425/17238 objects misplaced (2.465%); 4.0 MiB/s, 74 objects/s recovering 2024-08-30T23:29:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:37 smithi173 ceph-mon[20554]: pgmap v511: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 360 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 744 op/s; 296/17238 objects degraded (1.717%); 425/17238 objects misplaced (2.465%); 4.0 MiB/s, 74 objects/s recovering 2024-08-30T23:29:38.387 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:29:38.387 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:29:38.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:38.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:38.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:29:38.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:38.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:38.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:38.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:38.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:38.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:38.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:38.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:38.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:38.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:38.983 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:29:39.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.062 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:39.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:39.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:29:39.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:39.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:39.277 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2032881090' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:39.278 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:38 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:39.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2032881090' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:39.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:38 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:39.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2032881090' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:38 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:29:39.348 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:39.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:29:39.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:39.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:39.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:39.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:39.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.798 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:39.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:39.858 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:39.861 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:39.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:39.934 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:39.948 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:39.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:39.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:39 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:39.973 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:39 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:39.973 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:39 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:39.973 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:39 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:29:39.973 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:39 smithi106 ceph-mon[16603]: osdmap e199: 12 total, 12 up, 11 in 2024-08-30T23:29:39.973 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:39 smithi106 ceph-mon[16603]: pgmap v513: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 259 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 296/12432 objects degraded (2.381%); 146/12432 objects misplaced (1.174%); 6.9 MiB/s, 174 objects/s recovering 2024-08-30T23:29:40.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:40.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:40.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:40.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:39 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:40.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:39 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:39 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:39 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:29:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:39 smithi136 ceph-mon[20207]: osdmap e199: 12 total, 12 up, 11 in 2024-08-30T23:29:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:39 smithi136 ceph-mon[20207]: pgmap v513: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 259 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 296/12432 objects degraded (2.381%); 146/12432 objects misplaced (1.174%); 6.9 MiB/s, 174 objects/s recovering 2024-08-30T23:29:40.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:39 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:40.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:39 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:40.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:39 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:40.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:39 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:29:40.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:39 smithi173 ceph-mon[20554]: osdmap e199: 12 total, 12 up, 11 in 2024-08-30T23:29:40.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:39 smithi173 ceph-mon[20554]: pgmap v513: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 259 MiB data, 2.0 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 750 op/s; 296/12432 objects degraded (2.381%); 146/12432 objects misplaced (1.174%); 6.9 MiB/s, 174 objects/s recovering 2024-08-30T23:29:40.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:40.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:40.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:40.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:40.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:40.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:40.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:40.712 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:40.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:40.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:40.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:40.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:40.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:41.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:41.094 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:40 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:41.095 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:40 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:41.095 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:40 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 296/12432 objects degraded (2.381%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:41.095 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:40 smithi106 ceph-mon[16603]: osdmap e200: 12 total, 12 up, 11 in 2024-08-30T23:29:41.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:41.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:41.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:40 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:40 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:40 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 296/12432 objects degraded (2.381%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:40 smithi136 ceph-mon[20207]: osdmap e200: 12 total, 12 up, 11 in 2024-08-30T23:29:41.300 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:41.300 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:40 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:40 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:40 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 296/12432 objects degraded (2.381%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:40 smithi173 ceph-mon[20554]: osdmap e200: 12 total, 12 up, 11 in 2024-08-30T23:29:41.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:41.339 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:41.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:41.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:41.366 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:41.367 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:41.377 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:41.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:41.422 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:41.422 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:41.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:41.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:41.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:41.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:41.658 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:41.659 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:41.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:41.695 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:41.723 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:41.723 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:41.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:41.746 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:41.747 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:41.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:41.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:29:41.778 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:41.778 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:41.813 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:41.814 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:41.849 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:41.849 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:41.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:41.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:41.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:41.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:41.883 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:41.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:42.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:42.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:42.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:42.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:42.288 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/691378543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:41 smithi136 ceph-mon[20207]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:41 smithi136 ceph-mon[20207]: pgmap v515: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 189 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 296/9063 objects degraded (3.266%); 146/9063 objects misplaced (1.611%); 2.9 MiB/s, 102 objects/s recovering 2024-08-30T23:29:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/691378543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:41 smithi173 ceph-mon[20554]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:42.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:41 smithi173 ceph-mon[20554]: pgmap v515: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 189 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 296/9063 objects degraded (3.266%); 146/9063 objects misplaced (1.611%); 2.9 MiB/s, 102 objects/s recovering 2024-08-30T23:29:42.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/691378543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:42.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:41 smithi106 ceph-mon[16603]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:42.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:41 smithi106 ceph-mon[16603]: pgmap v515: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 189 MiB data, 1.9 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 296/9063 objects degraded (3.266%); 146/9063 objects misplaced (1.611%); 2.9 MiB/s, 102 objects/s recovering 2024-08-30T23:29:42.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:42.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:42.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:42.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:42.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:42.678 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:42.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:42.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:42.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:42.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:42.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:43.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:43.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:43.147 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:29:43.147 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:29:43.147 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:29:43.147 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:29:43.147 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:29:43.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:43.173 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:43.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:43.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.299 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:43.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:43.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:43.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:43.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:43.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:43.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:43.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:43.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:43.790 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 97005 objects 2024-08-30T23:29:43.790 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :134.615 2024-08-30T23:29:43.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:43.814 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-08-30T23:29:40.545627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":258277944,"num_objects":3935,"num_object_clones":0,"num_object_copies":11805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":296,"num_objects_misplaced":146,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":190202,"num_write_kb":6210471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29407,"num_bytes_recovered":1901278816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":2034084,"kb_used_data":915044,"kb_used_omap":17,"kb_used_meta":1118830,"kb_avail":1029162588,"statfs":{"total":1055945392128,"available":1053862490112,"internally_reserved":0,"allocated":937005056,"data_stored":933433363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1145682809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-330956800,"num_objects":-5050,"num_object_clones":0,"num_object_copies":-15150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":44,"num_objects_misplaced":-760,"num_objects_unfound":0,"num_objects_dirty":-5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1125,"num_bytes_recovered":46923776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.151092"},"pg_stats":[{"pgid":"2.f","version":"198'11862","reported_seq":12181,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:35.654290+0000","last_change":"2024-08-30T23:28:16.218035+0000","last_active":"2024-08-30T23:29:35.654290+0000","last_peered":"2024-08-30T23:29:35.654290+0000","last_clean":"2024-08-30T23:29:35.654290+0000","last_became_active":"2024-08-30T23:25:50.450886+0000","last_became_peered":"2024-08-30T23:25:50.450886+0000","last_unstale":"2024-08-30T23:29:35.654290+0000","last_undegraded":"2024-08-30T23:29:35.654290+0000","last_fullsized":"2024-08-30T23:29:35.654290+0000","mapping_epoch":97,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'8409","last_scrub_stamp":"2024-08-30T23:28:16.217972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.217972+0000","objects_scrubbed":3813,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:45.741305+0000","scrub_duration":782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11862,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"198'11747","reported_seq":11984,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:38.538025+0000","last_change":"2024-08-30T23:29:13.368781+0000","last_active":"2024-08-30T23:29:38.538025+0000","last_peered":"2024-08-30T23:29:38.538025+0000","last_clean":"2024-08-30T23:29:38.538025+0000","last_became_active":"2024-08-30T23:25:50.473957+0000","last_became_peered":"2024-08-30T23:25:50.473957+0000","last_unstale":"2024-08-30T23:29:38.538025+0000","last_undegraded":"2024-08-30T23:29:38.538025+0000","last_fullsized":"2024-08-30T23:29:38.538025+0000","mapping_epoch":97,"log_start":"115'1700","ondisk_log_start":"115'1700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'10663","last_scrub_stamp":"2024-08-30T23:29:13.368419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:13.368419+0000","objects_scrubbed":1283,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:14.410101+0000","scrub_duration":729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11747,"num_write_kb":382016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"199'12101","reported_seq":12359,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-08-30T23:29:40.071140+0000","last_change":"2024-08-30T23:28:27.803716+0000","last_active":"2024-08-30T23:29:40.071140+0000","last_peered":"2024-08-30T23:29:40.071140+0000","last_clean":"2024-08-30T23:29:40.071140+0000","last_became_active":"2024-08-30T23:25:50.473931+0000","last_became_peered":"2024-08-30T23:25:50.473931+0000","last_unstale":"2024-08-30T23:29:40.071140+0000","last_undegraded":"2024-08-30T23:29:40.071140+0000","last_fullsized":"2024-08-30T23:29:40.071140+0000","mapping_epoch":97,"log_start":"116'2100","ondisk_log_start":"116'2100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"171'9007","last_scrub_stamp":"2024-08-30T23:28:27.803577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.803577+0000","objects_scrubbed":3238,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:55.835178+0000","scrub_duration":1248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7471128,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12101,"num_write_kb":390849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"198'11721","reported_seq":11976,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:36.087331+0000","last_change":"2024-08-30T23:28:27.113440+0000","last_active":"2024-08-30T23:29:36.087331+0000","last_peered":"2024-08-30T23:29:36.087331+0000","last_clean":"2024-08-30T23:29:36.087331+0000","last_became_active":"2024-08-30T23:25:50.531720+0000","last_became_peered":"2024-08-30T23:25:50.531720+0000","last_unstale":"2024-08-30T23:29:36.087331+0000","last_undegraded":"2024-08-30T23:29:36.087331+0000","last_fullsized":"2024-08-30T23:29:36.087331+0000","mapping_epoch":97,"log_start":"115'1700","ondisk_log_start":"115'1700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'8728","last_scrub_stamp":"2024-08-30T23:28:27.113155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:27.113155+0000","objects_scrubbed":3323,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19595264,"num_objects":299,"num_object_clones":0,"num_object_copies":897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11721,"num_write_kb":384640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":356,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:35.552411+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:35.552411+0000","last_peered":"2024-08-30T23:29:35.552411+0000","last_clean":"2024-08-30T23:29:35.552411+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:35.552411+0000","last_undegraded":"2024-08-30T23:29:35.552411+0000","last_fullsized":"2024-08-30T23:29:35.552411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"198'12116","reported_seq":12375,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:38.614440+0000","last_change":"2024-08-30T23:28:16.855547+0000","last_active":"2024-08-30T23:29:38.614440+0000","last_peered":"2024-08-30T23:29:38.614440+0000","last_clean":"2024-08-30T23:29:38.614440+0000","last_became_active":"2024-08-30T23:25:50.451935+0000","last_became_peered":"2024-08-30T23:25:50.451935+0000","last_unstale":"2024-08-30T23:29:38.614440+0000","last_undegraded":"2024-08-30T23:29:38.614440+0000","last_fullsized":"2024-08-30T23:29:38.614440+0000","mapping_epoch":97,"log_start":"116'2100","ondisk_log_start":"116'2100","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'8559","last_scrub_stamp":"2024-08-30T23:28:16.855286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:16.855286+0000","objects_scrubbed":3791,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12116,"num_write_kb":394496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"198'11946","reported_seq":22344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:37.387831+0000","last_change":"2024-08-30T23:28:43.056625+0000","last_active":"2024-08-30T23:29:37.387831+0000","last_peered":"2024-08-30T23:29:37.387831+0000","last_clean":"2024-08-30T23:29:37.387831+0000","last_became_active":"2024-08-30T23:28:25.514631+0000","last_became_peered":"2024-08-30T23:28:25.514631+0000","last_unstale":"2024-08-30T23:29:37.387831+0000","last_undegraded":"2024-08-30T23:29:37.387831+0000","last_fullsized":"2024-08-30T23:29:37.387831+0000","mapping_epoch":168,"log_start":"116'1900","ondisk_log_start":"116'1900","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9573","last_scrub_stamp":"2024-08-30T23:28:43.056366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.056366+0000","objects_scrubbed":2652,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:58.660703+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17170432,"num_objects":262,"num_object_clones":0,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11946,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5068,"num_bytes_recovered":332136448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"198'11988","reported_seq":12232,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:38.532407+0000","last_change":"2024-08-30T23:28:53.329917+0000","last_active":"2024-08-30T23:29:38.532407+0000","last_peered":"2024-08-30T23:29:38.532407+0000","last_clean":"2024-08-30T23:29:38.532407+0000","last_became_active":"2024-08-30T23:25:50.531387+0000","last_became_peered":"2024-08-30T23:25:50.531387+0000","last_unstale":"2024-08-30T23:29:38.532407+0000","last_undegraded":"2024-08-30T23:29:38.532407+0000","last_fullsized":"2024-08-30T23:29:38.532407+0000","mapping_epoch":97,"log_start":"115'1900","ondisk_log_start":"115'1900","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9999","last_scrub_stamp":"2024-08-30T23:28:53.329808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:53.329808+0000","objects_scrubbed":2220,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:04.773357+0000","scrub_duration":704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14155776,"num_objects":216,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11988,"num_write_kb":390528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"198'11973","reported_seq":12232,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:38.555476+0000","last_change":"2024-08-30T23:28:23.087871+0000","last_active":"2024-08-30T23:29:38.555476+0000","last_peered":"2024-08-30T23:29:38.555476+0000","last_clean":"2024-08-30T23:29:38.555476+0000","last_became_active":"2024-08-30T23:25:50.451119+0000","last_became_peered":"2024-08-30T23:25:50.451119+0000","last_unstale":"2024-08-30T23:29:38.555476+0000","last_undegraded":"2024-08-30T23:29:38.555476+0000","last_fullsized":"2024-08-30T23:29:38.555476+0000","mapping_epoch":97,"log_start":"116'1900","ondisk_log_start":"116'1900","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'8647","last_scrub_stamp":"2024-08-30T23:28:23.087486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:23.087486+0000","objects_scrubbed":3550,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:29:51.858797+0000","scrub_duration":1061,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":211,"num_object_clones":0,"num_object_copies":633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11973,"num_write_kb":389888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"198'11881","reported_seq":22246,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:38.382992+0000","last_change":"2024-08-30T23:28:51.930260+0000","last_active":"2024-08-30T23:29:38.382992+0000","last_peered":"2024-08-30T23:29:38.382992+0000","last_clean":"2024-08-30T23:29:38.382992+0000","last_became_active":"2024-08-30T23:27:15.150053+0000","last_became_peered":"2024-08-30T23:27:15.150053+0000","last_unstale":"2024-08-30T23:29:38.382992+0000","last_undegraded":"2024-08-30T23:29:38.382992+0000","last_fullsized":"2024-08-30T23:29:38.382992+0000","mapping_epoch":143,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13172736,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11881,"num_write_kb":386624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5071,"num_bytes_recovered":332333056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"198'11879","reported_seq":25559,"reported_epoch":198,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:29:38.437021+0000","last_change":"2024-08-30T23:28:47.992578+0000","last_active":"2024-08-30T23:29:38.437021+0000","last_peered":"2024-08-30T23:29:38.437021+0000","last_clean":"2024-08-30T23:28:45.671248+0000","last_became_active":"2024-08-30T23:28:47.905585+0000","last_became_peered":"2024-08-30T23:28:47.905585+0000","last_unstale":"2024-08-30T23:29:38.437021+0000","last_undegraded":"2024-08-30T23:28:47.716401+0000","last_fullsized":"2024-08-30T23:28:47.688373+0000","mapping_epoch":182,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:25:49.429788+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":205,"num_object_clones":0,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":296,"num_objects_misplaced":107,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":386688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6727,"num_bytes_recovered":440795136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[0,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":107},{"shards":"0,4","objects":-198}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"198'11846","reported_seq":12130,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:35.671765+0000","last_change":"2024-08-30T23:27:43.200813+0000","last_active":"2024-08-30T23:29:35.671765+0000","last_peered":"2024-08-30T23:29:35.671765+0000","last_clean":"2024-08-30T23:29:35.671765+0000","last_became_active":"2024-08-30T23:27:14.138510+0000","last_became_peered":"2024-08-30T23:27:14.138510+0000","last_unstale":"2024-08-30T23:29:35.671765+0000","last_undegraded":"2024-08-30T23:29:35.671765+0000","last_fullsized":"2024-08-30T23:29:35.671765+0000","mapping_epoch":142,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'6776","last_scrub_stamp":"2024-08-30T23:27:43.200650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:27:43.200650+0000","objects_scrubbed":5424,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21233664,"num_objects":324,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11846,"num_write_kb":389440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,3],"acting":[8,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.7","version":"198'11863","reported_seq":22241,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:36.841666+0000","last_change":"2024-08-30T23:29:11.641650+0000","last_active":"2024-08-30T23:29:36.841666+0000","last_peered":"2024-08-30T23:29:36.841666+0000","last_clean":"2024-08-30T23:29:36.841666+0000","last_became_active":"2024-08-30T23:28:47.690562+0000","last_became_peered":"2024-08-30T23:28:47.690562+0000","last_unstale":"2024-08-30T23:29:36.841666+0000","last_undegraded":"2024-08-30T23:29:36.841666+0000","last_fullsized":"2024-08-30T23:29:36.841666+0000","mapping_epoch":182,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10738","last_scrub_stamp":"2024-08-30T23:29:11.641462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:11.641462+0000","objects_scrubbed":1414,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:34.059229+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18284544,"num_objects":279,"num_object_clones":0,"num_object_copies":837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":388544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5057,"num_bytes_recovered":331415552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"198'11782","reported_seq":12036,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:37.299873+0000","last_change":"2024-08-30T23:28:24.690367+0000","last_active":"2024-08-30T23:29:37.299873+0000","last_peered":"2024-08-30T23:29:37.299873+0000","last_clean":"2024-08-30T23:29:37.299873+0000","last_became_active":"2024-08-30T23:25:50.531596+0000","last_became_peered":"2024-08-30T23:25:50.531596+0000","last_unstale":"2024-08-30T23:29:37.299873+0000","last_undegraded":"2024-08-30T23:29:37.299873+0000","last_fullsized":"2024-08-30T23:29:37.299873+0000","mapping_epoch":97,"log_start":"115'1700","ondisk_log_start":"115'1700","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'8760","last_scrub_stamp":"2024-08-30T23:28:24.690182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:24.690182+0000","objects_scrubbed":3297,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11782,"num_write_kb":385536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"198'12013","reported_seq":12248,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:37.598106+0000","last_change":"2024-08-30T23:29:10.813505+0000","last_active":"2024-08-30T23:29:37.598106+0000","last_peered":"2024-08-30T23:29:37.598106+0000","last_clean":"2024-08-30T23:29:37.598106+0000","last_became_active":"2024-08-30T23:25:50.474118+0000","last_became_peered":"2024-08-30T23:25:50.474118+0000","last_unstale":"2024-08-30T23:29:37.598106+0000","last_undegraded":"2024-08-30T23:29:37.598106+0000","last_fullsized":"2024-08-30T23:29:37.598106+0000","mapping_epoch":97,"log_start":"116'2000","ondisk_log_start":"116'2000","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10887","last_scrub_stamp":"2024-08-30T23:29:10.813301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:29:10.813301+0000","objects_scrubbed":1415,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:27.781500+0000","scrub_duration":591,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17760256,"num_objects":271,"num_object_clones":0,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12013,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"198'11519","reported_seq":11767,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-08-30T23:29:36.101782+0000","last_change":"2024-08-30T23:28:39.652116+0000","last_active":"2024-08-30T23:29:36.101782+0000","last_peered":"2024-08-30T23:29:36.101782+0000","last_clean":"2024-08-30T23:29:36.101782+0000","last_became_active":"2024-08-30T23:25:50.473870+0000","last_became_peered":"2024-08-30T23:25:50.473870+0000","last_unstale":"2024-08-30T23:29:36.101782+0000","last_undegraded":"2024-08-30T23:29:36.101782+0000","last_fullsized":"2024-08-30T23:29:36.101782+0000","mapping_epoch":97,"log_start":"114'1500","ondisk_log_start":"114'1500","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9191","last_scrub_stamp":"2024-08-30T23:28:39.651956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:39.651956+0000","objects_scrubbed":2655,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.058479+0000","scrub_duration":745,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19988480,"num_objects":305,"num_object_clones":0,"num_object_copies":915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11519,"num_write_kb":378368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"198'11842","reported_seq":27056,"reported_epoch":198,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:29:38.620513+0000","last_change":"2024-08-30T23:28:47.980377+0000","last_active":"2024-08-30T23:29:38.620513+0000","last_peered":"2024-08-30T23:29:38.620513+0000","last_clean":"2024-08-30T23:28:45.654798+0000","last_became_active":"2024-08-30T23:28:47.911739+0000","last_became_peered":"2024-08-30T23:28:47.911739+0000","last_unstale":"2024-08-30T23:29:38.620513+0000","last_undegraded":"2024-08-30T23:29:38.620513+0000","last_fullsized":"2024-08-30T23:29:38.620513+0000","mapping_epoch":182,"log_start":"115'1800","ondisk_log_start":"115'1800","created":97,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'9517","last_scrub_stamp":"2024-08-30T23:28:43.578020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:25:49.429788+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.578020+0000","objects_scrubbed":2573,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15335424,"num_objects":234,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11842,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7477,"num_bytes_recovered":462946304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[5,11,7],"avail_no_missing":["5","7","11"],"object_location_counts":[{"shards":"5,7,11","objects":195}],"blocked_by":[],"up_primary":8,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":257687576,"num_objects":3933,"num_object_clones":0,"num_object_copies":11799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":296,"num_objects_misplaced":146,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190079,"num_write_kb":6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29401,"num_bytes_recovered":1899692032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1540698112,"data_stored":1540685896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187517,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190200,"kb_used_data":82660,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93554952,"statfs":{"total":95995035648,"available":95800270848,"internally_reserved":0,"allocated":84643840,"data_stored":84318050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":110098884},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449155,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710792,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168188,"kb_used_data":60652,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93576964,"statfs":{"total":95995035648,"available":95822811136,"internally_reserved":0,"allocated":62107648,"data_stored":61776955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972429,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162212,"kb_used_data":72720,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93582940,"statfs":{"total":95995035648,"available":95828930560,"internally_reserved":0,"allocated":74465280,"data_stored":74147465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234064,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204980,"kb_used_data":97444,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93540172,"statfs":{"total":95995035648,"available":95785136128,"internally_reserved":0,"allocated":99782656,"data_stored":99456886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528406,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":103608,"kb_used_data":41892,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93641544,"statfs":{"total":95995035648,"available":95888941056,"internally_reserved":0,"allocated":42897408,"data_stored":42571638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724634,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":121708,"kb_used_data":46108,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93623444,"statfs":{"total":95995035648,"available":95870406656,"internally_reserved":0,"allocated":47214592,"data_stored":46892374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109056,"kb_used_data":33452,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93636096,"statfs":{"total":95995035648,"available":95883362304,"internally_reserved":0,"allocated":34254848,"data_stored":33924155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":240688,"kb_used_data":97056,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93504464,"statfs":{"total":95995035648,"available":95748571136,"internally_reserved":0,"allocated":99385344,"data_stored":99059054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177196,"kb_used_data":73820,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93567956,"statfs":{"total":95995035648,"available":95813586944,"internally_reserved":0,"allocated":75591680,"data_stored":75269462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384556,"kb_used_data":240924,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93360596,"statfs":{"total":95995035648,"available":95601250304,"internally_reserved":0,"allocated":246706176,"data_stored":246383958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171692,"kb_used_data":68316,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93573460,"statfs":{"total":95995035648,"available":95819223040,"internally_reserved":0,"allocated":69955584,"data_stored":69633366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68747264,"data_stored":68747264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239861760,"data_stored":239861760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67108864,"data_stored":67108864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":91623424,"data_stored":91619352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29954048,"data_stored":29949976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":42991616,"data_stored":42991616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":37683200,"data_stored":37683200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":88670208,"data_stored":88670208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":70975488,"data_stored":70975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49680384,"data_stored":49676312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":671678464,"data_stored":671678464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":81723392,"data_stored":81723392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:29:43.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-08-30T23:29:43.837 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-08-30T23:29:43.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-08-30T23:29:43.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:43.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:43.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:44.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:44.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:44.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:44.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:44.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:44.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:44.358 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:44.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:29:44.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:44.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:44.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:44.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:44.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:44 smithi106 ceph-mon[16603]: pgmap v516: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 128 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 296/6126 objects degraded (4.832%); 146/6126 objects misplaced (2.383%); 2.2 MiB/s, 78 objects/s recovering 2024-08-30T23:29:44.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:44.662 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:44.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:44 smithi136 ceph-mon[20207]: pgmap v516: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 128 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 296/6126 objects degraded (4.832%); 146/6126 objects misplaced (2.383%); 2.2 MiB/s, 78 objects/s recovering 2024-08-30T23:29:44.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:44 smithi173 ceph-mon[20554]: pgmap v516: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 128 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 296/6126 objects degraded (4.832%); 146/6126 objects misplaced (2.383%); 2.2 MiB/s, 78 objects/s recovering 2024-08-30T23:29:44.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:44.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:29:44.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:44.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:45.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:45.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:45.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:45.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:45.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:45.196 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:45.237 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:45.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:45.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:45.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:45.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:45.454 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:29:45.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:45.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=78570887,config=2353167246,health=2562820933,kv=1150238030}) 2024-08-30T23:29:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3680398200,logm=2456832599}) 2024-08-30T23:29:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=791618012}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161494948}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109074847}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703635336}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471886914}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347463917}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1855201445}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283594521}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437148952}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2963762053}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529343146}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2316337901}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108585457}) 2024-08-30T23:29:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353318521}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3357422834}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091818561}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257757769}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1180376857}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300866450}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586859139}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=96945041}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847112209}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2267980098}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3344930276}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=67,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9} crc {logm=1396638225,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=1575914582}) 2024-08-30T23:29:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=2970290153,mon_config_key=2966830316}) 2024-08-30T23:29:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1059387542}) 2024-08-30T23:29:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=97,monmap=3} crc {mon_config_key=2765739967,monmap=494457144}) 2024-08-30T23:29:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {monmap=2,osd_metadata=12,osd_pg_creating=1,osdmap=85} crc {monmap=692619927,osd_metadata=3634387181,osd_pg_creating=727646340,osdmap=2882104306}) 2024-08-30T23:29:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=55} crc {osdmap=377366385}) 2024-08-30T23:29:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4202170072' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:45.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.657 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:45.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:45.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:45.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:45.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=78570887,config=2353167246,health=2562820933,kv=1150238030}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3680398200,logm=2456832599}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=791618012}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161494948}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109074847}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703635336}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471886914}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347463917}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1855201445}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283594521}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437148952}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2963762053}) 2024-08-30T23:29:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529343146}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2316337901}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108585457}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353318521}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3357422834}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091818561}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257757769}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1180376857}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300866450}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586859139}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=96945041}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847112209}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2267980098}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3344930276}) 2024-08-30T23:29:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=67,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9} crc {logm=1396638225,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=1575914582}) 2024-08-30T23:29:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=2970290153,mon_config_key=2966830316}) 2024-08-30T23:29:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1059387542}) 2024-08-30T23:29:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=97,monmap=3} crc {mon_config_key=2765739967,monmap=494457144}) 2024-08-30T23:29:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {monmap=2,osd_metadata=12,osd_pg_creating=1,osdmap=85} crc {monmap=692619927,osd_metadata=3634387181,osd_pg_creating=727646340,osdmap=2882104306}) 2024-08-30T23:29:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=55} crc {osdmap=377366385}) 2024-08-30T23:29:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4202170072' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=78570887,config=2353167246,health=2562820933,kv=1150238030}) 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3680398200,logm=2456832599}) 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=791618012}) 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4161494948}) 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109074847}) 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703635336}) 2024-08-30T23:29:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471886914}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3347463917}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1855201445}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1283594521}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2437148952}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2963762053}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529343146}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2316337901}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108585457}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1353318521}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3357422834}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091818561}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=257757769}) 2024-08-30T23:29:45.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1180376857}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3300866450}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586859139}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=96945041}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847112209}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2267980098}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3344930276}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=67,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=9} crc {logm=1396638225,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=1575914582}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=6,mon_config_key=94} crc {mgrstat=2970290153,mon_config_key=2966830316}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1059387542}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=97,monmap=3} crc {mon_config_key=2765739967,monmap=494457144}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {monmap=2,osd_metadata=12,osd_pg_creating=1,osdmap=85} crc {monmap=692619927,osd_metadata=3634387181,osd_pg_creating=727646340,osdmap=2882104306}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=55} crc {osdmap=377366385}) 2024-08-30T23:29:45.826 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4202170072' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:29:45.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:45.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:45.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:46.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:46.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:46.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:46.151 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:46.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:46.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:46.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:46.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:46.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:46.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:46.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:46 smithi173 ceph-mon[20554]: pgmap v517: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 29 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 296/1371 objects degraded (21.590%); 39/1371 objects misplaced (2.845%); 3.0 MiB/s, 155 objects/s recovering 2024-08-30T23:29:46.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:46 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:29:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4202170072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:46 smithi173 ceph-mon[20554]: osdmap e201: 12 total, 12 up, 11 in 2024-08-30T23:29:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 296/1371 objects degraded (21.590%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:46 smithi173 ceph-mon[20554]: osdmap e202: 12 total, 12 up, 11 in 2024-08-30T23:29:46.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:46.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:46.646 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.14157.0:80 1.0 e23/20 1:30306672:devicehealth::main.db.0000000000000000:head v 23'58, mlcod=23'53)", 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:29:46.650 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 512.939770501, 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94186119800000001, 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:29:46.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708862+0000", 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5799999999999999e-07 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708893+0000", 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1081999999999998e-05 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650093+0000", 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94119932299999998 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650101+0000", 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7900000000000005e-06 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650169+0000", 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650197+0000", 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.72e-05 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650681+0000", 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00048408700000000002 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650723+0000", 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:29:46.654 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14244 2.3 e109/97 2:c35428fa:::benchmark_data_smithi173_47195_object14243:head v 109'957, mlcod=109'954)", 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 217.764851459, 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213971894, 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883782+0000", 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999999e-07 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883844+0000", 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2580000000000001e-05 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096781+0000", 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212936878 2024-08-30T23:29:46.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096785+0000", 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0389999999999998e-06 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096831+0000", 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6261999999999999e-05 2024-08-30T23:29:46.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096844+0000", 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2883e-05 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097712+0000", 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086761599999999998 2024-08-30T23:29:46.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097753+0000", 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0729999999999998e-05 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:56060 2.e 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.86915544499999, 2024-08-30T23:29:46.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21409088000000001, 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 56060 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779478+0000", 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4869999999999999e-06 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779513+0000", 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4384999999999997e-05 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779529+0000", 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6447999999999999e-05 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779537+0000", 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999993e-06 2024-08-30T23:29:46.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779580+0000", 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3254999999999998e-05 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779658+0000", 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7711000000000002e-05 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:29:46.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779738+0000", 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9775000000000001e-05 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781534+0000", 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017964820000000001 2024-08-30T23:29:46.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992319+0000", 2024-08-30T23:29:46.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21078449599999999 2024-08-30T23:29:46.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993464+0000", 2024-08-30T23:29:46.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001145294 2024-08-30T23:29:46.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993511+0000", 2024-08-30T23:29:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6344000000000002e-05 2024-08-30T23:29:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993568+0000", 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6993000000000001e-05 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:56060 2.e e124/97 ondisk, result = 0)", 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.866815476, 2024-08-30T23:29:46.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051657200000001, 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:29:46.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9999999999999997e-07 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781821+0000", 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9480000000000001e-06 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990657+0000", 2024-08-30T23:29:46.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20883629200000001 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990664+0000", 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.793e-06 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992305+0000", 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016408480000000001 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992313+0000", 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1100000000000003e-06 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992333+0000", 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9981000000000001e-05 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67116 2.e 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:29:46.673 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 177.03670165899999, 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22641219700000001, 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67116 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611931+0000", 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612007+0000", 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5883999999999994e-05 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612021+0000", 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4007e-05 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612028+0000", 2024-08-30T23:29:46.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4099999999999996e-06 2024-08-30T23:29:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614110+0000", 2024-08-30T23:29:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020827520000000002 2024-08-30T23:29:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614270+0000", 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000159235 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614420+0000", 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015048199999999999 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617176+0000", 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027561619999999999 2024-08-30T23:29:46.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617722+0000", 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054594799999999998 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838237+0000", 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22051536999999999 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838285+0000", 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7808e-05 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838343+0000", 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7340000000000003e-05 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75429 2.1 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:29:46.683 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 169.69019364299999, 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24538217700000001, 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75429 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958440+0000", 2024-08-30T23:29:46.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9410000000000001e-06 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958471+0000", 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0919e-05 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958494+0000", 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3078000000000002e-05 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958504+0000", 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1190000000000003e-06 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958565+0000", 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1364000000000004e-05 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958669+0000", 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010414199999999999 2024-08-30T23:29:46.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958772+0000", 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102979 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.960745+0000", 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019727600000000001 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.066102+0000", 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.105356845 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203744+0000", 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13764249000000001 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203787+0000", 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.286e-05 2024-08-30T23:29:46.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203821+0000", 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3680000000000003e-05 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head, version: 114'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head@114'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:29:46.690 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 121.75647474100001, 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19944226600000001, 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892158+0000", 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892186+0000", 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7710000000000001e-05 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089119+0000", 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19693307299999999 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089137+0000", 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7533e-05 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089255+0000", 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118318 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089275+0000", 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9828000000000001e-05 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091600+0000", 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023249030000000001 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.694 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:126363 2.7 e160/138 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head v 160'7890, mlcod=119'2657)", 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 99.919009434000003, 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.217439837, 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:29:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729630+0000", 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.674e-06 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940470+0000", 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21084006199999999 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940478+0000", 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4390000000000003e-06 2024-08-30T23:29:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940541+0000", 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3115999999999997e-05 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940571+0000", 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9981e-05 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947012+0000", 2024-08-30T23:29:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064411269999999996 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947062+0000", 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0708999999999998e-05 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144840 2.1 2:8bdc20ef:::benchmark_data_smithi173_47195_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 75.055153771999997, 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21736361900000001, 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144840 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593481+0000", 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3460000000000001e-06 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593490+0000", 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593512+0000", 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2410000000000001e-05 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593522+0000", 2024-08-30T23:29:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4129999999999995e-06 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593705+0000", 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018286800000000001 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593993+0000", 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028869300000000002 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.594314+0000", 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032096700000000002 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.599306+0000", 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049920049999999999 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603619+0000", 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043124419999999997 2024-08-30T23:29:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810726+0000", 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20710743000000001 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810784+0000", 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7893999999999998e-05 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810842+0000", 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7791000000000003e-05 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144845 2.1 2:80fd4430:::benchmark_data_smithi173_47195_object47837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 75.045308790000007, 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20768498299999999, 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144845 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:29:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603326+0000", 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.373e-06 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603335+0000", 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5449999999999992e-06 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603359+0000", 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3377e-05 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603370+0000", 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1759e-05 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603649+0000", 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027892500000000002 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603755+0000", 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010549500000000001 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603833+0000", 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8150000000000002e-05 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.609421+0000", 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0055884799999999998 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.640300+0000", 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030878974999999999 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810910+0000", 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.170609233 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810964+0000", 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4549999999999998e-05 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.811008+0000", 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4121e-05 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155294 2.e 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.524749880000002, 2024-08-30T23:29:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26817000400000002, 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155294 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123884+0000", 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3960000000000001e-06 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123889+0000", 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4820000000000002e-06 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123902+0000", 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3202e-05 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123910+0000", 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8869999999999992e-06 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123962+0000", 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1959000000000002e-05 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124045+0000", 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2850999999999997e-05 2024-08-30T23:29:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124115+0000", 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9513999999999999e-05 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.127474+0000", 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033596149999999998 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.128135+0000", 2024-08-30T23:29:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00066107399999999999 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.391962+0000", 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26382699500000001 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392009+0000", 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6669999999999999e-05 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392052+0000", 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3359000000000003e-05 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 182/143 [PushOp(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head, version: 132'4708, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head@132'4708, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 59.209071115999997, 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25797871999999999, 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:29:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439562+0000", 2024-08-30T23:29:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439584+0000", 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2395000000000002e-05 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693170+0000", 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25358568300000001 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693191+0000", 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1192999999999999e-05 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693349+0000", 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015837300000000001 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693372+0000", 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2869000000000001e-05 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.697540+0000", 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004167558 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168870 2.e 2:74d7d6f3:::benchmark_data_smithi173_47195_object71862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.032994918999997, 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223579366, 2024-08-30T23:29:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168870 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615640+0000", 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3590000000000002e-06 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615648+0000", 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8559999999999996e-06 2024-08-30T23:29:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615665+0000", 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6563999999999999e-05 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615676+0000", 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1075e-05 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615800+0000", 2024-08-30T23:29:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123905 2024-08-30T23:29:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615963+0000", 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016347900000000001 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616104+0000", 2024-08-30T23:29:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140846 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620671+0000", 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045667540000000001 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620686+0000", 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.484e-05 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839134+0000", 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21844793100000001 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839172+0000", 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7737999999999997e-05 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839217+0000", 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5018999999999997e-05 2024-08-30T23:29:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168873 2.1 2:8ff1cc28:::benchmark_data_smithi173_47195_object71865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.032927008000001, 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22707487400000001, 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168873 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615706+0000", 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2500000000000001e-07 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615708+0000", 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6970000000000002e-06 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615715+0000", 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8499999999999996e-06 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615720+0000", 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.395e-06 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616068+0000", 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034794699999999998 2024-08-30T23:29:46.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616173+0000", 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104145 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616295+0000", 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012253000000000001 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620669+0000", 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043740690000000004 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.621058+0000", 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038845600000000002 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842677+0000", 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22161980100000001 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842735+0000", 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7420000000000003e-05 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842780+0000", 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5139000000000003e-05 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.738 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168892 2.e 2:7a358138:::benchmark_data_smithi173_47195_object71884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.010502707999997, 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208213383, 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168892 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638131+0000", 2024-08-30T23:29:46.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638136+0000", 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638147+0000", 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1119e-05 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638153+0000", 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8320000000000002e-06 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638212+0000", 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9219999999999999e-05 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638342+0000", 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012929599999999999 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638476+0000", 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013422799999999999 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643016+0000", 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045403120000000003 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643358+0000", 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034153500000000002 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846238+0000", 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202880476 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846298+0000", 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9453000000000002e-05 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846343+0000", 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5059999999999999e-05 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168895 2.1 2:802748b8:::benchmark_data_smithi173_47195_object71887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.006266414999999, 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21216137199999999, 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.745 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168895 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:29:46.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642368+0000", 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3080000000000002e-06 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642376+0000", 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6769999999999995e-06 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642392+0000", 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642400+0000", 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6329999999999996e-06 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642458+0000", 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8498000000000001e-05 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642574+0000", 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115822 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642662+0000", 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655709+0000", 2024-08-30T23:29:46.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013046791 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655767+0000", 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8464000000000001e-05 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854414+0000", 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19864683999999999 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854479+0000", 2024-08-30T23:29:46.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5042999999999994e-05 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854527+0000", 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8195999999999997e-05 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168896 2.1 2:82321f29:::benchmark_data_smithi173_47195_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:29:46.751 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.005433392999997, 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211510109, 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168896 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643200+0000", 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6899999999999996e-07 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643204+0000", 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643215+0000", 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.041e-05 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643220+0000", 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646422+0000", 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032022190000000001 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646565+0000", 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142564 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646696+0000", 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130877 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655755+0000", 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090594430000000004 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655773+0000", 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7263999999999999e-05 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854613+0000", 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19883995700000001 2024-08-30T23:29:46.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854672+0000", 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9522999999999999e-05 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854709+0000", 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6693000000000003e-05 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168897 2.1 2:8f1372f3:::benchmark_data_smithi173_47195_object71889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 39.004908295, 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21982235999999999, 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168897 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.758 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643725+0000", 2024-08-30T23:29:46.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643729+0000", 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643740+0000", 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1079e-05 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643745+0000", 2024-08-30T23:29:46.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2009999999999998e-06 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651404+0000", 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007658916 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651490+0000", 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5811999999999998e-05 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651572+0000", 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2472999999999995e-05 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656527+0000", 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049544910000000001 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656537+0000", 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0497e-05 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863418+0000", 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20688067199999999 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863501+0000", 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2806000000000006e-05 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863546+0000", 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5430999999999999e-05 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:193992 2.7 e200/182 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head v 200'12142, mlcod=200'12139)", 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 3.8730604290000001, 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60714938399999996, 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9599999999999999e-07 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775578+0000", 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8089999999999998e-06 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378923+0000", 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60334498999999997 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378934+0000", 2024-08-30T23:29:46.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0772e-05 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379004+0000", 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9783000000000006e-05 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379024+0000", 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0044e-05 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382681+0000", 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036575549999999998 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382721+0000", 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9734999999999998e-05 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:194011 2.1 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:29:46.768 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 3.8513463560000001, 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82807010000000003, 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194011 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797287+0000", 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.277e-06 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797292+0000", 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6419999999999998e-06 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797301+0000", 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0070000000000003e-06 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797307+0000", 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2809999999999997e-06 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797355+0000", 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8557000000000003e-05 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797473+0000", 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117904 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797597+0000", 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012318100000000001 2024-08-30T23:29:46.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.802546+0000", 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004948956 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.939024+0000", 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13647832200000001 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625288+0000", 2024-08-30T23:29:46.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.686263866 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625326+0000", 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8028999999999998e-05 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625356+0000", 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0077999999999999e-05 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:29:46.775 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:29:46.775 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:29:46.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:46 smithi106 ceph-mon[16603]: pgmap v517: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 29 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 296/1371 objects degraded (21.590%); 39/1371 objects misplaced (2.845%); 3.0 MiB/s, 155 objects/s recovering 2024-08-30T23:29:46.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:46 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:29:46.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4202170072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:46.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:46 smithi106 ceph-mon[16603]: osdmap e201: 12 total, 12 up, 11 in 2024-08-30T23:29:46.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 296/1371 objects degraded (21.590%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:46.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:46 smithi106 ceph-mon[16603]: osdmap e202: 12 total, 12 up, 11 in 2024-08-30T23:29:46.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:46 smithi136 ceph-mon[20207]: pgmap v517: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 29 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 296/1371 objects degraded (21.590%); 39/1371 objects misplaced (2.845%); 3.0 MiB/s, 155 objects/s recovering 2024-08-30T23:29:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:46 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:29:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4202170072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:29:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:46 smithi136 ceph-mon[20207]: osdmap e201: 12 total, 12 up, 11 in 2024-08-30T23:29:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 296/1371 objects degraded (21.590%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:29:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:46 smithi136 ceph-mon[20207]: osdmap e202: 12 total, 12 up, 11 in 2024-08-30T23:29:46.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:46.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:46.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:46.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:47.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:47.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:47.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:47.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:47.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:47.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:47.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.415 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:47.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:47.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:47.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:47.552 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/973675624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:47.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:47.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:47.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:29:47.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:29:47.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:29:47.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:29:47.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:29:47.731 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:29:47.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:29:47.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:29:47.734 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:29:47.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:47.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/973675624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/973675624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:29:47.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:47.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:47.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:48.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:48.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:48.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:48.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:48.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:48.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:48.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:48.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:48.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:29:48.491 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:48 smithi106 ceph-mon[16603]: pgmap v520: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 572 op/s; 296/42 objects degraded (704.762%); 39/42 objects misplaced (92.857%); 999 KiB/s, 91 objects/s recovering 2024-08-30T23:29:48.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:48.530 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:48.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:48.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:48.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:48.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:48.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:48 smithi136 ceph-mon[20207]: pgmap v520: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 572 op/s; 296/42 objects degraded (704.762%); 39/42 objects misplaced (92.857%); 999 KiB/s, 91 objects/s recovering 2024-08-30T23:29:48.800 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:48.800 INFO:teuthology.orchestra.run.smithi106.stdout:{"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 296/42 objects degraded (704.762%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":604,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":202,"num_osds":12,"num_up_osds":12,"osd_up_since":1725060285,"num_in_osds":11,"osd_in_since":1725060424,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":17,"num_pools":2,"num_objects":14,"data_bytes":1376800,"bytes_used":1307189248,"bytes_avail":1054638202880,"bytes_total":1055945392128,"degraded_objects":296,"degraded_total":42,"degraded_ratio":7.0476190476190474,"misplaced_objects":39,"misplaced_total":42,"misplaced_ratio":0.9285714285714286,"recovering_objects_per_sec":91,"recovering_bytes_per_sec":1023419,"recovering_keys_per_sec":0,"num_objects_recovered":628,"num_bytes_recovered":7012352,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":572},"fsmap":{"epoch":1,"btime":"2024-08-30T23:18:02:086399+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.106:8443/"}},"servicemap":{"epoch":19,"modified":"2024-08-30T23:29:23.392381+0000","services":{"osd":{"daemons":{"summary":"","4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-30T23:29:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:48 smithi173 ceph-mon[20554]: pgmap v520: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.3 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 572 op/s; 296/42 objects degraded (704.762%); 39/42 objects misplaced (92.857%); 999 KiB/s, 91 objects/s recovering 2024-08-30T23:29:48.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:48.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:48.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:48.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:49.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:49.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:49.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:49.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:49.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:29:49.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:49.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:49.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:49.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:49.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:49.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:49.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:49.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:49.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:49.763 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1679425792' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1679425792' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:49.840 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1679425792' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:29:49.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:49.876 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:49.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:49.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:49.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:50.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:50.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:50.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:50.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:50.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:50.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:50.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:50.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:50.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:50.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:50.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:50.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:50.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:50.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:50 smithi136 ceph-mon[20207]: pgmap v521: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 856 KiB/s, 209 objects/s recovering 2024-08-30T23:29:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:50 smithi173 ceph-mon[20554]: pgmap v521: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 856 KiB/s, 209 objects/s recovering 2024-08-30T23:29:50.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:50.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:50 smithi106 ceph-mon[16603]: pgmap v521: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 856 KiB/s, 209 objects/s recovering 2024-08-30T23:29:50.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:50.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:50.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:51.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:51.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:51.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:51.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:51.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:51.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:51.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:51.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:51.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:51.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:51.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:51.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:29:51.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:51.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:51.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 296/6 objects degraded (4933.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:29:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 296/6 objects degraded (4933.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:29:51.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:51.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 296/6 objects degraded (4933.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:29:51.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:51.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:51.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:51.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:52.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:52.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:52.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:52.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:52.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:52.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:52.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:52.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:52.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.613 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:52.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:52.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:52.736 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:52.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:52 smithi136 ceph-mon[20207]: pgmap v522: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 254 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 856 KiB/s, 209 objects/s recovering 2024-08-30T23:29:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:52 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:29:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:52 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:29:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:52 smithi173 ceph-mon[20554]: pgmap v522: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 254 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 856 KiB/s, 209 objects/s recovering 2024-08-30T23:29:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:52 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:29:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:52 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:29:52.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:52 smithi106 ceph-mon[16603]: pgmap v522: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 254 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 856 KiB/s, 209 objects/s recovering 2024-08-30T23:29:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:52 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:29:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:52 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:29:52.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:52.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:29:52.984 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:52.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:29:53.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:53.109 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:53.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:53.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:53.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:53.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:53.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:53.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:53.483 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_0' removed 2024-08-30T23:29:53.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:53.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:53 smithi106 ceph-mon[16603]: osdmap e203: 12 total, 12 up, 11 in 2024-08-30T23:29:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2380830011' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:29:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:53 smithi106 ceph-mon[16603]: pgmap v524: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 0 B/s, 131 objects/s recovering 2024-08-30T23:29:53.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:53.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:53.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:53 smithi136 ceph-mon[20207]: osdmap e203: 12 total, 12 up, 11 in 2024-08-30T23:29:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2380830011' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:29:53.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:53 smithi136 ceph-mon[20207]: pgmap v524: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 0 B/s, 131 objects/s recovering 2024-08-30T23:29:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:53 smithi173 ceph-mon[20554]: osdmap e203: 12 total, 12 up, 11 in 2024-08-30T23:29:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2380830011' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:29:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:53 smithi173 ceph-mon[20554]: pgmap v524: 17 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 296/6 objects degraded (4933.333%); 39/6 objects misplaced (650.000%); 0 B/s, 131 objects/s recovering 2024-08-30T23:29:53.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:53.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:53.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:53.989 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:53.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:54.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:54.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:54.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:54.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:54.296 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-30T23:29:54.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:29:54.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:54.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:54.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:54.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:54.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2380830011' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:29:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:54 smithi106 ceph-mon[16603]: osdmap e204: 12 total, 12 up, 11 in 2024-08-30T23:29:54.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:54.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:54.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2380830011' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:29:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:54 smithi136 ceph-mon[20207]: osdmap e204: 12 total, 12 up, 11 in 2024-08-30T23:29:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2380830011' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:29:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:54 smithi173 ceph-mon[20554]: osdmap e204: 12 total, 12 up, 11 in 2024-08-30T23:29:54.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:54.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:54.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:54.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:54.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:55.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:55.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:55.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:55.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:55.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:55.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:55.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:55.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:55.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:55.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:55.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:55.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:29:55.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:55 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:55 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:55 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:55 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.837 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:55 smithi136 ceph-mon[20207]: pgmap v526: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:55.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:55.838 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:29:55.838 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:55 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.838 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:55 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:55 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:55 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.839 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:55 smithi173 ceph-mon[20554]: pgmap v526: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:29:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:55 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:55 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:55 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:55 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:55 smithi106 ceph-mon[16603]: pgmap v526: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:55.861 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:55.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:55.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:55.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:56.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:56.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:56.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:56.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:56.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:56.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:56.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:56.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:56.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:56.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:56.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:29:56.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:29:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:56 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 296/6 objects degraded (4933.333%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:29:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:56 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:29:56.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:56 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 296/6 objects degraded (4933.333%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:29:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:56 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:29:56.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:56.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:56.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:56.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:56 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 296/6 objects degraded (4933.333%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:29:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:56 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:29:57.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:57.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:57.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:57.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:57.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:57.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:57.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:57.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:57.632 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:57.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:57.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:57 smithi173 ceph-mon[20554]: pgmap v527: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:57.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:57 smithi106 ceph-mon[16603]: pgmap v527: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:57.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:57.888 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:57.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:57.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:58.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:57 smithi136 ceph-mon[20207]: pgmap v527: 1 pgs: 1 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:58.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:58.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:29:58.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.264 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:58.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:58.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:29:58.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:29:58.438 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:29:58.438 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:29:58.439 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-08-30T23:29:58.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-08-30T23:29:58.457 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:58.457 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:58.460 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:58.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:29:58.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:29:58.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:58.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:58.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:58.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:58 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:29:58.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:58 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:29:58.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:58 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:58.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3683002727' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:29:58.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.767 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:58 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:29:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:58 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:29:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:58 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3683002727' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:29:58.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:29:58.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:58.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:58.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:29:58.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:59.004 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:29:59.013 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:29:59.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:58 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:29:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:58 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:29:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:58 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:29:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3683002727' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:29:59.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:29:59.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:59.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:29:59.172 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:29:59.173 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:29:59.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:29:59.269 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:29:59.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:29:59.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:29:59.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:29:59.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:29:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.525 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:29:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:59 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3683002727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:29:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:59 smithi106 ceph-mon[16603]: osdmap e205: 12 total, 12 up, 11 in 2024-08-30T23:29:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:59 smithi106 ceph-mon[16603]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:29:59 smithi106 ceph-mon[16603]: pgmap v529: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:59.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:29:59.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:29:59.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:29:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.771 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:29:59.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:29:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:59 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:29:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3683002727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:29:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:59 smithi173 ceph-mon[20554]: osdmap e205: 12 total, 12 up, 11 in 2024-08-30T23:29:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:59 smithi173 ceph-mon[20554]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:29:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:29:59 smithi173 ceph-mon[20554]: pgmap v529: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:29:59.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:29:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:29:59.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:29:59.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:00.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:00.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:59 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3683002727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:59 smithi136 ceph-mon[20207]: osdmap e205: 12 total, 12 up, 11 in 2024-08-30T23:30:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:59 smithi136 ceph-mon[20207]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:29:59 smithi136 ceph-mon[20207]: pgmap v529: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:00.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:00.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:00.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:00.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:00.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:00.405 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:00.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:00.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:00.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:00.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:00.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:00.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:00 smithi173 ceph-mon[20554]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-08-30T23:30:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:00 smithi173 ceph-mon[20554]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-30T23:30:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:00 smithi106 ceph-mon[16603]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-08-30T23:30:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:00 smithi106 ceph-mon[16603]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-30T23:30:00.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:00.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:00.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:01.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:00 smithi136 ceph-mon[20207]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-08-30T23:30:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:00 smithi136 ceph-mon[20207]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-30T23:30:01.044 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:01.055 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-08-30T23:29:59.014493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":4,"kb":1031196672,"kb_used":1149484,"kb_used_data":8780,"kb_used_omap":17,"kb_used_meta":1138286,"kb_avail":1030047188,"statfs":{"total":1055945392128,"available":1054768320512,"internally_reserved":0,"allocated":8990720,"data_stored":6241356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1165605753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-296,"num_objects_misplaced":-39,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-194012,"num_write_kb":-6208321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-31076,"num_bytes_recovered":-1906704384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.616064"},"pg_stats":[{"pgid":"1.0","version":"23'92","reported_seq":367,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-30T23:29:53.487353+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:53.487353+0000","last_peered":"2024-08-30T23:29:53.487353+0000","last_clean":"2024-08-30T23:29:53.487353+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:53.487353+0000","last_undegraded":"2024-08-30T23:29:53.487353+0000","last_fullsized":"2024-08-30T23:29:53.487353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112976,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632176,"statfs":{"total":95995035648,"available":95879348224,"internally_reserved":0,"allocated":1196032,"data_stored":946308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449159,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710795,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":112456,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632696,"statfs":{"total":95995035648,"available":95879880704,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972433,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":86272,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93658880,"statfs":{"total":95995035648,"available":95906693120,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234068,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113160,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93631992,"statfs":{"total":95995035648,"available":95879159808,"internally_reserved":0,"allocated":1196032,"data_stored":946328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528410,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64584,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680568,"statfs":{"total":95995035648,"available":95928901632,"internally_reserved":0,"allocated":1196032,"data_stored":946328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":63880,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681272,"statfs":{"total":95995035648,"available":95929622528,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953571,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64044,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681108,"statfs":{"total":95995035648,"available":95929454592,"internally_reserved":0,"allocated":757760,"data_stored":480592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":148552,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596600,"statfs":{"total":95995035648,"available":95842918400,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":126528,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618624,"statfs":{"total":95995035648,"available":95865470976,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":148552,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596600,"statfs":{"total":95995035648,"available":95842918400,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":108480,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636672,"statfs":{"total":95995035648,"available":95883952128,"internally_reserved":0,"allocated":663552,"data_stored":417400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:30:01.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:01.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:01.223 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:01.224 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:01.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:01.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:01.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:01.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:01.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:01.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:01.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:01.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:01.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:01.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:01 smithi173 ceph-mon[20554]: pgmap v530: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:01 smithi106 ceph-mon[16603]: pgmap v530: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:01.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:01.929 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:01.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:02.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:02.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:01 smithi136 ceph-mon[20207]: pgmap v530: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:02.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:30:02.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:02.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:30:02.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:02.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:02.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:02.309 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:02.330 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:02.330 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:02.356 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:02.356 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:02.386 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:02.386 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:02.389 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:02.393 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:02.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:02.418 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:02.418 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:02.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:02.440 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:02.440 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:02.461 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:02.462 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:02.482 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:02.483 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:02.504 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:02.504 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:02.526 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:02.526 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:02.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:02.549 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:02.549 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:02.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:02.571 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:02.571 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:02.591 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_1' created 2024-08-30T23:30:02.591 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:02.592 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:02.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:02.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:02.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:02.810 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/534721367' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-30T23:30:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3021893446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:02 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:02 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/534721367' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-30T23:30:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3021893446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:02 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:02 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:02.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:02.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:02.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:02.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:03.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/534721367' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-30T23:30:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3021893446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:02 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:02 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:03.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:03.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:03.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:03.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:03.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:03.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:03.436 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:03.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:03.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:03.560 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:03.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:03.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:03.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:03.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:03.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/534721367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-30T23:30:03.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:03 smithi106 ceph-mon[16603]: osdmap e206: 12 total, 12 up, 11 in 2024-08-30T23:30:03.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:03 smithi106 ceph-mon[16603]: pgmap v532: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:03.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:03.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-08-30T23:30:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:03.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:03.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:03.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:03.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/534721367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-30T23:30:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:03 smithi136 ceph-mon[20207]: osdmap e206: 12 total, 12 up, 11 in 2024-08-30T23:30:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:03 smithi136 ceph-mon[20207]: pgmap v532: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:04.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:04.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:30:04.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/534721367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-30T23:30:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:03 smithi173 ceph-mon[20554]: osdmap e206: 12 total, 12 up, 11 in 2024-08-30T23:30:04.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:03 smithi173 ceph-mon[20554]: pgmap v532: 17 pgs: 16 unknown, 1 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:04.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:04.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:04.261 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:30:04.261 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:30:04.261 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:30:04.261 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:30:04.262 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:30:04.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:04.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:04.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:04.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:04.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:04.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:04.603 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:30:04.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:04.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:04.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:04.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:04.829 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:04 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:30:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:04 smithi106 ceph-mon[16603]: osdmap e207: 12 total, 12 up, 11 in 2024-08-30T23:30:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3898415607' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:04.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:04.950 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:04.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:05.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:04 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:30:05.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:04 smithi136 ceph-mon[20207]: osdmap e207: 12 total, 12 up, 11 in 2024-08-30T23:30:05.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3898415607' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:05.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:05.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:04 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:30:05.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:04 smithi173 ceph-mon[20554]: osdmap e207: 12 total, 12 up, 11 in 2024-08-30T23:30:05.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3898415607' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:05.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:05.180 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:05.202 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:05.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:30:05.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:05.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:05.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:05.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:05.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:05.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:05.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:05.708 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:05.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:05 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:05.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3898415607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:05.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:05 smithi106 ceph-mon[16603]: osdmap e208: 12 total, 12 up, 11 in 2024-08-30T23:30:05.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:05 smithi106 ceph-mon[16603]: pgmap v535: 17 pgs: 2 creating+activating, 2 creating+peering, 8 unknown, 5 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:05.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:05.830 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:05.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:30:05.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:05.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:05 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:05.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3898415607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:05.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:05 smithi136 ceph-mon[20207]: osdmap e208: 12 total, 12 up, 11 in 2024-08-30T23:30:05.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:05 smithi136 ceph-mon[20207]: pgmap v535: 17 pgs: 2 creating+activating, 2 creating+peering, 8 unknown, 5 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:05.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:05.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:05.974 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:05 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:05.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3898415607' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:05.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:05 smithi173 ceph-mon[20554]: osdmap e208: 12 total, 12 up, 11 in 2024-08-30T23:30:05.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:05 smithi173 ceph-mon[20554]: pgmap v535: 17 pgs: 2 creating+activating, 2 creating+peering, 8 unknown, 5 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:06.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:06.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:06.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:06.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:06.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:06.328 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:06.341 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:06.374 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:06.375 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:06.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:06.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:06.510 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:30:06.511 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:30:06.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:06.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:06.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:06.721 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:06.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:06.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.851 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:06.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:06.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:06.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:07.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:07.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:07.159 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-08-30T23:30:05.400935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":8,"kb":1031196672,"kb_used":1147444,"kb_used_data":8916,"kb_used_omap":17,"kb_used_meta":1138286,"kb_avail":1030049228,"statfs":{"total":1055945392128,"available":1054770409472,"internally_reserved":0,"allocated":9129984,"data_stored":6324065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1165605753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001087"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-30T23:30:03.608966+0000","last_change":"2024-08-30T23:30:03.608169+0000","last_active":"2024-08-30T23:30:03.608966+0000","last_peered":"2024-08-30T23:30:03.608966+0000","last_clean":"2024-08-30T23:30:03.608966+0000","last_became_active":"2024-08-30T23:30:03.607837+0000","last_became_peered":"2024-08-30T23:30:03.607837+0000","last_unstale":"2024-08-30T23:30:03.608966+0000","last_undegraded":"2024-08-30T23:30:03.608966+0000","last_fullsized":"2024-08-30T23:30:03.608966+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:20.363179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-30T23:30:03.604145+0000","last_change":"2024-08-30T23:30:03.603560+0000","last_active":"2024-08-30T23:30:03.604145+0000","last_peered":"2024-08-30T23:30:03.604145+0000","last_clean":"2024-08-30T23:30:03.604145+0000","last_became_active":"2024-08-30T23:30:03.603254+0000","last_became_peered":"2024-08-30T23:30:03.603254+0000","last_unstale":"2024-08-30T23:30:03.604145+0000","last_undegraded":"2024-08-30T23:30:03.604145+0000","last_fullsized":"2024-08-30T23:30:03.604145+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:04.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":4,"reported_epoch":207,"state":"creating+activating","last_fresh":"2024-08-30T23:30:03.602676+0000","last_change":"2024-08-30T23:30:03.601107+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:03.602676+0000","last_undegraded":"2024-08-30T23:30:03.602676+0000","last_fullsized":"2024-08-30T23:30:03.602676+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":["3","7","11"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":206,"state":"creating+peering","last_fresh":"2024-08-30T23:30:02.597124+0000","last_change":"2024-08-30T23:30:02.597124+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.597124+0000","last_undegraded":"2024-08-30T23:30:02.597124+0000","last_fullsized":"2024-08-30T23:30:02.597124+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-30T23:30:03.609026+0000","last_change":"2024-08-30T23:30:03.608434+0000","last_active":"2024-08-30T23:30:03.609026+0000","last_peered":"2024-08-30T23:30:03.609026+0000","last_clean":"2024-08-30T23:30:03.609026+0000","last_became_active":"2024-08-30T23:30:03.607817+0000","last_became_peered":"2024-08-30T23:30:03.607817+0000","last_unstale":"2024-08-30T23:30:03.609026+0000","last_undegraded":"2024-08-30T23:30:03.609026+0000","last_fullsized":"2024-08-30T23:30:03.609026+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:21.051198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-30T23:30:03.609264+0000","last_change":"2024-08-30T23:30:03.608350+0000","last_active":"2024-08-30T23:30:03.609264+0000","last_peered":"2024-08-30T23:30:03.609264+0000","last_clean":"2024-08-30T23:30:03.609264+0000","last_became_active":"2024-08-30T23:30:03.607728+0000","last_became_peered":"2024-08-30T23:30:03.607728+0000","last_unstale":"2024-08-30T23:30:03.609264+0000","last_undegraded":"2024-08-30T23:30:03.609264+0000","last_fullsized":"2024-08-30T23:30:03.609264+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:23.785001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":206,"state":"creating+peering","last_fresh":"2024-08-30T23:30:02.597067+0000","last_change":"2024-08-30T23:30:02.597067+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.597067+0000","last_undegraded":"2024-08-30T23:30:02.597067+0000","last_fullsized":"2024-08-30T23:30:02.597067+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:30:02.569516+0000","last_change":"2024-08-30T23:30:02.569516+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:02.569516+0000","last_undegraded":"2024-08-30T23:30:02.569516+0000","last_fullsized":"2024-08-30T23:30:02.569516+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":3,"reported_epoch":207,"state":"creating+activating","last_fresh":"2024-08-30T23:30:03.601124+0000","last_change":"2024-08-30T23:30:03.601124+0000","last_active":"2024-08-30T23:30:02.569516+0000","last_peered":"2024-08-30T23:30:02.569516+0000","last_clean":"2024-08-30T23:30:02.569516+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:30:03.601124+0000","last_undegraded":"2024-08-30T23:30:03.601124+0000","last_fullsized":"2024-08-30T23:30:03.601124+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":["3","6","9"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":369,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-30T23:29:59.339919+0000","last_change":"2024-08-30T23:28:43.209434+0000","last_active":"2024-08-30T23:29:59.339919+0000","last_peered":"2024-08-30T23:29:59.339919+0000","last_clean":"2024-08-30T23:29:59.339919+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:29:59.339919+0000","last_undegraded":"2024-08-30T23:29:59.339919+0000","last_fullsized":"2024-08-30T23:29:59.339919+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:28:43.209372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:28:43.209372+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:30:03.472752+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187522,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112880,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632272,"statfs":{"total":95995035648,"available":95879446528,"internally_reserved":0,"allocated":1208320,"data_stored":953683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449160,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710797,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112384,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632768,"statfs":{"total":95995035648,"available":95879954432,"internally_reserved":0,"allocated":700416,"data_stored":440765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972434,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":85992,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93659160,"statfs":{"total":95995035648,"available":95906979840,"internally_reserved":0,"allocated":675840,"data_stored":424775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112880,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632272,"statfs":{"total":95995035648,"available":95879446528,"internally_reserved":0,"allocated":1208320,"data_stored":953703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528411,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64304,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680848,"statfs":{"total":95995035648,"available":95929188352,"internally_reserved":0,"allocated":1208320,"data_stored":953703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724639,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63796,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681356,"statfs":{"total":95995035648,"available":95929708544,"internally_reserved":0,"allocated":688128,"data_stored":432974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953573,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63812,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681340,"statfs":{"total":95995035648,"available":95929692160,"internally_reserved":0,"allocated":700416,"data_stored":440765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148468,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596684,"statfs":{"total":95995035648,"available":95843004416,"internally_reserved":0,"allocated":688128,"data_stored":432974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":126260,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618892,"statfs":{"total":95995035648,"available":95865745408,"internally_reserved":0,"allocated":688128,"data_stored":432974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":148456,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596696,"statfs":{"total":95995035648,"available":95843016704,"internally_reserved":0,"allocated":675840,"data_stored":424775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108212,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636940,"statfs":{"total":95995035648,"available":95884226560,"internally_reserved":0,"allocated":688128,"data_stored":432974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:30:07.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-08-30T23:30:07.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:07.229 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:07.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:07.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:30:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:07.358 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:07.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:07.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:07.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:07.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:07.637 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-08-30T23:30:07.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:07.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:07.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:07.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:07.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:07 smithi106 ceph-mon[16603]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:07.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:07 smithi106 ceph-mon[16603]: osdmap e209: 12 total, 12 up, 11 in 2024-08-30T23:30:07.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:07 smithi106 ceph-mon[16603]: pgmap v537: 17 pgs: 2 creating+activating, 2 creating+peering, 13 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:07.744 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2008902650' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:30:07.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:07.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:07.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:07.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:07.785 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:07.785 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:07.805 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:07.805 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:07.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:07.827 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:07.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:07.848 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:07.848 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:07.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:07.869 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:07.869 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:07.890 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:07.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:07.911 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:07.911 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:07.932 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:07.932 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:07.953 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:07.953 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:07.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:07.975 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:07.975 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:07.987 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:07.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:07 smithi136 ceph-mon[20207]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:07 smithi136 ceph-mon[20207]: osdmap e209: 12 total, 12 up, 11 in 2024-08-30T23:30:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:07 smithi136 ceph-mon[20207]: pgmap v537: 17 pgs: 2 creating+activating, 2 creating+peering, 13 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2008902650' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:30:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:07 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:07 smithi173 ceph-mon[20554]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:07 smithi173 ceph-mon[20554]: osdmap e209: 12 total, 12 up, 11 in 2024-08-30T23:30:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:07 smithi173 ceph-mon[20554]: pgmap v537: 17 pgs: 2 creating+activating, 2 creating+peering, 13 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2008902650' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:30:08.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:08.109 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:08.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:08.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:08.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:08.360 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:08.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:08.483 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:08.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:08.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:08.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:08.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:08.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2008902650' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:30:08.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:08 smithi106 ceph-mon[16603]: osdmap e210: 12 total, 12 up, 11 in 2024-08-30T23:30:08.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1759716931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:08.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:08.809 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:08.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:08.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:08.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:08.989 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:08.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:09.025 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:09.026 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2008902650' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:30:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:08 smithi136 ceph-mon[20207]: osdmap e210: 12 total, 12 up, 11 in 2024-08-30T23:30:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1759716931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2008902650' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:30:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:08 smithi173 ceph-mon[20554]: osdmap e210: 12 total, 12 up, 11 in 2024-08-30T23:30:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1759716931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:09.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:09.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.118 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:09.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:09.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:09.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:09.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:09.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:09.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:09.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.618 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:09.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:09.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:09.744 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:09.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:30:09.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:09.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:09.907 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:09 smithi106 ceph-mon[16603]: osdmap e211: 12 total, 12 up, 11 in 2024-08-30T23:30:09.908 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:09 smithi106 ceph-mon[16603]: pgmap v540: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:09.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:09.991 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:09.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:10.010 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:09 smithi136 ceph-mon[20207]: osdmap e211: 12 total, 12 up, 11 in 2024-08-30T23:30:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:09 smithi136 ceph-mon[20207]: pgmap v540: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:09 smithi173 ceph-mon[20554]: osdmap e211: 12 total, 12 up, 11 in 2024-08-30T23:30:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:09 smithi173 ceph-mon[20554]: pgmap v540: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:10.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:10.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:10.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:10.245 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:10.265 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:10.269 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:10.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:10.368 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:10.469 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:10.469 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:10.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:10.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:10.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:10.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:10.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:10 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:30:10.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:10.698 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:10.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:10.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:10.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:10 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:30:10.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:10.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:10.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:30:10.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:10.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:10.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:11.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:11.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:10 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:30:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:11.111 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:11.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:11.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:30:11.221 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-08-30T23:30:11.224 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:30:11.224 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:30:11.234 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:30:11.235 INFO:teuthology.orchestra.run.smithi106.stdout:{"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":626,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":212,"num_osds":12,"num_up_osds":12,"osd_up_since":1725060285,"num_in_osds":11,"osd_in_since":1725060424,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17}],"num_pgs":17,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":1175388160,"bytes_avail":1054770003968,"bytes_total":1055945392128},"fsmap":{"epoch":1,"btime":"2024-08-30T23:18:02:086399+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.106:8443/"}},"servicemap":{"epoch":19,"modified":"2024-08-30T23:29:23.392381+0000","services":{"osd":{"daemons":{"summary":"","4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-08-30T23:30:11.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:11.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:11.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:30:11.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:30:11.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_60780 2024-08-30T23:30:11.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:30:11.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:30:11.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:11.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:11.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:11.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:11.536 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:11.539 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:11.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:11.660 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:30:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:11.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:11.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:11.804 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:11.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:11.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:11.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:11.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:11 smithi136 ceph-mon[20207]: osdmap e212: 12 total, 12 up, 11 in 2024-08-30T23:30:11.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1785660659' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:11.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2874836416' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:30:11.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:11 smithi136 ceph-mon[20207]: pgmap v542: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:12.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:12.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:12.072 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:12.072 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:11 smithi173 ceph-mon[20554]: osdmap e212: 12 total, 12 up, 11 in 2024-08-30T23:30:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1785660659' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2874836416' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:30:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:11 smithi173 ceph-mon[20554]: pgmap v542: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:11 smithi106 ceph-mon[16603]: osdmap e212: 12 total, 12 up, 11 in 2024-08-30T23:30:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1785660659' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2874836416' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:30:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:11 smithi106 ceph-mon[16603]: pgmap v542: 17 pgs: 17 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-08-30T23:30:12.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:12.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.183 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:12.226 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-30T23:30:12.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:30:12.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:12.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 2088 2072 129.493 129.5 0.0226946 0.00766472 2024-08-30T23:30:12.307 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:12.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:12.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:12.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:12.558 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:12.588 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:12.589 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:12.611 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:12.612 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:12.615 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:12.640 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:12.640 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:12.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:12.669 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:12.669 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:12.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:12.691 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:12.692 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:12.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:12.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:12.734 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:12.734 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:12.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:12 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:12.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1785660659' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:12.734 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:12 smithi106 ceph-mon[16603]: osdmap e213: 12 total, 12 up, 11 in 2024-08-30T23:30:12.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:12 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:30:12.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:12 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:30:12.735 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3432427196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:12.754 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:12.755 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:12.776 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:12.777 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:12.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:12 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1785660659' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:12 smithi136 ceph-mon[20207]: osdmap e213: 12 total, 12 up, 11 in 2024-08-30T23:30:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:12 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:30:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:12 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:30:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3432427196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:12.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:12.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:12.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:12.818 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:12.818 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:12.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:30:12.839 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:12.839 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:12.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:12.940 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:12.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:13.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:13.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:12 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1785660659' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:12 smithi173 ceph-mon[20554]: osdmap e213: 12 total, 12 up, 11 in 2024-08-30T23:30:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:12 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:30:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:12 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:30:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3432427196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:13.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:13.182 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:13.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:13.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3273 3257 101.772 74.0625 0.00112635 0.00907075 2024-08-30T23:30:13.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:13.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.329 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:13.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:13.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:13.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:13.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:13.669 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:839 2.d e104/97 2:b8994a18:::benchmark_data_smithi173_47195_object838:head v 104'56, mlcod=104'53)", 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:58.576149+0000", 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 255.07510176100001, 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14537533, 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.576149+0000", 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.576149+0000", 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.576150+0000", 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2600000000000002e-07 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.576218+0000", 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7910999999999994e-05 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721088+0000", 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14487076400000001 2024-08-30T23:30:13.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721096+0000", 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7680000000000008e-06 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721146+0000", 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0204999999999998e-05 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721168+0000", 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1239000000000001e-05 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721482+0000", 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031484699999999999 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:58.721524+0000", 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.197e-05 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.674 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:16926 2.d e111/97 2:b54c02df:::benchmark_data_smithi173_47195_object16925:head v 111'1092, mlcod=111'1089)", 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.682888+0000", 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 242.96836276499999, 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.166138548, 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.682888+0000", 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.682888+0000", 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.682889+0000", 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9399999999999993e-07 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.682955+0000", 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5554000000000003e-05 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.848229+0000", 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.165274593 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.848241+0000", 2024-08-30T23:30:13.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1626e-05 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.848335+0000", 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3931999999999994e-05 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.848349+0000", 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4452e-05 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.848976+0000", 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00062670900000000003 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.849027+0000", 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0688e-05 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:17196 2.d e111/97 2:b96cb6c6:::benchmark_data_smithi173_47195_object17195:head v 111'1106, mlcod=111'1102)", 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.990789+0000", 2024-08-30T23:30:13.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 242.660461615, 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13463890000000001, 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.990789+0000", 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.990789+0000", 2024-08-30T23:30:13.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.990790+0000", 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2399999999999998e-07 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.990861+0000", 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0748999999999996e-05 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.124503+0000", 2024-08-30T23:30:13.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13364197999999999 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.124514+0000", 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1663e-05 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.124624+0000", 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010966100000000001 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.124657+0000", 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2759999999999998e-05 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.125393+0000", 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000736076 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.125428+0000", 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5386999999999998e-05 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.683 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:17740 2.f e111/97 2:f20a1960:::benchmark_data_smithi173_47195_object17739:head v 111'1101, mlcod=111'1098)", 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:11.576452+0000", 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 242.07479841099999, 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13546117599999999, 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.576452+0000", 2024-08-30T23:30:13.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.576452+0000", 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.576453+0000", 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3200000000000004e-07 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.576473+0000", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9715999999999999e-05 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.710935+0000", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.134462156 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.710944+0000", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4999999999999999e-06 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.711000+0000", 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6919000000000002e-05 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.711016+0000", 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5322e-05 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.711871+0000", 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00085535899999999998 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.711914+0000", 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2472000000000003e-05 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:22638 2.d e112/97 2:bbd97f5b:::benchmark_data_smithi173_47195_object22637:head v 112'1453, mlcod=112'1449)", 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:14.890065+0000", 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 238.76118630400001, 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.115172424, 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.688 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:14.890065+0000", 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:14.890065+0000", 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:14.890065+0000", 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8400000000000003e-07 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:14.890099+0000", 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3436000000000003e-05 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.004426+0000", 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.114327708 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.004431+0000", 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7550000000000002e-06 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.004477+0000", 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6156999999999999e-05 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.004495+0000", 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.73e-05 2024-08-30T23:30:13.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.005210+0000", 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000715735 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.005237+0000", 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6548999999999999e-05 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:22833 2.5 e112/97 2:a1501afa:::benchmark_data_smithi173_47195_object22832:head v 112'1424, mlcod=112'1421)", 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:15.168389+0000", 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 238.48286192800001, 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.140571108, 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.168389+0000", 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.168389+0000", 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.168390+0000", 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7599999999999997e-07 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.168419+0000", 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9445e-05 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.308124+0000", 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13970529500000001 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.308136+0000", 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1691999999999999e-05 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.308198+0000", 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2430999999999999e-05 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.308212+0000", 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.309e-05 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.308928+0000", 2024-08-30T23:30:13.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071610200000000004 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.308960+0000", 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2376999999999998e-05 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:22836 2.d e112/97 2:b62fe4d2:::benchmark_data_smithi173_47195_object22835:head v 112'1468, mlcod=112'1464)", 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:15.173033+0000", 2024-08-30T23:30:13.697 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 238.47821808200001, 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12638198, 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.173033+0000", 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.173033+0000", 2024-08-30T23:30:13.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.173033+0000", 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9800000000000004e-07 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.173055+0000", 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1807999999999999e-05 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.298431+0000", 2024-08-30T23:30:13.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12537636699999999 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.298439+0000", 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0579999999999998e-06 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.298511+0000", 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1760000000000004e-05 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.298529+0000", 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7578999999999999e-05 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.299379+0000", 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084986399999999998 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:15.299415+0000", 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6046e-05 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:49996 2.5 e122/97 2:aad518df:::benchmark_data_smithi173_47195_object49995:head v 122'3107, mlcod=122'3104)", 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 218.39295990900001, 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20284867600000001, 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.702 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.63e-07 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258370+0000", 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8464e-05 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459751+0000", 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.201380686 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459757+0000", 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.319e-06 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459801+0000", 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3912999999999998e-05 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459814+0000", 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3548e-05 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.461105+0000", 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012902320000000001 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.461140+0000", 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5051e-05 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.706 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:56067 2.f e124/97 2:f6065497:::benchmark_data_smithi173_47195_object56066:head v 124'3491, mlcod=124'3489)", 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:39.786082+0000", 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 213.865169245, 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.175848113, 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.786082+0000", 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.786082+0000", 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.786082+0000", 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.786142+0000", 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9747999999999998e-05 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.960582+0000", 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17444029599999999 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.960605+0000", 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2711999999999999e-05 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.960654+0000", 2024-08-30T23:30:13.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9484999999999998e-05 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.960668+0000", 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4002000000000001e-05 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.961901+0000", 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001232866 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.961930+0000", 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.853e-05 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:82957 2.d e140/97 2:b2e53dd7:::benchmark_data_smithi173_47195_object82956:head v 140'5254, mlcod=140'5251)", 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:08.307365+0000", 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 185.34388632100001, 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14140802, 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.711 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.307365+0000", 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.307365+0000", 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.307366+0000", 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.307406+0000", 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0062999999999999e-05 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.446129+0000", 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13872331700000001 2024-08-30T23:30:13.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.446135+0000", 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.083e-06 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.446187+0000", 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1447999999999999e-05 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.446201+0000", 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4119e-05 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.448753+0000", 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002552122 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:08.448773+0000", 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9644e-05 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:107355 2.d e151/97 2:b04d3eb3:::benchmark_data_smithi173_47195_object10347:head v 151'6767, mlcod=151'6763)", 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:42.120614+0000", 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 151.53063716299999, 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13865723199999999, 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.120614+0000", 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.120614+0000", 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.120614+0000", 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2200000000000003e-07 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.120620+0000", 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.253843+0000", 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13322297299999999 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.253850+0000", 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9890000000000003e-06 2024-08-30T23:30:13.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.253913+0000", 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2606000000000003e-05 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.253937+0000", 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4082000000000001e-05 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.259181+0000", 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052439870000000003 2024-08-30T23:30:13.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.259271+0000", 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0260000000000007e-05 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:107456 2.d e152/97 2:ba35afef:::benchmark_data_smithi173_47195_object10448:head v 152'6772, mlcod=151'6768)", 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:42.324849+0000", 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 151.32640177100001, 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12116316000000001, 2024-08-30T23:30:13.720 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.324849+0000", 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.324849+0000", 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.324850+0000", 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1039999999999999e-06 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.324858+0000", 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7950000000000008e-06 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.442978+0000", 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.118119865 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.442983+0000", 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1240000000000004e-06 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.443031+0000", 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8401e-05 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.443048+0000", 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6866000000000001e-05 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.445971+0000", 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029225589999999999 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:42.446012+0000", 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1446000000000003e-05 2024-08-30T23:30:13.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:109587 2.d e153/97 2:b377b128:::benchmark_data_smithi173_47195_object12579:head v 153'6909, mlcod=153'6905)", 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 148.746136315, 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18410296100000001, 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905116+0000", 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5770000000000001e-06 2024-08-30T23:30:13.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905126+0000", 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0356999999999999e-05 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086753+0000", 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18162683099999999 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086760+0000", 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5379999999999999e-06 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086804+0000", 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4310000000000001e-05 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086821+0000", 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7135000000000001e-05 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.089190+0000", 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023685709999999999 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.089217+0000", 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7642000000000001e-05 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:144834 2.f e174/97 2:f4dee189:::benchmark_data_smithi173_47195_object47826:head v 174'9111, mlcod=173'9107)", 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:31.588880+0000", 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.06237053, 2024-08-30T23:30:13.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13007226899999999, 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.588880+0000", 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.588880+0000", 2024-08-30T23:30:13.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.588881+0000", 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6100000000000001e-07 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.588886+0000", 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.152e-06 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.715813+0000", 2024-08-30T23:30:13.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12692650599999999 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.715821+0000", 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3420000000000008e-06 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.715876+0000", 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5566999999999998e-05 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.715895+0000", 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8601999999999999e-05 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.718923+0000", 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030275250000000001 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:31.718953+0000", 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0014000000000001e-05 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.733 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:145440 2.f e174/97 2:fe697e68:::benchmark_data_smithi173_47195_object48432:head v 174'9158, mlcod=174'9155)", 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:32.438073+0000", 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.213177539, 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16924828, 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.438073+0000", 2024-08-30T23:30:13.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.438073+0000", 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.438075+0000", 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0899999999999999e-06 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.438088+0000", 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.287e-05 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.604352+0000", 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.16626407800000001 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.604360+0000", 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6729999999999992e-06 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.604410+0000", 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9925000000000001e-05 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.604427+0000", 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6735000000000002e-05 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.607284+0000", 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028577630000000001 2024-08-30T23:30:13.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:32.607322+0000", 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7146e-05 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:2679 3.3 3:c7f6fa7e:::benchmark_data_smithi173_60780_object2678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.692616+0000", 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.95863453600000004, 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18271148200000001, 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.738 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 2679 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692616+0000", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692616+0000", 2024-08-30T23:30:13.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692617+0000", 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6199999999999998e-07 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692797+0000", 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000180337 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692802+0000", 2024-08-30T23:30:13.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0499999999999999e-06 2024-08-30T23:30:13.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:13.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692806+0000", 2024-08-30T23:30:13.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9990000000000002e-06 2024-08-30T23:30:13.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692847+0000", 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0677999999999999e-05 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692931+0000", 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4337000000000002e-05 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:30:13.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693030+0000", 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8945999999999998e-05 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693817+0000", 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078704800000000004 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694304+0000", 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048707999999999998 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875226+0000", 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180922157 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875283+0000", 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6572e-05 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875328+0000", 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4715999999999997e-05 2024-08-30T23:30:13.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:2681 3.3 3:c917533a:::benchmark_data_smithi173_60780_object2680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.95843636700000001, 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20960189000000001, 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 2681 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692815+0000", 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0100000000000005e-07 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692841+0000", 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6024e-05 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692847+0000", 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3520000000000003e-06 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692853+0000", 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2209999999999996e-06 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693408+0000", 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055510499999999996 2024-08-30T23:30:13.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693502+0000", 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3960000000000002e-05 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693587+0000", 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5558999999999999e-05 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:13.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694000+0000", 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041311100000000002 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.701466+0000", 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074651769999999999 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902340+0000", 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200874052 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902381+0000", 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0929000000000001e-05 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902416+0000", 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5799000000000001e-05 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:2679 3.3 e213/206 ondisk, result = 0)", 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.694015+0000", 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.95723579199999997, 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181320273, 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694015+0000", 2024-08-30T23:30:13.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694015+0000", 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694015+0000", 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7399999999999999e-07 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694016+0000", 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1119999999999999e-06 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875150+0000", 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18113343800000001 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875156+0000", 2024-08-30T23:30:13.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2929999999999997e-06 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875214+0000", 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7775000000000002e-05 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875220+0000", 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3940000000000001e-06 2024-08-30T23:30:13.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875335+0000", 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114987 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:2681 3.3 e213/206 ondisk, result = 0)", 2024-08-30T23:30:13.757 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.95647499800000002, 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207646681, 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1e-07 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694779+0000", 2024-08-30T23:30:13.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5129999999999999e-06 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902275+0000", 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207495916 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902280+0000", 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8850000000000004e-06 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902329+0000", 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8656999999999998e-05 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902336+0000", 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.528e-06 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902423+0000", 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6872000000000004e-05 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.761 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:3271 3.3 3:c1f88b2b:::benchmark_data_smithi173_60780_object3270:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:13.161111+0000", 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.49014015100000002, 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14434951900000001, 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:13.762 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3271 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161111+0000", 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161111+0000", 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:13.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161112+0000", 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.423e-06 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161160+0000", 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7485999999999999e-05 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161174+0000", 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4589999999999999e-05 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161182+0000", 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9009999999999994e-06 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161234+0000", 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1798000000000001e-05 2024-08-30T23:30:13.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161318+0000", 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4007999999999999e-05 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161428+0000", 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109676 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.161903+0000", 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00047532 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.162158+0000", 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025562000000000002 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.305145+0000", 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.142986008 2024-08-30T23:30:13.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.305416+0000", 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000271208 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:13.305460+0000", 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4481000000000003e-05 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:13.769 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:13.769 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:30:13.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:13.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:13.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:13.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:14.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:13 smithi136 ceph-mon[20207]: pgmap v544: 17 pgs: 17 active+clean; 25 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 4.2 MiB/s wr, 66 op/s 2024-08-30T23:30:14.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:14.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:13 smithi173 ceph-mon[20554]: pgmap v544: 17 pgs: 17 active+clean; 25 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 4.2 MiB/s wr, 66 op/s 2024-08-30T23:30:14.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:13 smithi106 ceph-mon[16603]: pgmap v544: 17 pgs: 17 active+clean; 25 MiB data, 1.1 GiB used, 982 GiB / 983 GiB avail; 4.2 MiB/s wr, 66 op/s 2024-08-30T23:30:14.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:14.176 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:14.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:14.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 4842 4826 100.533 98.0625 0.0295626 0.00986563 2024-08-30T23:30:14.299 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:14.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:14.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:14.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:14.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:14.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:14.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:14.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:14.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:14.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:30:14.881 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:30:14.881 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:30:14.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:14.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:14.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:15.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:15.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:15.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:15.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:15.183 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:15.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:30:15.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:15.287 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6504 6488 101.367 103.875 0.0323518 0.00983912 2024-08-30T23:30:15.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:15.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:15.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:15.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:15.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:15.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:15.670 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":544,"stamp":"2024-08-30T23:30:13.403050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25690656,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":506,"num_write_kb":26662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":475,"ondisk_log_size":475,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1196988,"kb_used_data":66752,"kb_used_omap":17,"kb_used_meta":1129966,"kb_avail":1029999684,"statfs":{"total":1055945392128,"available":1054719676416,"internally_reserved":0,"allocated":68354048,"data_stored":65299358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1157086073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25100288,"num_objects":383,"num_object_clones":0,"num_object_copies":1149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":383,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.764247"},"pg_stats":[{"pgid":"3.f","version":"213'142","reported_seq":167,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:12.379631+0000","last_change":"2024-08-30T23:30:03.608229+0000","last_active":"2024-08-30T23:30:12.379631+0000","last_peered":"2024-08-30T23:30:12.379631+0000","last_clean":"2024-08-30T23:30:12.379631+0000","last_became_active":"2024-08-30T23:30:03.607819+0000","last_became_peered":"2024-08-30T23:30:03.607819+0000","last_unstale":"2024-08-30T23:30:12.379631+0000","last_undegraded":"2024-08-30T23:30:12.379631+0000","last_fullsized":"2024-08-30T23:30:12.379631+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:24.377557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":142,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":142,"num_write_kb":9088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-30T23:30:10.043538+0000","last_change":"2024-08-30T23:30:03.603745+0000","last_active":"2024-08-30T23:30:10.043538+0000","last_peered":"2024-08-30T23:30:10.043538+0000","last_clean":"2024-08-30T23:30:10.043538+0000","last_became_active":"2024-08-30T23:30:03.603411+0000","last_became_peered":"2024-08-30T23:30:03.603411+0000","last_unstale":"2024-08-30T23:30:10.043538+0000","last_undegraded":"2024-08-30T23:30:10.043538+0000","last_fullsized":"2024-08-30T23:30:10.043538+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:32.386418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"213'141","reported_seq":165,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:12.367230+0000","last_change":"2024-08-30T23:30:03.786790+0000","last_active":"2024-08-30T23:30:12.367230+0000","last_peered":"2024-08-30T23:30:12.367230+0000","last_clean":"2024-08-30T23:30:12.367230+0000","last_became_active":"2024-08-30T23:30:03.786437+0000","last_became_peered":"2024-08-30T23:30:03.786437+0000","last_unstale":"2024-08-30T23:30:12.367230+0000","last_undegraded":"2024-08-30T23:30:12.367230+0000","last_fullsized":"2024-08-30T23:30:12.367230+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:21.692769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9240576,"num_objects":141,"num_object_clones":0,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.411435+0000","last_change":"2024-08-30T23:30:03.608169+0000","last_active":"2024-08-30T23:30:08.411435+0000","last_peered":"2024-08-30T23:30:08.411435+0000","last_clean":"2024-08-30T23:30:08.411435+0000","last_became_active":"2024-08-30T23:30:03.607837+0000","last_became_peered":"2024-08-30T23:30:03.607837+0000","last_unstale":"2024-08-30T23:30:08.411435+0000","last_undegraded":"2024-08-30T23:30:08.411435+0000","last_fullsized":"2024-08-30T23:30:08.411435+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:07.552005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.411055+0000","last_change":"2024-08-30T23:30:03.603560+0000","last_active":"2024-08-30T23:30:08.411055+0000","last_peered":"2024-08-30T23:30:08.411055+0000","last_clean":"2024-08-30T23:30:08.411055+0000","last_became_active":"2024-08-30T23:30:03.603254+0000","last_became_peered":"2024-08-30T23:30:03.603254+0000","last_unstale":"2024-08-30T23:30:08.411055+0000","last_undegraded":"2024-08-30T23:30:08.411055+0000","last_fullsized":"2024-08-30T23:30:08.411055+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:26.308266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"213'100","reported_seq":125,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:11.855290+0000","last_change":"2024-08-30T23:30:03.609682+0000","last_active":"2024-08-30T23:30:11.855290+0000","last_peered":"2024-08-30T23:30:11.855290+0000","last_clean":"2024-08-30T23:30:11.855290+0000","last_became_active":"2024-08-30T23:30:03.608966+0000","last_became_peered":"2024-08-30T23:30:03.608966+0000","last_unstale":"2024-08-30T23:30:11.855290+0000","last_undegraded":"2024-08-30T23:30:11.855290+0000","last_fullsized":"2024-08-30T23:30:11.855290+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:17.096730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.645906+0000","last_change":"2024-08-30T23:30:03.608129+0000","last_active":"2024-08-30T23:30:08.645906+0000","last_peered":"2024-08-30T23:30:08.645906+0000","last_clean":"2024-08-30T23:30:08.645906+0000","last_became_active":"2024-08-30T23:30:03.607776+0000","last_became_peered":"2024-08-30T23:30:03.607776+0000","last_unstale":"2024-08-30T23:30:08.645906+0000","last_undegraded":"2024-08-30T23:30:08.645906+0000","last_fullsized":"2024-08-30T23:30:08.645906+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:17.505536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-30T23:30:10.043428+0000","last_change":"2024-08-30T23:30:03.603803+0000","last_active":"2024-08-30T23:30:10.043428+0000","last_peered":"2024-08-30T23:30:10.043428+0000","last_clean":"2024-08-30T23:30:10.043428+0000","last_became_active":"2024-08-30T23:30:03.603520+0000","last_became_peered":"2024-08-30T23:30:03.603520+0000","last_unstale":"2024-08-30T23:30:10.043428+0000","last_undegraded":"2024-08-30T23:30:10.043428+0000","last_fullsized":"2024-08-30T23:30:10.043428+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:10.762997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.411048+0000","last_change":"2024-08-30T23:30:03.607742+0000","last_active":"2024-08-30T23:30:08.411048+0000","last_peered":"2024-08-30T23:30:08.411048+0000","last_clean":"2024-08-30T23:30:08.411048+0000","last_became_active":"2024-08-30T23:30:03.607480+0000","last_became_peered":"2024-08-30T23:30:03.607480+0000","last_unstale":"2024-08-30T23:30:08.411048+0000","last_undegraded":"2024-08-30T23:30:08.411048+0000","last_fullsized":"2024-08-30T23:30:08.411048+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:24.671675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:07.640840+0000","last_change":"2024-08-30T23:30:03.603169+0000","last_active":"2024-08-30T23:30:07.640840+0000","last_peered":"2024-08-30T23:30:07.640840+0000","last_clean":"2024-08-30T23:30:07.640840+0000","last_became_active":"2024-08-30T23:30:03.602886+0000","last_became_peered":"2024-08-30T23:30:03.602886+0000","last_unstale":"2024-08-30T23:30:07.640840+0000","last_undegraded":"2024-08-30T23:30:07.640840+0000","last_fullsized":"2024-08-30T23:30:07.640840+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:25.318931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.411718+0000","last_change":"2024-08-30T23:30:03.608434+0000","last_active":"2024-08-30T23:30:08.411718+0000","last_peered":"2024-08-30T23:30:08.411718+0000","last_clean":"2024-08-30T23:30:08.411718+0000","last_became_active":"2024-08-30T23:30:03.607817+0000","last_became_peered":"2024-08-30T23:30:03.607817+0000","last_unstale":"2024-08-30T23:30:08.411718+0000","last_undegraded":"2024-08-30T23:30:08.411718+0000","last_fullsized":"2024-08-30T23:30:08.411718+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:25.631790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-30T23:30:10.043565+0000","last_change":"2024-08-30T23:30:03.607114+0000","last_active":"2024-08-30T23:30:10.043565+0000","last_peered":"2024-08-30T23:30:10.043565+0000","last_clean":"2024-08-30T23:30:10.043565+0000","last_became_active":"2024-08-30T23:30:03.606790+0000","last_became_peered":"2024-08-30T23:30:03.606790+0000","last_unstale":"2024-08-30T23:30:10.043565+0000","last_undegraded":"2024-08-30T23:30:10.043565+0000","last_fullsized":"2024-08-30T23:30:10.043565+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:24.306172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.646681+0000","last_change":"2024-08-30T23:30:03.608350+0000","last_active":"2024-08-30T23:30:08.646681+0000","last_peered":"2024-08-30T23:30:08.646681+0000","last_clean":"2024-08-30T23:30:08.646681+0000","last_became_active":"2024-08-30T23:30:03.607728+0000","last_became_peered":"2024-08-30T23:30:03.607728+0000","last_unstale":"2024-08-30T23:30:08.646681+0000","last_undegraded":"2024-08-30T23:30:08.646681+0000","last_fullsized":"2024-08-30T23:30:08.646681+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:15.144213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:07.640537+0000","last_change":"2024-08-30T23:30:03.603357+0000","last_active":"2024-08-30T23:30:07.640537+0000","last_peered":"2024-08-30T23:30:07.640537+0000","last_clean":"2024-08-30T23:30:07.640537+0000","last_became_active":"2024-08-30T23:30:03.603024+0000","last_became_peered":"2024-08-30T23:30:03.603024+0000","last_unstale":"2024-08-30T23:30:07.640537+0000","last_undegraded":"2024-08-30T23:30:07.640537+0000","last_fullsized":"2024-08-30T23:30:07.640537+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:05.481890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-30T23:30:10.043440+0000","last_change":"2024-08-30T23:30:03.786568+0000","last_active":"2024-08-30T23:30:10.043440+0000","last_peered":"2024-08-30T23:30:10.043440+0000","last_clean":"2024-08-30T23:30:10.043440+0000","last_became_active":"2024-08-30T23:30:03.786205+0000","last_became_peered":"2024-08-30T23:30:03.786205+0000","last_unstale":"2024-08-30T23:30:10.043440+0000","last_undegraded":"2024-08-30T23:30:10.043440+0000","last_fullsized":"2024-08-30T23:30:10.043440+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:04.697651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-30T23:30:08.411053+0000","last_change":"2024-08-30T23:30:03.786557+0000","last_active":"2024-08-30T23:30:08.411053+0000","last_peered":"2024-08-30T23:30:08.411053+0000","last_clean":"2024-08-30T23:30:08.411053+0000","last_became_active":"2024-08-30T23:30:03.786203+0000","last_became_peered":"2024-08-30T23:30:03.786203+0000","last_unstale":"2024-08-30T23:30:08.411053+0000","last_undegraded":"2024-08-30T23:30:08.411053+0000","last_fullsized":"2024-08-30T23:30:08.411053+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:16.702887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":393,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:11.775523+0000","last_change":"2024-08-30T23:30:11.775523+0000","last_active":"2024-08-30T23:30:11.775523+0000","last_peered":"2024-08-30T23:30:11.775523+0000","last_clean":"2024-08-30T23:30:11.775523+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:30:11.775523+0000","last_undegraded":"2024-08-30T23:30:11.775523+0000","last_fullsized":"2024-08-30T23:30:11.775523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:30:11.775464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:30:11.775464+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:21.723007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":25100288,"num_objects":383,"num_object_clones":0,"num_object_copies":1149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":383,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107937792,"data_stored":107937792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":383,"ondisk_log_size":383,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187524,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112952,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632200,"statfs":{"total":95995035648,"available":95879372800,"internally_reserved":0,"allocated":1282048,"data_stored":1001374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449162,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710798,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112440,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93632712,"statfs":{"total":95995035648,"available":95879897088,"internally_reserved":0,"allocated":749568,"data_stored":472466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972436,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86072,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93659080,"statfs":{"total":95995035648,"available":95906897920,"internally_reserved":0,"allocated":749568,"data_stored":472466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234071,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122072,"kb_used_data":22844,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93623080,"statfs":{"total":95995035648,"available":95870033920,"internally_reserved":0,"allocated":23392256,"data_stored":23102556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528413,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64376,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680776,"statfs":{"total":95995035648,"available":95929114624,"internally_reserved":0,"allocated":1282048,"data_stored":1001394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724640,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63844,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681308,"statfs":{"total":95995035648,"available":95929659392,"internally_reserved":0,"allocated":737280,"data_stored":464676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953574,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63860,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93681292,"statfs":{"total":95995035648,"available":95929643008,"internally_reserved":0,"allocated":749568,"data_stored":472466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148520,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596632,"statfs":{"total":95995035648,"available":95842951168,"internally_reserved":0,"allocated":737280,"data_stored":464676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126296,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618856,"statfs":{"total":95995035648,"available":95865708544,"internally_reserved":0,"allocated":724992,"data_stored":456596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":188292,"kb_used_data":36340,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":93556860,"statfs":{"total":95995035648,"available":95802224640,"internally_reserved":0,"allocated":37212160,"data_stored":36926012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":155580869},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108264,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636888,"statfs":{"total":95995035648,"available":95884173312,"internally_reserved":0,"allocated":737280,"data_stored":464676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39256064,"data_stored":39256064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32309248,"data_stored":32309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36372480,"data_stored":36372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:30:15.694 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:7230 3.5 3:aa33fdba:::benchmark_data_smithi173_60780_object7229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:15.716112+0000", 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0017800159999999999, 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017992780000000001, 2024-08-30T23:30:15.718 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7230 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716112+0000", 2024-08-30T23:30:15.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716112+0000", 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716113+0000", 2024-08-30T23:30:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716162+0000", 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8597000000000002e-05 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716175+0000", 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3416e-05 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716182+0000", 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3170000000000004e-06 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716236+0000", 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4085999999999997e-05 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716356+0000", 2024-08-30T23:30:15.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119862 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.716496+0000", 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014011500000000001 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.717023+0000", 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052645800000000005 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:15.717486+0000", 2024-08-30T23:30:15.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00046363099999999999 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:30:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:30:15.731 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:15.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:15.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:15.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:15.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:15.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:16.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:16.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:16.116 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:30:16.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:16.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:16.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 8054 8038 100.467 96.875 0.00277216 0.00993234 2024-08-30T23:30:16.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:16.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:16.424 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:16.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:16.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:16.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:30:16.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:16.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:16.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:16.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:16.723 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:16 smithi106 ceph-mon[16603]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:16.723 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:16 smithi106 ceph-mon[16603]: pgmap v545: 17 pgs: 17 active+clean; 153 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 23 MiB/s wr, 361 op/s 2024-08-30T23:30:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:16 smithi136 ceph-mon[20207]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:16 smithi136 ceph-mon[20207]: pgmap v545: 17 pgs: 17 active+clean; 153 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 23 MiB/s wr, 361 op/s 2024-08-30T23:30:16.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:16.815 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:16 smithi173 ceph-mon[20554]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:16 smithi173 ceph-mon[20554]: pgmap v545: 17 pgs: 17 active+clean; 153 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 23 MiB/s wr, 361 op/s 2024-08-30T23:30:16.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:16.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:16.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:16.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:17.014 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:17.014 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:17.015 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:17.015 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:17.042 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:17.066 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:17.066 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:17.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:17.096 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:17.096 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:17.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:17.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:17.121 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:17.142 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:17.143 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:17.166 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:17.166 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:17.190 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:17.190 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:17.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:17.215 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:17.215 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:17.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:17.239 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:17.239 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:17.261 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:17.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 9739 9723 101.273 105.312 0.00133057 0.00984439 2024-08-30T23:30:17.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:17.342 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:17.342 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:17.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:17.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:17.470 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:30:17.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:17.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/905870285' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:17.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:17 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3036382469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:17.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/905870285' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:17.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:17 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:17.579 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3036382469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:17.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:17.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:17.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.734 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:17.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/905870285' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:17 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3036382469' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:17.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:17.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:17.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:17.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:17.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:18.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.112 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:18.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:18.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:18.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:30:18.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:18.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 15 11178 11163 99.6618 90 0.00250707 0.0100227 2024-08-30T23:30:18.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:18.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:18.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:18.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:18.574 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:18.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:18 smithi106 ceph-mon[16603]: pgmap v546: 17 pgs: 17 active+clean; 389 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 776 op/s 2024-08-30T23:30:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:18 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:30:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:18 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:30:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:18 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:18 smithi106 ceph-mon[16603]: osdmap e214: 12 total, 12 up, 11 in 2024-08-30T23:30:18.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:18.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:30:18.721 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:18 smithi136 ceph-mon[20207]: pgmap v546: 17 pgs: 17 active+clean; 389 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 776 op/s 2024-08-30T23:30:18.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:18 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:30:18.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:18 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:30:18.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:18 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:18.722 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:18 smithi136 ceph-mon[20207]: osdmap e214: 12 total, 12 up, 11 in 2024-08-30T23:30:18.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:18.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:18 smithi173 ceph-mon[20554]: pgmap v546: 17 pgs: 17 active+clean; 389 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 776 op/s 2024-08-30T23:30:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:18 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:30:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:18 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:30:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:18 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:18 smithi173 ceph-mon[20554]: osdmap e214: 12 total, 12 up, 11 in 2024-08-30T23:30:18.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:18.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:18.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:18.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:19.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:19.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:19.133 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:19.133 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:19.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:19.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:19.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:19.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 12957 12941 101.094 111.125 0.00514413 0.00988492 2024-08-30T23:30:19.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:19.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:19.412 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:19.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:19.519 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:19.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:19.644 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:19.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:19.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:19.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:30:19.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:19.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:19.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:19.957 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:30:20.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:20.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:20.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:20.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:20.167 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:30:20.167 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:30:20.168 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:20.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:20.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:20.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 14390 14374 99.8117 89.5625 0.0162087 0.0100101 2024-08-30T23:30:20.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:20.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:20.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:20.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:20.553 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:20 smithi136 ceph-mon[20207]: pgmap v548: 17 pgs: 17 active+clean; 607 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-08-30T23:30:20.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:20.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:20.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:20.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:20.802 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:20.802 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:20 smithi173 ceph-mon[20554]: pgmap v548: 17 pgs: 17 active+clean; 607 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-08-30T23:30:20.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:20 smithi106 ceph-mon[16603]: pgmap v548: 17 pgs: 17 active+clean; 607 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-08-30T23:30:20.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:20.911 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:20.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:20.953 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":548,"stamp":"2024-08-30T23:30:19.404263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":636682784,"num_objects":9708,"num_object_clones":0,"num_object_copies":29124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9708,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":9829,"num_write_kb":623334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9798,"ondisk_log_size":9798,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":2850944,"kb_used_data":1663812,"kb_used_omap":17,"kb_used_meta":1186862,"kb_avail":1028345728,"statfs":{"total":1055945392128,"available":1053026025472,"internally_reserved":0,"allocated":1703743488,"data_stored":1700566992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17543,"internal_metadata":1215347577},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,2,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":636092416,"num_objects":9706,"num_object_clones":0,"num_object_copies":29118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":621184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001753"},"pg_stats":[{"pgid":"3.f","version":"213'635","reported_seq":660,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:17.378073+0000","last_change":"2024-08-30T23:30:03.608229+0000","last_active":"2024-08-30T23:30:17.378073+0000","last_peered":"2024-08-30T23:30:17.378073+0000","last_clean":"2024-08-30T23:30:17.378073+0000","last_became_active":"2024-08-30T23:30:03.607819+0000","last_became_peered":"2024-08-30T23:30:03.607819+0000","last_unstale":"2024-08-30T23:30:17.378073+0000","last_undegraded":"2024-08-30T23:30:17.378073+0000","last_fullsized":"2024-08-30T23:30:17.378073+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:24.377557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41615360,"num_objects":635,"num_object_clones":0,"num_object_copies":1905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":635,"num_write_kb":40640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"213'505","reported_seq":530,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:16.113025+0000","last_change":"2024-08-30T23:30:03.603745+0000","last_active":"2024-08-30T23:30:16.113025+0000","last_peered":"2024-08-30T23:30:16.113025+0000","last_clean":"2024-08-30T23:30:16.113025+0000","last_became_active":"2024-08-30T23:30:03.603411+0000","last_became_peered":"2024-08-30T23:30:03.603411+0000","last_unstale":"2024-08-30T23:30:16.113025+0000","last_undegraded":"2024-08-30T23:30:16.113025+0000","last_fullsized":"2024-08-30T23:30:16.113025+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:32.386418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33095680,"num_objects":505,"num_object_clones":0,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":505,"num_write_kb":32320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"213'620","reported_seq":645,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:17.384487+0000","last_change":"2024-08-30T23:30:03.786790+0000","last_active":"2024-08-30T23:30:17.384487+0000","last_peered":"2024-08-30T23:30:17.384487+0000","last_clean":"2024-08-30T23:30:17.384487+0000","last_became_active":"2024-08-30T23:30:03.786437+0000","last_became_peered":"2024-08-30T23:30:03.786437+0000","last_unstale":"2024-08-30T23:30:17.384487+0000","last_undegraded":"2024-08-30T23:30:17.384487+0000","last_fullsized":"2024-08-30T23:30:17.384487+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":620,"log_dups_size":0,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:21.692769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":620,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":620,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"214'738","reported_seq":764,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.624014+0000","last_change":"2024-08-30T23:30:03.608169+0000","last_active":"2024-08-30T23:30:18.624014+0000","last_peered":"2024-08-30T23:30:18.624014+0000","last_clean":"2024-08-30T23:30:18.624014+0000","last_became_active":"2024-08-30T23:30:03.607837+0000","last_became_peered":"2024-08-30T23:30:03.607837+0000","last_unstale":"2024-08-30T23:30:18.624014+0000","last_undegraded":"2024-08-30T23:30:18.624014+0000","last_fullsized":"2024-08-30T23:30:18.624014+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":738,"log_dups_size":0,"ondisk_log_size":738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:07.552005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48365568,"num_objects":738,"num_object_clones":0,"num_object_copies":2214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":738,"num_write_kb":47232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"214'741","reported_seq":767,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.550491+0000","last_change":"2024-08-30T23:30:03.603560+0000","last_active":"2024-08-30T23:30:18.550491+0000","last_peered":"2024-08-30T23:30:18.550491+0000","last_clean":"2024-08-30T23:30:18.550491+0000","last_became_active":"2024-08-30T23:30:03.603254+0000","last_became_peered":"2024-08-30T23:30:03.603254+0000","last_unstale":"2024-08-30T23:30:18.550491+0000","last_undegraded":"2024-08-30T23:30:18.550491+0000","last_fullsized":"2024-08-30T23:30:18.550491+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:26.308266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":47424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"213'525","reported_seq":550,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:16.853506+0000","last_change":"2024-08-30T23:30:03.609682+0000","last_active":"2024-08-30T23:30:16.853506+0000","last_peered":"2024-08-30T23:30:16.853506+0000","last_clean":"2024-08-30T23:30:16.853506+0000","last_became_active":"2024-08-30T23:30:03.608966+0000","last_became_peered":"2024-08-30T23:30:03.608966+0000","last_unstale":"2024-08-30T23:30:16.853506+0000","last_undegraded":"2024-08-30T23:30:16.853506+0000","last_fullsized":"2024-08-30T23:30:16.853506+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:17.096730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":525,"num_write_kb":33600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"213'415","reported_seq":440,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:15.687228+0000","last_change":"2024-08-30T23:30:03.608129+0000","last_active":"2024-08-30T23:30:15.687228+0000","last_peered":"2024-08-30T23:30:15.687228+0000","last_clean":"2024-08-30T23:30:15.687228+0000","last_became_active":"2024-08-30T23:30:03.607776+0000","last_became_peered":"2024-08-30T23:30:03.607776+0000","last_unstale":"2024-08-30T23:30:15.687228+0000","last_undegraded":"2024-08-30T23:30:15.687228+0000","last_fullsized":"2024-08-30T23:30:15.687228+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:17.505536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27197440,"num_objects":415,"num_object_clones":0,"num_object_copies":1245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":415,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"213'486","reported_seq":511,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:16.125080+0000","last_change":"2024-08-30T23:30:03.603803+0000","last_active":"2024-08-30T23:30:16.125080+0000","last_peered":"2024-08-30T23:30:16.125080+0000","last_clean":"2024-08-30T23:30:16.125080+0000","last_became_active":"2024-08-30T23:30:03.603520+0000","last_became_peered":"2024-08-30T23:30:03.603520+0000","last_unstale":"2024-08-30T23:30:16.125080+0000","last_undegraded":"2024-08-30T23:30:16.125080+0000","last_fullsized":"2024-08-30T23:30:16.125080+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:10.762997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"214'777","reported_seq":804,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.575787+0000","last_change":"2024-08-30T23:30:03.607742+0000","last_active":"2024-08-30T23:30:18.575787+0000","last_peered":"2024-08-30T23:30:18.575787+0000","last_clean":"2024-08-30T23:30:18.575787+0000","last_became_active":"2024-08-30T23:30:03.607480+0000","last_became_peered":"2024-08-30T23:30:03.607480+0000","last_unstale":"2024-08-30T23:30:18.575787+0000","last_undegraded":"2024-08-30T23:30:18.575787+0000","last_fullsized":"2024-08-30T23:30:18.575787+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:24.671675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50921472,"num_objects":777,"num_object_clones":0,"num_object_copies":2331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":777,"num_write_kb":49728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"214'725","reported_seq":751,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.440833+0000","last_change":"2024-08-30T23:30:03.603169+0000","last_active":"2024-08-30T23:30:18.440833+0000","last_peered":"2024-08-30T23:30:18.440833+0000","last_clean":"2024-08-30T23:30:18.440833+0000","last_became_active":"2024-08-30T23:30:03.602886+0000","last_became_peered":"2024-08-30T23:30:03.602886+0000","last_unstale":"2024-08-30T23:30:18.440833+0000","last_undegraded":"2024-08-30T23:30:18.440833+0000","last_fullsized":"2024-08-30T23:30:18.440833+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:25.318931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47513600,"num_objects":725,"num_object_clones":0,"num_object_copies":2175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":725,"num_write_kb":46400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"214'766","reported_seq":793,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.597681+0000","last_change":"2024-08-30T23:30:03.608434+0000","last_active":"2024-08-30T23:30:18.597681+0000","last_peered":"2024-08-30T23:30:18.597681+0000","last_clean":"2024-08-30T23:30:18.597681+0000","last_became_active":"2024-08-30T23:30:03.607817+0000","last_became_peered":"2024-08-30T23:30:03.607817+0000","last_unstale":"2024-08-30T23:30:18.597681+0000","last_undegraded":"2024-08-30T23:30:18.597681+0000","last_fullsized":"2024-08-30T23:30:18.597681+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:25.631790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":2298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":766,"num_write_kb":49024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"213'473","reported_seq":498,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:16.121613+0000","last_change":"2024-08-30T23:30:03.607114+0000","last_active":"2024-08-30T23:30:16.121613+0000","last_peered":"2024-08-30T23:30:16.121613+0000","last_clean":"2024-08-30T23:30:16.121613+0000","last_became_active":"2024-08-30T23:30:03.606790+0000","last_became_peered":"2024-08-30T23:30:03.606790+0000","last_unstale":"2024-08-30T23:30:16.121613+0000","last_undegraded":"2024-08-30T23:30:16.121613+0000","last_fullsized":"2024-08-30T23:30:16.121613+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:24.306172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":473,"num_object_clones":0,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":473,"num_write_kb":30272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"213'339","reported_seq":364,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:14.526446+0000","last_change":"2024-08-30T23:30:03.608350+0000","last_active":"2024-08-30T23:30:14.526446+0000","last_peered":"2024-08-30T23:30:14.526446+0000","last_clean":"2024-08-30T23:30:14.526446+0000","last_became_active":"2024-08-30T23:30:03.607728+0000","last_became_peered":"2024-08-30T23:30:03.607728+0000","last_unstale":"2024-08-30T23:30:14.526446+0000","last_undegraded":"2024-08-30T23:30:14.526446+0000","last_fullsized":"2024-08-30T23:30:14.526446+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:15.144213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22216704,"num_objects":339,"num_object_clones":0,"num_object_copies":1017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":339,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"214'740","reported_seq":766,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.445337+0000","last_change":"2024-08-30T23:30:03.603357+0000","last_active":"2024-08-30T23:30:18.445337+0000","last_peered":"2024-08-30T23:30:18.445337+0000","last_clean":"2024-08-30T23:30:18.445337+0000","last_became_active":"2024-08-30T23:30:03.603024+0000","last_became_peered":"2024-08-30T23:30:03.603024+0000","last_unstale":"2024-08-30T23:30:18.445337+0000","last_undegraded":"2024-08-30T23:30:18.445337+0000","last_fullsized":"2024-08-30T23:30:18.445337+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:05.481890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":740,"num_write_kb":47360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"213'500","reported_seq":525,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:16.129603+0000","last_change":"2024-08-30T23:30:03.786568+0000","last_active":"2024-08-30T23:30:16.129603+0000","last_peered":"2024-08-30T23:30:16.129603+0000","last_clean":"2024-08-30T23:30:16.129603+0000","last_became_active":"2024-08-30T23:30:03.786205+0000","last_became_peered":"2024-08-30T23:30:03.786205+0000","last_unstale":"2024-08-30T23:30:16.129603+0000","last_undegraded":"2024-08-30T23:30:16.129603+0000","last_fullsized":"2024-08-30T23:30:16.129603+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:04.697651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":500,"num_object_clones":0,"num_object_copies":1500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":500,"num_write_kb":32000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"214'721","reported_seq":747,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-08-30T23:30:18.589000+0000","last_change":"2024-08-30T23:30:03.786557+0000","last_active":"2024-08-30T23:30:18.589000+0000","last_peered":"2024-08-30T23:30:18.589000+0000","last_clean":"2024-08-30T23:30:18.589000+0000","last_became_active":"2024-08-30T23:30:03.786203+0000","last_became_peered":"2024-08-30T23:30:03.786203+0000","last_unstale":"2024-08-30T23:30:18.589000+0000","last_undegraded":"2024-08-30T23:30:18.589000+0000","last_fullsized":"2024-08-30T23:30:18.589000+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:30:02.569516+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:16.702887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47251456,"num_objects":721,"num_object_clones":0,"num_object_copies":2163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":721,"num_write_kb":46144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":393,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-30T23:30:11.775523+0000","last_change":"2024-08-30T23:30:11.775523+0000","last_active":"2024-08-30T23:30:11.775523+0000","last_peered":"2024-08-30T23:30:11.775523+0000","last_clean":"2024-08-30T23:30:11.775523+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:30:11.775523+0000","last_undegraded":"2024-08-30T23:30:11.775523+0000","last_fullsized":"2024-08-30T23:30:11.775523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:30:11.775464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:30:11.775464+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:31:21.723007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":636092416,"num_objects":9706,"num_object_clones":0,"num_object_copies":29118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":621184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1776615424,"data_stored":1776615424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9706,"ondisk_log_size":9706,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187525,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353228,"kb_used_data":241532,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93391924,"statfs":{"total":95995035648,"available":95633330176,"internally_reserved":0,"allocated":247328768,"data_stored":247039048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":114358724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449163,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710799,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":167044,"kb_used_data":67828,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93578108,"statfs":{"total":95995035648,"available":95823982592,"internally_reserved":0,"allocated":69455872,"data_stored":69169724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":134980,"kb_used_data":49652,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93610172,"statfs":{"total":95995035648,"available":95856816128,"internally_reserved":0,"allocated":50843648,"data_stored":50557500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234072,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249420,"kb_used_data":141884,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93495732,"statfs":{"total":95995035648,"available":95739629568,"internally_reserved":0,"allocated":145289216,"data_stored":144999516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528414,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":235660,"kb_used_data":160060,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93509492,"statfs":{"total":95995035648,"available":95753719808,"internally_reserved":0,"allocated":163901440,"data_stored":163611740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301332,"kb_used_data":189632,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93443820,"statfs":{"total":95995035648,"available":95686471680,"internally_reserved":0,"allocated":194183168,"data_stored":193892472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953575,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217284,"kb_used_data":145844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93527868,"statfs":{"total":95995035648,"available":95772536832,"internally_reserved":0,"allocated":149344256,"data_stored":149058108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":387088,"kb_used_data":239296,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93358064,"statfs":{"total":95995035648,"available":95598657536,"internally_reserved":0,"allocated":245039104,"data_stored":244748408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171412,"kb_used_data":41664,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93573740,"statfs":{"total":95995035648,"available":95819509760,"internally_reserved":0,"allocated":42663936,"data_stored":42373240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299972,"kb_used_data":156340,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93445180,"statfs":{"total":95995035648,"available":95687864320,"internally_reserved":0,"allocated":160092160,"data_stored":159806012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333524,"kb_used_data":230080,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93411628,"statfs":{"total":95995035648,"available":95653507072,"internally_reserved":0,"allocated":235601920,"data_stored":235311224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239206400,"data_stored":239206400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162660352,"data_stored":162660352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50069504,"data_stored":50069504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245039104,"data_stored":245039104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":155058176,"data_stored":155058176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":193658880,"data_stored":193658880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":179109888,"data_stored":179109888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":164167680,"data_stored":164167680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57212928,"data_stored":57212928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":73728000,"data_stored":73728000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":256704512,"data_stored":256704512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:30:20.955 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-30T23:30:20.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:30:20.955 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-08-30T23:30:20.955 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:30:20.955 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:30:20.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-08-30T23:30:21.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:21.025 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:21.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:21.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:21.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:21.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:21.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 15653 15637 97.7237 78.9375 0.00872481 0.0102236 2024-08-30T23:30:21.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:21.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:30:21.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:21.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:21.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:21.541 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:21.543 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:21.547 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:21.547 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:21.547 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:21 smithi106 ceph-mon[16603]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:21.577 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:21.577 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:21.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:30:21.600 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:21.600 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:21.622 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:21.622 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:21.644 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:21.645 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:21.645 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:21.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:21.671 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:21.671 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:21.697 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:21.697 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:21.721 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:21.721 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:21.747 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:21.747 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:21.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:21.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:21.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:21 smithi136 ceph-mon[20207]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:21.796 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:21.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:21.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:21.822 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:21.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:21 smithi173 ceph-mon[20554]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:30:21.879 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:21.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:21.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:21.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:22.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:22.050 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:22.054 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:30:22.055 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:30:22.055 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:30:22.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:22.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:22.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:22.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 16810 16794 95.4131 72.3125 0.00267566 0.0104686 2024-08-30T23:30:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:22.403 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:30:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:30:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:30:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:30:22.403 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:30:22.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:22.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:22.501 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:22.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:22 smithi136 ceph-mon[20207]: pgmap v549: 17 pgs: 17 active+clean; 790 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-30T23:30:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/587729831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:22.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:22.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:22.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:22.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:22.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:22 smithi173 ceph-mon[20554]: pgmap v549: 17 pgs: 17 active+clean; 790 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-30T23:30:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/587729831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:22.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:22.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:22 smithi106 ceph-mon[16603]: pgmap v549: 17 pgs: 17 active+clean; 790 MiB data, 3.3 GiB used, 980 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-30T23:30:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/587729831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:22.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:22.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:22.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:22.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:30:23.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:23.078 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:23.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:23.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:23.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 15 18757 18742 97.6071 121.75 0.00194674 0.0102371 2024-08-30T23:30:23.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:23.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:23.379 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:23.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:23.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:23.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:23.589 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:23.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:23.713 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:23.814 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:23.842 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:23.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:23.951 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:30:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1905 2.7 2:e3c3e2b6:::benchmark_data_smithi173_47195_object1904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.63027395400002, 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.214235494, 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1905 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:30:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320114+0000", 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8139999999999999e-06 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320140+0000", 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5607999999999999e-05 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320149+0000", 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1779999999999992e-06 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320156+0000", 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320214+0000", 2024-08-30T23:30:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7816999999999999e-05 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320293+0000", 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9528000000000002e-05 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320375+0000", 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1314000000000001e-05 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320722+0000", 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034726800000000002 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.321314+0000", 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059229900000000002 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534242+0000", 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21292789500000001 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534303+0000", 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0656000000000003e-05 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534348+0000", 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5039e-05 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1915 2.7 2:e31a5d2a:::benchmark_data_smithi173_47195_object1914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.62809168000001, 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21216159100000001, 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1915 2024-08-30T23:30:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3600000000000002e-07 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322318+0000", 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2801000000000001e-05 2024-08-30T23:30:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322326+0000", 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3629999999999996e-06 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322330+0000", 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8069999999999999e-06 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322371+0000", 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0200000000000001e-05 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322445+0000", 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4750000000000001e-05 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322526+0000", 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0927000000000006e-05 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322865+0000", 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033871900000000002 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.323144+0000", 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027930899999999999 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534388+0000", 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21124331399999999 2024-08-30T23:30:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534422+0000", 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4087000000000001e-05 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534456+0000", 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4477999999999998e-05 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1922 2.7 2:ee8ad553:::benchmark_data_smithi173_47195_object1921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.623708738, 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207961537, 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1922 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326753+0000", 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4973000000000002e-05 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326764+0000", 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0787e-05 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326769+0000", 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0189999999999997e-06 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326810+0000", 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.125e-05 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326889+0000", 2024-08-30T23:30:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8110999999999998e-05 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326971+0000", 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2132999999999994e-05 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.327277+0000", 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030633899999999998 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.327759+0000", 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048190199999999998 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534555+0000", 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20679639799999999 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534604+0000", 2024-08-30T23:30:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8588999999999998e-05 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534639+0000", 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5185000000000003e-05 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1930 2.7 2:ea8dc1bc:::benchmark_data_smithi173_47195_object1929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.62021390299998, 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20455900900000001, 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1930 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330174+0000", 2024-08-30T23:30:23.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1540000000000001e-06 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330209+0000", 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5206000000000001e-05 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330220+0000", 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1443999999999999e-05 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330225+0000", 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2299999999999999e-06 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330272+0000", 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6168000000000002e-05 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330386+0000", 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114523 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330484+0000", 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7739999999999996e-05 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:23.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330815+0000", 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000331202 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.331211+0000", 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039613199999999999 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534670+0000", 2024-08-30T23:30:23.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203458423 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534698+0000", 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8432000000000002e-05 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534731+0000", 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3355000000000002e-05 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:23.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1967 2.7 2:e55fccac:::benchmark_data_smithi173_47195_object1966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.60209072200001, 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18683412999999999, 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1967 2024-08-30T23:30:23.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348297+0000", 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.548e-06 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348327+0000", 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0312e-05 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348342+0000", 2024-08-30T23:30:23.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4647000000000001e-05 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348365+0000", 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3098000000000001e-05 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348409+0000", 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4261000000000002e-05 2024-08-30T23:30:23.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348496+0000", 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6846000000000002e-05 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348576+0000", 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9411000000000003e-05 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:23.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348908+0000", 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033185099999999998 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.349356+0000", 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044820899999999998 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535032+0000", 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18567658200000001 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535085+0000", 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2559000000000003e-05 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535130+0000", 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4805999999999998e-05 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:23.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1975 2.7 2:eeb7a2cc:::benchmark_data_smithi173_47195_object1974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.59714460999999, 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18200930000000001, 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:23.993 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1975 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353243+0000", 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5899999999999995e-07 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353322+0000", 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9729999999999997e-05 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353335+0000", 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2719e-05 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353340+0000", 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:23.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353381+0000", 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0828000000000003e-05 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353459+0000", 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8151000000000004e-05 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353543+0000", 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3549999999999998e-05 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353838+0000", 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029477600000000001 2024-08-30T23:30:23.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.354159+0000", 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000321007 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535175+0000", 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181016702 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:23.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535227+0000", 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1307000000000002e-05 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535251+0000", 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4453000000000001e-05 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1977 2.7 2:e9772656:::benchmark_data_smithi173_47195_object1976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:30:24.000 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:30:24.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 264.59519464200002, 2024-08-30T23:30:24.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180200314, 2024-08-30T23:30:24.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.001 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:24.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:24.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1977 2024-08-30T23:30:24.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:30:24.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355192+0000", 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.09e-07 2024-08-30T23:30:24.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355252+0000", 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9171e-05 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355262+0000", 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0259e-05 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355267+0000", 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.938e-06 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355296+0000", 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9252e-05 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355364+0000", 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7589000000000006e-05 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355445+0000", 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1457000000000003e-05 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.355716+0000", 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027039099999999998 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.356116+0000", 2024-08-30T23:30:24.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039998099999999999 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535289+0000", 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17917321 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535356+0000", 2024-08-30T23:30:24.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7482000000000002e-05 2024-08-30T23:30:24.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535392+0000", 2024-08-30T23:30:24.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5775000000000002e-05 2024-08-30T23:30:24.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.010 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.011 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:36436 2.2 e116/97 2:4f13755f:::benchmark_data_smithi173_47195_object36435:head v 116'2235, mlcod=116'2231)", 2024-08-30T23:30:24.011 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:30:24.011 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 238.96681801299999, 2024-08-30T23:30:24.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23314933600000001, 2024-08-30T23:30:24.011 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983569+0000", 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2799999999999995e-07 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983591+0000", 2024-08-30T23:30:24.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1899000000000002e-05 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215327+0000", 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23173639900000001 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215335+0000", 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.216e-06 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215397+0000", 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2348999999999995e-05 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215421+0000", 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4114e-05 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216674+0000", 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012534379999999999 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216718+0000", 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3192999999999998e-05 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.016 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:36447 2.2 e116/97 2:42c8382e:::benchmark_data_smithi173_47195_object36446:head v 116'2236, mlcod=116'2232)", 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 238.96176693000001, 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310671684, 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988620+0000", 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.099e-06 2024-08-30T23:30:24.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988646+0000", 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5996999999999999e-05 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297448+0000", 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30880194900000002 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297456+0000", 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5059999999999998e-06 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297540+0000", 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4086999999999996e-05 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297565+0000", 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5304e-05 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299247+0000", 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016817690000000001 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299291+0000", 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3973000000000001e-05 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:40558 2.7 2:ec8f91a5:::benchmark_data_smithi173_47195_object40557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 236.02055912700001, 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18168003199999999, 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 40558 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929828+0000", 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.23e-07 2024-08-30T23:30:24.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929928+0000", 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100546 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929938+0000", 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8260000000000005e-06 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929945+0000", 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6669999999999999e-06 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930530+0000", 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058469100000000003 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930611+0000", 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1532000000000005e-05 2024-08-30T23:30:24.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930702+0000", 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0371000000000006e-05 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931513+0000", 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081120499999999996 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111320+0000", 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17980744600000001 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111443+0000", 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123242 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111479+0000", 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5734e-05 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111507+0000", 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8048999999999999e-05 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:40558 2.7 e119/97 ondisk, result = 0)", 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 236.01851166899999, 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17945143799999999, 2024-08-30T23:30:24.028 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:30:24.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931875+0000", 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931878+0000", 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5519999999999999e-06 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111265+0000", 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179387094 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111270+0000", 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1100000000000002e-06 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111310+0000", 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0417999999999998e-05 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111316+0000", 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0809999999999999e-06 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111326+0000", 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7559999999999994e-06 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:40558 2.7 e119/97 ondisk, result = 0)", 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 236.018484734, 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179610452, 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931902+0000", 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-08-30T23:30:24.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931905+0000", 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7190000000000001e-06 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111399+0000", 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17949393599999999 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111403+0000", 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3719999999999998e-06 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111434+0000", 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1375999999999999e-05 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111440+0000", 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8209999999999998e-06 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111512+0000", 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1800999999999999e-05 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50002 2.7 2:e84aa78e:::benchmark_data_smithi173_47195_object50001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 228.681947092, 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.199583015, 2024-08-30T23:30:24.037 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:24.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:24.038 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:24.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50002 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268441+0000", 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9139999999999998e-06 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268476+0000", 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4665000000000001e-05 2024-08-30T23:30:24.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268490+0000", 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4104000000000001e-05 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268498+0000", 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2619999999999998e-06 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268559+0000", 2024-08-30T23:30:24.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0537e-05 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268646+0000", 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7587e-05 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268736+0000", 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9363999999999993e-05 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.269952+0000", 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012166290000000001 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.270470+0000", 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051778800000000004 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.467915+0000", 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.197445024 2024-08-30T23:30:24.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.467979+0000", 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.41e-05 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.468022+0000", 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3040999999999997e-05 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:56072 2.7 2:e7493b3b:::benchmark_data_smithi173_47195_object56071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:39.795290+0000", 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 224.155096246, 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17400230799999999, 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:24.046 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 56072 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795290+0000", 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795290+0000", 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795296+0000", 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2960000000000004e-06 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795343+0000", 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6869000000000001e-05 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795374+0000", 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.04e-05 2024-08-30T23:30:24.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795390+0000", 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6711999999999999e-05 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795482+0000", 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1751999999999998e-05 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795571+0000", 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8468999999999996e-05 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.795655+0000", 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4734000000000006e-05 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.797013+0000", 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013579130000000001 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.797466+0000", 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045263200000000001 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.969189+0000", 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17172362799999999 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.969250+0000", 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0625000000000003e-05 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.969292+0000", 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2277999999999997e-05 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 146/138 [PushReplyOp(2:ee294341:::benchmark_data_smithi173_47195_object15777:head)])", 2024-08-30T23:30:24.052 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 175.75935468500001, 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.001463438, 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191037+0000", 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2749999999999997e-06 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181108+0000", 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99007171599999999 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181115+0000", 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.189300+0000", 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081848760000000007 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.192495+0000", 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031953250000000002 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 153/138 [PushReplyOp(2:eebfff31:::benchmark_data_smithi173_47195_object26886:head)])", 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.025355773, 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21501724999999999, 2024-08-30T23:30:24.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.09e-07 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925033+0000", 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.419e-06 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136148+0000", 2024-08-30T23:30:24.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.211115204 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136156+0000", 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1880000000000002e-06 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136222+0000", 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5320999999999993e-05 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.140048+0000", 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003826009 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:126363 2.7 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 137.221149879, 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221802834, 2024-08-30T23:30:24.060 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 126363 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:30:24.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729238+0000", 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9360000000000002e-06 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729244+0000", 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8819999999999998e-06 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729258+0000", 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3994e-05 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729265+0000", 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0570000000000004e-06 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729318+0000", 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2410000000000001e-05 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729417+0000", 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9414999999999996e-05 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729508+0000", 2024-08-30T23:30:24.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1209e-05 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.732169+0000", 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026607919999999999 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.737250+0000", 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050804370000000001 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949780+0000", 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21253043999999999 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949815+0000", 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5308000000000001e-05 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.951039+0000", 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001223954 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.066 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193992 2.7 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.175135550999997, 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60785984599999998, 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:24.067 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193992 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775252+0000", 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.528e-06 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775258+0000", 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.485e-06 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775273+0000", 2024-08-30T23:30:24.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5355000000000001e-05 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775281+0000", 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5140000000000004e-06 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775342+0000", 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1334000000000006e-05 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775448+0000", 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010587 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775514+0000", 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5747000000000006e-05 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.779105+0000", 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035911110000000001 2024-08-30T23:30:24.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.780025+0000", 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092040100000000005 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383045+0000", 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60302017900000005 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383083+0000", 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7670999999999999e-05 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383111+0000", 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7651e-05 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:2678 3.0 e213/206 3:0b08ec04:::benchmark_data_smithi173_60780_object2677:head v 213'154, mlcod=213'151)", 2024-08-30T23:30:24.073 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.686750+0000", 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 11.263636019, 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18360559000000001, 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686750+0000", 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686750+0000", 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686751+0000", 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9399999999999995e-07 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686816+0000", 2024-08-30T23:30:24.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4803999999999998e-05 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869906+0000", 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18309029199999999 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869910+0000", 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4440000000000001e-06 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869960+0000", 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9372000000000001e-05 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869986+0000", 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6452999999999999e-05 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.870330+0000", 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034373599999999998 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.870356+0000", 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5995000000000001e-05 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:17991 3.a 3:55af03fe:::benchmark_data_smithi173_60780_object17990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.775010+0000", 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.1753765810000001, 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18266866800000001, 2024-08-30T23:30:24.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17991 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775010+0000", 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775010+0000", 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775012+0000", 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9259999999999999e-06 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:24.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775054+0000", 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2778000000000003e-05 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775067+0000", 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.255e-05 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775074+0000", 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4189999999999996e-06 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775141+0000", 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6618999999999994e-05 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775260+0000", 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011933 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775400+0000", 2024-08-30T23:30:24.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013916500000000001 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.776302+0000", 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090262000000000003 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.776808+0000", 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050635199999999995 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.957613+0000", 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18080433000000001 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.957648+0000", 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5076000000000001e-05 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.957678+0000", 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0502999999999999e-05 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:24.085 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:30:24.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:24.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:24.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:24.284 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:24.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 20058 20042 96.3484 81.25 0.0405276 0.0103666 2024-08-30T23:30:24.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:24.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:24.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:24.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:24.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:24.666 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:24.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:24 smithi106 ceph-mon[16603]: pgmap v550: 17 pgs: 17 active+clean; 872 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-08-30T23:30:24.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:24.781 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:24 smithi136 ceph-mon[20207]: pgmap v550: 17 pgs: 17 active+clean; 872 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-08-30T23:30:24.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:24 smithi173 ceph-mon[20554]: pgmap v550: 17 pgs: 17 active+clean; 872 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-08-30T23:30:24.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:30:24.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:24.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:24.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:25.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:25.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:25.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:25.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:25.195 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:25.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 22005 21989 98.1576 121.688 0.0165604 0.0101741 2024-08-30T23:30:25.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:25.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:25.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:25.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:25.486 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:30:25.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:25.566 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:25.580 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4106809061' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:30:25.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:25.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:25.714 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:30:25.714 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:30:25.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.714 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22406 3.f 3:f94f5d43:::benchmark_data_smithi173_60780_object22405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-30T23:30:25.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.702645+0000", 2024-08-30T23:30:25.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011105043, 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011129784, 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22406 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:30:25.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702645+0000", 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702645+0000", 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702657+0000", 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1263999999999999e-05 2024-08-30T23:30:25.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702693+0000", 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6100999999999999e-05 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702712+0000", 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9684e-05 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:30:25.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702720+0000", 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2540000000000003e-06 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702777+0000", 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7051e-05 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702881+0000", 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104497 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.702975+0000", 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4163e-05 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.703917+0000", 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094191499999999998 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.704433+0000", 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051624199999999998 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:25.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:30:25.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:25.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:30:25.720 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:30:25.720 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:30:25.720 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:30:25.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:25.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4106809061' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:30:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4106809061' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:30:25.824 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:25.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:30:25.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:25.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:26.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:26.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:26.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:26.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:26.288 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 22914 22898 95.4007 56.8125 0.0286246 0.0104166 2024-08-30T23:30:26.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:26.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:26.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:26.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:26.444 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:26.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:30:26.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:26.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:26 smithi106 ceph-mon[16603]: pgmap v551: 17 pgs: 17 active+clean; 1.1 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:30:26.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:26 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:26.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4106809061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:26.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:26 smithi106 ceph-mon[16603]: osdmap e215: 12 total, 12 up, 11 in 2024-08-30T23:30:26.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:26.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:26.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:26 smithi136 ceph-mon[20207]: pgmap v551: 17 pgs: 17 active+clean; 1.1 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:30:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:26 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4106809061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:26 smithi136 ceph-mon[20207]: osdmap e215: 12 total, 12 up, 11 in 2024-08-30T23:30:26.818 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:26.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:26 smithi173 ceph-mon[20554]: pgmap v551: 17 pgs: 17 active+clean; 1.1 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:30:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:26 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4106809061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:26 smithi173 ceph-mon[20554]: osdmap e215: 12 total, 12 up, 11 in 2024-08-30T23:30:26.835 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:26.835 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:26.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:26.860 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:26.860 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:26.884 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:26.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:26.907 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:26.908 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:26.933 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:26.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:26.964 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:26.964 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:26.977 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:26.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:26.990 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:26.990 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:27.012 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:27.013 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:27.033 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:27.034 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:27.057 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:27.057 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:27.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:27.081 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:27.081 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:27.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:27.106 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:27.107 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:27.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:27.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:27.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 24196 24180 94.4455 80.125 0.0246044 0.010578 2024-08-30T23:30:27.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:27.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:27.382 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:27.383 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:27.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:27.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3059264272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:27 smithi136 ceph-mon[20207]: pgmap v553: 17 pgs: 17 active+clean; 1.3 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:30:27.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:27.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:27.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:27.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3059264272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:27 smithi173 ceph-mon[20554]: pgmap v553: 17 pgs: 17 active+clean; 1.3 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:30:27.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:27.849 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3059264272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:27.849 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:27 smithi106 ceph-mon[16603]: pgmap v553: 17 pgs: 17 active+clean; 1.3 GiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-08-30T23:30:27.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:27.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:27.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:27.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:28.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:30:28.015 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:30:28.015 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-08-30T23:30:28.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd in 10 2024-08-30T23:30:28.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:30:28.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:28.128 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:28.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:28.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:28.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:30:28.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 25629 25613 94.1581 89.5625 0.00239365 0.0106052 2024-08-30T23:30:28.332 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:28.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:28.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:28.383 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:28.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:28.507 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:28.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:28.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:28.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:28.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:28.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:28.890 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:28.890 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:28.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:28.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:28.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:29.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:29.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:29.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:29.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:29.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:29.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 26663 26647 92.5171 64.625 0.00196556 0.0107536 2024-08-30T23:30:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:30:29.064+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e215 definitely_dead 0 2024-08-30T23:30:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3118726984' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:30:29.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:29.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:29.479 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3118726984' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:30:29.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:29.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:30:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:29.526 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:29.544 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3118726984' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:30:29.579 INFO:teuthology.orchestra.run.smithi106.stderr:marked in osd.10. 2024-08-30T23:30:29.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:29.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.650 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:29.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:29.773 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:29.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:29.779 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:29.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:29.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:30:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:29.903 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:29.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:30.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:30.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:30.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:30.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:30.175 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3118726984' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:30:30.176 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:30 smithi173 ceph-mon[20554]: osdmap e216: 12 total, 12 up, 12 in 2024-08-30T23:30:30.176 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:30 smithi173 ceph-mon[20554]: pgmap v555: 17 pgs: 17 active+clean; 1.5 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-30T23:30:30.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3118726984' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:30:30.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:30 smithi106 ceph-mon[16603]: osdmap e216: 12 total, 12 up, 12 in 2024-08-30T23:30:30.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:30 smithi106 ceph-mon[16603]: pgmap v555: 17 pgs: 17 active+clean; 1.5 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-30T23:30:30.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:30.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:30.281 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:30.281 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:30.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 26824 26808 88.1773 10.0625 0.00205965 0.0108856 2024-08-30T23:30:30.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.399 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:30.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:30.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.523 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:30.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3118726984' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:30:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:30 smithi136 ceph-mon[20207]: osdmap e216: 12 total, 12 up, 12 in 2024-08-30T23:30:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:30 smithi136 ceph-mon[20207]: pgmap v555: 17 pgs: 17 active+clean; 1.5 GiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-08-30T23:30:30.555 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-08-30T23:30:30.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:30.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:30.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:30.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:30.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:30.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:30.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:31.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:31.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:30:31.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:31.097 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:31.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:31.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:31.225 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:31 smithi106 ceph-mon[16603]: osdmap e217: 12 total, 12 up, 12 in 2024-08-30T23:30:31.225 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:31 smithi106 ceph-mon[16603]: osdmap e218: 12 total, 12 up, 12 in 2024-08-30T23:30:31.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:31.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:30:31.290198+0000 min lat: 0.00105037 max lat: 1.62853 avg lat: 0.0115403 2024-08-30T23:30:31.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:30:31.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 27722 27706 86.5745 56.125 0.00293501 0.0115403 2024-08-30T23:30:31.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:31.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:31 smithi173 ceph-mon[20554]: osdmap e217: 12 total, 12 up, 12 in 2024-08-30T23:30:31.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:31 smithi173 ceph-mon[20554]: osdmap e218: 12 total, 12 up, 12 in 2024-08-30T23:30:31.364 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:31.403 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:31.425 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:31.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:31 smithi136 ceph-mon[20207]: osdmap e217: 12 total, 12 up, 12 in 2024-08-30T23:30:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:31 smithi136 ceph-mon[20207]: osdmap e218: 12 total, 12 up, 12 in 2024-08-30T23:30:31.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:31.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:31.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:31.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:31.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:31.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:31.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:31.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:31.940 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:31.940 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:32.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:32.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:32.130 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:30:32.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:32.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:32.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 29062 29046 86.4397 83.75 0.0057516 0.0115629 2024-08-30T23:30:32.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:32.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:32.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:32.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:32.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:32.460 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:32.461 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3679528436' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:32.461 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:32 smithi106 ceph-mon[16603]: pgmap v558: 17 pgs: 17 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-30T23:30:32.461 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/875415309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:32.482 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:32.482 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:32.502 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:32.503 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:32.523 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:32.523 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:32.533 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3679528436' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:32.533 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:32 smithi173 ceph-mon[20554]: pgmap v558: 17 pgs: 17 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-30T23:30:32.533 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/875415309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:32.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3679528436' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:32 smithi136 ceph-mon[20207]: pgmap v558: 17 pgs: 17 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-30T23:30:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/875415309' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:32.544 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:32.544 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:32.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:32.565 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:32.565 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:32.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:32.586 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:32.586 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:32.607 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:32.607 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:32.628 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:32.628 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:32.654 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:32.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:32.689 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:32.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:32.704 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:32.721 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:32.722 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:32.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:32.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:32.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:32.965 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:32.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:33.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.097 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:33.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:33.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:30:33.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:33.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:33.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 30867 30851 87.638 112.812 0.00393519 0.011404 2024-08-30T23:30:33.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:33.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:33.435 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:33 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:33.436 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3679528436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:33.436 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:33 smithi106 ceph-mon[16603]: osdmap e219: 12 total, 12 up, 12 in 2024-08-30T23:30:33.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:33.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:33.503 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:33.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:33 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:33.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3679528436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:33.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:33 smithi136 ceph-mon[20207]: osdmap e219: 12 total, 12 up, 12 in 2024-08-30T23:30:33.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:33 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:33.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3679528436' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:33.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:33 smithi173 ceph-mon[20554]: osdmap e219: 12 total, 12 up, 12 in 2024-08-30T23:30:33.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:33.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:33.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:33.747 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:33.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:33.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:33.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:33.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:34.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:14225 2.5 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 265.17229468699998, 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24031856100000001, 2024-08-30T23:30:34.048 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14225 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:30:34.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873763+0000", 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7887999999999999e-05 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873773+0000", 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8700000000000004e-06 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873780+0000", 2024-08-30T23:30:34.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6850000000000004e-06 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873819+0000", 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9190999999999998e-05 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873906+0000", 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7612000000000001e-05 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:30:34.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873999+0000", 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2088000000000003e-05 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.874429+0000", 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043006099999999997 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108141+0000", 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23371239399999999 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.113956+0000", 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058150340000000002 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114021+0000", 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500999999999998e-05 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114063+0000", 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2512999999999998e-05 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:14225 2.5 e109/97 ondisk, result = 0)", 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 265.11127820399997, 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.173386651, 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.055 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934762+0000", 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9700000000000005e-07 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934766+0000", 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5630000000000002e-06 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108075+0000", 2024-08-30T23:30:34.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17330804799999999 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108080+0000", 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4049999999999999e-06 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108127+0000", 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7547999999999999e-05 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108135+0000", 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8059999999999995e-06 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108148+0000", 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2384e-05 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.059 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:17728 2.5 2:a3d96d17:::benchmark_data_smithi173_47195_object17727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 262.54211589699997, 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20912092800000001, 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.060 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 17728 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:30:34.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503924+0000", 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.158e-06 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503947+0000", 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2192999999999999e-05 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503960+0000", 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3098999999999999e-05 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503975+0000", 2024-08-30T23:30:34.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5167999999999999e-05 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504020+0000", 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5464999999999999e-05 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504106+0000", 2024-08-30T23:30:34.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5754000000000006e-05 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504188+0000", 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1489000000000005e-05 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504942+0000", 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075475800000000001 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.577391+0000", 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072448769999999996 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712945+0000", 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13555429899999999 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713005+0000", 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9135000000000002e-05 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713044+0000", 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.964e-05 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.067 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:17728 2.5 e111/97 ondisk, result = 0)", 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 262.54086037399998, 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20787409000000001, 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505185+0000", 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712478+0000", 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20729294100000001 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712485+0000", 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712934+0000", 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044926100000000001 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712941+0000", 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4999999999999996e-06 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713053+0000", 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011231900000000001 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:49996 2.5 2:aad518df:::benchmark_data_smithi173_47195_object49995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 238.78817871000001, 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20351808099999999, 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.072 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 49996 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257862+0000", 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.33e-06 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257920+0000", 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8489000000000002e-05 2024-08-30T23:30:34.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257934+0000", 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3372e-05 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257941+0000", 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6459999999999993e-06 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257996+0000", 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5043999999999997e-05 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258082+0000", 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5890000000000006e-05 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258176+0000", 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3366e-05 2024-08-30T23:30:34.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259466+0000", 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012900190000000001 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259518+0000", 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2787999999999997e-05 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461274+0000", 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201755238 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461333+0000", 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9394000000000001e-05 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461379+0000", 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5504999999999999e-05 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:56067 2.f 2:f6065497:::benchmark_data_smithi173_47195_object56066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 234.26024273300001, 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17632836900000001, 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 56067 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785797+0000", 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785815+0000", 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8224e-05 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785825+0000", 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9869999999999994e-06 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785832+0000", 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.029e-06 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785866+0000", 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3729999999999997e-05 2024-08-30T23:30:34.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785937+0000", 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1027999999999998e-05 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786021+0000", 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3511999999999996e-05 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787483+0000", 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014626260000000001 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787561+0000", 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7565999999999996e-05 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962059+0000", 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17449777999999999 2024-08-30T23:30:34.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962099+0000", 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9953000000000002e-05 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962125+0000", 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6278000000000001e-05 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56068 2.6 e124/97 2:62a84442:::benchmark_data_smithi173_47195_object56067:head v 124'3528, mlcod=124'3525)", 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 234.25964881100001, 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17324930699999999, 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786391+0000", 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3200000000000004e-07 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786446+0000", 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4661999999999999e-05 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958168+0000", 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171722186 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958175+0000", 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7009999999999999e-06 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958216+0000", 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1102000000000001e-05 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958230+0000", 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4688999999999999e-05 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.959607+0000", 2024-08-30T23:30:34.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013761559999999999 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.959640+0000", 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3079e-05 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:62555 2.5 2:a45a541d:::benchmark_data_smithi173_47195_object62554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:30:34.090 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 228.608153719, 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.175746766, 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62555 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437886+0000", 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5000000000000002e-07 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437908+0000", 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2069000000000002e-05 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437919+0000", 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.098e-05 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437927+0000", 2024-08-30T23:30:34.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9519999999999994e-06 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438090+0000", 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016235300000000001 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438190+0000", 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010084199999999999 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438298+0000", 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107347 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.439985+0000", 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016870419999999999 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.440527+0000", 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054198600000000005 2024-08-30T23:30:34.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613517+0000", 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17299020600000001 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613579+0000", 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1791000000000005e-05 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613632+0000", 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3448000000000001e-05 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:96989 2.f 2:f906d66e:::benchmark_data_smithi173_47195_object96988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 185.903348107, 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84153836199999998, 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.097 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 96989 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142692+0000", 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142718+0000", 2024-08-30T23:30:34.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5548999999999999e-05 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142729+0000", 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1126e-05 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142736+0000", 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1840000000000002e-06 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148635+0000", 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058993520000000001 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148767+0000", 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013146699999999999 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148916+0000", 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149083 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152326+0000", 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034105860000000002 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152337+0000", 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1100999999999999e-05 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984109+0000", 2024-08-30T23:30:34.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83177168400000001 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984180+0000", 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0542000000000003e-05 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984229+0000", 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9808000000000002e-05 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:96989 2.f e146/97 ondisk, result = 0)", 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 185.894740748, 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83293943299999995, 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:30:34.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151299+0000", 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5600000000000005e-07 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151308+0000", 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984021+0000", 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83271241100000004 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984029+0000", 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9379999999999992e-06 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984091+0000", 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.211e-05 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984101+0000", 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0368e-05 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984238+0000", 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013670199999999999 2024-08-30T23:30:34.107 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100665 2.6 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:30:34.108 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 180.702224261, 2024-08-30T23:30:34.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:34.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21497229600000001, 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100665 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:30:34.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343816+0000", 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.494e-06 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343824+0000", 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2570000000000001e-06 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343841+0000", 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7107000000000001e-05 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343850+0000", 2024-08-30T23:30:34.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4299999999999995e-06 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343921+0000", 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0264000000000004e-05 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344020+0000", 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9296999999999995e-05 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344100+0000", 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0465999999999998e-05 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.346504+0000", 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002403723 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347743+0000", 2024-08-30T23:30:34.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001238549 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558717+0000", 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21097406299999999 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558756+0000", 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9728000000000003e-05 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558787+0000", 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0917999999999998e-05 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100667 2.6 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 180.698257775, 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27402247099999999, 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100667 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347782+0000", 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:30:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347787+0000", 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3039999999999998e-06 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347798+0000", 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0834e-05 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347803+0000", 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.242e-06 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347840+0000", 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7630999999999999e-05 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347918+0000", 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7778e-05 2024-08-30T23:30:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347989+0000", 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0988999999999994e-05 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.349336+0000", 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013472670000000001 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.351224+0000", 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018879960000000001 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621737+0000", 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27051204000000001 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621775+0000", 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.875e-05 2024-08-30T23:30:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621804+0000", 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8620999999999999e-05 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:102301 2.f 2:fc0af6c0:::benchmark_data_smithi173_47195_object5293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:35.541666+0000", 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 178.504373456, 2024-08-30T23:30:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169542736, 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102301 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541666+0000", 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541666+0000", 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541667+0000", 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.209e-06 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541672+0000", 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.617e-06 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541685+0000", 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3052e-05 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.541691+0000", 2024-08-30T23:30:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6200000000000001e-06 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.542436+0000", 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074515699999999998 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.542542+0000", 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106061 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.542612+0000", 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9160999999999997e-05 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.544890+0000", 2024-08-30T23:30:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022781839999999999 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.545054+0000", 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164196 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.711142+0000", 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16608751799999999 2024-08-30T23:30:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.711184+0000", 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2862999999999999e-05 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:35.711208+0000", 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4097999999999999e-05 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:127040 2.6 2:6091191f:::benchmark_data_smithi173_47195_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 146.19489235, 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18230701299999999, 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127040 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851148+0000", 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851155+0000", 2024-08-30T23:30:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8990000000000004e-06 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851175+0000", 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9711000000000002e-05 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851184+0000", 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6780000000000006e-06 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851247+0000", 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2780000000000005e-05 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851360+0000", 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112946 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851485+0000", 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012559 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.854952+0000", 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034665049999999999 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.862420+0000", 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0074685050000000003 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033372+0000", 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17095152199999999 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033407+0000", 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5567e-05 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033454+0000", 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681000000000002e-05 2024-08-30T23:30:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:144898 2.6 2:692f2db4:::benchmark_data_smithi173_47195_object47890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 122.366871592, 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18135380100000001, 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144898 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679169+0000", 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679174+0000", 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.198e-06 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679187+0000", 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.31e-05 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679194+0000", 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679236+0000", 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1828e-05 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679316+0000", 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0026999999999998e-05 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679383+0000", 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6925e-05 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.682228+0000", 2024-08-30T23:30:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028449149999999999 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.829406+0000", 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14717798100000001 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860431+0000", 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031025377 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860482+0000", 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0779000000000003e-05 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860521+0000", 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9307999999999997e-05 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:145440 2.f 2:fe697e68:::benchmark_data_smithi173_47195_object48432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:30:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:32.437723+0000", 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.608316247, 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16980590700000001, 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 145440 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437723+0000", 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437723+0000", 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437724+0000", 2024-08-30T23:30:34.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1850000000000001e-06 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437730+0000", 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8939999999999998e-06 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437745+0000", 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4686e-05 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437752+0000", 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8040000000000004e-06 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437798+0000", 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6683e-05 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437878+0000", 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0160000000000005e-05 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.437947+0000", 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8508000000000002e-05 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.442029+0000", 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040821110000000002 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.442043+0000", 2024-08-30T23:30:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4347e-05 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.607466+0000", 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16542242900000001 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.607500+0000", 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4694999999999999e-05 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:32.607529+0000", 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8405e-05 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:147254 2.f 2:f940aba9:::benchmark_data_smithi173_47195_object50246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 118.926534159, 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17365630000000001, 2024-08-30T23:30:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 147254 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119506+0000", 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3069999999999999e-06 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119512+0000", 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5450000000000003e-06 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119525+0000", 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2979999999999999e-05 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119532+0000", 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.99e-06 2024-08-30T23:30:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119579+0000", 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7336999999999998e-05 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119668+0000", 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8710999999999998e-05 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119740+0000", 2024-08-30T23:30:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2224999999999994e-05 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.123093+0000", 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033526760000000002 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.123100+0000", 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5610000000000001e-06 2024-08-30T23:30:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293060+0000", 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169959361 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293114+0000", 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3825000000000001e-05 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293161+0000", 2024-08-30T23:30:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7781999999999998e-05 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:d5896595:::benchmark_data_smithi173_47195_object12946:head,153'6861)", 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 50.308592036999997, 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59714269600000003, 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737449+0000", 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737471+0000", 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2631999999999999e-05 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334475+0000", 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59700367600000004 2024-08-30T23:30:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334480+0000", 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2719999999999997e-06 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334536+0000", 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5430999999999998e-05 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334546+0000", 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9389999999999992e-06 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334590+0000", 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4265999999999999e-05 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:dc4db0cb:::benchmark_data_smithi173_47195_object13755:head,153'6902)", 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 49.496782887999998, 2024-08-30T23:30:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85400190600000003, 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:30:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549257+0000", 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549260+0000", 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.45e-06 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403003+0000", 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85374268200000003 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403016+0000", 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3033e-05 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403163+0000", 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014668400000000001 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403180+0000", 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6813e-05 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403258+0000", 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8710999999999999e-05 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.5 e202,182 2:ac5fcd3c:::benchmark_data_smithi173_47195_object8560:head,150'6576)", 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.057550560999999, 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91766887600000002, 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1099999999999996e-07 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988496+0000", 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4989999999999999e-06 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906006+0000", 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91750992200000003 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906016+0000", 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0199e-05 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906087+0000", 2024-08-30T23:30:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1316999999999995e-05 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906098+0000", 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0808000000000001e-05 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906158+0000", 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9620000000000002e-05 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:34.171 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:34.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:34.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 32422 32406 88.0529 97.1875 0.00190997 0.0113455 2024-08-30T23:30:34.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:34.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:34 smithi173 ceph-mon[20554]: pgmap v560: 17 pgs: 17 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 634 op/s 2024-08-30T23:30:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:34 smithi106 ceph-mon[16603]: pgmap v560: 17 pgs: 17 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 634 op/s 2024-08-30T23:30:34.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:34.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:34.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:34 smithi136 ceph-mon[20207]: pgmap v560: 17 pgs: 17 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 634 op/s 2024-08-30T23:30:34.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:34.576 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:34.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:34.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:34.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:34.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:34.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:34.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:34.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:34.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:30:35.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:35.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:35.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:35.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:35.247 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:35.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 33398 33382 86.9255 61 0.00223835 0.0114947 2024-08-30T23:30:35.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:35.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:35.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:35.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:35.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:30:35.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:35.556 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:30:35.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:30:35.556 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:30:35.556 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.360636 2024-08-30T23:30:35.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 0 0.3606358361096095 2024-08-30T23:30:35.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:35.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:35.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:35.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:35.829 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:35.844 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:35.845 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:35.845 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:30:35.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.845 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:34082 3.d e221/206 3:bee49eab:::benchmark_data_smithi173_60780_object34081:head v 221'2105, mlcod=221'2103)", 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:35.796012+0000", 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00029424200000000001, 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032978499999999997, 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796012+0000", 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796012+0000", 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796013+0000", 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0500000000000003e-07 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796051+0000", 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7836000000000002e-05 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796081+0000", 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0499999999999999e-05 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796088+0000", 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2840000000000003e-06 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796138+0000", 2024-08-30T23:30:35.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0605000000000001e-05 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.796153+0000", 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4392e-05 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-08-30T23:30:35.849 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:35.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:35.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:30:35.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:35.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:36.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:36.100 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:36.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:36.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:36.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:36.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:36.236 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:36 smithi106 ceph-mon[16603]: osdmap e220: 12 total, 12 up, 12 in 2024-08-30T23:30:36.236 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:36 smithi106 ceph-mon[16603]: pgmap v562: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s; 1354/89214 objects misplaced (1.518%); 3.6 MiB/s, 56 objects/s recovering 2024-08-30T23:30:36.236 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:36 smithi106 ceph-mon[16603]: osdmap e221: 12 total, 12 up, 12 in 2024-08-30T23:30:36.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 34479 34463 86.1507 67.5625 0.00183443 0.0115762 2024-08-30T23:30:36.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:36.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:36.350 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:36.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:36.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:36.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:36 smithi136 ceph-mon[20207]: osdmap e220: 12 total, 12 up, 12 in 2024-08-30T23:30:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:36 smithi136 ceph-mon[20207]: pgmap v562: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s; 1354/89214 objects misplaced (1.518%); 3.6 MiB/s, 56 objects/s recovering 2024-08-30T23:30:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:36 smithi136 ceph-mon[20207]: osdmap e221: 12 total, 12 up, 12 in 2024-08-30T23:30:36.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:36.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:36 smithi173 ceph-mon[20554]: osdmap e220: 12 total, 12 up, 12 in 2024-08-30T23:30:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:36 smithi173 ceph-mon[20554]: pgmap v562: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 1.8 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s; 1354/89214 objects misplaced (1.518%); 3.6 MiB/s, 56 objects/s recovering 2024-08-30T23:30:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:36 smithi173 ceph-mon[20554]: osdmap e221: 12 total, 12 up, 12 in 2024-08-30T23:30:36.585 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:36.616 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:36.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:36.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:30:36.747 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:36.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:36.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:36.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:36.907 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:36.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:37.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:37.062 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:37.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:37.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:37.147 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:30:37.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:37.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:37.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 35887 35871 86.2216 88 0.00254525 0.0115828 2024-08-30T23:30:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3222082607' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3222082607' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:37.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:37.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:37.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:37.498 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3222082607' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:37.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:37.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:37.612 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:37.612 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:30:37.612 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:30:37.612 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:30:37.612 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:37.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:37.692 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:37.692 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:37.717 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:37.717 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:37.744 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:37.744 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:37.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:37.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:37.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:37.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:37.796 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:37.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:37.822 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:37.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:37.854 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:37.854 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:37.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:37.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:37.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:37.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:37.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:37.912 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:37.912 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:37.939 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:37.939 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:37.964 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:37.964 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:37.990 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:37.990 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:38.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:38.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:38.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:38.149 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.0 primary-affinity to 0.360636 (8236342) 2024-08-30T23:30:38.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:38.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:38.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:38.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 36876 36860 85.3174 61.8125 0.0448718 0.0117002 2024-08-30T23:30:38.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:30:38.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:38.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:38.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:38.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3222082607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: osdmap e222: 12 total, 12 up, 12 in 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4228295039' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.3606358361096095} : dispatch 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: pgmap v565: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s; 1354/99516 objects misplaced (1.361%); 3.6 MiB/s, 57 objects/s recovering 2024-08-30T23:30:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3583946598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3222082607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: osdmap e222: 12 total, 12 up, 12 in 2024-08-30T23:30:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4228295039' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.3606358361096095} : dispatch 2024-08-30T23:30:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: pgmap v565: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s; 1354/99516 objects misplaced (1.361%); 3.6 MiB/s, 57 objects/s recovering 2024-08-30T23:30:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3583946598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:38.591 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3222082607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: osdmap e222: 12 total, 12 up, 12 in 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4228295039' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.3606358361096095} : dispatch 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: pgmap v565: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s; 1354/99516 objects misplaced (1.361%); 3.6 MiB/s, 57 objects/s recovering 2024-08-30T23:30:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3583946598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:38.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:38.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:38.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:38.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:38.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:38.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:38.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:39.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:39.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:39.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:39.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:39.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:39.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:39.289 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 37096 37080 82.7613 13.75 0.00380847 0.0117525 2024-08-30T23:30:39.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:39.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:39.453 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4228295039' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.3606358361096095}]': finished 2024-08-30T23:30:39.453 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:39 smithi106 ceph-mon[16603]: osdmap e223: 12 total, 12 up, 12 in 2024-08-30T23:30:39.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:39.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4228295039' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.3606358361096095}]': finished 2024-08-30T23:30:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:39 smithi136 ceph-mon[20207]: osdmap e223: 12 total, 12 up, 12 in 2024-08-30T23:30:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4228295039' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.3606358361096095}]': finished 2024-08-30T23:30:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:39 smithi173 ceph-mon[20554]: osdmap e223: 12 total, 12 up, 12 in 2024-08-30T23:30:39.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:39.642 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:39.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.767 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:39.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:39.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:39.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:39.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:39.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:39.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:40.018 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:40.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:40.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:40.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:40.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:40.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 37353 37337 80.4612 16.0625 0.00740921 0.0124231 2024-08-30T23:30:40.296 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:40.296 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:40.296 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:40 smithi173 ceph-mon[20554]: osdmap e224: 12 total, 12 up, 12 in 2024-08-30T23:30:40.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:40 smithi173 ceph-mon[20554]: pgmap v568: 17 pgs: 1 remapped+peering, 16 active+clean; 2.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 5.7 MiB/s, 90 objects/s recovering 2024-08-30T23:30:40.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:40.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:30:40.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:40.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:40.530 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:40 smithi136 ceph-mon[20207]: osdmap e224: 12 total, 12 up, 12 in 2024-08-30T23:30:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:40 smithi136 ceph-mon[20207]: pgmap v568: 17 pgs: 1 remapped+peering, 16 active+clean; 2.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 5.7 MiB/s, 90 objects/s recovering 2024-08-30T23:30:40.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:40 smithi106 ceph-mon[16603]: osdmap e224: 12 total, 12 up, 12 in 2024-08-30T23:30:40.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:40 smithi106 ceph-mon[16603]: pgmap v568: 17 pgs: 1 remapped+peering, 16 active+clean; 2.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s; 5.7 MiB/s, 90 objects/s recovering 2024-08-30T23:30:40.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:40.660 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:40.704 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:40.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:40.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:40.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:40.913 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:40.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:41.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:41.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:41.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:30:41.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:41.187 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:41.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:41.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 38692 38676 80.5685 83.6875 0.0134349 0.0124039 2024-08-30T23:30:41.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:41 smithi173 ceph-mon[20554]: osdmap e225: 12 total, 12 up, 12 in 2024-08-30T23:30:41.390 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:41.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:41.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:41 smithi136 ceph-mon[20207]: osdmap e225: 12 total, 12 up, 12 in 2024-08-30T23:30:41.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:41.550 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:41 smithi106 ceph-mon[16603]: osdmap e225: 12 total, 12 up, 12 in 2024-08-30T23:30:41.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:41.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:41.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:41.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:41.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:41.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:30:41.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:41.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:41.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:41.971 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:41.971 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:42.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:42.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:42.167 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:30:42.179 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:42.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:42.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:42.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 39759 39743 80.1206 66.6875 0.0387598 0.0124722 2024-08-30T23:30:42.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:42.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:42.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:42.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:42.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:42 smithi106 ceph-mon[16603]: pgmap v570: 17 pgs: 1 remapped+peering, 16 active+clean; 2.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s; 5.3 MiB/s, 84 objects/s recovering 2024-08-30T23:30:42.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3058818943' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:42 smithi136 ceph-mon[20207]: pgmap v570: 17 pgs: 1 remapped+peering, 16 active+clean; 2.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s; 5.3 MiB/s, 84 objects/s recovering 2024-08-30T23:30:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3058818943' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:42.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:42 smithi173 ceph-mon[20554]: pgmap v570: 17 pgs: 1 remapped+peering, 16 active+clean; 2.3 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s; 5.3 MiB/s, 84 objects/s recovering 2024-08-30T23:30:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3058818943' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:42.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:42.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:42.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:42.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:30:42.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:42.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:42.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:42.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:42.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:42.985 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:42.985 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:43.009 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:43.009 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:43.035 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:43.035 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:43.061 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:43.061 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:43.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:43.088 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:43.089 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:43.106 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:43.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:43.119 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:43.119 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:43.146 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:43.146 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:43.175 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:43.175 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:43.202 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:43.202 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:43.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:43.226 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:43.226 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:43.237 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:43.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:43.249 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:43.249 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:43.269 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:43.269 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:43.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 41099 41083 80.2338 83.75 0.0290383 0.0124321 2024-08-30T23:30:43.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:43.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:43.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:43.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:43.507 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:43 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:30:43.507 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:43 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:30:43.507 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3058818943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:43.507 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:43 smithi173 ceph-mon[20554]: osdmap e226: 12 total, 12 up, 12 in 2024-08-30T23:30:43.507 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2644096385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:43 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:30:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:43 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:30:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3058818943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:43 smithi136 ceph-mon[20207]: osdmap e226: 12 total, 12 up, 12 in 2024-08-30T23:30:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2644096385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:43.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:43 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:30:43.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:43 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:30:43.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3058818943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:43.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:43 smithi106 ceph-mon[16603]: osdmap e226: 12 total, 12 up, 12 in 2024-08-30T23:30:43.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2644096385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:43.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:43.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:43.618 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:43.618 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:30:43.618 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:30:43.618 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:30:43.618 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:43.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:43.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:43.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:43.866 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:43.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:43.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:43.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.001 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:44.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:44.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:44.143 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:30:44.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:30:44.143 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-08-30T23:30:44.143 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:30:44.143 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:30:44.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-08-30T23:30:44.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:44.250 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:44 smithi106 ceph-mon[16603]: pgmap v572: 17 pgs: 1 remapped+peering, 16 active+clean; 2.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 896 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:30:44.250 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:44 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:44.251 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:44 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:44.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:44.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 42170 42154 79.8307 66.9375 0.00790653 0.0125177 2024-08-30T23:30:44.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:30:44.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:44.386 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:44.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:44.512 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:30:44.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:44 smithi136 ceph-mon[20207]: pgmap v572: 17 pgs: 1 remapped+peering, 16 active+clean; 2.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 896 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:30:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:44 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:44 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:44 smithi173 ceph-mon[20554]: pgmap v572: 17 pgs: 1 remapped+peering, 16 active+clean; 2.3 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 896 op/s; 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:30:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:44 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:44 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:44.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:44.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:44.653 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:44.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:44.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:44.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:44.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:44.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:45.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:45.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:45.074 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:30:45.074 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:30:45.074 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:30:45.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:45.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:45.217 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:30:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:30:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:30:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:16911 2.4 e111/97 2:26b3b856:::benchmark_data_smithi173_47195_object16910:head v 111'1012, mlcod=111'1009)", 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 274.54265823700001, 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225497685, 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672856+0000", 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0200000000000003e-07 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672918+0000", 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1801e-05 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897644+0000", 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22472613699999999 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897651+0000", 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.99e-06 2024-08-30T23:30:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897700+0000", 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9373000000000003e-05 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897721+0000", 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1314e-05 2024-08-30T23:30:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.898318+0000", 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059680200000000001 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.898353+0000", 2024-08-30T23:30:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4866000000000001e-05 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:19019 2.7 e111/97 2:e1c9e35c:::benchmark_data_smithi173_47195_object19018:head v 111'1208, mlcod=111'1204)", 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 272.74386188900002, 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.113963728, 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471652+0000", 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7700000000000004e-07 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.471730+0000", 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7249000000000005e-05 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584625+0000", 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11289540200000001 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584630+0000", 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9470000000000002e-06 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584749+0000", 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011939599999999999 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.584777+0000", 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7206e-05 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.585586+0000", 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00080899100000000001 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:12.585616+0000", 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9960000000000001e-05 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:20668 2.0 e112/97 2:033ae7b0:::benchmark_data_smithi173_47195_object20667:head v 112'1312, mlcod=112'1308)", 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:13.671018+0000", 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.544495262, 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.088517390000000001, 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671018+0000", 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671018+0000", 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671019+0000", 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1900000000000002e-07 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.671073+0000", 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4072e-05 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758556+0000", 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.087482480000000001 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758563+0000", 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4139999999999997e-06 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758635+0000", 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1639999999999998e-05 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.758659+0000", 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3864999999999999e-05 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.759476+0000", 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081773200000000005 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:13.759536+0000", 2024-08-30T23:30:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9568000000000003e-05 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:22849 2.7 e112/97 2:ebcd612e:::benchmark_data_smithi173_47195_object22848:head v 112'1442, mlcod=112'1439)", 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:15.244998+0000", 2024-08-30T23:30:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 269.97051535499997, 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117383665, 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.244998+0000", 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.244998+0000", 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.244999+0000", 2024-08-30T23:30:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.245059+0000", 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9746e-05 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361366+0000", 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.116306929 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361370+0000", 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6070000000000001e-06 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361409+0000", 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8924000000000002e-05 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.361423+0000", 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3698e-05 2024-08-30T23:30:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.362348+0000", 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000925521 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.362382+0000", 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.362e-05 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:31164 2.4 e116/97 2:2f94b145:::benchmark_data_smithi173_47195_object31163:head v 116'1913, mlcod=116'1909)", 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:21.144978+0000", 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 264.07053607799998, 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080170878000000001, 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.144978+0000", 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.144978+0000", 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.144979+0000", 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2410000000000001e-06 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.145008+0000", 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9189000000000001e-05 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224228+0000", 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.079219519000000002 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224233+0000", 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8250000000000001e-06 2024-08-30T23:30:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224272+0000", 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8659999999999999e-05 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.224286+0000", 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4358999999999999e-05 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.225109+0000", 2024-08-30T23:30:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082302500000000002 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:21.225149+0000", 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9060000000000002e-05 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:39829 2.7 e119/97 2:ea0d6c5f:::benchmark_data_smithi173_47195_object39828:head v 119'2487, mlcod=119'2483)", 2024-08-30T23:30:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:27.415758+0000", 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 257.79975522400002, 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083043754999999997, 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415758+0000", 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415758+0000", 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415759+0000", 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1200000000000002e-07 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.415875+0000", 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011565000000000001 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497633+0000", 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.081757904000000006 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497642+0000", 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8179999999999993e-06 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497694+0000", 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2194000000000002e-05 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.497711+0000", 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7136e-05 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.498758+0000", 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010474569999999999 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.498802+0000", 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3884000000000002e-05 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:39861 2.4 e119/97 2:2f9ea911:::benchmark_data_smithi173_47195_object39860:head v 119'2492, mlcod=119'2489)", 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 257.77893710900003, 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.090130358999999993, 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:30:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436577+0000", 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7100000000000001e-07 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.436632+0000", 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5006000000000002e-05 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525705+0000", 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.089072993000000003 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525710+0000", 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6310000000000002e-06 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525758+0000", 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8069999999999999e-05 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.525773+0000", 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4843e-05 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.526680+0000", 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090692099999999999 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:27.526707+0000", 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7223999999999999e-05 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:43503 2.0 e119/97 2:0f4dae72:::benchmark_data_smithi173_47195_object43502:head v 119'2693, mlcod=119'2690)", 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:30.305617+0000", 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 254.909896535, 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104864759, 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.252 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305617+0000", 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305617+0000", 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305618+0000", 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.051e-06 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.305661+0000", 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3087999999999998e-05 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409257+0000", 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.103595863 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409264+0000", 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4119999999999998e-06 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409321+0000", 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7048e-05 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.409344+0000", 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3490999999999999e-05 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.410451+0000", 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011067150000000001 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:30.410482+0000", 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1090999999999998e-05 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:50002 2.7 e122/97 2:e84aa78e:::benchmark_data_smithi173_47195_object50001:head v 122'3082, mlcod=122'3079)", 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:35.268911+0000", 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 249.946602426, 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19888059999999999, 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268911+0000", 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268911+0000", 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268913+0000", 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.305e-06 2024-08-30T23:30:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.268950+0000", 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7697000000000002e-05 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466526+0000", 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19757559199999999 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466531+0000", 2024-08-30T23:30:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3940000000000004e-06 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466584+0000", 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349000000000003e-05 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.466605+0000", 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1880000000000001e-05 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.467760+0000", 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011541419999999999 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.467792+0000", 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2240999999999998e-05 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:54264 2.4 e123/97 2:260bcf68:::benchmark_data_smithi173_47195_object54263:head v 123'3380, mlcod=123'3376)", 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:38.314919+0000", 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 246.90059475999999, 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.098021554999999996, 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314919+0000", 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314919+0000", 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314920+0000", 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.314981+0000", 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1614999999999999e-05 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411576+0000", 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096594261000000001 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411581+0000", 2024-08-30T23:30:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5799999999999999e-06 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411629+0000", 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.74e-05 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.411643+0000", 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.473e-05 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.412895+0000", 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001251963 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:38.412941+0000", 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5126000000000001e-05 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56070 2.0 e124/97 2:06759afd:::benchmark_data_smithi173_47195_object56069:head v 124'3466, mlcod=124'3463)", 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.791057+0000", 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 245.424456365, 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19941077600000001, 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791057+0000", 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791057+0000", 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791059+0000", 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.167e-06 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.791090+0000", 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1420000000000001e-05 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988501+0000", 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.197411267 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988508+0000", 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0829999999999998e-06 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988600+0000", 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1765e-05 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.988620+0000", 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9877999999999998e-05 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.990423+0000", 2024-08-30T23:30:45.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018031080000000001 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.990468+0000", 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5087999999999999e-05 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56072 2.7 e124/97 2:e7493b3b:::benchmark_data_smithi173_47195_object56071:head v 124'3469, mlcod=124'3466)", 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.795821+0000", 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 245.41969244800001, 2024-08-30T23:30:45.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17323951900000001, 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795821+0000", 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795821+0000", 2024-08-30T23:30:45.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795822+0000", 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7599999999999996e-07 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.795863+0000", 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0525000000000002e-05 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967597+0000", 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17173391800000001 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967603+0000", 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.674e-06 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967659+0000", 2024-08-30T23:30:45.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6178000000000003e-05 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.967682+0000", 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2158e-05 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.969029+0000", 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013469790000000001 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.969061+0000", 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2211e-05 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62661 2.7 e127/97 2:e0aff4c8:::benchmark_data_smithi173_47195_object62660:head v 127'3889, mlcod=127'3886)", 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.608751+0000", 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.60676251199999, 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.085590695999999994, 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.276 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608751+0000", 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608751+0000", 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608752+0000", 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7000000000000003e-07 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.608824+0000", 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692767+0000", 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083942873000000001 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692772+0000", 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4280000000000004e-06 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692821+0000", 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8829000000000003e-05 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.692835+0000", 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4347e-05 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.694308+0000", 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00147241 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.694342+0000", 2024-08-30T23:30:45.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4131999999999998e-05 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62809 2.7 e127/97 2:e4464b6d:::benchmark_data_smithi173_47195_object62808:head v 127'3894, mlcod=127'3891)", 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.781150+0000", 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.43436414799999, 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080382227000000001, 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.281 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781150+0000", 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781150+0000", 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781151+0000", 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1799999999999999e-06 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.781213+0000", 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2241000000000003e-05 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.859966+0000", 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078752848 2024-08-30T23:30:45.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.859975+0000", 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1060000000000006e-06 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.860026+0000", 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0804999999999999e-05 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.860042+0000", 2024-08-30T23:30:45.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6072000000000001e-05 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.861502+0000", 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014604100000000001 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.861532+0000", 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9564999999999999e-05 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.285 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62959 2.7 e127/97 2:e1c982e1:::benchmark_data_smithi173_47195_object62958:head v 127'3909, mlcod=127'3906)", 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:46.086449+0000", 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.129064551, 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.080663077, 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086449+0000", 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086449+0000", 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086450+0000", 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:30:45.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.086546+0000", 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6031000000000003e-05 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165445+0000", 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.078899104999999997 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165452+0000", 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4669999999999998e-06 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165514+0000", 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1694999999999998e-05 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.165535+0000", 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1128000000000002e-05 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.167079+0000", 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015441529999999999 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.167112+0000", 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2777999999999997e-05 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:77053 2.0 e134/97 2:06a5d6c0:::benchmark_data_smithi173_47195_object77052:head v 134'4820, mlcod=134'4817)", 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:00.133250+0000", 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.08226394100001, 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.086050852999999997, 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133250+0000", 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133250+0000", 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133251+0000", 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2799999999999995e-07 2024-08-30T23:30:45.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.133280+0000", 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8938999999999999e-05 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216909+0000", 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.083629259999999997 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216916+0000", 2024-08-30T23:30:45.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:45.295 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:45 smithi136 ceph-mon[20207]: osdmap e227: 12 total, 12 up, 12 in 2024-08-30T23:30:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:30:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216976+0000", 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9905999999999999e-05 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.216998+0000", 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2308000000000001e-05 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.219257+0000", 2024-08-30T23:30:45.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022584319999999999 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:00.219301+0000", 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3939000000000001e-05 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(3.1 220/217 [PushOp(3:86782519:::benchmark_data_smithi173_60780_object6046:head, version: 213'407, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:86782519:::benchmark_data_smithi173_60780_object6046:head@213'407, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:35.148081+0000", 2024-08-30T23:30:45.297 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.067433066, 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15428572300000001, 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.148081+0000", 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.148081+0000", 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.148082+0000", 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6499999999999998e-07 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.148132+0000", 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0565000000000001e-05 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.301490+0000", 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.153358356 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.301507+0000", 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6748e-05 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.301591+0000", 2024-08-30T23:30:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3784999999999998e-05 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.301600+0000", 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9129999999999992e-06 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.302366+0000", 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000766491 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.301 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:34400 3.1 e221/217 3:8a2236d9:::benchmark_data_smithi173_60780_object34399:head v 221'2095, mlcod=213'476)", 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:36.101526+0000", 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.1139878779999997, 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16274208800000001, 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.101526+0000", 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.101526+0000", 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.101527+0000", 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3489999999999999e-06 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.101571+0000", 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3727000000000003e-05 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.262848+0000", 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16127717 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.262856+0000", 2024-08-30T23:30:45.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9629999999999998e-06 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.262911+0000", 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4962e-05 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.262930+0000", 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8989000000000001e-05 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:30:45.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.264207+0000", 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001276744 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.264268+0000", 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1184000000000002e-05 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.306 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(3.1 221/217 [PushOp(3:83fee770:::benchmark_data_smithi173_60780_object7264:head, version: 213'478, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:83fee770:::benchmark_data_smithi173_60780_object7264:head@213'478, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:36.106277+0000", 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.1092363229999993, 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16377182200000001, 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.106277+0000", 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.106277+0000", 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.106278+0000", 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6800000000000009e-07 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.106348+0000", 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9270999999999995e-05 2024-08-30T23:30:45.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.269277+0000", 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.162929504 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.269289+0000", 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2289e-05 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.269384+0000", 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5026999999999997e-05 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.269400+0000", 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5591e-05 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.270049+0000", 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00064917200000000005 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.310 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(3.1 222/217 [PushOp(3:832577c8:::benchmark_data_smithi173_60780_object8675:head, version: 213'560, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:832577c8:::benchmark_data_smithi173_60780_object8675:head@213'560, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:37.373994+0000", 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 7.8415201300000001, 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.097458509999999998, 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:45.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.373994+0000", 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.373994+0000", 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.373995+0000", 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.085e-06 2024-08-30T23:30:45.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.374035+0000", 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0126000000000002e-05 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.470681+0000", 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.096645731999999998 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:45.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.470693+0000", 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2689e-05 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.470765+0000", 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1544999999999993e-05 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.470775+0000", 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0332999999999999e-05 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:45.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:37.471452+0000", 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067699999999999998 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:45.315 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:45.315 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 43618 43602 80.1443 90.5 0.00896731 0.012469 2024-08-30T23:30:45.338 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:45.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:45.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:45.556 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:45 smithi173 ceph-mon[20554]: osdmap e227: 12 total, 12 up, 12 in 2024-08-30T23:30:45.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:45.591 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:45.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:30:45.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:45 smithi106 ceph-mon[16603]: osdmap e227: 12 total, 12 up, 12 in 2024-08-30T23:30:45.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:45.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:45.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:45.908 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:45.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:45.983 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:45.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:46.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:30:46.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:46.105 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:46.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:46.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:46.269 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:46 smithi173 ceph-mon[20554]: pgmap v574: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 763 op/s; 497/120657 objects misplaced (0.412%); 5.1 MiB/s, 81 objects/s recovering 2024-08-30T23:30:46.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:30:46.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 15 44615 44600 79.6364 62.375 0.00360436 0.0125498 2024-08-30T23:30:46.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:46 smithi106 ceph-mon[16603]: pgmap v574: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 763 op/s; 497/120657 objects misplaced (0.412%); 5.1 MiB/s, 81 objects/s recovering 2024-08-30T23:30:46.326 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:46.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:46.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:46.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:46.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:46.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:46 smithi136 ceph-mon[20207]: pgmap v574: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 763 op/s; 497/120657 objects misplaced (0.412%); 5.1 MiB/s, 81 objects/s recovering 2024-08-30T23:30:46.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:46.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:46.627 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:46.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:46.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:46.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:46.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:46.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:45547 3.8 e228/206 ondisk, result = 0)", 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:46.900665+0000", 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00059464100000000003, 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061561999999999999, 2024-08-30T23:30:46.902 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.900665+0000", 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:46.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.900665+0000", 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.900665+0000", 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9500000000000002e-07 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.900669+0000", 2024-08-30T23:30:46.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5700000000000001e-06 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.900695+0000", 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5795999999999998e-05 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.900704+0000", 2024-08-30T23:30:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2769999999999994e-06 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901083+0000", 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000379372 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901087+0000", 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0199999999999996e-06 2024-08-30T23:30:46.906 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:45554 3.e 3:7826815e:::benchmark_data_smithi173_60780_object45553:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:46.901115+0000", 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00014444600000000001, 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023065999999999999, 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:46.907 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45554 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901115+0000", 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:46.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901115+0000", 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901116+0000", 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3340000000000001e-06 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901139+0000", 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2320999999999999e-05 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901149+0000", 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0441999999999999e-05 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901156+0000", 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8900000000000001e-06 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901211+0000", 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5281000000000001e-05 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901348+0000", 2024-08-30T23:30:46.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013681500000000001 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:45548 3.0 3:022dcd3f:::benchmark_data_smithi173_60780_object45547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:46.898279+0000", 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0029801580000000001, 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031264299999999999, 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:30:46.912 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45548 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898279+0000", 2024-08-30T23:30:46.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898279+0000", 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898280+0000", 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7300000000000005e-07 2024-08-30T23:30:46.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898300+0000", 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9615000000000001e-05 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898311+0000", 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.167e-05 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898318+0000", 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4779999999999996e-06 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898360+0000", 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2092000000000003e-05 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898490+0000", 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013010699999999999 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.898575+0000", 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4884999999999998e-05 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:30:46.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901063+0000", 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024880169999999999 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:46.901095+0000", 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2224000000000002e-05 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:30:46.918 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:30:46.919 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-08-30T23:30:46.919 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:46.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:47.014 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:47.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:47.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:47.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:47.179 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:47 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:47.180 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:47 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:47.186 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:30:47.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:47.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:47.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 46083 46067 79.971 91.6875 0.004168 0.0124942 2024-08-30T23:30:47.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:47 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:47.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:47 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:47.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:47 smithi106 ceph-mon[16603]: osdmap e228: 12 total, 12 up, 12 in 2024-08-30T23:30:47.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1396508028' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:30:47.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.400 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:47.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:47.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:47 smithi173 ceph-mon[20554]: osdmap e228: 12 total, 12 up, 12 in 2024-08-30T23:30:47.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1396508028' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:30:47.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:47.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:47.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:47 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:47.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:47 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:47.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:47 smithi136 ceph-mon[20207]: osdmap e228: 12 total, 12 up, 12 in 2024-08-30T23:30:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1396508028' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:30:47.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:47.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:30:47.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:47.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:47.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:47.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:47.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:47.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:48.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:48.023 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:48.024 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:48.024 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:48.024 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:48.024 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:48.025 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:48.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:48.027 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:48.027 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:48.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:48.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:48.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:48.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:48.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:48 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1396508028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:48 smithi136 ceph-mon[20207]: osdmap e229: 12 total, 12 up, 12 in 2024-08-30T23:30:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1120426251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:48 smithi136 ceph-mon[20207]: pgmap v577: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.6 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 497/129855 objects misplaced (0.383%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:30:48.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 47426 47410 80.078 83.9375 0.017264 0.0124809 2024-08-30T23:30:48.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:48.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:48.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:48.493 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:48 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:48.494 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1396508028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:48.494 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:48 smithi173 ceph-mon[20554]: osdmap e229: 12 total, 12 up, 12 in 2024-08-30T23:30:48.494 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1120426251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:48.494 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:48 smithi173 ceph-mon[20554]: pgmap v577: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.6 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 497/129855 objects misplaced (0.383%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:30:48.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:48.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:48 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1396508028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:30:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:48 smithi106 ceph-mon[16603]: osdmap e229: 12 total, 12 up, 12 in 2024-08-30T23:30:48.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1120426251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:48.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:48 smithi106 ceph-mon[16603]: pgmap v577: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.6 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 497/129855 objects misplaced (0.383%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:30:48.621 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:48.621 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:48.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:48.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:48.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:48.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:48.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:48.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:48.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:49.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:49.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:49.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:49.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:49.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:30:49.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 48577 48561 79.8636 71.9375 0.0117627 0.0125111 2024-08-30T23:30:49.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:49.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:49.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:49.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:49.543 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:49.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:49.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.574 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:49.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:49.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:49.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:49.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:49.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:49.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:49.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:49.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:30:50.038 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:50.038 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:50.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:50.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:50.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:50.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:50.195 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:50.290 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 48844 48828 78.2437 16.6875 0.0027718 0.012776 2024-08-30T23:30:50.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:50.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:50 smithi106 ceph-mon[16603]: osdmap e230: 12 total, 12 up, 12 in 2024-08-30T23:30:50.316 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:50 smithi106 ceph-mon[16603]: pgmap v579: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.8 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 497/137241 objects misplaced (0.362%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:30:50.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:50.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:50.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:50.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:50 smithi136 ceph-mon[20207]: osdmap e230: 12 total, 12 up, 12 in 2024-08-30T23:30:50.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:50 smithi136 ceph-mon[20207]: pgmap v579: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.8 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 497/137241 objects misplaced (0.362%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:30:50.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:50 smithi173 ceph-mon[20554]: osdmap e230: 12 total, 12 up, 12 in 2024-08-30T23:30:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:50 smithi173 ceph-mon[20554]: pgmap v579: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 2.8 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s; 497/137241 objects misplaced (0.362%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:30:50.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:50.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.715 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:50.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:30:50.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:50.726 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:30:50.726 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:30:50.726 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-08-30T23:30:50.726 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:30:50.727 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:30:50.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-08-30T23:30:50.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:30:50.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:50.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:50.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:50.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:50.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:51.041 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:30:51.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:51.087 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:51.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:51.125 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:51.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:51.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:51.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:30:51.291855+0000 min lat: 0.00105037 max lat: 1.98739 avg lat: 0.012761 2024-08-30T23:30:51.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:30:51.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 50149 50133 78.3265 81.5625 0.00497936 0.012761 2024-08-30T23:30:51.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:51.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:51.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:51.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:51.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:51 smithi173 ceph-mon[20554]: osdmap e231: 12 total, 12 up, 12 in 2024-08-30T23:30:51.489 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/734070224' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:51 smithi136 ceph-mon[20207]: osdmap e231: 12 total, 12 up, 12 in 2024-08-30T23:30:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/734070224' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:51.562 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:30:51.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:51 smithi106 ceph-mon[16603]: osdmap e231: 12 total, 12 up, 12 in 2024-08-30T23:30:51.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/734070224' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:30:51.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:51.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:51.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:51.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:51.783 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:30:51.783 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:30:51.783 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:30:51.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:51.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:51.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:51.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:52.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:52.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:52.129 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:52.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:52.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:52.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 51668 51652 78.7315 94.9375 0.00201623 0.0126947 2024-08-30T23:30:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:52 smithi106 ceph-mon[16603]: pgmap v581: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 5.9 MiB/s, 95 objects/s recovering 2024-08-30T23:30:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/734070224' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:52 smithi106 ceph-mon[16603]: osdmap e232: 12 total, 12 up, 12 in 2024-08-30T23:30:52.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:52.385 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:52.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:52.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:52 smithi173 ceph-mon[20554]: pgmap v581: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 5.9 MiB/s, 95 objects/s recovering 2024-08-30T23:30:52.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/734070224' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:52.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:52 smithi173 ceph-mon[20554]: osdmap e232: 12 total, 12 up, 12 in 2024-08-30T23:30:52.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:52.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:52.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:52 smithi136 ceph-mon[20207]: pgmap v581: 17 pgs: 1 peering, 16 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 5.9 MiB/s, 95 objects/s recovering 2024-08-30T23:30:52.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/734070224' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:30:52.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:52 smithi136 ceph-mon[20207]: osdmap e232: 12 total, 12 up, 12 in 2024-08-30T23:30:52.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:52.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:52.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:52.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:52.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:52.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:52.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:53.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:53.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:53.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:53.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:53.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:53.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 53045 53029 78.9059 86.0625 0.0385889 0.0126655 2024-08-30T23:30:53.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:53.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:53.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:53.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:53.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:53.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:53.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:53.759 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:30:53.760 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:30:53.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:53.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:53.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:53.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:53.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:54.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:54.036 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:54.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:54.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:54.193 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:30:54.194 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:30:54.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:30:54.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:30:54.237 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:30:54.237 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:30:54.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:30:54.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:30:54.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:54.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:30:54.278 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:30:54.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:54.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 53854 53838 78.2467 50.5625 0.203983 0.0127312 2024-08-30T23:30:54.307 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:30:54.307 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:30:54.337 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:30:54.337 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:30:54.368 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:30:54.369 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:30:54.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:54.398 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:30:54.398 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:30:54.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.411 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:54.430 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:30:54.430 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:30:54.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:30:54.460 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:30:54.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:30:54.489 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:30:54.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:54.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:54 smithi106 ceph-mon[16603]: pgmap v583: 17 pgs: 1 peering, 16 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 983 op/s; 5.2 MiB/s, 82 objects/s recovering 2024-08-30T23:30:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3126218618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:54.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:54.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:54.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:54 smithi136 ceph-mon[20207]: pgmap v583: 17 pgs: 1 peering, 16 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 983 op/s; 5.2 MiB/s, 82 objects/s recovering 2024-08-30T23:30:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3126218618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:54.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:30:54.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:54 smithi173 ceph-mon[20554]: pgmap v583: 17 pgs: 1 peering, 16 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 983 op/s; 5.2 MiB/s, 82 objects/s recovering 2024-08-30T23:30:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3126218618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:30:54.897 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:54.922 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:54.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:55.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:55.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:55.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:30:55.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:55.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:55.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:55.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 55273 55257 78.4839 88.6875 0.00504894 0.012736 2024-08-30T23:30:55.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:55.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:55.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:55.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:55.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:55.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:55.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:55.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:55.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:55.894 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:55.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:55.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:55.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:56.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:56.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:56.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:56.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:56.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 56540 56524 78.4994 79.1875 0.0104357 0.0127312 2024-08-30T23:30:56.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:56.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:56.406 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:30:56.406 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:30:56.407 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:30:56.407 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:30:56.407 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:30:56.407 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:56.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:30:56.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:30:56.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:56.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:56.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:56.697 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: pgmap v584: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: osdmap e233: 12 total, 12 up, 12 in 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:30:56.760 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:30:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: pgmap v584: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:30:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: osdmap e233: 12 total, 12 up, 12 in 2024-08-30T23:30:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:30:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:30:56.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:30:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: pgmap v584: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:30:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:30:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: osdmap e233: 12 total, 12 up, 12 in 2024-08-30T23:30:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:30:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:30:56.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:30:56.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:56.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:56.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:57.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:57.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:57.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:57.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:57.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 57639 57623 78.2859 68.6875 0.00493481 0.0127685 2024-08-30T23:30:57.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:30:57.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:30:57.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:57.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:30:57.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:57.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:30:57.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-08-30T23:30:57.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:57.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:57.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:30:57.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:30:57.809 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:57.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:57.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:57.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:30:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:57 smithi106 ceph-mon[16603]: osdmap e234: 12 total, 12 up, 12 in 2024-08-30T23:30:57.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:57 smithi106 ceph-mon[16603]: pgmap v587: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-30T23:30:57.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:30:57.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:57.977 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:57.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:57 smithi136 ceph-mon[20207]: osdmap e234: 12 total, 12 up, 12 in 2024-08-30T23:30:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:57 smithi136 ceph-mon[20207]: pgmap v587: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-30T23:30:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:57 smithi173 ceph-mon[20554]: osdmap e234: 12 total, 12 up, 12 in 2024-08-30T23:30:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:57 smithi173 ceph-mon[20554]: pgmap v587: 17 pgs: 17 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-30T23:30:58.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:30:58.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:30:58.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:58.234 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:30:58.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 58967 58951 78.3861 83 0.00418593 0.0127518 2024-08-30T23:30:58.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:58.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:58.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:58.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:58.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:58.581 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:30:58.589 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.14157.0:52 1.0 e22/20 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'34, mlcod=22'29)", 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 586.60849494800004, 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33784646499999998, 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.590 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977573+0000", 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977574+0000", 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7100000000000002e-07 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977579+0000", 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1080000000000001e-06 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.314904+0000", 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33732482499999999 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.314915+0000", 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1576000000000001e-05 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315011+0000", 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5877000000000004e-05 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315031+0000", 2024-08-30T23:30:58.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9718e-05 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315389+0000", 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000358184 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315420+0000", 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0605999999999998e-05 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.594 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14225 2.5 e109/97 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head v 109'898, mlcod=109'894)", 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.71205789700002, 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.239755616, 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:30:58.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874011+0000", 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874108+0000", 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6961999999999998e-05 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112976+0000", 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23886803600000001 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112983+0000", 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.951e-06 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113055+0000", 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113079+0000", 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4754000000000001e-05 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113724+0000", 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064475700000000003 2024-08-30T23:30:58.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113766+0000", 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1664e-05 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16156 2.4 e109/97 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head v 109'956, mlcod=109'953)", 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 288.48797720099998, 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213651339, 2024-08-30T23:30:58.599 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098092+0000", 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4300000000000002e-07 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098114+0000", 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2234000000000001e-05 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310963+0000", 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21284856199999999 2024-08-30T23:30:58.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310969+0000", 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311028+0000", 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8857999999999998e-05 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311051+0000", 2024-08-30T23:30:58.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2538999999999999e-05 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311702+0000", 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065106500000000004 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311742+0000", 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0318000000000003e-05 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 241.95685544, 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207325907, 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629214+0000", 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.294e-06 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629241+0000", 2024-08-30T23:30:58.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6767999999999999e-05 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834158+0000", 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20491721900000001 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834164+0000", 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.268e-06 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834227+0000", 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2712000000000005e-05 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2120000000000002e-05 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836507+0000", 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022580769999999998 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836539+0000", 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1449000000000003e-05 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 210.08830257899999, 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.608 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:30:58.611 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109556 2.5 2:a6b86250:::benchmark_data_smithi173_47195_object12548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.731719594, 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220264298, 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.612 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109556 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854350+0000", 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8840000000000001e-06 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854356+0000", 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6590000000000004e-06 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854369+0000", 2024-08-30T23:30:58.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.289e-05 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854377+0000", 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4649999999999994e-06 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854454+0000", 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6791000000000004e-05 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854600+0000", 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145864 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854751+0000", 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015046499999999999 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.857557+0000", 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028060419999999999 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.873657+0000", 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016100032 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073675+0000", 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200018474 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073721+0000", 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6072999999999997e-05 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.074613+0000", 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089165900000000005 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109564 2.4 2:242bfffa:::benchmark_data_smithi173_47195_object12556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.72427557399999, 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220047624, 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.620 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109564 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861794+0000", 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5889999999999999e-06 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861800+0000", 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861809+0000", 2024-08-30T23:30:58.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999997e-06 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861815+0000", 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861910+0000", 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5278000000000005e-05 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862048+0000", 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137911 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862173+0000", 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012423299999999999 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878997+0000", 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016824741000000001 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879048+0000", 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0890999999999998e-05 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080272+0000", 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20122369600000001 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080313+0000", 2024-08-30T23:30:58.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0785999999999999e-05 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.081840+0000", 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015274679999999999 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109567 2.4 2:235a232e:::benchmark_data_smithi173_47195_object12559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:30:58.626 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.71922074899999, 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218097968, 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109567 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866849+0000", 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866855+0000", 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1199999999999999e-06 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866877+0000", 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1954e-05 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866885+0000", 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9070000000000007e-06 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869749+0000", 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028635940000000001 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869826+0000", 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7583999999999994e-05 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869907+0000", 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0408999999999995e-05 2024-08-30T23:30:58.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879006+0000", 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090992710000000008 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883826+0000", 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048203719999999999 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083592+0000", 2024-08-30T23:30:58.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.199765677 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083632+0000", 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0262000000000002e-05 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.084945+0000", 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013130590000000001 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109569 2.5 2:a8435ff8:::benchmark_data_smithi173_47195_object12561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.71657839599999, 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22083613699999999, 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.633 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109569 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869491+0000", 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4270000000000001e-06 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869498+0000", 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869510+0000", 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1630999999999999e-05 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869519+0000", 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869586+0000", 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6699999999999995e-05 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869741+0000", 2024-08-30T23:30:58.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155141 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869888+0000", 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014657099999999999 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879878+0000", 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099902989999999994 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879907+0000", 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8612999999999999e-05 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089168+0000", 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20926119900000001 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089267+0000", 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9097000000000003e-05 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090326+0000", 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010589670000000001 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109574 2.5 2:aaf91c06:::benchmark_data_smithi173_47195_object12566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.710012573, 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21517555899999999, 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.639 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109574 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876058+0000", 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.046e-06 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876065+0000", 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876084+0000", 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8702000000000002e-05 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876094+0000", 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5850000000000006e-06 2024-08-30T23:30:58.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876163+0000", 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9441000000000001e-05 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876332+0000", 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016932699999999999 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:58.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876473+0000", 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014066600000000001 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882795+0000", 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063221149999999997 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883230+0000", 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043490299999999999 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090374+0000", 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20714431699999999 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090425+0000", 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0865000000000002e-05 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091231+0000", 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00080588100000000005 2024-08-30T23:30:58.645 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109576 2.5 2:ae1552f5:::benchmark_data_smithi173_47195_object12568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.70756875800001, 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21498098400000001, 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.646 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109576 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878501+0000", 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.751e-06 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878508+0000", 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8340000000000001e-06 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878521+0000", 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669e-05 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878529+0000", 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.055e-06 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880019+0000", 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001490686 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880144+0000", 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012418800000000001 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880266+0000", 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000122552 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882939+0000", 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026733849999999999 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.917160+0000", 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034220360999999998 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092559+0000", 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.175398734 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092623+0000", 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4208999999999995e-05 2024-08-30T23:30:58.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.093480+0000", 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00085756000000000005 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109577 2.4 2:26051de8:::benchmark_data_smithi173_47195_object12569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.706598031, 2024-08-30T23:30:58.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20693977799999999, 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109577 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879471+0000", 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.077e-06 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879476+0000", 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8289999999999998e-06 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879488+0000", 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1883e-05 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879497+0000", 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6570000000000003e-06 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879535+0000", 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8574e-05 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879631+0000", 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5890999999999994e-05 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879719+0000", 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8331999999999994e-05 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882451+0000", 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027319240000000002 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.893310+0000", 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01085908 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.085013+0000", 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19170245499999999 2024-08-30T23:30:58.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.085071+0000", 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7884999999999999e-05 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.086410+0000", 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001339191 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144821 2.4 2:2415a84c:::benchmark_data_smithi173_47195_object47813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:30:58.658 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 147.02268442100001, 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23107453999999999, 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144821 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563385+0000", 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.59e-06 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563391+0000", 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563404+0000", 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2982e-05 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563411+0000", 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565303+0000", 2024-08-30T23:30:58.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001891846 2024-08-30T23:30:58.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565431+0000", 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012746400000000001 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565550+0000", 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972900000000001 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.573426+0000", 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078756799999999995 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588851+0000", 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015424679 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793322+0000", 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20447127600000001 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793387+0000", 2024-08-30T23:30:58.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4586999999999997e-05 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.794458+0000", 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010715060000000001 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144836 2.4 2:2d7ce86c:::benchmark_data_smithi173_47195_object47828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:30:58.666 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 146.99795105499999, 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23589892500000001, 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144836 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588118+0000", 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3829999999999999e-06 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588123+0000", 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588169+0000", 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199999999999998e-05 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588182+0000", 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2748999999999999e-05 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.592926+0000", 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047443779999999996 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595020+0000", 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002094243 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595429+0000", 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040842299999999999 2024-08-30T23:30:58.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603861+0000", 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0084324050000000005 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.604206+0000", 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034452400000000002 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.822951+0000", 2024-08-30T23:30:58.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21874496900000001 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.823010+0000", 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9601000000000001e-05 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824016+0000", 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010057499999999999 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.672 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155204 2.4 2:2613b3a5:::benchmark_data_smithi173_47195_object58196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 132.80967017899999, 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21707409, 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155204 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.673 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776400+0000", 2024-08-30T23:30:58.674 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776406+0000", 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1890000000000004e-06 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776421+0000", 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449999999999999e-05 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.675 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776430+0000", 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0420000000000008e-06 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776499+0000", 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776643+0000", 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000144721 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.676 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776820+0000", 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017706199999999999 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.783777+0000", 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006956322 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.786557+0000", 2024-08-30T23:30:58.677 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027807270000000002 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992894+0000", 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20633659600000001 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992950+0000", 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5664000000000001e-05 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.678 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.993472+0000", 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052240299999999995 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:193978 2.5 2:a2ee5e67:::benchmark_data_smithi173_47195_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 75.829921603000003, 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.283382829, 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.679 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193978 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.680 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756148+0000", 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756154+0000", 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.681 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756166+0000", 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041e-05 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756172+0000", 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2349999999999996e-06 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756333+0000", 2024-08-30T23:30:58.682 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000160767 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756484+0000", 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015133799999999999 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756627+0000", 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014263700000000001 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.683 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764342+0000", 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077147439999999999 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.769055+0000", 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047136399999999998 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039012+0000", 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26995667699999998 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.684 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039055+0000", 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2567000000000001e-05 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039529+0000", 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047478400000000001 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:193978 2.5 e200/182 ondisk, result = 0)", 2024-08-30T23:30:58.685 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 75.821966337000006, 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27543190299999998, 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:30:58.686 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764103+0000", 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764108+0000", 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9080000000000003e-06 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.687 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038928+0000", 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27481996400000003 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038937+0000", 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7860000000000008e-06 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038998+0000", 2024-08-30T23:30:58.688 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0915999999999997e-05 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039007+0000", 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4319999999999999e-06 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039534+0000", 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052686200000000001 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.689 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e200,182 2:a1c9afc1:::benchmark_data_smithi173_47195_object1378:head,146'6135)", 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 75.803258134999993, 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42837196799999999, 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.690 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782811+0000", 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782816+0000", 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6969999999999998e-06 2024-08-30T23:30:58.691 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210395+0000", 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42757847500000001 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210404+0000", 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0000000000000002e-06 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210493+0000", 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9438000000000007e-05 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.211182+0000", 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068873000000000003 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2684 3.6 3:677e81c8:::benchmark_data_smithi173_60780_object2683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.691858+0000", 2024-08-30T23:30:58.693 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 45.894209695999997, 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20452972899999999, 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2684 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691858+0000", 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691858+0000", 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691860+0000", 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1739999999999999e-06 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691911+0000", 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1649e-05 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691922+0000", 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1037e-05 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691928+0000", 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6760000000000004e-06 2024-08-30T23:30:58.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691969+0000", 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1341000000000003e-05 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.692058+0000", 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8888999999999995e-05 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.692142+0000", 2024-08-30T23:30:58.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3455000000000006e-05 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.692545+0000", 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000402984 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.695211+0000", 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026659549999999998 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896323+0000", 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.2011126 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896358+0000", 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5274999999999997e-05 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896388+0000", 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9694000000000001e-05 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2685 3.6 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 45.892534408000003, 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22443937, 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2685 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693535+0000", 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.009e-06 2024-08-30T23:30:58.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693573+0000", 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7843999999999999e-05 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693589+0000", 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6359e-05 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693603+0000", 2024-08-30T23:30:58.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.412e-05 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693657+0000", 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3757e-05 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693735+0000", 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8444999999999999e-05 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:30:58.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693820+0000", 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4290999999999996e-05 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.694189+0000", 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036947500000000002 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.874073+0000", 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.179884508 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:30:58.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917876+0000", 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043802633000000001 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917931+0000", 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5297999999999998e-05 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917973+0000", 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1631000000000002e-05 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.705 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.706 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:30:58.706 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:30:58.706 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:30:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:58.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:58.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:30:58.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:30:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/393352440' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:58.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:58 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:30:58.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:30:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:58.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:30:58.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:30:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/393352440' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:58 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:30:59.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/393352440' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:30:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:58 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:30:59.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:59.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:30:59.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:30:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:30:59.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 60230 60214 78.3974 78.9375 0.0029886 0.0127494 2024-08-30T23:30:59.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:59.338 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:59.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:30:59.466 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:30:59.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:30:59.591 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:30:59.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:59.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:59.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:30:59.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: osdmap e235: 12 total, 12 up, 12 in 2024-08-30T23:30:59.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:30:59.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:30:59.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:30:59.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2112166353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:30:59.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:30:59 smithi106 ceph-mon[16603]: pgmap v589: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-08-30T23:30:59.780 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T23:30:59.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:30:59.839 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:30:59.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:30:59.962 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:30:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:30:59.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: osdmap e235: 12 total, 12 up, 12 in 2024-08-30T23:31:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2112166353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:31:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:30:59 smithi136 ceph-mon[20207]: pgmap v589: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-08-30T23:31:00.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: osdmap e235: 12 total, 12 up, 12 in 2024-08-30T23:31:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2112166353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:31:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:30:59 smithi173 ceph-mon[20554]: pgmap v589: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-08-30T23:31:00.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:00.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:00.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:00.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 61092 61076 77.8969 53.875 0.0208654 0.0128316 2024-08-30T23:31:00.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:00.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:00.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:00.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:00.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:00.591 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:00.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:31:00.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:00.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:00.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:00.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:00.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:00.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:00.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:01.018 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2112166353' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:31:01.018 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:00 smithi106 ceph-mon[16603]: osdmap e236: 12 total, 12 up, 12 in 2024-08-30T23:31:01.018 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:00 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:01.018 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:00 smithi106 ceph-mon[16603]: osdmap e237: 12 total, 12 up, 12 in 2024-08-30T23:31:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2112166353' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:31:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:00 smithi136 ceph-mon[20207]: osdmap e236: 12 total, 12 up, 12 in 2024-08-30T23:31:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:00 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:00 smithi136 ceph-mon[20207]: osdmap e237: 12 total, 12 up, 12 in 2024-08-30T23:31:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2112166353' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:31:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:00 smithi173 ceph-mon[20554]: osdmap e236: 12 total, 12 up, 12 in 2024-08-30T23:31:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:00 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:00 smithi173 ceph-mon[20554]: osdmap e237: 12 total, 12 up, 12 in 2024-08-30T23:31:01.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:01.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:01.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:01.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:01.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 62424 62408 78.0038 83.25 0.00358105 0.0128141 2024-08-30T23:31:01.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:01.355 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:01.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-08-30T23:31:01.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.483 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:01.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:01.490 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:01.534 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:01.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:01.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:01.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:01.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:01.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:01.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:01.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:01.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:02.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:02.052 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:01 smithi106 ceph-mon[16603]: pgmap v592: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:31:02.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:31:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:01 smithi173 ceph-mon[20554]: pgmap v592: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:31:02.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:02.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:02.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:02.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:01 smithi136 ceph-mon[20207]: pgmap v592: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:31:02.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 63680 63664 78.0134 78.5 0.00504975 0.0128131 2024-08-30T23:31:02.351 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:02.351 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:02.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:02.376 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:02.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:02.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:02.405 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:02.405 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:02.433 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:02.434 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:02.463 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:02.490 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:02.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:02.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:02.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:02.522 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:02.522 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:02.555 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:02.559 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:02.559 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:02.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:02.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:02.653 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:02.653 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:02.679 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:02.679 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:02.704 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:02.704 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:02.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:02.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:02.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:02.758 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:02.758 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:02.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3586269656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:02.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:02.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:02.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:03.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:03.009 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:03.032 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:03.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3586269656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:03.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:03.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:03.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:03.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:03.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3586269656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:03.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 64805 64789 77.8652 70.3125 0.0954764 0.012831 2024-08-30T23:31:03.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:03.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:03.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:03.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:03.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:03.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:03.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:03.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:03.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:03.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:03.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: pgmap v593: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:04.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:31:04.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:04.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:04.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:04.170 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:04.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: pgmap v593: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-08-30T23:31:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:31:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:31:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:31:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:31:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:31:04.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 65936 65920 77.7296 70.6875 0.00249848 0.0128585 2024-08-30T23:31:04.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: pgmap v593: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-08-30T23:31:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:31:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:31:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:31:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:31:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:31:04.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:04.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:04.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:04.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:04.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:04.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:04.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:04.804 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:04.846 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T23:31:04.846 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:31:04.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:04.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:04.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:05.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:05.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3912746636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:31:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:04 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:31:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:04 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:04 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2487026248' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:04 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:05.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:05.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:05.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3912746636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:31:05.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:04 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:31:05.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:04 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:05.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:04 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:05.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2487026248' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:05.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:04 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:05.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:31:05.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3912746636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:31:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:04 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:31:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:04 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:04 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2487026248' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:04 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:05.291 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 66937 66921 77.4487 62.5625 0.0267374 0.0129036 2024-08-30T23:31:05.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:05.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:05.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:05.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:05.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:05.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:05.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-08-30T23:31:05.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:05.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:05.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:05.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:05.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:05.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:05.993 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:31:05.993 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:31:05.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:31:05.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:05.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: osdmap e238: 12 total, 12 up, 12 in 2024-08-30T23:31:05.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: pgmap v595: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:31:05.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:05 smithi106 ceph-mon[16603]: osdmap e239: 12 total, 12 up, 12 in 2024-08-30T23:31:06.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:06.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:06.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:06.190 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:31:06.190 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:31:06.190 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:31:06.190 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:06.190 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: osdmap e238: 12 total, 12 up, 12 in 2024-08-30T23:31:06.191 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: pgmap v595: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:31:06.191 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:05 smithi136 ceph-mon[20207]: osdmap e239: 12 total, 12 up, 12 in 2024-08-30T23:31:06.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:06.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 15 67931 67916 77.1711 62.1875 0.00629746 0.0129528 2024-08-30T23:31:06.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:31:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:31:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:31:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: osdmap e238: 12 total, 12 up, 12 in 2024-08-30T23:31:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: pgmap v595: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:31:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:05 smithi173 ceph-mon[20554]: osdmap e239: 12 total, 12 up, 12 in 2024-08-30T23:31:06.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:06.379 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:06.410 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:06.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:06.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:06.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:06.581 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:06.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:06.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:06.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:06.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:06.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:31:06.899 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:06 smithi106 ceph-mon[16603]: 3.0 scrub starts 2024-08-30T23:31:06.922 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:06 smithi136 ceph-mon[20207]: 3.0 scrub starts 2024-08-30T23:31:06.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:06.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:06.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:06.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:07.020 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:07.020 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:07.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:06 smithi173 ceph-mon[20554]: 3.0 scrub starts 2024-08-30T23:31:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:07.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:07.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:07.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:07.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 69080 69064 77.0742 71.75 0.00981753 0.0129692 2024-08-30T23:31:07.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:07.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:07.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:07.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:07.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:07.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:07.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:07.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:07.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:07.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:07.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:07.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:08.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:07 smithi106 ceph-mon[16603]: 3.0 scrub ok 2024-08-30T23:31:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:07 smithi106 ceph-mon[16603]: pgmap v597: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-08-30T23:31:08.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:08.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:08.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:31:08.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:08.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:08.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:07 smithi136 ceph-mon[20207]: 3.0 scrub ok 2024-08-30T23:31:08.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:07 smithi136 ceph-mon[20207]: pgmap v597: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-08-30T23:31:08.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 70362 70346 77.1276 80.125 0.00323527 0.012959 2024-08-30T23:31:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:07 smithi173 ceph-mon[20554]: 3.0 scrub ok 2024-08-30T23:31:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:07 smithi173 ceph-mon[20554]: pgmap v597: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-08-30T23:31:08.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:31:08.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:08.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:08.479 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:08.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:08.515 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:08.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:08.644 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:08.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:08.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:08.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:08.854 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T23:31:08.871 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:08 smithi106 ceph-mon[16603]: 3.c scrub starts 2024-08-30T23:31:08.871 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2335861968' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:31:08.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:08.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:08.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:09.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:09.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:09.061 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:09.061 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:09.086 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:09.086 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:09.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:09.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:09.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:08 smithi136 ceph-mon[20207]: 3.c scrub starts 2024-08-30T23:31:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2335861968' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:31:09.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 71497 71481 77.0208 70.9375 0.0163369 0.0129753 2024-08-30T23:31:09.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:09.310 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:09.310 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:08 smithi173 ceph-mon[20554]: 3.c scrub starts 2024-08-30T23:31:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2335861968' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:31:09.333 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:09.333 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:09.358 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:09.358 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:09.384 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:09.384 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:09.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:09.406 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:09.406 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:09.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:09.431 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:09.431 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:09.456 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:09.457 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:09.482 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:09.482 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:09.510 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:09.510 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:09.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:09.541 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:09.542 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:09.549 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:09.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:09.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:09.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:09.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-08-30T23:31:09.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:09.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:09.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:09.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:10.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:10.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:10.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2335861968' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:31:10.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:09 smithi106 ceph-mon[16603]: osdmap e240: 12 total, 12 up, 12 in 2024-08-30T23:31:10.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2155112856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:10.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:09 smithi106 ceph-mon[16603]: 3.c scrub ok 2024-08-30T23:31:10.088 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:09 smithi106 ceph-mon[16603]: pgmap v599: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:31:10.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:10.174 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:10.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2335861968' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:31:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:09 smithi136 ceph-mon[20207]: osdmap e240: 12 total, 12 up, 12 in 2024-08-30T23:31:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2155112856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:09 smithi136 ceph-mon[20207]: 3.c scrub ok 2024-08-30T23:31:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:09 smithi136 ceph-mon[20207]: pgmap v599: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:31:10.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 72487 72471 76.764 61.875 0.00326421 0.0130206 2024-08-30T23:31:10.301 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2335861968' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:31:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:09 smithi173 ceph-mon[20554]: osdmap e240: 12 total, 12 up, 12 in 2024-08-30T23:31:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2155112856' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:09 smithi173 ceph-mon[20554]: 3.c scrub ok 2024-08-30T23:31:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:09 smithi173 ceph-mon[20554]: pgmap v599: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:31:10.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:10.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:10.516 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:10.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:10.549 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:10.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:10.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:10.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:10.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:10.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:10.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:10.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:10.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:11.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:11.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:11.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:10 smithi106 ceph-mon[16603]: osdmap e241: 12 total, 12 up, 12 in 2024-08-30T23:31:11.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:11.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:11.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:11.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:11.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:11.178 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:11.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:10 smithi136 ceph-mon[20207]: osdmap e241: 12 total, 12 up, 12 in 2024-08-30T23:31:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:11.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:31:11.293422+0000 min lat: 0.00105037 max lat: 1.98739 avg lat: 0.0130269 2024-08-30T23:31:11.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:31:11.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 73691 73675 76.7387 75.25 0.0060165 0.0130269 2024-08-30T23:31:11.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:10 smithi173 ceph-mon[20554]: osdmap e241: 12 total, 12 up, 12 in 2024-08-30T23:31:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:11.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:11.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.452 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:11.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:11.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:11.581 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:11.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:11.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:11.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:11.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:11.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:11.874 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:31:11.889 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:11 smithi106 ceph-mon[16603]: osdmap e242: 12 total, 12 up, 12 in 2024-08-30T23:31:11.889 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:11 smithi106 ceph-mon[16603]: pgmap v602: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:31:11.889 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2553087351' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:31:11.895 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 599.92528508999999, 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34705745500000001, 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.896 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 52 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968589+0000", 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968590+0000", 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6499999999999999e-07 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968594+0000", 2024-08-30T23:31:11.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0400000000000003e-06 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968602+0000", 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9850000000000006e-06 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.968607+0000", 2024-08-30T23:31:11.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2050000000000001e-06 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977321+0000", 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0087134099999999996 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977400+0000", 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8967000000000005e-05 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977514+0000", 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114164 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.977774+0000", 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026058500000000003 2024-08-30T23:31:11.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:11.978024+0000", 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00024982299999999998 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315575+0000", 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33755099 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315615+0000", 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9705e-05 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:12.315647+0000", 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2016e-05 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.14157.0:80 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 598.18703994500004, 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94419764299999998, 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.14157", 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.106:0/3573563326", 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 80 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706835+0000", 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7400000000000003e-07 2024-08-30T23:31:11.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706923+0000", 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8308999999999998e-05 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706932+0000", 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7409999999999998e-06 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.706938+0000", 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.263e-06 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708514+0000", 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015753620000000001 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708601+0000", 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6849999999999997e-05 2024-08-30T23:31:11.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.708731+0000", 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013043399999999999 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.709427+0000", 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069585600000000001 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:13.709468+0000", 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1125999999999999e-05 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650912+0000", 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94144422800000005 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.650975+0000", 2024-08-30T23:31:11.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2707999999999997e-05 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:21:14.651032+0000", 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7191999999999997e-05 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14229 2.6 2:681e88d6:::benchmark_data_smithi173_47195_object14228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 303.01767703600001, 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21536640000000001, 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.911 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14229 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876197+0000", 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876198+0000", 2024-08-30T23:31:11.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6899999999999996e-07 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876298+0000", 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9622000000000003e-05 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876310+0000", 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1817999999999999e-05 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876317+0000", 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8009999999999998e-06 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876377+0000", 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0489999999999999e-05 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876507+0000", 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012945900000000001 2024-08-30T23:31:11.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.876637+0000", 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013032300000000001 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878792+0000", 2024-08-30T23:31:11.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002154689 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883495+0000", 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047034950000000002 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091459+0000", 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207963857 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091526+0000", 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6610000000000001e-05 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.091564+0000", 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8266999999999998e-05 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14233 2.6 2:682ac747:::benchmark_data_smithi173_47195_object14232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 303.01614263699997, 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.215809479, 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14233 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877732+0000", 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877733+0000", 2024-08-30T23:31:11.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.215e-06 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877761+0000", 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7671999999999999e-05 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877776+0000", 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5352000000000001e-05 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.877783+0000", 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2660000000000004e-06 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878221+0000", 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043807200000000002 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878304+0000", 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2769999999999995e-05 2024-08-30T23:31:11.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.878392+0000", 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7997999999999993e-05 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.879128+0000", 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00073602100000000001 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883772+0000", 2024-08-30T23:31:11.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046439580000000001 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093486+0000", 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20971398699999999 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093518+0000", 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2047e-05 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093541+0000", 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3121000000000001e-05 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14240 2.6 2:6ab00b19:::benchmark_data_smithi173_47195_object14239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 303.01210069699999, 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21184964100000001, 2024-08-30T23:31:11.925 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14240 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:31:11.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881774+0000", 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881775+0000", 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.536e-06 2024-08-30T23:31:11.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881838+0000", 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2528999999999997e-05 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881849+0000", 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1606999999999999e-05 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881857+0000", 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5349999999999999e-06 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.881915+0000", 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8459999999999999e-05 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882042+0000", 2024-08-30T23:31:11.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012617300000000001 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882187+0000", 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014551399999999999 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883462+0000", 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012752149999999999 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.898625+0000", 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015162491 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093570+0000", 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.194944849 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093601+0000", 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0817e-05 2024-08-30T23:31:11.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093623+0000", 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2915e-05 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:14242 2.6 2:6fdab316:::benchmark_data_smithi173_47195_object14241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:31:11.932 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 303.01173656600002, 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21156007900000001, 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 14242 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882138+0000", 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882139+0000", 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2900000000000002e-07 2024-08-30T23:31:11.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882196+0000", 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7549000000000001e-05 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882204+0000", 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4189999999999996e-06 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882208+0000", 2024-08-30T23:31:11.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4390000000000002e-06 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882770+0000", 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056187700000000001 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.882879+0000", 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108856 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883022+0000", 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014315899999999999 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883584+0000", 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056226400000000004 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.938528+0000", 2024-08-30T23:31:11.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054943433 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093653+0000", 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.155125759 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093680+0000", 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6239000000000001e-05 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.093698+0000", 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8255999999999999e-05 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16156 2.4 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 301.79630217300002, 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21552370500000001, 2024-08-30T23:31:11.939 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16156 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097572+0000", 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097573+0000", 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3699999999999999e-07 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097632+0000", 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8870999999999999e-05 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097642+0000", 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0219e-05 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097649+0000", 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1269999999999997e-06 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097805+0000", 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015530900000000001 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.097941+0000", 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000135847 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:31:11.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098034+0000", 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3246999999999997e-05 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098764+0000", 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000730133 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099110+0000", 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034588700000000002 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313035+0000", 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21392467000000001 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313062+0000", 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7013999999999999e-05 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.313096+0000", 2024-08-30T23:31:11.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4443999999999998e-05 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16161 2.6 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 301.79452116800002, 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212960288, 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.946 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16161 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099353+0000", 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099354+0000", 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6199999999999997e-07 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099425+0000", 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0605999999999995e-05 2024-08-30T23:31:11.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099432+0000", 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6850000000000001e-06 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099438+0000", 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7069999999999997e-06 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099491+0000", 2024-08-30T23:31:11.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3038000000000003e-05 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099620+0000", 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012866900000000001 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099790+0000", 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170266 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100492+0000", 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070156400000000001 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.100884+0000", 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039193399999999998 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312191+0000", 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211307562 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312253+0000", 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2150000000000006e-05 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312314+0000", 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0344999999999999e-05 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:16911 2.4 2:26b3b856:::benchmark_data_smithi173_47195_object16910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 301.22148996499999, 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22617736999999999, 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 16911 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.953 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672385+0000", 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672455+0000", 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9855000000000001e-05 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672467+0000", 2024-08-30T23:31:11.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1528e-05 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672473+0000", 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5400000000000001e-06 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672516+0000", 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2645999999999998e-05 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672585+0000", 2024-08-30T23:31:11.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:11.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8751999999999995e-05 2024-08-30T23:31:11.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.672675+0000", 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0543000000000004e-05 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.673389+0000", 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071381299999999999 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.673738+0000", 2024-08-30T23:31:11.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034890399999999998 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898465+0000", 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22472745399999999 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898518+0000", 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2428e-05 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898562+0000", 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3989000000000002e-05 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16921 2.2 e111/97 2:4bb89a5b:::benchmark_data_smithi173_47195_object16920:head v 111'1004, mlcod=111'999)", 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 301.21509501200001, 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21942291, 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.961 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678779+0000", 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678780+0000", 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5699999999999998e-07 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.678834+0000", 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3992000000000001e-05 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.896861+0000", 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218027111 2024-08-30T23:31:11.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.896896+0000", 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5326e-05 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.897044+0000", 2024-08-30T23:31:11.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001473 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.897089+0000", 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5806000000000002e-05 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898158+0000", 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010683559999999999 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.898202+0000", 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4561999999999998e-05 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40525 2.9 2:93a0bad4:::benchmark_data_smithi173_47195_object40524:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 283.985923953, 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21130669299999999, 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.966 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40525 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907951+0000", 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907952+0000", 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8700000000000001e-06 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.907991+0000", 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8658999999999997e-05 2024-08-30T23:31:11.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908011+0000", 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9877999999999998e-05 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908020+0000", 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6500000000000002e-06 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908076+0000", 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6359e-05 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908193+0000", 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011733199999999999 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.908326+0000", 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013302999999999999 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.909927+0000", 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016005909999999999 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.910234+0000", 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030682700000000002 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119175+0000", 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20894082 2024-08-30T23:31:11.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119206+0000", 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1652000000000001e-05 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.119257+0000", 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1025e-05 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40537 2.9 2:97981a73:::benchmark_data_smithi173_47195_object40536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 283.97962502299998, 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207993344, 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.973 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40537 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914249+0000", 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914251+0000", 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.339e-06 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914366+0000", 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011518699999999999 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914379+0000", 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3424999999999999e-05 2024-08-30T23:31:11.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914388+0000", 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6070000000000008e-06 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914447+0000", 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8585999999999998e-05 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914600+0000", 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015365800000000001 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.914763+0000", 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016261300000000001 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.915944+0000", 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011807250000000001 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.916216+0000", 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000272692 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122145+0000", 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205928427 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122190+0000", 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5785000000000003e-05 2024-08-30T23:31:11.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.122243+0000", 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2299999999999997e-05 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:40552 2.6 2:67bfbe41:::benchmark_data_smithi173_47195_object40551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 283.96673373700003, 2024-08-30T23:31:11.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19393617399999999, 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 40552 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927141+0000", 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927142+0000", 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8899999999999998e-07 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927204+0000", 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2008000000000007e-05 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927213+0000", 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8409999999999995e-06 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927219+0000", 2024-08-30T23:31:11.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8690000000000001e-06 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927264+0000", 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4690000000000001e-05 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927338+0000", 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3831000000000005e-05 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:31:11.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.927468+0000", 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012972299999999999 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.928902+0000", 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014344970000000001 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:27.929243+0000", 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034044300000000001 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121004+0000", 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19176105600000001 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121035+0000", 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1269999999999997e-05 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:28.121077+0000", 2024-08-30T23:31:11.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2057000000000001e-05 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:62934 2.2 e127/97 2:479e8ab6:::benchmark_data_smithi173_47195_object62933:head v 127'3887, mlcod=127'3884)", 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 265.94370413500002, 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20906233900000001, 2024-08-30T23:31:11.986 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950170+0000", 2024-08-30T23:31:11.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950171+0000", 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2500000000000005e-07 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:45.950233+0000", 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2382e-05 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157402+0000", 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20716816099999999 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157407+0000", 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7479999999999999e-06 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157462+0000", 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4840999999999999e-05 2024-08-30T23:31:11.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.157485+0000", 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3244999999999999e-05 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.159190+0000", 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017050120000000001 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:46.159233+0000", 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2225000000000003e-05 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67558 2.9 2:98b6d163:::benchmark_data_smithi173_47195_object67557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 261.81770621499999, 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214383824, 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.991 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67558 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076168+0000", 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076169+0000", 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2300000000000001e-07 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076247+0000", 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8326999999999997e-05 2024-08-30T23:31:11.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076255+0000", 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2660000000000001e-06 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.076261+0000", 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.784e-06 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:11.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077492+0000", 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001231027 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077607+0000", 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011488 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.077733+0000", 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012587500000000001 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.079761+0000", 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020281829999999998 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.080335+0000", 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057335700000000003 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290485+0000", 2024-08-30T23:31:11.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21015015400000001 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290523+0000", 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8250000000000001e-05 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.290552+0000", 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9098000000000002e-05 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:11.997 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67582 2.9 2:97773903:::benchmark_data_smithi173_47195_object67581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 261.79710884000002, 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19555588700000001, 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:11.998 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67582 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096766+0000", 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:11.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096767+0000", 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096823+0000", 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6773e-05 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096835+0000", 2024-08-30T23:31:12.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1556000000000001e-05 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096842+0000", 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0770000000000002e-06 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096892+0000", 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0256000000000001e-05 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:12.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.096974+0000", 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1879999999999995e-05 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.097064+0000", 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9821999999999995e-05 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.098777+0000", 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017132149999999999 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.099120+0000", 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034311699999999999 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292262+0000", 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19314146600000001 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292296+0000", 2024-08-30T23:31:12.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3748000000000003e-05 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:50.292322+0000", 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5986000000000002e-05 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:188148 2.c e197/97 2:3df8a074:::benchmark_data_smithi173_47195_object91140:head v 197'11667, mlcod=197'11662)", 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:34.664737+0000", 2024-08-30T23:31:12.004 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.229137097999995, 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220687035, 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.664737+0000", 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:12.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.664737+0000", 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.664739+0000", 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3540000000000001e-06 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.664746+0000", 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2579999999999998e-06 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.879756+0000", 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21501003699999999 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.879764+0000", 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2409999999999995e-06 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.880458+0000", 2024-08-30T23:31:12.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00069413600000000004 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.880478+0000", 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9837e-05 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.885392+0000", 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049134310000000002 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:34.885424+0000", 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2740999999999997e-05 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:2685 3.6 e213/206 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head v 213'167, mlcod=213'163)", 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693898+0000", 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 59.199976178999997, 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202306718, 2024-08-30T23:31:12.009 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693898+0000", 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693898+0000", 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693899+0000", 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4700000000000001e-07 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693920+0000", 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1138e-05 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.895413+0000", 2024-08-30T23:31:12.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20149298299999999 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.895425+0000", 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2019e-05 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.895554+0000", 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012840700000000001 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:12.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.895612+0000", 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7985999999999997e-05 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896172+0000", 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00055993799999999995 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896205+0000", 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3500000000000001e-05 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.013 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:21975 3.4 e214/206 3:2a1d7d2d:::benchmark_data_smithi173_60780_object21974:head v 214'1354, mlcod=214'1349)", 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:25.268848+0000", 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 46.625026198999997, 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20976334899999999, 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:12.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.268848+0000", 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.268848+0000", 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.268849+0000", 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5799999999999999e-07 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.268928+0000", 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9647999999999994e-05 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.476696+0000", 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20776792699999999 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.476714+0000", 2024-08-30T23:31:12.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8006e-05 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.476796+0000", 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1192999999999993e-05 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.476819+0000", 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3723999999999999e-05 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.478582+0000", 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017621939999999999 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.478612+0000", 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0099000000000001e-05 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:73836 3.6 e242/206 3:6f66e2b4:::benchmark_data_smithi173_60780_object73835:head v 242'4681, mlcod=242'4677)", 2024-08-30T23:31:12.018 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:11.414470+0000", 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.47940441700000003, 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055270490999999998, 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.414470+0000", 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.414470+0000", 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.414471+0000", 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5199999999999996e-07 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.414493+0000", 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2595e-05 2024-08-30T23:31:12.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.467401+0000", 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.052907132000000003 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.467407+0000", 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7970000000000003e-06 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:12.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.467460+0000", 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2410999999999997e-05 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.467480+0000", 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0007999999999999e-05 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.469711+0000", 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022307960000000002 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:12.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:11.469741+0000", 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0000000000000001e-05 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:12.023 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:31:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:12.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:12.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:12.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:12.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:11 smithi136 ceph-mon[20207]: osdmap e242: 12 total, 12 up, 12 in 2024-08-30T23:31:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:11 smithi136 ceph-mon[20207]: pgmap v602: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:31:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2553087351' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:31:12.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 74643 74627 76.456 59.5 0.00665186 0.0130749 2024-08-30T23:31:12.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:12.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:11 smithi173 ceph-mon[20554]: osdmap e242: 12 total, 12 up, 12 in 2024-08-30T23:31:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:11 smithi173 ceph-mon[20554]: pgmap v602: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-30T23:31:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2553087351' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:31:12.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:12.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:12.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:31:12.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:12.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:12.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:12.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:12.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:12.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:12.874 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T23:31:12.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:12.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:12.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:13.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:13.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:13.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:13.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:12 smithi136 ceph-mon[20207]: 3.8 scrub starts 2024-08-30T23:31:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:12 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2553087351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:31:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:12 smithi136 ceph-mon[20207]: osdmap e243: 12 total, 12 up, 12 in 2024-08-30T23:31:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/561599887' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:31:13.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 75560 75544 76.1472 57.3125 0.0282289 0.0131239 2024-08-30T23:31:13.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:13.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:12 smithi173 ceph-mon[20554]: 3.8 scrub starts 2024-08-30T23:31:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:12 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2553087351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:31:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:12 smithi173 ceph-mon[20554]: osdmap e243: 12 total, 12 up, 12 in 2024-08-30T23:31:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/561599887' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:31:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:12 smithi106 ceph-mon[16603]: 3.8 scrub starts 2024-08-30T23:31:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:12 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2553087351' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:31:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:12 smithi106 ceph-mon[16603]: osdmap e243: 12 total, 12 up, 12 in 2024-08-30T23:31:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/561599887' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:31:13.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:13.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:13.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:13.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:13.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:13.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:13.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:13.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:13.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:13.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:13.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:14.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:31:14.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:14.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:14.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:14.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:14.220 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/561599887' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:31:14.220 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:13 smithi106 ceph-mon[16603]: osdmap e244: 12 total, 12 up, 12 in 2024-08-30T23:31:14.220 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:13 smithi106 ceph-mon[16603]: pgmap v605: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 864 op/s 2024-08-30T23:31:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/561599887' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:31:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:13 smithi136 ceph-mon[20207]: osdmap e244: 12 total, 12 up, 12 in 2024-08-30T23:31:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:13 smithi136 ceph-mon[20207]: pgmap v605: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 864 op/s 2024-08-30T23:31:14.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 76544 76528 75.9146 61.5 0.00380354 0.0131679 2024-08-30T23:31:14.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/561599887' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:31:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:13 smithi173 ceph-mon[20554]: osdmap e244: 12 total, 12 up, 12 in 2024-08-30T23:31:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:13 smithi173 ceph-mon[20554]: pgmap v605: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 864 op/s 2024-08-30T23:31:14.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:14.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:14.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:14.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:14.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:14.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:14.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:14.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:14.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:14.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:14.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:14.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:15.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:15.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:15.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:15.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:15.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 77763 77747 75.9188 76.1875 0.00343144 0.0131663 2024-08-30T23:31:15.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:15.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:15.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:15.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:15.615 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:15.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:15.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:15.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:15.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:15.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-08-30T23:31:15.856 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:15.856 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:15.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:15.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:15.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:15.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:16.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:16.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:16.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:16.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:16.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:16.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 78803 78787 75.7507 65 0.00810458 0.0131912 2024-08-30T23:31:16.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:16.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:16.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:16.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:16.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:16.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:16 smithi106 ceph-mon[16603]: pgmap v606: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s 2024-08-30T23:31:16.627 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:16 smithi106 ceph-mon[16603]: osdmap e245: 12 total, 12 up, 12 in 2024-08-30T23:31:16.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:16.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:16.744 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:16.744 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:16.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:16 smithi136 ceph-mon[20207]: pgmap v606: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s 2024-08-30T23:31:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:16 smithi136 ceph-mon[20207]: osdmap e245: 12 total, 12 up, 12 in 2024-08-30T23:31:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:16 smithi173 ceph-mon[20554]: pgmap v606: 17 pgs: 17 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s 2024-08-30T23:31:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:16 smithi173 ceph-mon[20554]: osdmap e245: 12 total, 12 up, 12 in 2024-08-30T23:31:16.835 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:16.835 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:16.835 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:16.836 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:16.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:16.855 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:16.855 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:16.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:16.879 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:16.880 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:16.883 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:16.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:16.901 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:16.901 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:16.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:17.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:17.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:17.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:17.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:31:17.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:17.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:17.292 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 79602 79586 75.3595 49.9375 0.00709997 0.0132653 2024-08-30T23:31:17.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:17.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:17.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:17.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:17.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:31:17.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:17.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:17.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:17.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3500298222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3500298222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3500298222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:17.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:17.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:17.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:18.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:18.035 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:18.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:18.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:18.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:18.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:18.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 80556 80540 75.1246 59.625 0.0101662 0.0133052 2024-08-30T23:31:18.314 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:18.318 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:18.364 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:18.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:18.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:18.541 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:18 smithi106 ceph-mon[16603]: pgmap v608: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:31:18.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:18.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:18.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:18.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:18 smithi136 ceph-mon[20207]: pgmap v608: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:31:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:18 smithi173 ceph-mon[20554]: pgmap v608: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:31:18.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:18.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:18.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:19.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:19.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:19.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:19.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:19.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:19.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 81560 81544 74.9426 62.75 0.020333 0.0133324 2024-08-30T23:31:19.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:19.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:19.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:19.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:19.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:19.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:19.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:19.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:19.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:19.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:19.945 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:19.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:20.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:20.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:20.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:20.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:20.274 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:31:20.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 82814 82798 74.9922 78.375 0.0156283 0.0133299 2024-08-30T23:31:20.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:20.344 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:20.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:20.470 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:20.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:20.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:20.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:20 smithi106 ceph-mon[16603]: pgmap v609: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:31:20.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:20.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:20.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:20 smithi106 ceph-mon[16603]: osdmap e246: 12 total, 12 up, 12 in 2024-08-30T23:31:20.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:20.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:20.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:20.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:20 smithi136 ceph-mon[20207]: pgmap v609: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:31:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:20 smithi136 ceph-mon[20207]: osdmap e246: 12 total, 12 up, 12 in 2024-08-30T23:31:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:20 smithi173 ceph-mon[20554]: pgmap v609: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:31:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:20 smithi173 ceph-mon[20554]: osdmap e246: 12 total, 12 up, 12 in 2024-08-30T23:31:20.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:20.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:20.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:20.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:20.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:21.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:21.209 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:21.240 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:21.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 83784 83768 74.7869 60.625 0.00352433 0.0133656 2024-08-30T23:31:21.321 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:21.322 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:21.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:21.368 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:21.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:21.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:21.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:21.494 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T23:31:21.495 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:31:21.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:21.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:21.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:21.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:21.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:21 smithi136 ceph-mon[20207]: osdmap e247: 12 total, 12 up, 12 in 2024-08-30T23:31:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1491113269' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2274641140' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:31:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:21 smithi136 ceph-mon[20207]: pgmap v612: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:31:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:21 smithi173 ceph-mon[20554]: osdmap e247: 12 total, 12 up, 12 in 2024-08-30T23:31:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1491113269' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2274641140' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:31:21.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:21 smithi173 ceph-mon[20554]: pgmap v612: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:31:21.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:21 smithi106 ceph-mon[16603]: osdmap e247: 12 total, 12 up, 12 in 2024-08-30T23:31:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1491113269' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2274641140' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:31:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:21 smithi106 ceph-mon[16603]: pgmap v612: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:31:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:21.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:21.980 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:22.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:22.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:22.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:22.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:22.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:22.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 84965 84949 74.7731 73.8125 0.0198077 0.0133684 2024-08-30T23:31:22.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:22.398 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:22.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:22.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:22 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1491113269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2274641140' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:31:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:22 smithi106 ceph-mon[16603]: osdmap e248: 12 total, 12 up, 12 in 2024-08-30T23:31:22.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:22.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:22.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:22.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:22 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1491113269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2274641140' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:31:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:22 smithi136 ceph-mon[20207]: osdmap e248: 12 total, 12 up, 12 in 2024-08-30T23:31:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:22 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1491113269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2274641140' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:31:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:22 smithi173 ceph-mon[20554]: osdmap e248: 12 total, 12 up, 12 in 2024-08-30T23:31:22.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:22.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:31:22.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:22.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:23.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:23.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:23.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:23.156 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:23.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:23.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:23.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 85902 85886 74.5479 58.5625 0.00400018 0.0134095 2024-08-30T23:31:23.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:23.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:23.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:23.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:23.585 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-08-30T23:31:23.608 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:23.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:23.648 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:23.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:23.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:23.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:23.775 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:23 smithi106 ceph-mon[16603]: pgmap v614: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-08-30T23:31:23.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:23.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:23 smithi136 ceph-mon[20207]: pgmap v614: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-08-30T23:31:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:23 smithi173 ceph-mon[20554]: pgmap v614: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-08-30T23:31:23.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:23.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:23.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:24.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:24.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:24.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:24.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:24.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:24.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:24.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:24.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 86690 86674 74.2013 49.25 0.00491229 0.0134704 2024-08-30T23:31:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:24.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:31:24.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:24.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:24.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:24.546 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:31:24.546 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:31:24.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:31:24.546 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:31:24.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.546 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14244 2.3 e109/97 2:c35428fa:::benchmark_data_smithi173_47195_object14243:head v 109'957, mlcod=109'954)", 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 315.65593824699999, 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213971894, 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883782+0000", 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999999e-07 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883844+0000", 2024-08-30T23:31:24.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2580000000000001e-05 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096781+0000", 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212936878 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096785+0000", 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0389999999999998e-06 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096831+0000", 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6261999999999999e-05 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096844+0000", 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2883e-05 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097712+0000", 2024-08-30T23:31:24.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086761599999999998 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097753+0000", 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0729999999999998e-05 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:56060 2.e 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 284.76024223299999, 2024-08-30T23:31:24.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21409088000000001, 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 56060 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779478+0000", 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4869999999999999e-06 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779513+0000", 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4384999999999997e-05 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779529+0000", 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6447999999999999e-05 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779537+0000", 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999993e-06 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779580+0000", 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3254999999999998e-05 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779658+0000", 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7711000000000002e-05 2024-08-30T23:31:24.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779738+0000", 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9775000000000001e-05 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781534+0000", 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017964820000000001 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992319+0000", 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21078449599999999 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993464+0000", 2024-08-30T23:31:24.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001145294 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993511+0000", 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6344000000000002e-05 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993568+0000", 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6993000000000001e-05 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:56060 2.e e124/97 ondisk, result = 0)", 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 284.75790226399999, 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051657200000001, 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:31:24.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9999999999999997e-07 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781821+0000", 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9480000000000001e-06 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990657+0000", 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20883629200000001 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990664+0000", 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.793e-06 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992305+0000", 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016408480000000001 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992313+0000", 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1100000000000003e-06 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992333+0000", 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9981000000000001e-05 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67116 2.e 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 274.92778844700001, 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22641219700000001, 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.563 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67116 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611931+0000", 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612007+0000", 2024-08-30T23:31:24.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5883999999999994e-05 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612021+0000", 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4007e-05 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612028+0000", 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4099999999999996e-06 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614110+0000", 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020827520000000002 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614270+0000", 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000159235 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614420+0000", 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015048199999999999 2024-08-30T23:31:24.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617176+0000", 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027561619999999999 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617722+0000", 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054594799999999998 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838237+0000", 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22051536999999999 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838285+0000", 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7808e-05 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838343+0000", 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7340000000000003e-05 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.569 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75429 2.1 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 267.58128043099998, 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24538217700000001, 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.570 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75429 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958440+0000", 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9410000000000001e-06 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958471+0000", 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0919e-05 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958494+0000", 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3078000000000002e-05 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958504+0000", 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1190000000000003e-06 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958565+0000", 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1364000000000004e-05 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958669+0000", 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010414199999999999 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958772+0000", 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102979 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.960745+0000", 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019727600000000001 2024-08-30T23:31:24.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.066102+0000", 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.105356845 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203744+0000", 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13764249000000001 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203787+0000", 2024-08-30T23:31:24.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.286e-05 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203821+0000", 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3680000000000003e-05 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head, version: 114'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head@114'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:31:24.576 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 219.647561529, 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19944226600000001, 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892158+0000", 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892186+0000", 2024-08-30T23:31:24.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7710000000000001e-05 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089119+0000", 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19693307299999999 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089137+0000", 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7533e-05 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089255+0000", 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118318 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089275+0000", 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9828000000000001e-05 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091600+0000", 2024-08-30T23:31:24.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023249030000000001 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:126363 2.7 e160/138 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head v 160'7890, mlcod=119'2657)", 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 197.810096222, 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.217439837, 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.581 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:31:24.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729630+0000", 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.674e-06 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940470+0000", 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21084006199999999 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940478+0000", 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4390000000000003e-06 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940541+0000", 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3115999999999997e-05 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940571+0000", 2024-08-30T23:31:24.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9981e-05 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947012+0000", 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064411269999999996 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947062+0000", 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0708999999999998e-05 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.585 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144840 2.1 2:8bdc20ef:::benchmark_data_smithi173_47195_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 172.94624056000001, 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21736361900000001, 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.586 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144840 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593481+0000", 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3460000000000001e-06 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593490+0000", 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593512+0000", 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2410000000000001e-05 2024-08-30T23:31:24.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593522+0000", 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4129999999999995e-06 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593705+0000", 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018286800000000001 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593993+0000", 2024-08-30T23:31:24.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028869300000000002 2024-08-30T23:31:24.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.590 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.590 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.594314+0000", 2024-08-30T23:31:24.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032096700000000002 2024-08-30T23:31:24.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.599306+0000", 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049920049999999999 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603619+0000", 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043124419999999997 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810726+0000", 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20710743000000001 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810784+0000", 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7893999999999998e-05 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810842+0000", 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7791000000000003e-05 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144845 2.1 2:80fd4430:::benchmark_data_smithi173_47195_object47837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:31:24.594 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 172.936395578, 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20768498299999999, 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144845 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603326+0000", 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.373e-06 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603335+0000", 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5449999999999992e-06 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603359+0000", 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3377e-05 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603370+0000", 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1759e-05 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603649+0000", 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027892500000000002 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603755+0000", 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010549500000000001 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603833+0000", 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8150000000000002e-05 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.609421+0000", 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0055884799999999998 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.640300+0000", 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030878974999999999 2024-08-30T23:31:24.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810910+0000", 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.170609233 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810964+0000", 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4549999999999998e-05 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.811008+0000", 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4121e-05 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155294 2.e 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 158.415836668, 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26817000400000002, 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155294 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123884+0000", 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3960000000000001e-06 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123889+0000", 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4820000000000002e-06 2024-08-30T23:31:24.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123902+0000", 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3202e-05 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123910+0000", 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8869999999999992e-06 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123962+0000", 2024-08-30T23:31:24.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1959000000000002e-05 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124045+0000", 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2850999999999997e-05 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124115+0000", 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9513999999999999e-05 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.127474+0000", 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033596149999999998 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.128135+0000", 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00066107399999999999 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.391962+0000", 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26382699500000001 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392009+0000", 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6669999999999999e-05 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392052+0000", 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3359000000000003e-05 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 182/143 [PushOp(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head, version: 132'4708, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head@132'4708, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 157.10015790400001, 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25797871999999999, 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:31:24.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439562+0000", 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439584+0000", 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2395000000000002e-05 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693170+0000", 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25358568300000001 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693191+0000", 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1192999999999999e-05 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693349+0000", 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015837300000000001 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693372+0000", 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2869000000000001e-05 2024-08-30T23:31:24.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.697540+0000", 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004167558 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168870 2.e 2:74d7d6f3:::benchmark_data_smithi173_47195_object71862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.924081707, 2024-08-30T23:31:24.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223579366, 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168870 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615640+0000", 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3590000000000002e-06 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615648+0000", 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8559999999999996e-06 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615665+0000", 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6563999999999999e-05 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615676+0000", 2024-08-30T23:31:24.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1075e-05 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615800+0000", 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123905 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615963+0000", 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016347900000000001 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:31:24.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616104+0000", 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140846 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620671+0000", 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045667540000000001 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620686+0000", 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.484e-05 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839134+0000", 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21844793100000001 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839172+0000", 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7737999999999997e-05 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839217+0000", 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5018999999999997e-05 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.618 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168873 2.1 2:8ff1cc28:::benchmark_data_smithi173_47195_object71865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.924013796, 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22707487400000001, 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.619 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168873 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:31:24.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615706+0000", 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2500000000000001e-07 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615708+0000", 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6970000000000002e-06 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615715+0000", 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8499999999999996e-06 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615720+0000", 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.395e-06 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616068+0000", 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034794699999999998 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616173+0000", 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104145 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616295+0000", 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012253000000000001 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620669+0000", 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043740690000000004 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.621058+0000", 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038845600000000002 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842677+0000", 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22161980100000001 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842735+0000", 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7420000000000003e-05 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842780+0000", 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5139000000000003e-05 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168892 2.e 2:7a358138:::benchmark_data_smithi173_47195_object71884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.90158949600001, 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208213383, 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168892 2024-08-30T23:31:24.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638131+0000", 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638136+0000", 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638147+0000", 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1119e-05 2024-08-30T23:31:24.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638153+0000", 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8320000000000002e-06 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638212+0000", 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9219999999999999e-05 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638342+0000", 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012929599999999999 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638476+0000", 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013422799999999999 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643016+0000", 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045403120000000003 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643358+0000", 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034153500000000002 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846238+0000", 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202880476 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846298+0000", 2024-08-30T23:31:24.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9453000000000002e-05 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846343+0000", 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5059999999999999e-05 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168895 2.1 2:802748b8:::benchmark_data_smithi173_47195_object71887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.89735320299999, 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21216137199999999, 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168895 2024-08-30T23:31:24.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642368+0000", 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3080000000000002e-06 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642376+0000", 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6769999999999995e-06 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642392+0000", 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642400+0000", 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6329999999999996e-06 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642458+0000", 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8498000000000001e-05 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642574+0000", 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115822 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642662+0000", 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655709+0000", 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013046791 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655767+0000", 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8464000000000001e-05 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854414+0000", 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19864683999999999 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854479+0000", 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5042999999999994e-05 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854527+0000", 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8195999999999997e-05 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168896 2.1 2:82321f29:::benchmark_data_smithi173_47195_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:31:24.639 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.896520181, 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211510109, 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168896 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643200+0000", 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6899999999999996e-07 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643204+0000", 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643215+0000", 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.041e-05 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643220+0000", 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646422+0000", 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032022190000000001 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646565+0000", 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142564 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646696+0000", 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130877 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655755+0000", 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090594430000000004 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655773+0000", 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7263999999999999e-05 2024-08-30T23:31:24.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854613+0000", 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19883995700000001 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854672+0000", 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9522999999999999e-05 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854709+0000", 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6693000000000003e-05 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168897 2.1 2:8f1372f3:::benchmark_data_smithi173_47195_object71889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.895995083, 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21982235999999999, 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.646 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168897 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:31:24.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643725+0000", 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643729+0000", 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643740+0000", 2024-08-30T23:31:24.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1079e-05 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643745+0000", 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2009999999999998e-06 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651404+0000", 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007658916 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651490+0000", 2024-08-30T23:31:24.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5811999999999998e-05 2024-08-30T23:31:24.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:24.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:24.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651572+0000", 2024-08-30T23:31:24.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2472999999999995e-05 2024-08-30T23:31:24.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656527+0000", 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049544910000000001 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656537+0000", 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0497e-05 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863418+0000", 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20688067199999999 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863501+0000", 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2806000000000006e-05 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863546+0000", 2024-08-30T23:31:24.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5430999999999999e-05 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:193992 2.7 e200/182 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head v 200'12142, mlcod=200'12139)", 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 101.764147217, 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60714938399999996, 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.654 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9599999999999999e-07 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775578+0000", 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8089999999999998e-06 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378923+0000", 2024-08-30T23:31:24.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60334498999999997 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378934+0000", 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0772e-05 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379004+0000", 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9783000000000006e-05 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379024+0000", 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0044e-05 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382681+0000", 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036575549999999998 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382721+0000", 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9734999999999998e-05 2024-08-30T23:31:24.658 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:194011 2.1 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 101.742433144, 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82807010000000003, 2024-08-30T23:31:24.659 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194011 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:31:24.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797287+0000", 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.277e-06 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797292+0000", 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6419999999999998e-06 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797301+0000", 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0070000000000003e-06 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797307+0000", 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2809999999999997e-06 2024-08-30T23:31:24.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797355+0000", 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8557000000000003e-05 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797473+0000", 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117904 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797597+0000", 2024-08-30T23:31:24.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012318100000000001 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.802546+0000", 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004948956 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.939024+0000", 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13647832200000001 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625288+0000", 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.686263866 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625326+0000", 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8028999999999998e-05 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625356+0000", 2024-08-30T23:31:24.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0077999999999999e-05 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:78796 3.5 e245/206 3:a1bc8ace:::benchmark_data_smithi173_60780_object78795:head v 245'4935, mlcod=245'4931)", 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:16.273777+0000", 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.2659418250000005, 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.049185866000000002, 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:24.666 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.273777+0000", 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.273777+0000", 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:24.667 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.273778+0000", 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.11e-06 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.273886+0000", 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010741 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.320876+0000", 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.046990918 2024-08-30T23:31:24.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.320885+0000", 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9539999999999993e-06 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.320947+0000", 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1216000000000005e-05 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.320964+0000", 2024-08-30T23:31:24.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7683999999999999e-05 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.322916+0000", 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019519310000000001 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:16.322963+0000", 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6643e-05 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.670 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.671 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:24.671 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:24.671 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:31:24.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:24 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:24.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:24 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:24.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:24 smithi106 ceph-mon[16603]: osdmap e249: 12 total, 12 up, 12 in 2024-08-30T23:31:24.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4286362698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:24.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:24.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:24 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:24 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:24 smithi136 ceph-mon[20207]: osdmap e249: 12 total, 12 up, 12 in 2024-08-30T23:31:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4286362698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:24 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:24 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:24 smithi173 ceph-mon[20554]: osdmap e249: 12 total, 12 up, 12 in 2024-08-30T23:31:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4286362698' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:24.851 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:24.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:24.853 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:24.854 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:24.854 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:24.854 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:24.854 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:24.854 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:24.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:24.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:24.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:24.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:25.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:25.030 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:25.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:25.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:25.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:25.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:25.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 87834 87818 74.1647 71.5 0.0248519 0.0134778 2024-08-30T23:31:25.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:25.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:25.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:25.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:25.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:25.665 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:25.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:25 smithi136 ceph-mon[20207]: pgmap v616: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 971 op/s 2024-08-30T23:31:25.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:25 smithi173 ceph-mon[20554]: pgmap v616: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 971 op/s 2024-08-30T23:31:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:25 smithi106 ceph-mon[16603]: pgmap v616: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 971 op/s 2024-08-30T23:31:25.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:25.915 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:25.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:26.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:26.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:26.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:26.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:26.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:26.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:26.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 88842 88826 74.0158 63 0.0229073 0.0135036 2024-08-30T23:31:26.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:26.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:26.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:26.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:26.551 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:26.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:31:26.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:26.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:26.780 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:26.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:26.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:26.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:26.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:27.027 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:27.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:27.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:27.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:27.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:27.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:27.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 90028 90012 74.0171 74.125 0.017424 0.0135056 2024-08-30T23:31:27.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:27.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:27.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:27.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:27.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.576 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:27.616 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:27.651 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:27.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:27.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:27.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:27.835 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:27.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:27.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:28.035 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:28.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:28.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:28.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:28.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:28.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 91082 91066 73.9113 65.875 0.00385303 0.0135241 2024-08-30T23:31:28.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:28.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:28.469 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T23:31:28.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:28.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:28.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:28 smithi106 ceph-mon[16603]: pgmap v617: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:31:28.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/779839677' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:91332 3.5 e250/206 3:a74631ab:::benchmark_data_smithi173_60780_object91331:head v 250'5672, mlcod=250'5670)", 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:28.613520+0000", 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.005853334, 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0058780969999999997, 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:28.620 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613520+0000", 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613520+0000", 2024-08-30T23:31:28.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613521+0000", 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4499999999999996e-07 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613597+0000", 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5488999999999995e-05 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613617+0000", 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0927000000000001e-05 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613624+0000", 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9990000000000002e-06 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613700+0000", 2024-08-30T23:31:28.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5564999999999999e-05 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.613721+0000", 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0469999999999999e-05 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:91336 3.7 3:e7cf9bf3:::benchmark_data_smithi173_60780_object91335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-08-30T23:31:28.624 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:28.618305+0000", 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0010689370000000001, 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001151319, 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 91336 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618305+0000", 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618305+0000", 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:28.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618306+0000", 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6339999999999999e-06 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618350+0000", 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3646000000000002e-05 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618368+0000", 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7742999999999999e-05 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618377+0000", 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0529999999999996e-06 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618437+0000", 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0337000000000002e-05 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618567+0000", 2024-08-30T23:31:28.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000129591 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.618696+0000", 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000129808 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:91320 3.7 3:e09cb518:::benchmark_data_smithi173_60780_object91319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-08-30T23:31:28.629 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:28.604213+0000", 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015160929, 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015306913, 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 91320 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604213+0000", 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604213+0000", 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604214+0000", 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3850000000000001e-06 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604264+0000", 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0411000000000002e-05 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604278+0000", 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3173000000000001e-05 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:28.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604285+0000", 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3350000000000001e-06 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604348+0000", 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2937999999999999e-05 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604455+0000", 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107198 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604548+0000", 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2868999999999995e-05 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.608681+0000", 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041328149999999998 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.608698+0000", 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7152000000000001e-05 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.634 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:91323 3.b 3:d0319274:::benchmark_data_smithi173_60780_object91322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:28.604913+0000", 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.014460550000000001, 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014675862, 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:31:28.635 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 91323 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604913+0000", 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604913+0000", 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604914+0000", 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.06e-06 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:31:28.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604970+0000", 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6132999999999999e-05 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604981+0000", 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0857e-05 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.638 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.604988+0000", 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.81e-06 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.605044+0000", 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6297e-05 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.605175+0000", 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013038700000000001 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.605355+0000", 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018035699999999999 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.608558+0000", 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032031310000000001 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:31:28.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:28.608791+0000", 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00023328199999999999 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 4 2024-08-30T23:31:28.641 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:31:28.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:28.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:28.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:28 smithi136 ceph-mon[20207]: pgmap v617: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:31:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/779839677' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:31:28.793 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:28.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:28 smithi173 ceph-mon[20554]: pgmap v617: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:31:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/779839677' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:31:28.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:28.921 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:28.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:29.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:29.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:31:29.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:29.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:29.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:29.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:29.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 92116 92100 73.7922 64.625 0.0158486 0.0135466 2024-08-30T23:31:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.305 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:29.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:29.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:29.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:29.482 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:31:29.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:29.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/779839677' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:31:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:29 smithi106 ceph-mon[16603]: osdmap e250: 12 total, 12 up, 12 in 2024-08-30T23:31:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3784720891' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:31:29.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:29.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:29.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/779839677' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:31:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:29 smithi136 ceph-mon[20207]: osdmap e250: 12 total, 12 up, 12 in 2024-08-30T23:31:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3784720891' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:31:29.803 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/779839677' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:31:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:29 smithi173 ceph-mon[20554]: osdmap e250: 12 total, 12 up, 12 in 2024-08-30T23:31:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3784720891' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:31:29.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:29.927 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:29.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:30.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:30.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:30.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:30.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:30.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:30.293 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 92869 92853 73.4538 47.0625 0.00737014 0.0136096 2024-08-30T23:31:30.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:30.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:30.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:30.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:30.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:30.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:30.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:30.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-08-30T23:31:30.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:30 smithi106 ceph-mon[16603]: pgmap v619: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:31:30.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:30 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:30.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3784720891' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:30.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:30 smithi106 ceph-mon[16603]: osdmap e251: 12 total, 12 up, 12 in 2024-08-30T23:31:30.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:30 smithi106 ceph-mon[16603]: 3.a scrub starts 2024-08-30T23:31:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:30 smithi136 ceph-mon[20207]: pgmap v619: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:31:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:30 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3784720891' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:30 smithi136 ceph-mon[20207]: osdmap e251: 12 total, 12 up, 12 in 2024-08-30T23:31:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:30 smithi136 ceph-mon[20207]: 3.a scrub starts 2024-08-30T23:31:30.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:30 smithi173 ceph-mon[20554]: pgmap v619: 17 pgs: 17 active+clean; 5.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:31:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:30 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3784720891' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:30 smithi173 ceph-mon[20554]: osdmap e251: 12 total, 12 up, 12 in 2024-08-30T23:31:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:30 smithi173 ceph-mon[20554]: 3.a scrub starts 2024-08-30T23:31:30.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:30.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:30.965 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:30.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:31.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:31:31.026 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:31.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:31.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:31.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:31.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:31.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:31:31.294975+0000 min lat: 0.00105037 max lat: 1.98739 avg lat: 0.0136265 2024-08-30T23:31:31.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:31:31.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 93908 93892 73.3473 64.9375 0.0394434 0.0136265 2024-08-30T23:31:31.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:31.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:31.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:31.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:31.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:31.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:31.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:31.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:31.849 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:31 smithi106 ceph-mon[16603]: 3.9 scrub starts 2024-08-30T23:31:31.849 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:31 smithi106 ceph-mon[16603]: 3.3 scrub starts 2024-08-30T23:31:31.850 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:31 smithi106 ceph-mon[16603]: 3.d scrub starts 2024-08-30T23:31:31.850 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:31 smithi106 ceph-mon[16603]: pgmap v621: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-30T23:31:31.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.852 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:31.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:31.904 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:31.908 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:31.908 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:31.933 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:31.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:31.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:31.960 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:31.960 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:31.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:31.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:31.989 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:31.989 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:32.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:32.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:31 smithi136 ceph-mon[20207]: 3.9 scrub starts 2024-08-30T23:31:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:31 smithi136 ceph-mon[20207]: 3.3 scrub starts 2024-08-30T23:31:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:31 smithi136 ceph-mon[20207]: 3.d scrub starts 2024-08-30T23:31:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:31 smithi136 ceph-mon[20207]: pgmap v621: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-30T23:31:32.044 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:32.044 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:31 smithi173 ceph-mon[20554]: 3.9 scrub starts 2024-08-30T23:31:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:31 smithi173 ceph-mon[20554]: 3.3 scrub starts 2024-08-30T23:31:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:31 smithi173 ceph-mon[20554]: 3.d scrub starts 2024-08-30T23:31:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:31 smithi173 ceph-mon[20554]: pgmap v621: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-30T23:31:32.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:32.075 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:32.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:32.104 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:32.104 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:32.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:32.134 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:32.134 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:32.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:32.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:32.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:32.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:32.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:31:32.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:32.218 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:32.218 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:32.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:32.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 94736 94720 73.0806 51.75 0.00541985 0.0136787 2024-08-30T23:31:32.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:32.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:32.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:32.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:32.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:32.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:32.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:32.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:32.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:32.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:32 smithi106 ceph-mon[16603]: 3.a scrub ok 2024-08-30T23:31:32.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:32 smithi106 ceph-mon[16603]: 3.2 scrub starts 2024-08-30T23:31:32.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:32 smithi106 ceph-mon[16603]: 3.9 scrub ok 2024-08-30T23:31:32.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3651865712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:32.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:32 smithi106 ceph-mon[16603]: 3.3 scrub ok 2024-08-30T23:31:32.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:32.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:32.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:32.927 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:32.927 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:33.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:33.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:33.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:32 smithi136 ceph-mon[20207]: 3.a scrub ok 2024-08-30T23:31:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:32 smithi136 ceph-mon[20207]: 3.2 scrub starts 2024-08-30T23:31:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:32 smithi136 ceph-mon[20207]: 3.9 scrub ok 2024-08-30T23:31:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3651865712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:32 smithi136 ceph-mon[20207]: 3.3 scrub ok 2024-08-30T23:31:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:32 smithi173 ceph-mon[20554]: 3.a scrub ok 2024-08-30T23:31:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:32 smithi173 ceph-mon[20554]: 3.2 scrub starts 2024-08-30T23:31:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:32 smithi173 ceph-mon[20554]: 3.9 scrub ok 2024-08-30T23:31:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3651865712' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:32 smithi173 ceph-mon[20554]: 3.3 scrub ok 2024-08-30T23:31:33.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:33.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:33.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:33.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:33.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 95925 95909 73.0956 74.3125 0.0243916 0.0136748 2024-08-30T23:31:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:33.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:33.427 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:33.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:33.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:33.587 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.6 to 1 (10000) 2024-08-30T23:31:33.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: 3.d scrub ok 2024-08-30T23:31:33.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: 3.8 scrub starts 2024-08-30T23:31:33.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:31:33.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:31:33.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: pgmap v622: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-08-30T23:31:33.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1156191583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:31:33.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:31:33.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:33 smithi106 ceph-mon[16603]: 3.5 scrub starts 2024-08-30T23:31:33.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:33.682 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:33.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:33.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:33.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:31:33.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:33.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:33.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: 3.d scrub ok 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: 3.8 scrub starts 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: pgmap v622: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1156191583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:31:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:33 smithi136 ceph-mon[20207]: 3.5 scrub starts 2024-08-30T23:31:34.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: 3.d scrub ok 2024-08-30T23:31:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: 3.8 scrub starts 2024-08-30T23:31:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:31:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:31:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: pgmap v622: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-08-30T23:31:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1156191583' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:31:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:31:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:33 smithi173 ceph-mon[20554]: 3.5 scrub starts 2024-08-30T23:31:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:34.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:34.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:34.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:34.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 96913 96897 72.9588 61.75 0.0237958 0.0137011 2024-08-30T23:31:34.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:34.340 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:34.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:34.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:34.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:34.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:34.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:34.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-08-30T23:31:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:34.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:34.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:34 smithi106 ceph-mon[16603]: 3.6 scrub starts 2024-08-30T23:31:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:34 smithi106 ceph-mon[16603]: 3.2 scrub ok 2024-08-30T23:31:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:31:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:34 smithi106 ceph-mon[16603]: osdmap e252: 12 total, 12 up, 12 in 2024-08-30T23:31:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:34 smithi106 ceph-mon[16603]: 3.8 scrub ok 2024-08-30T23:31:34.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:34.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:34.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:35.000 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:35.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:34 smithi173 ceph-mon[20554]: 3.6 scrub starts 2024-08-30T23:31:35.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:34 smithi173 ceph-mon[20554]: 3.2 scrub ok 2024-08-30T23:31:35.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:31:35.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:34 smithi173 ceph-mon[20554]: osdmap e252: 12 total, 12 up, 12 in 2024-08-30T23:31:35.002 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:34 smithi173 ceph-mon[20554]: 3.8 scrub ok 2024-08-30T23:31:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:34 smithi136 ceph-mon[20207]: 3.6 scrub starts 2024-08-30T23:31:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:34 smithi136 ceph-mon[20207]: 3.2 scrub ok 2024-08-30T23:31:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:31:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:34 smithi136 ceph-mon[20207]: osdmap e252: 12 total, 12 up, 12 in 2024-08-30T23:31:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:34 smithi136 ceph-mon[20207]: 3.8 scrub ok 2024-08-30T23:31:35.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:35.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:35.193 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:35.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:35.254 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:35.293 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:35.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 97995 97979 72.8953 67.625 0.0130095 0.0137139 2024-08-30T23:31:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:35.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:35.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:35.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:35.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:35.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:35.679 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:35.747 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:35.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:35.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:35 smithi106 ceph-mon[16603]: 3.6 scrub ok 2024-08-30T23:31:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:35 smithi106 ceph-mon[16603]: 3.4 scrub starts 2024-08-30T23:31:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:35 smithi106 ceph-mon[16603]: 3.1 scrub starts 2024-08-30T23:31:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:35 smithi106 ceph-mon[16603]: pgmap v624: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-08-30T23:31:35.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:35.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:35.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:36.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:35 smithi136 ceph-mon[20207]: 3.6 scrub ok 2024-08-30T23:31:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:35 smithi136 ceph-mon[20207]: 3.4 scrub starts 2024-08-30T23:31:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:35 smithi136 ceph-mon[20207]: 3.1 scrub starts 2024-08-30T23:31:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:35 smithi136 ceph-mon[20207]: pgmap v624: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-08-30T23:31:36.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:35 smithi173 ceph-mon[20554]: 3.6 scrub ok 2024-08-30T23:31:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:35 smithi173 ceph-mon[20554]: 3.4 scrub starts 2024-08-30T23:31:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:35 smithi173 ceph-mon[20554]: 3.1 scrub starts 2024-08-30T23:31:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:35 smithi173 ceph-mon[20554]: pgmap v624: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-08-30T23:31:36.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:36.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:36.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:36.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 98934 98918 72.7281 58.6875 0.0214356 0.0137428 2024-08-30T23:31:36.316 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:36.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:36.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:36.531 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:36.531 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:36.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:36.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:36.597 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:31:36.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:36.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:36.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:36.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:36 smithi106 ceph-mon[16603]: 3.5 scrub ok 2024-08-30T23:31:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1176775125' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:36.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:36.940 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:36 smithi136 ceph-mon[20207]: 3.5 scrub ok 2024-08-30T23:31:36.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1176775125' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:36.964 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:36.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:37.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:36 smithi173 ceph-mon[20554]: 3.5 scrub ok 2024-08-30T23:31:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1176775125' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:37.091 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:37.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:37.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:37.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:31:37.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 99998 99982 72.6556 66.5 0.0189091 0.0137585 2024-08-30T23:31:37.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.351 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:37.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:37.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:37.478 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:37.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:37.599 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T23:31:37.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:37.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:37.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:37.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: 3.4 scrub ok 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1176775125' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: osdmap e253: 12 total, 12 up, 12 in 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: 3.1 scrub ok 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/474321220' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:31:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:37 smithi106 ceph-mon[16603]: pgmap v626: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-08-30T23:31:37.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:37.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:37.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:37.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: 3.4 scrub ok 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1176775125' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: osdmap e253: 12 total, 12 up, 12 in 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: 3.1 scrub ok 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/474321220' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:31:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:37 smithi136 ceph-mon[20207]: pgmap v626: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-08-30T23:31:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: 3.4 scrub ok 2024-08-30T23:31:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:31:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:31:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1176775125' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: osdmap e253: 12 total, 12 up, 12 in 2024-08-30T23:31:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: 3.1 scrub ok 2024-08-30T23:31:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/474321220' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:31:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:37 smithi173 ceph-mon[20554]: pgmap v626: 17 pgs: 4 active+clean+scrubbing, 13 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 951 op/s 2024-08-30T23:31:38.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:38.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:38.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:38.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:38.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 100770 100754 72.375 48.25 0.00501756 0.0138122 2024-08-30T23:31:38.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:38.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-08-30T23:31:38.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:38.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:38.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:38.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:38.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/474321220' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:31:38.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:38 smithi106 ceph-mon[16603]: osdmap e254: 12 total, 12 up, 12 in 2024-08-30T23:31:38.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:38 smithi106 ceph-mon[16603]: 3.7 scrub starts 2024-08-30T23:31:38.631 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:38.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:38.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:38.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:31:38.825 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:38.857 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:38.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:38.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:38.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:39.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:39.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:39.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/474321220' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:31:39.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:38 smithi136 ceph-mon[20207]: osdmap e254: 12 total, 12 up, 12 in 2024-08-30T23:31:39.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:38 smithi136 ceph-mon[20207]: 3.7 scrub starts 2024-08-30T23:31:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/474321220' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:31:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:38 smithi173 ceph-mon[20554]: osdmap e254: 12 total, 12 up, 12 in 2024-08-30T23:31:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:38 smithi173 ceph-mon[20554]: 3.7 scrub starts 2024-08-30T23:31:39.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.170 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:39.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:39.171 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:39.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:39.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:39.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 101710 101694 72.2201 58.75 0.0218086 0.0138291 2024-08-30T23:31:39.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:39.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:39.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:39.550 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:39.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:39.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:39.720 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:31:39.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:39 smithi106 ceph-mon[16603]: 3.b scrub starts 2024-08-30T23:31:39.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:39 smithi106 ceph-mon[16603]: 3.e scrub starts 2024-08-30T23:31:39.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:39 smithi106 ceph-mon[16603]: 3.f scrub starts 2024-08-30T23:31:39.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:39 smithi106 ceph-mon[16603]: pgmap v628: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-30T23:31:39.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:39.824 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:39.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:39.955 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:39.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:39.958 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:39.958 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:40.000 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:40.001 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:40.022 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:39 smithi136 ceph-mon[20207]: 3.b scrub starts 2024-08-30T23:31:40.022 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:39 smithi136 ceph-mon[20207]: 3.e scrub starts 2024-08-30T23:31:40.022 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:39 smithi136 ceph-mon[20207]: 3.f scrub starts 2024-08-30T23:31:40.022 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:39 smithi136 ceph-mon[20207]: pgmap v628: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-30T23:31:40.033 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:40.033 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:40.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:40.065 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:40.065 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:39 smithi173 ceph-mon[20554]: 3.b scrub starts 2024-08-30T23:31:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:39 smithi173 ceph-mon[20554]: 3.e scrub starts 2024-08-30T23:31:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:39 smithi173 ceph-mon[20554]: 3.f scrub starts 2024-08-30T23:31:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:39 smithi173 ceph-mon[20554]: pgmap v628: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-30T23:31:40.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:40.096 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:40.101 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:40.133 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:40.162 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:40.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:40.183 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:31:40.183 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:31:40.183 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:31:40.184 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:31:40.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:40.192 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:40.192 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:40.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:40.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:40.252 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:40.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:40.294 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 102531 102515 71.9851 51.3125 0.0279038 0.0138871 2024-08-30T23:31:40.309 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:40.310 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:40.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:40.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:40.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:40.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:40.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:40.604 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:40.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:40.616 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.8 to 1 (10000) 2024-08-30T23:31:40.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:40 smithi136 ceph-mon[20207]: 3.b scrub ok 2024-08-30T23:31:40.682 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1533880221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:40.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/225593458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:31:40.683 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:40 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:31:40.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:40.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:40.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:40.862 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:40.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:40 smithi106 ceph-mon[16603]: 3.b scrub ok 2024-08-30T23:31:40.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1533880221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:40.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/225593458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:31:40.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:40 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:31:40.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:40.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:40.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:40.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:31:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:40 smithi173 ceph-mon[20554]: 3.b scrub ok 2024-08-30T23:31:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1533880221' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/225593458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:31:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:40 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:31:41.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:41.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:41.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:41.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:41.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 7 103451 103444 71.8304 58.0625 0.00758629 0.0139159 2024-08-30T23:31:41.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:41.353 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:41.371 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:41.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:41.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:41.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:41.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:41.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:41.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:41 smithi136 ceph-mon[20207]: 3.e scrub ok 2024-08-30T23:31:41.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:41 smithi136 ceph-mon[20207]: 3.f scrub ok 2024-08-30T23:31:41.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:41 smithi136 ceph-mon[20207]: 3.7 scrub ok 2024-08-30T23:31:41.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:41 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:31:41.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:41 smithi136 ceph-mon[20207]: osdmap e255: 12 total, 12 up, 12 in 2024-08-30T23:31:41.740 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:41 smithi136 ceph-mon[20207]: pgmap v630: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.06k op/s 2024-08-30T23:31:41.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:31:41.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:41.872 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:41.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:41.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:42.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:42.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:41 smithi173 ceph-mon[20554]: 3.e scrub ok 2024-08-30T23:31:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:41 smithi173 ceph-mon[20554]: 3.f scrub ok 2024-08-30T23:31:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:41 smithi173 ceph-mon[20554]: 3.7 scrub ok 2024-08-30T23:31:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:41 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:31:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:41 smithi173 ceph-mon[20554]: osdmap e255: 12 total, 12 up, 12 in 2024-08-30T23:31:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:41 smithi173 ceph-mon[20554]: pgmap v630: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.06k op/s 2024-08-30T23:31:42.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:41 smithi106 ceph-mon[16603]: 3.e scrub ok 2024-08-30T23:31:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:41 smithi106 ceph-mon[16603]: 3.f scrub ok 2024-08-30T23:31:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:41 smithi106 ceph-mon[16603]: 3.7 scrub ok 2024-08-30T23:31:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:41 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:31:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:41 smithi106 ceph-mon[16603]: osdmap e255: 12 total, 12 up, 12 in 2024-08-30T23:31:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:41 smithi106 ceph-mon[16603]: pgmap v630: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.06k op/s 2024-08-30T23:31:42.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:42.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:42.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:42.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.6094 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 103451 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 71.3578 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 21.2769 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 129.5 2024-08-30T23:31:42.295 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 10.0625 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1141 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 340.43 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2072 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 161 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0139304 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0343006 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 1.98739 2024-08-30T23:31:42.296 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00105037 2024-08-30T23:31:42.301 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:31:42.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:42.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:42.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:42.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:42.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:42.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:42.656 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:31:42.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:42.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:42.853 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3109722483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:31:42.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-08-30T23:31:42.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:42.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:42.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:42.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:43.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:43.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3109722483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:31:43.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3109722483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:31:43.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:43.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:43.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:43.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:43.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:43.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:43.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.531 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:43.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:43.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:43.653 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:43.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.656 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:43.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:43.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:43 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3109722483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:31:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:43 smithi106 ceph-mon[16603]: osdmap e256: 12 total, 12 up, 12 in 2024-08-30T23:31:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:43 smithi106 ceph-mon[16603]: pgmap v632: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 53 MiB/s wr, 857 op/s 2024-08-30T23:31:43.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:43.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:43.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:44.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:44.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:43 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:44.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3109722483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:31:44.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:43 smithi136 ceph-mon[20207]: osdmap e256: 12 total, 12 up, 12 in 2024-08-30T23:31:44.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:43 smithi136 ceph-mon[20207]: pgmap v632: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 53 MiB/s wr, 857 op/s 2024-08-30T23:31:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:43 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3109722483' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:31:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:43 smithi173 ceph-mon[20554]: osdmap e256: 12 total, 12 up, 12 in 2024-08-30T23:31:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:43 smithi173 ceph-mon[20554]: pgmap v632: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 53 MiB/s wr, 857 op/s 2024-08-30T23:31:44.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:44.167 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:44.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:31:44.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:44.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:44.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:44.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:44.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:44.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:44.578 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:44.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:44.672 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.9 to 1 (10000) 2024-08-30T23:31:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:44.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:44.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:44.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:44.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:44.936 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:44.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:45.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4285447373' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:31:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:44 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:31:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:45.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:45.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:31:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4285447373' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:31:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:44 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:31:45.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4285447373' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:31:45.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:44 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:31:45.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:45.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:45.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:45.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:45.399 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:45.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:45.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:45.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:45.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:45.570 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:45.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:45.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-08-30T23:31:45.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:31:45.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:45.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:45 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:31:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:45 smithi136 ceph-mon[20207]: osdmap e257: 12 total, 12 up, 12 in 2024-08-30T23:31:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:45 smithi136 ceph-mon[20207]: pgmap v634: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 39 MiB/s wr, 752 op/s 2024-08-30T23:31:45.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:45.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:45 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:31:45.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:45 smithi106 ceph-mon[16603]: osdmap e257: 12 total, 12 up, 12 in 2024-08-30T23:31:45.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:45 smithi106 ceph-mon[16603]: pgmap v634: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 39 MiB/s wr, 752 op/s 2024-08-30T23:31:45.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:45.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:45.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:45.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:46.006 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:31:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:31:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:31:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:49996 2.5 e122/97 2:aad518df:::benchmark_data_smithi173_47195_object49995:head v 122'3107, mlcod=122'3104)", 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 310.74585159600002, 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20284867600000001, 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:31:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.63e-07 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258370+0000", 2024-08-30T23:31:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8464e-05 2024-08-30T23:31:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459751+0000", 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.201380686 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459757+0000", 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.319e-06 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459801+0000", 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3912999999999998e-05 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459814+0000", 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3548e-05 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.461105+0000", 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012902320000000001 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.461140+0000", 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5051e-05 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:109587 2.d e153/97 2:b377b128:::benchmark_data_smithi173_47195_object12579:head v 153'6909, mlcod=153'6905)", 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:31:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 241.09902800200001, 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18410296100000001, 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905116+0000", 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5770000000000001e-06 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905126+0000", 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0356999999999999e-05 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086753+0000", 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18162683099999999 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086760+0000", 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5379999999999999e-06 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086804+0000", 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4310000000000001e-05 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086821+0000", 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7135000000000001e-05 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.089190+0000", 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023685709999999999 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.089217+0000", 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7642000000000001e-05 2024-08-30T23:31:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:2679 3.3 3:c7f6fa7e:::benchmark_data_smithi173_60780_object2678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.692616+0000", 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.311526223000001, 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18271148200000001, 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 2679 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692616+0000", 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692616+0000", 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692617+0000", 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6199999999999998e-07 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692797+0000", 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000180337 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692802+0000", 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0499999999999999e-06 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692806+0000", 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9990000000000002e-06 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692847+0000", 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0677999999999999e-05 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692931+0000", 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4337000000000002e-05 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:31:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693030+0000", 2024-08-30T23:31:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8945999999999998e-05 2024-08-30T23:31:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693817+0000", 2024-08-30T23:31:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078704800000000004 2024-08-30T23:31:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.024 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:46.025 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694304+0000", 2024-08-30T23:31:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048707999999999998 2024-08-30T23:31:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875226+0000", 2024-08-30T23:31:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.180922157 2024-08-30T23:31:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875283+0000", 2024-08-30T23:31:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6572e-05 2024-08-30T23:31:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.875328+0000", 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4715999999999997e-05 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:2681 3.3 3:c917533a:::benchmark_data_smithi173_60780_object2680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.311328054000001, 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20960189000000001, 2024-08-30T23:31:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 2681 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692815+0000", 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0100000000000005e-07 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692841+0000", 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6024e-05 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692847+0000", 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3520000000000003e-06 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692853+0000", 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2209999999999996e-06 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693408+0000", 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055510499999999996 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693502+0000", 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3960000000000002e-05 2024-08-30T23:31:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693587+0000", 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5558999999999999e-05 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694000+0000", 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041311100000000002 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.701466+0000", 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074651769999999999 2024-08-30T23:31:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902340+0000", 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200874052 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902381+0000", 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0929000000000001e-05 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902416+0000", 2024-08-30T23:31:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5799000000000001e-05 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:2681 3.3 e213/206 ondisk, result = 0)", 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.309366685000001, 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207646681, 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:31:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1e-07 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694779+0000", 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5129999999999999e-06 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902275+0000", 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207495916 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902280+0000", 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8850000000000004e-06 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902329+0000", 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8656999999999998e-05 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902336+0000", 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.528e-06 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902423+0000", 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6872000000000004e-05 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:21973 3.8 e214/206 3:16383432:::benchmark_data_smithi173_60780_object21972:head v 214'1387, mlcod=214'1385)", 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.267943+0000", 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.736199475000006, 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21049256599999999, 2024-08-30T23:31:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.267943+0000", 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.267943+0000", 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.267944+0000", 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2399999999999998e-07 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.268056+0000", 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112543 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476599+0000", 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20854252600000001 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476608+0000", 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8579999999999995e-06 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476688+0000", 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9915999999999999e-05 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476715+0000", 2024-08-30T23:31:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6392000000000001e-05 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478396+0000", 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016809349999999999 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478436+0000", 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9872e-05 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22010 3.3 3:c1c2ae0b:::benchmark_data_smithi173_60780_object22009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.293349+0000", 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.710793811000002, 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18727933099999999, 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22010 2024-08-30T23:31:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293349+0000", 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293349+0000", 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293355+0000", 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3949999999999998e-06 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293391+0000", 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5880000000000002e-05 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293403+0000", 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1629e-05 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293409+0000", 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.229e-06 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293467+0000", 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7692000000000003e-05 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293597+0000", 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130495 2024-08-30T23:31:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293748+0000", 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015096900000000001 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294588+0000", 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083958899999999996 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294909+0000", 2024-08-30T23:31:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032116499999999999 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480521+0000", 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18561213200000001 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480584+0000", 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3246999999999999e-05 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480628+0000", 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3909000000000003e-05 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:22010 3.3 e214/206 ondisk, result = 0)", 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.709219879000003, 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18571167899999999, 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294925+0000", 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4109999999999999e-06 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480459+0000", 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.185533737 2024-08-30T23:31:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480466+0000", 2024-08-30T23:31:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4810000000000003e-06 2024-08-30T23:31:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480505+0000", 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9511999999999997e-05 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480514+0000", 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.636e-06 2024-08-30T23:31:46.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480634+0000", 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120469 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29340 3.b e219/206 3:d304e59a:::benchmark_data_smithi173_60780_object29339:head v 219'1853, mlcod=219'1849)", 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.433764+0000", 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.570378261000002, 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.208233679, 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.057 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433764+0000", 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433764+0000", 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433765+0000", 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433814+0000", 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.901e-05 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640719+0000", 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20690529099999999 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640728+0000", 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4509999999999992e-06 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640791+0000", 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2963e-05 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640817+0000", 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5927000000000001e-05 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.641951+0000", 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011341540000000001 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.641998+0000", 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7185000000000003e-05 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.061 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:29363 3.3 e219/206 ondisk, result = 0)", 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.454808+0000", 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.549334220000006, 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20210732300000001, 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454808+0000", 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454808+0000", 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454809+0000", 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.23e-07 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454813+0000", 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1389999999999997e-06 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656737+0000", 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20192416999999999 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656743+0000", 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9669999999999998e-06 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656785+0000", 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1836999999999999e-05 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656790+0000", 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3589999999999999e-06 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656916+0000", 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012512800000000001 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29375 3.6 e219/206 3:660484a6:::benchmark_data_smithi173_60780_object29374:head v 219'1852, mlcod=219'1848)", 2024-08-30T23:31:46.065 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.461797+0000", 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.542345666000003, 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18554573199999999, 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461797+0000", 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461797+0000", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461798+0000", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2200000000000003e-07 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461847+0000", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9172000000000003e-05 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645099+0000", 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183251885 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645104+0000", 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.378e-06 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645145+0000", 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0611999999999997e-05 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645158+0000", 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3193000000000001e-05 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.647309+0000", 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021510629999999999 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.647343+0000", 2024-08-30T23:31:46.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3606999999999999e-05 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29376 3.8 e219/206 3:1bf89ba2:::benchmark_data_smithi173_60780_object29375:head v 219'1864, mlcod=219'1860)", 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.462195+0000", 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.541947397000001, 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183932077, 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462195+0000", 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462195+0000", 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462196+0000", 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0800000000000005e-07 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462268+0000", 2024-08-30T23:31:46.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1931e-05 2024-08-30T23:31:46.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:46.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.644961+0000", 2024-08-30T23:31:46.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18269360300000001 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.644968+0000", 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3990000000000001e-06 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645014+0000", 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6350000000000002e-05 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645027+0000", 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2822e-05 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646090+0000", 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001063587 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646127+0000", 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6876999999999999e-05 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29377 3.6 e219/206 3:61a54e16:::benchmark_data_smithi173_60780_object29376:head v 219'1853, mlcod=219'1848)", 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.463773+0000", 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.540369968999997, 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183697216, 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.463773+0000", 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.463773+0000", 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.463773+0000", 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2199999999999999e-07 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.463830+0000", 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6827000000000003e-05 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646494+0000", 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18266418200000001 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646500+0000", 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9030000000000001e-06 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646551+0000", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0701000000000001e-05 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646568+0000", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7762999999999999e-05 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.647442+0000", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00087396800000000003 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.647470+0000", 2024-08-30T23:31:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.745e-05 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41096 3.3 3:c1280440:::benchmark_data_smithi173_60780_object41095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 62.764157009999998, 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19616752700000001, 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41096 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5000000000000001e-07 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240040+0000", 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3548999999999998e-05 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240049+0000", 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3219999999999999e-06 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240057+0000", 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.571e-06 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240104+0000", 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6649e-05 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240188+0000", 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4746999999999993e-05 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240321+0000", 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013225499999999999 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241598+0000", 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012777960000000001 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.242242+0000", 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064343400000000002 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436092+0000", 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.193849724 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436127+0000", 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.519e-05 2024-08-30T23:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436153+0000", 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6339999999999999e-05 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:41096 3.3 e226/206 ondisk, result = 0)", 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 62.762327548000002, 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19434237100000001, 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.89e-07 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241822+0000", 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4010000000000002e-06 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436053+0000", 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19423099599999999 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436057+0000", 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3529999999999997e-06 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436086+0000", 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-05 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436089+0000", 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0639999999999998e-06 2024-08-30T23:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436157+0000", 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8638000000000003e-05 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:60810 3.1 3:826953a7:::benchmark_data_smithi173_60780_object60809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:59.893893+0000", 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.110249801999998, 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.187316551, 2024-08-30T23:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60810 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893893+0000", 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893893+0000", 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893894+0000", 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0899999999999999e-06 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893943+0000", 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8826000000000003e-05 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893951+0000", 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4859999999999997e-06 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893958+0000", 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4169999999999997e-06 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.894004+0000", 2024-08-30T23:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6424000000000001e-05 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.894101+0000", 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7459000000000003e-05 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.894190+0000", 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8355000000000004e-05 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895484+0000", 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001294447 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.897905+0000", 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024205699999999999 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081147+0000", 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183242612 2024-08-30T23:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081181+0000", 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3503000000000001e-05 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081209+0000", 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8362000000000001e-05 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:60810 3.1 e236/230 ondisk, result = 0)", 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:59.895762+0000", 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.108380377000003, 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.185451543, 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895762+0000", 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895762+0000", 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895763+0000", 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.44e-07 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895769+0000", 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081103+0000", 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18533458799999999 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081108+0000", 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.69e-06 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081140+0000", 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1578999999999997e-05 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081144+0000", 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.437e-06 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081214+0000", 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9638e-05 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74470 3.3 3:c5e97d47:::benchmark_data_smithi173_60780_object74469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 34.019105904, 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20435176499999999, 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74470 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8299999999999996e-07 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985109+0000", 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2125000000000005e-05 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985120+0000", 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0346999999999999e-05 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985125+0000", 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0200000000000002e-06 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.986495+0000", 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013701309999999999 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.986625+0000", 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013006000000000001 2024-08-30T23:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.986775+0000", 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149863 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.988511+0000", 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017362969999999999 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.002799+0000", 2024-08-30T23:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014288175 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.189318+0000", 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18651902300000001 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.189352+0000", 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3485999999999998e-05 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.189388+0000", 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6554999999999998e-05 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74488 3.3 3:cadebabe:::benchmark_data_smithi173_60780_object74487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:12.004501+0000", 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.999641015999998, 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19292046400000001, 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74488 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004501+0000", 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004501+0000", 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004502+0000", 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9999999999999996e-07 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004566+0000", 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4166999999999997e-05 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004576+0000", 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-06 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004581+0000", 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8069999999999999e-06 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004629+0000", 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7555000000000001e-05 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004752+0000", 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012330199999999999 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004895+0000", 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014360999999999999 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.006892+0000", 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019964190000000001 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017722+0000", 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010830367 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197329+0000", 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179606565 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197381+0000", 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2658999999999998e-05 2024-08-30T23:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197422+0000", 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0513000000000003e-05 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:74498 3.b e243/206 3:d56fca87:::benchmark_data_smithi173_60780_object74497:head v 243'4726, mlcod=243'4724)", 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.986399996000003, 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20062028400000001, 2024-08-30T23:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017800+0000", 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6663999999999998e-05 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216278+0000", 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19847857899999999 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216285+0000", 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.37e-06 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216334+0000", 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9434999999999997e-05 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216349+0000", 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4932e-05 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.218333+0000", 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019840489999999999 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.218363+0000", 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9776000000000001e-05 2024-08-30T23:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:31:46.119 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:45 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:31:46.119 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:45 smithi173 ceph-mon[20554]: osdmap e257: 12 total, 12 up, 12 in 2024-08-30T23:31:46.119 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:45 smithi173 ceph-mon[20554]: pgmap v634: 17 pgs: 17 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 39 MiB/s wr, 752 op/s 2024-08-30T23:31:46.143 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:46.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:46.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:46.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:46.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:46.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:46.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:46.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:46.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:46.721 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:46.721 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:46.745 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:46.745 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:46.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:46.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:46.772 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:46.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:46.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:46.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:46.822 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:46.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:46.853 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:46.853 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:46.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:46.880 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:46.880 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:46.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:46.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:46.904 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:46.904 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:46.925 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:46.926 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:46.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/900748567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:46.946 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:46.947 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:46.967 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:46.968 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:46.974 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/900748567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:46.991 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:46.991 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:47.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:47.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:31:47.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/900748567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:47.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:47.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:47.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:47.279 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:47.356 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:47.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:47.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:47.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:47.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:47.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:47.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:47.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:47 smithi136 ceph-mon[20207]: pgmap v635: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 150 B/s rd, 30 MiB/s wr, 882 op/s 2024-08-30T23:31:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:47 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:31:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:47 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:31:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:47 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:31:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:47 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:31:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:47.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:47.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:47.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:47.955 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:47.955 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:48.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:48.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:47 smithi173 ceph-mon[20554]: pgmap v635: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 150 B/s rd, 30 MiB/s wr, 882 op/s 2024-08-30T23:31:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:47 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:31:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:47 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:31:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:47 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:31:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:47 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:31:48.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:47 smithi106 ceph-mon[16603]: pgmap v635: 17 pgs: 17 active+clean; 6.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 150 B/s rd, 30 MiB/s wr, 882 op/s 2024-08-30T23:31:48.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:47 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:31:48.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:47 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:31:48.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:47 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:31:48.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:47 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:31:48.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:48.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:48.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:48.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:48.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:48.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:48.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:48.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:48.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:48.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:48.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:31:48.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:48.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:48.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:48.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:48.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:48.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:49.049 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:49.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:49.089 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:49.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:49.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:49.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:49.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:49.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:49.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:49.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:49.598 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:49.598 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:49.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:49.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:49.733 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:49.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.861 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:49.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:49.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:49.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:49.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:50.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:50.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:50.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:50.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:50.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:50.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:31:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.373 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:50.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:50.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:50.479 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.11 to 1 (10000) 2024-08-30T23:31:50.517 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:50.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:50.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:50.727 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:50.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:50 smithi136 ceph-mon[20207]: pgmap v636: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 14 MiB/s wr, 798 op/s 2024-08-30T23:31:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1844029321' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:31:50.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:50.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:50 smithi106 ceph-mon[16603]: pgmap v636: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 14 MiB/s wr, 798 op/s 2024-08-30T23:31:50.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1844029321' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:31:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:50 smithi173 ceph-mon[20554]: pgmap v636: 17 pgs: 17 active+clean; 6.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 14 MiB/s wr, 798 op/s 2024-08-30T23:31:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1844029321' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-30T23:31:50.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:50.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:50.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:51.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:51.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:51.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:51.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:51.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:51.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:51.342 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:51.342 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:51.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:51.446 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:51.484 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:31:51.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:51.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:31:51.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:51.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:51.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1844029321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:31:51.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:51 smithi136 ceph-mon[20207]: osdmap e258: 12 total, 12 up, 12 in 2024-08-30T23:31:51.764 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3707195809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:51.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1844029321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:31:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:51 smithi173 ceph-mon[20554]: osdmap e258: 12 total, 12 up, 12 in 2024-08-30T23:31:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3707195809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:51.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1844029321' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-30T23:31:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:51 smithi106 ceph-mon[16603]: osdmap e258: 12 total, 12 up, 12 in 2024-08-30T23:31:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3707195809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:31:51.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-08-30T23:31:51.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:51.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:51.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:52.075 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:52.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:31:52.101 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:52.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.225 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:52.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:52.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:52.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:52.431 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:52.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:52.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:52.590 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:52.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:52.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:52.749 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:52 smithi136 ceph-mon[20207]: pgmap v638: 17 pgs: 17 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 986 op/s 2024-08-30T23:31:52.749 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:52 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:52.749 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3707195809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:52.749 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:52 smithi136 ceph-mon[20207]: osdmap e259: 12 total, 12 up, 12 in 2024-08-30T23:31:52.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:52 smithi173 ceph-mon[20554]: pgmap v638: 17 pgs: 17 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 986 op/s 2024-08-30T23:31:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:52 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3707195809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:52 smithi173 ceph-mon[20554]: osdmap e259: 12 total, 12 up, 12 in 2024-08-30T23:31:52.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:52 smithi106 ceph-mon[16603]: pgmap v638: 17 pgs: 17 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s wr, 986 op/s 2024-08-30T23:31:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:52 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3707195809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:31:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:52 smithi106 ceph-mon[16603]: osdmap e259: 12 total, 12 up, 12 in 2024-08-30T23:31:52.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:52.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:52.944 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:52.979 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:31:52.979 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:31:52.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:53.015 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:53.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:53.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.142 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:53.236 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:53.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:53.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:53.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:53.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:53.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:53.495 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:53.495 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:53.531 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:53.533 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:53.534 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:53.566 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:53.567 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:53.594 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:53.594 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:53.618 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:53.618 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:53.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:53.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:53.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:31:53.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:53.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:53 smithi136 ceph-mon[20207]: pgmap v640: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s wr, 878 op/s 2024-08-30T23:31:53.623 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/421104928' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:53.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:53.646 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:53.646 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:53.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:53.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:31:53.674 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:53.675 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:53.708 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:31:53.708 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:31:53.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:31:53.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:31:53.754 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:53.754 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:53.754 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:31:53.754 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:53.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:53 smithi106 ceph-mon[16603]: pgmap v640: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s wr, 878 op/s 2024-08-30T23:31:53.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/421104928' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:53.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:53.763 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:31:53.764 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:31:53.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:53.789 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:31:53.789 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:31:53.813 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:31:53.813 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:31:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:31:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:53 smithi173 ceph-mon[20554]: pgmap v640: 17 pgs: 17 active+clean; 5.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s wr, 878 op/s 2024-08-30T23:31:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/421104928' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:53.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:53.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:53.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:54.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:54.030 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:54.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:54.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:54.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:54.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:54.307 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:54.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:54.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:54.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:54.558 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.10 to 1 (10000) 2024-08-30T23:31:54.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:54.570 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:31:54.570 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:31:54.570 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:31:54.570 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:31:54.570 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: osdmap e260: 12 total, 12 up, 12 in 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:31:54.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3771517972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:31:54.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:54.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: osdmap e260: 12 total, 12 up, 12 in 2024-08-30T23:31:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:31:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3771517972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:31:54.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:54.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: osdmap e260: 12 total, 12 up, 12 in 2024-08-30T23:31:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:31:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:31:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:31:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:31:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3771517972' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:31:54.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:54.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:54.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:54.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:54.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:31:55.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:55.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:55.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:55.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:55.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:55.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:55.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:55.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:55.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:55.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:55.591 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T23:31:55.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:31:55.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:55.678 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:55.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:55.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:55.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3771517972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:31:55.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:55 smithi106 ceph-mon[16603]: osdmap e261: 12 total, 12 up, 12 in 2024-08-30T23:31:55.805 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:55 smithi106 ceph-mon[16603]: pgmap v643: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-08-30T23:31:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3771517972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:31:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:55 smithi173 ceph-mon[20554]: osdmap e261: 12 total, 12 up, 12 in 2024-08-30T23:31:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:55 smithi173 ceph-mon[20554]: pgmap v643: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-08-30T23:31:55.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:55.950 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:55.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:56.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3771517972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:31:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:55 smithi136 ceph-mon[20207]: osdmap e261: 12 total, 12 up, 12 in 2024-08-30T23:31:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:55 smithi136 ceph-mon[20207]: pgmap v643: 17 pgs: 17 active+clean; 5.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s 2024-08-30T23:31:56.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:56.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:56.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:31:56.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:56.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:56.251 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:56.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:56.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:56.413 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:56.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:56.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:56.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:56.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:56.590 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:31:56.634 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/301363643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:31:56.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:56.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:56.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:31:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/301363643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:31:56.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:56.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:56.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:56.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:56.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1905 2.7 2:e3c3e2b6:::benchmark_data_smithi173_47195_object1904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.74963828, 2024-08-30T23:31:57.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.214235494, 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1905 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320112+0000", 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320114+0000", 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8139999999999999e-06 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320140+0000", 2024-08-30T23:31:57.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5607999999999999e-05 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320149+0000", 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1779999999999992e-06 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320156+0000", 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320214+0000", 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7816999999999999e-05 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320293+0000", 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9528000000000002e-05 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320375+0000", 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1314000000000001e-05 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.320722+0000", 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034726800000000002 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.321314+0000", 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059229900000000002 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534242+0000", 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21292789500000001 2024-08-30T23:31:57.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534303+0000", 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0656000000000003e-05 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534348+0000", 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5039e-05 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1915 2.7 2:e31a5d2a:::benchmark_data_smithi173_47195_object1914:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.74745600599999, 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21216159100000001, 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1915 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322295+0000", 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3600000000000002e-07 2024-08-30T23:31:57.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322318+0000", 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2801000000000001e-05 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322326+0000", 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3629999999999996e-06 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322330+0000", 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8069999999999999e-06 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322371+0000", 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0200000000000001e-05 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322445+0000", 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4750000000000001e-05 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322526+0000", 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0927000000000006e-05 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.322865+0000", 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033871900000000002 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.323144+0000", 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027930899999999999 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534388+0000", 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21124331399999999 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534422+0000", 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4087000000000001e-05 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534456+0000", 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4477999999999998e-05 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1922 2.7 2:ee8ad553:::benchmark_data_smithi173_47195_object1921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:31:57.084 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.74307306399999, 2024-08-30T23:31:57.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207961537, 2024-08-30T23:31:57.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:57.086 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/301363643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:31:57.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1922 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326678+0000", 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326753+0000", 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4973000000000002e-05 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326764+0000", 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0787e-05 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326769+0000", 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0189999999999997e-06 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326810+0000", 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.125e-05 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326889+0000", 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8110999999999998e-05 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.326971+0000", 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2132999999999994e-05 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.327277+0000", 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030633899999999998 2024-08-30T23:31:57.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.327759+0000", 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048190199999999998 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534555+0000", 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20679639799999999 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534604+0000", 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8588999999999998e-05 2024-08-30T23:31:57.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534639+0000", 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5185000000000003e-05 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1930 2.7 2:ea8dc1bc:::benchmark_data_smithi173_47195_object1929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.73957822900002, 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20455900900000001, 2024-08-30T23:31:57.092 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1930 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330172+0000", 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330174+0000", 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1540000000000001e-06 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330209+0000", 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5206000000000001e-05 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330220+0000", 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1443999999999999e-05 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330225+0000", 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2299999999999999e-06 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330272+0000", 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6168000000000002e-05 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330386+0000", 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114523 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330484+0000", 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7739999999999996e-05 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.330815+0000", 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000331202 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.331211+0000", 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039613199999999999 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534670+0000", 2024-08-30T23:31:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203458423 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534698+0000", 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8432000000000002e-05 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.534731+0000", 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3355000000000002e-05 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1967 2.7 2:e55fccac:::benchmark_data_smithi173_47195_object1966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.721455048, 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18683412999999999, 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1967 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348296+0000", 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348297+0000", 2024-08-30T23:31:57.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.548e-06 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348327+0000", 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0312e-05 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348342+0000", 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4647000000000001e-05 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348365+0000", 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3098000000000001e-05 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348409+0000", 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4261000000000002e-05 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348496+0000", 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6846000000000002e-05 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348576+0000", 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9411000000000003e-05 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.348908+0000", 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033185099999999998 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.349356+0000", 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044820899999999998 2024-08-30T23:31:57.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535032+0000", 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18567658200000001 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535085+0000", 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2559000000000003e-05 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535130+0000", 2024-08-30T23:31:57.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4805999999999998e-05 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:1975 2.7 2:eeb7a2cc:::benchmark_data_smithi173_47195_object1974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.71650893600003, 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18200930000000001, 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1975 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353242+0000", 2024-08-30T23:31:57.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353243+0000", 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5899999999999995e-07 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353322+0000", 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9729999999999997e-05 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353335+0000", 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2719e-05 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353340+0000", 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353381+0000", 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0828000000000003e-05 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353459+0000", 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8151000000000004e-05 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353543+0000", 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3549999999999998e-05 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.353838+0000", 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029477600000000001 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.354159+0000", 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000321007 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535175+0000", 2024-08-30T23:31:57.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181016702 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535227+0000", 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1307000000000002e-05 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:25:59.535251+0000", 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4453000000000001e-05 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:36436 2.2 e116/97 2:4f13755f:::benchmark_data_smithi173_47195_object36435:head v 116'2235, mlcod=116'2231)", 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 332.086182339, 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23314933600000001, 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983568+0000", 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983569+0000", 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2799999999999995e-07 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983591+0000", 2024-08-30T23:31:57.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1899000000000002e-05 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215327+0000", 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23173639900000001 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215335+0000", 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.216e-06 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215397+0000", 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2348999999999995e-05 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.215421+0000", 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4114e-05 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216674+0000", 2024-08-30T23:31:57.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012534379999999999 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216718+0000", 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3192999999999998e-05 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:36447 2.2 e116/97 2:42c8382e:::benchmark_data_smithi173_47195_object36446:head v 116'2236, mlcod=116'2232)", 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 332.08113125599999, 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310671684, 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:31:57.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988619+0000", 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988620+0000", 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.099e-06 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988646+0000", 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5996999999999999e-05 2024-08-30T23:31:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297448+0000", 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30880194900000002 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297456+0000", 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5059999999999998e-06 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297540+0000", 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4086999999999996e-05 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.297565+0000", 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5304e-05 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299247+0000", 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016817690000000001 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299291+0000", 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3973000000000001e-05 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:40558 2.7 2:ec8f91a5:::benchmark_data_smithi173_47195_object40557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 329.13992345299999, 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18168003199999999, 2024-08-30T23:31:57.121 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 40558 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:31:57.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929827+0000", 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929828+0000", 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.23e-07 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929928+0000", 2024-08-30T23:31:57.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100546 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929938+0000", 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8260000000000005e-06 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.929945+0000", 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6669999999999999e-06 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930530+0000", 2024-08-30T23:31:57.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058469100000000003 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930611+0000", 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1532000000000005e-05 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.930702+0000", 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0371000000000006e-05 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:27.931513+0000", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081120499999999996 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111320+0000", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17980744600000001 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111443+0000", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123242 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111479+0000", 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5734e-05 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:28.111507+0000", 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8048999999999999e-05 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:50002 2.7 2:e84aa78e:::benchmark_data_smithi173_47195_object50001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 321.80131141800001, 2024-08-30T23:31:57.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.199583015, 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 50002 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:31:57.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268439+0000", 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268441+0000", 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9139999999999998e-06 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268476+0000", 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4665000000000001e-05 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268490+0000", 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4104000000000001e-05 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268498+0000", 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2619999999999998e-06 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268559+0000", 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0537e-05 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268646+0000", 2024-08-30T23:31:57.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7587e-05 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.268736+0000", 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9363999999999993e-05 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.269952+0000", 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012166290000000001 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.270470+0000", 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051778800000000004 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.467915+0000", 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.197445024 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.467979+0000", 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.41e-05 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.468022+0000", 2024-08-30T23:31:57.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3040999999999997e-05 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 146/138 [PushReplyOp(2:ee294341:::benchmark_data_smithi173_47195_object15777:head)])", 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 268.87871901099999, 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.001463438, 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.136 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191037+0000", 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2749999999999997e-06 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181108+0000", 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99007171599999999 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181115+0000", 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.189300+0000", 2024-08-30T23:31:57.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081848760000000007 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.192495+0000", 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031953250000000002 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 153/138 [PushReplyOp(2:eebfff31:::benchmark_data_smithi173_47195_object26886:head)])", 2024-08-30T23:31:57.140 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 252.14472009900001, 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21501724999999999, 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.09e-07 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925033+0000", 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.419e-06 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136148+0000", 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.211115204 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136156+0000", 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1880000000000002e-06 2024-08-30T23:31:57.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136222+0000", 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5320999999999993e-05 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.140048+0000", 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003826009 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:126363 2.7 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 230.34051420500001, 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221802834, 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.145 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 126363 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:31:57.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729238+0000", 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9360000000000002e-06 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729244+0000", 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8819999999999998e-06 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729258+0000", 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3994e-05 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729265+0000", 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0570000000000004e-06 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729318+0000", 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2410000000000001e-05 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729417+0000", 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9414999999999996e-05 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:31:57.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729508+0000", 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1209e-05 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.732169+0000", 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026607919999999999 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.737250+0000", 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050804370000000001 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949780+0000", 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21253043999999999 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949815+0000", 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5308000000000001e-05 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.951039+0000", 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001223954 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193992 2.7 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:31:57.152 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 134.29449987699999, 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60785984599999998, 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193992 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775252+0000", 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.528e-06 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775258+0000", 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.485e-06 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775273+0000", 2024-08-30T23:31:57.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5355000000000001e-05 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775281+0000", 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5140000000000004e-06 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775342+0000", 2024-08-30T23:31:57.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1334000000000006e-05 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775448+0000", 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010587 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775514+0000", 2024-08-30T23:31:57.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5747000000000006e-05 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.779105+0000", 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035911110000000001 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.780025+0000", 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092040100000000005 2024-08-30T23:31:57.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383045+0000", 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60302017900000005 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383083+0000", 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7670999999999999e-05 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383111+0000", 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7651e-05 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:2678 3.0 e213/206 3:0b08ec04:::benchmark_data_smithi173_60780_object2677:head v 213'154, mlcod=213'151)", 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.686750+0000", 2024-08-30T23:31:57.160 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 104.383000345, 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18360559000000001, 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686750+0000", 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686750+0000", 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686751+0000", 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9399999999999995e-07 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.686816+0000", 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4803999999999998e-05 2024-08-30T23:31:57.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869906+0000", 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18309029199999999 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869910+0000", 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4440000000000001e-06 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869960+0000", 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9372000000000001e-05 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.869986+0000", 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6452999999999999e-05 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.870330+0000", 2024-08-30T23:31:57.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034373599999999998 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.870356+0000", 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5995000000000001e-05 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:17991 3.a 3:55af03fe:::benchmark_data_smithi173_60780_object17990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:31:57.165 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.775010+0000", 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.294740907000005, 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18266866800000001, 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17991 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.166 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775010+0000", 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775010+0000", 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775012+0000", 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9259999999999999e-06 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775054+0000", 2024-08-30T23:31:57.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2778000000000003e-05 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775067+0000", 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.255e-05 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775074+0000", 2024-08-30T23:31:57.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4189999999999996e-06 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775141+0000", 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6618999999999994e-05 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775260+0000", 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011933 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.775400+0000", 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013916500000000001 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.776302+0000", 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090262000000000003 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.776808+0000", 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050635199999999995 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.957613+0000", 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18080433000000001 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.957648+0000", 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5076000000000001e-05 2024-08-30T23:31:57.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.957678+0000", 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0502999999999999e-05 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:22811 3.c e215/206 3:362f756c:::benchmark_data_smithi173_60780_object22810:head v 215'1421, mlcod=215'1418)", 2024-08-30T23:31:57.173 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 91.064362747000004, 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32690804400000001, 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005389+0000", 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005420+0000", 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1340000000000001e-05 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331407+0000", 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32598661899999998 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331414+0000", 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7409999999999992e-06 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331466+0000", 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1319000000000001e-05 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331490+0000", 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4544000000000001e-05 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332257+0000", 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076687200000000004 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332296+0000", 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8828000000000002e-05 2024-08-30T23:31:57.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:60801 3.c e236/206 3:371497b1:::benchmark_data_smithi173_60780_object60800:head v 236'3740, mlcod=236'3736)", 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:59.878008+0000", 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.191742539000003, 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.195066446, 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.179 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.878008+0000", 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.878008+0000", 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.878009+0000", 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7300000000000005e-07 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.878028+0000", 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8876999999999999e-05 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.071226+0000", 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19319872799999999 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.071232+0000", 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9789999999999998e-06 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.071286+0000", 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3857000000000003e-05 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.071307+0000", 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1049000000000001e-05 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.073020+0000", 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001713011 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.073075+0000", 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4172000000000002e-05 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:74488 3.3 e243/206 3:cadebabe:::benchmark_data_smithi173_60780_object74487:head v 243'4664, mlcod=243'4659)", 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:12.004998+0000", 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 45.064753033999999, 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19222183100000001, 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004998+0000", 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004998+0000", 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004999+0000", 2024-08-30T23:31:57.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.005023+0000", 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4723e-05 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.194556+0000", 2024-08-30T23:31:57.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.189532277 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.194560+0000", 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7539999999999997e-06 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.194607+0000", 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6684000000000002e-05 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.194622+0000", 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4815e-05 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197189+0000", 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002567288 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197219+0000", 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0150000000000001e-05 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:74497 3.3 e243/206 3:cf55ef95:::benchmark_data_smithi173_60780_object74496:head v 243'4665, mlcod=243'4660)", 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:12.015390+0000", 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 45.054360264000003, 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181956859, 2024-08-30T23:31:57.190 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.015390+0000", 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.015390+0000", 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:57.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.015391+0000", 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699999999999997e-07 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.015413+0000", 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1594e-05 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:57.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.194679+0000", 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17926646500000001 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.194683+0000", 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3450000000000002e-06 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.196173+0000", 2024-08-30T23:31:57.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001490131 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.196191+0000", 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8649000000000001e-05 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197324+0000", 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001132428 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:57.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197347+0000", 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3430000000000001e-05 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:57.195 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:31:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.197 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:57.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:57.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:57.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:57.372 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:31:57.373 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:31:57.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:31:57.446 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:31:57.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:31:57.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:31:57.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:57.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:57.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:57.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:57 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/301363643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:57 smithi106 ceph-mon[16603]: osdmap e262: 12 total, 12 up, 12 in 2024-08-30T23:31:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:57 smithi106 ceph-mon[16603]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:31:57.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:57 smithi106 ceph-mon[16603]: pgmap v645: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-08-30T23:31:57.872 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:57 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:57.872 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/301363643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:57.872 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:57 smithi136 ceph-mon[20207]: osdmap e262: 12 total, 12 up, 12 in 2024-08-30T23:31:57.872 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:57 smithi136 ceph-mon[20207]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:31:57.872 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:57 smithi136 ceph-mon[20207]: pgmap v645: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-08-30T23:31:57.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:31:57.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:57.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:57.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:58.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:31:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:57 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:31:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/301363643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:31:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:57 smithi173 ceph-mon[20554]: osdmap e262: 12 total, 12 up, 12 in 2024-08-30T23:31:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:57 smithi173 ceph-mon[20554]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:31:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:57 smithi173 ceph-mon[20554]: pgmap v645: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-08-30T23:31:58.089 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:31:58.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:58.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:58.317 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:58.319 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:31:58.344 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:31:58.350 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:58.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:31:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:58.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:31:58.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:58.599 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:58.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:58.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:58.776 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-08-30T23:31:56.593512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6007423544,"num_objects":91660,"num_object_clones":0,"num_object_copies":274980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91660,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":115369,"num_write_kb":6623015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":109655648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115338,"ondisk_log_size":115338,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19419936,"kb_used_data":17736508,"kb_used_omap":18,"kb_used_meta":1683117,"kb_avail":1105521888,"statfs":{"total":1151940427776,"available":1132054413312,"internally_reserved":0,"allocated":18162184192,"data_stored":18158682912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":1723512121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-327876608,"num_objects":-5003,"num_object_clones":0,"num_object_copies":-15009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5003,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.171541"},"pg_stats":[{"pgid":"3.f","version":"259'7080","reported_seq":7269,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-08-30T23:31:52.402545+0000","last_change":"2024-08-30T23:31:40.413353+0000","last_active":"2024-08-30T23:31:52.402545+0000","last_peered":"2024-08-30T23:31:52.402545+0000","last_clean":"2024-08-30T23:31:52.402545+0000","last_became_active":"2024-08-30T23:30:03.607819+0000","last_became_peered":"2024-08-30T23:30:03.607819+0000","last_unstale":"2024-08-30T23:31:52.402545+0000","last_undegraded":"2024-08-30T23:31:52.402545+0000","last_fullsized":"2024-08-30T23:31:52.402545+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6390","last_scrub_stamp":"2024-08-30T23:31:40.413273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:40.413273+0000","objects_scrubbed":6353,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:49.474478+0000","scrub_duration":1483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379584512,"num_objects":5792,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7080,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"261'7390","reported_seq":7582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.149675+0000","last_change":"2024-08-30T23:31:40.361324+0000","last_active":"2024-08-30T23:31:56.149675+0000","last_peered":"2024-08-30T23:31:56.149675+0000","last_clean":"2024-08-30T23:31:56.149675+0000","last_became_active":"2024-08-30T23:30:03.603411+0000","last_became_peered":"2024-08-30T23:30:03.603411+0000","last_unstale":"2024-08-30T23:31:56.149675+0000","last_undegraded":"2024-08-30T23:31:56.149675+0000","last_fullsized":"2024-08-30T23:31:56.149675+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6447","last_scrub_stamp":"2024-08-30T23:31:40.361199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:40.361199+0000","objects_scrubbed":6393,"log_size":7390,"log_dups_size":0,"ondisk_log_size":7390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.133871+0000","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7390,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"259'7102","reported_seq":7287,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-08-30T23:31:52.396161+0000","last_change":"2024-08-30T23:31:32.601330+0000","last_active":"2024-08-30T23:31:52.396161+0000","last_peered":"2024-08-30T23:31:52.396161+0000","last_clean":"2024-08-30T23:31:52.396161+0000","last_became_active":"2024-08-30T23:30:03.786437+0000","last_became_peered":"2024-08-30T23:30:03.786437+0000","last_unstale":"2024-08-30T23:31:52.396161+0000","last_undegraded":"2024-08-30T23:31:52.396161+0000","last_fullsized":"2024-08-30T23:31:52.396161+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5949","last_scrub_stamp":"2024-08-30T23:31:32.601199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:32.601199+0000","objects_scrubbed":5901,"log_size":7102,"log_dups_size":0,"ondisk_log_size":7102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:52.695898+0000","scrub_duration":1656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382795800,"num_objects":5842,"num_object_clones":0,"num_object_copies":17526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7102,"num_write_kb":414145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"260'7082","reported_seq":7253,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-08-30T23:31:53.654415+0000","last_change":"2024-08-30T23:31:09.240650+0000","last_active":"2024-08-30T23:31:53.654415+0000","last_peered":"2024-08-30T23:31:53.654415+0000","last_clean":"2024-08-30T23:31:53.654415+0000","last_became_active":"2024-08-30T23:30:03.607837+0000","last_became_peered":"2024-08-30T23:30:03.607837+0000","last_unstale":"2024-08-30T23:31:53.654415+0000","last_undegraded":"2024-08-30T23:31:53.654415+0000","last_fullsized":"2024-08-30T23:31:53.654415+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'4408","last_scrub_stamp":"2024-08-30T23:31:09.240557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:09.240557+0000","objects_scrubbed":4371,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:37.856644+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"260'7183","reported_seq":7373,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-08-30T23:31:53.626976+0000","last_change":"2024-08-30T23:31:39.673799+0000","last_active":"2024-08-30T23:31:53.626976+0000","last_peered":"2024-08-30T23:31:53.626976+0000","last_clean":"2024-08-30T23:31:53.626976+0000","last_became_active":"2024-08-30T23:30:03.603254+0000","last_became_peered":"2024-08-30T23:30:03.603254+0000","last_unstale":"2024-08-30T23:31:53.626976+0000","last_undegraded":"2024-08-30T23:31:53.626976+0000","last_fullsized":"2024-08-30T23:31:53.626976+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6408","last_scrub_stamp":"2024-08-30T23:31:39.673633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:39.673633+0000","objects_scrubbed":6378,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:51.600135+0000","scrub_duration":1145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379781120,"num_objects":5795,"num_object_clones":0,"num_object_copies":17385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7183,"num_write_kb":415296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"259'7005","reported_seq":7186,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-08-30T23:31:51.878597+0000","last_change":"2024-08-30T23:31:31.181603+0000","last_active":"2024-08-30T23:31:51.878597+0000","last_peered":"2024-08-30T23:31:51.878597+0000","last_clean":"2024-08-30T23:31:51.878597+0000","last_became_active":"2024-08-30T23:30:03.608966+0000","last_became_peered":"2024-08-30T23:30:03.608966+0000","last_unstale":"2024-08-30T23:31:51.878597+0000","last_undegraded":"2024-08-30T23:31:51.878597+0000","last_fullsized":"2024-08-30T23:31:51.878597+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5796","last_scrub_stamp":"2024-08-30T23:31:31.181453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:31.181453+0000","objects_scrubbed":5764,"log_size":7005,"log_dups_size":0,"ondisk_log_size":7005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.069588+0000","scrub_duration":1620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383582208,"num_objects":5853,"num_object_clones":0,"num_object_copies":17559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7005,"num_write_kb":411456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"261'7284","reported_seq":7472,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:55.688209+0000","last_change":"2024-08-30T23:31:31.714070+0000","last_active":"2024-08-30T23:31:55.688209+0000","last_peered":"2024-08-30T23:31:55.688209+0000","last_clean":"2024-08-30T23:31:55.688209+0000","last_became_active":"2024-08-30T23:30:03.607776+0000","last_became_peered":"2024-08-30T23:30:03.607776+0000","last_unstale":"2024-08-30T23:31:55.688209+0000","last_undegraded":"2024-08-30T23:31:55.688209+0000","last_fullsized":"2024-08-30T23:31:55.688209+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5882","last_scrub_stamp":"2024-08-30T23:31:31.713921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:31.713921+0000","objects_scrubbed":5840,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:38.625685+0000","scrub_duration":1438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370802688,"num_objects":5658,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":414144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"261'7309","reported_seq":7504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.159047+0000","last_change":"2024-08-30T23:31:33.747342+0000","last_active":"2024-08-30T23:31:56.159047+0000","last_peered":"2024-08-30T23:31:56.159047+0000","last_clean":"2024-08-30T23:31:56.159047+0000","last_became_active":"2024-08-30T23:30:03.603520+0000","last_became_peered":"2024-08-30T23:30:03.603520+0000","last_unstale":"2024-08-30T23:31:56.159047+0000","last_undegraded":"2024-08-30T23:31:56.159047+0000","last_fullsized":"2024-08-30T23:31:56.159047+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6008","last_scrub_stamp":"2024-08-30T23:31:33.747232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:33.747232+0000","objects_scrubbed":5972,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:44.820492+0000","scrub_duration":1130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368377856,"num_objects":5621,"num_object_clones":0,"num_object_copies":16863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":413760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"260'7273","reported_seq":7465,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-08-30T23:31:53.583955+0000","last_change":"2024-08-30T23:31:40.505498+0000","last_active":"2024-08-30T23:31:53.583955+0000","last_peered":"2024-08-30T23:31:53.583955+0000","last_clean":"2024-08-30T23:31:53.583955+0000","last_became_active":"2024-08-30T23:30:03.607480+0000","last_became_peered":"2024-08-30T23:30:03.607480+0000","last_unstale":"2024-08-30T23:31:53.583955+0000","last_undegraded":"2024-08-30T23:31:53.583955+0000","last_fullsized":"2024-08-30T23:31:53.583955+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6492","last_scrub_stamp":"2024-08-30T23:31:40.505434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:40.505434+0000","objects_scrubbed":6416,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:46.966295+0000","scrub_duration":2958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380305408,"num_objects":5803,"num_object_clones":0,"num_object_copies":17409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7273,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"259'7173","reported_seq":7359,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-08-30T23:31:53.461436+0000","last_change":"2024-08-30T23:31:34.440798+0000","last_active":"2024-08-30T23:31:53.461436+0000","last_peered":"2024-08-30T23:31:53.461436+0000","last_clean":"2024-08-30T23:31:53.461436+0000","last_became_active":"2024-08-30T23:30:03.602886+0000","last_became_peered":"2024-08-30T23:30:03.602886+0000","last_unstale":"2024-08-30T23:31:53.461436+0000","last_undegraded":"2024-08-30T23:31:53.461436+0000","last_fullsized":"2024-08-30T23:31:53.461436+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6085","last_scrub_stamp":"2024-08-30T23:31:34.440629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:34.440629+0000","objects_scrubbed":6053,"log_size":7173,"log_dups_size":0,"ondisk_log_size":7173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:52.628384+0000","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380043264,"num_objects":5799,"num_object_clones":0,"num_object_copies":17397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":415104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"260'7153","reported_seq":7342,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-08-30T23:31:53.626248+0000","last_change":"2024-08-30T23:31:35.600556+0000","last_active":"2024-08-30T23:31:53.626248+0000","last_peered":"2024-08-30T23:31:53.626248+0000","last_clean":"2024-08-30T23:31:53.626248+0000","last_became_active":"2024-08-30T23:30:03.607817+0000","last_became_peered":"2024-08-30T23:30:03.607817+0000","last_unstale":"2024-08-30T23:31:53.626248+0000","last_undegraded":"2024-08-30T23:31:53.626248+0000","last_fullsized":"2024-08-30T23:31:53.626248+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6088","last_scrub_stamp":"2024-08-30T23:31:35.600439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:35.600439+0000","objects_scrubbed":6034,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:59.401775+0000","scrub_duration":2088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374013952,"num_objects":5707,"num_object_clones":0,"num_object_copies":17121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7153,"num_write_kb":411520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"261'7259","reported_seq":7451,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.168103+0000","last_change":"2024-08-30T23:31:36.313489+0000","last_active":"2024-08-30T23:31:56.168103+0000","last_peered":"2024-08-30T23:31:56.168103+0000","last_clean":"2024-08-30T23:31:56.168103+0000","last_became_active":"2024-08-30T23:30:03.606790+0000","last_became_peered":"2024-08-30T23:30:03.606790+0000","last_unstale":"2024-08-30T23:31:56.168103+0000","last_undegraded":"2024-08-30T23:31:56.168103+0000","last_fullsized":"2024-08-30T23:31:56.168103+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6149","last_scrub_stamp":"2024-08-30T23:31:36.313244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:36.313244+0000","objects_scrubbed":6083,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.932289+0000","scrub_duration":1728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368640000,"num_objects":5625,"num_object_clones":0,"num_object_copies":16875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":412288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"260'7241","reported_seq":7429,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:54.564282+0000","last_change":"2024-08-30T23:31:32.280147+0000","last_active":"2024-08-30T23:31:54.564282+0000","last_peered":"2024-08-30T23:31:54.564282+0000","last_clean":"2024-08-30T23:31:54.564282+0000","last_became_active":"2024-08-30T23:30:03.607728+0000","last_became_peered":"2024-08-30T23:30:03.607728+0000","last_unstale":"2024-08-30T23:31:54.564282+0000","last_undegraded":"2024-08-30T23:31:54.564282+0000","last_fullsized":"2024-08-30T23:31:54.564282+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5952","last_scrub_stamp":"2024-08-30T23:31:32.280051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:32.280051+0000","objects_scrubbed":5919,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:55.889484+0000","scrub_duration":1501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376504320,"num_objects":5745,"num_object_clones":0,"num_object_copies":17235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":415552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"260'7217","reported_seq":10722,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:54.561907+0000","last_change":"2024-08-30T23:31:36.869408+0000","last_active":"2024-08-30T23:31:54.561907+0000","last_peered":"2024-08-30T23:31:54.561907+0000","last_clean":"2024-08-30T23:31:54.561907+0000","last_became_active":"2024-08-30T23:30:50.207891+0000","last_became_peered":"2024-08-30T23:30:50.207891+0000","last_unstale":"2024-08-30T23:31:54.561907+0000","last_undegraded":"2024-08-30T23:31:54.561907+0000","last_fullsized":"2024-08-30T23:31:54.561907+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'6226","last_scrub_stamp":"2024-08-30T23:31:36.869350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:36.869350+0000","objects_scrubbed":6156,"log_size":7217,"log_dups_size":0,"ondisk_log_size":7217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:33:04.285914+0000","scrub_duration":2151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372441088,"num_objects":5683,"num_object_clones":0,"num_object_copies":17049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":108068864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,10],"acting":[4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"261'7393","reported_seq":7564,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.141712+0000","last_change":"2024-08-30T23:31:06.885757+0000","last_active":"2024-08-30T23:31:56.141712+0000","last_peered":"2024-08-30T23:31:56.141712+0000","last_clean":"2024-08-30T23:31:56.141712+0000","last_became_active":"2024-08-30T23:30:03.786205+0000","last_became_peered":"2024-08-30T23:30:03.786205+0000","last_unstale":"2024-08-30T23:31:56.141712+0000","last_undegraded":"2024-08-30T23:31:56.141712+0000","last_fullsized":"2024-08-30T23:31:56.141712+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'4301","last_scrub_stamp":"2024-08-30T23:31:06.885679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:06.885679+0000","objects_scrubbed":4258,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:30.208126+0000","scrub_duration":1053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371130368,"num_objects":5663,"num_object_clones":0,"num_object_copies":16989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7393,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"260'7102","reported_seq":7290,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-08-30T23:31:53.603593+0000","last_change":"2024-08-30T23:31:33.556142+0000","last_active":"2024-08-30T23:31:53.603593+0000","last_peered":"2024-08-30T23:31:53.603593+0000","last_clean":"2024-08-30T23:31:53.603593+0000","last_became_active":"2024-08-30T23:30:03.786203+0000","last_became_peered":"2024-08-30T23:30:03.786203+0000","last_unstale":"2024-08-30T23:31:53.603593+0000","last_undegraded":"2024-08-30T23:31:53.603593+0000","last_fullsized":"2024-08-30T23:31:53.603593+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5991","last_scrub_stamp":"2024-08-30T23:31:33.556078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:33.556078+0000","objects_scrubbed":5908,"log_size":7102,"log_dups_size":0,"ondisk_log_size":7102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:50.553291+0000","scrub_duration":2048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378011648,"num_objects":5768,"num_object_clones":0,"num_object_copies":17304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7102,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":494,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-08-30T23:31:51.491283+0000","last_change":"2024-08-30T23:31:32.963315+0000","last_active":"2024-08-30T23:31:51.491283+0000","last_peered":"2024-08-30T23:31:51.491283+0000","last_clean":"2024-08-30T23:31:51.491283+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:31:51.491283+0000","last_undegraded":"2024-08-30T23:31:51.491283+0000","last_fullsized":"2024-08-30T23:31:51.491283+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:31:32.963245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:31:32.963245+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.395422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6006833176,"num_objects":91658,"num_object_clones":0,"num_object_copies":274974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91658,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115246,"num_write_kb":6620865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":108068864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18071105536,"data_stored":18071093320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115246,"ondisk_log_size":115246,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2724376,"kb_used_data":2540424,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91020776,"statfs":{"total":95995035648,"available":93205274624,"internally_reserved":0,"allocated":2601394176,"data_stored":2601095521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":188283332},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449184,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415124,"kb_used_data":361728,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93330028,"statfs":{"total":95995035648,"available":95569948672,"internally_reserved":0,"allocated":370409472,"data_stored":370114389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232084,"kb_used_data":1096772,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92513068,"statfs":{"total":95995035648,"available":94733381632,"internally_reserved":0,"allocated":1123094528,"data_stored":1122795373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855956,"kb_used_data":726208,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92889196,"statfs":{"total":95995035648,"available":95118536704,"internally_reserved":0,"allocated":743636992,"data_stored":743341909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234091,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1653000,"kb_used_data":1505204,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92092152,"statfs":{"total":95995035648,"available":94302363648,"internally_reserved":0,"allocated":1541328896,"data_stored":1541035127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528433,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1986184,"kb_used_data":1878640,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91758968,"statfs":{"total":95995035648,"available":93961183232,"internally_reserved":0,"allocated":1923727360,"data_stored":1923437663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724661,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1600268,"kb_used_data":1470520,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92144884,"statfs":{"total":95995035648,"available":94356361216,"internally_reserved":0,"allocated":1505812480,"data_stored":1505517815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953595,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2723720,"kb_used_data":2588404,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91021432,"statfs":{"total":95995035648,"available":93205946368,"internally_reserved":0,"allocated":2650525696,"data_stored":2650235103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2041660,"kb_used_data":1843880,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91703492,"statfs":{"total":95995035648,"available":93904375808,"internally_reserved":0,"allocated":1888133120,"data_stored":1887846975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":202504647},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411461,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":489592,"kb_used_data":368168,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93255560,"statfs":{"total":95995035648,"available":95493693440,"internally_reserved":0,"allocated":377004032,"data_stored":376717887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1685244,"kb_used_data":1501352,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92059908,"statfs":{"total":95995035648,"available":94269345792,"internally_reserved":0,"allocated":1537384448,"data_stored":1537098303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2012728,"kb_used_data":1855208,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91732424,"statfs":{"total":95995035648,"available":93934002176,"internally_reserved":0,"allocated":1899732992,"data_stored":1899446847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1897332736,"data_stored":1897332736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1526005760,"data_stored":1526005760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374276096,"data_stored":374276096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1884225536,"data_stored":1884225536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2623733760,"data_stored":2623733760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1503006720,"data_stored":1503002648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1920663552,"data_stored":1920663552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1531383808,"data_stored":1531379736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":738525184,"data_stored":738525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1121456128,"data_stored":1121452056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":367460352,"data_stored":367460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2583035904,"data_stored":2583035904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:31:58.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:119547 3.f 3:f14ea2de:::benchmark_data_smithi173_60780_object16093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:58.739404+0000", 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.092943837000000001, 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.092964924000000004, 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 119547 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739404+0000", 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739404+0000", 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739406+0000", 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2390000000000001e-06 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739415+0000", 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9269999999999994e-06 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739435+0000", 2024-08-30T23:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9942e-05 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739443+0000", 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4580000000000004e-06 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739503+0000", 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0198999999999997e-05 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739631+0000", 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000128665 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.739759+0000", 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012795699999999999 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.742844+0000", 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003084388 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:58.752890+0000", 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010046169000000001 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:31:58.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:31:58.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:31:58.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:31:58.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:58.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:31:59.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:31:59.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:31:59.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:59.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:59.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:59.342 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:31:59.348 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:31:59.348 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:31:59.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:31:59.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:31:59.382 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:31:59.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:31:59.407 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:31:59.407 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:31:59.433 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:31:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3039794284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:59.439 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:31:59.439 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:31:59.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:31:59.469 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:31:59.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:31:59.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:31:59.500 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:31:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:31:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:31:59.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:31:59.523 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:31:59.524 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:31:59.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:31:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3039794284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.627 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:59.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:59.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:31:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:31:59.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:31:59.815 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:31:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:31:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3039794284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:31:59.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:31:59.884 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:31:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:31:59.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:31:59.986 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:00.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:00.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:32:00.099 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:32:00.099 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:32:00.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:00.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:00.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:00.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:00.353 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:00.353 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:00.372 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:00.372 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:00.375 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:00.375 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:00.396 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:00.396 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:00.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:00.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:00.418 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:00.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:00.443 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:00.443 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:00.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:00 smithi136 ceph-mon[20207]: pgmap v646: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:00.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:00.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:00 smithi106 ceph-mon[16603]: pgmap v646: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:00.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:00.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:00.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:00.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:00 smithi173 ceph-mon[20554]: pgmap v646: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:00.885 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2024-08-30T23:31:59.423354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5837226552,"num_objects":89063,"num_object_clones":0,"num_object_copies":267189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89063,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":117966,"num_write_kb":6623015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":109655648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117935,"ondisk_log_size":117935,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18867296,"kb_used_data":17212412,"kb_used_omap":18,"kb_used_meta":1653997,"kb_avail":1106074528,"statfs":{"total":1151940427776,"available":1132620316672,"internally_reserved":0,"allocated":17625509888,"data_stored":17622292461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":1693693241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-434765824,"num_objects":-6634,"num_object_clones":0,"num_object_copies":-19902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6634,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001065"},"pg_stats":[{"pgid":"3.f","version":"262'7361","reported_seq":7556,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:57.409255+0000","last_change":"2024-08-30T23:31:40.413353+0000","last_active":"2024-08-30T23:31:57.409255+0000","last_peered":"2024-08-30T23:31:57.409255+0000","last_clean":"2024-08-30T23:31:57.409255+0000","last_became_active":"2024-08-30T23:30:03.607819+0000","last_became_peered":"2024-08-30T23:30:03.607819+0000","last_unstale":"2024-08-30T23:31:57.409255+0000","last_undegraded":"2024-08-30T23:31:57.409255+0000","last_fullsized":"2024-08-30T23:31:57.409255+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6390","last_scrub_stamp":"2024-08-30T23:31:40.413273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:40.413273+0000","objects_scrubbed":6353,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:49.474478+0000","scrub_duration":1483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361168896,"num_objects":5511,"num_object_clones":0,"num_object_copies":16533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"261'7390","reported_seq":7582,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.149675+0000","last_change":"2024-08-30T23:31:40.361324+0000","last_active":"2024-08-30T23:31:56.149675+0000","last_peered":"2024-08-30T23:31:56.149675+0000","last_clean":"2024-08-30T23:31:56.149675+0000","last_became_active":"2024-08-30T23:30:03.603411+0000","last_became_peered":"2024-08-30T23:30:03.603411+0000","last_unstale":"2024-08-30T23:31:56.149675+0000","last_undegraded":"2024-08-30T23:31:56.149675+0000","last_fullsized":"2024-08-30T23:31:56.149675+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6447","last_scrub_stamp":"2024-08-30T23:31:40.361199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:40.361199+0000","objects_scrubbed":6393,"log_size":7390,"log_dups_size":0,"ondisk_log_size":7390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.133871+0000","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7390,"num_write_kb":416576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"262'7376","reported_seq":7567,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:57.378157+0000","last_change":"2024-08-30T23:31:32.601330+0000","last_active":"2024-08-30T23:31:57.378157+0000","last_peered":"2024-08-30T23:31:57.378157+0000","last_clean":"2024-08-30T23:31:57.378157+0000","last_became_active":"2024-08-30T23:30:03.786437+0000","last_became_peered":"2024-08-30T23:30:03.786437+0000","last_unstale":"2024-08-30T23:31:57.378157+0000","last_undegraded":"2024-08-30T23:31:57.378157+0000","last_fullsized":"2024-08-30T23:31:57.378157+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5949","last_scrub_stamp":"2024-08-30T23:31:32.601199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:32.601199+0000","objects_scrubbed":5901,"log_size":7376,"log_dups_size":0,"ondisk_log_size":7376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:52.695898+0000","scrub_duration":1656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364838936,"num_objects":5568,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5568,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7376,"num_write_kb":414145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"262'7372","reported_seq":7548,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:58.638281+0000","last_change":"2024-08-30T23:31:09.240650+0000","last_active":"2024-08-30T23:31:58.638281+0000","last_peered":"2024-08-30T23:31:58.638281+0000","last_clean":"2024-08-30T23:31:58.638281+0000","last_became_active":"2024-08-30T23:30:03.607837+0000","last_became_peered":"2024-08-30T23:30:03.607837+0000","last_unstale":"2024-08-30T23:31:58.638281+0000","last_undegraded":"2024-08-30T23:31:58.638281+0000","last_fullsized":"2024-08-30T23:31:58.638281+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'4408","last_scrub_stamp":"2024-08-30T23:31:09.240557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:09.240557+0000","objects_scrubbed":4371,"log_size":7372,"log_dups_size":0,"ondisk_log_size":7372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:37.856644+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352976896,"num_objects":5386,"num_object_clones":0,"num_object_copies":16158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7372,"num_write_kb":408256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"262'7494","reported_seq":7689,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:58.599051+0000","last_change":"2024-08-30T23:31:39.673799+0000","last_active":"2024-08-30T23:31:58.599051+0000","last_peered":"2024-08-30T23:31:58.599051+0000","last_clean":"2024-08-30T23:31:58.599051+0000","last_became_active":"2024-08-30T23:30:03.603254+0000","last_became_peered":"2024-08-30T23:30:03.603254+0000","last_unstale":"2024-08-30T23:31:58.599051+0000","last_undegraded":"2024-08-30T23:31:58.599051+0000","last_fullsized":"2024-08-30T23:31:58.599051+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6408","last_scrub_stamp":"2024-08-30T23:31:39.673633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:39.673633+0000","objects_scrubbed":6378,"log_size":7494,"log_dups_size":0,"ondisk_log_size":7494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:51.600135+0000","scrub_duration":1145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359399424,"num_objects":5484,"num_object_clones":0,"num_object_copies":16452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":415296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"262'7267","reported_seq":7455,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.873905+0000","last_change":"2024-08-30T23:31:31.181603+0000","last_active":"2024-08-30T23:31:56.873905+0000","last_peered":"2024-08-30T23:31:56.873905+0000","last_clean":"2024-08-30T23:31:56.873905+0000","last_became_active":"2024-08-30T23:30:03.608966+0000","last_became_peered":"2024-08-30T23:30:03.608966+0000","last_unstale":"2024-08-30T23:31:56.873905+0000","last_undegraded":"2024-08-30T23:31:56.873905+0000","last_fullsized":"2024-08-30T23:31:56.873905+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5796","last_scrub_stamp":"2024-08-30T23:31:31.181453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:31.181453+0000","objects_scrubbed":5764,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.069588+0000","scrub_duration":1620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366411776,"num_objects":5591,"num_object_clones":0,"num_object_copies":16773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7267,"num_write_kb":411456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"261'7284","reported_seq":7472,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:55.688209+0000","last_change":"2024-08-30T23:31:31.714070+0000","last_active":"2024-08-30T23:31:55.688209+0000","last_peered":"2024-08-30T23:31:55.688209+0000","last_clean":"2024-08-30T23:31:55.688209+0000","last_became_active":"2024-08-30T23:30:03.607776+0000","last_became_peered":"2024-08-30T23:30:03.607776+0000","last_unstale":"2024-08-30T23:31:55.688209+0000","last_undegraded":"2024-08-30T23:31:55.688209+0000","last_fullsized":"2024-08-30T23:31:55.688209+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5882","last_scrub_stamp":"2024-08-30T23:31:31.713921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:31.713921+0000","objects_scrubbed":5840,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:38.625685+0000","scrub_duration":1438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370802688,"num_objects":5658,"num_object_clones":0,"num_object_copies":16974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7284,"num_write_kb":414144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.8","version":"261'7309","reported_seq":7504,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.159047+0000","last_change":"2024-08-30T23:31:33.747342+0000","last_active":"2024-08-30T23:31:56.159047+0000","last_peered":"2024-08-30T23:31:56.159047+0000","last_clean":"2024-08-30T23:31:56.159047+0000","last_became_active":"2024-08-30T23:30:03.603520+0000","last_became_peered":"2024-08-30T23:30:03.603520+0000","last_unstale":"2024-08-30T23:31:56.159047+0000","last_undegraded":"2024-08-30T23:31:56.159047+0000","last_fullsized":"2024-08-30T23:31:56.159047+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6008","last_scrub_stamp":"2024-08-30T23:31:33.747232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:33.747232+0000","objects_scrubbed":5972,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:44.820492+0000","scrub_duration":1130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368377856,"num_objects":5621,"num_object_clones":0,"num_object_copies":16863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7309,"num_write_kb":413760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"262'7545","reported_seq":7740,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:58.613944+0000","last_change":"2024-08-30T23:31:40.505498+0000","last_active":"2024-08-30T23:31:58.613944+0000","last_peered":"2024-08-30T23:31:58.613944+0000","last_clean":"2024-08-30T23:31:58.613944+0000","last_became_active":"2024-08-30T23:30:03.607480+0000","last_became_peered":"2024-08-30T23:30:03.607480+0000","last_unstale":"2024-08-30T23:31:58.613944+0000","last_undegraded":"2024-08-30T23:31:58.613944+0000","last_fullsized":"2024-08-30T23:31:58.613944+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'6492","last_scrub_stamp":"2024-08-30T23:31:40.505434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:40.505434+0000","objects_scrubbed":6416,"log_size":7545,"log_dups_size":0,"ondisk_log_size":7545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:46.966295+0000","scrub_duration":2958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362479616,"num_objects":5531,"num_object_clones":0,"num_object_copies":16593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7545,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"262'7465","reported_seq":7656,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:58.459364+0000","last_change":"2024-08-30T23:31:34.440798+0000","last_active":"2024-08-30T23:31:58.459364+0000","last_peered":"2024-08-30T23:31:58.459364+0000","last_clean":"2024-08-30T23:31:58.459364+0000","last_became_active":"2024-08-30T23:30:03.602886+0000","last_became_peered":"2024-08-30T23:30:03.602886+0000","last_unstale":"2024-08-30T23:31:58.459364+0000","last_undegraded":"2024-08-30T23:31:58.459364+0000","last_fullsized":"2024-08-30T23:31:58.459364+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6085","last_scrub_stamp":"2024-08-30T23:31:34.440629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:34.440629+0000","objects_scrubbed":6053,"log_size":7465,"log_dups_size":0,"ondisk_log_size":7465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:52.628384+0000","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360906752,"num_objects":5507,"num_object_clones":0,"num_object_copies":16521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7465,"num_write_kb":415104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"262'7453","reported_seq":7646,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:58.630542+0000","last_change":"2024-08-30T23:31:35.600556+0000","last_active":"2024-08-30T23:31:58.630542+0000","last_peered":"2024-08-30T23:31:58.630542+0000","last_clean":"2024-08-30T23:31:58.630542+0000","last_became_active":"2024-08-30T23:30:03.607817+0000","last_became_peered":"2024-08-30T23:30:03.607817+0000","last_unstale":"2024-08-30T23:31:58.630542+0000","last_undegraded":"2024-08-30T23:31:58.630542+0000","last_fullsized":"2024-08-30T23:31:58.630542+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6088","last_scrub_stamp":"2024-08-30T23:31:35.600439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:35.600439+0000","objects_scrubbed":6034,"log_size":7453,"log_dups_size":0,"ondisk_log_size":7453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:59.401775+0000","scrub_duration":2088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354353152,"num_objects":5407,"num_object_clones":0,"num_object_copies":16221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":411520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"261'7259","reported_seq":7451,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.168103+0000","last_change":"2024-08-30T23:31:36.313489+0000","last_active":"2024-08-30T23:31:56.168103+0000","last_peered":"2024-08-30T23:31:56.168103+0000","last_clean":"2024-08-30T23:31:56.168103+0000","last_became_active":"2024-08-30T23:30:03.606790+0000","last_became_peered":"2024-08-30T23:30:03.606790+0000","last_unstale":"2024-08-30T23:31:56.168103+0000","last_undegraded":"2024-08-30T23:31:56.168103+0000","last_fullsized":"2024-08-30T23:31:56.168103+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'6149","last_scrub_stamp":"2024-08-30T23:31:36.313244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:36.313244+0000","objects_scrubbed":6083,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.932289+0000","scrub_duration":1728,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368640000,"num_objects":5625,"num_object_clones":0,"num_object_copies":16875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":412288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"260'7241","reported_seq":7429,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:54.564282+0000","last_change":"2024-08-30T23:31:32.280147+0000","last_active":"2024-08-30T23:31:54.564282+0000","last_peered":"2024-08-30T23:31:54.564282+0000","last_clean":"2024-08-30T23:31:54.564282+0000","last_became_active":"2024-08-30T23:30:03.607728+0000","last_became_peered":"2024-08-30T23:30:03.607728+0000","last_unstale":"2024-08-30T23:31:54.564282+0000","last_undegraded":"2024-08-30T23:31:54.564282+0000","last_fullsized":"2024-08-30T23:31:54.564282+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5952","last_scrub_stamp":"2024-08-30T23:31:32.280051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:32.280051+0000","objects_scrubbed":5919,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:55.889484+0000","scrub_duration":1501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376504320,"num_objects":5745,"num_object_clones":0,"num_object_copies":17235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":415552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1","version":"260'7217","reported_seq":10722,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:54.561907+0000","last_change":"2024-08-30T23:31:36.869408+0000","last_active":"2024-08-30T23:31:54.561907+0000","last_peered":"2024-08-30T23:31:54.561907+0000","last_clean":"2024-08-30T23:31:54.561907+0000","last_became_active":"2024-08-30T23:30:50.207891+0000","last_became_peered":"2024-08-30T23:30:50.207891+0000","last_unstale":"2024-08-30T23:31:54.561907+0000","last_undegraded":"2024-08-30T23:31:54.561907+0000","last_fullsized":"2024-08-30T23:31:54.561907+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'6226","last_scrub_stamp":"2024-08-30T23:31:36.869350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:36.869350+0000","objects_scrubbed":6156,"log_size":7217,"log_dups_size":0,"ondisk_log_size":7217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:33:04.285914+0000","scrub_duration":2151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372441088,"num_objects":5683,"num_object_clones":0,"num_object_copies":17049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7217,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":108068864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,10],"acting":[4,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"261'7393","reported_seq":7564,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.141712+0000","last_change":"2024-08-30T23:31:06.885757+0000","last_active":"2024-08-30T23:31:56.141712+0000","last_peered":"2024-08-30T23:31:56.141712+0000","last_clean":"2024-08-30T23:31:56.141712+0000","last_became_active":"2024-08-30T23:30:03.786205+0000","last_became_peered":"2024-08-30T23:30:03.786205+0000","last_unstale":"2024-08-30T23:31:56.141712+0000","last_undegraded":"2024-08-30T23:31:56.141712+0000","last_fullsized":"2024-08-30T23:31:56.141712+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'4301","last_scrub_stamp":"2024-08-30T23:31:06.885679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:06.885679+0000","objects_scrubbed":4258,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:30.208126+0000","scrub_duration":1053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371130368,"num_objects":5663,"num_object_clones":0,"num_object_copies":16989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7393,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"262'7417","reported_seq":7609,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:58.618500+0000","last_change":"2024-08-30T23:31:33.556142+0000","last_active":"2024-08-30T23:31:58.618500+0000","last_peered":"2024-08-30T23:31:58.618500+0000","last_clean":"2024-08-30T23:31:58.618500+0000","last_became_active":"2024-08-30T23:30:03.786203+0000","last_became_peered":"2024-08-30T23:30:03.786203+0000","last_unstale":"2024-08-30T23:31:58.618500+0000","last_undegraded":"2024-08-30T23:31:58.618500+0000","last_fullsized":"2024-08-30T23:31:58.618500+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'5991","last_scrub_stamp":"2024-08-30T23:31:33.556078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:30:02.569516+0000","last_clean_scrub_stamp":"2024-08-30T23:31:33.556078+0000","objects_scrubbed":5908,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:50.553291+0000","scrub_duration":2048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357367808,"num_objects":5453,"num_object_clones":0,"num_object_copies":16359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7417,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":500,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-08-30T23:31:56.596083+0000","last_change":"2024-08-30T23:31:32.963315+0000","last_active":"2024-08-30T23:31:56.596083+0000","last_peered":"2024-08-30T23:31:56.596083+0000","last_clean":"2024-08-30T23:31:56.596083+0000","last_became_active":"2024-08-30T23:24:47.869550+0000","last_became_peered":"2024-08-30T23:24:47.869550+0000","last_unstale":"2024-08-30T23:31:56.596083+0000","last_undegraded":"2024-08-30T23:31:56.596083+0000","last_fullsized":"2024-08-30T23:31:56.596083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:31:32.963245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:31:32.963245+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:32:57.395422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5836636184,"num_objects":89061,"num_object_clones":0,"num_object_copies":267183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117843,"num_write_kb":6620865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":108068864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17532596224,"data_stored":17532584008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117843,"ondisk_log_size":117843,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2724376,"kb_used_data":2540424,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91020776,"statfs":{"total":95995035648,"available":93205274624,"internally_reserved":0,"allocated":2601394176,"data_stored":2601095521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":188283332},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449184,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415124,"kb_used_data":361728,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93330028,"statfs":{"total":95995035648,"available":95569948672,"internally_reserved":0,"allocated":370409472,"data_stored":370114389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1232084,"kb_used_data":1096772,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92513068,"statfs":{"total":95995035648,"available":94733381632,"internally_reserved":0,"allocated":1123094528,"data_stored":1122795373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":138541504},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855956,"kb_used_data":726208,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92889196,"statfs":{"total":95995035648,"available":95118536704,"internally_reserved":0,"allocated":743636992,"data_stored":743341909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234092,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564656,"kb_used_data":1429192,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92180496,"statfs":{"total":95995035648,"available":94392827904,"internally_reserved":0,"allocated":1463492608,"data_stored":1463238754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":138541514},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528434,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1888360,"kb_used_data":1788996,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91856792,"statfs":{"total":95995035648,"available":94061355008,"internally_reserved":0,"allocated":1831931904,"data_stored":1831682122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724662,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1517476,"kb_used_data":1395904,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92227676,"statfs":{"total":95995035648,"available":94441140224,"internally_reserved":0,"allocated":1429405696,"data_stored":1429155394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953595,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2723720,"kb_used_data":2588404,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91021432,"statfs":{"total":95995035648,"available":93205946368,"internally_reserved":0,"allocated":2650525696,"data_stored":2650235103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1953292,"kb_used_data":1747196,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":91791860,"statfs":{"total":95995035648,"available":93994864640,"internally_reserved":0,"allocated":1789128704,"data_stored":1788882474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":211024327},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":470412,"kb_used_data":348988,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93274740,"statfs":{"total":95995035648,"available":95513333760,"internally_reserved":0,"allocated":357363712,"data_stored":357117482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601444,"kb_used_data":1429884,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92143708,"statfs":{"total":95995035648,"available":94355156992,"internally_reserved":0,"allocated":1464201216,"data_stored":1463954986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1920396,"kb_used_data":1758716,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91824756,"statfs":{"total":95995035648,"available":94028550144,"internally_reserved":0,"allocated":1800925184,"data_stored":1800678954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1798766592,"data_stored":1798766592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1452474368,"data_stored":1452474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354549760,"data_stored":354549760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1786642432,"data_stored":1786642432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2623733760,"data_stored":2623733760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1423380480,"data_stored":1423376408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1826422784,"data_stored":1826422784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1456148480,"data_stored":1456144408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":738525184,"data_stored":738525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1121456128,"data_stored":1121452056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":367460352,"data_stored":367460352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2583035904,"data_stored":2583035904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:32:00.886 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-30T23:32:00.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:32:00.887 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:32:00.887 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 0.968221 2024-08-30T23:32:00.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 8 0.9682206909884691 2024-08-30T23:32:00.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:00.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:00.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:01.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:01.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:01.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:01.159 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:01.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:01.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:01.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:01.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:01.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:01.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:01.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:01.589 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:01.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:01 smithi106 ceph-mon[16603]: from='client.25309 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:32:01.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:01.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:01.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:01 smithi106 ceph-mon[16603]: osdmap e263: 12 total, 12 up, 12 in 2024-08-30T23:32:01.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:32:01.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:01 smithi136 ceph-mon[20207]: from='client.25309 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:32:01.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:01.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:01.647 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:01 smithi136 ceph-mon[20207]: osdmap e263: 12 total, 12 up, 12 in 2024-08-30T23:32:01.687 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:01.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.722 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:01.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:01.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:01 smithi173 ceph-mon[20554]: from='client.25309 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:32:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:01 smithi173 ceph-mon[20554]: osdmap e263: 12 total, 12 up, 12 in 2024-08-30T23:32:01.852 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:01.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:01.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:01.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:01.991 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:32:02.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:02.106 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:02.209 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:02.234 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:02.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:02.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:02.377 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:02 smithi106 ceph-mon[16603]: pgmap v648: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s 2024-08-30T23:32:02.377 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2164939433' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:02.377 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:02 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:02.381 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:32:02.456 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:02 smithi136 ceph-mon[20207]: pgmap v648: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s 2024-08-30T23:32:02.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2164939433' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:02.457 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:02 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:02.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:02.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:02.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:02.614 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:02.707 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:02.707 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:02.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:02.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:02.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:02 smithi173 ceph-mon[20554]: pgmap v648: 17 pgs: 17 active+clean; 5.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s 2024-08-30T23:32:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2164939433' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:02 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:02.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:02.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:02.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:02.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:03.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:03.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:03.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:03.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:03.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:03.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:03.377 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.8 primary-affinity to 0.968221 (8634532) 2024-08-30T23:32:03.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:03.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:32:03.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:03.519 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:03.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:03.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:03.736 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:03.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:03.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:03.788 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:03 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:03.788 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:03 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:32:03.788 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:03 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:03.788 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:03 smithi106 ceph-mon[16603]: osdmap e264: 12 total, 12 up, 12 in 2024-08-30T23:32:03.788 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2914600536' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 0.9682206909884691} : dispatch 2024-08-30T23:32:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:03 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:03 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:32:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:03 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:03 smithi136 ceph-mon[20207]: osdmap e264: 12 total, 12 up, 12 in 2024-08-30T23:32:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2914600536' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 0.9682206909884691} : dispatch 2024-08-30T23:32:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:03 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:03.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:03 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:32:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:03 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:03 smithi173 ceph-mon[20554]: osdmap e264: 12 total, 12 up, 12 in 2024-08-30T23:32:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2914600536' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 0.9682206909884691} : dispatch 2024-08-30T23:32:03.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:03.903 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:03.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:04.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:04.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:04.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:04.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:32:04.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:04.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:04.285 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:04.286 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:04.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:04.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:04.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:04.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:04.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2914600536' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.9682206909884691}]': finished 2024-08-30T23:32:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:04 smithi173 ceph-mon[20554]: osdmap e265: 12 total, 12 up, 12 in 2024-08-30T23:32:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:04 smithi173 ceph-mon[20554]: pgmap v651: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-08-30T23:32:04.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2914600536' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.9682206909884691}]': finished 2024-08-30T23:32:04.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:04 smithi106 ceph-mon[16603]: osdmap e265: 12 total, 12 up, 12 in 2024-08-30T23:32:04.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:04 smithi106 ceph-mon[16603]: pgmap v651: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-08-30T23:32:04.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:04.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:04.713 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2914600536' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 0.9682206909884691}]': finished 2024-08-30T23:32:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:04 smithi136 ceph-mon[20207]: osdmap e265: 12 total, 12 up, 12 in 2024-08-30T23:32:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:04 smithi136 ceph-mon[20207]: pgmap v651: 17 pgs: 17 active+clean; 5.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s 2024-08-30T23:32:04.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:04.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:04.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:04.942 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:04.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:05.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:32:05.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:05.068 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:05.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:05.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:05.256 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:05.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:05.279 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:05.280 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:05.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:05.304 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:05.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:05.332 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:05.332 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:05.334 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:05.359 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:05.360 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:05.386 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:05.386 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:05.387 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:05.411 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:05.411 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:05.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:05.438 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:05.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:05.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:05.464 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:05.465 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:05.490 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:05.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:05.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:05.518 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:05.546 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:05.547 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:05.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:05.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:05.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1527580087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:05.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:05.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1527580087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:05.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:05.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:32:05.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:05.850 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1527580087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:05.924 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:05.925 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:32:05.925 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:32:05.925 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:32:05.925 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:05.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:05.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:05.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:06.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:06.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:06.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:06.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:06.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:06.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:06.381 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:06.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:06.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:06.535 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:06 smithi173 ceph-mon[20554]: pgmap v652: 17 pgs: 17 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-08-30T23:32:06.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:06 smithi106 ceph-mon[16603]: pgmap v652: 17 pgs: 17 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-08-30T23:32:06.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:06.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:06.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:06 smithi136 ceph-mon[20207]: pgmap v652: 17 pgs: 17 active+clean; 5.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s 2024-08-30T23:32:06.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:06.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:06.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:32:06.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:06.886 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:06.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:06.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:07.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:07.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:07.149 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:07.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:07.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:07.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:07.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:07.401 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:32:07.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:07.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:07.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2762306877' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:07.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:07.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:14225 2.5 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:32:07.723 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 358.84723564400002, 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24031856100000001, 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14225 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:32:07.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873763+0000", 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7887999999999999e-05 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873773+0000", 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8700000000000004e-06 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873780+0000", 2024-08-30T23:32:07.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6850000000000004e-06 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873819+0000", 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9190999999999998e-05 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873906+0000", 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7612000000000001e-05 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:32:07.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873999+0000", 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2088000000000003e-05 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.874429+0000", 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043006099999999997 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108141+0000", 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23371239399999999 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.113956+0000", 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058150340000000002 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114021+0000", 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500999999999998e-05 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114063+0000", 2024-08-30T23:32:07.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2512999999999998e-05 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:14225 2.5 e109/97 ondisk, result = 0)", 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 358.78621916100002, 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.173386651, 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934761+0000", 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934762+0000", 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9700000000000005e-07 2024-08-30T23:32:07.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.934766+0000", 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5630000000000002e-06 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108075+0000", 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17330804799999999 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108080+0000", 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4049999999999999e-06 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108127+0000", 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7547999999999999e-05 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108135+0000", 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8059999999999995e-06 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108148+0000", 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2384e-05 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:17728 2.5 2:a3d96d17:::benchmark_data_smithi173_47195_object17727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 356.21705685400002, 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20912092800000001, 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.734 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 17728 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503924+0000", 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.158e-06 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503947+0000", 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2192999999999999e-05 2024-08-30T23:32:07.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503960+0000", 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3098999999999999e-05 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503975+0000", 2024-08-30T23:32:07.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5167999999999999e-05 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504020+0000", 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5464999999999999e-05 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504106+0000", 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5754000000000006e-05 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504188+0000", 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1489000000000005e-05 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504942+0000", 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075475800000000001 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.577391+0000", 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072448769999999996 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712945+0000", 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13555429899999999 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713005+0000", 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9135000000000002e-05 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713044+0000", 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.964e-05 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:17728 2.5 e111/97 ondisk, result = 0)", 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 356.21580133100002, 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20787409000000001, 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505185+0000", 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712478+0000", 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20729294100000001 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712485+0000", 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712934+0000", 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044926100000000001 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712941+0000", 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4999999999999996e-06 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713053+0000", 2024-08-30T23:32:07.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011231900000000001 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:49996 2.5 2:aad518df:::benchmark_data_smithi173_47195_object49995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 332.463119667, 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20351808099999999, 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.746 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 49996 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:32:07.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257862+0000", 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.33e-06 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257920+0000", 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8489000000000002e-05 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257934+0000", 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3372e-05 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257941+0000", 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6459999999999993e-06 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257996+0000", 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5043999999999997e-05 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258082+0000", 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5890000000000006e-05 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258176+0000", 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3366e-05 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259466+0000", 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012900190000000001 2024-08-30T23:32:07.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259518+0000", 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2787999999999997e-05 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461274+0000", 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201755238 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461333+0000", 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9394000000000001e-05 2024-08-30T23:32:07.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461379+0000", 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5504999999999999e-05 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:56067 2.f 2:f6065497:::benchmark_data_smithi173_47195_object56066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:32:07.752 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.93518368999997, 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17632836900000001, 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 56067 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:32:07.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785797+0000", 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785815+0000", 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8224e-05 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785825+0000", 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9869999999999994e-06 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785832+0000", 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.029e-06 2024-08-30T23:32:07.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785866+0000", 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3729999999999997e-05 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785937+0000", 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1027999999999998e-05 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:32:07.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786021+0000", 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3511999999999996e-05 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787483+0000", 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014626260000000001 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787561+0000", 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7565999999999996e-05 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962059+0000", 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17449777999999999 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962099+0000", 2024-08-30T23:32:07.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9953000000000002e-05 2024-08-30T23:32:07.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962125+0000", 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6278000000000001e-05 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:56068 2.6 e124/97 2:62a84442:::benchmark_data_smithi173_47195_object56067:head v 124'3528, mlcod=124'3525)", 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.93458976800002, 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17324930699999999, 2024-08-30T23:32:07.760 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786390+0000", 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786391+0000", 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3200000000000004e-07 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786446+0000", 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4661999999999999e-05 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958168+0000", 2024-08-30T23:32:07.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171722186 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958175+0000", 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7009999999999999e-06 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958216+0000", 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1102000000000001e-05 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.958230+0000", 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4688999999999999e-05 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.959607+0000", 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013761559999999999 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.959640+0000", 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3079e-05 2024-08-30T23:32:07.764 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:62555 2.5 2:a45a541d:::benchmark_data_smithi173_47195_object62554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 322.28309467600002, 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.175746766, 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.765 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62555 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437885+0000", 2024-08-30T23:32:07.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437886+0000", 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5000000000000002e-07 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437908+0000", 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2069000000000002e-05 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437919+0000", 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.098e-05 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.437927+0000", 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9519999999999994e-06 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438090+0000", 2024-08-30T23:32:07.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016235300000000001 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438190+0000", 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010084199999999999 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.438298+0000", 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107347 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.439985+0000", 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016870419999999999 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.440527+0000", 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00054198600000000005 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613517+0000", 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17299020600000001 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613579+0000", 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1791000000000005e-05 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.613632+0000", 2024-08-30T23:32:07.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3448000000000001e-05 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:96989 2.f 2:f906d66e:::benchmark_data_smithi173_47195_object96988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 279.57828906399999, 2024-08-30T23:32:07.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84153836199999998, 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 96989 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:32:07.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142692+0000", 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142718+0000", 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5548999999999999e-05 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142729+0000", 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1126e-05 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142736+0000", 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1840000000000002e-06 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148635+0000", 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058993520000000001 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148767+0000", 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013146699999999999 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148916+0000", 2024-08-30T23:32:07.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149083 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152326+0000", 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034105860000000002 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152337+0000", 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1100999999999999e-05 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984109+0000", 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83177168400000001 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984180+0000", 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0542000000000003e-05 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984229+0000", 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9808000000000002e-05 2024-08-30T23:32:07.778 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:96989 2.f e146/97 ondisk, result = 0)", 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 279.56968170499999, 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83293943299999995, 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151299+0000", 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5600000000000005e-07 2024-08-30T23:32:07.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151308+0000", 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984021+0000", 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83271241100000004 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984029+0000", 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9379999999999992e-06 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984091+0000", 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.211e-05 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984101+0000", 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0368e-05 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984238+0000", 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013670199999999999 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100665 2.6 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:32:07.783 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:32:07.784 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2762306877' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 274.37716521800002, 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21497229600000001, 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100665 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.785 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.786 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343816+0000", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.494e-06 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343824+0000", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2570000000000001e-06 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343841+0000", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7107000000000001e-05 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343850+0000", 2024-08-30T23:32:07.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4299999999999995e-06 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343921+0000", 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0264000000000004e-05 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344020+0000", 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9296999999999995e-05 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:32:07.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344100+0000", 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0465999999999998e-05 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.346504+0000", 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002403723 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347743+0000", 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001238549 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558717+0000", 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21097406299999999 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558756+0000", 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9728000000000003e-05 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558787+0000", 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0917999999999998e-05 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.790 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100667 2.6 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 274.37319873199999, 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27402247099999999, 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100667 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.791 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347782+0000", 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347787+0000", 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3039999999999998e-06 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347798+0000", 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0834e-05 2024-08-30T23:32:07.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347803+0000", 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.242e-06 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347840+0000", 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7630999999999999e-05 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347918+0000", 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7778e-05 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347989+0000", 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0988999999999994e-05 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.349336+0000", 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013472670000000001 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.351224+0000", 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018879960000000001 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621737+0000", 2024-08-30T23:32:07.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27051204000000001 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621775+0000", 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.875e-05 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621804+0000", 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8620999999999999e-05 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.797 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:127040 2.6 2:6091191f:::benchmark_data_smithi173_47195_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 239.86983330699999, 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18230701299999999, 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.798 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127040 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851148+0000", 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851155+0000", 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8990000000000004e-06 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851175+0000", 2024-08-30T23:32:07.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9711000000000002e-05 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851184+0000", 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6780000000000006e-06 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851247+0000", 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2780000000000005e-05 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851360+0000", 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112946 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851485+0000", 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012559 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.854952+0000", 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034665049999999999 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.862420+0000", 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0074685050000000003 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033372+0000", 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17095152199999999 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033407+0000", 2024-08-30T23:32:07.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5567e-05 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033454+0000", 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681000000000002e-05 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:144898 2.6 2:692f2db4:::benchmark_data_smithi173_47195_object47890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:32:07.804 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 216.04181254900001, 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18135380100000001, 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144898 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:32:07.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679169+0000", 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679174+0000", 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.198e-06 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679187+0000", 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.31e-05 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679194+0000", 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:32:07.807 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679236+0000", 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1828e-05 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679316+0000", 2024-08-30T23:32:07.808 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0026999999999998e-05 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679383+0000", 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6925e-05 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.682228+0000", 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028449149999999999 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.829406+0000", 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14717798100000001 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860431+0000", 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031025377 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860482+0000", 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0779000000000003e-05 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860521+0000", 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9307999999999997e-05 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:147254 2.f 2:f940aba9:::benchmark_data_smithi173_47195_object50246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 212.60147511599999, 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17365630000000001, 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.811 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 147254 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119505+0000", 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119506+0000", 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3069999999999999e-06 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119512+0000", 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5450000000000003e-06 2024-08-30T23:32:07.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119525+0000", 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2979999999999999e-05 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119532+0000", 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.99e-06 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119579+0000", 2024-08-30T23:32:07.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7336999999999998e-05 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119668+0000", 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8710999999999998e-05 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.119740+0000", 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2224999999999994e-05 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.123093+0000", 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033526760000000002 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.123100+0000", 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5610000000000001e-06 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293060+0000", 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.169959361 2024-08-30T23:32:07.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293114+0000", 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3825000000000001e-05 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:35.293161+0000", 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7781999999999998e-05 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:d5896595:::benchmark_data_smithi173_47195_object12946:head,153'6861)", 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.983532994, 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59714269600000003, 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737449+0000", 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737471+0000", 2024-08-30T23:32:07.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2631999999999999e-05 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334475+0000", 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59700367600000004 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334480+0000", 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2719999999999997e-06 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334536+0000", 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5430999999999998e-05 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334546+0000", 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9389999999999992e-06 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334590+0000", 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4265999999999999e-05 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.821 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:dc4db0cb:::benchmark_data_smithi173_47195_object13755:head,153'6902)", 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.171723845, 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85400190600000003, 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549257+0000", 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549260+0000", 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.45e-06 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403003+0000", 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85374268200000003 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403016+0000", 2024-08-30T23:32:07.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3033e-05 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403163+0000", 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014668400000000001 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403180+0000", 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6813e-05 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403258+0000", 2024-08-30T23:32:07.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8710999999999999e-05 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.5 e202,182 2:ac5fcd3c:::benchmark_data_smithi173_47195_object8560:head,150'6576)", 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 141.73249151799999, 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91766887600000002, 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:32:07.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1099999999999996e-07 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988496+0000", 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4989999999999999e-06 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906006+0000", 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91750992200000003 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906016+0000", 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0199e-05 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906087+0000", 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1316999999999995e-05 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906098+0000", 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0808000000000001e-05 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906158+0000", 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9620000000000002e-05 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103438 3.9 3:9093f9df:::benchmark_data_smithi173_60780_object103437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.446114018999999, 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38848340100000001, 2024-08-30T23:32:07.830 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103438 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274868+0000", 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274894+0000", 2024-08-30T23:32:07.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5692000000000001e-05 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274909+0000", 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5378000000000001e-05 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274918+0000", 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6489999999999997e-06 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274990+0000", 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2335000000000005e-05 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275112+0000", 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012186 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275235+0000", 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123585 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278451+0000", 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003215954 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.279271+0000", 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081947200000000002 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663147+0000", 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38387581399999998 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663253+0000", 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106125 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663350+0000", 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6832999999999999e-05 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.836 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:103438 3.9 e255/206 ondisk, result = 0)", 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.442864745000001, 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38524248500000002, 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278116+0000", 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278121+0000", 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7609999999999998e-06 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662960+0000", 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38483874699999998 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662985+0000", 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663120+0000", 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013537999999999999 2024-08-30T23:32:07.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663136+0000", 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6220000000000001e-05 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663358+0000", 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022142900000000001 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.840 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:07.841 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:07.841 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:07.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:07.841 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2762306877' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:07.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:07.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:07.993 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:07.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:08.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:08.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:08.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:08.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:08.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:32:08.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:08.385 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:08.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:08.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:08 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2762306877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:08 smithi173 ceph-mon[20554]: osdmap e266: 12 total, 12 up, 12 in 2024-08-30T23:32:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:08 smithi173 ceph-mon[20554]: pgmap v654: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-08-30T23:32:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:08 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2762306877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:08 smithi106 ceph-mon[16603]: osdmap e266: 12 total, 12 up, 12 in 2024-08-30T23:32:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:08 smithi106 ceph-mon[16603]: pgmap v654: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-08-30T23:32:08.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:08.615 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:08.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:08.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:08.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:08 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2762306877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:08 smithi136 ceph-mon[20207]: osdmap e266: 12 total, 12 up, 12 in 2024-08-30T23:32:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:08 smithi136 ceph-mon[20207]: pgmap v654: 17 pgs: 17 active+clean; 5.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s 2024-08-30T23:32:08.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:08.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:08.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:09.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:09.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:09.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:09.153 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:09.154 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:09.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:09.180 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:09.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:09.209 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:09.209 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:09.242 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:09.242 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:09.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:09.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:09.268 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:09.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:09.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:09.296 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:09.297 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:09.325 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:09.325 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:09.356 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:09.356 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:09.379 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:09.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:09.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:09.414 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:09.414 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:09.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:09.445 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:09.445 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:09.470 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:09.471 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:09.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:09.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3295108059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:09.637 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:32:09.637 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:32:09.638 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-08-30T23:32:09.638 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:32:09.638 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:32:09.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-08-30T23:32:09.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:09.665 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3295108059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:09.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:09.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3295108059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:09.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:09.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:09.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:09.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:10.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:10.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:10.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:10.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:10.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:32:10.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:10.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:32:10.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:10.391 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:10.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:10.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:10.523 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:10 smithi173 ceph-mon[20554]: pgmap v655: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:10.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:10.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:10.656 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:10.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:10.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:10.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:10 smithi106 ceph-mon[16603]: pgmap v655: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:10 smithi136 ceph-mon[20207]: pgmap v655: 17 pgs: 17 active+clean; 4.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:10.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:10.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:10.944 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:10.944 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "" 2024-08-30T23:32:10.945 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:10.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:10.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:11.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:11.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:11.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:11.258 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:11.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:11.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:11.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:11.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.521 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:11.601 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:32:11.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:11.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:11.698 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:11.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:11.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:11 smithi173 ceph-mon[20554]: pgmap v656: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/388135442' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:11 smithi106 ceph-mon[16603]: pgmap v656: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/388135442' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:11.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:11.912 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:11.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:12.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:12.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:11 smithi136 ceph-mon[20207]: pgmap v656: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/388135442' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:12.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:12.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:12.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:12.252 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:12.252 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:12.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:12.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:12.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:12.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:12.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:12.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:12.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:32:12.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:12.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:12.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:12.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/388135442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:12 smithi106 ceph-mon[16603]: osdmap e267: 12 total, 12 up, 12 in 2024-08-30T23:32:12.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:12.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:12.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:12.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/388135442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:12.934 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:12 smithi173 ceph-mon[20554]: osdmap e267: 12 total, 12 up, 12 in 2024-08-30T23:32:12.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:12.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:12.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:32:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/388135442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:12 smithi136 ceph-mon[20207]: osdmap e267: 12 total, 12 up, 12 in 2024-08-30T23:32:13.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:13.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:13.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.216 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:13.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:13.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:13.327 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:13.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:13.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:13.461 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:13.461 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:13.479 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:13.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:13.499 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:13.525 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:13.526 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:13.553 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:13.554 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:13.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:13.583 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:13.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:13.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:13.612 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:13.613 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:13.638 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:13.638 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:13.668 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:13.669 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:13.699 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:13.699 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:13.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:13.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:13.726 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:13.737 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:13.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:13.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:13 smithi173 ceph-mon[20554]: osdmap e268: 12 total, 12 up, 12 in 2024-08-30T23:32:13.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:13 smithi173 ceph-mon[20554]: pgmap v659: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-08-30T23:32:13.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3539415926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:13.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:13.757 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:13.785 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:13.785 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:13.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:13 smithi106 ceph-mon[16603]: osdmap e268: 12 total, 12 up, 12 in 2024-08-30T23:32:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:13 smithi106 ceph-mon[16603]: pgmap v659: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-08-30T23:32:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3539415926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:13.865 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:13.866 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:32:13.866 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:32:13.866 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:32:13.866 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:13.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:13.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:13.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:13.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:13 smithi136 ceph-mon[20207]: osdmap e268: 12 total, 12 up, 12 in 2024-08-30T23:32:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:13 smithi136 ceph-mon[20207]: pgmap v659: 17 pgs: 17 active+clean; 4.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-08-30T23:32:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3539415926' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:14.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.128 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:14.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:14.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:14.254 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:14.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:14.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:32:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:14.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:14.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:14.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:14.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:32:14.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:14.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:14.690 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:14.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:14.805 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:14.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:14.916 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:14.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:14.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:15.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:15.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:15.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:15.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:15.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:15.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:15.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:15.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:15.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:15.468 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:15.468 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:32:15.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:32:15.468 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62896 2.a e127/97 2:5bc7d7b5:::benchmark_data_smithi173_47195_object62895:head v 127'3849, mlcod=127'3846)", 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 329.55410521700003, 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.271516058, 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912243+0000", 2024-08-30T23:32:15.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8800000000000002e-07 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912264+0000", 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1222000000000001e-05 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182196+0000", 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26993194300000001 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182200+0000", 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4549999999999997e-06 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182246+0000", 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5404000000000001e-05 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182263+0000", 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7167999999999999e-05 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183724+0000", 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014614000000000001 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183758+0000", 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3677999999999998e-05 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.473 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67113 2.e e129/97 2:7c5fddf0:::benchmark_data_smithi173_47195_object67112:head v 129'4149, mlcod=129'4145)", 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.85434773399999, 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.177302496, 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.612000+0000", 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5600000000000004e-07 2024-08-30T23:32:15.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.612070+0000", 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9492000000000005e-05 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787357+0000", 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17528732899999999 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787369+0000", 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2272999999999999e-05 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787501+0000", 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013128700000000001 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787559+0000", 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8606000000000001e-05 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.789267+0000", 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017075619999999999 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.789302+0000", 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4990999999999998e-05 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67116 2.e e129/97 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head v 129'4150, mlcod=129'4146)", 2024-08-30T23:32:15.478 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.85173980000002, 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22312085200000001, 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614608+0000", 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614657+0000", 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8501000000000002e-05 2024-08-30T23:32:15.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835863+0000", 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.221206134 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835869+0000", 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6359999999999997e-06 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835916+0000", 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6378999999999998e-05 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835929+0000", 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3583e-05 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.837698+0000", 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001769112 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.837728+0000", 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9808999999999999e-05 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67586 2.e e129/97 2:726358de:::benchmark_data_smithi173_47195_object67585:head v 129'4182, mlcod=129'4178)", 2024-08-30T23:32:15.483 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 325.36453881599999, 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.182122652, 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101809+0000", 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:32:15.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101870+0000", 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1013e-05 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281568+0000", 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17969818400000001 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281577+0000", 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6270000000000006e-06 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281653+0000", 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5925000000000002e-05 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281681+0000", 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8076999999999999e-05 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.283872+0000", 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021911510000000001 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.283931+0000", 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9083000000000003e-05 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.488 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:75419 2.0 e132/97 2:02abbf90:::benchmark_data_smithi173_47195_object75418:head v 132'4710, mlcod=132'4708)", 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.61687864999999, 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87299763699999999, 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849469+0000", 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9299999999999996e-07 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849566+0000", 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6235000000000003e-05 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719661+0000", 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87009526500000001 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719680+0000", 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8785999999999999e-05 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719819+0000", 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139491 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719862+0000", 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2382000000000002e-05 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.722427+0000", 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025652909999999999 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.722466+0000", 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9294e-05 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:109533 2.d 2:b9e7a252:::benchmark_data_smithi173_47195_object12525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:44.821491+0000", 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 270.64485633800001, 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.267912015, 2024-08-30T23:32:15.493 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 109533 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821491+0000", 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821491+0000", 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821492+0000", 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0589999999999999e-06 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821498+0000", 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7760000000000003e-06 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821512+0000", 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4106e-05 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821519+0000", 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6079999999999998e-06 2024-08-30T23:32:15.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.822991+0000", 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014714400000000001 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.823097+0000", 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010626800000000001 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.823163+0000", 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5671000000000002e-05 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.825511+0000", 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023481499999999998 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.831879+0000", 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0063679000000000001 2024-08-30T23:32:15.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089328+0000", 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25744958400000001 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089362+0000", 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3571000000000001e-05 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089403+0000", 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0881999999999999e-05 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:109587 2.d 2:b377b128:::benchmark_data_smithi173_47195_object12579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:44.904763+0000", 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 270.56158371999999, 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20665070099999999, 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.500 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 109587 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904763+0000", 2024-08-30T23:32:15.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904763+0000", 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904765+0000", 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.691e-06 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904770+0000", 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2429999999999996e-06 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904782+0000", 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1609e-05 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904789+0000", 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.075e-06 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904840+0000", 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1447999999999999e-05 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904926+0000", 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5690000000000001e-05 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.905012+0000", 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5675999999999998e-05 2024-08-30T23:32:15.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.907884+0000", 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028722420000000001 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908020+0000", 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013599400000000001 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.111348+0000", 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20332840799999999 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.111382+0000", 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3562000000000002e-05 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.111414+0000", 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2063000000000001e-05 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.506 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:109533 2.d e153/97 ondisk, result = 0)", 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:44.908189+0000", 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 270.55815785300001, 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18121869800000001, 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908189+0000", 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908189+0000", 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908191+0000", 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.517e-06 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908206+0000", 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4816e-05 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089281+0000", 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18107588399999999 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089286+0000", 2024-08-30T23:32:15.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4870000000000004e-06 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089319+0000", 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2898000000000002e-05 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089325+0000", 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6699999999999999e-06 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089408+0000", 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3425999999999997e-05 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:126602 2.e e160/97 2:75861acd:::benchmark_data_smithi173_47195_object29594:head v 160'7755, mlcod=160'7752)", 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:07.139995+0000", 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 248.32635191400001, 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17440773300000001, 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.511 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.139995+0000", 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.139995+0000", 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.139996+0000", 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3099999999999997e-07 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.140002+0000", 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6810000000000003e-06 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.311573+0000", 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17157167300000001 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.311580+0000", 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9410000000000001e-06 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.311629+0000", 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9268000000000003e-05 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.311645+0000", 2024-08-30T23:32:15.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5741999999999999e-05 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.314368+0000", 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027233190000000001 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.314403+0000", 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4477999999999998e-05 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:146079 2.d 2:b3e1325e:::benchmark_data_smithi173_47195_object49071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:33.409320+0000", 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 222.05702669900001, 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202734725, 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 146079 2024-08-30T23:32:15.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409320+0000", 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409320+0000", 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409322+0000", 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4079999999999999e-06 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409327+0000", 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5879999999999997e-06 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409340+0000", 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2369e-05 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409345+0000", 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5609999999999998e-06 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409390+0000", 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5062999999999999e-05 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409471+0000", 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0115000000000001e-05 2024-08-30T23:32:15.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409538+0000", 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7527999999999994e-05 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.412680+0000", 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031420770000000001 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.413386+0000", 2024-08-30T23:32:15.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070555399999999995 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.611977+0000", 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198590822 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.612011+0000", 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4116999999999999e-05 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.612055+0000", 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4523e-05 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:152975 2.e e179/97 2:7aa5cae5:::benchmark_data_smithi173_47195_object55967:head v 179'9430, mlcod=178'9427)", 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 212.701512992, 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17485895900000001, 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.522 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8900000000000001e-07 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764838+0000", 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.258e-06 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936271+0000", 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171433059 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936278+0000", 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8210000000000004e-06 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936332+0000", 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4724e-05 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936357+0000", 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4671000000000001e-05 2024-08-30T23:32:15.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.939654+0000", 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032971770000000001 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.939693+0000", 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.896e-05 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:155294 2.e e181/97 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head v 181'9573, mlcod=181'9570)", 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 209.34211555799999, 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26756636299999997, 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:32:15.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.85e-07 2024-08-30T23:32:15.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124238+0000", 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4129999999999997e-06 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.386998+0000", 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26276001399999999 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.387007+0000", 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.679e-06 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.387084+0000", 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6469000000000003e-05 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.387115+0000", 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0574000000000002e-05 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.391740+0000", 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046257349999999997 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.391798+0000", 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7694e-05 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.532 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:194002 2.d 2:bd11e86a:::benchmark_data_smithi173_47195_object96994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 152.682649309, 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0074568129999999, 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 194002 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0100000000000005e-07 2024-08-30T23:32:15.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783702+0000", 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.371e-06 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783710+0000", 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3909999999999998e-06 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783715+0000", 2024-08-30T23:32:15.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4399999999999998e-06 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.792687+0000", 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0089719310000000007 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.792761+0000", 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4054000000000006e-05 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.792832+0000", 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1705999999999994e-05 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797127+0000", 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042944649999999999 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113523+0000", 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31639595199999998 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791079+0000", 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67755674899999996 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791121+0000", 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1829000000000002e-05 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791155+0000", 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3324000000000001e-05 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:194002 2.d e200/97 ondisk, result = 0)", 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 152.669029628, 2024-08-30T23:32:15.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.316210565, 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797322+0000", 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.839e-06 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113409+0000", 2024-08-30T23:32:15.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31608707200000002 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113425+0000", 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6099999999999998e-05 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113509+0000", 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3580999999999998e-05 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113517+0000", 2024-08-30T23:32:15.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1859999999999992e-06 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113528+0000", 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1143e-05 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:194002 2.d e200/97 ondisk, result = 0)", 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:42.797432+0000", 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 152.66891484199999, 2024-08-30T23:32:15.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.993726836, 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797432+0000", 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797432+0000", 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797433+0000", 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1600000000000001e-07 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797437+0000", 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.613e-06 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791015+0000", 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99357831299999999 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791022+0000", 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3289999999999996e-06 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791069+0000", 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6609e-05 2024-08-30T23:32:15.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791076+0000", 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8029999999999999e-06 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791159+0000", 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3553000000000005e-05 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:2664 3.d e213/206 3:b4e3f87a:::benchmark_data_smithi173_60780_object2663:head v 213'157, mlcod=213'153)", 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 122.792051419, 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23855293999999999, 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:32:15.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9099999999999999e-07 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674443+0000", 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014676500000000001 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912387+0000", 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23794400299999999 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912393+0000", 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3409999999999999e-06 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912445+0000", 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1572e-05 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912468+0000", 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3334000000000001e-05 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912808+0000", 2024-08-30T23:32:15.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034020600000000002 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912849+0000", 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0327999999999997e-05 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:19959 3.d e214/206 3:b8cd0889:::benchmark_data_smithi173_60780_object19958:head v 214'1243, mlcod=214'1239)", 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:24.063884+0000", 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 111.402462755, 2024-08-30T23:32:15.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18932943499999999, 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063884+0000", 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063884+0000", 2024-08-30T23:32:15.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063885+0000", 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063912+0000", 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6775e-05 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252412+0000", 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188500011 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252435+0000", 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2844e-05 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252473+0000", 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8583e-05 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252488+0000", 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4544e-05 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.253189+0000", 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070125500000000002 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.253214+0000", 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4698999999999998e-05 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.556 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:19970 3.d e214/206 3:b8325352:::benchmark_data_smithi173_60780_object19969:head v 214'1244, mlcod=214'1240)", 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:24.073819+0000", 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 111.392528176, 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21071751499999999, 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073819+0000", 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073819+0000", 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073820+0000", 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5600000000000004e-07 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073896+0000", 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5807000000000001e-05 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283768+0000", 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20987212099999999 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283772+0000", 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7339999999999999e-06 2024-08-30T23:32:15.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283817+0000", 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4555999999999998e-05 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283831+0000", 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.284504+0000", 2024-08-30T23:32:15.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067363199999999996 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.284536+0000", 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2106000000000001e-05 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:74504 3.2 e243/206 3:43d7c069:::benchmark_data_smithi173_60780_object74503:head v 243'4601, mlcod=243'4598)", 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:12.030559+0000", 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.435787689000001, 2024-08-30T23:32:15.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17484989000000001, 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.030559+0000", 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.030559+0000", 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.030560+0000", 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4099999999999997e-07 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.030648+0000", 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7494e-05 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.203352+0000", 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17270375900000001 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.203357+0000", 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4179999999999996e-06 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.203401+0000", 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4373999999999999e-05 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.203417+0000", 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5600999999999999e-05 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.205375+0000", 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019579279999999998 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:12.205409+0000", 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4474999999999998e-05 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:92238 3.0 e250/206 3:0862a24f:::benchmark_data_smithi173_60780_object92237:head v 250'5807, mlcod=250'5804)", 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:29.455781+0000", 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.010566251999997, 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18914341600000001, 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:15.566 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455781+0000", 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455781+0000", 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455782+0000", 2024-08-30T23:32:15.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5900000000000005e-07 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455803+0000", 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1282999999999999e-05 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642542+0000", 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18673842900000001 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642547+0000", 2024-08-30T23:32:15.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0150000000000003e-06 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642626+0000", 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9908999999999997e-05 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642641+0000", 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4542000000000001e-05 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:15.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.644886+0000", 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022451049999999998 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.644924+0000", 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8173999999999997e-05 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:15.570 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:15.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:15.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:15.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:15.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:15 smithi136 ceph-mon[20207]: pgmap v660: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:32:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1628693762' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:32:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:15 smithi173 ceph-mon[20554]: pgmap v660: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:32:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1628693762' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:32:15.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:15 smithi106 ceph-mon[16603]: pgmap v660: 17 pgs: 17 active+clean; 4.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:32:15.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1628693762' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:32:15.865 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:15.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:15.994 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:15.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:16.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:16.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:16.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:16.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:16.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:32:16.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:16.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:16.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:16.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:16.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:16.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:32:16.634 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:16.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:16.723 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:32:16.723 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:32:16.723 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:32:16.723 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-08-30T23:32:16.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 2 0 2024-08-30T23:32:16.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:16.782 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:16 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:16.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1628693762' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:32:16.783 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:16 smithi173 ceph-mon[20554]: osdmap e269: 12 total, 12 up, 12 in 2024-08-30T23:32:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:16.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:16 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1628693762' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:32:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:16 smithi136 ceph-mon[20207]: osdmap e269: 12 total, 12 up, 12 in 2024-08-30T23:32:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:16 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1628693762' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:32:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:16 smithi106 ceph-mon[16603]: osdmap e269: 12 total, 12 up, 12 in 2024-08-30T23:32:16.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:16.921 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:16.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:16.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:17.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:17.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:17.057 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:17.113 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:17.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:17.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:17.191 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:32:17.191 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:32:17.192 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:17.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:17.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:17.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:17.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:17.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:17.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:17.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:17.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:17.786 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:17 smithi173 ceph-mon[20554]: pgmap v662: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-08-30T23:32:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:17 smithi136 ceph-mon[20207]: pgmap v662: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-08-30T23:32:17.790 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:17.791 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:17.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:17 smithi106 ceph-mon[16603]: pgmap v662: 17 pgs: 17 active+clean; 4.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s 2024-08-30T23:32:17.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:17.819 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:17.820 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:17.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:17.848 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:17.848 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:17.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:17.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:17.902 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:17.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:32:17.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:17.932 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:17.932 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:17.958 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:17.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:17.966 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:17.966 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:17.996 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:17.996 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:18.030 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:18.031 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:18.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:18.063 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:18.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:18.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.092 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:18.099 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:18.099 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:18.131 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:18.131 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:18.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:18.216 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.217 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:18.262 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:18.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:18.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:18.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:18.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:18.530 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.2 primary-affinity to 0 (802) 2024-08-30T23:32:18.574 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:18.601 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:18.688 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4169276365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:18.688 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3282651521' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-08-30T23:32:18.688 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:18 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-08-30T23:32:18.704 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:18.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:18.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4169276365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:18.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3282651521' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-08-30T23:32:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:18 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-08-30T23:32:18.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:18.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4169276365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3282651521' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-08-30T23:32:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:18 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.0} : dispatch 2024-08-30T23:32:18.860 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:18.860 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:18.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:18.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:18.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:18.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:32:19.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:19.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:19.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:19.228 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:19.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:19.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:19.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:19.483 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:19.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:32:19.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:19.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:19.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:19.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:19.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:19 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-08-30T23:32:19.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:19 smithi106 ceph-mon[16603]: osdmap e270: 12 total, 12 up, 12 in 2024-08-30T23:32:19.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:19 smithi106 ceph-mon[16603]: pgmap v664: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:19 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-08-30T23:32:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:19 smithi173 ceph-mon[20554]: osdmap e270: 12 total, 12 up, 12 in 2024-08-30T23:32:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:19 smithi173 ceph-mon[20554]: pgmap v664: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:19.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:19.838 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:19.861 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:19.873 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:19.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:19.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:19.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:19 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.0}]': finished 2024-08-30T23:32:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:19 smithi136 ceph-mon[20207]: osdmap e270: 12 total, 12 up, 12 in 2024-08-30T23:32:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:19 smithi136 ceph-mon[20207]: pgmap v664: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:20.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:20.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:20.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:20.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:20.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:20.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:20.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:20.511 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:20.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:20.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:20.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:20.647 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:20.647 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:20.700 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:32:20.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:20.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/157962294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:20.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:20.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:20.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:20.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:21.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:21.032 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/157962294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/157962294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:21.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:21.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:21.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:21.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:21.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:32:21.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:21.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:21.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:32:21.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:21.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:21.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:21.637 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:21.663 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:21.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:21.777 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:21.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:21.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:21.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:21.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:22.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:21 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:21 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:32:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/157962294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:21 smithi136 ceph-mon[20207]: osdmap e271: 12 total, 12 up, 12 in 2024-08-30T23:32:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:21 smithi136 ceph-mon[20207]: pgmap v666: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:22.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:22.065 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:21 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:22.066 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:21 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:32:22.066 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/157962294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:22.066 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:21 smithi173 ceph-mon[20554]: osdmap e271: 12 total, 12 up, 12 in 2024-08-30T23:32:22.066 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:21 smithi173 ceph-mon[20554]: pgmap v666: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:22.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:21 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:22.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:21 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:32:22.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/157962294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:22.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:21 smithi106 ceph-mon[16603]: osdmap e271: 12 total, 12 up, 12 in 2024-08-30T23:32:22.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:21 smithi106 ceph-mon[16603]: pgmap v666: 17 pgs: 17 active+clean; 4.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s 2024-08-30T23:32:22.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:22.182 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:22.182 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:22.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:22.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:22.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:22.334 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:22.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:22.357 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:22.357 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:22.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:22.381 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:22.407 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:22.407 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:22.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:22.434 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:22.434 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:22.436 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:22.464 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:22.464 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:22.491 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:22.491 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:22.517 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:22.517 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:22.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:22.543 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:22.543 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:22.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:22.567 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:22.567 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:22.591 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:22.592 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:22.615 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:22.615 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:22.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:22.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:22.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:22.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:22.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3226793986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:22.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:22.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:22.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3226793986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:23.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3226793986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:23.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:23.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:32:23.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:23.204 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:23.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:23.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:23.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:32:23.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:23.438 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:23.471 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:23.573 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:23.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:23.705 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:23.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:23.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:23 smithi173 ceph-mon[20554]: pgmap v667: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 937 op/s 2024-08-30T23:32:23.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:23.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:23.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:23.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:23.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:23.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:23 smithi136 ceph-mon[20207]: pgmap v667: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 937 op/s 2024-08-30T23:32:24.073 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:23 smithi106 ceph-mon[16603]: pgmap v667: 17 pgs: 17 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 937 op/s 2024-08-30T23:32:24.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:24.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:24.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:24.226 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:24.226 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:32:24.226 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:32:24.226 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:32:24.226 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:24.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:24.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:24.374 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:24.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:24.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:24.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:24.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:24.727 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:32:24.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:24.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:24.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:24.864 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:32:24.865 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:32:24.865 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.9818312754734808 2024-08-30T23:32:24.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 0.9818312754734808 2024-08-30T23:32:24.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:24.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:24.957 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2579528578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:24.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:24.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:32:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:25.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2579528578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:25.104 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2579528578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:25.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:25.146 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:25.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:25.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:25.310 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:32:25.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:25.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:32:25.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:25.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:25.520 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:25.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:25.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:25.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:25.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:25.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:25.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:25.878 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:22817 2.c 2:33be2d6c:::benchmark_data_smithi173_47195_object22816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:15.136335+0000", 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.74072520599998, 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23528854799999999, 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.879 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22817 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136335+0000", 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136335+0000", 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136337+0000", 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3360000000000002e-06 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136361+0000", 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3737999999999999e-05 2024-08-30T23:32:25.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136377+0000", 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6540000000000001e-05 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136387+0000", 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136441+0000", 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4045000000000002e-05 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136531+0000", 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9345999999999995e-05 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136621+0000", 2024-08-30T23:32:25.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0836999999999998e-05 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.137400+0000", 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077874499999999996 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.137799+0000", 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039926799999999999 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.371543+0000", 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233743644 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.371579+0000", 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5593999999999998e-05 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.371623+0000", 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4706000000000002e-05 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:62896 2.a 2:5bc7d7b5:::benchmark_data_smithi173_47195_object62895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 339.96518572999997, 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.272135707, 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.887 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62896 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911875+0000", 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.099e-06 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911895+0000", 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9724999999999998e-05 2024-08-30T23:32:25.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911907+0000", 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1423000000000001e-05 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911912+0000", 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5160000000000002e-06 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911972+0000", 2024-08-30T23:32:25.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9698999999999999e-05 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912062+0000", 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0204999999999994e-05 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912153+0000", 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0696e-05 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.913669+0000", 2024-08-30T23:32:25.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001516807 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.914495+0000", 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082544200000000002 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183885+0000", 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26939059300000001 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183937+0000", 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1586999999999999e-05 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.184010+0000", 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2915000000000002e-05 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.893 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:75402 2.c e132/97 ondisk, result = 0)", 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 329.25405958599998, 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.335240237, 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:32:25.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:25.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623001+0000", 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2099999999999996e-07 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623007+0000", 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7710000000000003e-06 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.957925+0000", 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33491853599999999 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.957938+0000", 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3142999999999999e-05 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958067+0000", 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012853699999999999 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958082+0000", 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4620000000000001e-05 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958241+0000", 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015900899999999999 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75297 2.b 2:da865ed2:::benchmark_data_smithi173_47195_object75296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276273+0000", 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.60078659800001, 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22091398400000001, 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75297 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276273+0000", 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276273+0000", 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276276+0000", 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.136e-06 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276303+0000", 2024-08-30T23:32:25.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7433999999999999e-05 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276320+0000", 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6922000000000002e-05 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276334+0000", 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4049999999999999e-05 2024-08-30T23:32:25.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277420+0000", 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010862039999999999 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277434+0000", 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3750999999999999e-05 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.496363+0000", 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21892957199999999 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.497187+0000", 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082391500000000004 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75311 2.b 2:d9413190:::benchmark_data_smithi173_47195_object75310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:32:25.904 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276361+0000", 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.600698803, 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23327812000000001, 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75311 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276361+0000", 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276361+0000", 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276362+0000", 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9299999999999998e-07 2024-08-30T23:32:25.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276381+0000", 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8938999999999999e-05 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276390+0000", 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8710000000000003e-06 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276397+0000", 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2479999999999999e-06 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277483+0000", 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010856990000000001 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277493+0000", 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.083e-05 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.498510+0000", 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22101696900000001 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.498976+0000", 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000465797 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.499083+0000", 2024-08-30T23:32:25.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010696000000000001 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506114+0000", 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0070305740000000004 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506156+0000", 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2466000000000003e-05 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.508423+0000", 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022673319999999999 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.508461+0000", 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7796000000000002e-05 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.509639+0000", 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011780460000000001 2024-08-30T23:32:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75355 2.b 2:df42354f:::benchmark_data_smithi173_47195_object75354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276407+0000", 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.60065270799998, 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23431305799999999, 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75355 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276407+0000", 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276407+0000", 2024-08-30T23:32:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276408+0000", 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276439+0000", 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0859999999999999e-05 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276447+0000", 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6759999999999996e-06 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276453+0000", 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0619999999999997e-06 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277535+0000", 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010818259999999999 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277544+0000", 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0690000000000008e-06 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.501083+0000", 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.223538336 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.501222+0000", 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139386 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.501382+0000", 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016007400000000001 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506199+0000", 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048170679999999999 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506207+0000", 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3010000000000007e-06 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.509692+0000", 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034849220000000001 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.509732+0000", 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9243999999999999e-05 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510720+0000", 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098875500000000002 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75381 2.b 2:d6004728:::benchmark_data_smithi173_47195_object75380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.60059808, 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24515187899999999, 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75381 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.920 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4000000000000002e-07 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276511+0000", 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8109000000000003e-05 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276520+0000", 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3710000000000006e-06 2024-08-30T23:32:25.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276527+0000", 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8129999999999998e-06 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277587+0000", 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001060088 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277595+0000", 2024-08-30T23:32:25.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7950000000000008e-06 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.503269+0000", 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22567431399999999 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.503364+0000", 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4963999999999994e-05 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.503466+0000", 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101851 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506212+0000", 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027464659999999999 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510783+0000", 2024-08-30T23:32:25.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045707569999999999 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.520094+0000", 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0093106439999999999 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.520133+0000", 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9777999999999998e-05 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.521614+0000", 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014803889999999999 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75430 2.b 2:d9f4526d:::benchmark_data_smithi173_47195_object75429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.600525205, 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24608438699999999, 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.927 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75430 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5200000000000002e-07 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276577+0000", 2024-08-30T23:32:25.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1709999999999999e-05 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276585+0000", 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.541e-06 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276592+0000", 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9269999999999999e-06 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277626+0000", 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010331089999999999 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277630+0000", 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3579999999999996e-06 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.504828+0000", 2024-08-30T23:32:25.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.227198493 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.504914+0000", 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5351000000000003e-05 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.505024+0000", 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110031 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.507311+0000", 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00228766 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510833+0000", 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035211420000000001 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.521663+0000", 2024-08-30T23:32:25.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010829979999999999 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.521700+0000", 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7666000000000002e-05 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.522619+0000", 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091896700000000003 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.934 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75433 2.b 2:d49bdc78:::benchmark_data_smithi173_47195_object75432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 327.60045846499997, 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24810786800000001, 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.935 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75433 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9799999999999999e-07 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276644+0000", 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2070000000000002e-05 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276652+0000", 2024-08-30T23:32:25.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4789999999999996e-06 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276660+0000", 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1520000000000003e-06 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277668+0000", 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010085370000000001 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:32:25.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277673+0000", 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8300000000000003e-06 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506298+0000", 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22862464800000001 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506376+0000", 2024-08-30T23:32:25.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:25 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:25.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2579528578' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:25.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:25 smithi106 ceph-mon[16603]: osdmap e272: 12 total, 12 up, 12 in 2024-08-30T23:32:25.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:25 smithi106 ceph-mon[16603]: pgmap v669: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s 2024-08-30T23:32:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8415000000000001e-05 2024-08-30T23:32:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:32:25.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506472+0000", 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5729000000000005e-05 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510841+0000", 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043695649999999997 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.515766+0000", 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049242940000000001 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.523643+0000", 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078772089999999996 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.523685+0000", 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1755000000000003e-05 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.524709+0000", 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010247870000000001 2024-08-30T23:32:25.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:110499 2.b 2:d409eb3a:::benchmark_data_smithi173_47195_object13491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 279.80225052999998, 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25912935999999998, 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 110499 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074810+0000", 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7499999999999998e-07 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074815+0000", 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5700000000000003e-06 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074826+0000", 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.101e-05 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074832+0000", 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.643e-06 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074884+0000", 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2614999999999997e-05 2024-08-30T23:32:25.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074986+0000", 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101808 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.075067+0000", 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0661000000000006e-05 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.077720+0000", 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026533749999999999 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.078416+0000", 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069541700000000004 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333266+0000", 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25485053000000002 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333335+0000", 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8868999999999994e-05 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333939+0000", 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000603887 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:153181 2.a 2:5b6beff2:::benchmark_data_smithi173_47195_object56173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-08-30T23:32:25.950 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:43.131212+0000", 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 222.74584773500001, 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22176643100000001, 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 153181 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131212+0000", 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131212+0000", 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131213+0000", 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131219+0000", 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7300000000000002e-06 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131230+0000", 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0305999999999999e-05 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131237+0000", 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5839999999999997e-06 2024-08-30T23:32:25.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131298+0000", 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.071e-05 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131447+0000", 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014884300000000001 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.131570+0000", 2024-08-30T23:32:25.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012309899999999999 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.135356+0000", 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037859790000000001 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.136398+0000", 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00104177 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352855+0000", 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21645704700000001 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352926+0000", 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1563000000000006e-05 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352979+0000", 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2575999999999999e-05 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.956 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:153181 2.a e179/97 ondisk, result = 0)", 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:43.136028+0000", 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 222.741031706, 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.216957082, 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.136028+0000", 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.136028+0000", 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.136030+0000", 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2309999999999999e-06 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.136037+0000", 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.007e-06 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352779+0000", 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21674211700000001 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352785+0000", 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2140000000000001e-06 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352840+0000", 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4780000000000001e-05 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352849+0000", 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5140000000000007e-06 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:43.352985+0000", 2024-08-30T23:32:25.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000136219 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:188148 2.c 2:3df8a074:::benchmark_data_smithi173_47195_object91140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:34.664403+0000", 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 171.21265742599999, 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22132776200000001, 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.961 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188148 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664403+0000", 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664403+0000", 2024-08-30T23:32:25.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664404+0000", 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3209999999999999e-06 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664409+0000", 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7339999999999999e-06 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664422+0000", 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3302000000000001e-05 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664429+0000", 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2570000000000001e-06 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664476+0000", 2024-08-30T23:32:25.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6508999999999998e-05 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664552+0000", 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6674000000000005e-05 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.664622+0000", 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9575999999999999e-05 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.669327+0000", 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047055370000000001 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.669856+0000", 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00052879900000000004 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.885602+0000", 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21574571200000001 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.885657+0000", 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4639999999999999e-05 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:34.885730+0000", 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3701000000000005e-05 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:21973 3.8 3:16383432:::benchmark_data_smithi173_60780_object21972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:25.267447+0000", 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.609613076, 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22566488500000001, 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21973 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267447+0000", 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267447+0000", 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267448+0000", 2024-08-30T23:32:25.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5099999999999999e-07 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267469+0000", 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1212999999999998e-05 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267477+0000", 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8089999999999993e-06 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267482+0000", 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1699999999999996e-06 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267528+0000", 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6592999999999999e-05 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267640+0000", 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111897 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267771+0000", 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013064500000000001 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.268634+0000", 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086273700000000003 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.269874+0000", 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001240732 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.493035+0000", 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22316090799999999 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.493077+0000", 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1498999999999997e-05 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.493112+0000", 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4931000000000001e-05 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.974 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:21975 3.4 3:2a1d7d2d:::benchmark_data_smithi173_60780_object21974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:25.267712+0000", 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 120.609348372, 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.215978423, 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21975 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.975 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267712+0000", 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267712+0000", 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267712+0000", 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.96e-07 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267727+0000", 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5111e-05 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267732+0000", 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7190000000000001e-06 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267737+0000", 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6600000000000003e-06 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.268569+0000", 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083242699999999995 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.268666+0000", 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7108000000000006e-05 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5", 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.268754+0000", 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7653999999999998e-05 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.269604+0000", 2024-08-30T23:32:25.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085073199999999999 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.270024+0000", 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00042002499999999998 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.483573+0000", 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21354864000000001 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.483629+0000", 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6378000000000001e-05 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.483690+0000", 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0572999999999998e-05 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:33278 3.4 3:20bdef47:::benchmark_data_smithi173_60780_object33277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-08-30T23:32:25.981 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:35.054358+0000", 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 110.82270209399999, 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21600087900000001, 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 33278 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054358+0000", 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054358+0000", 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054359+0000", 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3090000000000001e-06 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054380+0000", 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1128000000000002e-05 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054399+0000", 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8604e-05 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054426+0000", 2024-08-30T23:32:25.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6934999999999999e-05 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054488+0000", 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2470000000000003e-05 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054647+0000", 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015819799999999999 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5", 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.054824+0000", 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017753499999999999 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.056497+0000", 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001672906 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.056634+0000", 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013671000000000001 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.270280+0000", 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21364587800000001 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.270323+0000", 2024-08-30T23:32:25.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3229999999999998e-05 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:35.270359+0000", 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5976000000000002e-05 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:34423 3.4 3:28475229:::benchmark_data_smithi173_60780_object34422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:36.141947+0000", 2024-08-30T23:32:25.988 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 109.73511261, 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22328045799999999, 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 34423 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.141947+0000", 2024-08-30T23:32:25.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.141947+0000", 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.141948+0000", 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0550000000000001e-06 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.141983+0000", 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4823000000000002e-05 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.141996+0000", 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2342e-05 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.142003+0000", 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.452e-06 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.142055+0000", 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2419000000000001e-05 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.142153+0000", 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7501e-05 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5", 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.142246+0000", 2024-08-30T23:32:25.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2557000000000002e-05 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.143262+0000", 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010163959999999999 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.143666+0000", 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000403841 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.365165+0000", 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22149970499999999 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.365201+0000", 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5778999999999997e-05 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:36.365228+0000", 2024-08-30T23:32:25.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6588e-05 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:41062 3.f e226/206 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head v 226'2525, mlcod=226'2521)", 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 102.67472821299999, 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32836641100000002, 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:25.995 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:25.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202340+0000", 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9529999999999999e-06 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202390+0000", 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0473999999999998e-05 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529209+0000", 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32681829099999998 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529215+0000", 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8379999999999996e-06 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529257+0000", 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2036999999999997e-05 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529272+0000", 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4331999999999999e-05 2024-08-30T23:32:25.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530655+0000", 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001382961 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530698+0000", 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3525000000000001e-05 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:25.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103636 3.e 3:73e363f7:::benchmark_data_smithi173_60780_object182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:42.426866+0000", 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.450193642000002, 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.229269273, 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103636 2024-08-30T23:32:26.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426866+0000", 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426866+0000", 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426867+0000", 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426872+0000", 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5510000000000001e-06 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426881+0000", 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0790000000000007e-06 2024-08-30T23:32:26.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426897+0000", 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5829e-05 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.430114+0000", 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032173000000000002 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:26.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.430305+0000", 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019137900000000001 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.430427+0000", 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121854 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.434119+0000", 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036919349999999999 2024-08-30T23:32:26.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:26.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.436376+0000", 2024-08-30T23:32:26.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022569069999999998 2024-08-30T23:32:26.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656042+0000", 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21966624400000001 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656081+0000", 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9029000000000002e-05 2024-08-30T23:32:26.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656136+0000", 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4397000000000001e-05 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:103636 3.e e255/206 ondisk, result = 0)", 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:42.434954+0000", 2024-08-30T23:32:26.009 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.442106051000003, 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22119518699999999, 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.434954+0000", 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.434954+0000", 2024-08-30T23:32:26.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.434955+0000", 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.434969+0000", 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.455e-05 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.653840+0000", 2024-08-30T23:32:26.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21887090100000001 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.653846+0000", 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.127e-06 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656033+0000", 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021869020000000001 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:32:26.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656038+0000", 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1830000000000002e-06 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656149+0000", 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110894 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:32:26.013 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:32:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.015 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:26.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:25 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2579528578' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:25 smithi136 ceph-mon[20207]: osdmap e272: 12 total, 12 up, 12 in 2024-08-30T23:32:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:25 smithi136 ceph-mon[20207]: pgmap v669: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s 2024-08-30T23:32:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:25 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2579528578' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:25 smithi173 ceph-mon[20554]: osdmap e272: 12 total, 12 up, 12 in 2024-08-30T23:32:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:25 smithi173 ceph-mon[20554]: pgmap v669: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s 2024-08-30T23:32:26.086 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:26.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:26.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:26.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:26.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:26.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:26.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:26.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:26.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:26.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:26.632 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:26.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:26.650 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:26.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:26.665 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:26.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:26.695 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:26.720 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:26.720 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:26.731 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.6 to 0.981831 (fb59) 2024-08-30T23:32:26.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3088721981' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.9818312754734808} : dispatch 2024-08-30T23:32:26.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/845374147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:26.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:26.755 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:26.755 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:26.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:26.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:26.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:26.816 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:26.816 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:26.844 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:26.844 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:26.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:32:26.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:26.872 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:26.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:26.901 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:26.901 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:26.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:26.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:26.931 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:26.931 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:26.960 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:27.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3088721981' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.9818312754734808} : dispatch 2024-08-30T23:32:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/845374147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:27.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3088721981' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 0.9818312754734808} : dispatch 2024-08-30T23:32:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/845374147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:27.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:27.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:27.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:27.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.304 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:27.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:27.432 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:27.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:32:27.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:27.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:27.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:27.685 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:27.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3088721981' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.9818312754734808}]': finished 2024-08-30T23:32:27.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:27 smithi106 ceph-mon[16603]: osdmap e273: 12 total, 12 up, 12 in 2024-08-30T23:32:27.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:27 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:27.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:27 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:27.755 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:27 smithi106 ceph-mon[16603]: pgmap v671: 17 pgs: 17 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s 2024-08-30T23:32:27.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:27.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:27.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:27.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:27.939 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3088721981' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.9818312754734808}]': finished 2024-08-30T23:32:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:27 smithi136 ceph-mon[20207]: osdmap e273: 12 total, 12 up, 12 in 2024-08-30T23:32:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:27 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:27 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:27 smithi136 ceph-mon[20207]: pgmap v671: 17 pgs: 17 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s 2024-08-30T23:32:28.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:28.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:28.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3088721981' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.9818312754734808}]': finished 2024-08-30T23:32:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:27 smithi173 ceph-mon[20554]: osdmap e273: 12 total, 12 up, 12 in 2024-08-30T23:32:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:27 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:27 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:27 smithi173 ceph-mon[20554]: pgmap v671: 17 pgs: 17 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 932 op/s 2024-08-30T23:32:28.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:28.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:28.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:28.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:28.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:28.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:28.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:28.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:28.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:28.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:28.759 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:28.768 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:28.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:28.850 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:28.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:28.953 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:28 smithi106 ceph-mon[16603]: osdmap e274: 12 total, 12 up, 12 in 2024-08-30T23:32:28.977 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:28.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:28 smithi136 ceph-mon[20207]: osdmap e274: 12 total, 12 up, 12 in 2024-08-30T23:32:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:28 smithi173 ceph-mon[20554]: osdmap e274: 12 total, 12 up, 12 in 2024-08-30T23:32:29.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:29.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:29.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:29.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:29.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:29.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:29.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:29.494 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:29.528 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:29.528 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:29.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:29.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:29.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:29.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:29.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:29.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:29.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:29.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:30.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:29 smithi136 ceph-mon[20207]: pgmap v673: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-08-30T23:32:30.073 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:30.073 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:30.073 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:29 smithi106 ceph-mon[16603]: pgmap v673: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-08-30T23:32:30.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:29 smithi173 ceph-mon[20554]: pgmap v673: 17 pgs: 17 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-08-30T23:32:30.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:30.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:30.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:30.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:30.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:30.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:30.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:30.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:30.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:32:30.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:30.668 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:30.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:32:30.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:30.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:30.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:30.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:30.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:30.931 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:31.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:30 smithi136 ceph-mon[20207]: osdmap e275: 12 total, 12 up, 12 in 2024-08-30T23:32:31.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:31.065 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:31.065 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:31.065 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:30 smithi106 ceph-mon[16603]: osdmap e275: 12 total, 12 up, 12 in 2024-08-30T23:32:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:30 smithi173 ceph-mon[20554]: osdmap e275: 12 total, 12 up, 12 in 2024-08-30T23:32:31.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:31.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:31.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:31.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:31.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:31.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:31.455 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:31.455 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:31.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:31.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:31.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:31.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:31.777 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:32:31.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:31.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:31.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:31.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:31.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1036026610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:31 smithi136 ceph-mon[20207]: pgmap v675: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:32.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:32.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1036026610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:31 smithi173 ceph-mon[20554]: pgmap v675: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1036026610' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:31 smithi106 ceph-mon[16603]: pgmap v675: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-08-30T23:32:32.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:32.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:32.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:32:32.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:32.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:32.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:32.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:32.553 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:32.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:32.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:32.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:32.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:32.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:32.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:32.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:32.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:32.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:32 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1036026610' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:32 smithi136 ceph-mon[20207]: osdmap e276: 12 total, 12 up, 12 in 2024-08-30T23:32:33.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:32 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1036026610' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:32 smithi173 ceph-mon[20554]: osdmap e276: 12 total, 12 up, 12 in 2024-08-30T23:32:33.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:32 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1036026610' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:32 smithi106 ceph-mon[16603]: osdmap e276: 12 total, 12 up, 12 in 2024-08-30T23:32:33.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:33.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:33.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:33.321 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:32:33.321 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:32:33.321 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:32:33.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-08-30T23:32:33.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:33.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:33.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:33.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:33.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:33.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:33.720 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:33.723 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:33.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:33.761 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:33.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:33.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:33.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:33.825 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:33.826 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:33.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:33.854 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:33.855 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:33.881 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:33.881 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:33.904 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:33.905 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:33.929 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:33.930 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:33.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:33.956 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:33.956 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:33.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:33.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:34.027 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:34.027 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:34.027 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:33 smithi106 ceph-mon[16603]: osdmap e277: 12 total, 12 up, 12 in 2024-08-30T23:32:34.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:33 smithi106 ceph-mon[16603]: pgmap v678: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s 2024-08-30T23:32:34.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1774978943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:33 smithi136 ceph-mon[20207]: osdmap e277: 12 total, 12 up, 12 in 2024-08-30T23:32:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:33 smithi136 ceph-mon[20207]: pgmap v678: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s 2024-08-30T23:32:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1774978943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:33 smithi173 ceph-mon[20554]: osdmap e277: 12 total, 12 up, 12 in 2024-08-30T23:32:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:33 smithi173 ceph-mon[20554]: pgmap v678: 17 pgs: 17 active+clean; 3.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 859 op/s 2024-08-30T23:32:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1774978943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:34.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:34.111 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:34.182 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:34.183 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:34.211 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:34.211 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:34.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:34.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:34.241 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:34.241 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:34.245 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:34.259 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:32:34.259 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:32:34.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:32:34.259 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:32:34.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14225 2.5 e109/97 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head v 109'898, mlcod=109'894)", 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 385.38099438099999, 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.239755616, 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874011+0000", 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874108+0000", 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6961999999999998e-05 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112976+0000", 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23886803600000001 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112983+0000", 2024-08-30T23:32:34.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.951e-06 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113055+0000", 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113079+0000", 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4754000000000001e-05 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113724+0000", 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064475700000000003 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113766+0000", 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1664e-05 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.264 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16156 2.4 e109/97 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head v 109'956, mlcod=109'953)", 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 384.15691368500001, 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213651339, 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098092+0000", 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4300000000000002e-07 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098114+0000", 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2234000000000001e-05 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310963+0000", 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21284856199999999 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310969+0000", 2024-08-30T23:32:34.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311028+0000", 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8857999999999998e-05 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311051+0000", 2024-08-30T23:32:34.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2538999999999999e-05 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311702+0000", 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065106500000000004 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311742+0000", 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0318000000000003e-05 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 337.625791924, 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.207325907, 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.270 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629213+0000", 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629214+0000", 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.294e-06 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629241+0000", 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6767999999999999e-05 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834158+0000", 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20491721900000001 2024-08-30T23:32:34.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834164+0000", 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.268e-06 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834227+0000", 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2712000000000005e-05 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:32:34.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2120000000000002e-05 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836507+0000", 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022580769999999998 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836539+0000", 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1449000000000003e-05 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 305.75723906299999, 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:32:34.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:32:34.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.278 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109556 2.5 2:a6b86250:::benchmark_data_smithi173_47195_object12548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.400656078, 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220264298, 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.279 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109556 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854350+0000", 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8840000000000001e-06 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854356+0000", 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6590000000000004e-06 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854369+0000", 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.289e-05 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854377+0000", 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4649999999999994e-06 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854454+0000", 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6791000000000004e-05 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854600+0000", 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145864 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854751+0000", 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015046499999999999 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.857557+0000", 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028060419999999999 2024-08-30T23:32:34.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.873657+0000", 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016100032 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073675+0000", 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200018474 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073721+0000", 2024-08-30T23:32:34.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6072999999999997e-05 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.074613+0000", 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089165900000000005 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109564 2.4 2:242bfffa:::benchmark_data_smithi173_47195_object12556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:32:34.285 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.39321205800002, 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220047624, 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109564 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:32:34.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861794+0000", 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5889999999999999e-06 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861800+0000", 2024-08-30T23:32:34.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861809+0000", 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999997e-06 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861815+0000", 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861910+0000", 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5278000000000005e-05 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862048+0000", 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137911 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862173+0000", 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012423299999999999 2024-08-30T23:32:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878997+0000", 2024-08-30T23:32:34.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016824741000000001 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879048+0000", 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0890999999999998e-05 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080272+0000", 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20122369600000001 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080313+0000", 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0785999999999999e-05 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.081840+0000", 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015274679999999999 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109567 2.4 2:235a232e:::benchmark_data_smithi173_47195_object12559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.38815723300002, 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218097968, 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109567 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.293 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866849+0000", 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866855+0000", 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1199999999999999e-06 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866877+0000", 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1954e-05 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866885+0000", 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9070000000000007e-06 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869749+0000", 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028635940000000001 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869826+0000", 2024-08-30T23:32:34.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7583999999999994e-05 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869907+0000", 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0408999999999995e-05 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879006+0000", 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090992710000000008 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883826+0000", 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048203719999999999 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083592+0000", 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.199765677 2024-08-30T23:32:34.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083632+0000", 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0262000000000002e-05 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.084945+0000", 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013130590000000001 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109569 2.5 2:a8435ff8:::benchmark_data_smithi173_47195_object12561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.38551488000002, 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22083613699999999, 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109569 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.300 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869491+0000", 2024-08-30T23:32:34.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4270000000000001e-06 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869498+0000", 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869510+0000", 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1630999999999999e-05 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869519+0000", 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869586+0000", 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6699999999999995e-05 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869741+0000", 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155141 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869888+0000", 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014657099999999999 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879878+0000", 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099902989999999994 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879907+0000", 2024-08-30T23:32:34.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8612999999999999e-05 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089168+0000", 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20926119900000001 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089267+0000", 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9097000000000003e-05 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090326+0000", 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010589670000000001 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109574 2.5 2:aaf91c06:::benchmark_data_smithi173_47195_object12566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.378949057, 2024-08-30T23:32:34.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21517555899999999, 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109574 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:32:34.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876058+0000", 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.046e-06 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876065+0000", 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876084+0000", 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8702000000000002e-05 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876094+0000", 2024-08-30T23:32:34.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5850000000000006e-06 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876163+0000", 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9441000000000001e-05 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876332+0000", 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016932699999999999 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876473+0000", 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014066600000000001 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882795+0000", 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063221149999999997 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883230+0000", 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043490299999999999 2024-08-30T23:32:34.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090374+0000", 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20714431699999999 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090425+0000", 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0865000000000002e-05 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091231+0000", 2024-08-30T23:32:34.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00080588100000000005 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109576 2.5 2:ae1552f5:::benchmark_data_smithi173_47195_object12568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.37650524200001, 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21498098400000001, 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.313 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109576 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878501+0000", 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.751e-06 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878508+0000", 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8340000000000001e-06 2024-08-30T23:32:34.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878521+0000", 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669e-05 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878529+0000", 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.055e-06 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880019+0000", 2024-08-30T23:32:34.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001490686 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880144+0000", 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012418800000000001 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880266+0000", 2024-08-30T23:32:34.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000122552 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882939+0000", 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026733849999999999 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.917160+0000", 2024-08-30T23:32:34.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034220360999999998 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092559+0000", 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.175398734 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092623+0000", 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4208999999999995e-05 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.093480+0000", 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00085756000000000005 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109577 2.4 2:26051de8:::benchmark_data_smithi173_47195_object12569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:32:34.320 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 289.37553451500003, 2024-08-30T23:32:34.321 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:34.321 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20693977799999999, 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109577 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879470+0000", 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879471+0000", 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.077e-06 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879476+0000", 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8289999999999998e-06 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879488+0000", 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1883e-05 2024-08-30T23:32:34.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879497+0000", 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6570000000000003e-06 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879535+0000", 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8574e-05 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879631+0000", 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5890999999999994e-05 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879719+0000", 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8331999999999994e-05 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882451+0000", 2024-08-30T23:32:34.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027319240000000002 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.893310+0000", 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01085908 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.085013+0000", 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19170245499999999 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.085071+0000", 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7884999999999999e-05 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.086410+0000", 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001339191 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144821 2.4 2:2415a84c:::benchmark_data_smithi173_47195_object47813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:32:34.328 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.69162090500001, 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23107453999999999, 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144821 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563385+0000", 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.59e-06 2024-08-30T23:32:34.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563391+0000", 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563404+0000", 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2982e-05 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563411+0000", 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565303+0000", 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001891846 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565431+0000", 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012746400000000001 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565550+0000", 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972900000000001 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.573426+0000", 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078756799999999995 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588851+0000", 2024-08-30T23:32:34.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015424679 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793322+0000", 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20447127600000001 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793387+0000", 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4586999999999997e-05 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.794458+0000", 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010715060000000001 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144836 2.4 2:2d7ce86c:::benchmark_data_smithi173_47195_object47828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.66688753899999, 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23589892500000001, 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.335 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144836 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:32:34.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588118+0000", 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3829999999999999e-06 2024-08-30T23:32:34.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588123+0000", 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588169+0000", 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199999999999998e-05 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588182+0000", 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2748999999999999e-05 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.592926+0000", 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047443779999999996 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595020+0000", 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002094243 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595429+0000", 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040842299999999999 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603861+0000", 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0084324050000000005 2024-08-30T23:32:34.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.604206+0000", 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034452400000000002 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.822951+0000", 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21874496900000001 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.823010+0000", 2024-08-30T23:32:34.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9601000000000001e-05 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824016+0000", 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010057499999999999 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155204 2.4 2:2613b3a5:::benchmark_data_smithi173_47195_object58196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:32:34.342 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.47860666299999, 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21707409, 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155204 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:32:34.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776400+0000", 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776406+0000", 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1890000000000004e-06 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776421+0000", 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449999999999999e-05 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776430+0000", 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0420000000000008e-06 2024-08-30T23:32:34.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776499+0000", 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776643+0000", 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000144721 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776820+0000", 2024-08-30T23:32:34.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017706199999999999 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.783777+0000", 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006956322 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.786557+0000", 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027807270000000002 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992894+0000", 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20633659600000001 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992950+0000", 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5664000000000001e-05 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.993472+0000", 2024-08-30T23:32:34.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052240299999999995 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:193978 2.5 2:a2ee5e67:::benchmark_data_smithi173_47195_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.498858087, 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.283382829, 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.349 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193978 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:32:34.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756148+0000", 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756154+0000", 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756166+0000", 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041e-05 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756172+0000", 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2349999999999996e-06 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756333+0000", 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000160767 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756484+0000", 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015133799999999999 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756627+0000", 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014263700000000001 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764342+0000", 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077147439999999999 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.769055+0000", 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047136399999999998 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039012+0000", 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26995667699999998 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039055+0000", 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2567000000000001e-05 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039529+0000", 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047478400000000001 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:193978 2.5 e200/182 ondisk, result = 0)", 2024-08-30T23:32:34.355 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.49090282099999, 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27543190299999998, 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764103+0000", 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764108+0000", 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9080000000000003e-06 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038928+0000", 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27481996400000003 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038937+0000", 2024-08-30T23:32:34.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7860000000000008e-06 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038998+0000", 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0915999999999997e-05 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039007+0000", 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4319999999999999e-06 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039534+0000", 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052686200000000001 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e200,182 2:a1c9afc1:::benchmark_data_smithi173_47195_object1378:head,146'6135)", 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.47219461899999, 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42837196799999999, 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.360 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782811+0000", 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782816+0000", 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6969999999999998e-06 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210395+0000", 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42757847500000001 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210404+0000", 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0000000000000002e-06 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210493+0000", 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9438000000000007e-05 2024-08-30T23:32:34.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.211182+0000", 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068873000000000003 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2684 3.6 3:677e81c8:::benchmark_data_smithi173_60780_object2683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.691858+0000", 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 141.56314617999999, 2024-08-30T23:32:34.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20452972899999999, 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2684 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691858+0000", 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691858+0000", 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691860+0000", 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1739999999999999e-06 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.366 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691911+0000", 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1649e-05 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691922+0000", 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1037e-05 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691928+0000", 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6760000000000004e-06 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.691969+0000", 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1341000000000003e-05 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.692058+0000", 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8888999999999995e-05 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.692142+0000", 2024-08-30T23:32:34.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3455000000000006e-05 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.692545+0000", 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000402984 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.695211+0000", 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026659549999999998 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896323+0000", 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.2011126 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896358+0000", 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5274999999999997e-05 2024-08-30T23:32:34.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.896388+0000", 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9694000000000001e-05 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2685 3.6 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 141.56147089199999, 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22443937, 2024-08-30T23:32:34.371 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2685 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:32:34.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693535+0000", 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.009e-06 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693573+0000", 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7843999999999999e-05 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693589+0000", 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6359e-05 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693603+0000", 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.412e-05 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693657+0000", 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3757e-05 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693735+0000", 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8444999999999999e-05 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693820+0000", 2024-08-30T23:32:34.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4290999999999996e-05 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.694189+0000", 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036947500000000002 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.874073+0000", 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.179884508 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917876+0000", 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043802633000000001 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917931+0000", 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5297999999999998e-05 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917973+0000", 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1631000000000002e-05 2024-08-30T23:32:34.377 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:103697 3.1 e255/230 3:81054a1b:::benchmark_data_smithi173_60780_object243:head v 255'6462, mlcod=255'6460)", 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:42.480286+0000", 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.774718178000001, 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17397622500000001, 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.378 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.480286+0000", 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.480286+0000", 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.480287+0000", 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.89e-07 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.480290+0000", 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4860000000000002e-06 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.651146+0000", 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.170855692 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.651152+0000", 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9699999999999996e-06 2024-08-30T23:32:34.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.651216+0000", 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4200000000000002e-05 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.651247+0000", 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0967999999999999e-05 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:34.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.654230+0000", 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029824880000000002 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:42.654263+0000", 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3031999999999998e-05 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:34.382 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:32:34.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:34.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:34.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:34.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:34.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:34.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:34.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:34.799 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:34.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:34.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:34.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:35.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:35.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:35.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:35.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:35.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:32:35.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:35.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:35.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:moved 0 / 51 (0%) 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.12623 -> 2.12623 (expected baseline 1.97379) 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 1 -> 1 pgs (0.235294 -> 0.235294 * mean) 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.11 with 8 -> 8 pgs (1.88235 -> 1.88235 * mean) 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:32:35.421 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 1 2024-08-30T23:32:35.422 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:32:35.422 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0114 2024-08-30T23:32:35.422 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.1135 2024-08-30T23:32:35.422 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:32:35.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:35.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:35.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:35.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:35.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:35.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:35.831 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:35.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:35.958 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:35.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:36.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:36.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:36.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:36.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:32:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.222 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:36.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:36.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:36.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:36.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:36.479 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:36.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:36.608 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:36.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:36.741 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:36.741 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:36 smithi136 ceph-mon[20207]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:36.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:36 smithi136 ceph-mon[20207]: pgmap v679: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s 2024-08-30T23:32:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:36.775 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:36 smithi173 ceph-mon[20554]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:36.775 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:36 smithi173 ceph-mon[20554]: pgmap v679: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s 2024-08-30T23:32:36.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:36 smithi106 ceph-mon[16603]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:36 smithi106 ceph-mon[16603]: pgmap v679: 17 pgs: 17 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 872 op/s 2024-08-30T23:32:36.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:36.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:36.995 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:36.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:37.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:37.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:37.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:37.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:37.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:37.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:37.428 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-30T23:32:37.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-08-30T23:32:37.480 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:37.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:37.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:37.539 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:37.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:37.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:37.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:37.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:37.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:37.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:37.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:37.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:38.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:38.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:38.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:38.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:38.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:38.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:38.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:38.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:157327 3.4 e277/206 3:2f3a313d:::benchmark_data_smithi173_60780_object53873:head v 277'9764, mlcod=277'9761)", 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:32:38.486382+0000", 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0067119830000000004, 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0067405779999999997, 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:38.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486382+0000", 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486382+0000", 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:38.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486383+0000", 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3399999999999997e-07 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486388+0000", 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5459999999999999e-06 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486416+0000", 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7515999999999999e-05 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486423+0000", 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3479999999999998e-06 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486475+0000", 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2086999999999998e-05 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.486498+0000", 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2597999999999999e-05 2024-08-30T23:32:38.498 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:157331 3.a e277/206 3:59ef22f9:::benchmark_data_smithi173_60780_object53877:head v 277'9732, mlcod=277'9730)", 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:32:38.489297+0000", 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0037969150000000001, 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003877166, 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489297+0000", 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489297+0000", 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489299+0000", 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5829999999999999e-06 2024-08-30T23:32:38.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489308+0000", 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0469999999999999e-06 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489358+0000", 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0118999999999999e-05 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489368+0000", 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0064000000000001e-05 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489439+0000", 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1678000000000001e-05 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.489464+0000", 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4964999999999999e-05 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.502 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:157334 3.4 e277/206)", 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:32:38.490011+0000", 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.003083138, 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003206137, 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:38.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.490011+0000", 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.490011+0000", 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.490012+0000", 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6099999999999999e-07 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.490017+0000", 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.694e-06 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.490052+0000", 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4514000000000002e-05 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:32:38.490058+0000", 2024-08-30T23:32:38.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9680000000000003e-06 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 3 2024-08-30T23:32:38.506 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:32:38.509 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:32:38.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:38 smithi173 ceph-mon[20554]: pgmap v680: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-08-30T23:32:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/604126534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:32:38.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:38 smithi106 ceph-mon[16603]: pgmap v680: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-08-30T23:32:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/604126534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:32:38.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:32:38.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:38.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:38.774 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:38.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:38 smithi136 ceph-mon[20207]: pgmap v680: 17 pgs: 17 active+clean; 3.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 811 op/s 2024-08-30T23:32:38.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/604126534' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:32:38.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:38.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:38.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:38.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:38.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:39.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:39.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:39.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:39.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:32:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:39.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:39.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:39.373 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:39.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:39.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:39 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/604126534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:32:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:39 smithi106 ceph-mon[16603]: osdmap e278: 12 total, 12 up, 12 in 2024-08-30T23:32:39.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:39.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:39.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:39.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:39 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/604126534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:32:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:39 smithi136 ceph-mon[20207]: osdmap e278: 12 total, 12 up, 12 in 2024-08-30T23:32:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:39 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/604126534' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:32:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:39 smithi173 ceph-mon[20554]: osdmap e278: 12 total, 12 up, 12 in 2024-08-30T23:32:39.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:39.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:39.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:39.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:40.020 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:40.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:40.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:40.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:40.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:40.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:40.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:40.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:40.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:40.548 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:40.551 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:40.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:40.680 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:40 smithi106 ceph-mon[16603]: pgmap v682: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-08-30T23:32:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:40.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:40.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:40 smithi136 ceph-mon[20207]: pgmap v682: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-08-30T23:32:40.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:40 smithi173 ceph-mon[20554]: pgmap v682: 17 pgs: 17 active+clean; 3.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-08-30T23:32:40.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:40.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:40.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:41.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:41.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:41.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:41.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:41.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:41.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:41.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.449 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:41.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:41.458 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:41.459 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:41.487 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:41.487 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:41.510 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:41.510 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:41.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:41.533 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:41.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:41.559 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:41.560 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:41.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:41.589 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:41.590 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:moved 3 / 51 (5.88235%) 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.12623 -> 2.0052 (expected baseline 1.97379) 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 1 -> 1 pgs (0.235294 -> 0.235294 * mean) 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.11 with 8 -> 8 pgs (1.88235 -> 1.88235 * mean) 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:oload 110 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0104 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0114 2024-08-30T23:32:41.595 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 weight 1.0000 -> 0.5898 2024-08-30T23:32:41.596 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 weight 1.0000 -> 0.5971 2024-08-30T23:32:41.596 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 weight 1.0000 -> 0.8072 2024-08-30T23:32:41.596 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 1.0000 -> 0.8264 2024-08-30T23:32:41.625 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:41.626 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:41.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:41.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:41.672 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:41.672 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:41.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:41.681 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:41 smithi106 ceph-mon[16603]: 3.c scrub starts 2024-08-30T23:32:41.682 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:41 smithi106 ceph-mon[16603]: 3.9 scrub starts 2024-08-30T23:32:41.682 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"} : dispatch 2024-08-30T23:32:41.682 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:41 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"} : dispatch 2024-08-30T23:32:41.682 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:41 smithi106 ceph-mon[16603]: 3.9 scrub ok 2024-08-30T23:32:41.682 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3875005098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:41.688 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:41.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:41.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:41.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:41.711 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:41.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:41.733 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:41.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:41.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:41 smithi136 ceph-mon[20207]: 3.c scrub starts 2024-08-30T23:32:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:41 smithi136 ceph-mon[20207]: 3.9 scrub starts 2024-08-30T23:32:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"} : dispatch 2024-08-30T23:32:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:41 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"} : dispatch 2024-08-30T23:32:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:41 smithi136 ceph-mon[20207]: 3.9 scrub ok 2024-08-30T23:32:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3875005098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:41.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:41.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:41 smithi173 ceph-mon[20554]: 3.c scrub starts 2024-08-30T23:32:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:41 smithi173 ceph-mon[20554]: 3.9 scrub starts 2024-08-30T23:32:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"} : dispatch 2024-08-30T23:32:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:41 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"} : dispatch 2024-08-30T23:32:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:41 smithi173 ceph-mon[20554]: 3.9 scrub ok 2024-08-30T23:32:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3875005098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:41.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:41.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:41.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:42.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:42.068 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:42.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:32:42.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:42.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:42.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:42.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:42.360 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:32:42.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-08-30T23:32:42.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:42.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:42.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:42.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:42.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:42.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:42.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:42.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:42 smithi136 ceph-mon[20207]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:42 smithi136 ceph-mon[20207]: 3.c scrub ok 2024-08-30T23:32:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:42 smithi136 ceph-mon[20207]: pgmap v683: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-08-30T23:32:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:42 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"}]': finished 2024-08-30T23:32:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:42 smithi136 ceph-mon[20207]: osdmap e279: 12 total, 12 up, 12 in 2024-08-30T23:32:42.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:42.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:42 smithi173 ceph-mon[20554]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:42 smithi173 ceph-mon[20554]: 3.c scrub ok 2024-08-30T23:32:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:42 smithi173 ceph-mon[20554]: pgmap v683: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-08-30T23:32:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:42 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"}]': finished 2024-08-30T23:32:42.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:42 smithi173 ceph-mon[20554]: osdmap e279: 12 total, 12 up, 12 in 2024-08-30T23:32:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:42 smithi106 ceph-mon[16603]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:42 smithi106 ceph-mon[16603]: 3.c scrub ok 2024-08-30T23:32:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:42 smithi106 ceph-mon[16603]: pgmap v683: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 922 op/s 2024-08-30T23:32:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:42 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54157','3':'52901','4':'39130','11':'38654'}"}]': finished 2024-08-30T23:32:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:42 smithi106 ceph-mon[16603]: osdmap e279: 12 total, 12 up, 12 in 2024-08-30T23:32:42.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:42.932 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:42.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:43.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:43.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:43.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:43.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:43.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:32:43.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:43.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:43.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:43.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.429 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:43.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:43.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:43.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:43.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:43.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:43.784 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:43.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:43.809 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:43 smithi106 ceph-mon[16603]: osdmap e280: 12 total, 12 up, 12 in 2024-08-30T23:32:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:43 smithi106 ceph-mon[16603]: 3.0 scrub starts 2024-08-30T23:32:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:43 smithi106 ceph-mon[16603]: 3.0 scrub ok 2024-08-30T23:32:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:43 smithi106 ceph-mon[16603]: pgmap v686: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 823 op/s 2024-08-30T23:32:43.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:43.927 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:43.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:44.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:44.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:43 smithi136 ceph-mon[20207]: osdmap e280: 12 total, 12 up, 12 in 2024-08-30T23:32:44.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:43 smithi136 ceph-mon[20207]: 3.0 scrub starts 2024-08-30T23:32:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:43 smithi136 ceph-mon[20207]: 3.0 scrub ok 2024-08-30T23:32:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:43 smithi136 ceph-mon[20207]: pgmap v686: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 823 op/s 2024-08-30T23:32:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:44.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:43 smithi173 ceph-mon[20554]: osdmap e280: 12 total, 12 up, 12 in 2024-08-30T23:32:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:43 smithi173 ceph-mon[20554]: 3.0 scrub starts 2024-08-30T23:32:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:43 smithi173 ceph-mon[20554]: 3.0 scrub ok 2024-08-30T23:32:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:43 smithi173 ceph-mon[20554]: pgmap v686: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 823 op/s 2024-08-30T23:32:44.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:44.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:44.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:44.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:44.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:44.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:44.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:44.567 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:44.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:44.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:44.756 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:44 smithi106 ceph-mon[16603]: osdmap e281: 12 total, 12 up, 12 in 2024-08-30T23:32:44.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:44 smithi106 ceph-mon[16603]: 3.e starting backfill to osd.3 from (0'0,0'0] MAX to 278'10133 2024-08-30T23:32:44.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:44 smithi106 ceph-mon[16603]: 3.e starting backfill to osd.9 from (0'0,0'0] MAX to 278'10133 2024-08-30T23:32:44.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:44.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:44.865 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:44.891 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:32:44.892 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:32:44.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:44.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:44.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:44 smithi136 ceph-mon[20207]: osdmap e281: 12 total, 12 up, 12 in 2024-08-30T23:32:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:44 smithi136 ceph-mon[20207]: 3.e starting backfill to osd.3 from (0'0,0'0] MAX to 278'10133 2024-08-30T23:32:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:44 smithi136 ceph-mon[20207]: 3.e starting backfill to osd.9 from (0'0,0'0] MAX to 278'10133 2024-08-30T23:32:45.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:44 smithi173 ceph-mon[20554]: osdmap e281: 12 total, 12 up, 12 in 2024-08-30T23:32:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:44 smithi173 ceph-mon[20554]: 3.e starting backfill to osd.3 from (0'0,0'0] MAX to 278'10133 2024-08-30T23:32:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:44 smithi173 ceph-mon[20554]: 3.e starting backfill to osd.9 from (0'0,0'0] MAX to 278'10133 2024-08-30T23:32:45.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:45.185 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:45.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:45.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:45.343 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:45.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:45.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:45.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:45.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:45.609 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:32:45.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:45.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:45.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:32:45.831 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:45.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:45.933 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1043320689' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:45.933 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:45 smithi106 ceph-mon[16603]: pgmap v688: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 15 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s 2024-08-30T23:32:45.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:45.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:32:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:45.995 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:45.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1043320689' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:45 smithi136 ceph-mon[20207]: pgmap v688: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 15 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s 2024-08-30T23:32:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1043320689' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:32:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:45 smithi173 ceph-mon[20554]: pgmap v688: 17 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 15 active+clean; 2.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s 2024-08-30T23:32:46.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:46.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:46.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:46.253 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:46.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:46.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:46.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:46.449 INFO:teuthology.orchestra.run.smithi106.stdout:moved 0 / 51 (0%) 2024-08-30T23:32:46.449 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:32:46.449 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.0052 -> 2.0052 (expected baseline 1.97379) 2024-08-30T23:32:46.449 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 1 -> 1 pgs (0.235294 -> 0.235294 * mean) 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.11 with 8 -> 8 pgs (1.88235 -> 1.88235 * mean) 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0096 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0959 2024-08-30T23:32:46.450 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:32:46.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:46.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:46.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:46.645 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:46.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:46.776 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:32:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1043320689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: osdmap e282: 12 total, 12 up, 12 in 2024-08-30T23:32:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:46.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:46.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:46 smithi106 ceph-mon[16603]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:46.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:46.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:46.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:47.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:47.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.029 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:32:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1043320689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: osdmap e282: 12 total, 12 up, 12 in 2024-08-30T23:32:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:46 smithi136 ceph-mon[20207]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:32:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:32:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1043320689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:32:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: osdmap e282: 12 total, 12 up, 12 in 2024-08-30T23:32:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:46 smithi173 ceph-mon[20554]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:47.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:47.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:47.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:47.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:47.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:47.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:47.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.408 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:47.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:47.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:47.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:47.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:47.695 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-30T23:32:47.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-08-30T23:32:47.757 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:47.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:47.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:47.796 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:47.882 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:47 smithi106 ceph-mon[16603]: osdmap e283: 12 total, 12 up, 12 in 2024-08-30T23:32:47.882 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:47 smithi106 ceph-mon[16603]: pgmap v691: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 437 op/s; 65/137985 objects degraded (0.047%); 2820/137985 objects misplaced (2.044%); 728 KiB/s, 11 objects/s recovering 2024-08-30T23:32:47.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:47.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:47.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:48.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:47 smithi136 ceph-mon[20207]: osdmap e283: 12 total, 12 up, 12 in 2024-08-30T23:32:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:47 smithi136 ceph-mon[20207]: pgmap v691: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 437 op/s; 65/137985 objects degraded (0.047%); 2820/137985 objects misplaced (2.044%); 728 KiB/s, 11 objects/s recovering 2024-08-30T23:32:48.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:47 smithi173 ceph-mon[20554]: osdmap e283: 12 total, 12 up, 12 in 2024-08-30T23:32:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:47 smithi173 ceph-mon[20554]: pgmap v691: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 15 active+clean; 2.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 437 op/s; 65/137985 objects degraded (0.047%); 2820/137985 objects misplaced (2.044%); 728 KiB/s, 11 objects/s recovering 2024-08-30T23:32:48.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:48.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:48.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:48.306 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:48.335 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:48.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:48.364 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:48.365 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:48.386 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:48.386 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:48.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:48.415 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:48.416 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:48.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:48.444 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:48.444 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:48.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:48.469 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:48.494 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:48.494 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:48.526 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:48.527 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:48.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:48.555 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:48.555 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:48.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:48.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:48.582 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:48.607 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:48.607 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:48.633 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:48.634 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:48.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:48 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 65/137985 objects degraded (0.047%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:48.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/988011009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:48.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:48.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:48.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:48.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:48 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 65/137985 objects degraded (0.047%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/988011009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:48.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:48.947 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:48.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:48.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:32:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:48 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 65/137985 objects degraded (0.047%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/988011009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:49.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:49.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:49.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:49.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:49.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:49.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:49.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:49.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:49.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.602 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:49.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:32:49.657 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:32:49.657 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:32:49.657 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:32:49.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:49 smithi106 ceph-mon[16603]: pgmap v692: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 65/133668 objects degraded (0.049%); 7516/133668 objects misplaced (5.623%); 3.9 MiB/s, 62 objects/s recovering 2024-08-30T23:32:49.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:49.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:49.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:49.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:49.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:49.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:49.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:49 smithi136 ceph-mon[20207]: pgmap v692: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 65/133668 objects degraded (0.049%); 7516/133668 objects misplaced (5.623%); 3.9 MiB/s, 62 objects/s recovering 2024-08-30T23:32:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:49 smithi173 ceph-mon[20554]: pgmap v692: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 592 op/s; 65/133668 objects degraded (0.049%); 7516/133668 objects misplaced (5.623%); 3.9 MiB/s, 62 objects/s recovering 2024-08-30T23:32:50.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:50.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:50.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:50.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:50.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:50.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:50.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:50.484 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:50.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:50.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:50.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:50.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:50.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:32:50.783 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:50.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:50.892 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:50.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:50.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:51.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:51.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:51.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:51.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:51.350 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:51.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:51.417 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:51.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:32:51.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:51.543 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:51.624 INFO:teuthology.orchestra.run.smithi106.stdout:moved 0 / 51 (0%) 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.0052 -> 2.0052 (expected baseline 1.97379) 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 1 -> 1 pgs (0.235294 -> 0.235294 * mean) 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.11 with 8 -> 8 pgs (1.88235 -> 1.88235 * mean) 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 0.05 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0091 2024-08-30T23:32:51.625 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0908 2024-08-30T23:32:51.626 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:32:51.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:51.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:51.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:51.804 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:51 smithi106 ceph-mon[16603]: pgmap v693: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.6 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 136/128121 objects degraded (0.106%); 7286/128121 objects misplaced (5.687%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:32:51.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:51.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:51.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:52.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:51 smithi136 ceph-mon[20207]: pgmap v693: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.6 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 136/128121 objects degraded (0.106%); 7286/128121 objects misplaced (5.687%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:32:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:52.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:51 smithi173 ceph-mon[20554]: pgmap v693: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.6 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 136/128121 objects degraded (0.106%); 7286/128121 objects misplaced (5.687%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:32:52.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:52.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:52.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:52.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:52.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:52.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:52.480 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:32:52.481 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:32:52.481 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:32:52.481 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:32:52.481 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:32:52.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:52.589 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:52.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:52.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:52.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:52.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:52.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:52 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:52 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:52 smithi106 ceph-mon[16603]: osdmap e284: 12 total, 12 up, 12 in 2024-08-30T23:32:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:52 smithi106 ceph-mon[16603]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:52.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:52.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:52.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:52 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:52 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:52 smithi136 ceph-mon[20207]: osdmap e284: 12 total, 12 up, 12 in 2024-08-30T23:32:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:52 smithi136 ceph-mon[20207]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:53.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:52 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:53.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:52 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:53.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:52 smithi173 ceph-mon[20554]: osdmap e284: 12 total, 12 up, 12 in 2024-08-30T23:32:53.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:52 smithi173 ceph-mon[20554]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:32:53.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:53.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:53.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:53.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:53.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:32:53.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:53.378 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.396 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:53.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:53.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.525 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:53.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:53.617 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:32:53.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:53.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:53.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:53.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:53.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:53.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:53.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:53.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1153139941' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:53.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:53 smithi106 ceph-mon[16603]: pgmap v695: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.6 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 136/126057 objects degraded (0.108%); 7286/126057 objects misplaced (5.780%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:32:54.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:54.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1153139941' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:54.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:53 smithi136 ceph-mon[20207]: pgmap v695: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.6 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 136/126057 objects degraded (0.108%); 7286/126057 objects misplaced (5.780%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:32:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:54.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1153139941' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:32:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:53 smithi173 ceph-mon[20554]: pgmap v695: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.6 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 136/126057 objects degraded (0.108%); 7286/126057 objects misplaced (5.780%); 4.2 MiB/s, 67 objects/s recovering 2024-08-30T23:32:54.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:54.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:54.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:54.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:54.333 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:32:54.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-08-30T23:32:54.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:54.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:54.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:54.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:54.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:54.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:54.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:54.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 136/126057 objects degraded (0.108%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1153139941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: osdmap e285: 12 total, 12 up, 12 in 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:54.881 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:54.923 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:54.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:32:54.980 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:54.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:32:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 136/126057 objects degraded (0.108%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1153139941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: osdmap e285: 12 total, 12 up, 12 in 2024-08-30T23:32:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:32:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 136/126057 objects degraded (0.108%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:32:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1153139941' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:32:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: osdmap e285: 12 total, 12 up, 12 in 2024-08-30T23:32:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:32:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:55.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:55.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:55.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:55.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:55.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:55.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:55.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:55.379 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:55.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:55.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:55.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:55.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:55.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:55.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:55.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:55.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:55.740 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:55 smithi106 ceph-mon[16603]: osdmap e286: 12 total, 12 up, 12 in 2024-08-30T23:32:55.740 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:32:55.740 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:32:55.740 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:55 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:32:55.740 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:55 smithi106 ceph-mon[16603]: pgmap v698: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 136/120312 objects degraded (0.113%); 5906/120312 objects misplaced (4.909%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:32:55.740 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3746798553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14244 2.3 e109/97 2:c35428fa:::benchmark_data_smithi173_47195_object14243:head v 109'957, mlcod=109'954)", 2024-08-30T23:32:55.751 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 406.843549354, 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213971894, 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883781+0000", 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883782+0000", 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999999e-07 2024-08-30T23:32:55.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.883844+0000", 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2580000000000001e-05 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096781+0000", 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212936878 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096785+0000", 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0389999999999998e-06 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096831+0000", 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6261999999999999e-05 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.096844+0000", 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2883e-05 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097712+0000", 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086761599999999998 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.097753+0000", 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0729999999999998e-05 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:56060 2.e 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 375.94785333999999, 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21409088000000001, 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 56060 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779478+0000", 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4869999999999999e-06 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779513+0000", 2024-08-30T23:32:55.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4384999999999997e-05 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779529+0000", 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6447999999999999e-05 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779537+0000", 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999993e-06 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779580+0000", 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3254999999999998e-05 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779658+0000", 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7711000000000002e-05 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779738+0000", 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9775000000000001e-05 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781534+0000", 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017964820000000001 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992319+0000", 2024-08-30T23:32:55.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21078449599999999 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993464+0000", 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001145294 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993511+0000", 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6344000000000002e-05 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993568+0000", 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6993000000000001e-05 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:56060 2.e e124/97 ondisk, result = 0)", 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 375.945513371, 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051657200000001, 2024-08-30T23:32:55.764 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:32:55.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9999999999999997e-07 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781821+0000", 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9480000000000001e-06 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990657+0000", 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20883629200000001 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990664+0000", 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.793e-06 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992305+0000", 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016408480000000001 2024-08-30T23:32:55.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992313+0000", 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1100000000000003e-06 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992333+0000", 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9981000000000001e-05 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.768 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67116 2.e 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 366.11539955400002, 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22641219700000001, 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.769 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67116 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611931+0000", 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612007+0000", 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5883999999999994e-05 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612021+0000", 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4007e-05 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612028+0000", 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4099999999999996e-06 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614110+0000", 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020827520000000002 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614270+0000", 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000159235 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614420+0000", 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015048199999999999 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617176+0000", 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027561619999999999 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617722+0000", 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054594799999999998 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838237+0000", 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22051536999999999 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838285+0000", 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7808e-05 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838343+0000", 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7340000000000003e-05 2024-08-30T23:32:55.775 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75429 2.1 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 358.76889153799999, 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24538217700000001, 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.776 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75429 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958440+0000", 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9410000000000001e-06 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958471+0000", 2024-08-30T23:32:55.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0919e-05 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958494+0000", 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3078000000000002e-05 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958504+0000", 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1190000000000003e-06 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958565+0000", 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1364000000000004e-05 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958669+0000", 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010414199999999999 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958772+0000", 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102979 2024-08-30T23:32:55.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.960745+0000", 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019727600000000001 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.066102+0000", 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.105356845 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203744+0000", 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13764249000000001 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203787+0000", 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.286e-05 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203821+0000", 2024-08-30T23:32:55.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3680000000000003e-05 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head, version: 114'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head@114'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 310.83517263599998, 2024-08-30T23:32:55.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19944226600000001, 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:32:55.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892158+0000", 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892186+0000", 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7710000000000001e-05 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089119+0000", 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19693307299999999 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089137+0000", 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7533e-05 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089255+0000", 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118318 2024-08-30T23:32:55.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089275+0000", 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9828000000000001e-05 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091600+0000", 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023249030000000001 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:126363 2.7 e160/138 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head v 160'7890, mlcod=119'2657)", 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 288.99770732899998, 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.217439837, 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:32:55.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729630+0000", 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.674e-06 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940470+0000", 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21084006199999999 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940478+0000", 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4390000000000003e-06 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940541+0000", 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3115999999999997e-05 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940571+0000", 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9981e-05 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947012+0000", 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064411269999999996 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947062+0000", 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0708999999999998e-05 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.792 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144840 2.1 2:8bdc20ef:::benchmark_data_smithi173_47195_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 264.13385166699999, 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21736361900000001, 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144840 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593481+0000", 2024-08-30T23:32:55.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3460000000000001e-06 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593490+0000", 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593512+0000", 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2410000000000001e-05 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593522+0000", 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4129999999999995e-06 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593705+0000", 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018286800000000001 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593993+0000", 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028869300000000002 2024-08-30T23:32:55.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.594314+0000", 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032096700000000002 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.599306+0000", 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049920049999999999 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603619+0000", 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043124419999999997 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810726+0000", 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20710743000000001 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810784+0000", 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7893999999999998e-05 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810842+0000", 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7791000000000003e-05 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144845 2.1 2:80fd4430:::benchmark_data_smithi173_47195_object47837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 264.12400668499998, 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20768498299999999, 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.799 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144845 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603326+0000", 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.373e-06 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603335+0000", 2024-08-30T23:32:55.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5449999999999992e-06 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603359+0000", 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3377e-05 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603370+0000", 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1759e-05 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603649+0000", 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027892500000000002 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603755+0000", 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010549500000000001 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603833+0000", 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8150000000000002e-05 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.609421+0000", 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0055884799999999998 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.640300+0000", 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030878974999999999 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810910+0000", 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.170609233 2024-08-30T23:32:55.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810964+0000", 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4549999999999998e-05 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.811008+0000", 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4121e-05 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.805 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155294 2.e 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 249.60344777500001, 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26817000400000002, 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.806 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155294 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123884+0000", 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3960000000000001e-06 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123889+0000", 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4820000000000002e-06 2024-08-30T23:32:55.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123902+0000", 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3202e-05 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123910+0000", 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8869999999999992e-06 2024-08-30T23:32:55.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123962+0000", 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1959000000000002e-05 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124045+0000", 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2850999999999997e-05 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124115+0000", 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9513999999999999e-05 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.127474+0000", 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033596149999999998 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.128135+0000", 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00066107399999999999 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.391962+0000", 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26382699500000001 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392009+0000", 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6669999999999999e-05 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392052+0000", 2024-08-30T23:32:55.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3359000000000003e-05 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 182/143 [PushOp(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head, version: 132'4708, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head@132'4708, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 248.28776901099999, 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25797871999999999, 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:32:55.813 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439562+0000", 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439584+0000", 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2395000000000002e-05 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693170+0000", 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25358568300000001 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693191+0000", 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1192999999999999e-05 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693349+0000", 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015837300000000001 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693372+0000", 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2869000000000001e-05 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.697540+0000", 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004167558 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168870 2.e 2:74d7d6f3:::benchmark_data_smithi173_47195_object71862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.11169281400001, 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223579366, 2024-08-30T23:32:55.817 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168870 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615640+0000", 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3590000000000002e-06 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615648+0000", 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8559999999999996e-06 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615665+0000", 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6563999999999999e-05 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615676+0000", 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1075e-05 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615800+0000", 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123905 2024-08-30T23:32:55.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615963+0000", 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016347900000000001 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616104+0000", 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140846 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620671+0000", 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045667540000000001 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620686+0000", 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.484e-05 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839134+0000", 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21844793100000001 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839172+0000", 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7737999999999997e-05 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839217+0000", 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5018999999999997e-05 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.823 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168873 2.1 2:8ff1cc28:::benchmark_data_smithi173_47195_object71865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.11162490300001, 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22707487400000001, 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168873 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:32:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615706+0000", 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2500000000000001e-07 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615708+0000", 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6970000000000002e-06 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615715+0000", 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8499999999999996e-06 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615720+0000", 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.395e-06 2024-08-30T23:32:55.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616068+0000", 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034794699999999998 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616173+0000", 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104145 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616295+0000", 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012253000000000001 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620669+0000", 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043740690000000004 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.621058+0000", 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038845600000000002 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842677+0000", 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22161980100000001 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842735+0000", 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7420000000000003e-05 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842780+0000", 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5139000000000003e-05 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168892 2.e 2:7a358138:::benchmark_data_smithi173_47195_object71884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.08920060299999, 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208213383, 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.832 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168892 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638131+0000", 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638136+0000", 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638147+0000", 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1119e-05 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638153+0000", 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8320000000000002e-06 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638212+0000", 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9219999999999999e-05 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638342+0000", 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012929599999999999 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638476+0000", 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013422799999999999 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643016+0000", 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045403120000000003 2024-08-30T23:32:55.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643358+0000", 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034153500000000002 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846238+0000", 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202880476 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846298+0000", 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9453000000000002e-05 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846343+0000", 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5059999999999999e-05 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168895 2.1 2:802748b8:::benchmark_data_smithi173_47195_object71887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.08496431, 2024-08-30T23:32:55.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21216137199999999, 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168895 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642368+0000", 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3080000000000002e-06 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642376+0000", 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6769999999999995e-06 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642392+0000", 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642400+0000", 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6329999999999996e-06 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642458+0000", 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8498000000000001e-05 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642574+0000", 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115822 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642662+0000", 2024-08-30T23:32:55.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655709+0000", 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013046791 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655767+0000", 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8464000000000001e-05 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854414+0000", 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19864683999999999 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854479+0000", 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5042999999999994e-05 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854527+0000", 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8195999999999997e-05 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.844 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168896 2.1 2:82321f29:::benchmark_data_smithi173_47195_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.08413128800001, 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211510109, 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.845 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168896 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643200+0000", 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6899999999999996e-07 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643204+0000", 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643215+0000", 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.041e-05 2024-08-30T23:32:55.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643220+0000", 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646422+0000", 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032022190000000001 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646565+0000", 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142564 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646696+0000", 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130877 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655755+0000", 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090594430000000004 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655773+0000", 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7263999999999999e-05 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854613+0000", 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19883995700000001 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854672+0000", 2024-08-30T23:32:55.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9522999999999999e-05 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854709+0000", 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6693000000000003e-05 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168897 2.1 2:8f1372f3:::benchmark_data_smithi173_47195_object71889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.08360619000001, 2024-08-30T23:32:55.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21982235999999999, 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168897 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643725+0000", 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643729+0000", 2024-08-30T23:32:55.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643740+0000", 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1079e-05 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643745+0000", 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2009999999999998e-06 2024-08-30T23:32:55.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651404+0000", 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007658916 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651490+0000", 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5811999999999998e-05 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651572+0000", 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2472999999999995e-05 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656527+0000", 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049544910000000001 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656537+0000", 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0497e-05 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863418+0000", 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20688067199999999 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863501+0000", 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2806000000000006e-05 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863546+0000", 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5430999999999999e-05 2024-08-30T23:32:55.857 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:193992 2.7 e200/182 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head v 200'12142, mlcod=200'12139)", 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 192.951758324, 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60714938399999996, 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9599999999999999e-07 2024-08-30T23:32:55.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775578+0000", 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8089999999999998e-06 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378923+0000", 2024-08-30T23:32:55.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60334498999999997 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378934+0000", 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0772e-05 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379004+0000", 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9783000000000006e-05 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379024+0000", 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0044e-05 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382681+0000", 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036575549999999998 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382721+0000", 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9734999999999998e-05 2024-08-30T23:32:55.862 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:194011 2.1 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 192.930044251, 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82807010000000003, 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:32:55.863 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194011 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:32:55.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797287+0000", 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.277e-06 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797292+0000", 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6419999999999998e-06 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797301+0000", 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0070000000000003e-06 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797307+0000", 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2809999999999997e-06 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797355+0000", 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8557000000000003e-05 2024-08-30T23:32:55.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797473+0000", 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117904 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797597+0000", 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012318100000000001 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:32:55.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.802546+0000", 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004948956 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.939024+0000", 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13647832200000001 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625288+0000", 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.686263866 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625326+0000", 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8028999999999998e-05 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625356+0000", 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0077999999999999e-05 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.869 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:92251 3.5 e251/206 3:a291f91a:::benchmark_data_smithi173_60780_object92250:head v 251'5738, mlcod=250'5735)", 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:29.501548+0000", 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 86.225781721999994, 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19789310800000001, 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.501548+0000", 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.501548+0000", 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.501549+0000", 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1999999999999998e-07 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.501651+0000", 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102058 2024-08-30T23:32:55.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.696610+0000", 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19495896400000001 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.696618+0000", 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3699999999999997e-06 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:32:55.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.696679+0000", 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1241000000000005e-05 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.696702+0000", 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3011e-05 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.699395+0000", 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00269352 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:32:55.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:29.699441+0000", 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6023999999999998e-05 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:32:55.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:55.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:55.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:56.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:56.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:55 smithi136 ceph-mon[20207]: osdmap e286: 12 total, 12 up, 12 in 2024-08-30T23:32:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:32:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:32:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:55 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:32:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:55 smithi136 ceph-mon[20207]: pgmap v698: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 136/120312 objects degraded (0.113%); 5906/120312 objects misplaced (4.909%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:32:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3746798553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:55 smithi173 ceph-mon[20554]: osdmap e286: 12 total, 12 up, 12 in 2024-08-30T23:32:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:32:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:32:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:55 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:32:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:55 smithi173 ceph-mon[20554]: pgmap v698: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 741 op/s; 136/120312 objects degraded (0.113%); 5906/120312 objects misplaced (4.909%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:32:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3746798553' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:32:56.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:56.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:56.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:56.261 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:56.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:56.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:56.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:56.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:56.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:56.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:32:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:32:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:32:56.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:32:56.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:32:56.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:32:56.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:32:56.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:32:56.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:32:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:32:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:32:56.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:32:56.667 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:32:56.668 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:32:56.737 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:56.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:56.741 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:32:56.741 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:32:56.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:56.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:32:56.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:32:56.802 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:32:56.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:32:56.829 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:32:56.829 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:32:56.859 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:32:56.859 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:32:56.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:56.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:32:56.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:56.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:56.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:57.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:57.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:32:57.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:32:57.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:32:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:57.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:32:57.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:32:57.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:57.401 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:57.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:57.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:57.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:57.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:57.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:57.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:57.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:57.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:57.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:58.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:58.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:58.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:32:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.185 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:58.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:32:58.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:58.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:58.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:58.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:58.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:32:58.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:32:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:58 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:32:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:58 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:32:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:58 smithi106 ceph-mon[16603]: pgmap v699: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 140/115809 objects degraded (0.121%); 5669/115809 objects misplaced (4.895%); 7.6 MiB/s, 121 objects/s recovering 2024-08-30T23:32:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:58 smithi106 ceph-mon[16603]: osdmap e287: 12 total, 12 up, 12 in 2024-08-30T23:32:58.655 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:32:58.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:58.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:58 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:32:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:58 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:32:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:58 smithi136 ceph-mon[20207]: pgmap v699: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 140/115809 objects degraded (0.121%); 5669/115809 objects misplaced (4.895%); 7.6 MiB/s, 121 objects/s recovering 2024-08-30T23:32:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:58 smithi136 ceph-mon[20207]: osdmap e287: 12 total, 12 up, 12 in 2024-08-30T23:32:58.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:58.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:58 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:32:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:58 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:32:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:58 smithi173 ceph-mon[20554]: pgmap v699: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 683 op/s; 140/115809 objects degraded (0.121%); 5669/115809 objects misplaced (4.895%); 7.6 MiB/s, 121 objects/s recovering 2024-08-30T23:32:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:58 smithi173 ceph-mon[20554]: osdmap e287: 12 total, 12 up, 12 in 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:moved 3 / 51 (5.88235%) 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.0052 -> 1.78536 (expected baseline 1.97379) 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 1 -> 1 pgs (0.235294 -> 0.235294 * mean) 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.11 with 8 -> 7 pgs (1.88235 -> 1.64706 * mean) 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:oload 110 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 0.05 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:32:58.873 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0084 2024-08-30T23:32:58.874 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0092 2024-08-30T23:32:58.874 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 weight 0.5971 -> 0.5471 2024-08-30T23:32:58.874 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 weight 0.5898 -> 0.5398 2024-08-30T23:32:58.874 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 weight 0.8072 -> 0.7572 2024-08-30T23:32:58.874 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 0.8264 -> 0.7764 2024-08-30T23:32:58.874 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:32:58.925 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:58.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:58.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:59.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:32:59.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:32:59.107 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:32:58 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:32:58.871+0000 7f2243404640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-08-30T23:32:59.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:32:59.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:32:59.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:59.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:32:59.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:32:59.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:32:59.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:32:59.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:32:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:32:59.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:32:59.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:32:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:32:59 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 140/108993 objects degraded (0.128%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:59.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:32:59.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:32:59 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 140/108993 objects degraded (0.128%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:59.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:32:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:32:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:32:59 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 140/108993 objects degraded (0.128%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:32:59.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:32:59.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:32:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:32:59.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:00.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:00.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:00.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:00.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:00.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:00.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:00.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:33:00.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:00.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:00.509 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:00.509 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:33:00.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:00.633 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:00.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:00.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:00 smithi136 ceph-mon[20207]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:33:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:00 smithi136 ceph-mon[20207]: pgmap v701: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 140/108993 objects degraded (0.128%); 4693/108993 objects misplaced (4.306%); 11 MiB/s, 175 objects/s recovering 2024-08-30T23:33:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3702351512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:00 smithi173 ceph-mon[20554]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:33:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:00 smithi173 ceph-mon[20554]: pgmap v701: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 140/108993 objects degraded (0.128%); 4693/108993 objects misplaced (4.306%); 11 MiB/s, 175 objects/s recovering 2024-08-30T23:33:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3702351512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:00 smithi106 ceph-mon[16603]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:33:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:00 smithi106 ceph-mon[16603]: pgmap v701: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 2.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 140/108993 objects degraded (0.128%); 4693/108993 objects misplaced (4.306%); 11 MiB/s, 175 objects/s recovering 2024-08-30T23:33:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3702351512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:00.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:00.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:00.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:01.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:01.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:01.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:01.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:01.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:01.290 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:01.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:01.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:01.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:01.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:01.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:01.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:01.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:01.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:01.721 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:01 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:01.721 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3702351512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:01.721 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:01 smithi106 ceph-mon[16603]: osdmap e288: 12 total, 12 up, 12 in 2024-08-30T23:33:01.747 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:01.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:01.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:01 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3702351512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:01 smithi136 ceph-mon[20207]: osdmap e288: 12 total, 12 up, 12 in 2024-08-30T23:33:01.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:01 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3702351512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:01.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:01 smithi173 ceph-mon[20554]: osdmap e288: 12 total, 12 up, 12 in 2024-08-30T23:33:01.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:01.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:01.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:02.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:02.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:02.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:02.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:02.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:02.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:02.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:02.463 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:02.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:02.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:02.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:02.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:02.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:02 smithi136 ceph-mon[20207]: pgmap v703: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 212/106749 objects degraded (0.199%); 4433/106749 objects misplaced (4.153%); 6.9 MiB/s, 109 objects/s recovering 2024-08-30T23:33:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:02 smithi173 ceph-mon[20554]: pgmap v703: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 212/106749 objects degraded (0.199%); 4433/106749 objects misplaced (4.153%); 6.9 MiB/s, 109 objects/s recovering 2024-08-30T23:33:02.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:02 smithi106 ceph-mon[16603]: pgmap v703: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 212/106749 objects degraded (0.199%); 4433/106749 objects misplaced (4.153%); 6.9 MiB/s, 109 objects/s recovering 2024-08-30T23:33:02.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:02.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:02.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:02.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:03.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:03.115 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:03.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:03.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:03.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:03.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:03.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:03.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:03.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:03.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:03.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:03.665 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:03.665 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:03.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:03.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:03.771 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:33:03.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:03.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:03.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:03.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:04.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:04.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:04.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:04.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:04.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:04.345 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:04.346 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:04.375 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:04.376 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:04.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:04.404 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:04.404 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:04.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:04.430 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:04.431 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:04.463 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:04.463 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:04.495 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:04.495 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:04.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:04.521 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:04.521 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:04.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:04.545 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:04.545 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:04.570 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:04.570 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:04.597 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:04.597 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:04.622 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:04.622 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:04.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:04.648 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:04.649 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:04.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:04.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:04 smithi136 ceph-mon[20207]: pgmap v704: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 755 op/s; 212/102168 objects degraded (0.208%); 4433/102168 objects misplaced (4.339%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:33:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/786625632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:04.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:04 smithi173 ceph-mon[20554]: pgmap v704: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 755 op/s; 212/102168 objects degraded (0.208%); 4433/102168 objects misplaced (4.339%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:33:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/786625632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:04.855 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:04 smithi106 ceph-mon[16603]: pgmap v704: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 2.1 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 755 op/s; 212/102168 objects degraded (0.208%); 4433/102168 objects misplaced (4.339%); 5.9 MiB/s, 93 objects/s recovering 2024-08-30T23:33:04.855 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/786625632' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:04.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:33:04.897 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:04.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:04.950 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:04.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:05.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:05.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:05.093 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:05.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:05.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:05.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:05.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:05.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:05.473 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:05.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:05.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:05.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:05.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:05.811 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:05 smithi106 ceph-mon[16603]: pgmap v705: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s; 212/95778 objects degraded (0.221%); 3407/95778 objects misplaced (3.557%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:33:05.811 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 212/95778 objects degraded (0.221%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:05 smithi173 ceph-mon[20554]: pgmap v705: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s; 212/95778 objects degraded (0.221%); 3407/95778 objects misplaced (3.557%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:33:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 212/95778 objects degraded (0.221%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:05.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:05.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:05.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:05.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:05.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:05.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:06.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:33:06.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:33:06.027 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:33:06.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:33:06.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:05 smithi136 ceph-mon[20207]: pgmap v705: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.9 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s; 212/95778 objects degraded (0.221%); 3407/95778 objects misplaced (3.557%); 7.3 MiB/s, 116 objects/s recovering 2024-08-30T23:33:06.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 212/95778 objects degraded (0.221%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:06.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:06.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:06.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:06.241 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:06.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:06.354 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:06.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:06.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:06.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:06.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:06.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:06.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:06.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:06.747 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:33:06.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1077964132' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:06.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:06 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:06.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:06.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:06.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:06.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:06.956 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:33:06.956 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:33:06.956 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:33:06.957 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:33:06.957 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:33:06.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:07.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1077964132' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:06 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1077964132' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:06 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:07.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:07.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:07.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:07.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:07.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:07.406 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:07.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:07.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:07.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:07.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:07.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:07.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:07.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:07.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:33:07.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:07.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:07.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:07.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:07 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:07.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:07 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:07.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:07 smithi173 ceph-mon[20554]: osdmap e289: 12 total, 12 up, 12 in 2024-08-30T23:33:07.990 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:07 smithi173 ceph-mon[20554]: pgmap v707: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 303/91278 objects degraded (0.332%); 3118/91278 objects misplaced (3.416%); 6.2 MiB/s, 99 objects/s recovering 2024-08-30T23:33:08.002 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:08.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:07 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:07 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:07 smithi136 ceph-mon[20207]: osdmap e289: 12 total, 12 up, 12 in 2024-08-30T23:33:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:07 smithi136 ceph-mon[20207]: pgmap v707: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 303/91278 objects degraded (0.332%); 3118/91278 objects misplaced (3.416%); 6.2 MiB/s, 99 objects/s recovering 2024-08-30T23:33:08.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.060 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:08.099 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:07 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:08.099 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:07 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:08.099 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:07 smithi106 ceph-mon[16603]: osdmap e289: 12 total, 12 up, 12 in 2024-08-30T23:33:08.099 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:07 smithi106 ceph-mon[16603]: pgmap v707: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.9 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 737 op/s; 303/91278 objects degraded (0.332%); 3118/91278 objects misplaced (3.416%); 6.2 MiB/s, 99 objects/s recovering 2024-08-30T23:33:08.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:08.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:08.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:08.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:08.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:08.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:08.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:33:08.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:08.582 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:08.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:08.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:08.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:08.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:08.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:08.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:08.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:09.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:09.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:09.073 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:09.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:09.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:09.121 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:09.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:09.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:09.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:09.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:09.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:09.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:09.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:09.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:09.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:09.745 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:33:09.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:09 smithi106 ceph-mon[16603]: osdmap e290: 12 total, 12 up, 12 in 2024-08-30T23:33:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:09 smithi106 ceph-mon[16603]: pgmap v709: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 303/85143 objects degraded (0.356%); 2284/85143 objects misplaced (2.683%); 7.6 MiB/s, 122 objects/s recovering 2024-08-30T23:33:09.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:09.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:09.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:09.993 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:09.994 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:09.994 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:09.994 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:09.994 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:09.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:10.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:10.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:09 smithi136 ceph-mon[20207]: osdmap e290: 12 total, 12 up, 12 in 2024-08-30T23:33:10.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:09 smithi136 ceph-mon[20207]: pgmap v709: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 303/85143 objects degraded (0.356%); 2284/85143 objects misplaced (2.683%); 7.6 MiB/s, 122 objects/s recovering 2024-08-30T23:33:10.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-30T23:33:10.069+0000 7ff9eaffd640 0 --2- 172.21.15.106:0/777387673 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9ec10ef70 0x7ff9ec19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-30T23:33:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:09 smithi173 ceph-mon[20554]: osdmap e290: 12 total, 12 up, 12 in 2024-08-30T23:33:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:09 smithi173 ceph-mon[20554]: pgmap v709: 17 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 303/85143 objects degraded (0.356%); 2284/85143 objects misplaced (2.683%); 7.6 MiB/s, 122 objects/s recovering 2024-08-30T23:33:10.115 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:33:10.115 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:33:10.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:33:10.115 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:33:10.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.115 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16161 2.6 e109/97 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head v 109'1027, mlcod=109'1024)", 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.01257222999999, 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212155128, 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099911+0000", 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5499999999999997e-07 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099933+0000", 2024-08-30T23:33:10.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1793999999999998e-05 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311098+0000", 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21116489199999999 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311105+0000", 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6920000000000002e-06 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311160+0000", 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4716999999999998e-05 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311179+0000", 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8814e-05 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312023+0000", 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00084434600000000005 2024-08-30T23:33:10.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312065+0000", 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2117999999999999e-05 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:56060 2.e e124/97 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head v 124'3464, mlcod=124'3461)", 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 390.33265333000003, 2024-08-30T23:33:10.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21259787399999999, 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:33:10.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779830+0000", 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-07 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779851+0000", 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1341000000000001e-05 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988300+0000", 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208448405 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988311+0000", 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1141000000000001e-05 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988383+0000", 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2733999999999999e-05 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988404+0000", 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0509999999999998e-05 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992259+0000", 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003854704 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992427+0000", 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016835099999999999 2024-08-30T23:33:10.124 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75397 2.1 e132/97 2:8c19b29d:::benchmark_data_smithi173_47195_object75396:head v 132'4751, mlcod=132'4748)", 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 373.50347130199998, 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22761530499999999, 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609012+0000", 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8199999999999998e-07 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609036+0000", 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4414999999999999e-05 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834165+0000", 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22512892100000001 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834172+0000", 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2589999999999997e-06 2024-08-30T23:33:10.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834226+0000", 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3919999999999999e-05 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.298e-05 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836584+0000", 2024-08-30T23:33:10.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023353750000000002 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836626+0000", 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2552999999999997e-05 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 373.48324429100001, 2024-08-30T23:33:10.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.240592844, 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:33:10.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629239+0000", 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629261+0000", 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2293000000000002e-05 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867840+0000", 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23857915299999999 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867849+0000", 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4940000000000003e-06 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867913+0000", 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3818000000000005e-05 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867935+0000", 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2024000000000001e-05 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869794+0000", 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001859018 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869831+0000", 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7239e-05 2024-08-30T23:33:10.133 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75429 2.1 e132/97 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head v 132'4755, mlcod=132'4752)", 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 373.15358427799998, 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24468055399999999, 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958899+0000", 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1939999999999999e-06 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958944+0000", 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4851999999999997e-05 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201529+0000", 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24258502100000001 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201540+0000", 2024-08-30T23:33:10.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0744000000000001e-05 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201630+0000", 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0168999999999997e-05 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201650+0000", 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9477999999999999e-05 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203542+0000", 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00189236 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203579+0000", 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6736000000000002e-05 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100665 2.6 e147/142 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head v 147'6316, mlcod=147'6313)", 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:33:10.138 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 336.76828970000003, 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214342806, 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:33:10.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344194+0000", 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344200+0000", 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5200000000000003e-06 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555870+0000", 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21167006199999999 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555878+0000", 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8420000000000005e-06 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555957+0000", 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8789999999999996e-05 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555979+0000", 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2206999999999999e-05 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.558491+0000", 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025112960000000001 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.558536+0000", 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5039e-05 2024-08-30T23:33:10.142 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100667 2.6 e147/142 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head v 147'6317, mlcod=147'6313)", 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 336.76440443400003, 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.273264648, 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:33:10.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348079+0000", 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1459999999999999e-06 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348086+0000", 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6649999999999997e-06 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615008+0000", 2024-08-30T23:33:10.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26692174899999999 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615016+0000", 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0909999999999993e-06 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615082+0000", 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6359000000000006e-05 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615102+0000", 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9945999999999998e-05 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621308+0000", 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006205828 2024-08-30T23:33:10.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621343+0000", 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4863999999999997e-05 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:146079 2.d e174/97 2:b3e1325e:::benchmark_data_smithi173_47195_object49071:head v 174'9276, mlcod=174'9273)", 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:33.409853+0000", 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.70262903000003, 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20166622200000001, 2024-08-30T23:33:10.149 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.409853+0000", 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.409853+0000", 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.409854+0000", 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.63e-07 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.409859+0000", 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6759999999999998e-06 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.587232+0000", 2024-08-30T23:33:10.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17737370699999999 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.587237+0000", 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3599999999999998e-06 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.587350+0000", 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112874 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.587374+0000", 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4678e-05 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.611232+0000", 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023857545000000001 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:33.611520+0000", 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028791899999999998 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.153 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:194011 2.1 e200/97 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head v 200'12204, mlcod=200'12201)", 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 207.314805643, 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82744930299999997, 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797678+0000", 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1800000000000005e-07 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797683+0000", 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6529999999999999e-06 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620982+0000", 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82329911700000002 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620996+0000", 2024-08-30T23:33:10.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3995999999999999e-05 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621133+0000", 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013682700000000001 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621198+0000", 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4368000000000004e-05 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625091+0000", 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003893194 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625126+0000", 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5429999999999998e-05 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:18000 3.7 3:e954d40a:::benchmark_data_smithi173_60780_object17999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:22.778914+0000", 2024-08-30T23:33:10.158 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 167.33356855700001, 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.209583305, 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 18000 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.778914+0000", 2024-08-30T23:33:10.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:10.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.778914+0000", 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.778915+0000", 2024-08-30T23:33:10.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.779098+0000", 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018369299999999999 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.779107+0000", 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9530000000000005e-06 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.779112+0000", 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6190000000000002e-06 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.779164+0000", 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1847e-05 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.779254+0000", 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0808000000000002e-05 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.779366+0000", 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011118 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.780188+0000", 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082260999999999999 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.785768+0000", 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0055796750000000001 2024-08-30T23:33:10.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.988401+0000", 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202632757 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.988455+0000", 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4613e-05 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.988497+0000", 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1922000000000003e-05 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:18011 3.7 3:eb57f387:::benchmark_data_smithi173_60780_object18010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:22.784527+0000", 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 167.327955804, 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205216592, 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.166 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 18011 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784527+0000", 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784527+0000", 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784527+0000", 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6499999999999998e-07 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784556+0000", 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8949999999999999e-05 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784567+0000", 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0684e-05 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784572+0000", 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9289999999999997e-06 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784620+0000", 2024-08-30T23:33:10.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8105000000000001e-05 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784734+0000", 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114133 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.784883+0000", 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014865599999999999 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.170 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.785535+0000", 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065251299999999996 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.988308+0000", 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20277241600000001 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.989643+0000", 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001335204 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.989703+0000", 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0155000000000003e-05 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:22.989743+0000", 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0082e-05 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:21992 3.b 3:df4f75f5:::benchmark_data_smithi173_60780_object21991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:10.172 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:25.276436+0000", 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.836046807, 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21213258500000001, 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 21992 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276436+0000", 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276436+0000", 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276437+0000", 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276498+0000", 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1155000000000007e-05 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276508+0000", 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0454e-05 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276513+0000", 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9690000000000002e-06 2024-08-30T23:33:10.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276553+0000", 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9753999999999999e-05 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276633+0000", 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9760999999999997e-05 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:33:10.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276728+0000", 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5237999999999999e-05 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.277492+0000", 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00076429099999999995 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.277989+0000", 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049673899999999995 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.177 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488499+0000", 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051017 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488536+0000", 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6732e-05 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488568+0000", 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.243e-05 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.178 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.15351.0:21992 3.b e214/206 ondisk, result = 0)", 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:25.278094+0000", 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.83438873200001, 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21048071700000001, 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.278094+0000", 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.278094+0000", 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.278094+0000", 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.278097+0000", 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0259999999999999e-06 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488439+0000", 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21034142 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488446+0000", 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8850000000000001e-06 2024-08-30T23:33:10.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488489+0000", 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3588999999999999e-05 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488495+0000", 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3550000000000004e-06 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488574+0000", 2024-08-30T23:33:10.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9908999999999997e-05 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:22013 3.7 3:ee8a83f6:::benchmark_data_smithi173_60780_object22012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:25.298529+0000", 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.813953459, 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20154677400000001, 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.183 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 22013 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298529+0000", 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298529+0000", 2024-08-30T23:33:10.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298530+0000", 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5799999999999999e-07 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298597+0000", 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6957000000000003e-05 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298604+0000", 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8029999999999997e-06 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298609+0000", 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0770000000000001e-06 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298647+0000", 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8727999999999999e-05 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298726+0000", 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8307999999999996e-05 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.298808+0000", 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2507000000000001e-05 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.299578+0000", 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00076976900000000001 2024-08-30T23:33:10.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.457717+0000", 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15813909800000001 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.499979+0000", 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.042261900999999998 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.500038+0000", 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8826000000000002e-05 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.500076+0000", 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8142000000000002e-05 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.15351.0:29340 3.b e219/206 ondisk, result = 0)", 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:32.435160+0000", 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 157.67732292900001, 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20893367199999999, 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.435160+0000", 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.435160+0000", 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.435160+0000", 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1900000000000002e-07 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.435164+0000", 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6260000000000001e-06 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.643311+0000", 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20814745700000001 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.643318+0000", 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3110000000000002e-06 2024-08-30T23:33:10.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.643961+0000", 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064337900000000002 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.643964+0000", 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.14e-06 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:32.644093+0000", 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012914 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:41088 3.2 3:4a9dd21f:::benchmark_data_smithi173_60780_object41087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:43.230043+0000", 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 146.882439393, 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20611860200000001, 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.194 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 41088 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230043+0000", 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230043+0000", 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230044+0000", 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3090000000000001e-06 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230098+0000", 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3667e-05 2024-08-30T23:33:10.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230110+0000", 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1642e-05 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230117+0000", 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1720000000000001e-06 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230165+0000", 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8228e-05 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230262+0000", 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6779000000000003e-05 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.230355+0000", 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3255000000000001e-05 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.231532+0000", 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011763870000000001 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.232074+0000", 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054265200000000002 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436104+0000", 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20402946499999999 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436138+0000", 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3997e-05 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436162+0000", 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4049e-05 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.15351.0:41088 3.2 e226/206 ondisk, result = 0)", 2024-08-30T23:33:10.201 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:43.231884+0000", 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 146.880598413, 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20428158099999999, 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.231884+0000", 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.231884+0000", 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.231885+0000", 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8700000000000005e-07 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.231889+0000", 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1409999999999998e-06 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436049+0000", 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.204159913 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436054+0000", 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2050000000000001e-06 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436094+0000", 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0117000000000002e-05 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.205 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:10.205 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436100+0000", 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0120000000000002e-06 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:43.436166+0000", 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5406000000000004e-05 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.206 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:74498 3.b 3:d56fca87:::benchmark_data_smithi173_60780_object74497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:12.017241+0000", 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 118.09524141199999, 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.201377155, 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.207 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 74498 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017241+0000", 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017241+0000", 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017243+0000", 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.621e-06 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017294+0000", 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0976000000000001e-05 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017309+0000", 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5443000000000001e-05 2024-08-30T23:33:10.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017316+0000", 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3390000000000004e-06 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017366+0000", 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9567999999999997e-05 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017469+0000", 2024-08-30T23:33:10.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010282500000000001 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.017571+0000", 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102013 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.019593+0000", 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002022598 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.019984+0000", 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039081700000000001 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.218548+0000", 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.198563458 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.218585+0000", 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7749000000000001e-05 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:12.218618+0000", 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2747999999999999e-05 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:75545 3.7 3:e894f768:::benchmark_data_smithi173_60780_object75544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:13.271744+0000", 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 116.84073836899999, 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200142875, 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75545 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271744+0000", 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271744+0000", 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271745+0000", 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9699999999999994e-07 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271772+0000", 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6543999999999999e-05 2024-08-30T23:33:10.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271783+0000", 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1208e-05 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271788+0000", 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3120000000000001e-06 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271838+0000", 2024-08-30T23:33:10.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0084999999999999e-05 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.271930+0000", 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1524e-05 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.272016+0000", 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5807e-05 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.274307+0000", 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022911239999999999 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.274793+0000", 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00048587800000000003 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.471819+0000", 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.197026808 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.471851+0000", 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1593000000000001e-05 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:13.471887+0000", 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5994999999999997e-05 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.219 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:103424 3.2 3:4caab458:::benchmark_data_smithi173_60780_object103423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 88.852859038000005, 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.221222537, 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103424 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:33:10.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259624+0000", 2024-08-30T23:33:10.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0800000000000004e-07 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259682+0000", 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7364999999999997e-05 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259692+0000", 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0686000000000001e-05 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259698+0000", 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.361e-06 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261702+0000", 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020041289999999999 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261778+0000", 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6148000000000003e-05 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261872+0000", 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4598000000000005e-05 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.263845+0000", 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001972419 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.264707+0000", 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086252099999999999 2024-08-30T23:33:10.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480753+0000", 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.216045457 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480795+0000", 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2605000000000003e-05 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480846+0000", 2024-08-30T23:33:10.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0540000000000001e-05 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:33:10.226 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:10.227 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:10.227 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:10.227 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:10.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:10.246 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:10.247 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:10.276 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:10.276 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:10.306 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:10.306 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:10.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:10.336 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:10.337 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:10.358 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:10.369 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:10.369 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:10.394 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:10.394 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:10.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:10.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:10.452 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:10.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:10.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:10.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.493 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:10.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:10.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:10.621 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:10.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:10.747 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:10.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:10.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:10.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:10.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:11.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:11.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:10 smithi136 ceph-mon[20207]: 3.f scrub starts 2024-08-30T23:33:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/777387673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:10 smithi136 ceph-mon[20207]: 3.3 scrub starts 2024-08-30T23:33:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 303/85143 objects degraded (0.356%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:10 smithi173 ceph-mon[20554]: 3.f scrub starts 2024-08-30T23:33:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/777387673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:10 smithi173 ceph-mon[20554]: 3.3 scrub starts 2024-08-30T23:33:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 303/85143 objects degraded (0.356%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:10 smithi106 ceph-mon[16603]: 3.f scrub starts 2024-08-30T23:33:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/777387673' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:10 smithi106 ceph-mon[16603]: 3.3 scrub starts 2024-08-30T23:33:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 303/85143 objects degraded (0.356%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:11.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:11.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:11.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:11.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:11.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:11.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:11.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:11.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:11.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:11.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:11.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:11.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:11.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:11.911 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:11.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:12.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:11 smithi136 ceph-mon[20207]: 3.f scrub ok 2024-08-30T23:33:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:11 smithi136 ceph-mon[20207]: 3.3 scrub ok 2024-08-30T23:33:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:11 smithi136 ceph-mon[20207]: pgmap v710: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 337/80664 objects degraded (0.418%); 2054/80664 objects misplaced (2.546%); 9.1 MiB/s, 146 objects/s recovering 2024-08-30T23:33:12.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:11 smithi173 ceph-mon[20554]: 3.f scrub ok 2024-08-30T23:33:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:11 smithi173 ceph-mon[20554]: 3.3 scrub ok 2024-08-30T23:33:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:11 smithi173 ceph-mon[20554]: pgmap v710: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 337/80664 objects degraded (0.418%); 2054/80664 objects misplaced (2.546%); 9.1 MiB/s, 146 objects/s recovering 2024-08-30T23:33:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:11 smithi106 ceph-mon[16603]: 3.f scrub ok 2024-08-30T23:33:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:11 smithi106 ceph-mon[16603]: 3.3 scrub ok 2024-08-30T23:33:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:11 smithi106 ceph-mon[16603]: pgmap v710: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 895 op/s; 337/80664 objects degraded (0.418%); 2054/80664 objects misplaced (2.546%); 9.1 MiB/s, 146 objects/s recovering 2024-08-30T23:33:12.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:12.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:12.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:12.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:12.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:12.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:12.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:12.557 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:12.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:33:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:12.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:12.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:12.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:12.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:12.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:12 smithi106 ceph-mon[16603]: 3.d scrub starts 2024-08-30T23:33:12.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:12.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:12.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:13.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:33:13.031 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:12 smithi136 ceph-mon[20207]: 3.d scrub starts 2024-08-30T23:33:13.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:13.045 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.067 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:13.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:12 smithi173 ceph-mon[20554]: 3.d scrub starts 2024-08-30T23:33:13.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:13.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:13.270 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:13.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:13.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:13.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:13.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:13.511 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:13.527 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:13.527 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:13.527 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:13.528 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:13.529 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:13.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:13.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:13.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:13.685 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:13.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:13.805 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:13.805 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:13.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.813 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:13.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:13.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:13.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:13 smithi136 ceph-mon[20207]: 3.d scrub ok 2024-08-30T23:33:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:13 smithi136 ceph-mon[20207]: osdmap e291: 12 total, 12 up, 12 in 2024-08-30T23:33:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:13 smithi136 ceph-mon[20207]: pgmap v712: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 337/78441 objects degraded (0.430%); 2054/78441 objects misplaced (2.619%); 5.2 MiB/s, 83 objects/s recovering 2024-08-30T23:33:14.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:13 smithi173 ceph-mon[20554]: 3.d scrub ok 2024-08-30T23:33:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:13 smithi173 ceph-mon[20554]: osdmap e291: 12 total, 12 up, 12 in 2024-08-30T23:33:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:13 smithi173 ceph-mon[20554]: pgmap v712: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 337/78441 objects degraded (0.430%); 2054/78441 objects misplaced (2.619%); 5.2 MiB/s, 83 objects/s recovering 2024-08-30T23:33:14.076 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:13 smithi106 ceph-mon[16603]: 3.d scrub ok 2024-08-30T23:33:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:13 smithi106 ceph-mon[16603]: osdmap e291: 12 total, 12 up, 12 in 2024-08-30T23:33:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:13 smithi106 ceph-mon[16603]: pgmap v712: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfilling, 15 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 337/78441 objects degraded (0.430%); 2054/78441 objects misplaced (2.619%); 5.2 MiB/s, 83 objects/s recovering 2024-08-30T23:33:14.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:33:14.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:14.232 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:14.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:14.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:14.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:14.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:14.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:33:14.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.625 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:14.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:14.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:14.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:14.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:14 smithi136 ceph-mon[20207]: 3.a scrub starts 2024-08-30T23:33:14.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:14 smithi136 ceph-mon[20207]: 3.a scrub ok 2024-08-30T23:33:14.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:14 smithi136 ceph-mon[20207]: 3.5 scrub starts 2024-08-30T23:33:14.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:14 smithi136 ceph-mon[20207]: osdmap e292: 12 total, 12 up, 12 in 2024-08-30T23:33:14.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:14 smithi136 ceph-mon[20207]: 3.5 scrub ok 2024-08-30T23:33:14.853 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:14.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:14.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:14.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:14.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:15.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:14 smithi173 ceph-mon[20554]: 3.a scrub starts 2024-08-30T23:33:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:14 smithi173 ceph-mon[20554]: 3.a scrub ok 2024-08-30T23:33:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:14 smithi173 ceph-mon[20554]: 3.5 scrub starts 2024-08-30T23:33:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:14 smithi173 ceph-mon[20554]: osdmap e292: 12 total, 12 up, 12 in 2024-08-30T23:33:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:14 smithi173 ceph-mon[20554]: 3.5 scrub ok 2024-08-30T23:33:15.089 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:14 smithi106 ceph-mon[16603]: 3.a scrub starts 2024-08-30T23:33:15.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:14 smithi106 ceph-mon[16603]: 3.a scrub ok 2024-08-30T23:33:15.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:14 smithi106 ceph-mon[16603]: 3.5 scrub starts 2024-08-30T23:33:15.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:14 smithi106 ceph-mon[16603]: osdmap e292: 12 total, 12 up, 12 in 2024-08-30T23:33:15.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:14 smithi106 ceph-mon[16603]: 3.5 scrub ok 2024-08-30T23:33:15.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:15.124 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:15.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:33:15.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:15.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:15.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:15.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:15.388 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:15.388 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:15.428 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:15.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:15.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:15.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:15.673 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:15.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:15.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:15.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:15.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:15.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:15.951 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:15 smithi106 ceph-mon[16603]: 3.7 scrub starts 2024-08-30T23:33:15.951 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:15 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:33:15.951 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:15 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:33:15.951 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:15 smithi106 ceph-mon[16603]: pgmap v714: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 337/73293 objects degraded (0.460%); 1574/73293 objects misplaced (2.148%); 3.9 MiB/s, 63 objects/s recovering 2024-08-30T23:33:15.952 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 337/78441 objects degraded (0.430%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:16.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:15 smithi136 ceph-mon[20207]: 3.7 scrub starts 2024-08-30T23:33:16.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:15 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:33:16.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:15 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:33:16.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:15 smithi136 ceph-mon[20207]: pgmap v714: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 337/73293 objects degraded (0.460%); 1574/73293 objects misplaced (2.148%); 3.9 MiB/s, 63 objects/s recovering 2024-08-30T23:33:16.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 337/78441 objects degraded (0.430%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:16.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:16.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:33:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.049 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:16.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:15 smithi173 ceph-mon[20554]: 3.7 scrub starts 2024-08-30T23:33:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:15 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:33:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:15 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:33:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:15 smithi173 ceph-mon[20554]: pgmap v714: 17 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 15 active+clean; 1.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 337/73293 objects degraded (0.460%); 1574/73293 objects misplaced (2.148%); 3.9 MiB/s, 63 objects/s recovering 2024-08-30T23:33:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 337/78441 objects degraded (0.430%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:16.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:16.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:16.206 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:16.234 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:16.234 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:16.265 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:16.265 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:16.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:16.294 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:16.294 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:16.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:16.322 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:16.322 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:16.356 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:16.356 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:16.388 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:16.390 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:16.390 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:16.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:16.422 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:16.422 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:16.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:16.452 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:16.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:16.486 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:16.486 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:16.519 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:16.519 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:16.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:16.554 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:16.554 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:16.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:16.584 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:16.584 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:16.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:16.711 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:16.793 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:33:16.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:16.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:16.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:16.963 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:16.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:16.993 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:49996 2.5 e122/97 2:aad518df:::benchmark_data_smithi173_47195_object49995:head v 122'3107, mlcod=122'3104)", 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 401.73324525999999, 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20284867600000001, 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:16.994 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258291+0000", 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.63e-07 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.258370+0000", 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8464e-05 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459751+0000", 2024-08-30T23:33:16.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.201380686 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459757+0000", 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.319e-06 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459801+0000", 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3912999999999998e-05 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.459814+0000", 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3548e-05 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.461105+0000", 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012902320000000001 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.461140+0000", 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5051e-05 2024-08-30T23:33:16.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.24460.0:109587 2.d e153/97 2:b377b128:::benchmark_data_smithi173_47195_object12579:head v 153'6909, mlcod=153'6905)", 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 332.08642166599998, 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18410296100000001, 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:16.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905115+0000", 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905116+0000", 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5770000000000001e-06 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.905126+0000", 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0356999999999999e-05 2024-08-30T23:33:17.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086753+0000", 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18162683099999999 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086760+0000", 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5379999999999999e-06 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086804+0000", 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4310000000000001e-05 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.086821+0000", 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7135000000000001e-05 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.089190+0000", 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023685709999999999 2024-08-30T23:33:17.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.089217+0000", 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7642000000000001e-05 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:2681 3.3 3:c917533a:::benchmark_data_smithi173_60780_object2680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 184.298721718, 2024-08-30T23:33:17.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20960189000000001, 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 2681 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692814+0000", 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692815+0000", 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0100000000000005e-07 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692841+0000", 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6024e-05 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692847+0000", 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3520000000000003e-06 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.692853+0000", 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2209999999999996e-06 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693408+0000", 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055510499999999996 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693502+0000", 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3960000000000002e-05 2024-08-30T23:33:17.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.693587+0000", 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5558999999999999e-05 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694000+0000", 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041311100000000002 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.701466+0000", 2024-08-30T23:33:17.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074651769999999999 2024-08-30T23:33:17.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902340+0000", 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200874052 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902381+0000", 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0929000000000001e-05 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902416+0000", 2024-08-30T23:33:17.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5799000000000001e-05 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:2681 3.3 e213/206 ondisk, result = 0)", 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 184.29676034900001, 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207646681, 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.012 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694776+0000", 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1e-07 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.694779+0000", 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5129999999999999e-06 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902275+0000", 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.207495916 2024-08-30T23:33:17.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902280+0000", 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8850000000000004e-06 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902329+0000", 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8656999999999998e-05 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902336+0000", 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.528e-06 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:12.902423+0000", 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6872000000000004e-05 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:21973 3.8 e214/206 3:16383432:::benchmark_data_smithi173_60780_object21972:head v 214'1387, mlcod=214'1385)", 2024-08-30T23:33:17.016 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.267943+0000", 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 171.723593139, 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21049256599999999, 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.267943+0000", 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.267943+0000", 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.267944+0000", 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2399999999999998e-07 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.268056+0000", 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112543 2024-08-30T23:33:17.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476599+0000", 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20854252600000001 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476608+0000", 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8579999999999995e-06 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476688+0000", 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9915999999999999e-05 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.476715+0000", 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6392000000000001e-05 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478396+0000", 2024-08-30T23:33:17.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016809349999999999 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478436+0000", 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9872e-05 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22010 3.3 3:c1c2ae0b:::benchmark_data_smithi173_60780_object22009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.293349+0000", 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 171.698187475, 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18727933099999999, 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22010 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293349+0000", 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293349+0000", 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293355+0000", 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3949999999999998e-06 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293391+0000", 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5880000000000002e-05 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293403+0000", 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1629e-05 2024-08-30T23:33:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293409+0000", 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.229e-06 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293467+0000", 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7692000000000003e-05 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293597+0000", 2024-08-30T23:33:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130495 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.293748+0000", 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015096900000000001 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294588+0000", 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083958899999999996 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294909+0000", 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032116499999999999 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480521+0000", 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18561213200000001 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480584+0000", 2024-08-30T23:33:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3246999999999999e-05 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480628+0000", 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3909000000000003e-05 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:22010 3.3 e214/206 ondisk, result = 0)", 2024-08-30T23:33:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 171.69661354300001, 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18571167899999999, 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294923+0000", 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.294925+0000", 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4109999999999999e-06 2024-08-30T23:33:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480459+0000", 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.185533737 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480466+0000", 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4810000000000003e-06 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480505+0000", 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9511999999999997e-05 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480514+0000", 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.636e-06 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.480634+0000", 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120469 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29340 3.b e219/206 3:d304e59a:::benchmark_data_smithi173_60780_object29339:head v 219'1853, mlcod=219'1849)", 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.433764+0000", 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.557771925, 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.208233679, 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433764+0000", 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433764+0000", 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433765+0000", 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.433814+0000", 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.901e-05 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640719+0000", 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20690529099999999 2024-08-30T23:33:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640728+0000", 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4509999999999992e-06 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640791+0000", 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2963e-05 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.640817+0000", 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5927000000000001e-05 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.641951+0000", 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011341540000000001 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.641998+0000", 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7185000000000003e-05 2024-08-30T23:33:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:29363 3.3 e219/206 ondisk, result = 0)", 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.454808+0000", 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.53672788399999, 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20210732300000001, 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454808+0000", 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454808+0000", 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454809+0000", 2024-08-30T23:33:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.23e-07 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.454813+0000", 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1389999999999997e-06 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656737+0000", 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20192416999999999 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656743+0000", 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9669999999999998e-06 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656785+0000", 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1836999999999999e-05 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656790+0000", 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3589999999999999e-06 2024-08-30T23:33:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.656916+0000", 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012512800000000001 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29375 3.6 e219/206 3:660484a6:::benchmark_data_smithi173_60780_object29374:head v 219'1852, mlcod=219'1848)", 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.461797+0000", 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.52973933000001, 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18554573199999999, 2024-08-30T23:33:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461797+0000", 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461797+0000", 2024-08-30T23:33:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461798+0000", 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2200000000000003e-07 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.461847+0000", 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9172000000000003e-05 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645099+0000", 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183251885 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645104+0000", 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.378e-06 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645145+0000", 2024-08-30T23:33:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0611999999999997e-05 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645158+0000", 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3193000000000001e-05 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.647309+0000", 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021510629999999999 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.647343+0000", 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3606999999999999e-05 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:29376 3.8 e219/206 3:1bf89ba2:::benchmark_data_smithi173_60780_object29375:head v 219'1864, mlcod=219'1860)", 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:32.462195+0000", 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.529341061, 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183932077, 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462195+0000", 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462195+0000", 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462196+0000", 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0800000000000005e-07 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.462268+0000", 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1931e-05 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.644961+0000", 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18269360300000001 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.644968+0000", 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3990000000000001e-06 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645014+0000", 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6350000000000002e-05 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.645027+0000", 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2822e-05 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646090+0000", 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001063587 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:32.646127+0000", 2024-08-30T23:33:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6876999999999999e-05 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41096 3.3 3:c1280440:::benchmark_data_smithi173_60780_object41095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.75155067399999, 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19616752700000001, 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41096 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.239986+0000", 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5000000000000001e-07 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240040+0000", 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3548999999999998e-05 2024-08-30T23:33:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240049+0000", 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3219999999999999e-06 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240057+0000", 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.571e-06 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240104+0000", 2024-08-30T23:33:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6649e-05 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240188+0000", 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4746999999999993e-05 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:33:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.240321+0000", 2024-08-30T23:33:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013225499999999999 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241598+0000", 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012777960000000001 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.242242+0000", 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064343400000000002 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436092+0000", 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.193849724 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436127+0000", 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.519e-05 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436153+0000", 2024-08-30T23:33:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6339999999999999e-05 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:41096 3.3 e226/206 ondisk, result = 0)", 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.749721212, 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19434237100000001, 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241815+0000", 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.89e-07 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.241822+0000", 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4010000000000002e-06 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436053+0000", 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19423099599999999 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436057+0000", 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3529999999999997e-06 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436086+0000", 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-05 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436089+0000", 2024-08-30T23:33:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0639999999999998e-06 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.436157+0000", 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8638000000000003e-05 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:60810 3.1 3:826953a7:::benchmark_data_smithi173_60780_object60809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:59.893893+0000", 2024-08-30T23:33:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 137.09764346599999, 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.187316551, 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60810 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893893+0000", 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893893+0000", 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893894+0000", 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0899999999999999e-06 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893943+0000", 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8826000000000003e-05 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893951+0000", 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4859999999999997e-06 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.893958+0000", 2024-08-30T23:33:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4169999999999997e-06 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.894004+0000", 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6424000000000001e-05 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.894101+0000", 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7459000000000003e-05 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.894190+0000", 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8355000000000004e-05 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895484+0000", 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001294447 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.897905+0000", 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024205699999999999 2024-08-30T23:33:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081147+0000", 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183242612 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081181+0000", 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3503000000000001e-05 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081209+0000", 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8362000000000001e-05 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:60810 3.1 e236/230 ondisk, result = 0)", 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:59.895762+0000", 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 137.095774041, 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.185451543, 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895762+0000", 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895762+0000", 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895763+0000", 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.44e-07 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:59.895769+0000", 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081103+0000", 2024-08-30T23:33:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18533458799999999 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081108+0000", 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.69e-06 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081140+0000", 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1578999999999997e-05 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081144+0000", 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.437e-06 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:00.081214+0000", 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9638e-05 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74470 3.3 3:c5e97d47:::benchmark_data_smithi173_60780_object74469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.006499568, 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20435176499999999, 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74470 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985037+0000", 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8299999999999996e-07 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985109+0000", 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2125000000000005e-05 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985120+0000", 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0346999999999999e-05 2024-08-30T23:33:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.985125+0000", 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0200000000000002e-06 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.986495+0000", 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013701309999999999 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.986625+0000", 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013006000000000001 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.986775+0000", 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149863 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.988511+0000", 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017362969999999999 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.002799+0000", 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014288175 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.189318+0000", 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18651902300000001 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.189352+0000", 2024-08-30T23:33:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3485999999999998e-05 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.189388+0000", 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6554999999999998e-05 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74488 3.3 3:cadebabe:::benchmark_data_smithi173_60780_object74487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:12.004501+0000", 2024-08-30T23:33:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 124.98703467999999, 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19292046400000001, 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74488 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004501+0000", 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004501+0000", 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004502+0000", 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9999999999999996e-07 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004566+0000", 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4166999999999997e-05 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004576+0000", 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-06 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004581+0000", 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8069999999999999e-06 2024-08-30T23:33:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004629+0000", 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7555000000000001e-05 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004752+0000", 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012330199999999999 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:33:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.004895+0000", 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014360999999999999 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.006892+0000", 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019964190000000001 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017722+0000", 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010830367 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197329+0000", 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.179606565 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197381+0000", 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2658999999999998e-05 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.197422+0000", 2024-08-30T23:33:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0513000000000003e-05 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:74498 3.b e243/206 3:d56fca87:::benchmark_data_smithi173_60780_object74497:head v 243'4726, mlcod=243'4724)", 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 124.97379366, 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20062028400000001, 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017743+0000", 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.017800+0000", 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6663999999999998e-05 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216278+0000", 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19847857899999999 2024-08-30T23:33:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216285+0000", 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.37e-06 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216334+0000", 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9434999999999997e-05 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.216349+0000", 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4932e-05 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.218333+0000", 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019840489999999999 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.218363+0000", 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9776000000000001e-05 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:129305 3.1 3:8cf64449:::benchmark_data_smithi173_60780_object25851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:32:08.868358+0000", 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.123178052, 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18904068900000001, 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 129305 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868358+0000", 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868358+0000", 2024-08-30T23:33:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868359+0000", 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.16e-07 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868363+0000", 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8650000000000003e-06 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868375+0000", 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2536e-05 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868382+0000", 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8220000000000001e-06 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868434+0000", 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1870999999999999e-05 2024-08-30T23:33:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868545+0000", 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111222 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.868648+0000", 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102883 2024-08-30T23:33:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.872019+0000", 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003370938 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.887867+0000", 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015848086000000001 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.057300+0000", 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.169433104 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.057357+0000", 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6376999999999999e-05 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.057399+0000", 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2069e-05 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:129305 3.1 e266/230 ondisk, result = 0)", 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:32:08.872484+0000", 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.119051861000003, 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.184921367, 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.872484+0000", 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.872484+0000", 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.872485+0000", 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7899999999999998e-07 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:08.872492+0000", 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.883e-06 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.054071+0000", 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18157912300000001 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.054077+0000", 2024-08-30T23:33:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2210000000000002e-06 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.057285+0000", 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032080110000000002 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.057294+0000", 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7809999999999994e-06 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:32:09.057406+0000", 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111669 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:17.108 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:17.108 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:16 smithi136 ceph-mon[20207]: 3.7 scrub ok 2024-08-30T23:33:17.108 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2135666160' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:17.108 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:16 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:17.109 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3767647663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:17.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:17.110 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:17.111 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:17.111 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:17.111 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:17.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:16 smithi173 ceph-mon[20554]: 3.7 scrub ok 2024-08-30T23:33:17.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2135666160' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:17.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:16 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:17.111 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3767647663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:17.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:16 smithi106 ceph-mon[16603]: 3.7 scrub ok 2024-08-30T23:33:17.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2135666160' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:17.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:16 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:17.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3767647663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:17.140 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:17.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:17.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:17.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:17.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:17.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:17.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:17.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:17.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:17.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:17.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:17.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:17.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:17.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:17.953 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:17 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:33:17.954 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:17 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:17.954 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:17 smithi106 ceph-mon[16603]: osdmap e293: 12 total, 12 up, 12 in 2024-08-30T23:33:17.954 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:17 smithi106 ceph-mon[16603]: pgmap v716: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 15 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 337/69834 objects degraded (0.483%); 1317/69834 objects misplaced (1.886%); 5.0 MiB/s, 80 objects/s recovering 2024-08-30T23:33:18.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:18.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:17 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:33:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:17 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:17 smithi173 ceph-mon[20554]: osdmap e293: 12 total, 12 up, 12 in 2024-08-30T23:33:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:17 smithi173 ceph-mon[20554]: pgmap v716: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 15 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 337/69834 objects degraded (0.483%); 1317/69834 objects misplaced (1.886%); 5.0 MiB/s, 80 objects/s recovering 2024-08-30T23:33:18.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:18.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:18.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:18.186 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:17 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:33:18.186 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:17 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:18.186 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:17 smithi136 ceph-mon[20207]: osdmap e293: 12 total, 12 up, 12 in 2024-08-30T23:33:18.186 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:17 smithi136 ceph-mon[20207]: pgmap v716: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 15 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 601 op/s; 337/69834 objects degraded (0.483%); 1317/69834 objects misplaced (1.886%); 5.0 MiB/s, 80 objects/s recovering 2024-08-30T23:33:18.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:33:18.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:18.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:18.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.430 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:18.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:18.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:18.556 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:18.556 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:18.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:18.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:18.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:18.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:18.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:18.925 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:18.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:33:18.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:18.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:19.036 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:18 smithi136 ceph-mon[20207]: 3.2 scrub starts 2024-08-30T23:33:19.036 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:18 smithi136 ceph-mon[20207]: 3.2 scrub ok 2024-08-30T23:33:19.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:18 smithi106 ceph-mon[16603]: 3.2 scrub starts 2024-08-30T23:33:19.071 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:18 smithi106 ceph-mon[16603]: 3.2 scrub ok 2024-08-30T23:33:19.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:18 smithi173 ceph-mon[20554]: 3.2 scrub starts 2024-08-30T23:33:19.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:18 smithi173 ceph-mon[20554]: 3.2 scrub ok 2024-08-30T23:33:19.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:187047 3.5 3:ac4df910:::benchmark_data_smithi173_60780_object83593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:19.166599+0000", 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0020565620000000001, 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002079787, 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:19.169 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 187047 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166599+0000", 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166599+0000", 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166600+0000", 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166604+0000", 2024-08-30T23:33:19.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6179999999999997e-06 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166613+0000", 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6840000000000002e-06 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166618+0000", 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.826e-06 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166656+0000", 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166775+0000", 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118622 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:19.166887+0000", 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112277 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:19.173 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:19.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:19.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:19.174 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:33:19.174 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:33:19.174 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:19.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:19.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:19.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.357 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:19.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:19.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:19.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:19.573 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:19.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:19.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:19.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:19.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:19.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:19.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:19.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:33:19.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:19.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:19 smithi106 ceph-mon[16603]: 3.e scrub starts 2024-08-30T23:33:19.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:19 smithi106 ceph-mon[16603]: 3.e scrub ok 2024-08-30T23:33:19.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:19 smithi106 ceph-mon[16603]: pgmap v717: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 337/64167 objects degraded (0.525%); 1317/64167 objects misplaced (2.052%); 4.5 MiB/s, 72 objects/s recovering 2024-08-30T23:33:19.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:19.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:19 smithi173 ceph-mon[20554]: 3.e scrub starts 2024-08-30T23:33:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:19 smithi173 ceph-mon[20554]: 3.e scrub ok 2024-08-30T23:33:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:19 smithi173 ceph-mon[20554]: pgmap v717: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 337/64167 objects degraded (0.525%); 1317/64167 objects misplaced (2.052%); 4.5 MiB/s, 72 objects/s recovering 2024-08-30T23:33:20.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:20.121 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:20.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:20.174 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:20.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:20.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:20.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:19 smithi136 ceph-mon[20207]: 3.e scrub starts 2024-08-30T23:33:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:19 smithi136 ceph-mon[20207]: 3.e scrub ok 2024-08-30T23:33:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:19 smithi136 ceph-mon[20207]: pgmap v717: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 713 op/s; 337/64167 objects degraded (0.525%); 1317/64167 objects misplaced (2.052%); 4.5 MiB/s, 72 objects/s recovering 2024-08-30T23:33:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:20.290 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:20.291 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:20.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:20.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:20.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:33:20.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:20.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:20.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:20.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:20.739 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:20.739 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:33:20.739 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:33:20.739 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:33:20.739 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:20.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:20.777 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:20.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:20 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 337/64167 objects degraded (0.525%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:20.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:20.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:20.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:20.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:20 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 337/64167 objects degraded (0.525%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:20.990 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:20.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:33:21.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:20 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 337/64167 objects degraded (0.525%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:21.077 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:21.110 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:21.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:21.208 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:21.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:21.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:21.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:21.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:21.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:33:21.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:33:21.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:21.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:21.665 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:21.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:21.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:21.781 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:21.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:21.833 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:21.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:21.972 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:21.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:22.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:21 smithi173 ceph-mon[20554]: pgmap v718: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 337/59064 objects degraded (0.571%); 1100/59064 objects misplaced (1.862%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:33:22.078 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:21 smithi136 ceph-mon[20207]: pgmap v718: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 337/59064 objects degraded (0.571%); 1100/59064 objects misplaced (1.862%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:33:22.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:21 smithi106 ceph-mon[16603]: pgmap v718: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.2 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 807 op/s; 337/59064 objects degraded (0.571%); 1100/59064 objects misplaced (1.862%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:33:22.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:22.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:22.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:22.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:22.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:22.259 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:22.260 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:22.260 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:22.260 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:22.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:14227 2.2 2:4af79d78:::benchmark_data_smithi173_47195_object14226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:33:22.341 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 433.464169648, 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24349405099999999, 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14227 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875039+0000", 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046e-06 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875091+0000", 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2556999999999998e-05 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875103+0000", 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1331e-05 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875108+0000", 2024-08-30T23:33:22.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1549999999999998e-06 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875152+0000", 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4161e-05 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875235+0000", 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2814999999999999e-05 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875316+0000", 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699000000000005e-05 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875841+0000", 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052407300000000001 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.037068+0000", 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.161227964 2024-08-30T23:33:22.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118454+0000", 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.081385499 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118498+0000", 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3513000000000002e-05 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118532+0000", 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4238e-05 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:14227 2.2 e109/97 ondisk, result = 0)", 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 433.46321305700002, 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.242546031, 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875995+0000", 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0100000000000004e-07 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875999+0000", 2024-08-30T23:33:22.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.177e-06 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.117926+0000", 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.241927006 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.117934+0000", 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5290000000000003e-06 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118441+0000", 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050753599999999995 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118448+0000", 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6619999999999999e-06 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118540+0000", 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2419999999999999e-05 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:16921 2.2 2:4bb89a5b:::benchmark_data_smithi173_47195_object16920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:10.678480+0000", 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 431.66072758799999, 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22023874399999999, 2024-08-30T23:33:22.353 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 16921 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678480+0000", 2024-08-30T23:33:22.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678480+0000", 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678481+0000", 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.125e-06 2024-08-30T23:33:22.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678534+0000", 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2828000000000003e-05 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678545+0000", 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0908e-05 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678550+0000", 2024-08-30T23:33:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1259999999999997e-06 2024-08-30T23:33:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678592+0000", 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2175999999999997e-05 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678667+0000", 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4757000000000003e-05 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.678743+0000", 2024-08-30T23:33:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5879999999999999e-05 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.679411+0000", 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066885499999999999 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.679624+0000", 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021280499999999999 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.898640+0000", 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.219015931 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.898685+0000", 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4543999999999999e-05 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:10.898718+0000", 2024-08-30T23:33:22.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3809000000000001e-05 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:17737 2.2 2:4fd7dd2a:::benchmark_data_smithi173_47195_object17736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 430.80895781999999, 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22122945399999999, 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17737 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530250+0000", 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5199999999999995e-07 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530297+0000", 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6873999999999999e-05 2024-08-30T23:33:22.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530308+0000", 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1147e-05 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530315+0000", 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:33:22.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530355+0000", 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0649999999999999e-05 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530438+0000", 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3207999999999993e-05 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.530520+0000", 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1971999999999994e-05 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531149+0000", 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000628139 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531405+0000", 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000256142 2024-08-30T23:33:22.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751373+0000", 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.219967995 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751433+0000", 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0591000000000003e-05 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751479+0000", 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5661999999999998e-05 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:17737 2.2 e111/97 ondisk, result = 0)", 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:11.531482+0000", 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 430.80772520400001, 2024-08-30T23:33:22.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22000273300000001, 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531482+0000", 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531482+0000", 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531483+0000", 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6800000000000001e-07 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.531486+0000", 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0520000000000002e-06 2024-08-30T23:33:22.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751313+0000", 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.219827576 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751319+0000", 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2750000000000003e-06 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751358+0000", 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9786999999999997e-05 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751367+0000", 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5939999999999994e-06 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:11.751485+0000", 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011798099999999999 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.372 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:36436 2.2 2:4f13755f:::benchmark_data_smithi173_47195_object36435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 417.35604066100001, 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23381564699999999, 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36436 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:33:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983169+0000", 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9470000000000002e-06 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983211+0000", 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1959000000000003e-05 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983229+0000", 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8508e-05 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983238+0000", 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9549999999999998e-06 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983294+0000", 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5605e-05 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983377+0000", 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3768000000000002e-05 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983458+0000", 2024-08-30T23:33:22.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0272000000000007e-05 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.984672+0000", 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214449 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.985077+0000", 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040527100000000002 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216885+0000", 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.231807244 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216945+0000", 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0545999999999999e-05 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216982+0000", 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7122999999999997e-05 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.379 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:36447 2.2 2:42c8382e:::benchmark_data_smithi173_47195_object36446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 417.35096926099999, 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31128452299999998, 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.380 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36447 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988239+0000", 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4e-07 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988315+0000", 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5585000000000002e-05 2024-08-30T23:33:22.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988325+0000", 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0549e-05 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988330+0000", 2024-08-30T23:33:22.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9729999999999997e-06 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988371+0000", 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1201000000000001e-05 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988453+0000", 2024-08-30T23:33:22.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1965000000000006e-05 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988536+0000", 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3088000000000001e-05 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.989696+0000", 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001159979 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.989973+0000", 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027723299999999999 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299419+0000", 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30944559100000002 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299477+0000", 2024-08-30T23:33:22.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7568000000000002e-05 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299523+0000", 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5951000000000001e-05 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.387 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(2.b 153/133 [PushOp(2:d76b2f20:::benchmark_data_smithi173_47195_object57342:head, version: 125'3537, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d76b2f20:::benchmark_data_smithi173_47195_object57342:head@125'3537, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:46.081333+0000", 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 336.25787481499998, 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.251162106, 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081333+0000", 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081333+0000", 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081334+0000", 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2699999999999999e-06 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081392+0000", 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8628999999999997e-05 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330558+0000", 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24916590999999999 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330571+0000", 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2774e-05 2024-08-30T23:33:22.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330638+0000", 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7082999999999995e-05 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330646+0000", 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0120000000000005e-06 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.332495+0000", 2024-08-30T23:33:22.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018484280000000001 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155144 2.b 2:dec0e07e:::benchmark_data_smithi173_47195_object58136:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.61795081700001, 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24062692899999999, 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.392 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155144 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:33:22.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721258+0000", 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.688e-06 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721264+0000", 2024-08-30T23:33:22.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5060000000000003e-06 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721277+0000", 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3345e-05 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721295+0000", 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7598e-05 2024-08-30T23:33:22.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721850+0000", 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055514699999999998 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721861+0000", 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1009000000000001e-05 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.932731+0000", 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.210870364 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.932895+0000", 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016375999999999999 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.933065+0000", 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000170104 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943014+0000", 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099494240000000001 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943076+0000", 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1239000000000001e-05 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943195+0000", 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118923 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.960869+0000", 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017674390000000002 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.960910+0000", 2024-08-30T23:33:22.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1467999999999997e-05 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.961883+0000", 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00097296399999999999 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155158 2.b 2:dcb1d3fb:::benchmark_data_smithi173_47195_object58150:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.61789437900001, 2024-08-30T23:33:22.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24126403199999999, 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155158 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4999999999999998e-07 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721316+0000", 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1900000000000002e-06 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721321+0000", 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4450000000000004e-06 2024-08-30T23:33:22.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721324+0000", 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721883+0000", 2024-08-30T23:33:22.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055905600000000005 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721891+0000", 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1509999999999998e-06 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937082+0000", 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.215191454 2024-08-30T23:33:22.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937182+0000", 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9567000000000004e-05 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937273+0000", 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1305999999999996e-05 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943145+0000", 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058718049999999999 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948639+0000", 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054946880000000002 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948721+0000", 2024-08-30T23:33:22.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1936999999999998e-05 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.961943+0000", 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013221568 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.962004+0000", 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0990999999999999e-05 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.962577+0000", 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057307400000000002 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155229 2.b 2:ddba2b2a:::benchmark_data_smithi173_47195_object58221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.61787726599999, 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.248255843, 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.413 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155229 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2000000000000001e-07 2024-08-30T23:33:22.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721332+0000", 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3260000000000001e-06 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721336+0000", 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0759999999999996e-06 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721339+0000", 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.569e-06 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721916+0000", 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057626300000000004 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721923+0000", 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5530000000000004e-06 2024-08-30T23:33:22.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940166+0000", 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21824272 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940264+0000", 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8392000000000003e-05 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940350+0000", 2024-08-30T23:33:22.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5668999999999996e-05 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.945936+0000", 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055864249999999999 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948776+0000", 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028393670000000002 2024-08-30T23:33:22.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957244+0000", 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084679639999999997 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.968623+0000", 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011379146999999999 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.968683+0000", 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0170999999999997e-05 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969586+0000", 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090298099999999997 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155284 2.b 2:d661bc27:::benchmark_data_smithi173_47195_object58276:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:33:22.421 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.61786471300002, 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24904695199999999, 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.422 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155284 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0200000000000001e-07 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721344+0000", 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.519e-06 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721347+0000", 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.118e-06 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721351+0000", 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5429999999999998e-06 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721957+0000", 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060615599999999995 2024-08-30T23:33:22.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721963+0000", 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3619999999999997e-06 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943241+0000", 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221278947 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943333+0000", 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1094000000000006e-05 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943419+0000", 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6588999999999994e-05 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948733+0000", 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053133649999999996 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951543+0000", 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002810819 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957285+0000", 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057415169999999998 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969633+0000", 2024-08-30T23:33:22.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012348181999999999 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969670+0000", 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6854000000000003e-05 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970390+0000", 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071968499999999996 2024-08-30T23:33:22.429 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155286 2.b 2:da5a53b4:::benchmark_data_smithi173_47195_object58278:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.61785269500001, 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24979741499999999, 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155286 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:33:22.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1199999999999999e-07 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721356+0000", 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2079999999999999e-06 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721360+0000", 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8750000000000002e-06 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721363+0000", 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7389999999999999e-06 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721987+0000", 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00062452099999999995 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721998+0000", 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277000000000001e-05 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.945970+0000", 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.223971225 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.946066+0000", 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6781999999999996e-05 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.946148+0000", 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1522999999999999e-05 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951509+0000", 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053612979999999996 2024-08-30T23:33:22.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.954537+0000", 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030273959999999999 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957323+0000", 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002785839 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970435+0000", 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013112342000000001 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970469+0000", 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4140000000000002e-05 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.971152+0000", 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068303799999999996 2024-08-30T23:33:22.437 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155298 2.b 2:dd53a97e:::benchmark_data_smithi173_47195_object58290:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.61784164199997, 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25834940200000001, 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.438 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155298 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2000000000000001e-07 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721367+0000", 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0950000000000001e-06 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721371+0000", 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.783e-06 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721373+0000", 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6280000000000001e-06 2024-08-30T23:33:22.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.722015+0000", 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064132200000000005 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.722020+0000", 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.028e-06 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948797+0000", 2024-08-30T23:33:22.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22677699400000001 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948892+0000", 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5272000000000005e-05 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948978+0000", 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6365000000000004e-05 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951655+0000", 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026768669999999999 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.954636+0000", 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029808159999999998 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957360+0000", 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002723821 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.978592+0000", 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021232183000000002 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.978633+0000", 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0460000000000002e-05 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.979715+0000", 2024-08-30T23:33:22.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001082548 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193943 2.b 2:d4b90c03:::benchmark_data_smithi173_47195_object96935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 219.631085353, 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42676301799999999, 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193943 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708123+0000", 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.09e-07 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708129+0000", 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5119999999999997e-06 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708141+0000", 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243999999999999e-05 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708147+0000", 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1920000000000002e-06 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709291+0000", 2024-08-30T23:33:22.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011439899999999999 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709428+0000", 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013711600000000001 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709522+0000", 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3832000000000006e-05 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.714473+0000", 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049515779999999999 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.714764+0000", 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029112000000000002 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.718006+0000", 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032420859999999999 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134491+0000", 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41648495699999999 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134534+0000", 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2568000000000003e-05 2024-08-30T23:33:22.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134885+0000", 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035101400000000002 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:193943 2.b e200/182 ondisk, result = 0)", 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 219.625263187, 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42094626899999998, 2024-08-30T23:33:22.453 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713945+0000", 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9500000000000003e-07 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713948+0000", 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134314+0000", 2024-08-30T23:33:22.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42036561300000003 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134335+0000", 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1803000000000001e-05 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134478+0000", 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014253700000000001 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134487+0000", 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5469999999999997e-06 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134890+0000", 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040388000000000002 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(2.b e200,182 2:d6a96fd5:::benchmark_data_smithi173_47195_object12191:head,153'6821)", 2024-08-30T23:33:22.457 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 219.478967023, 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66937705700000005, 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860247+0000", 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3439999999999997e-06 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860257+0000", 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0582e-05 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529079+0000", 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66882161500000004 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529086+0000", 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8430000000000004e-06 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529174+0000", 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8320999999999997e-05 2024-08-30T23:33:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529617+0000", 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044335199999999999 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22811 3.c 3:362f756c:::benchmark_data_smithi173_60780_object22810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 176.33416090899999, 2024-08-30T23:33:22.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32742973199999997, 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22811 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005047+0000", 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005105+0000", 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7797999999999998e-05 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005114+0000", 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0319999999999992e-06 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005119+0000", 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5820000000000003e-06 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005151+0000", 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2404999999999999e-05 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005230+0000", 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9212e-05 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005314+0000", 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3540000000000003e-05 2024-08-30T23:33:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005956+0000", 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064215699999999997 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.006179+0000", 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022317300000000001 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332412+0000", 2024-08-30T23:33:22.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.326233314 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332449+0000", 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6671999999999997e-05 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332476+0000", 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7161999999999998e-05 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:34430 3.5 3:a7266b30:::benchmark_data_smithi173_60780_object34429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:36.154464+0000", 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 166.184743677, 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.217586852, 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 34430 2024-08-30T23:33:22.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154464+0000", 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154464+0000", 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154464+0000", 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0100000000000004e-07 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154528+0000", 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4053000000000005e-05 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154537+0000", 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.507e-06 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154543+0000", 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7250000000000002e-06 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154572+0000", 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9028000000000001e-05 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154645+0000", 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3713000000000004e-05 2024-08-30T23:33:22.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.154725+0000", 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0074999999999995e-05 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.155342+0000", 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00061622800000000002 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.173182+0000", 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017840001000000001 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.371924+0000", 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19874204500000001 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.371988+0000", 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4366e-05 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:36.372051+0000", 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2409999999999994e-05 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:176692 3.c 3:3cbd1f02:::benchmark_data_smithi173_60780_object73238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:04.809934+0000", 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.529273625999998, 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22159526399999999, 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:22.474 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 176692 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.809934+0000", 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.809934+0000", 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.809935+0000", 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1090000000000001e-06 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.809940+0000", 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4049999999999999e-06 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.809953+0000", 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2337999999999999e-05 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.809959+0000", 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0349999999999998e-06 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.811153+0000", 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001194816 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.811278+0000", 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124657 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.811409+0000", 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013069700000000001 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:04.816017+0000", 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046081000000000004 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:05.000852+0000", 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18483464899999999 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:05.031428+0000", 2024-08-30T23:33:22.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030576969999999998 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:05.031481+0000", 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2899000000000003e-05 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:05.031529+0000", 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7589000000000001e-05 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.480 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:22.481 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:22.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:22.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:22.525 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:22.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:22.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:22.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:22.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:22.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:22.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:22.829 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:33:22.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:22.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:22.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:22.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:23.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3441125996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:23.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:22 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:23.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3441125996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:23.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:22 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:23.102 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:23.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-30T23:33:23.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:23.105 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3441125996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:23.105 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:22 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:23.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:23.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:23.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:23.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:23.395 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:23.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:23.524 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:23.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:23.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:23.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:33:23.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:23.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:23.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:23.916 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:23.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:24.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:23 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:23 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:33:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:23 smithi136 ceph-mon[20207]: osdmap e294: 12 total, 12 up, 12 in 2024-08-30T23:33:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:23 smithi136 ceph-mon[20207]: pgmap v720: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 337/57048 objects degraded (0.591%); 1100/57048 objects misplaced (1.928%); 3.7 MiB/s, 59 objects/s recovering 2024-08-30T23:33:24.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:24.062 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:23 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:23 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:33:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:23 smithi173 ceph-mon[20554]: osdmap e294: 12 total, 12 up, 12 in 2024-08-30T23:33:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:23 smithi173 ceph-mon[20554]: pgmap v720: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 337/57048 objects degraded (0.591%); 1100/57048 objects misplaced (1.928%); 3.7 MiB/s, 59 objects/s recovering 2024-08-30T23:33:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:23 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:23 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:33:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:23 smithi106 ceph-mon[16603]: osdmap e294: 12 total, 12 up, 12 in 2024-08-30T23:33:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:23 smithi106 ceph-mon[16603]: pgmap v720: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 676 op/s; 337/57048 objects degraded (0.591%); 1100/57048 objects misplaced (1.928%); 3.7 MiB/s, 59 objects/s recovering 2024-08-30T23:33:24.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:24.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:24.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:24.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:24.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:24.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:24.483 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:24.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:24.577 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:24.610 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:24.610 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:191250 3.a 3:5344c693:::benchmark_data_smithi173_60780_object87796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:24.605729+0000", 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0044733580000000002, 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045032659999999997, 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 191250 2024-08-30T23:33:24.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.605729+0000", 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.605729+0000", 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.605730+0000", 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.605737+0000", 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7510000000000002e-06 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.605753+0000", 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6472000000000001e-05 2024-08-30T23:33:24.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.605761+0000", 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4800000000000004e-06 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.607743+0000", 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019822709999999999 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.608015+0000", 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000272162 2024-08-30T23:33:24.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.608357+0000", 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000341799 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:191255 3.0 e294/206 3:0a80b3e4:::benchmark_data_smithi173_60780_object87801:head v 294'12040, mlcod=294'12038)", 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:24.609556+0000", 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00064612800000000005, 2024-08-30T23:33:24.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00073996100000000003, 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609556+0000", 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609556+0000", 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609557+0000", 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1170000000000001e-06 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609563+0000", 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0510000000000002e-06 2024-08-30T23:33:24.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609586+0000", 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3156999999999999e-05 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609594+0000", 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.875e-06 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:24.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609670+0000", 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6265000000000002e-05 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:24.609689+0000", 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8552000000000001e-05 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-08-30T23:33:24.619 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:24.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:24.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:24.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:24.840 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:24.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:24.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:24.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:25.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2150410005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:25.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:25.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:25.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:25.219 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:25.219 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:25.219 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:25.219 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:25.220 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2150410005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:25.232 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:25.293 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:25.293 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:25.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2150410005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:25.325 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:25.325 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:25.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:25.361 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:25.361 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:25.365 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:25.388 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:25.389 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:25.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:33:25.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:25.420 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:25.448 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:25.448 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:25.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:25.479 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:25.479 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:25.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:25.505 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:25.505 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:25.529 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:25.529 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:25.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:25.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:25.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:25.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:25.759 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:25.759 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:25.771 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:25.785 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:25.786 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:25.809 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:25.809 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:25.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:33:25.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:25.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:25.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:25.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:26.018 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:26.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:25 smithi106 ceph-mon[16603]: pgmap v721: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 337/51957 objects degraded (0.649%); 1100/51957 objects misplaced (2.117%); 1.6 MiB/s, 25 objects/s recovering 2024-08-30T23:33:26.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:25 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 337/57048 objects degraded (0.591%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:26.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:26.039 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:26.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-30T23:33:26.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:26.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:26.180 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:25 smithi136 ceph-mon[20207]: pgmap v721: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 337/51957 objects degraded (0.649%); 1100/51957 objects misplaced (2.117%); 1.6 MiB/s, 25 objects/s recovering 2024-08-30T23:33:26.180 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:25 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 337/57048 objects degraded (0.591%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:26.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:26.280 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:26.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:25 smithi173 ceph-mon[20554]: pgmap v721: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 709 op/s; 337/51957 objects degraded (0.649%); 1100/51957 objects misplaced (2.117%); 1.6 MiB/s, 25 objects/s recovering 2024-08-30T23:33:26.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:25 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 337/57048 objects degraded (0.591%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:26.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:26.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:26.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:26.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:26.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:26.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:26.761 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:26.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:26.804 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:26.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:26.825 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:26.825 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:26.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:26.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:26.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:27.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:27.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:27.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:27.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:27.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:27.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:27.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:27.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:27.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:27.603 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:27.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:33:27.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:27.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:27.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:27.955 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:27.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:27.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:27.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:28.011 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:28.012 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:28.012 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:28.012 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:28.012 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:28.089 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:28.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:28.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:28.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:28.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:28.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:28.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:28.477 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:28.477 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:28.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:28.503 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:33:28.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:28.614 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:28.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:28.740 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:28.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:33:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:28 smithi136 ceph-mon[20207]: pgmap v722: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 970 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 337/46527 objects degraded (0.724%); 844/46527 objects misplaced (1.814%); 3.0 MiB/s, 47 objects/s recovering 2024-08-30T23:33:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3310214555' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:28 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:28.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:28 smithi106 ceph-mon[16603]: pgmap v722: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 970 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 337/46527 objects degraded (0.724%); 844/46527 objects misplaced (1.814%); 3.0 MiB/s, 47 objects/s recovering 2024-08-30T23:33:28.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3310214555' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:28.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:28 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:28 smithi173 ceph-mon[20554]: pgmap v722: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 970 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 776 op/s; 337/46527 objects degraded (0.724%); 844/46527 objects misplaced (1.814%); 3.0 MiB/s, 47 objects/s recovering 2024-08-30T23:33:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3310214555' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:28 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:28.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:28.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:28.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:28.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:29.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:29.026 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:29.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-30T23:33:29.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.140 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:29.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:29.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:29.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:33:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.275 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:29.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:29.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:29.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:29.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:29.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:29.604 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:29.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:29.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:29.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:29 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:29.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:29 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:29.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:29 smithi106 ceph-mon[16603]: osdmap e295: 12 total, 12 up, 12 in 2024-08-30T23:33:29.754 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:29.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:29 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:29 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:29 smithi136 ceph-mon[20207]: osdmap e295: 12 total, 12 up, 12 in 2024-08-30T23:33:29.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:29 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:29 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:29 smithi173 ceph-mon[20554]: osdmap e295: 12 total, 12 up, 12 in 2024-08-30T23:33:29.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:29.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:29.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:30.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:30.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:30.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:30.177 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:30.177 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:30.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:30.255 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:30.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:30.300 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:30.300 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:30.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:30.326 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:30.327 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:30.352 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:30.353 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:30.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:30.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:30.408 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:30.408 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:30.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:30.441 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:30.441 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:30.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:30.471 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:30.471 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:30.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:30.499 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:30.528 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:30.528 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:30.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:30.561 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:30.561 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: pgmap v724: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 843 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 337/40455 objects degraded (0.833%); 844/40455 objects misplaced (2.086%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: osdmap e296: 12 total, 12 up, 12 in 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:30.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4120240013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:30.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:30.595 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:30.595 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:30.627 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:30.627 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:30.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:30.700 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: pgmap v724: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 843 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 337/40455 objects degraded (0.833%); 844/40455 objects misplaced (2.086%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:33:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: osdmap e296: 12 total, 12 up, 12 in 2024-08-30T23:33:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4120240013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:30.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: pgmap v724: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 843 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 337/40455 objects degraded (0.833%); 844/40455 objects misplaced (2.086%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:33:30.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:30.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:30.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: osdmap e296: 12 total, 12 up, 12 in 2024-08-30T23:33:30.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:30.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:30.870 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4120240013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.872 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:30.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:30.872 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:30.872 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:30.872 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:30.872 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:30.873 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:30.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:33:30.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:30.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:30.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:31.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:31.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:31.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:31.246 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:31.248 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:31.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:31.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:31.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:31.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:31.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:31.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:31.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:31.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:33:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.759 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:31.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:31.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:31 smithi106 ceph-mon[16603]: osdmap e297: 12 total, 12 up, 12 in 2024-08-30T23:33:31.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 337/40455 objects degraded (0.833%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:31.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:31 smithi106 ceph-mon[16603]: pgmap v727: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 728 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s; 384/34920 objects degraded (1.100%); 572/34920 objects misplaced (1.638%); 5.0 MiB/s, 80 objects/s recovering 2024-08-30T23:33:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:31 smithi136 ceph-mon[20207]: osdmap e297: 12 total, 12 up, 12 in 2024-08-30T23:33:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 337/40455 objects degraded (0.833%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:31 smithi136 ceph-mon[20207]: pgmap v727: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 728 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s; 384/34920 objects degraded (1.100%); 572/34920 objects misplaced (1.638%); 5.0 MiB/s, 80 objects/s recovering 2024-08-30T23:33:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:31 smithi173 ceph-mon[20554]: osdmap e297: 12 total, 12 up, 12 in 2024-08-30T23:33:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 337/40455 objects degraded (0.833%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:31 smithi173 ceph-mon[20554]: pgmap v727: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 728 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s; 384/34920 objects degraded (1.100%); 572/34920 objects misplaced (1.638%); 5.0 MiB/s, 80 objects/s recovering 2024-08-30T23:33:31.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:31.894 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:31.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:31.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:32.017 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:32.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:32.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:32.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:32.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:32.365 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:32.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-30T23:33:32.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:32.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:32.503 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:32.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:32.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:32.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:32.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:32.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:32.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:32.872 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:32.872 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:32.872 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:32 smithi106 ceph-mon[16603]: osdmap e298: 12 total, 12 up, 12 in 2024-08-30T23:33:32.883 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:33:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:33:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:33:32.884 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:32.884 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:32.884 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:32.884 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:32 smithi136 ceph-mon[20207]: osdmap e298: 12 total, 12 up, 12 in 2024-08-30T23:33:32.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:32.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:32.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:33.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:33.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:32 smithi173 ceph-mon[20554]: osdmap e298: 12 total, 12 up, 12 in 2024-08-30T23:33:33.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:33.175 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:33.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:33.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:33.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:33.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.313 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:33.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:33.441 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:33.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:33.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:33.623 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:33:33.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:33:33.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:33.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:33.708 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3074084907' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:33.708 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:33.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:33 smithi106 ceph-mon[16603]: pgmap v729: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 681 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 384/32682 objects degraded (1.175%); 572/32682 objects misplaced (1.750%); 2.8 MiB/s, 45 objects/s recovering 2024-08-30T23:33:33.774 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:33.774 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:33.774 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:33.775 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:33.775 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:33.776 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:33.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:33.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:33.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:33.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:33.976 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3074084907' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:33.977 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:33.977 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:33 smithi136 ceph-mon[20207]: pgmap v729: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 681 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 384/32682 objects degraded (1.175%); 572/32682 objects misplaced (1.750%); 2.8 MiB/s, 45 objects/s recovering 2024-08-30T23:33:34.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:33:34.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3074084907' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:33 smithi173 ceph-mon[20554]: pgmap v729: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 681 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s; 384/32682 objects degraded (1.175%); 572/32682 objects misplaced (1.750%); 2.8 MiB/s, 45 objects/s recovering 2024-08-30T23:33:34.080 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:34.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:34.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:34.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:34.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:34.349 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:34.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:34.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:34.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:34.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:34.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:34.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:34.753 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:34.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-30T23:33:34.781 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:34 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:34.781 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:34.782 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:34 smithi136 ceph-mon[20207]: osdmap e299: 12 total, 12 up, 12 in 2024-08-30T23:33:34.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:34.840 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:34.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:34.915 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:49992 2.0 2:0e70673a:::benchmark_data_smithi173_47195_object49991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 419.65883235799998, 2024-08-30T23:33:34.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20138367700000001, 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49992 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8400000000000004e-07 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254943+0000", 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5250999999999998e-05 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254950+0000", 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9349999999999994e-06 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254955+0000", 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2400000000000001e-06 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255011+0000", 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6350000000000001e-05 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255085+0000", 2024-08-30T23:33:34.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3522000000000005e-05 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255182+0000", 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7161e-05 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256087+0000", 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090517600000000005 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256836+0000", 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074912999999999998 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456148+0000", 2024-08-30T23:33:34.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19931177 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456236+0000", 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8363999999999996e-05 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456280+0000", 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4094000000000002e-05 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:49992 2.0 e122/97 ondisk, result = 0)", 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.256898+0000", 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 419.65683084900002, 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19938884900000001, 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:34.925 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256898+0000", 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256898+0000", 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256899+0000", 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2e-07 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256901+0000", 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4930000000000001e-06 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456045+0000", 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19914447900000001 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456054+0000", 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.952e-06 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456132+0000", 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7441999999999995e-05 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456144+0000", 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1806e-05 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456287+0000", 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014325699999999999 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.929 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:56070 2.0 2:06759afd:::benchmark_data_smithi173_47195_object56069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:39.790591+0000", 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 415.12313747100001, 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200068458, 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 56070 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790591+0000", 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790591+0000", 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790592+0000", 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-08-30T23:33:34.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790618+0000", 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5905999999999999e-05 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790631+0000", 2024-08-30T23:33:34.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2559e-05 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790636+0000", 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5570000000000003e-06 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790680+0000", 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3238000000000002e-05 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790776+0000", 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6687000000000004e-05 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.790865+0000", 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8517000000000006e-05 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.792279+0000", 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00141439 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.792724+0000", 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044493199999999999 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.990580+0000", 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19785618899999999 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.990617+0000", 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6553000000000001e-05 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:39.990660+0000", 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.295e-05 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.937 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:82550 2.8 2:108ea9a8:::benchmark_data_smithi173_47195_object82549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:07.727800+0000", 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 387.18592942100003, 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221382149, 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82550 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727800+0000", 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727800+0000", 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727801+0000", 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0380000000000001e-06 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727863+0000", 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2016999999999999e-05 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727872+0000", 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.533e-06 2024-08-30T23:33:34.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727879+0000", 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5899999999999996e-06 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.730036+0000", 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021568550000000001 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.730106+0000", 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0875000000000002e-05 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.730207+0000", 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100666 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.732953+0000", 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027455240000000001 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.733003+0000", 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0546e-05 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.949112+0000", 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.216108616 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.949151+0000", 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9619999999999997e-05 2024-08-30T23:33:34.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.949182+0000", 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0269000000000001e-05 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:97003 2.8 2:1251d5e9:::benchmark_data_smithi173_47195_object97002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.158196+0000", 2024-08-30T23:33:34.944 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 366.75553263099999, 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36850456500000001, 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 97003 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158196+0000", 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158196+0000", 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158197+0000", 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9599999999999998e-07 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158260+0000", 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2417999999999998e-05 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158269+0000", 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1749999999999994e-06 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158274+0000", 2024-08-30T23:33:34.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1839999999999998e-06 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.160501+0000", 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022267749999999998 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.160585+0000", 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3971e-05 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.160662+0000", 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6977999999999994e-05 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.166110+0000", 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054480869999999999 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.166123+0000", 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3463e-05 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526611+0000", 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36048797300000002 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526652+0000", 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0923000000000001e-05 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526701+0000", 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8721999999999999e-05 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:97003 2.8 e146/97 ondisk, result = 0)", 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 366.749584271, 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36256268200000002, 2024-08-30T23:33:34.951 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8799999999999998e-07 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164147+0000", 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3429999999999998e-06 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526534+0000", 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36238611900000001 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526539+0000", 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3499999999999996e-06 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526602+0000", 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2779000000000003e-05 2024-08-30T23:33:34.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526607+0000", 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8139999999999997e-06 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526707+0000", 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9889000000000005e-05 2024-08-30T23:33:34.955 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:187347 2.0 2:041180c2:::benchmark_data_smithi173_47195_object90339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:33.360063+0000", 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 241.55366608, 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.205202035, 2024-08-30T23:33:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 187347 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360063+0000", 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360063+0000", 2024-08-30T23:33:34.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360064+0000", 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4419999999999999e-06 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360070+0000", 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360082+0000", 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2482e-05 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360089+0000", 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1589999999999998e-06 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360127+0000", 2024-08-30T23:33:34.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8877000000000001e-05 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360209+0000", 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1476000000000004e-05 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,11", 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360281+0000", 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2119000000000005e-05 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.364622+0000", 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043406039999999996 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.364634+0000", 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2887999999999999e-05 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.565152+0000", 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20051793000000001 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.565212+0000", 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.965e-05 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.565265+0000", 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2818999999999997e-05 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:187383 2.0 2:0fe70041:::benchmark_data_smithi173_47195_object90375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:33.424456+0000", 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 241.48927246700001, 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200114871, 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.964 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 187383 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424456+0000", 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424456+0000", 2024-08-30T23:33:34.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424458+0000", 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6300000000000001e-06 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424465+0000", 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8530000000000003e-06 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424480+0000", 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5355000000000001e-05 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424487+0000", 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8399999999999997e-06 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424549+0000", 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1882000000000001e-05 2024-08-30T23:33:34.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424689+0000", 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139865 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,11", 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.424827+0000", 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013795100000000001 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.429255+0000", 2024-08-30T23:33:34.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044284499999999996 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.429644+0000", 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038828799999999998 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.624506+0000", 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19486246900000001 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.624543+0000", 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6560000000000002e-05 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.624571+0000", 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8728e-05 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.970 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:17943 3.f 3:f6359080:::benchmark_data_smithi173_60780_object17942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.755771+0000", 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 192.15795787499999, 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.204636807, 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17943 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755771+0000", 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755771+0000", 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755772+0000", 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-06 2024-08-30T23:33:34.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755823+0000", 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0385e-05 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755840+0000", 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755848+0000", 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9389999999999997e-06 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755886+0000", 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8501999999999998e-05 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.756004+0000", 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117446 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.756137+0000", 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013379900000000001 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.756884+0000", 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074688399999999996 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757473+0000", 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000589227 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960336+0000", 2024-08-30T23:33:34.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.202862818 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960374+0000", 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7598999999999997e-05 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960408+0000", 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4004999999999997e-05 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:17943 3.f e214/206 ondisk, result = 0)", 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.757206+0000", 2024-08-30T23:33:34.978 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 192.15652278600001, 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203207843, 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757206+0000", 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757206+0000", 2024-08-30T23:33:34.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757207+0000", 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0500000000000002e-07 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757214+0000", 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7059999999999998e-06 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960244+0000", 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203029869 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960251+0000", 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9500000000000004e-06 2024-08-30T23:33:34.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960328+0000", 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7737000000000005e-05 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960333+0000", 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5199999999999999e-06 2024-08-30T23:33:34.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960414+0000", 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1155999999999993e-05 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:18000 3.7 e214/206 3:e954d40a:::benchmark_data_smithi173_60780_object17999:head v 214'1140, mlcod=214'1136)", 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.779491+0000", 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 192.13423796399999, 2024-08-30T23:33:34.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20851170499999999, 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779491+0000", 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779491+0000", 2024-08-30T23:33:34.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779492+0000", 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9299999999999997e-07 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779580+0000", 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7880999999999994e-05 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986314+0000", 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20673482800000001 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986321+0000", 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8669999999999996e-06 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986374+0000", 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2571000000000002e-05 2024-08-30T23:33:34.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986394+0000", 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0010999999999999e-05 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987946+0000", 2024-08-30T23:33:34.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001551703 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.988003+0000", 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7151000000000002e-05 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:18011 3.7 e214/206 3:eb57f387:::benchmark_data_smithi173_60780_object18010:head v 214'1141, mlcod=214'1137)", 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.785032+0000", 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 192.128696883, 2024-08-30T23:33:34.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20333275100000001, 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785032+0000", 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785032+0000", 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785033+0000", 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8599999999999998e-07 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785089+0000", 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6268999999999999e-05 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987325+0000", 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.202235785 2024-08-30T23:33:34.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987335+0000", 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0219e-05 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987394+0000", 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9082000000000001e-05 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987412+0000", 2024-08-30T23:33:34.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.988324+0000", 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00091150499999999995 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.988365+0000", 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0926000000000001e-05 2024-08-30T23:33:34.993 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:19970 3.d 3:b8325352:::benchmark_data_smithi173_60780_object19969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:24.073294+0000", 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 190.84043479499999, 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21259018599999999, 2024-08-30T23:33:34.994 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19970 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.995 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073294+0000", 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073294+0000", 2024-08-30T23:33:34.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073295+0000", 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.316e-06 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073351+0000", 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5868000000000001e-05 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073366+0000", 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4260000000000001e-05 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073373+0000", 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2710000000000003e-06 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073426+0000", 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3372000000000003e-05 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073555+0000", 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012880199999999999 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073687+0000", 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013148200000000001 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.074547+0000", 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086078900000000004 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:34.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.075046+0000", 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00049826000000000004 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.285770+0000", 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21072489899999999 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.285820+0000", 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9518000000000003e-05 2024-08-30T23:33:35.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.285884+0000", 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4349000000000003e-05 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22000 3.d 3:bc62a708:::benchmark_data_smithi173_60780_object21999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.284812+0000", 2024-08-30T23:33:35.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 189.62891729099999, 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19380415300000001, 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22000 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284812+0000", 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284812+0000", 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284817+0000", 2024-08-30T23:33:35.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6500000000000001e-06 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284858+0000", 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0432999999999997e-05 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284869+0000", 2024-08-30T23:33:35.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1292e-05 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284874+0000", 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4439999999999999e-06 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.284920+0000", 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5466999999999997e-05 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.285009+0000", 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8943000000000005e-05 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.285096+0000", 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7155000000000002e-05 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.285882+0000", 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078637200000000003 2024-08-30T23:33:35.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.286164+0000", 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028139199999999999 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478473+0000", 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19230941100000001 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478554+0000", 2024-08-30T23:33:35.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0896999999999995e-05 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.478616+0000", 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1697000000000002e-05 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:22013 3.7 e214/206 3:ee8a83f6:::benchmark_data_smithi173_60780_object22012:head v 214'1410, mlcod=214'1407)", 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:25.298936+0000", 2024-08-30T23:33:35.008 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 189.614793378, 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200901993, 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.298936+0000", 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.298936+0000", 2024-08-30T23:33:35.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.298937+0000", 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.167e-06 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.299041+0000", 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103813 2024-08-30T23:33:35.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.498911+0000", 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19987073999999999 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.498916+0000", 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.498984+0000", 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8084999999999996e-05 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.499013+0000", 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9090000000000001e-05 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.499775+0000", 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076212399999999998 2024-08-30T23:33:35.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:25.499838+0000", 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2314e-05 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41039 3.f 3:fd8786e1:::benchmark_data_smithi173_60780_object41038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 171.72772564600001, 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26531318500000001, 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41039 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186005+0000", 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.815e-06 2024-08-30T23:33:35.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186051+0000", 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5473999999999999e-05 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186067+0000", 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6861999999999999e-05 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186076+0000", 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8880000000000003e-06 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186133+0000", 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7101000000000001e-05 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186258+0000", 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012487800000000001 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186377+0000", 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118268 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.187721+0000", 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013439979999999999 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.188919+0000", 2024-08-30T23:33:35.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001198574 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451218+0000", 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26229850199999999 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451272+0000", 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4500999999999999e-05 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451316+0000", 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4323999999999998e-05 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41062 3.f 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 171.711776606, 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32901242400000003, 2024-08-30T23:33:35.021 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41062 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201954+0000", 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202013+0000", 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9793999999999997e-05 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202024+0000", 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0502999999999999e-05 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202030+0000", 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6210000000000001e-06 2024-08-30T23:33:35.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202072+0000", 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2011999999999997e-05 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202154+0000", 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2306999999999996e-05 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:33:35.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202236+0000", 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2520000000000003e-05 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203586+0000", 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00134931 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.204090+0000", 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050403899999999996 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530901+0000", 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32681120000000002 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530936+0000", 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4857000000000002e-05 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530965+0000", 2024-08-30T23:33:35.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8999000000000001e-05 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:41039 3.f e226/206 ondisk, result = 0)", 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 171.709856022, 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24745020100000001, 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:35.028 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203874+0000", 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699999999999997e-07 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203878+0000", 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.941e-06 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451055+0000", 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24717699200000001 2024-08-30T23:33:35.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451067+0000", 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.186e-05 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451186+0000", 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118997 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451210+0000", 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4219e-05 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451323+0000", 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113375 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74450 3.d 3:b854981d:::benchmark_data_smithi173_60780_object74449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:33:35.032 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 142.947797982, 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29260630999999998, 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74450 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965932+0000", 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.71e-07 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965987+0000", 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5448000000000002e-05 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966002+0000", 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4419999999999999e-05 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966007+0000", 2024-08-30T23:33:35.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0119999999999996e-06 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967853+0000", 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001846547 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967977+0000", 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123404 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:33:35.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.968112+0000", 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013568200000000001 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970457+0000", 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023443460000000002 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.013531+0000", 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043074440999999998 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258300+0000", 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.244768449 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258497+0000", 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000197209 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258537+0000", 2024-08-30T23:33:35.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0481000000000001e-05 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:74450 3.d e243/206 ondisk, result = 0)", 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 142.94348406, 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28829785299999999, 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:33:35.039 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:33:35.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970246+0000", 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970250+0000", 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0330000000000002e-06 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258240+0000", 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28799039900000001 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258244+0000", 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1309999999999999e-06 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258287+0000", 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2347e-05 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258294+0000", 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2250000000000003e-06 2024-08-30T23:33:35.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258543+0000", 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000249001 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:33:35.043 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:35.070 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:34 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:34 smithi173 ceph-mon[20554]: osdmap e299: 12 total, 12 up, 12 in 2024-08-30T23:33:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:34 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:34 smithi106 ceph-mon[16603]: osdmap e299: 12 total, 12 up, 12 in 2024-08-30T23:33:35.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:35.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:35.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:35.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:35.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:35.454 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:35.485 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:35.497 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:35.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:35.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:35.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:35.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:35.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:35 smithi106 ceph-mon[16603]: pgmap v731: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 558 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 384/26778 objects degraded (1.434%); 572/26778 objects misplaced (2.136%); 2.4 MiB/s, 37 objects/s recovering 2024-08-30T23:33:35.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 384/32682 objects degraded (1.175%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:35.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:35.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:35.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:35.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:35.979 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:35.999 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:35 smithi136 ceph-mon[20207]: pgmap v731: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 558 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 384/26778 objects degraded (1.434%); 572/26778 objects misplaced (2.136%); 2.4 MiB/s, 37 objects/s recovering 2024-08-30T23:33:35.999 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 384/32682 objects degraded (1.175%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:36.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:33:36.070 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:35 smithi173 ceph-mon[20554]: pgmap v731: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 558 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s; 384/26778 objects degraded (1.434%); 572/26778 objects misplaced (2.136%); 2.4 MiB/s, 37 objects/s recovering 2024-08-30T23:33:36.070 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 384/32682 objects degraded (1.175%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:36.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:36.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:36.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:36.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:36.262 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:36.262 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:36.288 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:36.288 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:36.313 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:36.313 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:36.341 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:36.342 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:36.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:36.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:36.372 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:36.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:36.390 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:36.406 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:36.406 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:36.436 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:36.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:36.464 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:36.465 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:36.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:36.493 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:36.493 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.503 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:36.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:36.520 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:36.521 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:36.521 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:36.521 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:36.521 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:36.521 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:36.528 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:36.528 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:36.555 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:36.555 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:36.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:36.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:36.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:36.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:36.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:36.761 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:36.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2973873612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:36.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2973873612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:36.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:36.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:36.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:36.948 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:36.948 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:36.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:37.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:37.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2973873612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:37.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:37.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:37.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:37.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:37.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:37.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:37.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:37.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:33:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:37.470 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:37.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-30T23:33:37.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:37.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:37.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:37.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:37.797 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:37 smithi173 ceph-mon[20554]: 3.b scrub starts 2024-08-30T23:33:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:37 smithi173 ceph-mon[20554]: 3.b scrub ok 2024-08-30T23:33:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:37 smithi173 ceph-mon[20554]: pgmap v732: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 467 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 384/22413 objects degraded (1.713%); 248/22413 objects misplaced (1.107%); 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:33:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:37 smithi106 ceph-mon[16603]: 3.b scrub starts 2024-08-30T23:33:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:37 smithi106 ceph-mon[16603]: 3.b scrub ok 2024-08-30T23:33:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:37 smithi106 ceph-mon[16603]: pgmap v732: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 467 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 384/22413 objects degraded (1.713%); 248/22413 objects misplaced (1.107%); 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:33:37.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:37.927 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:37.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:38.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:37 smithi136 ceph-mon[20207]: 3.b scrub starts 2024-08-30T23:33:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:37 smithi136 ceph-mon[20207]: 3.b scrub ok 2024-08-30T23:33:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:37 smithi136 ceph-mon[20207]: pgmap v732: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 467 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 788 op/s; 384/22413 objects degraded (1.713%); 248/22413 objects misplaced (1.107%); 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:33:38.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:38.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:38.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:38.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:38.268 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:38.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:38.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:38.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:33:38.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:38.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:38.559 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:38.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:38.595 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:38.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:38.724 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:38.725 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:38.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:38.834 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:38.835 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:38.835 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:38.835 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:38.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:38.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:38.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:38.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:39.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:39.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:39.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:39.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:39.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:39.332 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:39.332 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:39.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:39.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:39.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:39.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:39.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:39.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:39.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:39.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:39.857 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:39.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-08-30T23:33:39.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:39.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:39.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:40.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:40.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:33:40.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:40.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:40.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:33:40.229 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:40.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:40.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:40.377 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:40.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:40.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:40.500 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:33:40.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:40.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:40.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:40.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:40.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:40 smithi136 ceph-mon[20207]: pgmap v733: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 347 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 384/16629 objects degraded (2.309%); 248/16629 objects misplaced (1.491%); 2.5 MiB/s, 40 objects/s recovering 2024-08-30T23:33:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2797257476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:40.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:40.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:40 smithi173 ceph-mon[20554]: pgmap v733: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 347 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 384/16629 objects degraded (2.309%); 248/16629 objects misplaced (1.491%); 2.5 MiB/s, 40 objects/s recovering 2024-08-30T23:33:40.822 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2797257476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:40 smithi106 ceph-mon[16603]: pgmap v733: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 347 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 384/16629 objects degraded (2.309%); 248/16629 objects misplaced (1.491%); 2.5 MiB/s, 40 objects/s recovering 2024-08-30T23:33:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2797257476' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:40.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:40.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:40.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:40.949 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:40.949 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:41.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:41.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:41.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:41.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:41.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:41.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:41.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:41.439 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:41.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:41.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:41 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:33:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2797257476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:41 smithi173 ceph-mon[20554]: osdmap e300: 12 total, 12 up, 12 in 2024-08-30T23:33:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 384/16629 objects degraded (2.309%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:41.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:41.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:41 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:33:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2797257476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:41 smithi136 ceph-mon[20207]: osdmap e300: 12 total, 12 up, 12 in 2024-08-30T23:33:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 384/16629 objects degraded (2.309%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:41.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:41.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:41 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:33:41.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2797257476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:41.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:41 smithi106 ceph-mon[16603]: osdmap e300: 12 total, 12 up, 12 in 2024-08-30T23:33:41.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 384/16629 objects degraded (2.309%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:33:41.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:33:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:41.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:41.849 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:41.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:41.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:41.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:33:42.028 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:33:42.029 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:33:42.029 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:33:42.029 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:33:42.029 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:33:42.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:42.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:42.165 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:42.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:42.212 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:42.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:42.340 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:42.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:42.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:42.559 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:42 smithi173 ceph-mon[20554]: pgmap v735: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s; 384/11319 objects degraded (3.393%); 20/11319 objects misplaced (0.177%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:33:42.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:42.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:42.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:42.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:42 smithi136 ceph-mon[20207]: pgmap v735: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s; 384/11319 objects degraded (3.393%); 20/11319 objects misplaced (0.177%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:33:42.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:42 smithi106 ceph-mon[16603]: pgmap v735: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 236 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s; 384/11319 objects degraded (3.393%); 20/11319 objects misplaced (0.177%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:33:42.859 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:42.891 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:42.892 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:33:42.905 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:33:42.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:42.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:42.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:43.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:43.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:43.139 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:43.139 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:43.170 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:43.171 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:43.198 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:43.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:43.228 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:43.228 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:43.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:43.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:43.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:43.287 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:43.288 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:43.327 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:43.327 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:43.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:43.355 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:43.356 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:43.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:43.384 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:43.384 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:43.414 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:43.415 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:43.442 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:43.443 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:43.467 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:43.467 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:43.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:43.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:43.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:33:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3955354965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:43.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:43.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:43.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:43.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:33:43.754 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3955354965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:43.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3955354965' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:43.883 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:43.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:43.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:44.010 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:44.074 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:44.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.134 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:44.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:44.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:44.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:44.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:44.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:44.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:44.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:44.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:44.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:44.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:44.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:44 smithi136 ceph-mon[20207]: pgmap v736: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 189 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 384/9072 objects degraded (4.233%); 20/9072 objects misplaced (0.220%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:33:44.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:44 smithi173 ceph-mon[20554]: pgmap v736: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 189 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 384/9072 objects degraded (4.233%); 20/9072 objects misplaced (0.220%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:33:44.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:44.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:44 smithi106 ceph-mon[16603]: pgmap v736: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 189 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 775 op/s; 384/9072 objects degraded (4.233%); 20/9072 objects misplaced (0.220%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:33:44.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:44.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:44.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:44.880 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 103451 objects 2024-08-30T23:33:44.880 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :122.579 2024-08-30T23:33:44.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:45.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:45.004 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:45.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:45.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:45.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:45.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:45.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:45.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:45.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:45.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:45.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:45.621 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:45.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:45.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:45.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:45 smithi173 ceph-mon[20554]: pgmap v737: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 69 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 384/3285 objects degraded (11.689%); 20/3285 objects misplaced (0.609%); 3.4 MiB/s, 55 objects/s recovering 2024-08-30T23:33:45.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:45 smithi106 ceph-mon[16603]: pgmap v737: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 69 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 384/3285 objects degraded (11.689%); 20/3285 objects misplaced (0.609%); 3.4 MiB/s, 55 objects/s recovering 2024-08-30T23:33:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:45.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:45.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:45.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:45.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:45 smithi136 ceph-mon[20207]: pgmap v737: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 69 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 384/3285 objects degraded (11.689%); 20/3285 objects misplaced (0.609%); 3.4 MiB/s, 55 objects/s recovering 2024-08-30T23:33:46.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:46.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:46.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:46.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:46.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:46.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:46.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:46.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:46.558 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:33:46.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:46.627 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:46.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:46.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 384/3285 objects degraded (11.689%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:33:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/122012590' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:46.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 384/3285 objects degraded (11.689%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:33:46.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/122012590' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:46.882 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:46.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:46.972 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 384/3285 objects degraded (11.689%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:33:46.973 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/122012590' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:33:46.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:47.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:47.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:33:47.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:47.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:47.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:47.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:47.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:47.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:47.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:47.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:47.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:47.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:47.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:47.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:47.753 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:33:47.754 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:33:47.754 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:33:47.754 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:33:47.754 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:33:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:47.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:47.811 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:47.811 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:47.812 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:47 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:47.812 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/122012590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:33:47.812 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:47 smithi106 ceph-mon[16603]: osdmap e301: 12 total, 12 up, 12 in 2024-08-30T23:33:47.812 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:47 smithi106 ceph-mon[16603]: pgmap v739: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 384/735 objects degraded (52.245%); 1.9 MiB/s, 78 objects/s recovering 2024-08-30T23:33:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:47 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/122012590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:33:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:47 smithi173 ceph-mon[20554]: osdmap e301: 12 total, 12 up, 12 in 2024-08-30T23:33:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:47 smithi173 ceph-mon[20554]: pgmap v739: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 384/735 objects degraded (52.245%); 1.9 MiB/s, 78 objects/s recovering 2024-08-30T23:33:47.832 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:47.832 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:47.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:47.853 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:47.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:47.873 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:47.873 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:47.885 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:47.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:47.897 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:47.897 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:47.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:33:47.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:33:47.907 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-08-30T23:33:47.907 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:33:47.907 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:33:47.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-08-30T23:33:47.918 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:47.918 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:47.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:47 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:47.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/122012590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:33:47.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:47 smithi136 ceph-mon[20207]: osdmap e301: 12 total, 12 up, 12 in 2024-08-30T23:33:47.930 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:47 smithi136 ceph-mon[20207]: pgmap v739: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 16 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 384/735 objects degraded (52.245%); 1.9 MiB/s, 78 objects/s recovering 2024-08-30T23:33:47.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:48.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:48.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:48.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:48.141 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:48.142 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:48.152 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:33:48.162 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:48.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:48.183 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:48.184 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:48.204 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:48.204 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:48.225 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:48.225 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:48.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:48.246 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:48.246 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:48.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:48.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:48.378 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:48.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:48.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:48.587 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:33:48.588 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "" 2024-08-30T23:33:48.588 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:33:48.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:48.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:48.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:48.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:48.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:48.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:48.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:48 smithi136 ceph-mon[20207]: osdmap e302: 12 total, 12 up, 12 in 2024-08-30T23:33:48.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/60762485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:48 smithi173 ceph-mon[20554]: osdmap e302: 12 total, 12 up, 12 in 2024-08-30T23:33:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/60762485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:48.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:48.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:48.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:48.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:48 smithi106 ceph-mon[16603]: osdmap e302: 12 total, 12 up, 12 in 2024-08-30T23:33:48.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/60762485' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:48.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:48.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:48.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:48.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:49.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:49.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:49.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:49.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:49.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:49.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:33:49.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:49.367 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:49.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:49.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:49.529 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:49.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:49.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:49.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:49.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:49 smithi173 ceph-mon[20554]: pgmap v741: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 384/6 objects degraded (6400.000%); 160 KiB/s, 49 objects/s recovering 2024-08-30T23:33:49.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:49 smithi106 ceph-mon[16603]: pgmap v741: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 384/6 objects degraded (6400.000%); 160 KiB/s, 49 objects/s recovering 2024-08-30T23:33:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:49.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:49.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:49.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:49.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:49 smithi136 ceph-mon[20207]: pgmap v741: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 384/6 objects degraded (6400.000%); 160 KiB/s, 49 objects/s recovering 2024-08-30T23:33:50.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:50.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:50.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:50.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:50.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:50.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:50.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:50.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:50.580 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:33:50.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:50.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.609 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:50.710 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:50.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:50.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:50.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4251284889' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:50.857 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:50.873 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-08-30T23:33:50.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-08-30T23:33:50.884 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4251284889' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:50.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:50.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:50.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:51.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:33:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4251284889' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:33:51.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:51.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:51.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:51.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:51.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:51.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:51.380 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:51.390 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:51.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:51.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:51.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:51.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:51.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:51.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:51.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:33:51.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:51.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4251284889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:51 smithi106 ceph-mon[16603]: osdmap e303: 12 total, 12 up, 12 in 2024-08-30T23:33:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 384/6 objects degraded (6400.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:33:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:51 smithi106 ceph-mon[16603]: pgmap v743: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 384/6 objects degraded (6400.000%); 213 KiB/s, 150 objects/s recovering 2024-08-30T23:33:51.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:51.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:51.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:52.038 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4251284889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:51 smithi136 ceph-mon[20207]: osdmap e303: 12 total, 12 up, 12 in 2024-08-30T23:33:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 384/6 objects degraded (6400.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:33:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:51 smithi136 ceph-mon[20207]: pgmap v743: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 384/6 objects degraded (6400.000%); 213 KiB/s, 150 objects/s recovering 2024-08-30T23:33:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4251284889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:33:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:51 smithi173 ceph-mon[20554]: osdmap e303: 12 total, 12 up, 12 in 2024-08-30T23:33:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 384/6 objects degraded (6400.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:33:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:51 smithi173 ceph-mon[20554]: pgmap v743: 17 pgs: 1 active+recovering+undersized+degraded+remapped, 16 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 182 op/s; 384/6 objects degraded (6400.000%); 213 KiB/s, 150 objects/s recovering 2024-08-30T23:33:52.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:52.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:52.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:52.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:52.315 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:52.315 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:52.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:52.335 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:52.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:52.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:52.360 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:52.360 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:52.380 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:52.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:52.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:52.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:52.422 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:52.423 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:52.443 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:52.443 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:52.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:52.465 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:52.465 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:52.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:52.486 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:52.486 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:52.507 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:52.507 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:52.528 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:52.528 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:52.548 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:52.549 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:52.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:52.599 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:52.607 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:33:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:33:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:33:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:33:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:62896 2.a e127/97 2:5bc7d7b5:::benchmark_data_smithi173_47195_object62895:head v 127'3849, mlcod=127'3846)", 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 426.62775770100001, 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.271516058, 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912242+0000", 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912243+0000", 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8800000000000002e-07 2024-08-30T23:33:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912264+0000", 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1222000000000001e-05 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182196+0000", 2024-08-30T23:33:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26993194300000001 2024-08-30T23:33:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182200+0000", 2024-08-30T23:33:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4549999999999997e-06 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182246+0000", 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5404000000000001e-05 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.182263+0000", 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7167999999999999e-05 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183724+0000", 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014614000000000001 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183758+0000", 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3677999999999998e-05 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67113 2.e e129/97 2:7c5fddf0:::benchmark_data_smithi173_47195_object67112:head v 129'4149, mlcod=129'4145)", 2024-08-30T23:33:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:33:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 422.92800021800002, 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.177302496, 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.611999+0000", 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.612000+0000", 2024-08-30T23:33:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5600000000000004e-07 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.612070+0000", 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9492000000000005e-05 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787357+0000", 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17528732899999999 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787369+0000", 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2272999999999999e-05 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787501+0000", 2024-08-30T23:33:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013128700000000001 2024-08-30T23:33:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.787559+0000", 2024-08-30T23:33:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8606000000000001e-05 2024-08-30T23:33:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.789267+0000", 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017075619999999999 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.789302+0000", 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4990999999999998e-05 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67116 2.e e129/97 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head v 129'4150, mlcod=129'4146)", 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 422.925392284, 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22312085200000001, 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614607+0000", 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614608+0000", 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.614657+0000", 2024-08-30T23:33:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8501000000000002e-05 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835863+0000", 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.221206134 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835869+0000", 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6359999999999997e-06 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835916+0000", 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6378999999999998e-05 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.835929+0000", 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3583e-05 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.837698+0000", 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001769112 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:49.837728+0000", 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9808999999999999e-05 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:67586 2.e e129/97 2:726358de:::benchmark_data_smithi173_47195_object67585:head v 129'4182, mlcod=129'4178)", 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 422.43819130000003, 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.182122652, 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101808+0000", 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101809+0000", 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.101870+0000", 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1013e-05 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281568+0000", 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17969818400000001 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281577+0000", 2024-08-30T23:33:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6270000000000006e-06 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281653+0000", 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5925000000000002e-05 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.281681+0000", 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8076999999999999e-05 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.283872+0000", 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021911510000000001 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:50.283931+0000", 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9083000000000003e-05 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:75419 2.0 e132/97 2:02abbf90:::benchmark_data_smithi173_47195_object75418:head v 132'4710, mlcod=132'4708)", 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 415.69053113400003, 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87299763699999999, 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849468+0000", 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849469+0000", 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9299999999999996e-07 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.849566+0000", 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6235000000000003e-05 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719661+0000", 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87009526500000001 2024-08-30T23:33:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719680+0000", 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8785999999999999e-05 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719819+0000", 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139491 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.719862+0000", 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2382000000000002e-05 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.722427+0000", 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025652909999999999 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:57.722466+0000", 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9294e-05 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:109533 2.d 2:b9e7a252:::benchmark_data_smithi173_47195_object12525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:44.821491+0000", 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 367.71850882199999, 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.267912015, 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 109533 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821491+0000", 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821491+0000", 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821492+0000", 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0589999999999999e-06 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821498+0000", 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7760000000000003e-06 2024-08-30T23:33:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821512+0000", 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4106e-05 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.821519+0000", 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6079999999999998e-06 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.822991+0000", 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014714400000000001 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.823097+0000", 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010626800000000001 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.823163+0000", 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5671000000000002e-05 2024-08-30T23:33:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.825511+0000", 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023481499999999998 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.831879+0000", 2024-08-30T23:33:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0063679000000000001 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089328+0000", 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25744958400000001 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089362+0000", 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3571000000000001e-05 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089403+0000", 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0881999999999999e-05 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:109587 2.d 2:b377b128:::benchmark_data_smithi173_47195_object12579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:33:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:44.904763+0000", 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 367.63523620400002, 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20665070099999999, 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 109587 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904763+0000", 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904763+0000", 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904765+0000", 2024-08-30T23:33:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.691e-06 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904770+0000", 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2429999999999996e-06 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904782+0000", 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1609e-05 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904789+0000", 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.075e-06 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904840+0000", 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1447999999999999e-05 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.904926+0000", 2024-08-30T23:33:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5690000000000001e-05 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.905012+0000", 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5675999999999998e-05 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.907884+0000", 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028722420000000001 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908020+0000", 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013599400000000001 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.111348+0000", 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20332840799999999 2024-08-30T23:33:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.111382+0000", 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3562000000000002e-05 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.111414+0000", 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2063000000000001e-05 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:109533 2.d e153/97 ondisk, result = 0)", 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:44.908189+0000", 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 367.63181033699999, 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18121869800000001, 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908189+0000", 2024-08-30T23:33:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908189+0000", 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908191+0000", 2024-08-30T23:33:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.517e-06 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:44.908206+0000", 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4816e-05 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089281+0000", 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18107588399999999 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089286+0000", 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4870000000000004e-06 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089319+0000", 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2898000000000002e-05 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089325+0000", 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6699999999999999e-06 2024-08-30T23:33:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:45.089408+0000", 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3425999999999997e-05 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:146079 2.d 2:b3e1325e:::benchmark_data_smithi173_47195_object49071:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:33.409320+0000", 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 319.13067918299998, 2024-08-30T23:33:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.202734725, 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 146079 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409320+0000", 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409320+0000", 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409322+0000", 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4079999999999999e-06 2024-08-30T23:33:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409327+0000", 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5879999999999997e-06 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409340+0000", 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2369e-05 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409345+0000", 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5609999999999998e-06 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409390+0000", 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5062999999999999e-05 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409471+0000", 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0115000000000001e-05 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.409538+0000", 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7527999999999994e-05 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.412680+0000", 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031420770000000001 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.413386+0000", 2024-08-30T23:33:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070555399999999995 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.611977+0000", 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.198590822 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.612011+0000", 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4116999999999999e-05 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:33.612055+0000", 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4523e-05 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:152975 2.e e179/97 2:7aa5cae5:::benchmark_data_smithi173_47195_object55967:head v 179'9430, mlcod=178'9427)", 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 309.77516547599998, 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17485895900000001, 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.663 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764834+0000", 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8900000000000001e-07 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.764838+0000", 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.258e-06 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936271+0000", 2024-08-30T23:33:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.171433059 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936278+0000", 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8210000000000004e-06 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936332+0000", 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4724e-05 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.936357+0000", 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4671000000000001e-05 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.939654+0000", 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032971770000000001 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:42.939693+0000", 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.896e-05 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:155294 2.e e181/97 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head v 181'9573, mlcod=181'9570)", 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 306.41576804200002, 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26756636299999997, 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124232+0000", 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.85e-07 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.124238+0000", 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4129999999999997e-06 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.386998+0000", 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26276001399999999 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.387007+0000", 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.679e-06 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.387084+0000", 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6469000000000003e-05 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.387115+0000", 2024-08-30T23:33:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0574000000000002e-05 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.391740+0000", 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046257349999999997 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:46.391798+0000", 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7694e-05 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:194002 2.d 2:bd11e86a:::benchmark_data_smithi173_47195_object96994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 249.75630179300001, 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0074568129999999, 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:33:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 194002 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783698+0000", 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0100000000000005e-07 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783702+0000", 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.371e-06 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783710+0000", 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3909999999999998e-06 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.783715+0000", 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4399999999999998e-06 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.792687+0000", 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0089719310000000007 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.792761+0000", 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4054000000000006e-05 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,4", 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.792832+0000", 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1705999999999994e-05 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:33:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797127+0000", 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042944649999999999 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113523+0000", 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31639595199999998 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791079+0000", 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67755674899999996 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791121+0000", 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1829000000000002e-05 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791155+0000", 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3324000000000001e-05 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:194002 2.d e200/97 ondisk, result = 0)", 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 249.74268211200001, 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.316210565, 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797318+0000", 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.44e-07 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797322+0000", 2024-08-30T23:33:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.839e-06 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113409+0000", 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31608707200000002 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113425+0000", 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6099999999999998e-05 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113509+0000", 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3580999999999998e-05 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113517+0000", 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1859999999999992e-06 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.113528+0000", 2024-08-30T23:33:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1143e-05 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:194002 2.d e200/97 ondisk, result = 0)", 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:42.797432+0000", 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 249.742567326, 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.993726836, 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:33:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797432+0000", 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797432+0000", 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797433+0000", 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1600000000000001e-07 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:42.797437+0000", 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.613e-06 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791015+0000", 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99357831299999999 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791022+0000", 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3289999999999996e-06 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791069+0000", 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6609e-05 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791076+0000", 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8029999999999999e-06 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.791159+0000", 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3553000000000005e-05 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:2664 3.d e213/206 3:b4e3f87a:::benchmark_data_smithi173_60780_object2663:head v 213'157, mlcod=213'153)", 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 219.865703903, 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23855293999999999, 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674296+0000", 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9099999999999999e-07 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.674443+0000", 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014676500000000001 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912387+0000", 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23794400299999999 2024-08-30T23:33:52.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912393+0000", 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3409999999999999e-06 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912445+0000", 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1572e-05 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912468+0000", 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3334000000000001e-05 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912808+0000", 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034020600000000002 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:12.912849+0000", 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0327999999999997e-05 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.695 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:19959 3.d e214/206 3:b8cd0889:::benchmark_data_smithi173_60780_object19958:head v 214'1243, mlcod=214'1239)", 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:24.063884+0000", 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 208.47611523899999, 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18932943499999999, 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063884+0000", 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063884+0000", 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063885+0000", 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.063912+0000", 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6775e-05 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252412+0000", 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188500011 2024-08-30T23:33:52.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252435+0000", 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2844e-05 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252473+0000", 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8583e-05 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.252488+0000", 2024-08-30T23:33:52.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4544e-05 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.253189+0000", 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00070125500000000002 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.253214+0000", 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4698999999999998e-05 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.700 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:19970 3.d e214/206 3:b8325352:::benchmark_data_smithi173_60780_object19969:head v 214'1244, mlcod=214'1240)", 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:24.073819+0000", 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 208.46618065999999, 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21071751499999999, 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073819+0000", 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073819+0000", 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073820+0000", 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5600000000000004e-07 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.073896+0000", 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5807000000000001e-05 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283768+0000", 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20987212099999999 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283772+0000", 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7339999999999999e-06 2024-08-30T23:33:52.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283817+0000", 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4555999999999998e-05 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.283831+0000", 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.284504+0000", 2024-08-30T23:33:52.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067363199999999996 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:24.284536+0000", 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2106000000000001e-05 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.705 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:92238 3.0 e250/206 3:0862a24f:::benchmark_data_smithi173_60780_object92237:head v 250'5807, mlcod=250'5804)", 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:29.455781+0000", 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 143.084218736, 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18914341600000001, 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455781+0000", 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455781+0000", 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455782+0000", 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5900000000000005e-07 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.455803+0000", 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1282999999999999e-05 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642542+0000", 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18673842900000001 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642547+0000", 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0150000000000003e-06 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642626+0000", 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9908999999999997e-05 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.642641+0000", 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4542000000000001e-05 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.644886+0000", 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022451049999999998 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:29.644924+0000", 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8173999999999997e-05 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:188943 3.e 3:7ad1823c:::benchmark_data_smithi173_60780_object85489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:21.673245+0000", 2024-08-30T23:33:52.710 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 30.866754284999999, 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22059119199999999, 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188943 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673245+0000", 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673245+0000", 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673247+0000", 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5579999999999999e-06 2024-08-30T23:33:52.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673253+0000", 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5459999999999997e-06 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673267+0000", 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3332e-05 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673273+0000", 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.297e-06 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673335+0000", 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1680000000000006e-05 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673454+0000", 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118787 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.673539+0000", 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5637999999999996e-05 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.675911+0000", 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023715749999999999 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.682362+0000", 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0064508179999999997 2024-08-30T23:33:52.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.893731+0000", 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.211369005 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.893789+0000", 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8218999999999999e-05 2024-08-30T23:33:52.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.893837+0000", 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7737000000000001e-05 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:188944 3.e 3:74c7d5f9:::benchmark_data_smithi173_60780_object85490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:21.677295+0000", 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 30.862704750999999, 2024-08-30T23:33:52.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21670371599999999, 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 188944 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677295+0000", 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677295+0000", 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677297+0000", 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.843e-06 2024-08-30T23:33:52.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677303+0000", 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.336e-06 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677318+0000", 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5038000000000001e-05 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677327+0000", 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8759999999999994e-06 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677388+0000", 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1421999999999996e-05 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677491+0000", 2024-08-30T23:33:52.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103008 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.677566+0000", 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4499000000000006e-05 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.680207+0000", 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026411479999999998 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.682394+0000", 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021867060000000001 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.893886+0000", 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21149233000000001 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.893946+0000", 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9786e-05 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:33:52.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:21.893999+0000", 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2723999999999999e-05 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:33:52.724 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:33:52.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:52.725 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_1' removed 2024-08-30T23:33:52.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1329983057' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:33:52.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/437404736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:52.746 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1329983057' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:33:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/437404736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:52.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:52.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:52.975 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:52.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:52.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1329983057' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:33:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/437404736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:53.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:53.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:53.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:53.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:53.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:33:53.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:53.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:53.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:53.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:53.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:53.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:53.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:53.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:53.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1329983057' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:33:53.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:53 smithi106 ceph-mon[16603]: osdmap e304: 12 total, 12 up, 12 in 2024-08-30T23:33:53.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:53 smithi106 ceph-mon[16603]: pgmap v745: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:53.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:53.861 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:33:53.862 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:33:53.862 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-08-30T23:33:53.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-08-30T23:33:53.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:53.902 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1329983057' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:33:53.902 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:53 smithi173 ceph-mon[20554]: osdmap e304: 12 total, 12 up, 12 in 2024-08-30T23:33:53.902 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:53 smithi173 ceph-mon[20554]: pgmap v745: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:53.913 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:53.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:33:53.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:53.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:53.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:54.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1329983057' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:33:54.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:53 smithi136 ceph-mon[20207]: osdmap e304: 12 total, 12 up, 12 in 2024-08-30T23:33:54.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:53 smithi136 ceph-mon[20207]: pgmap v745: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:54.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:54.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:54.174 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:54.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:54.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:54.328 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:33:54.329 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:33:54.329 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:33:54.329 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.347892 2024-08-30T23:33:54.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 7 0.34789233084308246 2024-08-30T23:33:54.338 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:54.394 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:54.460 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:54.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:54.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:54.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:54.637 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:33:54.645 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:33:54.645 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:33:54.646 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:33:54.646 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:33:54.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:54.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:54.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:54.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:54 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 384/6 objects degraded (6400.000%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:33:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2430375354' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:54 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 384/6 objects degraded (6400.000%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:33:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2430375354' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:54.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:54.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:54.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:54.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:54.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:55.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:54 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 384/6 objects degraded (6400.000%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:33:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2430375354' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:33:55.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:55.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:55.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:33:55.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:55.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:55.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:55.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:55.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:33:55.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:55.520 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:55.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:55.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:55.649 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_2' created 2024-08-30T23:33:55.725 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:55.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:55.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:55 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2430375354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:55 smithi106 ceph-mon[16603]: osdmap e305: 12 total, 12 up, 12 in 2024-08-30T23:33:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:33:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3720722303' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-30T23:33:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:55 smithi106 ceph-mon[16603]: pgmap v747: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:55.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:55.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:55.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:55.913 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:55.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:55 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:55.977 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2430375354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:55.977 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:55 smithi173 ceph-mon[20554]: osdmap e305: 12 total, 12 up, 12 in 2024-08-30T23:33:55.977 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:33:55.977 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3720722303' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-30T23:33:55.977 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:55 smithi173 ceph-mon[20554]: pgmap v747: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:55.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:55.999 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:55 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:33:56.000 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2430375354' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:33:56.000 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:55 smithi136 ceph-mon[20207]: osdmap e305: 12 total, 12 up, 12 in 2024-08-30T23:33:56.000 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:33:56.000 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3720722303' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-30T23:33:56.000 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:55 smithi136 ceph-mon[20207]: pgmap v747: 1 pgs: 1 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:56.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:56.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:56.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:56.200 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:33:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:33:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:33:56.200 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:33:56.200 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:33:56.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:56.266 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:56.279 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:56.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-08-30T23:33:56.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.402 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:56.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:56.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:56.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:56.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:56.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:56.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:56.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:56.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:33:56.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:56.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:33:56.930 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3720722303' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-30T23:33:56.930 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:56 smithi106 ceph-mon[16603]: osdmap e306: 12 total, 12 up, 12 in 2024-08-30T23:33:56.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:56 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:56.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:56 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:56.986 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:33:56.987 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:33:57.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:33:57.007 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:33:57.008 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:33:57.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:33:57.029 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:33:57.029 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:33:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3720722303' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-30T23:33:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:56 smithi136 ceph-mon[20207]: osdmap e306: 12 total, 12 up, 12 in 2024-08-30T23:33:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:56 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:56 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:57.048 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:33:57.049 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:33:57.070 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:33:57.070 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:33:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3720722303' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-30T23:33:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:56 smithi173 ceph-mon[20554]: osdmap e306: 12 total, 12 up, 12 in 2024-08-30T23:33:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:56 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:56 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:57.090 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:33:57.091 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:33:57.112 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:33:57.112 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:33:57.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:57.133 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:33:57.133 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:33:57.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:57.154 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:33:57.155 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:33:57.177 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:33:57.177 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:33:57.201 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:33:57.201 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:33:57.217 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:57.224 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:33:57.224 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:33:57.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:57.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:57.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:57.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:57.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:57.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:57.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:57.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:33:57.642 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:57.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:33:57.678 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.7 primary-affinity to 0.347892 (8227992) 2024-08-30T23:33:57.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:57.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: osdmap e307: 12 total, 12 up, 12 in 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2432699254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/623101263' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.34789233084308246} : dispatch 2024-08-30T23:33:57.854 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:57 smithi106 ceph-mon[16603]: pgmap v750: 17 pgs: 2 creating+peering, 11 unknown, 4 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:57.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:57.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:57.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:57.921 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:57.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.018 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:58.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: osdmap e307: 12 total, 12 up, 12 in 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2432699254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/623101263' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.34789233084308246} : dispatch 2024-08-30T23:33:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:57 smithi136 ceph-mon[20207]: pgmap v750: 17 pgs: 2 creating+peering, 11 unknown, 4 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:58.041 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:58.041 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:58.041 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: osdmap e307: 12 total, 12 up, 12 in 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2432699254' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/623101263' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.34789233084308246} : dispatch 2024-08-30T23:33:58.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:57 smithi173 ceph-mon[20554]: pgmap v750: 17 pgs: 2 creating+peering, 11 unknown, 4 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:33:58.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:58.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:58.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:58.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:58.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:33:58.388 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:33:58.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:58.502 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:58.502 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:58.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:58.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:58.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:58.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:58.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:58.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:33:58.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:58.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:33:58.957 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-08-30T23:33:58.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/623101263' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.34789233084308246}]': finished 2024-08-30T23:33:58.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:58 smithi106 ceph-mon[16603]: osdmap e308: 12 total, 12 up, 12 in 2024-08-30T23:33:58.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:58 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:33:58.958 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3961873026' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:33:58.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:33:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:59.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:33:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/623101263' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.34789233084308246}]': finished 2024-08-30T23:33:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:58 smithi136 ceph-mon[20207]: osdmap e308: 12 total, 12 up, 12 in 2024-08-30T23:33:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:58 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:33:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3961873026' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:33:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/623101263' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.34789233084308246}]': finished 2024-08-30T23:33:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:58 smithi173 ceph-mon[20554]: osdmap e308: 12 total, 12 up, 12 in 2024-08-30T23:33:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:58 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:33:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3961873026' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:33:59.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:33:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:59.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:33:59.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:33:59.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:33:59.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:33:59.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:33:59.378 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:33:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:33:59.451 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:33:59.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:33:59.489 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:33:59.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:33:59.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:33:59.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:33:59.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:33:59.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:33:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:33:59.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:33:59.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:33:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:33:59.888 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:33:59.960 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:33:59.960 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:33:59.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:00.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:00.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:00.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:00.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:00.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3961873026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:34:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:59 smithi136 ceph-mon[20207]: osdmap e309: 12 total, 12 up, 12 in 2024-08-30T23:34:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:33:59 smithi136 ceph-mon[20207]: pgmap v753: 17 pgs: 2 creating+peering, 5 unknown, 10 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:34:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3961873026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:34:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:59 smithi173 ceph-mon[20554]: osdmap e309: 12 total, 12 up, 12 in 2024-08-30T23:34:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:33:59 smithi173 ceph-mon[20554]: pgmap v753: 17 pgs: 2 creating+peering, 5 unknown, 10 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:34:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3961873026' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:34:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:59 smithi106 ceph-mon[16603]: osdmap e309: 12 total, 12 up, 12 in 2024-08-30T23:34:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:33:59 smithi106 ceph-mon[16603]: pgmap v753: 17 pgs: 2 creating+peering, 5 unknown, 10 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:34:00.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:00.383 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:00.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:00.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:00.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:34:00.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:00.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:00.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:34:00.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:00.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:00.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:00.829 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:00.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:00.884 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:00.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:00.925 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:34:00.968 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:34:00.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:01.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:01.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:01.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:01.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:01.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:00 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:00 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:00 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:34:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:00 smithi136 ceph-mon[20207]: osdmap e310: 12 total, 12 up, 12 in 2024-08-30T23:34:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2243067577' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:01.312 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:00 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:01.312 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:00 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:01.312 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:00 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:34:01.312 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:00 smithi173 ceph-mon[20554]: osdmap e310: 12 total, 12 up, 12 in 2024-08-30T23:34:01.312 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2243067577' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:00 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:00 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:00 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:34:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:00 smithi106 ceph-mon[16603]: osdmap e310: 12 total, 12 up, 12 in 2024-08-30T23:34:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2243067577' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:01.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:01.390 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-08-30T23:34:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:01.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:01.394 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:34:01.394 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:34:01.411 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:01.412 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:01.459 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:34:01.459 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:34:01.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_74914 2024-08-30T23:34:01.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:34:01.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:34:01.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:01.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:01.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:01.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:01.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:01.782 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:01.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:01.908 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:01.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:01.964 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:01.964 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:01.985 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:01.986 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:02.006 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:02.006 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:02.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:02.028 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:02.028 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:02.032 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:02.049 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:02.050 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:02.070 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:02.070 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:01 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:01 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:34:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2243067577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:01 smithi173 ceph-mon[20554]: osdmap e311: 12 total, 12 up, 12 in 2024-08-30T23:34:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:01 smithi173 ceph-mon[20554]: pgmap v756: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:34:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1243640511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:02.093 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:02.093 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:02.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:01 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:02.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:01 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:34:02.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2243067577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:02.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:01 smithi106 ceph-mon[16603]: osdmap e311: 12 total, 12 up, 12 in 2024-08-30T23:34:02.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:01 smithi106 ceph-mon[16603]: pgmap v756: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:34:02.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1243640511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:02.116 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:02.117 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:02.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:02.139 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:02.139 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:02.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:02.161 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:02.162 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:02.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:02.186 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:02.210 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:02.210 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:02.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:34:02.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:02.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:01 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:01 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:34:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2243067577' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:01 smithi136 ceph-mon[20207]: osdmap e311: 12 total, 12 up, 12 in 2024-08-30T23:34:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:01 smithi136 ceph-mon[20207]: pgmap v756: 17 pgs: 17 active+clean; 577 KiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:34:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1243640511' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:02.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:02.404 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:02.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1718 1702 106.369 106.375 0.0578226 0.00895243 2024-08-30T23:34:02.498 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:34:02.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:02.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:02.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:02.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:02.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:02.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:02.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:02.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:02.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:03.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:03.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:03.066 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:22817 2.c 2:33be2d6c:::benchmark_data_smithi173_47195_object22816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:15.136335+0000", 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 467.92866868599998, 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23528854799999999, 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.067 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22817 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136335+0000", 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136335+0000", 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136337+0000", 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3360000000000002e-06 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136361+0000", 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3737999999999999e-05 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136377+0000", 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6540000000000001e-05 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136387+0000", 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7489999999999993e-06 2024-08-30T23:34:03.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136441+0000", 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4045000000000002e-05 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136531+0000", 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9345999999999995e-05 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:34:03.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.136621+0000", 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0836999999999998e-05 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.137400+0000", 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077874499999999996 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.137799+0000", 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039926799999999999 2024-08-30T23:34:03.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.371543+0000", 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.233743644 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.371579+0000", 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5593999999999998e-05 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:15.371623+0000", 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4706000000000002e-05 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:62896 2.a 2:5bc7d7b5:::benchmark_data_smithi173_47195_object62895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 437.15312920999997, 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.272135707, 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.074 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62896 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911875+0000", 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.099e-06 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911895+0000", 2024-08-30T23:34:03.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9724999999999998e-05 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911907+0000", 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1423000000000001e-05 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911912+0000", 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5160000000000002e-06 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911972+0000", 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9698999999999999e-05 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912062+0000", 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0204999999999994e-05 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:34:03.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912153+0000", 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0696e-05 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.913669+0000", 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001516807 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.914495+0000", 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082544200000000002 2024-08-30T23:34:03.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183885+0000", 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26939059300000001 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183937+0000", 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1586999999999999e-05 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.184010+0000", 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2915000000000002e-05 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:75402 2.c e132/97 ondisk, result = 0)", 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 426.44200306599998, 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.335240237, 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.081 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:34:03.082 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623001+0000", 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2099999999999996e-07 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623007+0000", 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7710000000000003e-06 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.957925+0000", 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33491853599999999 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.957938+0000", 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3142999999999999e-05 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958067+0000", 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012853699999999999 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958082+0000", 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4620000000000001e-05 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958241+0000", 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015900899999999999 2024-08-30T23:34:03.086 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75311 2.b 2:d9413190:::benchmark_data_smithi173_47195_object75310:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276361+0000", 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.788642283, 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23327812000000001, 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.087 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75311 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276361+0000", 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276361+0000", 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276362+0000", 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9299999999999998e-07 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276381+0000", 2024-08-30T23:34:03.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8938999999999999e-05 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276390+0000", 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8710000000000003e-06 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276397+0000", 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2479999999999999e-06 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277483+0000", 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010856990000000001 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277493+0000", 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.083e-05 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.498510+0000", 2024-08-30T23:34:03.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22101696900000001 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.498976+0000", 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000465797 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.499083+0000", 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010696000000000001 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506114+0000", 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0070305740000000004 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506156+0000", 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2466000000000003e-05 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.508423+0000", 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022673319999999999 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.508461+0000", 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7796000000000002e-05 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.509639+0000", 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011780460000000001 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.094 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75355 2.b 2:df42354f:::benchmark_data_smithi173_47195_object75354:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276407+0000", 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.78859618799999, 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23431305799999999, 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.095 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75355 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276407+0000", 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276407+0000", 2024-08-30T23:34:03.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276408+0000", 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276439+0000", 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0859999999999999e-05 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276447+0000", 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6759999999999996e-06 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276453+0000", 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0619999999999997e-06 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277535+0000", 2024-08-30T23:34:03.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010818259999999999 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277544+0000", 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0690000000000008e-06 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.501083+0000", 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.223538336 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.501222+0000", 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139386 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.501382+0000", 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016007400000000001 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506199+0000", 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048170679999999999 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506207+0000", 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3010000000000007e-06 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.509692+0000", 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034849220000000001 2024-08-30T23:34:03.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.509732+0000", 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9243999999999999e-05 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510720+0000", 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098875500000000002 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.102 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75381 2.b 2:d6004728:::benchmark_data_smithi173_47195_object75380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.78854156, 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24515187899999999, 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75381 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276462+0000", 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4000000000000002e-07 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276511+0000", 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8109000000000003e-05 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276520+0000", 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3710000000000006e-06 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276527+0000", 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8129999999999998e-06 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277587+0000", 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001060088 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277595+0000", 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7950000000000008e-06 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.503269+0000", 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22567431399999999 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.503364+0000", 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4963999999999994e-05 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.503466+0000", 2024-08-30T23:34:03.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101851 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506212+0000", 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027464659999999999 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510783+0000", 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045707569999999999 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.520094+0000", 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0093106439999999999 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.520133+0000", 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9777999999999998e-05 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.521614+0000", 2024-08-30T23:34:03.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014803889999999999 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75430 2.b 2:d9f4526d:::benchmark_data_smithi173_47195_object75429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.788468685, 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24608438699999999, 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.110 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75430 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276535+0000", 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5200000000000002e-07 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276577+0000", 2024-08-30T23:34:03.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1709999999999999e-05 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276585+0000", 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.541e-06 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276592+0000", 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9269999999999999e-06 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277626+0000", 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010331089999999999 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277630+0000", 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3579999999999996e-06 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.504828+0000", 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.227198493 2024-08-30T23:34:03.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.504914+0000", 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5351000000000003e-05 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.505024+0000", 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110031 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.507311+0000", 2024-08-30T23:34:03.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00228766 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510833+0000", 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035211420000000001 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.521663+0000", 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010829979999999999 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.521700+0000", 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7666000000000002e-05 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.522619+0000", 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091896700000000003 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:75433 2.b 2:d49bdc78:::benchmark_data_smithi173_47195_object75432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e134)", 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.78840194499998, 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24810786800000001, 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75433 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276602+0000", 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9799999999999999e-07 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276644+0000", 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2070000000000002e-05 2024-08-30T23:34:03.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276652+0000", 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4789999999999996e-06 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.276660+0000", 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1520000000000003e-06 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277668+0000", 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010085370000000001 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.277673+0000", 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8300000000000003e-06 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506298+0000", 2024-08-30T23:34:03.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22862464800000001 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506376+0000", 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8415000000000001e-05 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.506472+0000", 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5729000000000005e-05 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.510841+0000", 2024-08-30T23:34:03.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043695649999999997 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.515766+0000", 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049242940000000001 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.523643+0000", 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078772089999999996 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.523685+0000", 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1755000000000003e-05 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:58.524709+0000", 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010247870000000001 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:110499 2.b 2:d409eb3a:::benchmark_data_smithi173_47195_object13491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 376.99019400999998, 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25912935999999998, 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 110499 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:34:03.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074810+0000", 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7499999999999998e-07 2024-08-30T23:34:03.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074815+0000", 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5700000000000003e-06 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074826+0000", 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.101e-05 2024-08-30T23:34:03.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074832+0000", 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.643e-06 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074884+0000", 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2614999999999997e-05 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074986+0000", 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101808 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.075067+0000", 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0661000000000006e-05 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.077720+0000", 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026533749999999999 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.078416+0000", 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069541700000000004 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333266+0000", 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25485053000000002 2024-08-30T23:34:03.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333335+0000", 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8868999999999994e-05 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333939+0000", 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000603887 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.134 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:21973 3.8 3:16383432:::benchmark_data_smithi173_60780_object21972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:25.267447+0000", 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 217.79755655599999, 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22566488500000001, 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:34:03.135 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21973 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267447+0000", 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267447+0000", 2024-08-30T23:34:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267448+0000", 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5099999999999999e-07 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267469+0000", 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1212999999999998e-05 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267477+0000", 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8089999999999993e-06 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267482+0000", 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1699999999999996e-06 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267528+0000", 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6592999999999999e-05 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267640+0000", 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111897 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.267771+0000", 2024-08-30T23:34:03.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013064500000000001 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.268634+0000", 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086273700000000003 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.269874+0000", 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001240732 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.493035+0000", 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22316090799999999 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.493077+0000", 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1498999999999997e-05 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:25.493112+0000", 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4931000000000001e-05 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.141 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:41062 3.f e226/206 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head v 226'2525, mlcod=226'2521)", 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 199.86267169300001, 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32836641100000002, 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202340+0000", 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9529999999999999e-06 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202390+0000", 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0473999999999998e-05 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529209+0000", 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32681829099999998 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529215+0000", 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8379999999999996e-06 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529257+0000", 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2036999999999997e-05 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529272+0000", 2024-08-30T23:34:03.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4331999999999999e-05 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530655+0000", 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001382961 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530698+0000", 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3525000000000001e-05 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103636 3.e 3:73e363f7:::benchmark_data_smithi173_60780_object182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:42.426866+0000", 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 140.63813712199999, 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.229269273, 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:03.147 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103636 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426866+0000", 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426866+0000", 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426867+0000", 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426872+0000", 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5510000000000001e-06 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426881+0000", 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0790000000000007e-06 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.426897+0000", 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5829e-05 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.430114+0000", 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032173000000000002 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.430305+0000", 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019137900000000001 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.430427+0000", 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121854 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.434119+0000", 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036919349999999999 2024-08-30T23:34:03.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.436376+0000", 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022569069999999998 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656042+0000", 2024-08-30T23:34:03.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21966624400000001 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656081+0000", 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9029000000000002e-05 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:42.656136+0000", 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4397000000000001e-05 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.153 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:199193 3.0 3:0da882ce:::benchmark_data_smithi173_60780_object95739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.707113104000001, 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30993041100000002, 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:34:03.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199193 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357892+0000", 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.189e-06 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357896+0000", 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5090000000000004e-06 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357907+0000", 2024-08-30T23:34:03.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0607e-05 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357912+0000", 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2190000000000003e-06 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357956+0000", 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3745000000000002e-05 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.358054+0000", 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7940000000000001e-05 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.358129+0000", 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5887e-05 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362642+0000", 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045129510000000003 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.364032+0000", 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00138984 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667719+0000", 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30368694699999998 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667778+0000", 2024-08-30T23:34:03.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8754e-05 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667821+0000", 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2823e-05 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:199193 3.0 e299/206 ondisk, result = 0)", 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:34:03.160 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.702650375000001, 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30548378700000001, 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362354+0000", 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1060000000000001e-06 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362361+0000", 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4230000000000002e-06 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667470+0000", 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30510947599999999 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667476+0000", 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8379999999999998e-06 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667704+0000", 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022796099999999999 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667712+0000", 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4749999999999993e-06 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667837+0000", 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012450800000000001 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.164 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:199216 3.0 3:0986db6c:::benchmark_data_smithi173_60780_object95762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.666044636999999, 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28454402699999998, 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199216 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398960+0000", 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0100000000000001e-06 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398972+0000", 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2381000000000001e-05 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398982+0000", 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0236e-05 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398988+0000", 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5940000000000001e-06 2024-08-30T23:34:03.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399037+0000", 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8613999999999999e-05 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399169+0000", 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132657 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:34:03.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399297+0000", 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127732 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.403718+0000", 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044206580000000001 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.403817+0000", 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9599999999999995e-05 2024-08-30T23:34:03.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683416+0000", 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27959865299999997 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683467+0000", 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1248999999999997e-05 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683503+0000", 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5642999999999997e-05 2024-08-30T23:34:03.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:199216 3.0 e299/206 ondisk, result = 0)", 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.660836290999999, 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27933989799999998, 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404168+0000", 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7700000000000004e-07 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404173+0000", 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8230000000000002e-06 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.679627+0000", 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27545433000000002 2024-08-30T23:34:03.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.679635+0000", 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9200000000000004e-06 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683406+0000", 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037705299999999998 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683412+0000", 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5640000000000002e-06 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683507+0000", 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4953999999999999e-05 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:206901 3.7 e300/206 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head v 300'13076, mlcod=300'13071)", 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:34:03.175 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 19.000858654999998, 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38146454200000002, 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064146+0000", 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8000000000000005e-07 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064151+0000", 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8270000000000003e-06 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441558+0000", 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37740696499999998 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441573+0000", 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4623e-05 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441672+0000", 2024-08-30T23:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9351999999999993e-05 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441691+0000", 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8134999999999998e-05 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445570+0000", 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003879511 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445609+0000", 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9348999999999999e-05 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87cb8fd9:::benchmark_data_smithi173_60780_object16345:head,262'7462)", 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 11.118752508, 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72540086000000004, 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.180 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946252+0000", 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3399999999999999e-07 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946257+0000", 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349999999999999e-06 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671354+0000", 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72509723199999998 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671379+0000", 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5279e-05 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671555+0000", 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000175203 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671572+0000", 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.774e-05 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671652+0000", 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9536999999999994e-05 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87b81cd9:::benchmark_data_smithi173_60780_object18028:head,263'7563)", 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:53.456103+0000", 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.6089005140000001, 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.238580758, 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.456103+0000", 2024-08-30T23:34:03.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.456103+0000", 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.456104+0000", 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7999999999999995e-07 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.456108+0000", 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5689999999999998e-06 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.694497+0000", 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23838899999999999 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.694509+0000", 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2334000000000001e-05 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.694613+0000", 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010367299999999999 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.694622+0000", 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9080000000000001e-06 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.694684+0000", 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1693999999999996e-05 2024-08-30T23:34:03.188 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87ea66dc:::benchmark_data_smithi173_60780_object18546:head,263'7595)", 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.1824041180000009, 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85628250299999997, 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:34:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882600+0000", 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2024-08-30T23:34:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882604+0000", 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2289999999999997e-06 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738700+0000", 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85609620099999995 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:34:03.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738719+0000", 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8343e-05 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738882+0000", 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016310000000000001 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:34:03.192 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:34:03.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:03.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:03.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:03.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:03.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3506 3490 109.055 111.75 0.0122692 0.00914019 2024-08-30T23:34:03.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:03.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:03.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:03.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:03.730 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:03.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:34:03.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:03.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:03.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:03.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:03.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:04.086 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:04.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:04.097 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:34:04.097 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:34:04.097 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:34:04.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-08-30T23:34:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:03 smithi106 ceph-mon[16603]: osdmap e312: 12 total, 12 up, 12 in 2024-08-30T23:34:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:03 smithi106 ceph-mon[16603]: pgmap v758: 17 pgs: 17 active+clean; 44 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 155 op/s 2024-08-30T23:34:04.116 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:04.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:34:04.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:04.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:03 smithi136 ceph-mon[20207]: osdmap e312: 12 total, 12 up, 12 in 2024-08-30T23:34:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:03 smithi136 ceph-mon[20207]: pgmap v758: 17 pgs: 17 active+clean; 44 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 155 op/s 2024-08-30T23:34:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:03 smithi173 ceph-mon[20554]: osdmap e312: 12 total, 12 up, 12 in 2024-08-30T23:34:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:03 smithi173 ceph-mon[20554]: pgmap v758: 17 pgs: 17 active+clean; 44 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s wr, 155 op/s 2024-08-30T23:34:04.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:04.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:04.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5572 5556 115.74 129.125 0.0337554 0.00846313 2024-08-30T23:34:04.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:04.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:04.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.628 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:04.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:04.655 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:04.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:04.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:04.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:04.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:04.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:04.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:34:04.939 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:04.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:04.986 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:34:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:05.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:05.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2467614332' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:34:05.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:05.151 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:05.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:05.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2467614332' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:34:05.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2467614332' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:34:05.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:05.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:moved 0 / 51 (0%) 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.16506 -> 2.16506 (expected baseline 1.97379) 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 2 -> 2 pgs (0.470588 -> 0.470588 * mean) 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.1 with 9 -> 9 pgs (2.11765 -> 2.11765 * mean) 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:34:05.457 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:34:05.458 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:34:05.458 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0023 2024-08-30T23:34:05.458 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0229 2024-08-30T23:34:05.458 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6 weight 0.9818 -> 1.0000 2024-08-30T23:34:05.458 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:34:05.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 7443 7427 116.036 116.938 0.00123844 0.00856743 2024-08-30T23:34:05.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:05.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:05.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:05.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:05.734 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:34:05.456+0000 7f2243404640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-08-30T23:34:05.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:05.787 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:05.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:05.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:05.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:06.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:06.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2467614332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:34:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: osdmap e313: 12 total, 12 up, 12 in 2024-08-30T23:34:06.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: pgmap v760: 17 pgs: 17 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 572 op/s 2024-08-30T23:34:06.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:05 smithi106 ceph-mon[16603]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:06.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:06.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:06.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2467614332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:34:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: osdmap e313: 12 total, 12 up, 12 in 2024-08-30T23:34:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: pgmap v760: 17 pgs: 17 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 572 op/s 2024-08-30T23:34:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:05 smithi136 ceph-mon[20207]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:06.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2467614332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:34:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: osdmap e313: 12 total, 12 up, 12 in 2024-08-30T23:34:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: pgmap v760: 17 pgs: 17 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 572 op/s 2024-08-30T23:34:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:05 smithi173 ceph-mon[20554]: from='client.25640 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:06.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:9012 4.b 4:d58c88d2:::benchmark_data_smithi173_74914_object9011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:06.410453+0000", 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0030980650000000001, 2024-08-30T23:34:06.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031192149999999998, 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 9012 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:06.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410453+0000", 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410453+0000", 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:06.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410454+0000", 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.55e-06 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410515+0000", 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1367999999999999e-05 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410529+0000", 2024-08-30T23:34:06.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3101e-05 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410536+0000", 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8329999999999994e-06 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410583+0000", 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6897000000000002e-05 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410672+0000", 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8468999999999996e-05 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.410759+0000", 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7346e-05 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.411334+0000", 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000574615 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:06.411800+0000", 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00046579100000000001 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:06.420 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:06.421 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:06.421 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:34:06.421 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-08-30T23:34:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:34:06.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:06.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 9144 9128 114.089 106.312 0.00170456 0.00875183 2024-08-30T23:34:06.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:06.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:06.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:06.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:06.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:06.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:06.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:06.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:06.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:07.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:07.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:07.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:07.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:07.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:07.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:07.320 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:34:07.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-08-30T23:34:07.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:07.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:07.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 10651 10635 110.771 94.1875 0.0386525 0.00898978 2024-08-30T23:34:07.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:07.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:07.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:07.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:07.826 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:07.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:07.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:07.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:08.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:08.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:08.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:34:08.155 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:08.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:08.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:34:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.227 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:08.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:08.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:08.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:08.419 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:08.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:08.460 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 12865 12849 114.713 138.375 0.0369978 0.00865985 2024-08-30T23:34:08.474 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:08.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:08.603 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:08.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:08.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:08.696 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:08.696 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:08 smithi106 ceph-mon[16603]: pgmap v761: 17 pgs: 17 active+clean; 374 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s 2024-08-30T23:34:08.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:08.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:08.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:08.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:08.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:08.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:08.774 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:08.774 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:08.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:08 smithi136 ceph-mon[20207]: pgmap v761: 17 pgs: 17 active+clean; 374 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s 2024-08-30T23:34:08.798 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:08.798 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:08.820 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:08.820 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:08 smithi173 ceph-mon[20554]: pgmap v761: 17 pgs: 17 active+clean; 374 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 920 op/s 2024-08-30T23:34:08.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:08.841 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:08.841 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:08.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:08.863 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:08.863 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:08.886 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:08.886 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:08.909 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:08.909 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:08.935 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:08.935 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:08.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:08.962 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:08.962 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:08.984 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:08.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:09.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:09.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:09.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:09.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:09.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:09.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:09.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 14419 14403 112.513 97.125 0.0287401 0.0088772 2024-08-30T23:34:09.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:09.491 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:09.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:09.616 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:09.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:09.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4097379259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4097379259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:09.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4097379259' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:09.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:09.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:09.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:09.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:10.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:10.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:10.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:10.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:10.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:10.375 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:10.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 15973 15957 110.803 97.125 0.0016613 0.00901108 2024-08-30T23:34:10.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:10.500 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:10 smithi173 ceph-mon[20554]: pgmap v762: 17 pgs: 17 active+clean; 588 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-08-30T23:34:10.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:10.603 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:10.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:10.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:10.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:10.728 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:10 smithi106 ceph-mon[16603]: pgmap v762: 17 pgs: 17 active+clean; 588 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-08-30T23:34:10.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:10 smithi136 ceph-mon[20207]: pgmap v762: 17 pgs: 17 active+clean; 588 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.18k op/s 2024-08-30T23:34:10.806 INFO:teuthology.orchestra.run.smithi106.stdout:moved 0 / 51 (0%) 2024-08-30T23:34:10.806 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:34:10.806 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.16506 -> 2.16506 (expected baseline 1.97379) 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 2 -> 2 pgs (0.470588 -> 0.470588 * mean) 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.1 with 9 -> 9 pgs (2.11765 -> 2.11765 * mean) 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 0.05 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0035 2024-08-30T23:34:10.807 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0352 2024-08-30T23:34:10.808 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:34:10.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:10.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:10.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:10.980 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:11.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:11.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:11.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:11.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:11.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:11.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:11.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:11.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 17515 17499 109.359 96.375 0.0194884 0.00912892 2024-08-30T23:34:11.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:11.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:11.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:11.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:11.680 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:34:11.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-08-30T23:34:11.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:11.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:11.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:11.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:11.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:11.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:11 smithi106 ceph-mon[16603]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:11.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:11 smithi106 ceph-mon[16603]: pgmap v763: 17 pgs: 17 active+clean; 827 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-08-30T23:34:11.990 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:11.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:12.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:12.022 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:11 smithi136 ceph-mon[20207]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:11 smithi136 ceph-mon[20207]: pgmap v763: 17 pgs: 17 active+clean; 827 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-08-30T23:34:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:11 smithi173 ceph-mon[20554]: from='client.16065 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:11 smithi173 ceph-mon[20554]: pgmap v763: 17 pgs: 17 active+clean; 827 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.48k op/s 2024-08-30T23:34:12.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:12.135 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:12.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:12.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:12.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:12.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:12.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:34:12.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 18674 18658 106.002 72.4375 0.00736561 0.0094241 2024-08-30T23:34:12.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:12.518 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:12.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:12.634 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:34:12.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:12.725 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:12.726 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:12.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2604537158' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:12.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:12.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:12.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:12.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:12.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:13.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:13.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2604537158' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2604537158' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:13.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:13.146 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:13.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:13.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:13.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:13.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:13.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:13.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 19808 19792 103.074 70.875 0.00165982 0.00968521 2024-08-30T23:34:13.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:13.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:13.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:13.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:13.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:13.777 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:13.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:13 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:13.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2604537158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:13.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:13 smithi106 ceph-mon[16603]: osdmap e314: 12 total, 12 up, 12 in 2024-08-30T23:34:13.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:13.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:13.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:13 smithi106 ceph-mon[16603]: pgmap v765: 17 pgs: 17 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-30T23:34:13.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:13.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:13.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:14.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:14.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:13 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2604537158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:13 smithi136 ceph-mon[20207]: osdmap e314: 12 total, 12 up, 12 in 2024-08-30T23:34:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:13 smithi136 ceph-mon[20207]: pgmap v765: 17 pgs: 17 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-30T23:34:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:13 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2604537158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:13 smithi173 ceph-mon[20554]: osdmap e314: 12 total, 12 up, 12 in 2024-08-30T23:34:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:13 smithi173 ceph-mon[20554]: pgmap v765: 17 pgs: 17 active+clean; 1.0 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-30T23:34:14.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:14.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:34:14.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:14.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:14.277 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:14.355 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:14.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:14.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:14.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 21081 21065 101.265 79.5625 0.00840643 0.00986318 2024-08-30T23:34:14.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:14.531 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:14.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:14.616 INFO:teuthology.orchestra.run.smithi106.stdout:moved 1 / 51 (1.96078%) 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.16506 -> 2.12623 (expected baseline 1.97379) 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 2 -> 2 pgs (0.470588 -> 0.470588 * mean) 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.1 with 9 -> 9 pgs (2.11765 -> 2.11765 * mean) 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 0.05 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0049 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0489 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 weight 0.8072 -> 0.8572 2024-08-30T23:34:14.617 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 0.8264 -> 0.8764 2024-08-30T23:34:14.618 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:34:14.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:14.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:14.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:14.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:14.857 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:34:14 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:34:14.616+0000 7f2243404640 -1 mgr.server reply reply (2) No such file or directory no change 2024-08-30T23:34:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:14 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:14 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:14 smithi106 ceph-mon[16603]: osdmap e315: 12 total, 12 up, 12 in 2024-08-30T23:34:14.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:14.922 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:14.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:15.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:14 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:14 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:14 smithi136 ceph-mon[20207]: osdmap e315: 12 total, 12 up, 12 in 2024-08-30T23:34:15.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:14 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:14 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:14 smithi173 ceph-mon[20554]: osdmap e315: 12 total, 12 up, 12 in 2024-08-30T23:34:15.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:15.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:15.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:15.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:15.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:15.429 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:15.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 22869 22853 102.013 111.75 0.00234859 0.00966422 2024-08-30T23:34:15.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:34:15.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:15.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:15.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:15.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:15.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:15.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:15.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:15.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:15.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:15.967 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:15 smithi106 ceph-mon[16603]: from='client.25693 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:15.967 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:15 smithi106 ceph-mon[16603]: pgmap v767: 17 pgs: 17 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-30T23:34:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:15 smithi136 ceph-mon[20207]: from='client.25693 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:15 smithi136 ceph-mon[20207]: pgmap v767: 17 pgs: 17 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-30T23:34:16.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:16.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:15 smithi173 ceph-mon[20554]: from='client.25693 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:15 smithi173 ceph-mon[20554]: pgmap v767: 17 pgs: 17 active+clean; 1.2 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-08-30T23:34:16.116 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:16.116 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:16.144 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:16.144 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:16.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:16.174 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:16.174 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:16.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:16.203 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:16.203 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:16.232 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:16.232 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:16.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:16.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:16.285 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:16.285 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:16.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:16.317 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:16.317 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:16.333 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:16.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:16.350 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:16.351 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:16.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:16.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:16.402 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:16.403 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:16.427 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:16.428 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:16.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:16.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:16.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 24658 24642 102.666 111.812 0.0696995 0.00972528 2024-08-30T23:34:16.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:16.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:16.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:16.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:16.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:16.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:16.893 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-30T23:34:16.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-08-30T23:34:16.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:16.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:16.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2196546420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:16.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2196546420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:17.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2196546420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:17.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:17.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.212 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:17.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:17.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:17.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:17.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:17.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 25729 25713 100.433 66.9375 0.0506323 0.00993757 2024-08-30T23:34:17.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:17.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:17.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:17.657 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:17 smithi106 ceph-mon[16603]: pgmap v768: 17 pgs: 17 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-30T23:34:17.674 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:17.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:17.713 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:17.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:17.717 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:17.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:17.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:17.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:17.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:17.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:17 smithi136 ceph-mon[20207]: pgmap v768: 17 pgs: 17 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-30T23:34:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:17 smithi173 ceph-mon[20554]: pgmap v768: 17 pgs: 17 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-30T23:34:18.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:18.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:18.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:18.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:18.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:18.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:18.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:18.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 26997 26981 99.1863 79.25 0.0803085 0.0100675 2024-08-30T23:34:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:18.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:18.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:18.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:18.669 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:34:18.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:18.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1380203859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:18.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:18.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:18.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14225 2.5 e109/97 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head v 109'898, mlcod=109'894)", 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 490.03645326499998, 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.239755616, 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:18.997 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:18.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874011+0000", 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874108+0000", 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6961999999999998e-05 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:18.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112976+0000", 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23886803600000001 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112983+0000", 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.951e-06 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113055+0000", 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113079+0000", 2024-08-30T23:34:19.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4754000000000001e-05 2024-08-30T23:34:19.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113724+0000", 2024-08-30T23:34:19.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064475700000000003 2024-08-30T23:34:19.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113766+0000", 2024-08-30T23:34:19.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1664e-05 2024-08-30T23:34:19.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.003 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16156 2.4 e109/97 2:25a56bae:::benchmark_data_smithi173_47195_object16155:head v 109'956, mlcod=109'953)", 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 488.81237256899999, 2024-08-30T23:34:19.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.213651339, 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:34:19.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098091+0000", 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098092+0000", 2024-08-30T23:34:19.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4300000000000002e-07 2024-08-30T23:34:19.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.098114+0000", 2024-08-30T23:34:19.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2234000000000001e-05 2024-08-30T23:34:19.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310963+0000", 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21284856199999999 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.310969+0000", 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:34:19.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311028+0000", 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8857999999999998e-05 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311051+0000", 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2538999999999999e-05 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311702+0000", 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065106500000000004 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311742+0000", 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0318000000000003e-05 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.41269794700003, 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:34:19.015 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109556 2.5 2:a6b86250:::benchmark_data_smithi173_47195_object12548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.05611496199998, 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220264298, 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.016 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109556 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854350+0000", 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8840000000000001e-06 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854356+0000", 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6590000000000004e-06 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854369+0000", 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.289e-05 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854377+0000", 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4649999999999994e-06 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854454+0000", 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6791000000000004e-05 2024-08-30T23:34:19.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854600+0000", 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145864 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854751+0000", 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015046499999999999 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.857557+0000", 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028060419999999999 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.873657+0000", 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016100032 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073675+0000", 2024-08-30T23:34:19.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200018474 2024-08-30T23:34:19.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073721+0000", 2024-08-30T23:34:19.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6072999999999997e-05 2024-08-30T23:34:19.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.074613+0000", 2024-08-30T23:34:19.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089165900000000005 2024-08-30T23:34:19.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.024 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.024 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.024 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109564 2.4 2:242bfffa:::benchmark_data_smithi173_47195_object12556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:19.024 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:34:19.025 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.048670942, 2024-08-30T23:34:19.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220047624, 2024-08-30T23:34:19.025 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.025 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.025 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109564 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:34:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:34:19.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861794+0000", 2024-08-30T23:34:19.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5889999999999999e-06 2024-08-30T23:34:19.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861800+0000", 2024-08-30T23:34:19.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:34:19.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861809+0000", 2024-08-30T23:34:19.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999997e-06 2024-08-30T23:34:19.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861815+0000", 2024-08-30T23:34:19.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:34:19.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861910+0000", 2024-08-30T23:34:19.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5278000000000005e-05 2024-08-30T23:34:19.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862048+0000", 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137911 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:34:19.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862173+0000", 2024-08-30T23:34:19.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012423299999999999 2024-08-30T23:34:19.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878997+0000", 2024-08-30T23:34:19.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016824741000000001 2024-08-30T23:34:19.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879048+0000", 2024-08-30T23:34:19.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0890999999999998e-05 2024-08-30T23:34:19.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080272+0000", 2024-08-30T23:34:19.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20122369600000001 2024-08-30T23:34:19.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080313+0000", 2024-08-30T23:34:19.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0785999999999999e-05 2024-08-30T23:34:19.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.081840+0000", 2024-08-30T23:34:19.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015274679999999999 2024-08-30T23:34:19.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.039 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.039 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.039 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109567 2.4 2:235a232e:::benchmark_data_smithi173_47195_object12559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:19.040 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:34:19.040 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.043616117, 2024-08-30T23:34:19.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.218097968, 2024-08-30T23:34:19.040 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.040 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.041 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109567 2024-08-30T23:34:19.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.041 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:34:19.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866847+0000", 2024-08-30T23:34:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866849+0000", 2024-08-30T23:34:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:34:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866855+0000", 2024-08-30T23:34:19.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1199999999999999e-06 2024-08-30T23:34:19.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866877+0000", 2024-08-30T23:34:19.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1954e-05 2024-08-30T23:34:19.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.866885+0000", 2024-08-30T23:34:19.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9070000000000007e-06 2024-08-30T23:34:19.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869749+0000", 2024-08-30T23:34:19.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028635940000000001 2024-08-30T23:34:19.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869826+0000", 2024-08-30T23:34:19.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7583999999999994e-05 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869907+0000", 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0408999999999995e-05 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879006+0000", 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090992710000000008 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883826+0000", 2024-08-30T23:34:19.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048203719999999999 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083592+0000", 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.199765677 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.083632+0000", 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0262000000000002e-05 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.084945+0000", 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013130590000000001 2024-08-30T23:34:19.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109569 2.5 2:a8435ff8:::benchmark_data_smithi173_47195_object12561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.040973764, 2024-08-30T23:34:19.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22083613699999999, 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109569 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869491+0000", 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4270000000000001e-06 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869498+0000", 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869510+0000", 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1630999999999999e-05 2024-08-30T23:34:19.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869519+0000", 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869586+0000", 2024-08-30T23:34:19.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6699999999999995e-05 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869741+0000", 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155141 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869888+0000", 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014657099999999999 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879878+0000", 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099902989999999994 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879907+0000", 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8612999999999999e-05 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089168+0000", 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20926119900000001 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089267+0000", 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9097000000000003e-05 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090326+0000", 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010589670000000001 2024-08-30T23:34:19.060 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109574 2.5 2:aaf91c06:::benchmark_data_smithi173_47195_object12566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.03440794099998, 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21517555899999999, 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.061 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109574 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876056+0000", 2024-08-30T23:34:19.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876058+0000", 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.046e-06 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876065+0000", 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876084+0000", 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8702000000000002e-05 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876094+0000", 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5850000000000006e-06 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876163+0000", 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9441000000000001e-05 2024-08-30T23:34:19.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876332+0000", 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016932699999999999 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.876473+0000", 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014066600000000001 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882795+0000", 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063221149999999997 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.883230+0000", 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043490299999999999 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090374+0000", 2024-08-30T23:34:19.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20714431699999999 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090425+0000", 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0865000000000002e-05 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091231+0000", 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00080588100000000005 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.067 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109576 2.5 2:ae1552f5:::benchmark_data_smithi173_47195_object12568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 394.03196412599999, 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21498098400000001, 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109576 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878499+0000", 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878501+0000", 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.751e-06 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878508+0000", 2024-08-30T23:34:19.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8340000000000001e-06 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878521+0000", 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2669e-05 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878529+0000", 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.055e-06 2024-08-30T23:34:19.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880019+0000", 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001490686 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880144+0000", 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012418800000000001 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.880266+0000", 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000122552 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.882939+0000", 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026733849999999999 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.917160+0000", 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034220360999999998 2024-08-30T23:34:19.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092559+0000", 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.175398734 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.092623+0000", 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4208999999999995e-05 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.093480+0000", 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00085756000000000005 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144821 2.4 2:2415a84c:::benchmark_data_smithi173_47195_object47813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 347.34707978900002, 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23107453999999999, 2024-08-30T23:34:19.075 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144821 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563385+0000", 2024-08-30T23:34:19.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.59e-06 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563391+0000", 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563404+0000", 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2982e-05 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563411+0000", 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565303+0000", 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001891846 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565431+0000", 2024-08-30T23:34:19.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012746400000000001 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565550+0000", 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972900000000001 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.573426+0000", 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078756799999999995 2024-08-30T23:34:19.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588851+0000", 2024-08-30T23:34:19.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015424679 2024-08-30T23:34:19.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793322+0000", 2024-08-30T23:34:19.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20447127600000001 2024-08-30T23:34:19.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793387+0000", 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4586999999999997e-05 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.794458+0000", 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010715060000000001 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.084 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144836 2.4 2:2d7ce86c:::benchmark_data_smithi173_47195_object47828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 347.322346423, 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23589892500000001, 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.085 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144836 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:34:19.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588118+0000", 2024-08-30T23:34:19.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3829999999999999e-06 2024-08-30T23:34:19.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588123+0000", 2024-08-30T23:34:19.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-08-30T23:34:19.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588169+0000", 2024-08-30T23:34:19.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199999999999998e-05 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588182+0000", 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2748999999999999e-05 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.592926+0000", 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047443779999999996 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595020+0000", 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002094243 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595429+0000", 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040842299999999999 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603861+0000", 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0084324050000000005 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.604206+0000", 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034452400000000002 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.822951+0000", 2024-08-30T23:34:19.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21874496900000001 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.823010+0000", 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9601000000000001e-05 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824016+0000", 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010057499999999999 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155204 2.4 2:2613b3a5:::benchmark_data_smithi173_47195_object58196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 333.13406554699998, 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21707409, 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155204 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.095 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776398+0000", 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776400+0000", 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8649999999999999e-06 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776406+0000", 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1890000000000004e-06 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776421+0000", 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449999999999999e-05 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776430+0000", 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0420000000000008e-06 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776499+0000", 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8048999999999998e-05 2024-08-30T23:34:19.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776643+0000", 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000144721 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.776820+0000", 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017706199999999999 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.783777+0000", 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006956322 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.786557+0000", 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027807270000000002 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992894+0000", 2024-08-30T23:34:19.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20633659600000001 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.992950+0000", 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5664000000000001e-05 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:45.993472+0000", 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052240299999999995 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:193978 2.5 2:a2ee5e67:::benchmark_data_smithi173_47195_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.15431697100001, 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.283382829, 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.104 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193978 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.105 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756148+0000", 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756154+0000", 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756166+0000", 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041e-05 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756172+0000", 2024-08-30T23:34:19.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2349999999999996e-06 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756333+0000", 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000160767 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756484+0000", 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015133799999999999 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:34:19.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756627+0000", 2024-08-30T23:34:19.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014263700000000001 2024-08-30T23:34:19.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764342+0000", 2024-08-30T23:34:19.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077147439999999999 2024-08-30T23:34:19.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.769055+0000", 2024-08-30T23:34:19.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047136399999999998 2024-08-30T23:34:19.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039012+0000", 2024-08-30T23:34:19.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26995667699999998 2024-08-30T23:34:19.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039055+0000", 2024-08-30T23:34:19.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2567000000000001e-05 2024-08-30T23:34:19.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039529+0000", 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047478400000000001 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:193978 2.5 e200/182 ondisk, result = 0)", 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.146361705, 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27543190299999998, 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:34:19.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764103+0000", 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764108+0000", 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9080000000000003e-06 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038928+0000", 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27481996400000003 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038937+0000", 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7860000000000008e-06 2024-08-30T23:34:19.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038998+0000", 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0915999999999997e-05 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039007+0000", 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4319999999999999e-06 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039534+0000", 2024-08-30T23:34:19.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052686200000000001 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e200,182 2:a1c9afc1:::benchmark_data_smithi173_47195_object1378:head,146'6135)", 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.12765350299998, 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42837196799999999, 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:34:19.119 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782811+0000", 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782816+0000", 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6969999999999998e-06 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210395+0000", 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42757847500000001 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210404+0000", 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0000000000000002e-06 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210493+0000", 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9438000000000007e-05 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.211182+0000", 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068873000000000003 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2685 3.6 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.216929776, 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22443937, 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.123 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2685 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693535+0000", 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.009e-06 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693573+0000", 2024-08-30T23:34:19.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7843999999999999e-05 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693589+0000", 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6359e-05 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693603+0000", 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.412e-05 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693657+0000", 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3757e-05 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693735+0000", 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8444999999999999e-05 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693820+0000", 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4290999999999996e-05 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.694189+0000", 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036947500000000002 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.874073+0000", 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.179884508 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917876+0000", 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043802633000000001 2024-08-30T23:34:19.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917931+0000", 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5297999999999998e-05 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917973+0000", 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1631000000000002e-05 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176646 3.e 3:78ef2980:::benchmark_data_smithi173_60780_object73192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 74.175629799999996, 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22395779099999999, 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:19.130 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176646 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734835+0000", 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7399999999999991e-07 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734839+0000", 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5800000000000002e-06 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734853+0000", 2024-08-30T23:34:19.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.375e-05 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734858+0000", 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738711+0000", 2024-08-30T23:34:19.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003852847 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738850+0000", 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000138818 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738954+0000", 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001037 2024-08-30T23:34:19.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748083+0000", 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091293829999999996 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748375+0000", 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000292427 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748430+0000", 2024-08-30T23:34:19.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4191999999999999e-05 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.756943+0000", 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085137770000000002 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958661+0000", 2024-08-30T23:34:19.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.201717853 2024-08-30T23:34:19.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.137 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958707+0000", 2024-08-30T23:34:19.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5840999999999997e-05 2024-08-30T23:34:19.138 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958791+0000", 2024-08-30T23:34:19.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4349000000000002e-05 2024-08-30T23:34:19.139 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.139 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.139 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.139 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176647 3.e 3:751af89f:::benchmark_data_smithi173_60780_object73193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:34:19.140 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:34:19.140 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 74.174381041999993, 2024-08-30T23:34:19.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223004061, 2024-08-30T23:34:19.140 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.140 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.140 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.141 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.141 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:19.141 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:19.141 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176647 2024-08-30T23:34:19.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.141 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:34:19.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:34:19.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736083+0000", 2024-08-30T23:34:19.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8299999999999995e-07 2024-08-30T23:34:19.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736088+0000", 2024-08-30T23:34:19.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2370000000000003e-06 2024-08-30T23:34:19.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736098+0000", 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0835e-05 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736104+0000", 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742826+0000", 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006721505 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742967+0000", 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140977 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.743123+0000", 2024-08-30T23:34:19.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015588399999999999 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748383+0000", 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052605059999999999 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.758513+0000", 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010130116 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767529+0000", 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090161789999999992 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771374+0000", 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038445660000000002 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958849+0000", 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18747534399999999 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958930+0000", 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0463000000000005e-05 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.959086+0000", 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000156701 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.150 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176663 3.e 3:70eacb77:::benchmark_data_smithi173_60780_object73209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.763338+0000", 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 74.147125453000001, 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211628873, 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176663 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763338+0000", 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763338+0000", 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763339+0000", 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.083e-06 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763343+0000", 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2560000000000004e-06 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763353+0000", 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0034000000000001e-05 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763358+0000", 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.792e-06 2024-08-30T23:34:19.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763406+0000", 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7868000000000003e-05 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763507+0000", 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100677 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.763593+0000", 2024-08-30T23:34:19.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5925000000000001e-05 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771343+0000", 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077507590000000003 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771912+0000", 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000568927 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.782646+0000", 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010733846999999999 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.782691+0000", 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4625e-05 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.974814+0000", 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19212274600000001 2024-08-30T23:34:19.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.974855+0000", 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1757e-05 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.974967+0000", 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011157700000000001 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176665 3.e 3:7a858eb9:::benchmark_data_smithi173_60780_object73211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.764487+0000", 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 74.145976008999995, 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21076267100000001, 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176665 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:19.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.764487+0000", 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.764487+0000", 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.764488+0000", 2024-08-30T23:34:19.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6899999999999996e-07 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.764493+0000", 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.121e-06 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.764502+0000", 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6590000000000002e-06 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:19.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.764508+0000", 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.28e-06 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767560+0000", 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003051144 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767653+0000", 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3270999999999995e-05 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767748+0000", 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5299999999999999e-05 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771496+0000", 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037479779999999999 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.782718+0000", 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011221971000000001 2024-08-30T23:34:19.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.783475+0000", 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00075669799999999998 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.815940+0000", 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032465330000000001 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.975023+0000", 2024-08-30T23:34:19.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15908291599999999 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.975096+0000", 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2754000000000002e-05 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.975250+0000", 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015427999999999999 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.164 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.165 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:19.165 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:19.165 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:34:19.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:19.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1380203859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:19.166 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1380203859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:19.192 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:19.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:19.320 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:19.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:19.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:19.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 28906 28890 100.304 119.312 0.0217374 0.00996288 2024-08-30T23:34:19.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:19.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:19.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:34:19.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:19.702 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:19.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.834 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:19.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:19.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:19.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:19.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:20.022 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:19 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:20.022 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1380203859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:34:20.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:19 smithi106 ceph-mon[16603]: osdmap e316: 12 total, 12 up, 12 in 2024-08-30T23:34:20.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:19 smithi106 ceph-mon[16603]: pgmap v770: 17 pgs: 17 active+clean; 1.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-30T23:34:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:19 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1380203859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:34:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:19 smithi136 ceph-mon[20207]: osdmap e316: 12 total, 12 up, 12 in 2024-08-30T23:34:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:19 smithi136 ceph-mon[20207]: pgmap v770: 17 pgs: 17 active+clean; 1.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-30T23:34:20.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:19 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1380203859' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:34:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:19 smithi173 ceph-mon[20554]: osdmap e316: 12 total, 12 up, 12 in 2024-08-30T23:34:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:19 smithi173 ceph-mon[20554]: pgmap v770: 17 pgs: 17 active+clean; 1.5 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-30T23:34:20.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:20.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:20.227 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:20.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:20.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:20.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:20.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 30766 30750 101.143 116.25 0.0221625 0.00987845 2024-08-30T23:34:20.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:34:20.486 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:20.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:20.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:20.681 INFO:teuthology.orchestra.run.smithi106.stdout:moved 8 / 51 (15.6863%) 2024-08-30T23:34:20.681 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:34:20.681 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.16506 -> 1.58771 (expected baseline 1.97379) 2024-08-30T23:34:20.681 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 2 -> 2 pgs (0.470588 -> 0.470588 * mean) 2024-08-30T23:34:20.681 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.1 with 9 -> 6 pgs (2.11765 -> 1.41176 * mean) 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:oload 110 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0062 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0068 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1 weight 1.0000 -> 0.5043 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6 weight 0.9818 -> 0.7102 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 weight 0.5971 -> 0.5274 2024-08-30T23:34:20.682 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 0.8264 -> 0.9238 2024-08-30T23:34:20.695 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:20.696 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:20.696 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:20 smithi106 ceph-mon[16603]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:20.696 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"} : dispatch 2024-08-30T23:34:20.696 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:20 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"} : dispatch 2024-08-30T23:34:20.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:20.734 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:20.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:20.857 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:20.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:20.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:20.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:20 smithi136 ceph-mon[20207]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:21.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"} : dispatch 2024-08-30T23:34:21.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:20 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"} : dispatch 2024-08-30T23:34:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:20 smithi173 ceph-mon[20554]: from='client.16095 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"} : dispatch 2024-08-30T23:34:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:20 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"} : dispatch 2024-08-30T23:34:21.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:21.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:21.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:21.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:21.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.355 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:21.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:21.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:21.461 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:34:21.462762+0000 min lat: 0.0010779 max lat: 0.766267 avg lat: 0.0100112 2024-08-30T23:34:21.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:34:21.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 31073 31057 97.0449 19.1875 0.766267 0.0100112 2024-08-30T23:34:21.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:21.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:21.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:21.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:21.713 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:34:21.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-08-30T23:34:21.718 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:21.732 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:21.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:21.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:21.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:21.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:21.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:22.000 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:21 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"}]': finished 2024-08-30T23:34:22.000 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:21 smithi106 ceph-mon[16603]: osdmap e317: 12 total, 12 up, 12 in 2024-08-30T23:34:22.000 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:21 smithi106 ceph-mon[16603]: pgmap v772: 17 pgs: 1 peering, 2 unknown, 14 active+clean; 1.7 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-08-30T23:34:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:21 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"}]': finished 2024-08-30T23:34:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:21 smithi136 ceph-mon[20207]: osdmap e317: 12 total, 12 up, 12 in 2024-08-30T23:34:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:21 smithi136 ceph-mon[20207]: pgmap v772: 17 pgs: 1 peering, 2 unknown, 14 active+clean; 1.7 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-08-30T23:34:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:21 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'60542','1':'33051','4':'34566','6':'46541'}"}]': finished 2024-08-30T23:34:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:21 smithi173 ceph-mon[20554]: osdmap e317: 12 total, 12 up, 12 in 2024-08-30T23:34:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:21 smithi173 ceph-mon[20554]: pgmap v772: 17 pgs: 1 peering, 2 unknown, 14 active+clean; 1.7 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-08-30T23:34:22.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:22.105 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:22.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:22.236 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:22.236 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:22.237 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:22.248 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:34:22.248 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:34:22.248 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:34:22.248 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:34:22.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:22.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:22.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 31073 31057 92.4235 0 - 0.0100112 2024-08-30T23:34:22.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:22.492 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:22.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:22.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:22.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:22.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:22.850 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:22 smithi106 ceph-mon[16603]: osdmap e318: 12 total, 12 up, 12 in 2024-08-30T23:34:22.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:22.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:22.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:34:22.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:23.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:22 smithi136 ceph-mon[20207]: osdmap e318: 12 total, 12 up, 12 in 2024-08-30T23:34:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:22 smithi173 ceph-mon[20554]: osdmap e318: 12 total, 12 up, 12 in 2024-08-30T23:34:23.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:23.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:23.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:23.253 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:23.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:23.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:23.277 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:23.277 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:23.301 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:23.301 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:23.322 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:23.322 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:23.356 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:23.357 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:23.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:23.380 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:23.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:23.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:23.405 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:23.405 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:23.429 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:23.429 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:23.451 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:23.451 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:23.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 31134 31118 88.3957 1.90625 0.00531183 0.0105916 2024-08-30T23:34:23.477 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:23.478 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:23.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:23.502 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:23.503 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:23.514 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:23.529 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:23.529 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:23.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:23.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:23.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:23.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:23.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:23.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:23.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:23.919 INFO:teuthology.orchestra.run.smithi106.stdout:moved 1 / 51 (1.96078%) 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:avg 4.25 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 1.58771 -> 1.73805 (expected baseline 1.97379) 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 2 -> 2 pgs (0.470588 -> 0.470588 * mean) 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.7 with 7 -> 7 pgs (1.64706 -> 1.64706 * mean) 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 0.05 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0072 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0720 2024-08-30T23:34:23.920 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 0.9238 -> 0.9738 2024-08-30T23:34:23.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 weight 0.8072 -> 0.8572 2024-08-30T23:34:23.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 weight 0.5898 -> 0.6018 2024-08-30T23:34:23.921 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:34:23.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:24.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:23 smithi136 ceph-mon[20207]: osdmap e319: 12 total, 12 up, 12 in 2024-08-30T23:34:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/280529193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:23 smithi136 ceph-mon[20207]: pgmap v775: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 2 unknown, 11 active+clean; 1.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 1914/90606 objects degraded (2.112%) 2024-08-30T23:34:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:23 smithi173 ceph-mon[20554]: osdmap e319: 12 total, 12 up, 12 in 2024-08-30T23:34:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/280529193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:23 smithi173 ceph-mon[20554]: pgmap v775: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 2 unknown, 11 active+clean; 1.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 1914/90606 objects degraded (2.112%) 2024-08-30T23:34:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:23 smithi106 ceph-mon[16603]: osdmap e319: 12 total, 12 up, 12 in 2024-08-30T23:34:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/280529193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:23 smithi106 ceph-mon[16603]: pgmap v775: 17 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 2 unknown, 11 active+clean; 1.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 1914/90606 objects degraded (2.112%) 2024-08-30T23:34:24.107 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:34:23 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:34:23.919+0000 7f2243404640 -1 mgr.server reply reply (3) No such process no change 2024-08-30T23:34:24.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:24.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:24.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:34:24.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:24.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:24.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:24.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:24.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 31213 31197 84.7671 4.9375 0.00419366 0.0111076 2024-08-30T23:34:24.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:24.520 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:24.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:24.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:24.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:24.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:24.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:24.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:24.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:24.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:24.992 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:24 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 1914/90606 objects degraded (2.112%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:24.992 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:24 smithi106 ceph-mon[16603]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:25.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:24 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 1914/90606 objects degraded (2.112%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:24 smithi136 ceph-mon[20207]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:25.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:25.042 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:24 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 1914/90606 objects degraded (2.112%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:25.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:24 smithi173 ceph-mon[20554]: from='client.16107 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:34:25.086 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:25.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:25.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:25.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:25.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:25.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:25.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:25.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 32294 32278 84.05 67.5625 0.00290327 0.0118897 2024-08-30T23:34:25.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:25.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:25.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:25.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:25.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:25.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:25.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:25.877 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:25 smithi106 ceph-mon[16603]: pgmap v776: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 2 unknown, 9 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s; 1914/93807 objects degraded (2.040%); 3546/93807 objects misplaced (3.780%); 3.4 MiB/s, 53 objects/s recovering 2024-08-30T23:34:25.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:25.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:25.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:25 smithi136 ceph-mon[20207]: pgmap v776: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 2 unknown, 9 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s; 1914/93807 objects degraded (2.040%); 3546/93807 objects misplaced (3.780%); 3.4 MiB/s, 53 objects/s recovering 2024-08-30T23:34:26.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:25 smithi173 ceph-mon[20554]: pgmap v776: 17 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 2 unknown, 9 active+clean; 1.9 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s; 1914/93807 objects degraded (2.040%); 3546/93807 objects misplaced (3.780%); 3.4 MiB/s, 53 objects/s recovering 2024-08-30T23:34:26.085 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:26.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:26.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:26.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:26.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:26.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:26.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 33203 33187 82.9604 56.8125 0.0441676 0.0120447 2024-08-30T23:34:26.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:26.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:26.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:26.676 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:26.676 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:26.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:26.716 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:34:26.736 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:26.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:26.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:26.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:26.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:26.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:27.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:34:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3691128165' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:27.055 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3691128165' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3691128165' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:27.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:27.122 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:27.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:27.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:27.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:27.375 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:27.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 34153 34137 82.053 59.375 0.0170173 0.0121765 2024-08-30T23:34:27.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:27.508 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:27.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:27.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:27.720 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:34:27.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:27.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:27.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:27.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:27.909 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:27.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:28.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:27 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3691128165' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:27 smithi136 ceph-mon[20207]: osdmap e320: 12 total, 12 up, 12 in 2024-08-30T23:34:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:27 smithi136 ceph-mon[20207]: pgmap v778: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 9 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s; 3891/95877 objects degraded (4.058%); 8609/95877 objects misplaced (8.979%); 11 MiB/s, 179 objects/s recovering 2024-08-30T23:34:28.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:27 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3691128165' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:27 smithi173 ceph-mon[20554]: osdmap e320: 12 total, 12 up, 12 in 2024-08-30T23:34:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:27 smithi173 ceph-mon[20554]: pgmap v778: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 9 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s; 3891/95877 objects degraded (4.058%); 8609/95877 objects misplaced (8.979%); 11 MiB/s, 179 objects/s recovering 2024-08-30T23:34:28.094 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:27 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:28.094 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3691128165' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:28.094 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:27 smithi106 ceph-mon[16603]: osdmap e320: 12 total, 12 up, 12 in 2024-08-30T23:34:28.094 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:27 smithi106 ceph-mon[16603]: pgmap v778: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 remapped+peering, 9 active+clean; 2.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 576 op/s; 3891/95877 objects degraded (4.058%); 8609/95877 objects misplaced (8.979%); 11 MiB/s, 179 objects/s recovering 2024-08-30T23:34:28.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:28.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:28.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:28.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:28.320 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:28.336 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:28.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:28.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:28.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 35484 35468 82.0948 83.1875 0.00504369 0.0121739 2024-08-30T23:34:28.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:28.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:28.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:28.696 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:28.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:28.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:28.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:28.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:28.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:29.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:29.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:29.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:29.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:29.217 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:29.228 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:29.228 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:29.319 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:29.349 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:29.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:29.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 36690 36674 81.8546 75.375 0.0380727 0.0122097 2024-08-30T23:34:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:29.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:29.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:29.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:29.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:29.739 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:29.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2167840950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2167840950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:29.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2167840950' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:29.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:29.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:29.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:29.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:30.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:30.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:30.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:30.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:34:30.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:30.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:30.271 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:30.271 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:30.309 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:30.309 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:30.342 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:30.342 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:30.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:30.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:34:30.375 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:30.375 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:30.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:30.412 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:30.447 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:30.447 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:30.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 37847 37831 81.5254 72.3125 0.0136251 0.012257 2024-08-30T23:34:30.483 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:30.484 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:30.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:30.525 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:30.525 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:30.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.557 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:30.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:30.563 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:30.563 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:30.599 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:30.599 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:30.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:30 smithi106 ceph-mon[16603]: pgmap v779: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 620 op/s; 1977/99975 objects degraded (1.977%); 11192/99975 objects misplaced (11.195%); 19 MiB/s, 302 objects/s recovering 2024-08-30T23:34:30.636 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:30 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1977/99975 objects degraded (1.977%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:30.637 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:30.637 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:30.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:34:30.655 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:34:30.655 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:34:30.655 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-08-30T23:34:30.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-30T23:34:30.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:30.667 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:30.672 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:30.673 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:30.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:30.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:30 smithi136 ceph-mon[20207]: pgmap v779: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 620 op/s; 1977/99975 objects degraded (1.977%); 11192/99975 objects misplaced (11.195%); 19 MiB/s, 302 objects/s recovering 2024-08-30T23:34:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:30 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1977/99975 objects degraded (1.977%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:30 smithi173 ceph-mon[20554]: pgmap v779: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 620 op/s; 1977/99975 objects degraded (1.977%); 11192/99975 objects misplaced (11.195%); 19 MiB/s, 302 objects/s recovering 2024-08-30T23:34:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:30 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1977/99975 objects degraded (1.977%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:30.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:30.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.857 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:30.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:30.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:30.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:30.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:31.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:31.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:31.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:31.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:31.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.371 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:31.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:31.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 38863 38847 80.9244 63.5 0.00933929 0.0123492 2024-08-30T23:34:31.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:31.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:31.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:31.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:31.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:31.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:31 smithi106 ceph-mon[16603]: osdmap e321: 12 total, 12 up, 12 in 2024-08-30T23:34:31.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:31 smithi106 ceph-mon[16603]: pgmap v781: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 784 op/s; 1977/109443 objects degraded (1.806%); 9832/109443 objects misplaced (8.984%); 29 MiB/s, 464 objects/s recovering 2024-08-30T23:34:31.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:31.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:31.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:32.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:32.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:31 smithi136 ceph-mon[20207]: osdmap e321: 12 total, 12 up, 12 in 2024-08-30T23:34:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:31 smithi136 ceph-mon[20207]: pgmap v781: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 784 op/s; 1977/109443 objects degraded (1.806%); 9832/109443 objects misplaced (8.984%); 29 MiB/s, 464 objects/s recovering 2024-08-30T23:34:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:31 smithi173 ceph-mon[20554]: osdmap e321: 12 total, 12 up, 12 in 2024-08-30T23:34:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:31 smithi173 ceph-mon[20554]: pgmap v781: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 784 op/s; 1977/109443 objects degraded (1.806%); 9832/109443 objects misplaced (8.984%); 29 MiB/s, 464 objects/s recovering 2024-08-30T23:34:32.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:32.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:32.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:32.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:32.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 40176 40160 80.9609 82.0625 0.0213452 0.0123438 2024-08-30T23:34:32.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:32.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:32.640 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:34:32.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:32.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:32.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:32.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:32.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:32.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:32.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:33.011 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3754566857' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:34:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3754566857' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:34:33.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3754566857' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:34:33.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:33.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:33.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:33.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:33.337 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:33.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:33.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:33.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:33.462 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 41174 41158 80.3799 62.375 0.00379179 0.0124299 2024-08-30T23:34:33.481 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:33.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:33.615 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:33.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:33.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:33.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:33.870 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:33 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:33.870 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3754566857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:34:33.870 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:33 smithi106 ceph-mon[16603]: osdmap e322: 12 total, 12 up, 12 in 2024-08-30T23:34:33.870 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:33 smithi106 ceph-mon[16603]: pgmap v783: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s; 1977/115056 objects degraded (1.718%); 8443/115056 objects misplaced (7.338%); 40 MiB/s, 638 objects/s recovering 2024-08-30T23:34:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:33.874 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:33.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:33.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:34.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:33 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3754566857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:34:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:33 smithi136 ceph-mon[20207]: osdmap e322: 12 total, 12 up, 12 in 2024-08-30T23:34:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:33 smithi136 ceph-mon[20207]: pgmap v783: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s; 1977/115056 objects degraded (1.718%); 8443/115056 objects misplaced (7.338%); 40 MiB/s, 638 objects/s recovering 2024-08-30T23:34:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:33 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3754566857' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:34:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:33 smithi173 ceph-mon[20554]: osdmap e322: 12 total, 12 up, 12 in 2024-08-30T23:34:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:33 smithi173 ceph-mon[20554]: pgmap v783: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 973 op/s; 1977/115056 objects degraded (1.718%); 8443/115056 objects misplaced (7.338%); 40 MiB/s, 638 objects/s recovering 2024-08-30T23:34:34.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:34.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.138 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:34.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:34.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:34.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:34.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:34.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 42505 42489 80.4648 83.1875 0.00292987 0.0124209 2024-08-30T23:34:34.493 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:34:34.493 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:34:34.493 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:34:34.493 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:34:34.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:34.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:34.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:34.655 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-30T23:34:34.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:34:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:34.673 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:34.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:34.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:34.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:34 smithi106 ceph-mon[16603]: osdmap e323: 12 total, 12 up, 12 in 2024-08-30T23:34:34.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4055458643' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:34:34.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:34:34.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:34.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:34.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:34.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:34.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:35.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:34 smithi136 ceph-mon[20207]: osdmap e323: 12 total, 12 up, 12 in 2024-08-30T23:34:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4055458643' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:34:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:34:35.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:34 smithi173 ceph-mon[20554]: osdmap e323: 12 total, 12 up, 12 in 2024-08-30T23:34:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4055458643' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:34:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:34:35.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:35.191 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:35.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:34:35.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:35.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:35.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:34:35.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:35.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:35.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 43669 43653 80.2377 72.75 0.00911413 0.0124553 2024-08-30T23:34:35.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:35.586 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.587 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:35.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:35.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:35.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:35.841 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:35.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:35.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:35.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:35 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:34:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:35 smithi136 ceph-mon[20207]: osdmap e324: 12 total, 12 up, 12 in 2024-08-30T23:34:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:35 smithi136 ceph-mon[20207]: pgmap v786: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 1977/122658 objects degraded (1.612%); 7399/122658 objects misplaced (6.032%); 43 MiB/s, 684 objects/s recovering 2024-08-30T23:34:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1977/115056 objects degraded (1.718%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:35 smithi136 ceph-mon[20207]: osdmap e325: 12 total, 12 up, 12 in 2024-08-30T23:34:36.044 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:35 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:34:36.044 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:35 smithi106 ceph-mon[16603]: osdmap e324: 12 total, 12 up, 12 in 2024-08-30T23:34:36.044 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:35 smithi106 ceph-mon[16603]: pgmap v786: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 1977/122658 objects degraded (1.612%); 7399/122658 objects misplaced (6.032%); 43 MiB/s, 684 objects/s recovering 2024-08-30T23:34:36.044 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1977/115056 objects degraded (1.718%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:36.044 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:35 smithi106 ceph-mon[16603]: osdmap e325: 12 total, 12 up, 12 in 2024-08-30T23:34:36.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:35 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:34:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:35 smithi173 ceph-mon[20554]: osdmap e324: 12 total, 12 up, 12 in 2024-08-30T23:34:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:35 smithi173 ceph-mon[20554]: pgmap v786: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 1977/122658 objects degraded (1.612%); 7399/122658 objects misplaced (6.032%); 43 MiB/s, 684 objects/s recovering 2024-08-30T23:34:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1977/115056 objects degraded (1.718%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:35 smithi173 ceph-mon[20554]: osdmap e325: 12 total, 12 up, 12 in 2024-08-30T23:34:36.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:36.101 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:36.160 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:36.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:36.221 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:36.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:36.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:36.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:36.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 44786 44770 79.9398 69.8125 0.00856787 0.0125035 2024-08-30T23:34:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:36.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:36.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:36.617 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:36.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:36.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:36.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:36.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:36.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:36.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:36.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:37.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:37.130 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:37.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:37.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:37.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:37.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:37.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 15 45991 45976 79.8128 75.375 0.00359144 0.0125224 2024-08-30T23:34:37.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:37.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.532 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:37.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:37.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:37.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:37.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:37.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:37.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:37.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:38.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.044 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:38.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:38.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:38.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:38.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:38.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:38.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:38.429 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:38.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 47005 46989 79.3667 63.3125 0.00573815 0.0125899 2024-08-30T23:34:38.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:38.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:38.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:38.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:38.775 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:38 smithi173 ceph-mon[20554]: pgmap v788: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 1977/128226 objects degraded (1.542%); 5554/128226 objects misplaced (4.331%); 43 MiB/s, 680 objects/s recovering 2024-08-30T23:34:38.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:38.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:38 smithi136 ceph-mon[20207]: pgmap v788: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 1977/128226 objects degraded (1.542%); 5554/128226 objects misplaced (4.331%); 43 MiB/s, 680 objects/s recovering 2024-08-30T23:34:38.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:38.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:38 smithi106 ceph-mon[16603]: pgmap v788: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 1977/128226 objects degraded (1.542%); 5554/128226 objects misplaced (4.331%); 43 MiB/s, 680 objects/s recovering 2024-08-30T23:34:38.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:38.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:38.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:39.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:39.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:39.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:39.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:39.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:39.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:39.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:39.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:39.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 48160 48144 79.1776 72.1875 0.0064444 0.0126226 2024-08-30T23:34:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.470 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:39.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:39.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:39.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:39.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:39.729 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:39.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:39.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:39.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:39.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:39.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:40.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:40.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:40.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:40.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:40.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:40.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:40.374 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:40.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 49323 49307 79.0111 72.6875 0.00688275 0.0126499 2024-08-30T23:34:40.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:40.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:40.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:40.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:40.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:40.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:40 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:40 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:40 smithi136 ceph-mon[20207]: pgmap v789: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 1977/134694 objects degraded (1.468%); 3999/134694 objects misplaced (2.969%); 46 MiB/s, 740 objects/s recovering 2024-08-30T23:34:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:40 smithi136 ceph-mon[20207]: osdmap e326: 12 total, 12 up, 12 in 2024-08-30T23:34:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1758606887' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:40 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:40 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:40 smithi173 ceph-mon[20554]: pgmap v789: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 1977/134694 objects degraded (1.468%); 3999/134694 objects misplaced (2.969%); 46 MiB/s, 740 objects/s recovering 2024-08-30T23:34:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:40 smithi173 ceph-mon[20554]: osdmap e326: 12 total, 12 up, 12 in 2024-08-30T23:34:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1758606887' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:40.841 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:34:40.841 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:40 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:40.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:40 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:40.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:40 smithi106 ceph-mon[16603]: pgmap v789: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 9 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 1977/134694 objects degraded (1.468%); 3999/134694 objects misplaced (2.969%); 46 MiB/s, 740 objects/s recovering 2024-08-30T23:34:40.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:40 smithi106 ceph-mon[16603]: osdmap e326: 12 total, 12 up, 12 in 2024-08-30T23:34:40.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1758606887' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:40.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:40.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:40.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:40.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:41.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:41.090 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:34:41.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:34:41.091 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-08-30T23:34:41.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:34:41.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:41.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:41.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:41.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:41.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:41.433 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:41.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:34:41.464390+0000 min lat: 0.0010779 max lat: 3.89318 avg lat: 0.0127083 2024-08-30T23:34:41.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:34:41.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 50362 50346 78.6591 64.9375 0.0141578 0.0127083 2024-08-30T23:34:41.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:41.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:41.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:41.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:41 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1758606887' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:41 smithi136 ceph-mon[20207]: osdmap e327: 12 total, 12 up, 12 in 2024-08-30T23:34:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1977/134694 objects degraded (1.468%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:41 smithi136 ceph-mon[20207]: osdmap e328: 12 total, 12 up, 12 in 2024-08-30T23:34:41.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:41.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:41 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1758606887' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:41 smithi173 ceph-mon[20554]: osdmap e327: 12 total, 12 up, 12 in 2024-08-30T23:34:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1977/134694 objects degraded (1.468%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:41 smithi173 ceph-mon[20554]: osdmap e328: 12 total, 12 up, 12 in 2024-08-30T23:34:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:41 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1758606887' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:41 smithi106 ceph-mon[16603]: osdmap e327: 12 total, 12 up, 12 in 2024-08-30T23:34:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1977/134694 objects degraded (1.468%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:34:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:41 smithi106 ceph-mon[16603]: osdmap e328: 12 total, 12 up, 12 in 2024-08-30T23:34:41.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:41.938 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:41.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:42.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:42.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:42.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:42.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:42.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:42.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:42.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:42.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:42.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 50709 50693 77.2695 21.6875 0.00248118 0.0127279 2024-08-30T23:34:42.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:42.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:42.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:42.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:42.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:42 smithi136 ceph-mon[20207]: pgmap v793: 17 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 9 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3927/144063 objects misplaced (2.726%); 58 MiB/s, 934 objects/s recovering 2024-08-30T23:34:42.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:42 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1977/134694 objects degraded (1.468%), 1 pg degraded) 2024-08-30T23:34:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:42 smithi136 ceph-mon[20207]: osdmap e329: 12 total, 12 up, 12 in 2024-08-30T23:34:42.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:42.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:42 smithi173 ceph-mon[20554]: pgmap v793: 17 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 9 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3927/144063 objects misplaced (2.726%); 58 MiB/s, 934 objects/s recovering 2024-08-30T23:34:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:42 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1977/134694 objects degraded (1.468%), 1 pg degraded) 2024-08-30T23:34:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:42 smithi173 ceph-mon[20554]: osdmap e329: 12 total, 12 up, 12 in 2024-08-30T23:34:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:42 smithi106 ceph-mon[16603]: pgmap v793: 17 pgs: 2 active+undersized+remapped, 6 active+recovering+undersized+remapped, 9 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 3927/144063 objects misplaced (2.726%); 58 MiB/s, 934 objects/s recovering 2024-08-30T23:34:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:42 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1977/134694 objects degraded (1.468%), 1 pg degraded) 2024-08-30T23:34:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:42 smithi106 ceph-mon[16603]: osdmap e329: 12 total, 12 up, 12 in 2024-08-30T23:34:42.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:42.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:42.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:43.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:43.078 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:43.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:43.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:43.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:43.328 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:43.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:43.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:43.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 51891 51875 77.1885 73.875 0.013312 0.012948 2024-08-30T23:34:43.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:43.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:43.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:43.717 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:43.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:43.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.850 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:43.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:43.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:43.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:44.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:43 smithi136 ceph-mon[20207]: osdmap e330: 12 total, 12 up, 12 in 2024-08-30T23:34:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:43 smithi136 ceph-mon[20207]: pgmap v796: 17 pgs: 1 peering, 2 active+undersized+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 2636/148641 objects misplaced (1.773%); 53 MiB/s, 847 objects/s recovering 2024-08-30T23:34:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:43 smithi173 ceph-mon[20554]: osdmap e330: 12 total, 12 up, 12 in 2024-08-30T23:34:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:43 smithi173 ceph-mon[20554]: pgmap v796: 17 pgs: 1 peering, 2 active+undersized+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 2636/148641 objects misplaced (1.773%); 53 MiB/s, 847 objects/s recovering 2024-08-30T23:34:44.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:43 smithi106 ceph-mon[16603]: osdmap e330: 12 total, 12 up, 12 in 2024-08-30T23:34:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:43 smithi106 ceph-mon[16603]: pgmap v796: 17 pgs: 1 peering, 2 active+undersized+remapped, 4 active+recovering+undersized+remapped, 10 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 2636/148641 objects misplaced (1.773%); 53 MiB/s, 847 objects/s recovering 2024-08-30T23:34:44.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:44.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:44.235 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:44.338 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:44.366 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:44.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 53191 53175 77.2828 81.25 0.0115882 0.0129345 2024-08-30T23:34:44.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:44.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:44.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:44.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:44.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:44.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:44.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2650786758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:44.846 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:44.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:44.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:44.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:44.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2650786758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2650786758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:45.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:45.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:45.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:45.217 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:45.218 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:45.219 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:45.219 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:45.219 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:45.219 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:45.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:45.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:45.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:45.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:45.220 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:45.220 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:45.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:45.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:45.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:45.361 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:45.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:45.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 53606 53590 76.1158 25.9375 0.0420696 0.0129965 2024-08-30T23:34:45.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:45.582 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:45.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:45.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:45.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:45.737 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:45.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:45.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:45.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:45.976 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: osdmap e331: 12 total, 12 up, 12 in 2024-08-30T23:34:45.976 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=3879910961,config=2353167246,health=179246142,kv=1940727612,logm=2328201200}) 2024-08-30T23:34:45.976 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3255597013}) 2024-08-30T23:34:45.976 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922396947}) 2024-08-30T23:34:45.976 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132872824}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2057902525}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=276265374}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585006014}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684336725}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777394118}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089242139}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612666901}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=31712457}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882604424}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603964739}) 2024-08-30T23:34:45.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179441750}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076717494}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702105279}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397296879}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887269384}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218545394}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224368310}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=766649925}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4190650613}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661146877}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535117261}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296933678}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2182351938}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406325103}) 2024-08-30T23:34:45.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079794566}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047143349}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=931374795}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=825146153}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426726696}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559139493}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=384061972,mdsmap=425822562,mgr=799420628}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mgr=11,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=71} crc {mgr=192676359,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=428186187,mon_config_key=619805772}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2339294601}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3375045919}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=20,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=62} crc {mon_config_key=3194612244,monmap=1193225519,osd_metadata=3634387181,osd_pg_creating=1771991257,osdmap=3305112035}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3431451652}) 2024-08-30T23:34:45.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: pgmap v798: 17 pgs: 1 peering, 2 active+undersized+remapped, 3 active+recovering+undersized+remapped, 11 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 2156/154812 objects misplaced (1.393%); 41 MiB/s, 649 objects/s recovering 2024-08-30T23:34:45.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:45 smithi106 ceph-mon[16603]: osdmap e332: 12 total, 12 up, 12 in 2024-08-30T23:34:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:45.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:45.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: osdmap e331: 12 total, 12 up, 12 in 2024-08-30T23:34:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=3879910961,config=2353167246,health=179246142,kv=1940727612,logm=2328201200}) 2024-08-30T23:34:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3255597013}) 2024-08-30T23:34:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922396947}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132872824}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2057902525}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=276265374}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585006014}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684336725}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777394118}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089242139}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612666901}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=31712457}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882604424}) 2024-08-30T23:34:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603964739}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179441750}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076717494}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702105279}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397296879}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887269384}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218545394}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224368310}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=766649925}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4190650613}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661146877}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535117261}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296933678}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2182351938}) 2024-08-30T23:34:46.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406325103}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079794566}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047143349}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=931374795}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=825146153}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426726696}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559139493}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=384061972,mdsmap=425822562,mgr=799420628}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mgr=11,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=71} crc {mgr=192676359,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=428186187,mon_config_key=619805772}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2339294601}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3375045919}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=20,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=62} crc {mon_config_key=3194612244,monmap=1193225519,osd_metadata=3634387181,osd_pg_creating=1771991257,osdmap=3305112035}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3431451652}) 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: pgmap v798: 17 pgs: 1 peering, 2 active+undersized+remapped, 3 active+recovering+undersized+remapped, 11 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 2156/154812 objects misplaced (1.393%); 41 MiB/s, 649 objects/s recovering 2024-08-30T23:34:46.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:45 smithi136 ceph-mon[20207]: osdmap e332: 12 total, 12 up, 12 in 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: osdmap e331: 12 total, 12 up, 12 in 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=3879910961,config=2353167246,health=179246142,kv=1940727612,logm=2328201200}) 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3255597013}) 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=922396947}) 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132872824}) 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2057902525}) 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=276265374}) 2024-08-30T23:34:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585006014}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684336725}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777394118}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089242139}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612666901}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=31712457}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882604424}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603964739}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179441750}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076717494}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702105279}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397296879}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887269384}) 2024-08-30T23:34:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4218545394}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4224368310}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=766649925}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4190650613}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3661146877}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1535117261}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296933678}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2182351938}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1406325103}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2079794566}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047143349}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=931374795}) 2024-08-30T23:34:46.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=825146153}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426726696}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=559139493}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=384061972,mdsmap=425822562,mgr=799420628}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mgr=11,mgr_command_descs=1,mgr_metadata=2,mgrstat=15,mon_config_key=71} crc {mgr=192676359,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=428186187,mon_config_key=619805772}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2339294601}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3375045919}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=20,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=62} crc {mon_config_key=3194612244,monmap=1193225519,osd_metadata=3634387181,osd_pg_creating=1771991257,osdmap=3305112035}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3431451652}) 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: pgmap v798: 17 pgs: 1 peering, 2 active+undersized+remapped, 3 active+recovering+undersized+remapped, 11 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 2156/154812 objects misplaced (1.393%); 41 MiB/s, 649 objects/s recovering 2024-08-30T23:34:46.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:45 smithi173 ceph-mon[20554]: osdmap e332: 12 total, 12 up, 12 in 2024-08-30T23:34:46.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:46.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:46.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:46.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:46.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:46.358 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:46.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:46.463 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 53815 53799 74.7146 13.0625 0.00230254 0.0132228 2024-08-30T23:34:46.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:46.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:46.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:46.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:46.726 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:46.726 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:46.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:46.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:46.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:46.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:46.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:46.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:47.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:47.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:47.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:47.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:47.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:47.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:47.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 54821 54805 74.4571 62.875 0.00569284 0.0134244 2024-08-30T23:34:47.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:47.499 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:47.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:47.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:47.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:34:47.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:47.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:47.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:47 smithi106 ceph-mon[16603]: osdmap e333: 12 total, 12 up, 12 in 2024-08-30T23:34:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:47 smithi106 ceph-mon[16603]: pgmap v801: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 13 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s; 1540/158262 objects misplaced (0.973%); 20 MiB/s, 314 objects/s recovering 2024-08-30T23:34:47.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:47.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:47.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:48.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:47 smithi136 ceph-mon[20207]: osdmap e333: 12 total, 12 up, 12 in 2024-08-30T23:34:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:47 smithi136 ceph-mon[20207]: pgmap v801: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 13 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s; 1540/158262 objects misplaced (0.973%); 20 MiB/s, 314 objects/s recovering 2024-08-30T23:34:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:47 smithi173 ceph-mon[20554]: osdmap e333: 12 total, 12 up, 12 in 2024-08-30T23:34:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:47 smithi173 ceph-mon[20554]: pgmap v801: 17 pgs: 1 peering, 3 active+recovering+undersized+remapped, 13 active+clean; 3.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s; 1540/158262 objects misplaced (0.973%); 20 MiB/s, 314 objects/s recovering 2024-08-30T23:34:48.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:48.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:48.202 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:48.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:48.240 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:48.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:48.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:48.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:48.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 55806 55790 74.1827 61.5625 0.00291428 0.0134693 2024-08-30T23:34:48.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:48.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.506 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:48.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:48.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:48.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:48.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:48.788 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:34:48.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:48.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:48.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:48.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:49.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:49.073 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:49.073 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:49.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:49.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:49.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:49.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:49.381 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:49.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:49.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 57018 57002 74.2152 75.75 0.0163426 0.0134687 2024-08-30T23:34:49.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:49.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:49.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:49.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:49.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:49.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:49.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:34:49.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:49.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:49.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:49.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:49 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:34:49.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:49 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:34:49.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:49 smithi106 ceph-mon[16603]: osdmap e334: 12 total, 12 up, 12 in 2024-08-30T23:34:49.926 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:49 smithi106 ceph-mon[16603]: pgmap v803: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 1330/163083 objects misplaced (0.816%); 11 MiB/s, 180 objects/s recovering 2024-08-30T23:34:50.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:49 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:34:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:49 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:34:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:49 smithi136 ceph-mon[20207]: osdmap e334: 12 total, 12 up, 12 in 2024-08-30T23:34:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:49 smithi136 ceph-mon[20207]: pgmap v803: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 1330/163083 objects misplaced (0.816%); 11 MiB/s, 180 objects/s recovering 2024-08-30T23:34:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:50.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:49 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:34:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:49 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:34:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:49 smithi173 ceph-mon[20554]: osdmap e334: 12 total, 12 up, 12 in 2024-08-30T23:34:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:49 smithi173 ceph-mon[20554]: pgmap v803: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 873 op/s; 1330/163083 objects misplaced (0.816%); 11 MiB/s, 180 objects/s recovering 2024-08-30T23:34:50.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:34:50.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:50.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:50.187 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:50.207 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:50.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:50.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:50.395 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:34:50.396 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:34:50.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:50.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:50.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 58162 58146 74.1597 71.5 0.0239862 0.0134778 2024-08-30T23:34:50.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:50.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:50.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:50.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:50.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:50.807 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:50.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:50.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:50.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:50.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:50 smithi106 ceph-mon[16603]: from='client.25759 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:34:51.032 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:50 smithi136 ceph-mon[20207]: from='client.25759 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:34:51.062 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:51.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:50 smithi173 ceph-mon[20554]: from='client.25759 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:34:51.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:51.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:51.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:51.349 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-08-30T23:34:49.461459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3563061792,"num_objects":54361,"num_object_clones":0,"num_object_copies":163083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1330,"num_objects_unfound":0,"num_objects_dirty":54361,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":54482,"num_write_kb":3481126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14202,"num_bytes_recovered":931935840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54451,"ondisk_log_size":54451,"up":51,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13183368,"kb_used_data":10601692,"kb_used_omap":18,"kb_used_meta":2581357,"kb_avail":1111758456,"statfs":{"total":1151940427776,"available":1138440658944,"internally_reserved":0,"allocated":10856132608,"data_stored":10851751186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":2643309881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,2,2,1,1,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":275185664,"num_objects":4199,"num_object_clones":0,"num_object_copies":12597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-869,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":268736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":56950784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.806536"},"pg_stats":[{"pgid":"4.f","version":"333'3430","reported_seq":3494,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:47.701561+0000","last_change":"2024-08-30T23:33:56.656974+0000","last_active":"2024-08-30T23:34:47.701561+0000","last_peered":"2024-08-30T23:34:47.701561+0000","last_clean":"2024-08-30T23:34:47.701561+0000","last_became_active":"2024-08-30T23:33:56.656639+0000","last_became_peered":"2024-08-30T23:33:56.656639+0000","last_unstale":"2024-08-30T23:34:47.701561+0000","last_undegraded":"2024-08-30T23:34:47.701561+0000","last_fullsized":"2024-08-30T23:34:47.701561+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:25.404526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":219520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"331'3328","reported_seq":7232,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-08-30T23:34:44.686596+0000","last_change":"2024-08-30T23:34:40.722642+0000","last_active":"2024-08-30T23:34:44.686596+0000","last_peered":"2024-08-30T23:34:44.686596+0000","last_clean":"2024-08-30T23:34:44.686596+0000","last_became_active":"2024-08-30T23:34:40.719708+0000","last_became_peered":"2024-08-30T23:34:40.719708+0000","last_unstale":"2024-08-30T23:34:44.686596+0000","last_undegraded":"2024-08-30T23:34:44.686596+0000","last_fullsized":"2024-08-30T23:34:44.686596+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3328,"log_dups_size":0,"ondisk_log_size":3328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:10.572477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3328,"num_write_kb":212992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":125632512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"331'3305","reported_seq":3366,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-08-30T23:34:45.149366+0000","last_change":"2024-08-30T23:33:56.656195+0000","last_active":"2024-08-30T23:34:45.149366+0000","last_peered":"2024-08-30T23:34:45.149366+0000","last_clean":"2024-08-30T23:34:45.149366+0000","last_became_active":"2024-08-30T23:33:56.655881+0000","last_became_peered":"2024-08-30T23:33:56.655881+0000","last_unstale":"2024-08-30T23:34:45.149366+0000","last_undegraded":"2024-08-30T23:34:45.149366+0000","last_fullsized":"2024-08-30T23:34:45.149366+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3305,"log_dups_size":0,"ondisk_log_size":3305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:02.913816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216596480,"num_objects":3305,"num_object_clones":0,"num_object_copies":9915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3305,"num_write_kb":211520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11],"acting":[9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"331'3183","reported_seq":3244,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-08-30T23:34:44.670126+0000","last_change":"2024-08-30T23:33:56.657408+0000","last_active":"2024-08-30T23:34:44.670126+0000","last_peered":"2024-08-30T23:34:44.670126+0000","last_clean":"2024-08-30T23:34:44.670126+0000","last_became_active":"2024-08-30T23:33:56.657074+0000","last_became_peered":"2024-08-30T23:33:56.657074+0000","last_unstale":"2024-08-30T23:34:44.670126+0000","last_undegraded":"2024-08-30T23:34:44.670126+0000","last_fullsized":"2024-08-30T23:34:44.670126+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3183,"log_dups_size":0,"ondisk_log_size":3183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:23.951270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208601088,"num_objects":3183,"num_object_clones":0,"num_object_copies":9549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3183,"num_write_kb":203712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"333'3544","reported_seq":3607,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:48.581729+0000","last_change":"2024-08-30T23:33:56.655706+0000","last_active":"2024-08-30T23:34:48.581729+0000","last_peered":"2024-08-30T23:34:48.581729+0000","last_clean":"2024-08-30T23:34:48.581729+0000","last_became_active":"2024-08-30T23:33:56.655416+0000","last_became_peered":"2024-08-30T23:33:56.655416+0000","last_unstale":"2024-08-30T23:34:48.581729+0000","last_undegraded":"2024-08-30T23:34:48.581729+0000","last_fullsized":"2024-08-30T23:34:48.581729+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3544,"log_dups_size":0,"ondisk_log_size":3544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:08.633565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232259584,"num_objects":3544,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":226816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"333'3456","reported_seq":7359,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:47.683333+0000","last_change":"2024-08-30T23:34:46.670467+0000","last_active":"2024-08-30T23:34:47.683333+0000","last_peered":"2024-08-30T23:34:47.683333+0000","last_clean":"2024-08-30T23:34:47.683333+0000","last_became_active":"2024-08-30T23:34:46.666422+0000","last_became_peered":"2024-08-30T23:34:46.666422+0000","last_unstale":"2024-08-30T23:34:47.683333+0000","last_undegraded":"2024-08-30T23:34:47.683333+0000","last_fullsized":"2024-08-30T23:34:47.683333+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:07.196858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3456,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":125435904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"333'3418","reported_seq":7373,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:46.955648+0000","last_change":"2024-08-30T23:34:41.063395+0000","last_active":"2024-08-30T23:34:46.955648+0000","last_peered":"2024-08-30T23:34:46.955648+0000","last_clean":"2024-08-30T23:34:46.955648+0000","last_became_active":"2024-08-30T23:34:40.834281+0000","last_became_peered":"2024-08-30T23:34:40.834281+0000","last_unstale":"2024-08-30T23:34:46.955648+0000","last_undegraded":"2024-08-30T23:34:46.955648+0000","last_fullsized":"2024-08-30T23:34:46.955648+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3418,"log_dups_size":0,"ondisk_log_size":3418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:11.912261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224002048,"num_objects":3418,"num_object_clones":0,"num_object_copies":10254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3418,"num_write_kb":218752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"332'3448","reported_seq":7495,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-08-30T23:34:45.771483+0000","last_change":"2024-08-30T23:34:42.645739+0000","last_active":"2024-08-30T23:34:45.771483+0000","last_peered":"2024-08-30T23:34:45.771483+0000","last_clean":"2024-08-30T23:34:45.771483+0000","last_became_active":"2024-08-30T23:34:42.641060+0000","last_became_peered":"2024-08-30T23:34:42.641060+0000","last_unstale":"2024-08-30T23:34:45.771483+0000","last_undegraded":"2024-08-30T23:34:45.771483+0000","last_fullsized":"2024-08-30T23:34:45.771483+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3448,"log_dups_size":0,"ondisk_log_size":3448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:07.815387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225968128,"num_objects":3448,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":220672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1986,"num_bytes_recovered":130154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"332'3267","reported_seq":7131,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-08-30T23:34:45.774018+0000","last_change":"2024-08-30T23:34:42.646300+0000","last_active":"2024-08-30T23:34:45.774018+0000","last_peered":"2024-08-30T23:34:45.774018+0000","last_clean":"2024-08-30T23:34:45.774018+0000","last_became_active":"2024-08-30T23:34:42.642239+0000","last_became_peered":"2024-08-30T23:34:42.642239+0000","last_unstale":"2024-08-30T23:34:45.774018+0000","last_undegraded":"2024-08-30T23:34:45.774018+0000","last_fullsized":"2024-08-30T23:34:45.774018+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:00.540319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214106112,"num_objects":3267,"num_object_clones":0,"num_object_copies":9801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3267,"num_write_kb":209088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1895,"num_bytes_recovered":124190720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"331'3394","reported_seq":3455,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-08-30T23:34:45.097789+0000","last_change":"2024-08-30T23:33:57.414494+0000","last_active":"2024-08-30T23:34:45.097789+0000","last_peered":"2024-08-30T23:34:45.097789+0000","last_clean":"2024-08-30T23:34:45.097789+0000","last_became_active":"2024-08-30T23:33:57.414169+0000","last_became_peered":"2024-08-30T23:33:57.414169+0000","last_unstale":"2024-08-30T23:34:45.097789+0000","last_undegraded":"2024-08-30T23:34:45.097789+0000","last_fullsized":"2024-08-30T23:34:45.097789+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:15.542146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222429184,"num_objects":3394,"num_object_clones":0,"num_object_copies":10182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3394,"num_write_kb":217216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"334'3501","reported_seq":7486,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-08-30T23:34:48.800158+0000","last_change":"2024-08-30T23:34:45.641113+0000","last_active":"2024-08-30T23:34:48.800158+0000","last_peered":"2024-08-30T23:34:48.800158+0000","last_clean":"2024-08-30T23:34:48.800158+0000","last_became_active":"2024-08-30T23:34:45.636628+0000","last_became_peered":"2024-08-30T23:34:45.636628+0000","last_unstale":"2024-08-30T23:34:48.800158+0000","last_undegraded":"2024-08-30T23:34:48.800158+0000","last_fullsized":"2024-08-30T23:34:48.800158+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3501,"log_dups_size":0,"ondisk_log_size":3501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:05.810294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229441536,"num_objects":3501,"num_object_clones":0,"num_object_copies":10503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3501,"num_write_kb":224064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1953,"num_bytes_recovered":127991808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10],"acting":[3,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"333'3398","reported_seq":3463,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:47.522286+0000","last_change":"2024-08-30T23:33:56.657715+0000","last_active":"2024-08-30T23:34:47.522286+0000","last_peered":"2024-08-30T23:34:47.522286+0000","last_clean":"2024-08-30T23:34:47.522286+0000","last_became_active":"2024-08-30T23:33:56.657367+0000","last_became_peered":"2024-08-30T23:33:56.657367+0000","last_unstale":"2024-08-30T23:34:47.522286+0000","last_undegraded":"2024-08-30T23:34:47.522286+0000","last_fullsized":"2024-08-30T23:34:47.522286+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3398,"log_dups_size":0,"ondisk_log_size":3398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:15.495167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691328,"num_objects":3398,"num_object_clones":0,"num_object_copies":10194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3398,"num_write_kb":217472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"331'3388","reported_seq":3449,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-08-30T23:34:45.168075+0000","last_change":"2024-08-30T23:33:57.275424+0000","last_active":"2024-08-30T23:34:45.168075+0000","last_peered":"2024-08-30T23:34:45.168075+0000","last_clean":"2024-08-30T23:34:45.168075+0000","last_became_active":"2024-08-30T23:33:57.274974+0000","last_became_peered":"2024-08-30T23:33:57.274974+0000","last_unstale":"2024-08-30T23:34:45.168075+0000","last_undegraded":"2024-08-30T23:34:45.168075+0000","last_fullsized":"2024-08-30T23:34:45.168075+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:22.657203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222035968,"num_objects":3388,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":216832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,8],"acting":[9,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"332'3405","reported_seq":3468,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-08-30T23:34:46.264681+0000","last_change":"2024-08-30T23:33:57.275412+0000","last_active":"2024-08-30T23:34:46.264681+0000","last_peered":"2024-08-30T23:34:46.264681+0000","last_clean":"2024-08-30T23:34:46.264681+0000","last_became_active":"2024-08-30T23:33:57.274888+0000","last_became_peered":"2024-08-30T23:33:57.274888+0000","last_unstale":"2024-08-30T23:34:46.264681+0000","last_undegraded":"2024-08-30T23:34:46.264681+0000","last_fullsized":"2024-08-30T23:34:46.264681+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3405,"log_dups_size":0,"ondisk_log_size":3405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:19.600781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223150080,"num_objects":3405,"num_object_clones":0,"num_object_copies":10215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3405,"num_write_kb":217920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"333'3464","reported_seq":7429,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:47.494860+0000","last_change":"2024-08-30T23:34:42.645841+0000","last_active":"2024-08-30T23:34:47.494860+0000","last_peered":"2024-08-30T23:34:47.494860+0000","last_clean":"2024-08-30T23:34:47.494860+0000","last_became_active":"2024-08-30T23:34:42.642044+0000","last_became_peered":"2024-08-30T23:34:42.642044+0000","last_unstale":"2024-08-30T23:34:47.494860+0000","last_undegraded":"2024-08-30T23:34:47.494860+0000","last_fullsized":"2024-08-30T23:34:47.494860+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3464,"log_dups_size":0,"ondisk_log_size":3464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:01.093499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227016704,"num_objects":3464,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3464,"num_write_kb":221696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1944,"num_bytes_recovered":127401984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"332'3430","reported_seq":4792,"reported_epoch":332,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:34:46.291925+0000","last_change":"2024-08-30T23:34:40.716538+0000","last_active":"2024-08-30T23:34:46.291925+0000","last_peered":"2024-08-30T23:34:46.291925+0000","last_clean":"2024-08-30T23:34:20.679105+0000","last_became_active":"2024-08-30T23:34:24.497972+0000","last_became_peered":"2024-08-30T23:34:24.497972+0000","last_unstale":"2024-08-30T23:34:46.291925+0000","last_undegraded":"2024-08-30T23:34:46.291925+0000","last_fullsized":"2024-08-30T23:34:22.685120+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1330,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":219520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":42467328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0],"acting":[10,7],"avail_no_missing":["10","6","7"],"object_location_counts":[{"shards":"6,7,10","objects":1330},{"shards":"7,10","objects":2100}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":626,"reported_epoch":333,"state":"active+clean","last_fresh":"2024-08-30T23:34:46.665758+0000","last_change":"2024-08-30T23:34:35.631256+0000","last_active":"2024-08-30T23:34:46.665758+0000","last_peered":"2024-08-30T23:34:46.665758+0000","last_clean":"2024-08-30T23:34:46.665758+0000","last_became_active":"2024-08-30T23:34:35.630884+0000","last_became_peered":"2024-08-30T23:34:35.630884+0000","last_unstale":"2024-08-30T23:34:46.665758+0000","last_undegraded":"2024-08-30T23:34:46.665758+0000","last_fullsized":"2024-08-30T23:34:46.665758+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:33:15.070383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:33:15.070383+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3562471424,"num_objects":54359,"num_object_clones":0,"num_object_copies":163077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1330,"num_objects_unfound":0,"num_objects_dirty":54359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54359,"num_write_kb":3478976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14196,"num_bytes_recovered":930349056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10825302016,"data_stored":10825302016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54359,"ondisk_log_size":54359,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187579,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954328,"kb_used_data":652416,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92790824,"statfs":{"total":95995035648,"available":95017803776,"internally_reserved":0,"allocated":668073984,"data_stored":667707323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":309131716},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449218,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1176200,"kb_used_data":1072824,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92568952,"statfs":{"total":95995035648,"available":94790606848,"internally_reserved":0,"allocated":1098571776,"data_stored":1098208687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":105839040},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1305904,"kb_used_data":1072032,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92439248,"statfs":{"total":95995035648,"available":94657789952,"internally_reserved":0,"allocated":1097760768,"data_stored":1097406622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":239466944},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":555844,"kb_used_data":434412,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93189308,"statfs":{"total":95995035648,"available":95425851392,"internally_reserved":0,"allocated":444837888,"data_stored":444479281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234126,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521120,"kb_used_data":1305292,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92224032,"statfs":{"total":95995035648,"available":94437408768,"internally_reserved":0,"allocated":1336619008,"data_stored":1336247856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":220985802},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528468,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951004,"kb_used_data":779596,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92794148,"statfs":{"total":95995035648,"available":95021207552,"internally_reserved":0,"allocated":798306304,"data_stored":797935152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724696,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623892,"kb_used_data":448324,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93121260,"statfs":{"total":95995035648,"available":95356170240,"internally_reserved":0,"allocated":459083776,"data_stored":458716176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953629,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1149432,"kb_used_data":925280,"kb_used_omap":1,"kb_used_meta":224126,"kb_avail":92595720,"statfs":{"total":95995035648,"available":94818017280,"internally_reserved":0,"allocated":947486720,"data_stored":947132574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":229505477},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1192424,"kb_used_data":886352,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92552728,"statfs":{"total":95995035648,"available":94773993472,"internally_reserved":0,"allocated":907624448,"data_stored":907252379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":313391559},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590808,"kb_used_data":447172,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93154344,"statfs":{"total":95995035648,"available":95390048256,"internally_reserved":0,"allocated":457904128,"data_stored":457536528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1610648,"kb_used_data":1318468,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92134504,"statfs":{"total":95995035648,"available":94345732096,"internally_reserved":0,"allocated":1350111232,"data_stored":1349743632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1551764,"kb_used_data":1259524,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92193388,"statfs":{"total":95995035648,"available":94406029312,"internally_reserved":0,"allocated":1289752576,"data_stored":1289384976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1313603584,"data_stored":1313603584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1354366976,"data_stored":1354366976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":460718080,"data_stored":460718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":908460032,"data_stored":908460032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":867827712,"data_stored":867827712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":463732736,"data_stored":463732736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":790560768,"data_stored":790560768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1352925184,"data_stored":1352925184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":444858368,"data_stored":444858368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1103953920,"data_stored":1103953920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1097662464,"data_stored":1097662464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":666632192,"data_stored":666632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:34:51.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-08-30T23:34:51.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:51.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:51.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 59314 59298 74.1164 72 0.0253581 0.0134861 2024-08-30T23:34:51.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:51.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:51.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:51.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:51.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:51.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:51.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:51 smithi106 ceph-mon[16603]: pgmap v804: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s; 670/170805 objects misplaced (0.392%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:34:51.928 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:34:51.929 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:34:51.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:51.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:51.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:51 smithi136 ceph-mon[20207]: pgmap v804: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s; 670/170805 objects misplaced (0.392%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:34:52.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:51 smithi173 ceph-mon[20554]: pgmap v804: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s; 670/170805 objects misplaced (0.392%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:34:52.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:52.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:52.227 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:52.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:52.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:52.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:52.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 60383 60367 73.973 66.8125 0.0332464 0.0135118 2024-08-30T23:34:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:52.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:52.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:52.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:52.690 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:34:52.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:34:52.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:52.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:52.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:52.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:52.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:52.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:52.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:53.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2619356801' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:53.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:52 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2619356801' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:52 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2619356801' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:52 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:34:53.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:53.110 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:53.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:53.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:53.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:53.352 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:53.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:53.388 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:53.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:53.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 61056 61040 73.3593 42.0625 0.0287788 0.0135599 2024-08-30T23:34:53.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:53.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:53.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:53.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:53.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:53.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:53 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:53 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:34:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:53 smithi106 ceph-mon[16603]: osdmap e335: 12 total, 12 up, 12 in 2024-08-30T23:34:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:53 smithi106 ceph-mon[16603]: pgmap v806: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s; 670/175815 objects misplaced (0.381%); 8.0 MiB/s, 127 objects/s recovering 2024-08-30T23:34:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:53.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:53.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:53.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:53.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:54.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:53 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:54.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:53 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:34:54.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:53 smithi136 ceph-mon[20207]: osdmap e335: 12 total, 12 up, 12 in 2024-08-30T23:34:54.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:53 smithi136 ceph-mon[20207]: pgmap v806: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s; 670/175815 objects misplaced (0.381%); 8.0 MiB/s, 127 objects/s recovering 2024-08-30T23:34:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:53 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:34:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:53 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:34:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:53 smithi173 ceph-mon[20554]: osdmap e335: 12 total, 12 up, 12 in 2024-08-30T23:34:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:53 smithi173 ceph-mon[20554]: pgmap v806: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s; 670/175815 objects misplaced (0.381%); 8.0 MiB/s, 127 objects/s recovering 2024-08-30T23:34:54.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:54.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:54.143 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:54.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:34:54.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:34:54.184 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:34:54.184 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:34:54.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:54.207 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:34:54.207 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:34:54.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:54.229 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:34:54.230 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:34:54.251 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:34:54.251 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:34:54.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:34:54.274 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:34:54.295 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:34:54.295 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:34:54.317 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:34:54.317 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:34:54.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:54.345 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:34:54.345 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:34:54.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:54.366 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:34:54.366 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:34:54.389 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:34:54.389 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:34:54.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:34:54.417 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:34:54.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:54.464 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 61137 61121 72.0706 5.0625 0.00323926 0.0135951 2024-08-30T23:34:54.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:54.577 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:54.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:54.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:54.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:54.722 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:54.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:54.845 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:54.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:54.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:54.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:54.977 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/193948878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/193948878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:55.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/193948878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:34:55.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:55.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:55.216 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:55.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:55.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:55.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:34:55.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:34:55.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:55.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 61138 61122 70.7371 0.0625 0.940242 0.0136102 2024-08-30T23:34:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:34:55.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:55.572 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:34:55.572 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:34:55.572 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:34:55.572 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:34:55.572 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:34:55.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:55.628 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 4 pgp_num to 16 2024-08-30T23:34:55.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:55.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:55.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:55.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:55.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:55.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:55.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:56.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4032704919' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"} : dispatch 2024-08-30T23:34:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:55 smithi136 ceph-mon[20207]: pgmap v807: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 948 op/s; 670/181023 objects misplaced (0.370%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:34:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4032704919' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-08-30T23:34:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:55 smithi136 ceph-mon[20207]: osdmap e336: 12 total, 12 up, 12 in 2024-08-30T23:34:56.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4032704919' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"} : dispatch 2024-08-30T23:34:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:55 smithi173 ceph-mon[20554]: pgmap v807: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 948 op/s; 670/181023 objects misplaced (0.370%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:34:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4032704919' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-08-30T23:34:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:55 smithi173 ceph-mon[20554]: osdmap e336: 12 total, 12 up, 12 in 2024-08-30T23:34:56.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4032704919' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"} : dispatch 2024-08-30T23:34:56.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:55 smithi106 ceph-mon[16603]: pgmap v807: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 948 op/s; 670/181023 objects misplaced (0.370%); 6.8 MiB/s, 108 objects/s recovering 2024-08-30T23:34:56.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4032704919' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-08-30T23:34:56.085 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:55 smithi106 ceph-mon[16603]: osdmap e336: 12 total, 12 up, 12 in 2024-08-30T23:34:56.089 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:56.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:56.212 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:56.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:34:56.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:56.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:34:56.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:56.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:56.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 61138 61122 69.4508 0 - 0.0136102 2024-08-30T23:34:56.465 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:56.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:56.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:56.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.732 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:56.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:56.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:56.859 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:56.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:56.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:34:56.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.024 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:57.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:57.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:57.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:57.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:57.166 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:57 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:34:57.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:57.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:57.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:34:57.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:57.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:57.464 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:57.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 62148 62132 69.3378 31.5625 0.124158 0.0143696 2024-08-30T23:34:57.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:34:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:57 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:34:57.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:34:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:57 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:34:57.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:34:57.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:34:57.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:57.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:57.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:57.916 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:57.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:16161 2.6 e109/97 2:60ee081e:::benchmark_data_smithi173_47195_object16160:head v 109'1027, mlcod=109'1024)", 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:34:58.003 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 527.90134789499996, 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.212155128, 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099910+0000", 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099911+0000", 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5499999999999997e-07 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.099933+0000", 2024-08-30T23:34:58.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1793999999999998e-05 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311098+0000", 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21116489199999999 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311105+0000", 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6920000000000002e-06 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311160+0000", 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4716999999999998e-05 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.311179+0000", 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8814e-05 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312023+0000", 2024-08-30T23:34:58.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00084434600000000005 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:10.312065+0000", 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2117999999999999e-05 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.008 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:56060 2.e e124/97 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head v 124'3464, mlcod=124'3461)", 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 498.221428995, 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21259787399999999, 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779829+0000", 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779830+0000", 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-07 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779851+0000", 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1341000000000001e-05 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988300+0000", 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208448405 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988311+0000", 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1141000000000001e-05 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988383+0000", 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2733999999999999e-05 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.988404+0000", 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0509999999999998e-05 2024-08-30T23:34:58.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992259+0000", 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003854704 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992427+0000", 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016835099999999999 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75397 2.1 e132/97 2:8c19b29d:::benchmark_data_smithi173_47195_object75396:head v 132'4751, mlcod=132'4748)", 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 481.39224696700001, 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22761530499999999, 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.014 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609012+0000", 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8199999999999998e-07 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609036+0000", 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4414999999999999e-05 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834165+0000", 2024-08-30T23:34:58.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22512892100000001 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834172+0000", 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2589999999999997e-06 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834226+0000", 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3919999999999999e-05 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.298e-05 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836584+0000", 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023353750000000002 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836626+0000", 2024-08-30T23:34:58.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2552999999999997e-05 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:34:58.020 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 481.37201995599997, 2024-08-30T23:34:58.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:58.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.240592844, 2024-08-30T23:34:58.021 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629239+0000", 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629261+0000", 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2293000000000002e-05 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867840+0000", 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23857915299999999 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867849+0000", 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4940000000000003e-06 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867913+0000", 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3818000000000005e-05 2024-08-30T23:34:58.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867935+0000", 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2024000000000001e-05 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869794+0000", 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001859018 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869831+0000", 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7239e-05 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75429 2.1 e132/97 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head v 132'4755, mlcod=132'4752)", 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 481.04235994300001, 2024-08-30T23:34:58.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24468055399999999, 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958899+0000", 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1939999999999999e-06 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958944+0000", 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4851999999999997e-05 2024-08-30T23:34:58.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201529+0000", 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24258502100000001 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201540+0000", 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0744000000000001e-05 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201630+0000", 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0168999999999997e-05 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201650+0000", 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9477999999999999e-05 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203542+0000", 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00189236 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203579+0000", 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6736000000000002e-05 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100665 2.6 e147/142 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head v 147'6316, mlcod=147'6313)", 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 444.65706536499999, 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214342806, 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.031 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344193+0000", 2024-08-30T23:34:58.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344194+0000", 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.344200+0000", 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5200000000000003e-06 2024-08-30T23:34:58.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555870+0000", 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21167006199999999 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555878+0000", 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8420000000000005e-06 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555957+0000", 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8789999999999996e-05 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.555979+0000", 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2206999999999999e-05 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.558491+0000", 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025112960000000001 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.558536+0000", 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5039e-05 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100667 2.6 e147/142 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head v 147'6317, mlcod=147'6313)", 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 444.653180099, 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.273264648, 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348079+0000", 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1459999999999999e-06 2024-08-30T23:34:58.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348086+0000", 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6649999999999997e-06 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615008+0000", 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26692174899999999 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615016+0000", 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0909999999999993e-06 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615082+0000", 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6359000000000006e-05 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615102+0000", 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9945999999999998e-05 2024-08-30T23:34:58.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621308+0000", 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006205828 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621343+0000", 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4863999999999997e-05 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:194011 2.1 e200/97 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head v 200'12204, mlcod=200'12201)", 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 315.20358130800003, 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82744930299999997, 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797678+0000", 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1800000000000005e-07 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797683+0000", 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6529999999999999e-06 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620982+0000", 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82329911700000002 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620996+0000", 2024-08-30T23:34:58.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3995999999999999e-05 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621133+0000", 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013682700000000001 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621198+0000", 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4368000000000004e-05 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625091+0000", 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003893194 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625126+0000", 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5429999999999998e-05 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.046 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:21992 3.b 3:df4f75f5:::benchmark_data_smithi173_60780_object21991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:25.276436+0000", 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 272.72482247200003, 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21213258500000001, 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 21992 2024-08-30T23:34:58.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276436+0000", 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276436+0000", 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276437+0000", 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276498+0000", 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1155000000000007e-05 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276508+0000", 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0454e-05 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276513+0000", 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9690000000000002e-06 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276553+0000", 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9753999999999999e-05 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276633+0000", 2024-08-30T23:34:58.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9760999999999997e-05 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.276728+0000", 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5237999999999999e-05 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.277492+0000", 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00076429099999999995 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.277989+0000", 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049673899999999995 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488499+0000", 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051017 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488536+0000", 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6732e-05 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:25.488568+0000", 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.243e-05 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:103424 3.2 3:4caab458:::benchmark_data_smithi173_60780_object103423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.74163470299999, 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.221222537, 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103424 2024-08-30T23:34:58.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259624+0000", 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0800000000000004e-07 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259682+0000", 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7364999999999997e-05 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259692+0000", 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0686000000000001e-05 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259698+0000", 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.361e-06 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261702+0000", 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020041289999999999 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261778+0000", 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6148000000000003e-05 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261872+0000", 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4598000000000005e-05 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.263845+0000", 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001972419 2024-08-30T23:34:58.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.264707+0000", 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086252099999999999 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480753+0000", 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.216045457 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480795+0000", 2024-08-30T23:34:58.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2605000000000003e-05 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480846+0000", 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0540000000000001e-05 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:188931 3.b 3:d0b06218:::benchmark_data_smithi173_60780_object85477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e293)", 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:21.650694+0000", 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 96.350564355000003, 2024-08-30T23:34:58.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.210908295, 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 188931 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650694+0000", 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650694+0000", 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650695+0000", 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0599999999999999e-07 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650697+0000", 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7659999999999999e-06 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650705+0000", 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7049999999999999e-06 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650711+0000", 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9989999999999997e-06 2024-08-30T23:34:58.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650759+0000", 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.833e-05 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.650887+0000", 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000127556 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:34:58.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.651011+0000", 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012356800000000001 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.656978+0000", 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0059675580000000004 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.657088+0000", 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010972300000000001 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.861527+0000", 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20443935399999999 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.861562+0000", 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4434000000000003e-05 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:21.861602+0000", 2024-08-30T23:34:58.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0496e-05 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198168 3.2 3:46c77281:::benchmark_data_smithi173_60780_object94714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 85.039042029000001, 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22735555199999999, 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.067 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.068 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198168 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962217+0000", 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2099999999999995e-07 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962221+0000", 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999999e-06 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962234+0000", 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2412000000000001e-05 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962241+0000", 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6789999999999999e-06 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962296+0000", 2024-08-30T23:34:58.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5347999999999999e-05 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962411+0000", 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011498299999999999 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962524+0000", 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011293700000000001 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.967040+0000", 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045164259999999996 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.968062+0000", 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010217270000000001 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189490+0000", 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22142835899999999 2024-08-30T23:34:58.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189535+0000", 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4338999999999997e-05 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189572+0000", 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6990999999999999e-05 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198178 3.2 3:47f13797:::benchmark_data_smithi173_60780_object94724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 85.028252601000005, 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21670980100000001, 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198178 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.075 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973007+0000", 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973013+0000", 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1140000000000002e-06 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973023+0000", 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0516000000000001e-05 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973030+0000", 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8390000000000001e-06 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973072+0000", 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1919000000000003e-05 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973158+0000", 2024-08-30T23:34:58.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6148000000000003e-05 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973234+0000", 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6165e-05 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.976842+0000", 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036077290000000001 2024-08-30T23:34:58.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.978153+0000", 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013109930000000001 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189617+0000", 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21146347300000001 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189674+0000", 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7082e-05 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189715+0000", 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1656999999999998e-05 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.081 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206885 3.5 e300/206 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head v 300'12859, mlcod=300'12855)", 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 73.960014861999994, 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23467711799999999, 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041244+0000", 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.083 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:34:58.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041249+0000", 2024-08-30T23:34:58.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2360000000000004e-06 2024-08-30T23:34:58.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271945+0000", 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.230695859 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271961+0000", 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5815999999999999e-05 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272088+0000", 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126721 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272127+0000", 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8872000000000003e-05 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275888+0000", 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037616350000000002 2024-08-30T23:34:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275920+0000", 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2255000000000002e-05 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206897 3.b 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 73.944173852999995, 2024-08-30T23:34:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58358057900000004, 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206897 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057085+0000", 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3399999999999999e-07 2024-08-30T23:34:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057088+0000", 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8609999999999998e-06 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057094+0000", 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5429999999999999e-06 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057098+0000", 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9720000000000003e-06 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057149+0000", 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0683000000000002e-05 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057264+0000", 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114772 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057387+0000", 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012322799999999999 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.061158+0000", 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037710170000000002 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062218+0000", 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001059948 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640580+0000", 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57836202199999998 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640625+0000", 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5116e-05 2024-08-30T23:34:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640665+0000", 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9882999999999997e-05 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206901 3.7 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 73.938492595, 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38317995100000002, 2024-08-30T23:34:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206901 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062767+0000", 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062771+0000", 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6779999999999999e-06 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062785+0000", 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3839000000000001e-05 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062793+0000", 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:34:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063873+0000", 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001080632 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063965+0000", 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1717000000000003e-05 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:34:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.064031+0000", 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5729999999999996e-05 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.067745+0000", 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037140430000000002 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.071385+0000", 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036407620000000001 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445862+0000", 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37447608799999998 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445912+0000", 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0059999999999998e-05 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445946+0000", 2024-08-30T23:34:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3927000000000002e-05 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206900 3.5 e300/206 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head v 300'12860, mlcod=300'12856)", 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 73.938051328, 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36104319299999998, 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063208+0000", 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063213+0000", 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0080000000000002e-06 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420406+0000", 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35719326299999998 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420426+0000", 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0253000000000001e-05 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420597+0000", 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170786 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420630+0000", 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3578999999999998e-05 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424218+0000", 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035875490000000002 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424250+0000", 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2030999999999999e-05 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:1172 4.4 e311/306 ondisk, result = 0)", 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:01.863109+0000", 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 56.138149189000004, 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214770404, 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.863109+0000", 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.863109+0000", 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.863110+0000", 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8800000000000002e-07 2024-08-30T23:34:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.863113+0000", 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3900000000000002e-06 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077752+0000", 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214638949 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077756+0000", 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5140000000000003e-06 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077794+0000", 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7805999999999997e-05 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077800+0000", 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7479999999999999e-06 2024-08-30T23:34:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077879+0000", 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9408999999999998e-05 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62091 4.4 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.83559036399999997, 2024-08-30T23:34:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318614905, 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62091 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:34:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165670+0000", 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165713+0000", 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3368000000000002e-05 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165732+0000", 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9031000000000001e-05 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165740+0000", 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8050000000000007e-06 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165802+0000", 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2279999999999993e-05 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165946+0000", 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014352899999999999 2024-08-30T23:34:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.166046+0000", 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100046 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169317+0000", 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003271267 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169909+0000", 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059201200000000005 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484178+0000", 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31426850299999998 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484234+0000", 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6678000000000002e-05 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484283+0000", 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8457e-05 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62105 4.4 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.822006032, 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34846845199999998, 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62105 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179254+0000", 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-08-30T23:34:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179297+0000", 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3108000000000001e-05 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179311+0000", 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.345e-05 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179316+0000", 2024-08-30T23:34:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179371+0000", 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5316999999999999e-05 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179508+0000", 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643600000000001 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179665+0000", 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015680499999999999 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182071+0000", 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024068090000000002 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182226+0000", 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015503899999999999 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527606+0000", 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34537996300000001 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527669+0000", 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2966999999999995e-05 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527721+0000", 2024-08-30T23:34:58.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1224000000000003e-05 2024-08-30T23:34:58.124 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.124 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.124 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.124 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:34:58.124 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:34:58.124 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:34:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:58.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:58.145 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:34:58.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:58 smithi106 ceph-mon[16603]: osdmap e337: 12 total, 12 up, 12 in 2024-08-30T23:34:58.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:58 smithi106 ceph-mon[16603]: pgmap v810: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 677 op/s; 83/183009 objects misplaced (0.045%); 6.1 MiB/s, 97 objects/s recovering 2024-08-30T23:34:58.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:58 smithi106 ceph-mon[16603]: 4.6 scrub starts 2024-08-30T23:34:58.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2668957190' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:58.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:34:58.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:34:58.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:34:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:58.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:34:58.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 62758 62742 68.7901 38.125 0.00635841 0.0145312 2024-08-30T23:34:58.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:58.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:58 smithi136 ceph-mon[20207]: osdmap e337: 12 total, 12 up, 12 in 2024-08-30T23:34:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:58 smithi136 ceph-mon[20207]: pgmap v810: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 677 op/s; 83/183009 objects misplaced (0.045%); 6.1 MiB/s, 97 objects/s recovering 2024-08-30T23:34:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:58 smithi136 ceph-mon[20207]: 4.6 scrub starts 2024-08-30T23:34:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2668957190' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:58 smithi173 ceph-mon[20554]: osdmap e337: 12 total, 12 up, 12 in 2024-08-30T23:34:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:58 smithi173 ceph-mon[20554]: pgmap v810: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 677 op/s; 83/183009 objects misplaced (0.045%); 6.1 MiB/s, 97 objects/s recovering 2024-08-30T23:34:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:58 smithi173 ceph-mon[20554]: 4.6 scrub starts 2024-08-30T23:34:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2668957190' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:34:58.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:58.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:58.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:58.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:58.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:58.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:34:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:58.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:58.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:34:58.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:59.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:59.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:34:59.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:34:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:34:59.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:59.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:59.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:59.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:34:59.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 63805 63789 68.7322 65.4375 0.00314939 0.0145441 2024-08-30T23:34:59.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:59 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:34:59.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:59 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:34:59.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:59 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:59.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:59 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:34:59.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2668957190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:59.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:34:59 smithi106 ceph-mon[16603]: osdmap e338: 12 total, 12 up, 12 in 2024-08-30T23:34:59.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:34:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:59 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:34:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:59 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:34:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:59 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:59 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:34:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2668957190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:34:59 smithi136 ceph-mon[20207]: osdmap e338: 12 total, 12 up, 12 in 2024-08-30T23:34:59.546 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:34:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:59 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:34:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:59 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:34:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:59 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:34:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:59 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:34:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2668957190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:34:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:34:59 smithi173 ceph-mon[20554]: osdmap e338: 12 total, 12 up, 12 in 2024-08-30T23:34:59.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:34:59.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:34:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:34:59.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:34:59.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:34:59.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:34:59.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:34:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:34:59.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:00.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:00.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:00.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:00.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:00.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:00.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:35:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:00.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:00.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:00.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:00.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 65015 64999 68.849 75.625 0.00294898 0.0145193 2024-08-30T23:35:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:00 smithi136 ceph-mon[20207]: 4.6 scrub ok 2024-08-30T23:35:00.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:00 smithi136 ceph-mon[20207]: pgmap v812: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 504 op/s; 83/184902 objects misplaced (0.045%); 6.1 MiB/s, 97 objects/s recovering 2024-08-30T23:35:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3382692917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:00.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:00.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:00 smithi173 ceph-mon[20554]: 4.6 scrub ok 2024-08-30T23:35:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:00 smithi173 ceph-mon[20554]: pgmap v812: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 504 op/s; 83/184902 objects misplaced (0.045%); 6.1 MiB/s, 97 objects/s recovering 2024-08-30T23:35:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3382692917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:00.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:00.594 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:00.598 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:00.599 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:00.599 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:00 smithi106 ceph-mon[16603]: 4.6 scrub ok 2024-08-30T23:35:00.599 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:00 smithi106 ceph-mon[16603]: pgmap v812: 17 pgs: 1 active+recovering+undersized+remapped, 16 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 504 op/s; 83/184902 objects misplaced (0.045%); 6.1 MiB/s, 97 objects/s recovering 2024-08-30T23:35:00.599 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3382692917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:00.628 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:00.629 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:00.659 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:00.659 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:00.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:00.686 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:00.686 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:00.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:00.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:00.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:00.741 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:00.742 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:00.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:00.772 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:00.802 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:00.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:00.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:00.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:00.833 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:00.834 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.835 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:00.863 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:00.863 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:00.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:00.957 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:00.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:01.021 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:01.022 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:01.051 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:01.051 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:01.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:01.088 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:01.115 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:01.115 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:01.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:01.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:01.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:35:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:01 smithi136 ceph-mon[20207]: osdmap e339: 12 total, 12 up, 12 in 2024-08-30T23:35:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:01 smithi136 ceph-mon[20207]: osdmap e340: 12 total, 12 up, 12 in 2024-08-30T23:35:01.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:01.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:01.411 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:35:01.411 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:35:01.412 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-30T23:35:01.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:35:01.433 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:01.433 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:01.433 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:01 smithi106 ceph-mon[16603]: osdmap e339: 12 total, 12 up, 12 in 2024-08-30T23:35:01.433 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:01 smithi106 ceph-mon[16603]: osdmap e340: 12 total, 12 up, 12 in 2024-08-30T23:35:01.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:01.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:35:01.466211+0000 min lat: 0.0010779 max lat: 3.89318 avg lat: 0.0145299 2024-08-30T23:35:01.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:35:01.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 65959 65943 68.6847 59 0.0808947 0.0145299 2024-08-30T23:35:01.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:01.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:01.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:01.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:01 smithi173 ceph-mon[20554]: osdmap e339: 12 total, 12 up, 12 in 2024-08-30T23:35:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:01 smithi173 ceph-mon[20554]: osdmap e340: 12 total, 12 up, 12 in 2024-08-30T23:35:01.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:01.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:01.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:01.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.871 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:01.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:01.914 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:01.916 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:01.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:01.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:01.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:02.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:02.130 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:02.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.269 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:02.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:02.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:35:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:02 smithi106 ceph-mon[16603]: pgmap v815: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 673 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-08-30T23:35:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:02 smithi106 ceph-mon[16603]: 4.a scrub starts 2024-08-30T23:35:02.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:02.392 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:02.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 66910 66894 68.533 59.4375 0.0033397 0.0145632 2024-08-30T23:35:02.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:02.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:02.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:02 smithi136 ceph-mon[20207]: pgmap v815: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 673 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-08-30T23:35:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:02 smithi136 ceph-mon[20207]: 4.a scrub starts 2024-08-30T23:35:02.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:02 smithi173 ceph-mon[20554]: pgmap v815: 17 pgs: 17 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 673 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-08-30T23:35:02.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:02 smithi173 ceph-mon[20554]: 4.a scrub starts 2024-08-30T23:35:02.579 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:02.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.665 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:02.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:02.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:02.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:02.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:02.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:02.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:02.953 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:35:02.954 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:35:03.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:03.458 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:03.458 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:03.458 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:03.459 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:03.459 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:03 smithi136 ceph-mon[20207]: osdmap e341: 12 total, 12 up, 12 in 2024-08-30T23:35:03.459 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:03 smithi136 ceph-mon[20207]: 4.a scrub ok 2024-08-30T23:35:03.459 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1826707792' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:03.459 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:35:03.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 67952 67936 68.4779 65.125 0.00649642 0.0145972 2024-08-30T23:35:03.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:03.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:03 smithi106 ceph-mon[16603]: osdmap e341: 12 total, 12 up, 12 in 2024-08-30T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:03 smithi106 ceph-mon[16603]: 4.a scrub ok 2024-08-30T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1826707792' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:03 smithi173 ceph-mon[20554]: osdmap e341: 12 total, 12 up, 12 in 2024-08-30T23:35:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:03 smithi173 ceph-mon[20554]: 4.a scrub ok 2024-08-30T23:35:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1826707792' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:03.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:03.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:03.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:03.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:03.741 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:03.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:03.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:03.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:03.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:03.869 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:35:03.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:35:03.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:03.988 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-08-30T23:35:02.160773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4207542816,"num_objects":64195,"num_object_clones":0,"num_object_copies":192585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64195,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":64316,"num_write_kb":4110502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15532,"num_bytes_recovered":1019098720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64285,"ondisk_log_size":64285,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14909548,"kb_used_data":12246000,"kb_used_omap":18,"kb_used_meta":2663277,"kb_avail":1110032276,"statfs":{"total":1151940427776,"available":1136673050624,"internally_reserved":0,"allocated":12539904000,"data_stored":12535206455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":2727195961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,1,2,4,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":209190912,"num_objects":3192,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-83,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3192,"num_write_kb":204288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":5439488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.697624"},"pg_stats":[{"pgid":"4.f","version":"337'3864","reported_seq":3937,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-30T23:34:57.519936+0000","last_change":"2024-08-30T23:33:56.656974+0000","last_active":"2024-08-30T23:34:57.519936+0000","last_peered":"2024-08-30T23:34:57.519936+0000","last_clean":"2024-08-30T23:34:57.519936+0000","last_became_active":"2024-08-30T23:33:56.656639+0000","last_became_peered":"2024-08-30T23:33:56.656639+0000","last_unstale":"2024-08-30T23:34:57.519936+0000","last_undegraded":"2024-08-30T23:34:57.519936+0000","last_fullsized":"2024-08-30T23:34:57.519936+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:17.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253231104,"num_objects":3864,"num_object_clones":0,"num_object_copies":11592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3864,"num_write_kb":247296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"338'3995","reported_seq":7914,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:34:59.667019+0000","last_change":"2024-08-30T23:34:40.722642+0000","last_active":"2024-08-30T23:34:59.667019+0000","last_peered":"2024-08-30T23:34:59.667019+0000","last_clean":"2024-08-30T23:34:59.667019+0000","last_became_active":"2024-08-30T23:34:40.719708+0000","last_became_peered":"2024-08-30T23:34:40.719708+0000","last_unstale":"2024-08-30T23:34:59.667019+0000","last_undegraded":"2024-08-30T23:34:59.667019+0000","last_fullsized":"2024-08-30T23:34:59.667019+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:22.082750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261816320,"num_objects":3995,"num_object_clones":0,"num_object_copies":11985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":255680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":125632512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"338'3990","reported_seq":4066,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:34:59.987985+0000","last_change":"2024-08-30T23:33:56.656195+0000","last_active":"2024-08-30T23:34:59.987985+0000","last_peered":"2024-08-30T23:34:59.987985+0000","last_clean":"2024-08-30T23:34:59.987985+0000","last_became_active":"2024-08-30T23:33:56.655881+0000","last_became_peered":"2024-08-30T23:33:56.655881+0000","last_unstale":"2024-08-30T23:34:59.987985+0000","last_undegraded":"2024-08-30T23:34:59.987985+0000","last_fullsized":"2024-08-30T23:34:59.987985+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3990,"log_dups_size":0,"ondisk_log_size":3990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:07.130146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261488640,"num_objects":3990,"num_object_clones":0,"num_object_copies":11970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":255360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11],"acting":[9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"338'3852","reported_seq":3927,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:34:59.690383+0000","last_change":"2024-08-30T23:33:56.657408+0000","last_active":"2024-08-30T23:34:59.690383+0000","last_peered":"2024-08-30T23:34:59.690383+0000","last_clean":"2024-08-30T23:34:59.690383+0000","last_became_active":"2024-08-30T23:33:56.657074+0000","last_became_peered":"2024-08-30T23:33:56.657074+0000","last_unstale":"2024-08-30T23:34:59.690383+0000","last_undegraded":"2024-08-30T23:34:59.690383+0000","last_fullsized":"2024-08-30T23:34:59.690383+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3852,"log_dups_size":0,"ondisk_log_size":3852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:17.110283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252444672,"num_objects":3852,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":246528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"338'3985","reported_seq":4060,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:34:58.572077+0000","last_change":"2024-08-30T23:33:56.655706+0000","last_active":"2024-08-30T23:34:58.572077+0000","last_peered":"2024-08-30T23:34:58.572077+0000","last_clean":"2024-08-30T23:34:58.572077+0000","last_became_active":"2024-08-30T23:33:56.655416+0000","last_became_peered":"2024-08-30T23:33:56.655416+0000","last_unstale":"2024-08-30T23:34:58.572077+0000","last_undegraded":"2024-08-30T23:34:58.572077+0000","last_fullsized":"2024-08-30T23:34:58.572077+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:14.875748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261160960,"num_objects":3985,"num_object_clones":0,"num_object_copies":11955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3985,"num_write_kb":255040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"337'3904","reported_seq":7817,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-30T23:34:57.494258+0000","last_change":"2024-08-30T23:34:46.670467+0000","last_active":"2024-08-30T23:34:57.494258+0000","last_peered":"2024-08-30T23:34:57.494258+0000","last_clean":"2024-08-30T23:34:57.494258+0000","last_became_active":"2024-08-30T23:34:46.666422+0000","last_became_peered":"2024-08-30T23:34:46.666422+0000","last_unstale":"2024-08-30T23:34:57.494258+0000","last_undegraded":"2024-08-30T23:34:57.494258+0000","last_fullsized":"2024-08-30T23:34:57.494258+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3904,"log_dups_size":0,"ondisk_log_size":3904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:00.960858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":3904,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3904,"num_write_kb":249856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":125435904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"340'4119","reported_seq":8089,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-08-30T23:35:01.938357+0000","last_change":"2024-08-30T23:34:41.063395+0000","last_active":"2024-08-30T23:35:01.938357+0000","last_peered":"2024-08-30T23:35:01.938357+0000","last_clean":"2024-08-30T23:35:01.938357+0000","last_became_active":"2024-08-30T23:34:40.834281+0000","last_became_peered":"2024-08-30T23:34:40.834281+0000","last_unstale":"2024-08-30T23:35:01.938357+0000","last_undegraded":"2024-08-30T23:35:01.938357+0000","last_fullsized":"2024-08-30T23:35:01.938357+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:02.094794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269942784,"num_objects":4119,"num_object_clones":0,"num_object_copies":12357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4119,"num_write_kb":263616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"340'4221","reported_seq":8284,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-08-30T23:35:01.328411+0000","last_change":"2024-08-30T23:34:42.645739+0000","last_active":"2024-08-30T23:35:01.328411+0000","last_peered":"2024-08-30T23:35:01.328411+0000","last_clean":"2024-08-30T23:35:01.328411+0000","last_became_active":"2024-08-30T23:34:42.641060+0000","last_became_peered":"2024-08-30T23:34:42.641060+0000","last_unstale":"2024-08-30T23:35:01.328411+0000","last_undegraded":"2024-08-30T23:35:01.328411+0000","last_fullsized":"2024-08-30T23:35:01.328411+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:21.817074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276627456,"num_objects":4221,"num_object_clones":0,"num_object_copies":12663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":270144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1986,"num_bytes_recovered":130154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"340'4062","reported_seq":7943,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-08-30T23:35:01.290496+0000","last_change":"2024-08-30T23:34:42.646300+0000","last_active":"2024-08-30T23:35:01.290496+0000","last_peered":"2024-08-30T23:35:01.290496+0000","last_clean":"2024-08-30T23:35:01.290496+0000","last_became_active":"2024-08-30T23:34:42.642239+0000","last_became_peered":"2024-08-30T23:34:42.642239+0000","last_unstale":"2024-08-30T23:35:01.290496+0000","last_undegraded":"2024-08-30T23:35:01.290496+0000","last_fullsized":"2024-08-30T23:35:01.290496+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:05.213893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266207232,"num_objects":4062,"num_object_clones":0,"num_object_copies":12186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":259968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1895,"num_bytes_recovered":124190720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"338'4067","reported_seq":4191,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:35:00.029501+0000","last_change":"2024-08-30T23:34:58.649574+0000","last_active":"2024-08-30T23:35:00.029501+0000","last_peered":"2024-08-30T23:35:00.029501+0000","last_clean":"2024-08-30T23:35:00.029501+0000","last_became_active":"2024-08-30T23:33:57.414169+0000","last_became_peered":"2024-08-30T23:33:57.414169+0000","last_unstale":"2024-08-30T23:35:00.029501+0000","last_undegraded":"2024-08-30T23:35:00.029501+0000","last_fullsized":"2024-08-30T23:35:00.029501+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'3948","last_scrub_stamp":"2024-08-30T23:34:58.649510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:34:58.649510+0000","objects_scrubbed":3911,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:36:18.903841+0000","scrub_duration":1186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266534912,"num_objects":4067,"num_object_clones":0,"num_object_copies":12201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":260288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"338'3925","reported_seq":7920,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:34:58.569374+0000","last_change":"2024-08-30T23:34:45.641113+0000","last_active":"2024-08-30T23:34:58.569374+0000","last_peered":"2024-08-30T23:34:58.569374+0000","last_clean":"2024-08-30T23:34:58.569374+0000","last_became_active":"2024-08-30T23:34:45.636628+0000","last_became_peered":"2024-08-30T23:34:45.636628+0000","last_unstale":"2024-08-30T23:34:58.569374+0000","last_undegraded":"2024-08-30T23:34:58.569374+0000","last_fullsized":"2024-08-30T23:34:58.569374+0000","mapping_epoch":331,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:11.330254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257228800,"num_objects":3925,"num_object_clones":0,"num_object_copies":11775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":251200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1953,"num_bytes_recovered":127991808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10],"acting":[3,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"337'3863","reported_seq":3937,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-30T23:34:57.511125+0000","last_change":"2024-08-30T23:33:56.657715+0000","last_active":"2024-08-30T23:34:57.511125+0000","last_peered":"2024-08-30T23:34:57.511125+0000","last_clean":"2024-08-30T23:34:57.511125+0000","last_became_active":"2024-08-30T23:33:56.657367+0000","last_became_peered":"2024-08-30T23:33:56.657367+0000","last_unstale":"2024-08-30T23:34:57.511125+0000","last_undegraded":"2024-08-30T23:34:57.511125+0000","last_fullsized":"2024-08-30T23:34:57.511125+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:12.708736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253165568,"num_objects":3863,"num_object_clones":0,"num_object_copies":11589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3863,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"338'4076","reported_seq":4152,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-08-30T23:35:00.030092+0000","last_change":"2024-08-30T23:33:57.275424+0000","last_active":"2024-08-30T23:35:00.030092+0000","last_peered":"2024-08-30T23:35:00.030092+0000","last_clean":"2024-08-30T23:35:00.030092+0000","last_became_active":"2024-08-30T23:33:57.274974+0000","last_became_peered":"2024-08-30T23:33:57.274974+0000","last_unstale":"2024-08-30T23:35:00.030092+0000","last_undegraded":"2024-08-30T23:35:00.030092+0000","last_fullsized":"2024-08-30T23:35:00.030092+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:05.717298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267124736,"num_objects":4076,"num_object_clones":0,"num_object_copies":12228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":260864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,8],"acting":[9,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"340'4165","reported_seq":4245,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-08-30T23:35:01.327894+0000","last_change":"2024-08-30T23:33:57.275412+0000","last_active":"2024-08-30T23:35:01.327894+0000","last_peered":"2024-08-30T23:35:01.327894+0000","last_clean":"2024-08-30T23:35:01.327894+0000","last_became_active":"2024-08-30T23:33:57.274888+0000","last_became_peered":"2024-08-30T23:33:57.274888+0000","last_unstale":"2024-08-30T23:35:01.327894+0000","last_undegraded":"2024-08-30T23:35:01.327894+0000","last_fullsized":"2024-08-30T23:35:01.327894+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:14.055546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272957440,"num_objects":4165,"num_object_clones":0,"num_object_copies":12495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4165,"num_write_kb":266560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"337'3905","reported_seq":7879,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-30T23:34:57.508957+0000","last_change":"2024-08-30T23:34:42.645841+0000","last_active":"2024-08-30T23:34:57.508957+0000","last_peered":"2024-08-30T23:34:57.508957+0000","last_clean":"2024-08-30T23:34:57.508957+0000","last_became_active":"2024-08-30T23:34:42.642044+0000","last_became_peered":"2024-08-30T23:34:42.642044+0000","last_unstale":"2024-08-30T23:34:57.508957+0000","last_undegraded":"2024-08-30T23:34:57.508957+0000","last_fullsized":"2024-08-30T23:34:57.508957+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:16.213837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255918080,"num_objects":3905,"num_object_clones":0,"num_object_copies":11715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3905,"num_write_kb":249920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1944,"num_bytes_recovered":127401984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"340'4200","reported_seq":8246,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-08-30T23:35:01.296805+0000","last_change":"2024-08-30T23:34:58.157418+0000","last_active":"2024-08-30T23:35:01.296805+0000","last_peered":"2024-08-30T23:35:01.296805+0000","last_clean":"2024-08-30T23:35:01.296805+0000","last_became_active":"2024-08-30T23:34:58.152061+0000","last_became_peered":"2024-08-30T23:34:58.152061+0000","last_unstale":"2024-08-30T23:35:01.296805+0000","last_undegraded":"2024-08-30T23:35:01.296805+0000","last_fullsized":"2024-08-30T23:35:01.296805+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":306,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:33:55.643119+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:35:12.029111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275251200,"num_objects":4200,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4200,"num_write_kb":268800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":129630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0],"acting":[10,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":634,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-30T23:34:57.144863+0000","last_change":"2024-08-30T23:34:35.631256+0000","last_active":"2024-08-30T23:34:57.144863+0000","last_peered":"2024-08-30T23:34:57.144863+0000","last_clean":"2024-08-30T23:34:57.144863+0000","last_became_active":"2024-08-30T23:34:35.630884+0000","last_became_peered":"2024-08-30T23:34:35.630884+0000","last_unstale":"2024-08-30T23:34:57.144863+0000","last_undegraded":"2024-08-30T23:34:57.144863+0000","last_fullsized":"2024-08-30T23:34:57.144863+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:33:15.070383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:33:15.070383+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4206952448,"num_objects":64193,"num_object_clones":0,"num_object_copies":192579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64193,"num_write_kb":4108352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15526,"num_bytes_recovered":1017511936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12549292032,"data_stored":12549292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64193,"ondisk_log_size":64193,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187582,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113208,"kb_used_data":798816,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92631944,"statfs":{"total":95995035648,"available":94855110656,"internally_reserved":0,"allocated":817987584,"data_stored":817585140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":321911236},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449221,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407272,"kb_used_data":1308056,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92337880,"statfs":{"total":95995035648,"available":94553989120,"internally_reserved":0,"allocated":1339449344,"data_stored":1339050472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564500,"kb_used_data":1280640,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":92180652,"statfs":{"total":95995035648,"available":94392987648,"internally_reserved":0,"allocated":1311375360,"data_stored":1310985534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":290650560},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666468,"kb_used_data":522828,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93078684,"statfs":{"total":95995035648,"available":95312572416,"internally_reserved":0,"allocated":535375872,"data_stored":534981548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234128,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725316,"kb_used_data":1495600,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92019836,"statfs":{"total":95995035648,"available":94228312064,"internally_reserved":0,"allocated":1531494400,"data_stored":1531110081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528471,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971120,"kb_used_data":787232,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92774032,"statfs":{"total":95995035648,"available":95000608768,"internally_reserved":0,"allocated":806125568,"data_stored":805723144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724698,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679372,"kb_used_data":503796,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93065780,"statfs":{"total":95995035648,"available":95299358720,"internally_reserved":0,"allocated":515887104,"data_stored":515501818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953632,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282264,"kb_used_data":1066432,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92462888,"statfs":{"total":95995035648,"available":94681997312,"internally_reserved":0,"allocated":1092026368,"data_stored":1091636542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292244,"kb_used_data":994496,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92452908,"statfs":{"total":95995035648,"available":94671777792,"internally_reserved":0,"allocated":1018363904,"data_stored":1017974078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":304871879},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642960,"kb_used_data":499328,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93102192,"statfs":{"total":95995035648,"available":95336644608,"internally_reserved":0,"allocated":511311872,"data_stored":510922046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1775628,"kb_used_data":1491764,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91969524,"statfs":{"total":95995035648,"available":94176792576,"internally_reserved":0,"allocated":1527566336,"data_stored":1527181050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738548,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1789196,"kb_used_data":1497012,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91955956,"statfs":{"total":95995035648,"available":94162898944,"internally_reserved":0,"allocated":1532940288,"data_stored":1532555002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1549271040,"data_stored":1549271040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1527447552,"data_stored":1527447552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":515702784,"data_stored":515702784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1017774080,"data_stored":1017774080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1042612224,"data_stored":1042612224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":520028160,"data_stored":520028160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":807600128,"data_stored":807600128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1535967232,"data_stored":1535967232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":539426816,"data_stored":539426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1328414720,"data_stored":1328414720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1348468736,"data_stored":1348468736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":816578560,"data_stored":816578560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:35:03.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-08-30T23:35:03.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:03.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:04.095 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:04.122 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:04.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:04.238 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:04.246 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:04.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:04.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:04.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:04.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 68930 68914 68.3611 61.125 0.0177637 0.0146197 2024-08-30T23:35:04.471 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:04 smithi106 ceph-mon[16603]: from='client.16182 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:35:04.471 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:04 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:04.471 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1826707792' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:04.471 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:04 smithi106 ceph-mon[16603]: osdmap e342: 12 total, 12 up, 12 in 2024-08-30T23:35:04.472 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:04 smithi106 ceph-mon[16603]: pgmap v818: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:35:04.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:04.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:04.518 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:04 smithi136 ceph-mon[20207]: from='client.16182 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:35:04.518 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:04 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:04.518 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1826707792' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:04.518 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:04 smithi136 ceph-mon[20207]: osdmap e342: 12 total, 12 up, 12 in 2024-08-30T23:35:04.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:04 smithi136 ceph-mon[20207]: pgmap v818: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:35:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:04 smithi173 ceph-mon[20554]: from='client.16182 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:35:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:04 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1826707792' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:04 smithi173 ceph-mon[20554]: osdmap e342: 12 total, 12 up, 12 in 2024-08-30T23:35:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:04 smithi173 ceph-mon[20554]: pgmap v818: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:35:04.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:04.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:04.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:04.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:04.835 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:04.853 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:04.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:04.894 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:04.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:04.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:05.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:05.037 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:05.037 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:05.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:05.156 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:05.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:05.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:05.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:05.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.415 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:05.465 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 70201 70185 68.534 79.4375 0.0241052 0.0145861 2024-08-30T23:35:05.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:05.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:05.613 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:05.613 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:05.641 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:05.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:05.668 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:05.671 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:05.697 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:05.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:35:05.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:05.731 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:05.763 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:05.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:05.793 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:05.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.800 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:05.827 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:05.851 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:05.851 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:05.880 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:05.880 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:05.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:05.919 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:05.919 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:05.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:05.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:05.948 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:05.948 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:06.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:06.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:06.061 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:06.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:06.187 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:06.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:06.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:06.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:06.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:06.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 71377 71361 68.6103 73.5 0.00661525 0.014568 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: pgmap v819: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 1000 KiB/s, 15 objects/s recovering 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2321971508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.484 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2927181981' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-30T23:35:06.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:06.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: pgmap v819: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 1000 KiB/s, 15 objects/s recovering 2024-08-30T23:35:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2321971508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:35:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:35:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2927181981' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-30T23:35:06.591 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:35:06.592 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:35:06.592 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:35:06.592 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:35:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: pgmap v819: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 1000 KiB/s, 15 objects/s recovering 2024-08-30T23:35:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2321971508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:35:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:35:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:35:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2927181981' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-30T23:35:06.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:06.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:06.706 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pgp_num to 1 2024-08-30T23:35:06.771 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:35:06.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:06.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:06.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:06.952 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:06.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:07.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:07.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:07.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:07.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:07.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:07.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:07.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:07.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 72500 72484 68.6341 70.1875 0.0346353 0.014563 2024-08-30T23:35:07.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:07.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:35:07.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:07.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:07.638 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:07.698 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:07.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2927181981' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-30T23:35:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:07 smithi136 ceph-mon[20207]: osdmap e343: 12 total, 12 up, 12 in 2024-08-30T23:35:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:07 smithi136 ceph-mon[20207]: pgmap v821: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 913 op/s 2024-08-30T23:35:07.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:07.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:07.853 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2927181981' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-30T23:35:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:07 smithi106 ceph-mon[16603]: osdmap e343: 12 total, 12 up, 12 in 2024-08-30T23:35:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:07 smithi106 ceph-mon[16603]: pgmap v821: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 913 op/s 2024-08-30T23:35:07.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:07.979 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:07.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:07.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2927181981' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-30T23:35:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:07 smithi173 ceph-mon[20554]: osdmap e343: 12 total, 12 up, 12 in 2024-08-30T23:35:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:07 smithi173 ceph-mon[20554]: pgmap v821: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 913 op/s 2024-08-30T23:35:08.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:08.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:08.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:08.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:08.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:08.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:08.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:08.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 73673 73657 68.7039 73.3125 0.00328272 0.0145416 2024-08-30T23:35:08.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:08.496 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:08.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:08.625 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:08.711 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:35:08.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:08.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:08.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:08.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:08.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:08.866 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1869919525' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:08.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:35:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:08.892 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:08.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:08.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:09.016 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1869919525' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1869919525' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:09.114 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:35:09.114 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:35:09.114 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:35:09.114 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:35:09.115 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:35:09.118 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:09.146 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:09.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:09.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:09.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:09.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 74848 74832 68.7733 73.4375 0.0110379 0.0145357 2024-08-30T23:35:09.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:09.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:09.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:09.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:09.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:09.801 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:09.864 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:09 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:09.864 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1869919525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:09.864 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:09 smithi136 ceph-mon[20207]: osdmap e344: 12 total, 12 up, 12 in 2024-08-30T23:35:09.864 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:09 smithi136 ceph-mon[20207]: pgmap v823: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s 2024-08-30T23:35:09.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:35:09.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:09.928 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:09.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:10.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:10.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:09 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1869919525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:09 smithi173 ceph-mon[20554]: osdmap e344: 12 total, 12 up, 12 in 2024-08-30T23:35:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:09 smithi173 ceph-mon[20554]: pgmap v823: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s 2024-08-30T23:35:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:09 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1869919525' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:09 smithi106 ceph-mon[16603]: osdmap e344: 12 total, 12 up, 12 in 2024-08-30T23:35:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:09 smithi106 ceph-mon[16603]: pgmap v823: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 968 op/s 2024-08-30T23:35:10.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:10.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:10.199 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:10.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:10.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:10.344 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:10.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:10.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:10.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 75997 75981 68.8173 71.8125 0.0141371 0.0145244 2024-08-30T23:35:10.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:10.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:10.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:10.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:10.722 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:14227 2.2 2:4af79d78:::benchmark_data_smithi173_47195_object14226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 541.84503169899995, 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24349405099999999, 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14227 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875038+0000", 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875039+0000", 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046e-06 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875091+0000", 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2556999999999998e-05 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875103+0000", 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1331e-05 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875108+0000", 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1549999999999998e-06 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875152+0000", 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4161e-05 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875235+0000", 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2814999999999999e-05 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875316+0000", 2024-08-30T23:35:10.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699000000000005e-05 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875841+0000", 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052407300000000001 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.037068+0000", 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.161227964 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118454+0000", 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.081385499 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118498+0000", 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3513000000000002e-05 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118532+0000", 2024-08-30T23:35:10.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4238e-05 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:14227 2.2 e109/97 ondisk, result = 0)", 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 541.84407510799997, 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.242546031, 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875994+0000", 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875995+0000", 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0100000000000004e-07 2024-08-30T23:35:10.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:08.875999+0000", 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.177e-06 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.117926+0000", 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.241927006 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.117934+0000", 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5290000000000003e-06 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118441+0000", 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050753599999999995 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118448+0000", 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6619999999999999e-06 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:09.118540+0000", 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2419999999999999e-05 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:36436 2.2 2:4f13755f:::benchmark_data_smithi173_47195_object36435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 525.73690271199996, 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23381564699999999, 2024-08-30T23:35:10.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36436 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983167+0000", 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983169+0000", 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9470000000000002e-06 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983211+0000", 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1959000000000003e-05 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983229+0000", 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8508e-05 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983238+0000", 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9549999999999998e-06 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983294+0000", 2024-08-30T23:35:10.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5605e-05 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983377+0000", 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3768000000000002e-05 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.983458+0000", 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0272000000000007e-05 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.984672+0000", 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214449 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.985077+0000", 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040527100000000002 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216885+0000", 2024-08-30T23:35:10.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.231807244 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216945+0000", 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0545999999999999e-05 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.216982+0000", 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7122999999999997e-05 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:36447 2.2 2:42c8382e:::benchmark_data_smithi173_47195_object36446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 525.73183131200005, 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31128452299999998, 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36447 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988238+0000", 2024-08-30T23:35:10.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988239+0000", 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4e-07 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988315+0000", 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5585000000000002e-05 2024-08-30T23:35:10.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988325+0000", 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0549e-05 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988330+0000", 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9729999999999997e-06 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988371+0000", 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1201000000000001e-05 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988453+0000", 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1965000000000006e-05 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.988536+0000", 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3088000000000001e-05 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.989696+0000", 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001159979 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:24.989973+0000", 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027723299999999999 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299419+0000", 2024-08-30T23:35:10.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30944559100000002 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299477+0000", 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7568000000000002e-05 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:25.299523+0000", 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5951000000000001e-05 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.748 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(2.b 153/133 [PushOp(2:d76b2f20:::benchmark_data_smithi173_47195_object57342:head, version: 125'3537, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d76b2f20:::benchmark_data_smithi173_47195_object57342:head@125'3537, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:46.081333+0000", 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 444.63873686599999, 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.251162106, 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081333+0000", 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081333+0000", 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081334+0000", 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2699999999999999e-06 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.081392+0000", 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8628999999999997e-05 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330558+0000", 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24916590999999999 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330571+0000", 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2774e-05 2024-08-30T23:35:10.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330638+0000", 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7082999999999995e-05 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.330646+0000", 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0120000000000005e-06 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:46.332495+0000", 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018484280000000001 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155144 2.b 2:dec0e07e:::benchmark_data_smithi173_47195_object58136:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.99881286800002, 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24062692899999999, 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.753 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155144 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721258+0000", 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.688e-06 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721264+0000", 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5060000000000003e-06 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721277+0000", 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3345e-05 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721295+0000", 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7598e-05 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721850+0000", 2024-08-30T23:35:10.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055514699999999998 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721861+0000", 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1009000000000001e-05 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.932731+0000", 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.210870364 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.932895+0000", 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016375999999999999 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.933065+0000", 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000170104 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943014+0000", 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099494240000000001 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943076+0000", 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1239000000000001e-05 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943195+0000", 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118923 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.960869+0000", 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017674390000000002 2024-08-30T23:35:10.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.960910+0000", 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1467999999999997e-05 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.961883+0000", 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00097296399999999999 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155158 2.b 2:dcb1d3fb:::benchmark_data_smithi173_47195_object58150:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.99875643000001, 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24126403199999999, 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155158 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.761 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:35:10.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4999999999999998e-07 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721316+0000", 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1900000000000002e-06 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721321+0000", 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4450000000000004e-06 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721324+0000", 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721883+0000", 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055905600000000005 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721891+0000", 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1509999999999998e-06 2024-08-30T23:35:10.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937082+0000", 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.215191454 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937182+0000", 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9567000000000004e-05 2024-08-30T23:35:10.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937273+0000", 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1305999999999996e-05 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943145+0000", 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058718049999999999 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948639+0000", 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054946880000000002 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948721+0000", 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1936999999999998e-05 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.961943+0000", 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013221568 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.962004+0000", 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0990999999999999e-05 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.962577+0000", 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057307400000000002 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155229 2.b 2:ddba2b2a:::benchmark_data_smithi173_47195_object58221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.998739317, 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.248255843, 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155229 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:35:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2000000000000001e-07 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721332+0000", 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3260000000000001e-06 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721336+0000", 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0759999999999996e-06 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721339+0000", 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.569e-06 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721916+0000", 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057626300000000004 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721923+0000", 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5530000000000004e-06 2024-08-30T23:35:10.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940166+0000", 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21824272 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940264+0000", 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8392000000000003e-05 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940350+0000", 2024-08-30T23:35:10.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5668999999999996e-05 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.945936+0000", 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055864249999999999 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948776+0000", 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028393670000000002 2024-08-30T23:35:10.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957244+0000", 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084679639999999997 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.968623+0000", 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011379146999999999 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.968683+0000", 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0170999999999997e-05 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969586+0000", 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090298099999999997 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.776 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155284 2.b 2:d661bc27:::benchmark_data_smithi173_47195_object58276:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.99872676400003, 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24904695199999999, 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155284 2024-08-30T23:35:10.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0200000000000001e-07 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721344+0000", 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.519e-06 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721347+0000", 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.118e-06 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721351+0000", 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5429999999999998e-06 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721957+0000", 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060615599999999995 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721963+0000", 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3619999999999997e-06 2024-08-30T23:35:10.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943241+0000", 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221278947 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943333+0000", 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1094000000000006e-05 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943419+0000", 2024-08-30T23:35:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6588999999999994e-05 2024-08-30T23:35:10.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948733+0000", 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053133649999999996 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951543+0000", 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002810819 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957285+0000", 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057415169999999998 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969633+0000", 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012348181999999999 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969670+0000", 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6854000000000003e-05 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970390+0000", 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071968499999999996 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155286 2.b 2:da5a53b4:::benchmark_data_smithi173_47195_object58278:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.99871474600002, 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24979741499999999, 2024-08-30T23:35:10.787 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155286 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1199999999999999e-07 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721356+0000", 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2079999999999999e-06 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721360+0000", 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8750000000000002e-06 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721363+0000", 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7389999999999999e-06 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721987+0000", 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00062452099999999995 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721998+0000", 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277000000000001e-05 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.945970+0000", 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.223971225 2024-08-30T23:35:10.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.946066+0000", 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6781999999999996e-05 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.946148+0000", 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1522999999999999e-05 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951509+0000", 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053612979999999996 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.954537+0000", 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030273959999999999 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957323+0000", 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002785839 2024-08-30T23:35:10.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970435+0000", 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013112342000000001 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970469+0000", 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4140000000000002e-05 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.971152+0000", 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068303799999999996 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155298 2.b 2:dd53a97e:::benchmark_data_smithi173_47195_object58290:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.99870369299998, 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25834940200000001, 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.795 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155298 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2000000000000001e-07 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721367+0000", 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0950000000000001e-06 2024-08-30T23:35:10.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721371+0000", 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.783e-06 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721373+0000", 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6280000000000001e-06 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.722015+0000", 2024-08-30T23:35:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064132200000000005 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.722020+0000", 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.028e-06 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948797+0000", 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22677699400000001 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948892+0000", 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5272000000000005e-05 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948978+0000", 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6365000000000004e-05 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951655+0000", 2024-08-30T23:35:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026768669999999999 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.954636+0000", 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029808159999999998 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957360+0000", 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002723821 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.978592+0000", 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021232183000000002 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.978633+0000", 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0460000000000002e-05 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.979715+0000", 2024-08-30T23:35:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001082548 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193943 2.b 2:d4b90c03:::benchmark_data_smithi173_47195_object96935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 328.01194740400001, 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42676301799999999, 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193943 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708123+0000", 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.09e-07 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708129+0000", 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5119999999999997e-06 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708141+0000", 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243999999999999e-05 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708147+0000", 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1920000000000002e-06 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709291+0000", 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011439899999999999 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709428+0000", 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013711600000000001 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709522+0000", 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3832000000000006e-05 2024-08-30T23:35:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.714473+0000", 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049515779999999999 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.714764+0000", 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029112000000000002 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.718006+0000", 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032420859999999999 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134491+0000", 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41648495699999999 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134534+0000", 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2568000000000003e-05 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134885+0000", 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035101400000000002 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:193943 2.b e200/182 ondisk, result = 0)", 2024-08-30T23:35:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 328.00612523799998, 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42094626899999998, 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713945+0000", 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9500000000000003e-07 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713948+0000", 2024-08-30T23:35:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134314+0000", 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42036561300000003 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134335+0000", 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1803000000000001e-05 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134478+0000", 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014253700000000001 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134487+0000", 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5469999999999997e-06 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134890+0000", 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040388000000000002 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(2.b e200,182 2:d6a96fd5:::benchmark_data_smithi173_47195_object12191:head,153'6821)", 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 327.859829074, 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66937705700000005, 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860247+0000", 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3439999999999997e-06 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860257+0000", 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0582e-05 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529079+0000", 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66882161500000004 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529086+0000", 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8430000000000004e-06 2024-08-30T23:35:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529174+0000", 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8320999999999997e-05 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529617+0000", 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044335199999999999 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22811 3.c 3:362f756c:::benchmark_data_smithi173_60780_object22810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 284.71502296, 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32742973199999997, 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22811 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005047+0000", 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:35:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005105+0000", 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7797999999999998e-05 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005114+0000", 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0319999999999992e-06 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005119+0000", 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5820000000000003e-06 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005151+0000", 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2404999999999999e-05 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005230+0000", 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9212e-05 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005314+0000", 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3540000000000003e-05 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005956+0000", 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064215699999999997 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.006179+0000", 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022317300000000001 2024-08-30T23:35:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332412+0000", 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.326233314 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332449+0000", 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6671999999999997e-05 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332476+0000", 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7161999999999998e-05 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:198842 3.5 3:ab155887:::benchmark_data_smithi173_60780_object95388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:33.844424+0000", 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 96.875644932, 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23186731199999999, 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 198842 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844424+0000", 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844424+0000", 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844426+0000", 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3e-06 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844431+0000", 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844444+0000", 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3559e-05 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844452+0000", 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3259999999999998e-06 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.847658+0000", 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032057539999999999 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.847800+0000", 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142497 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.847942+0000", 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141485 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.851731+0000", 2024-08-30T23:35:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037893169999999999 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.939608+0000", 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.087877409000000004 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076162+0000", 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13655384300000001 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076242+0000", 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0294000000000001e-05 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076292+0000", 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9394000000000002e-05 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:198842 3.5 e299/206 ondisk, result = 0)", 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:33.852776+0000", 2024-08-30T23:35:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 96.867293838999998, 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22352028700000001, 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.852776+0000", 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.852776+0000", 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.852776+0000", 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3000000000000002e-07 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.852779+0000", 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7099999999999999e-06 2024-08-30T23:35:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076051+0000", 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22327245400000001 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076058+0000", 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3590000000000002e-06 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076151+0000", 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2237999999999993e-05 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076157+0000", 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.365e-06 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076296+0000", 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013883100000000001 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206885 3.5 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.040796+0000", 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.679273046000006, 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.235763583, 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206885 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040796+0000", 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040796+0000", 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040798+0000", 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.536e-06 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040803+0000", 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040816+0000", 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3443e-05 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040823+0000", 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2400000000000001e-06 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040886+0000", 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2333000000000001e-05 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.041010+0000", 2024-08-30T23:35:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124573 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.041133+0000", 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012286299999999999 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.045228+0000", 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040943810000000002 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.045331+0000", 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103051 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.276401+0000", 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.231070047 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.276493+0000", 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1821000000000001e-05 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.276560+0000", 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7374999999999998e-05 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206900 3.5 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.062814+0000", 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 86.657255703000004, 2024-08-30T23:35:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36188622999999998, 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206900 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062814+0000", 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062814+0000", 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062815+0000", 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0640000000000001e-06 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062818+0000", 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2610000000000002e-06 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062828+0000", 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7159999999999998e-06 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062833+0000", 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0669999999999999e-06 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062883+0000", 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9716000000000003e-05 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.063029+0000", 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146766 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:35:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.063159+0000", 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012947100000000001 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.067813+0000", 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046544960000000002 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.068092+0000", 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027919499999999999 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.424625+0000", 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.356532126 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.424667+0000", 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2113000000000002e-05 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.424700+0000", 2024-08-30T23:35:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3238999999999998e-05 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:31015 4.2 e316/306 4:4a9291e8:::benchmark_data_smithi173_74914_object31014:head v 316'1935, mlcod=316'1932)", 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:20.657893+0000", 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.062176379, 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27371718299999998, 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657893+0000", 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657893+0000", 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657894+0000", 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1900000000000002e-07 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657996+0000", 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102334 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930224+0000", 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27222813099999998 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930248+0000", 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3320999999999999e-05 2024-08-30T23:35:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930347+0000", 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9932999999999993e-05 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930371+0000", 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3311000000000001e-05 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.931568+0000", 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001196927 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.931610+0000", 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2506999999999998e-05 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:10.853 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:35:10.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:10.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:10.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:10.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:11.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:11.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:11.135 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:11.215 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:11.215 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:11.215 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:11.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:11.217 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:11.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:11.222 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:11.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:11.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:11.258 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:11.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:11.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:11.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:11.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 15 77355 77340 69.0475 84.9375 0.0230491 0.0144777 2024-08-30T23:35:11.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:11.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:11.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:11.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:11.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:11.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:35:11.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:11.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:11.910 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:11.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:12.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:12.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1222049867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:11 smithi136 ceph-mon[20207]: pgmap v824: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-30T23:35:12.067 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1222049867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:11 smithi173 ceph-mon[20554]: pgmap v824: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-30T23:35:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1222049867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:11 smithi106 ceph-mon[16603]: pgmap v824: 17 pgs: 17 active+clean; 4.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-30T23:35:12.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:12.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:12.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:12.284 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:12.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:35:12.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:12.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:12.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 78312 78296 68.9165 59.75 0.00999727 0.0145057 2024-08-30T23:35:12.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:12.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:12.616 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:35:12.616 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:35:12.617 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-08-30T23:35:12.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set-full-ratio .001 2024-08-30T23:35:12.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:12.644 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:12.644 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:12.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:12.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:12.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:12.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:12.917 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:12.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:13.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:13.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:13.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:13.176 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:13.254 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:13.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:13.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:13.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:35:13.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:13.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:13.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 79374 79358 68.8811 66.375 0.0149569 0.0145129 2024-08-30T23:35:13.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:13.576 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:13.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:13.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:13.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:13.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:13 smithi106 ceph-mon[16603]: osdmap e345: 12 total, 12 up, 12 in 2024-08-30T23:35:13.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:13 smithi106 ceph-mon[16603]: pgmap v826: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:35:13.712 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:13.716 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:13.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:13.820 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:13.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:13.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:13.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:13.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:14.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:14.006 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:14.007 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:13 smithi136 ceph-mon[20207]: osdmap e345: 12 total, 12 up, 12 in 2024-08-30T23:35:14.007 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:13 smithi136 ceph-mon[20207]: pgmap v826: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:35:14.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:13 smithi173 ceph-mon[20554]: osdmap e345: 12 total, 12 up, 12 in 2024-08-30T23:35:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:13 smithi173 ceph-mon[20554]: pgmap v826: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:35:14.095 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:14.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:14.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:14.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:14.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:14.410 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:14.410 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:14.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:14.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 80545 80529 68.94 73.1875 0.0239264 0.0145005 2024-08-30T23:35:14.483 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:14.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:14.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:14.656 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:35:14.661 INFO:teuthology.orchestra.run.smithi106.stderr:osd set-full-ratio 0.001 2024-08-30T23:35:14.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:14.738 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:14.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:14.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:14.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:14.992 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1424412747' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-08-30T23:35:14.992 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:14 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-08-30T23:35:14.992 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4172368880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:35:14.992 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:14 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:35:14.649+0000 7f04ccefb640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-08-30T23:35:14.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:14.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:15.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1424412747' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-08-30T23:35:15.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:14 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-08-30T23:35:15.029 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4172368880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:35:15.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:35:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1424412747' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-08-30T23:35:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:14 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.001} : dispatch 2024-08-30T23:35:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4172368880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:35:15.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:15.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:15.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:35:15.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:15.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:15.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:15.410 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:15.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:15.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 81822 81806 69.0869 79.8125 0.00652541 0.0144692 2024-08-30T23:35:15.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:15.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:15.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:15.674 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:15.714 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:15 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:15.714 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:15 smithi106 ceph-mon[16603]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-08-30T23:35:15.714 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:15 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-08-30T23:35:15.715 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4172368880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:35:15.715 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:15 smithi106 ceph-mon[16603]: osdmap e346: 12 total, 12 up, 12 in 2024-08-30T23:35:15.715 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:15 smithi106 ceph-mon[16603]: pgmap v828: 17 pgs: 17 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:35:15.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:15 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:15 smithi136 ceph-mon[20207]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-08-30T23:35:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:15 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-08-30T23:35:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4172368880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:35:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:15 smithi136 ceph-mon[20207]: osdmap e346: 12 total, 12 up, 12 in 2024-08-30T23:35:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:15 smithi136 ceph-mon[20207]: pgmap v828: 17 pgs: 17 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:35:15.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:15.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:15.930 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:15.930 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:15.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:15.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:16.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:16.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:15 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:15 smithi173 ceph-mon[20554]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-08-30T23:35:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:15 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-08-30T23:35:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4172368880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:35:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:15 smithi173 ceph-mon[20554]: osdmap e346: 12 total, 12 up, 12 in 2024-08-30T23:35:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:15 smithi173 ceph-mon[20554]: pgmap v828: 17 pgs: 17 active+clean; 4.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:35:16.156 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:16.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:16.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:16.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:16.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:16.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:16.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:16.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 83048 83032 69.1873 76.625 0.00479863 0.0144474 2024-08-30T23:35:16.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:16.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:16.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:35:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.717 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:16.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:16.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:16.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:16.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:16.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:16.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:17.022 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:16 smithi136 ceph-mon[20207]: 4.5 scrub starts 2024-08-30T23:35:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:16 smithi173 ceph-mon[20554]: 4.5 scrub starts 2024-08-30T23:35:17.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:17.093 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-08-30T23:35:17.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set-full-ratio .95 2024-08-30T23:35:17.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:16 smithi106 ceph-mon[16603]: 4.5 scrub starts 2024-08-30T23:35:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:17.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:17.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:17.300 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:17.300 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:17.329 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:17.330 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:17.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:17.353 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:17.353 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:17.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:17.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:17.381 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:17.412 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:17.412 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:17.438 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:17.439 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:17.449 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:17.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:35:17.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:17.460 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:17.466 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 84238 84222 69.2555 74.375 0.0574851 0.0144272 2024-08-30T23:35:17.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:17.489 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:17.491 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:17.492 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:17.512 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:17.512 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:17.533 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:17.534 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:17.550 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:35:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:35:17.550 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:35:17.550 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:35:17.554 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:17.554 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:17.575 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:17.575 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:17.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:17.616 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:17.673 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:17 smithi106 ceph-mon[16603]: 4.b scrub starts 2024-08-30T23:35:17.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3337924243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:17.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:17 smithi106 ceph-mon[16603]: 4.5 scrub ok 2024-08-30T23:35:17.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:17 smithi106 ceph-mon[16603]: pgmap v829: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:35:17.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:17.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:17 smithi136 ceph-mon[20207]: 4.b scrub starts 2024-08-30T23:35:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3337924243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:17 smithi136 ceph-mon[20207]: 4.5 scrub ok 2024-08-30T23:35:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:17 smithi136 ceph-mon[20207]: pgmap v829: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:35:17.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:17.874 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:17.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:17.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:18.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:17 smithi173 ceph-mon[20554]: 4.b scrub starts 2024-08-30T23:35:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3337924243' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:17 smithi173 ceph-mon[20554]: 4.5 scrub ok 2024-08-30T23:35:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:17 smithi173 ceph-mon[20554]: pgmap v829: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:35:18.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:18.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:18.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:18.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:18.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:35:18.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:18.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:18.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 85279 85263 69.201 65.0625 0.0161857 0.0144453 2024-08-30T23:35:18.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:18.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:18.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:18.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:18.723 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:18 smithi106 ceph-mon[16603]: 4.b scrub ok 2024-08-30T23:35:18.723 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:18 smithi106 ceph-mon[16603]: 4.1 scrub starts 2024-08-30T23:35:18.723 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:18 smithi106 ceph-mon[16603]: 4.c scrub starts 2024-08-30T23:35:18.723 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:18 smithi106 ceph-mon[16603]: 4.0 scrub starts 2024-08-30T23:35:18.747 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:18.770 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:18.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:18.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:18.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:18.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:18 smithi136 ceph-mon[20207]: 4.b scrub ok 2024-08-30T23:35:18.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:18 smithi136 ceph-mon[20207]: 4.1 scrub starts 2024-08-30T23:35:18.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:18 smithi136 ceph-mon[20207]: 4.c scrub starts 2024-08-30T23:35:18.952 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:18 smithi136 ceph-mon[20207]: 4.0 scrub starts 2024-08-30T23:35:18.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:18.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:18 smithi173 ceph-mon[20554]: 4.b scrub ok 2024-08-30T23:35:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:18 smithi173 ceph-mon[20554]: 4.1 scrub starts 2024-08-30T23:35:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:18 smithi173 ceph-mon[20554]: 4.c scrub starts 2024-08-30T23:35:19.011 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:18 smithi173 ceph-mon[20554]: 4.0 scrub starts 2024-08-30T23:35:19.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:19.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:19.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:19.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:19.327 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:35:19.328 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:35:19.328 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:35:19.328 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:35:19.328 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:35:19.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.361 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:19.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:19.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:19.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 86243 86227 69.0861 60.25 0.00665197 0.0144694 2024-08-30T23:35:19.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:19.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:19.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:19.676 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:35:19.676 INFO:teuthology.orchestra.run.smithi106.stderr:osd set-full-ratio 0.95 2024-08-30T23:35:19.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:19.751 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:19.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:19.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:19.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:19.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: 4.1 scrub ok 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: 4.f scrub starts 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: 4.2 scrub starts 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: 4.c scrub ok 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3675725318' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: 4.7 scrub starts 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: pgmap v830: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/887893788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:20.008 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:19 smithi106 ceph-mon[16603]: 4.d scrub starts 2024-08-30T23:35:20.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: 4.1 scrub ok 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: 4.f scrub starts 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: 4.2 scrub starts 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: 4.c scrub ok 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3675725318' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: 4.7 scrub starts 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: pgmap v830: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/887893788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:19 smithi136 ceph-mon[20207]: 4.d scrub starts 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: 4.1 scrub ok 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: 4.f scrub starts 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: 4.2 scrub starts 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: 4.c scrub ok 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3675725318' entity='client.admin' cmd={"prefix": "osd set-full-ratio", "ratio": 0.95} : dispatch 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: 4.7 scrub starts 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: pgmap v830: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/887893788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:19 smithi173 ceph-mon[20554]: 4.d scrub starts 2024-08-30T23:35:20.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:20.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:20.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:20.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:20.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:35:20.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:20.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:20.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 87301 87285 69.0486 66.125 0.00424079 0.0144771 2024-08-30T23:35:20.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:20.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:35:20.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:20.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:20.660 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:20.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:20.707 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:35:20.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:20.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:20.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:20.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:20.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: 4.0 scrub ok 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3675725318' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/887893788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: osdmap e347: 12 total, 12 up, 12 in 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: 4.f scrub ok 2024-08-30T23:35:20.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: 4.3 scrub starts 2024-08-30T23:35:20.946 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: 4.2 scrub ok 2024-08-30T23:35:20.946 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: 4.7 scrub ok 2024-08-30T23:35:20.946 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:20 smithi136 ceph-mon[20207]: osdmap e348: 12 total, 12 up, 12 in 2024-08-30T23:35:21.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:21.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: 4.0 scrub ok 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3675725318' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/887893788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: osdmap e347: 12 total, 12 up, 12 in 2024-08-30T23:35:21.049 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: 4.f scrub ok 2024-08-30T23:35:21.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: 4.3 scrub starts 2024-08-30T23:35:21.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: 4.2 scrub ok 2024-08-30T23:35:21.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: 4.7 scrub ok 2024-08-30T23:35:21.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:20 smithi106 ceph-mon[16603]: osdmap e348: 12 total, 12 up, 12 in 2024-08-30T23:35:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: 4.0 scrub ok 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3675725318' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/887893788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: osdmap e347: 12 total, 12 up, 12 in 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: 4.f scrub ok 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: 4.3 scrub starts 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: 4.2 scrub ok 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: 4.7 scrub ok 2024-08-30T23:35:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:20 smithi173 ceph-mon[20554]: osdmap e348: 12 total, 12 up, 12 in 2024-08-30T23:35:21.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:21.160 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:21.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:21.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:21.197 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:35:21.197 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:49992 2.0 2:0e70673a:::benchmark_data_smithi173_47195_object49991:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 525.94097174399997, 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20138367700000001, 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49992 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254897+0000", 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8400000000000004e-07 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254943+0000", 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5250999999999998e-05 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254950+0000", 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9349999999999994e-06 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.254955+0000", 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2400000000000001e-06 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255011+0000", 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6350000000000001e-05 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255085+0000", 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3522000000000005e-05 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.255182+0000", 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7161e-05 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256087+0000", 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090517600000000005 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.256836+0000", 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074912999999999998 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456148+0000", 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19931177 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456236+0000", 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8363999999999996e-05 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:26:35.456280+0000", 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4094000000000002e-05 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:82550 2.8 2:108ea9a8:::benchmark_data_smithi173_47195_object82549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-08-30T23:35:21.204 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:07.727800+0000", 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 493.46806880700001, 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221382149, 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82550 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727800+0000", 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727800+0000", 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727801+0000", 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0380000000000001e-06 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727863+0000", 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2016999999999999e-05 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727872+0000", 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.533e-06 2024-08-30T23:35:21.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.727879+0000", 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5899999999999996e-06 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.730036+0000", 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021568550000000001 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.730106+0000", 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0875000000000002e-05 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.730207+0000", 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100666 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.732953+0000", 2024-08-30T23:35:21.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027455240000000001 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.733003+0000", 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0546e-05 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.949112+0000", 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.216108616 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.949151+0000", 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9619999999999997e-05 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:07.949182+0000", 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0269000000000001e-05 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.211 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:97003 2.8 2:1251d5e9:::benchmark_data_smithi173_47195_object97002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.158196+0000", 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 473.03767201699998, 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36850456500000001, 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 97003 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.212 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158196+0000", 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158196+0000", 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158197+0000", 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9599999999999998e-07 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158260+0000", 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2417999999999998e-05 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158269+0000", 2024-08-30T23:35:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1749999999999994e-06 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.158274+0000", 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1839999999999998e-06 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.160501+0000", 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022267749999999998 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.160585+0000", 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3971e-05 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.160662+0000", 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6977999999999994e-05 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.166110+0000", 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054480869999999999 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.166123+0000", 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3463e-05 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526611+0000", 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36048797300000002 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526652+0000", 2024-08-30T23:35:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0923000000000001e-05 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526701+0000", 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8721999999999999e-05 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:97003 2.8 e146/97 ondisk, result = 0)", 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:35:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 473.03172365699999, 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36256268200000002, 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164145+0000", 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8799999999999998e-07 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.164147+0000", 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3429999999999998e-06 2024-08-30T23:35:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526534+0000", 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36238611900000001 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526539+0000", 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3499999999999996e-06 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526602+0000", 2024-08-30T23:35:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2779000000000003e-05 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526607+0000", 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8139999999999997e-06 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.526707+0000", 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9889000000000005e-05 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:187347 2.0 2:041180c2:::benchmark_data_smithi173_47195_object90339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:33.360063+0000", 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 347.83580546600001, 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.205202035, 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 187347 2024-08-30T23:35:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360063+0000", 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360063+0000", 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360064+0000", 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4419999999999999e-06 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360070+0000", 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360082+0000", 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2482e-05 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360089+0000", 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1589999999999998e-06 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360127+0000", 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8877000000000001e-05 2024-08-30T23:35:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360209+0000", 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1476000000000004e-05 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,11", 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.360281+0000", 2024-08-30T23:35:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2119000000000005e-05 2024-08-30T23:35:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.364622+0000", 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043406039999999996 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.364634+0000", 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2887999999999999e-05 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.565152+0000", 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20051793000000001 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.565212+0000", 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.965e-05 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:33.565265+0000", 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2818999999999997e-05 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:17943 3.f 3:f6359080:::benchmark_data_smithi173_60780_object17942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.755771+0000", 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 298.44009726100001, 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.204636807, 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17943 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755771+0000", 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755771+0000", 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755772+0000", 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-06 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755823+0000", 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0385e-05 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755840+0000", 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755848+0000", 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9389999999999997e-06 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.755886+0000", 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8501999999999998e-05 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.756004+0000", 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117446 2024-08-30T23:35:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.756137+0000", 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013379900000000001 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.756884+0000", 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074688399999999996 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757473+0000", 2024-08-30T23:35:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000589227 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960336+0000", 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.202862818 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960374+0000", 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7598999999999997e-05 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960408+0000", 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4004999999999997e-05 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:17943 3.f e214/206 ondisk, result = 0)", 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.757206+0000", 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 298.43866217200002, 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203207843, 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757206+0000", 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757206+0000", 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757207+0000", 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0500000000000002e-07 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.757214+0000", 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7059999999999998e-06 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960244+0000", 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203029869 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960251+0000", 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9500000000000004e-06 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960328+0000", 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7737000000000005e-05 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960333+0000", 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5199999999999999e-06 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.960414+0000", 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1155999999999993e-05 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:18000 3.7 e214/206 3:e954d40a:::benchmark_data_smithi173_60780_object17999:head v 214'1140, mlcod=214'1136)", 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.779491+0000", 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 298.41637735, 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20851170499999999, 2024-08-30T23:35:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779491+0000", 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779491+0000", 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779492+0000", 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9299999999999997e-07 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.779580+0000", 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7880999999999994e-05 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986314+0000", 2024-08-30T23:35:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20673482800000001 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986321+0000", 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8669999999999996e-06 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986374+0000", 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2571000000000002e-05 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.986394+0000", 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0010999999999999e-05 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987946+0000", 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001551703 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.988003+0000", 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7151000000000002e-05 2024-08-30T23:35:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:18011 3.7 e214/206 3:eb57f387:::benchmark_data_smithi173_60780_object18010:head v 214'1141, mlcod=214'1137)", 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:22.785032+0000", 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 298.41083626900001, 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20333275100000001, 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785032+0000", 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785032+0000", 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785033+0000", 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8599999999999998e-07 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.785089+0000", 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6268999999999999e-05 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987325+0000", 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.202235785 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987335+0000", 2024-08-30T23:35:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0219e-05 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987394+0000", 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9082000000000001e-05 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.987412+0000", 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.988324+0000", 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00091150499999999995 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:22.988365+0000", 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0926000000000001e-05 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:19970 3.d 3:b8325352:::benchmark_data_smithi173_60780_object19969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-08-30T23:35:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:24.073294+0000", 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 297.122574181, 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21259018599999999, 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19970 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073294+0000", 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073294+0000", 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073295+0000", 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.316e-06 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073351+0000", 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5868000000000001e-05 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073366+0000", 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4260000000000001e-05 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073373+0000", 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2710000000000003e-06 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073426+0000", 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3372000000000003e-05 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073555+0000", 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012880199999999999 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:35:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.073687+0000", 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013148200000000001 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.074547+0000", 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086078900000000004 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.075046+0000", 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00049826000000000004 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.285770+0000", 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21072489899999999 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.285820+0000", 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9518000000000003e-05 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:24.285884+0000", 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4349000000000003e-05 2024-08-30T23:35:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41039 3.f 3:fd8786e1:::benchmark_data_smithi173_60780_object41038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 278.00986503199999, 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26531318500000001, 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41039 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:35:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186005+0000", 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.815e-06 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186051+0000", 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5473999999999999e-05 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186067+0000", 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6861999999999999e-05 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186076+0000", 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8880000000000003e-06 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186133+0000", 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7101000000000001e-05 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186258+0000", 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012487800000000001 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186377+0000", 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118268 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.187721+0000", 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013439979999999999 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.188919+0000", 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001198574 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451218+0000", 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26229850199999999 2024-08-30T23:35:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451272+0000", 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4500999999999999e-05 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451316+0000", 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4323999999999998e-05 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41062 3.f 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 277.99391599199998, 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32901242400000003, 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41062 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201954+0000", 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202013+0000", 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9793999999999997e-05 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202024+0000", 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0502999999999999e-05 2024-08-30T23:35:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202030+0000", 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6210000000000001e-06 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202072+0000", 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2011999999999997e-05 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202154+0000", 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2306999999999996e-05 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202236+0000", 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2520000000000003e-05 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203586+0000", 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00134931 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.204090+0000", 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050403899999999996 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530901+0000", 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32681120000000002 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530936+0000", 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4857000000000002e-05 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530965+0000", 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8999000000000001e-05 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:41039 3.f e226/206 ondisk, result = 0)", 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 277.99199540799998, 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24745020100000001, 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.270 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203874+0000", 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699999999999997e-07 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203878+0000", 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.941e-06 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451055+0000", 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24717699200000001 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451067+0000", 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.186e-05 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451186+0000", 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118997 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451210+0000", 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4219e-05 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451323+0000", 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113375 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74450 3.d 3:b854981d:::benchmark_data_smithi173_60780_object74449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:35:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 249.22993736800001, 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29260630999999998, 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74450 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965932+0000", 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.71e-07 2024-08-30T23:35:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965987+0000", 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5448000000000002e-05 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966002+0000", 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4419999999999999e-05 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966007+0000", 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0119999999999996e-06 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967853+0000", 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001846547 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967977+0000", 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123404 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.968112+0000", 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013568200000000001 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970457+0000", 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023443460000000002 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.013531+0000", 2024-08-30T23:35:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043074440999999998 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258300+0000", 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.244768449 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258497+0000", 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000197209 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258537+0000", 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0481000000000001e-05 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:74450 3.d e243/206 ondisk, result = 0)", 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 249.22562344599999, 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28829785299999999, 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:35:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970246+0000", 2024-08-30T23:35:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970250+0000", 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0330000000000002e-06 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258240+0000", 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28799039900000001 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258244+0000", 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1309999999999999e-06 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258287+0000", 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2347e-05 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258294+0000", 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2250000000000003e-06 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258543+0000", 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000249001 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:24761 4.5 e315/306 4:a203c2b9:::benchmark_data_smithi173_74914_object24760:head v 315'1583, mlcod=315'1579)", 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.542241+0000", 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.653626944999999, 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21030616799999999, 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542241+0000", 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542241+0000", 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542242+0000", 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1399999999999997e-07 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542277+0000", 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5036000000000001e-05 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.751481+0000", 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20920387200000001 2024-08-30T23:35:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.751488+0000", 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5870000000000004e-06 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.751547+0000", 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9144999999999997e-05 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.751570+0000", 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2999000000000001e-05 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752502+0000", 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00093175099999999998 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752548+0000", 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5364000000000001e-05 2024-08-30T23:35:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(4.0 321/318 [PushReplyOp(4:0f39a0dd:::benchmark_data_smithi173_74914_object12678:head)])", 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:31.155042+0000", 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.040826590000002, 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.211697572, 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.155042+0000", 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.155042+0000", 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.155042+0000", 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:35:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.155049+0000", 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4790000000000001e-06 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.365551+0000", 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.210501836 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.365558+0000", 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.187e-06 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.365591+0000", 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3033e-05 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:31.366739+0000", 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011483470000000001 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61137 4.0 4:05184379:::benchmark_data_smithi173_74914_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.782802122, 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0987384589999998, 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:35:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61137 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413067+0000", 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2640000000000001e-06 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413084+0000", 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6407999999999999e-05 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413095+0000", 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277999999999999e-05 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413117+0000", 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1498e-05 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413168+0000", 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1266e-05 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413265+0000", 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6620000000000007e-05 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413359+0000", 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4003000000000001e-05 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415078+0000", 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719478 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415468+0000", 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038999199999999998 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511679+0000", 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0962105000000002 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511758+0000", 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9426999999999997e-05 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511805+0000", 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6724999999999997e-05 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61138 4.0 4:0601b2bd:::benchmark_data_smithi173_74914_object61137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:35:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:35:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.042147363000002, 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582538660000001, 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61138 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:35:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153722+0000", 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-06 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153826+0000", 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103683 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153839+0000", 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3169e-05 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153863+0000", 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4326000000000001e-05 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153921+0000", 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7283e-05 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154051+0000", 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012986399999999999 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154154+0000", 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103188 2024-08-30T23:35:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156034+0000", 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018801009999999999 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156113+0000", 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8807000000000006e-05 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511881+0000", 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3557679949999999 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511932+0000", 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1134999999999997e-05 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511975+0000", 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2973999999999999e-05 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:62193 4.8 e337/329 4:198a939b:::benchmark_data_smithi173_74914_object62192:head v 337'3972, mlcod=337'3968)", 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.590742967000001, 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.349666123, 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605126+0000", 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.13e-06 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605148+0000", 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1759e-05 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952851+0000", 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34770309199999999 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952856+0000", 2024-08-30T23:35:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1220000000000003e-06 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952891+0000", 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4292999999999998e-05 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952908+0000", 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7277000000000001e-05 2024-08-30T23:35:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954758+0000", 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018501889999999999 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954791+0000", 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3260999999999999e-05 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:35:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:35:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:35:21.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:21.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:21.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:21.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:35:21.468026+0000 min lat: 0.0010779 max lat: 3.89318 avg lat: 0.0145348 2024-08-30T23:35:21.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:35:21.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 88053 88037 68.7729 47 0.0204466 0.0145348 2024-08-30T23:35:21.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:21.597 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:21.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:21.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:21.746 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:21.746 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:21.777 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:21.777 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:21.809 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:21.809 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:21.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:21.843 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:21.843 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:21.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:21 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:21.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:21 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:21.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:21 smithi106 ceph-mon[16603]: 4.4 scrub starts 2024-08-30T23:35:21.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:21 smithi106 ceph-mon[16603]: 4.d scrub ok 2024-08-30T23:35:21.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:21 smithi106 ceph-mon[16603]: 4.3 scrub ok 2024-08-30T23:35:21.844 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:21 smithi106 ceph-mon[16603]: pgmap v833: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-08-30T23:35:21.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:21.878 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:21.878 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:21.911 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:21.911 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:21.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:21.945 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:21.949 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:21 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:21.949 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:21 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:21.949 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:21 smithi136 ceph-mon[20207]: 4.4 scrub starts 2024-08-30T23:35:21.949 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:21 smithi136 ceph-mon[20207]: 4.d scrub ok 2024-08-30T23:35:21.949 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:21 smithi136 ceph-mon[20207]: 4.3 scrub ok 2024-08-30T23:35:21.949 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:21 smithi136 ceph-mon[20207]: pgmap v833: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-08-30T23:35:21.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:21.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:21.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:21.980 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:21.981 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:21 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:21 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:21 smithi173 ceph-mon[20554]: 4.4 scrub starts 2024-08-30T23:35:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:21 smithi173 ceph-mon[20554]: 4.d scrub ok 2024-08-30T23:35:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:21 smithi173 ceph-mon[20554]: 4.3 scrub ok 2024-08-30T23:35:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:21 smithi173 ceph-mon[20554]: pgmap v833: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.29k op/s 2024-08-30T23:35:22.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:22.109 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:22.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:22.213 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:22.213 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:22.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:35:22.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:22.251 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:22.290 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:22.290 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:22.319 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:22.320 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:22.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:22.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:22.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:35:22.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:22.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 89188 89172 68.7995 70.9375 0.00455126 0.0145294 2024-08-30T23:35:22.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:22.554 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:22.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:22.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:22.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:22.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:22.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:22.761 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4127427935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:22.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:22.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:22.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:23.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:23.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4127427935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:23.066 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4127427935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:23.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:23.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:23.235 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:23.235 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:23.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:23.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:23.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:23.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:23.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 90180 90164 68.7165 62 0.00346429 0.0145475 2024-08-30T23:35:23.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:23.591 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:23.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:23.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:23.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.785 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:23.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:23.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:23 smithi106 ceph-mon[16603]: 4.4 scrub ok 2024-08-30T23:35:23.842 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:23 smithi106 ceph-mon[16603]: pgmap v834: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:35:23.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:23.906 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:23 smithi173 ceph-mon[20554]: 4.4 scrub ok 2024-08-30T23:35:23.906 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:23 smithi173 ceph-mon[20554]: pgmap v834: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:35:23.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:35:23.923 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:23.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:24.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:23 smithi136 ceph-mon[20207]: 4.4 scrub ok 2024-08-30T23:35:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:23 smithi136 ceph-mon[20207]: pgmap v834: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:35:24.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:24.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:24.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:24.268 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:24.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:24.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:24.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:24.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:24.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 90962 90946 68.4774 48.875 0.0081062 0.0145982 2024-08-30T23:35:24.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:24.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:24.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:24.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:24.705 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:35:24.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:24.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:24.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:24.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:24.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:24.969 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:24 smithi173 ceph-mon[20554]: 4.9 scrub starts 2024-08-30T23:35:24.970 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:24 smithi173 ceph-mon[20554]: 4.e scrub starts 2024-08-30T23:35:24.970 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1063734' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:25.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:24 smithi106 ceph-mon[16603]: 4.9 scrub starts 2024-08-30T23:35:25.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:24 smithi106 ceph-mon[16603]: 4.e scrub starts 2024-08-30T23:35:25.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1063734' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:24 smithi136 ceph-mon[20207]: 4.9 scrub starts 2024-08-30T23:35:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:24 smithi136 ceph-mon[20207]: 4.e scrub starts 2024-08-30T23:35:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1063734' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:25.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:25.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:25.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:25.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:25.266 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:25.266 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:25.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:25.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:25.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:25.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 91972 91956 68.4136 63.125 0.00407012 0.0146115 2024-08-30T23:35:25.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:25.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:35:25.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:25.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:25.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:25.731 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:25.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:25.861 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:25.936 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:25.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:25.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:25.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1063734' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: osdmap e349: 12 total, 12 up, 12 in 2024-08-30T23:35:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: pgmap v836: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:35:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:25 smithi136 ceph-mon[20207]: osdmap e350: 12 total, 12 up, 12 in 2024-08-30T23:35:26.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:26.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1063734' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:26.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: osdmap e349: 12 total, 12 up, 12 in 2024-08-30T23:35:26.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:26.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:26.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: pgmap v836: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:35:26.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:25 smithi106 ceph-mon[16603]: osdmap e350: 12 total, 12 up, 12 in 2024-08-30T23:35:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1063734' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: osdmap e349: 12 total, 12 up, 12 in 2024-08-30T23:35:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:26.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:26.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: pgmap v836: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:35:26.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:25 smithi173 ceph-mon[20554]: osdmap e350: 12 total, 12 up, 12 in 2024-08-30T23:35:26.090 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:35:26.090 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:35:26.090 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:35:26.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:35:26.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:26.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:26.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:26.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:26.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:26.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:26.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 93032 93016 68.3881 66.25 0.0135812 0.0146175 2024-08-30T23:35:26.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:26.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:26.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:26.529 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:26.530 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:26.532 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:26.566 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:26.566 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:26.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:35:26.596 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:26.596 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:26.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:26.625 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:26.625 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:26.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:26.655 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:26.655 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:26.689 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:26.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:26.718 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:26.718 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:26.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:26.747 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:26.747 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:26.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:26.773 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:26.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:26.804 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:26.805 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2793929331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:26.837 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:26.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:26.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:26.870 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:26.870 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:26.884 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2793929331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:26.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:26.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:26.931 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:27.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:27.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2793929331' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:27.120 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:27.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:27.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:27.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:27.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:27.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:27.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:27.467 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 94030 94014 68.3181 62.375 0.00574811 0.0146326 2024-08-30T23:35:27.506 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:27.506 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:35:27.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:35:27.506 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:35:27.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:14225 2.5 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 558.63038575200005, 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24031856100000001, 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 14225 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873745+0000", 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873763+0000", 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7887999999999999e-05 2024-08-30T23:35:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873773+0000", 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8700000000000004e-06 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873780+0000", 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6850000000000004e-06 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873819+0000", 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9190999999999998e-05 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873906+0000", 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7612000000000001e-05 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:35:27.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.873999+0000", 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2088000000000003e-05 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:08.874429+0000", 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043006099999999997 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.108141+0000", 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23371239399999999 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.113956+0000", 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058150340000000002 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114021+0000", 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500999999999998e-05 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:09.114063+0000", 2024-08-30T23:35:27.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2512999999999998e-05 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:17728 2.5 2:a3d96d17:::benchmark_data_smithi173_47195_object17727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 556.00020696199999, 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20912092800000001, 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.514 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 17728 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503923+0000", 2024-08-30T23:35:27.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503924+0000", 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.158e-06 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503947+0000", 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2192999999999999e-05 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503960+0000", 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3098999999999999e-05 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.503975+0000", 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5167999999999999e-05 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504020+0000", 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5464999999999999e-05 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504106+0000", 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5754000000000006e-05 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504188+0000", 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1489000000000005e-05 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.504942+0000", 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075475800000000001 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.577391+0000", 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.072448769999999996 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712945+0000", 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13555429899999999 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713005+0000", 2024-08-30T23:35:27.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9135000000000002e-05 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713044+0000", 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.964e-05 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:17728 2.5 e111/97 ondisk, result = 0)", 2024-08-30T23:35:27.520 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 555.99895143900005, 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20787409000000001, 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505179+0000", 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9100000000000004e-07 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.505185+0000", 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:35:27.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712478+0000", 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20729294100000001 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712485+0000", 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712934+0000", 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00044926100000000001 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.712941+0000", 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4999999999999996e-06 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:11.713053+0000", 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011231900000000001 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:49996 2.5 2:aad518df:::benchmark_data_smithi173_47195_object49995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 532.24626977499997, 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20351808099999999, 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 49996 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257860+0000", 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257862+0000", 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.33e-06 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257920+0000", 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8489000000000002e-05 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257934+0000", 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3372e-05 2024-08-30T23:35:27.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257941+0000", 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6459999999999993e-06 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.257996+0000", 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5043999999999997e-05 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258082+0000", 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5890000000000006e-05 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.258176+0000", 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3366e-05 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259466+0000", 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012900190000000001 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.259518+0000", 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2787999999999997e-05 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461274+0000", 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.201755238 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461333+0000", 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9394000000000001e-05 2024-08-30T23:35:27.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:35.461379+0000", 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5504999999999999e-05 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:56067 2.f 2:f6065497:::benchmark_data_smithi173_47195_object56066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 527.718333798, 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17632836900000001, 2024-08-30T23:35:27.531 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 56067 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785796+0000", 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785797+0000", 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5600000000000005e-07 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785815+0000", 2024-08-30T23:35:27.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8224e-05 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785825+0000", 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9869999999999994e-06 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785832+0000", 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.029e-06 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785866+0000", 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3729999999999997e-05 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.785937+0000", 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1027999999999998e-05 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:35:27.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:27.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.786021+0000", 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3511999999999996e-05 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787483+0000", 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014626260000000001 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.787561+0000", 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7565999999999996e-05 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962059+0000", 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17449777999999999 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962099+0000", 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9953000000000002e-05 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:39.962125+0000", 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6278000000000001e-05 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:96989 2.f 2:f906d66e:::benchmark_data_smithi173_47195_object96988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 479.36143917200002, 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84153836199999998, 2024-08-30T23:35:27.539 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 96989 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142692+0000", 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142718+0000", 2024-08-30T23:35:27.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5548999999999999e-05 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142729+0000", 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1126e-05 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142736+0000", 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1840000000000002e-06 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148635+0000", 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058993520000000001 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148767+0000", 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013146699999999999 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148916+0000", 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149083 2024-08-30T23:35:27.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152326+0000", 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034105860000000002 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152337+0000", 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1100999999999999e-05 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984109+0000", 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83177168400000001 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984180+0000", 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0542000000000003e-05 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984229+0000", 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9808000000000002e-05 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.545 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:96989 2.f e146/97 ondisk, result = 0)", 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 479.35283181300002, 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83293943299999995, 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151299+0000", 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5600000000000005e-07 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151308+0000", 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984021+0000", 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83271241100000004 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984029+0000", 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9379999999999992e-06 2024-08-30T23:35:27.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984091+0000", 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.211e-05 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984101+0000", 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0368e-05 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984238+0000", 2024-08-30T23:35:27.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013670199999999999 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100665 2.6 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 474.16031532599999, 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21497229600000001, 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.550 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100665 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343816+0000", 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.494e-06 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343824+0000", 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2570000000000001e-06 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343841+0000", 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7107000000000001e-05 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343850+0000", 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4299999999999995e-06 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343921+0000", 2024-08-30T23:35:27.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0264000000000004e-05 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344020+0000", 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9296999999999995e-05 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344100+0000", 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0465999999999998e-05 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.346504+0000", 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002403723 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347743+0000", 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001238549 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558717+0000", 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21097406299999999 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558756+0000", 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9728000000000003e-05 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558787+0000", 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0917999999999998e-05 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100667 2.6 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 474.15634884000002, 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27402247099999999, 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100667 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.557 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347782+0000", 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347787+0000", 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3039999999999998e-06 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347798+0000", 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0834e-05 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347803+0000", 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.242e-06 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347840+0000", 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7630999999999999e-05 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347918+0000", 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7778e-05 2024-08-30T23:35:27.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347989+0000", 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0988999999999994e-05 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.349336+0000", 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013472670000000001 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.351224+0000", 2024-08-30T23:35:27.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018879960000000001 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621737+0000", 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27051204000000001 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621775+0000", 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.875e-05 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621804+0000", 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8620999999999999e-05 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:127040 2.6 2:6091191f:::benchmark_data_smithi173_47195_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 439.65298341499999, 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18230701299999999, 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.563 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127040 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851148+0000", 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851155+0000", 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8990000000000004e-06 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851175+0000", 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9711000000000002e-05 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851184+0000", 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6780000000000006e-06 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851247+0000", 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2780000000000005e-05 2024-08-30T23:35:27.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851360+0000", 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112946 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851485+0000", 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012559 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.854952+0000", 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034665049999999999 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.862420+0000", 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0074685050000000003 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033372+0000", 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17095152199999999 2024-08-30T23:35:27.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033407+0000", 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5567e-05 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033454+0000", 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681000000000002e-05 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:144898 2.6 2:692f2db4:::benchmark_data_smithi173_47195_object47890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 415.82496265700001, 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18135380100000001, 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144898 2024-08-30T23:35:27.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679169+0000", 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679174+0000", 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.198e-06 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679187+0000", 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.31e-05 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679194+0000", 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679236+0000", 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1828e-05 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679316+0000", 2024-08-30T23:35:27.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0026999999999998e-05 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679383+0000", 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6925e-05 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.682228+0000", 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028449149999999999 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.829406+0000", 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14717798100000001 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860431+0000", 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031025377 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860482+0000", 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0779000000000003e-05 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860521+0000", 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9307999999999997e-05 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:d5896595:::benchmark_data_smithi173_47195_object12946:head,153'6861)", 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 343.766683102, 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59714269600000003, 2024-08-30T23:35:27.576 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737449+0000", 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737471+0000", 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2631999999999999e-05 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334475+0000", 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59700367600000004 2024-08-30T23:35:27.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334480+0000", 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2719999999999997e-06 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334536+0000", 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5430999999999998e-05 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334546+0000", 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9389999999999992e-06 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334590+0000", 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4265999999999999e-05 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.580 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:dc4db0cb:::benchmark_data_smithi173_47195_object13755:head,153'6902)", 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 342.954873953, 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85400190600000003, 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:35:27.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549257+0000", 2024-08-30T23:35:27.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549260+0000", 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.45e-06 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403003+0000", 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85374268200000003 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403016+0000", 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3033e-05 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403163+0000", 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014668400000000001 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403180+0000", 2024-08-30T23:35:27.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6813e-05 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403258+0000", 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8710999999999999e-05 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.5 e202,182 2:ac5fcd3c:::benchmark_data_smithi173_47195_object8560:head,150'6576)", 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 341.51564162599999, 2024-08-30T23:35:27.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91766887600000002, 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1099999999999996e-07 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988496+0000", 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4989999999999999e-06 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906006+0000", 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91750992200000003 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906016+0000", 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0199e-05 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906087+0000", 2024-08-30T23:35:27.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1316999999999995e-05 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906098+0000", 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0808000000000001e-05 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906158+0000", 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9620000000000002e-05 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.590 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103438 3.9 3:9093f9df:::benchmark_data_smithi173_60780_object103437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 226.22926412699999, 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38848340100000001, 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:35:27.591 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103438 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274868+0000", 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274894+0000", 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5692000000000001e-05 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274909+0000", 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5378000000000001e-05 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274918+0000", 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6489999999999997e-06 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274990+0000", 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2335000000000005e-05 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275112+0000", 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012186 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275235+0000", 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123585 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278451+0000", 2024-08-30T23:35:27.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003215954 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.279271+0000", 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081947200000000002 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663147+0000", 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38387581399999998 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663253+0000", 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106125 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663350+0000", 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6832999999999999e-05 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.597 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:103438 3.9 e255/206 ondisk, result = 0)", 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 226.22601485300001, 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38524248500000002, 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278116+0000", 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278121+0000", 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7609999999999998e-06 2024-08-30T23:35:27.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662960+0000", 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38483874699999998 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662985+0000", 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663120+0000", 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013537999999999999 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663136+0000", 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6220000000000001e-05 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663358+0000", 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022142900000000001 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.601 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:163721 3.9 3:9f4ef878:::benchmark_data_smithi173_60780_object60267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 158.85089108299999, 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.179110358, 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:35:27.602 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 163721 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653241+0000", 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.013e-06 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653250+0000", 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653272+0000", 2024-08-30T23:35:27.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2860000000000001e-05 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653281+0000", 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8289999999999997e-06 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654712+0000", 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014306620000000001 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654854+0000", 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141755 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654980+0000", 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012682000000000001 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.658949+0000", 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039685739999999999 2024-08-30T23:35:27.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.714803+0000", 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055854251000000001 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832230+0000", 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117426525 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832301+0000", 2024-08-30T23:35:27.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0921999999999996e-05 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832350+0000", 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8835000000000003e-05 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:26909 4.0 e315/306 4:05316899:::benchmark_data_smithi173_74914_object26908:head v 315'1674, mlcod=315'1671)", 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:35:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 69.307644728, 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.194211306, 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196487+0000", 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196605+0000", 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011812299999999999 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389338+0000", 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19273368799999999 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389343+0000", 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6589999999999998e-06 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389405+0000", 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1645999999999999e-05 2024-08-30T23:35:27.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389423+0000", 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8641e-05 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390666+0000", 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001242993 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390697+0000", 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0565000000000003e-05 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:49431 4.2 e327/306 4:49678b2f:::benchmark_data_smithi173_74914_object49430:head v 327'3104, mlcod=326'3101)", 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 46.962062211999999, 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.203368459, 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0900000000000002e-07 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542091+0000", 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2452000000000001e-05 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743809+0000", 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20171845999999999 2024-08-30T23:35:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743817+0000", 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0960000000000001e-06 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743892+0000", 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4893000000000003e-05 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743911+0000", 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8479999999999999e-05 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745394+0000", 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001483617 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745437+0000", 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2052000000000003e-05 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.617 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:84204 4.0 e346/329 4:023d35bd:::benchmark_data_smithi173_74914_object84203:head v 346'5351, mlcod=346'5348)", 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 10.103198395, 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20227883999999999, 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:27.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400933+0000", 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1990000000000001e-06 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400986+0000", 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3118000000000002e-05 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601032+0000", 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200045414 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601038+0000", 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4330000000000001e-06 2024-08-30T23:35:27.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601086+0000", 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7720999999999999e-05 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601104+0000", 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8267999999999999e-05 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:27.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603163+0000", 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020593569999999999 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603211+0000", 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7330000000000003e-05 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:27.622 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:27.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:27.725 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:27.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:27.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:35:27.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:27.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:27.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:28.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:28.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:27 smithi136 ceph-mon[20207]: pgmap v838: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 945 op/s 2024-08-30T23:35:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:27 smithi173 ceph-mon[20554]: pgmap v838: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 945 op/s 2024-08-30T23:35:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:27 smithi106 ceph-mon[16603]: pgmap v838: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 945 op/s 2024-08-30T23:35:28.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:28.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:28.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:28.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:28.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:28.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:35:28.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:28.468 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 95244 95228 68.4049 75.875 0.0432768 0.0146136 2024-08-30T23:35:28.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:28.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:28.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:28.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:28.741 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:28.751 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:28.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:35:28.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:28.801 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:28.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:28.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:28.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:28.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:29.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:29.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:29.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:29.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:29.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:95940 4.d 4:b9d8314f:::benchmark_data_smithi173_74914_object95939:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:29.181112+0000", 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.086978385000000005, 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.087002104999999996, 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:29.276 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 95940 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181112+0000", 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181112+0000", 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181114+0000", 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.821e-06 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:29.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181188+0000", 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4299000000000001e-05 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181205+0000", 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7303e-05 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181215+0000", 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1819999999999995e-06 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181272+0000", 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7207999999999998e-05 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181380+0000", 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108206 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,11", 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.181479+0000", 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8997000000000001e-05 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.184053+0000", 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025737049999999999 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:29.184196+0000", 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000143249 2024-08-30T23:35:29.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:29.282 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:29.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:29.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:29.282 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:35:29.282 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-08-30T23:35:29.282 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:29.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:29.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:29.372 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:29.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:29.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:29.468 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 96094 96078 68.2312 53.125 0.034498 0.0146507 2024-08-30T23:35:29.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.576 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:29.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:29.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:29.704 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:29.740 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:35:29.796 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:29.796 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:29.796 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: osdmap e351: 12 total, 12 up, 12 in 2024-08-30T23:35:29.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:29.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:29.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: pgmap v840: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s 2024-08-30T23:35:29.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4043412038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:29.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:29 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:29.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:29.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:29.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:35:29.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: osdmap e351: 12 total, 12 up, 12 in 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: pgmap v840: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4043412038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:29.944 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:29 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:29.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:29.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:29.981 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:29.981 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:29.981 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:29.981 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:29.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:29.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:35:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: osdmap e351: 12 total, 12 up, 12 in 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: pgmap v840: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 848 op/s 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4043412038' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:29 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:30.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:30.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:30.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:30.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:30.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:30.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:30.376 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:30.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:30.468 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 97260 97244 68.2833 72.875 0.00642141 0.0146403 2024-08-30T23:35:30.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:30.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:30.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:30.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:30.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:30 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:30 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:30 smithi173 ceph-mon[20554]: osdmap e352: 12 total, 12 up, 12 in 2024-08-30T23:35:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:30.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:30.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:30.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:30.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:30.983 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:35:30.983 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:35:30.984 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:35:30.984 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:31.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:30 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:30 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:30 smithi136 ceph-mon[20207]: osdmap e352: 12 total, 12 up, 12 in 2024-08-30T23:35:31.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:31.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:30 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:30 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:30 smithi106 ceph-mon[16603]: osdmap e352: 12 total, 12 up, 12 in 2024-08-30T23:35:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:31.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:31.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:31.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:31.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:31.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:31.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:31.468 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 9 98239 98230 68.2093 61.625 0.00591095 0.0146542 2024-08-30T23:35:31.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.515 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:31.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:31.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:31.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:31.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:31.763 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:31.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:31.887 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:31.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:31.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:32.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:31 smithi136 ceph-mon[20207]: osdmap e353: 12 total, 12 up, 12 in 2024-08-30T23:35:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:31 smithi136 ceph-mon[20207]: pgmap v843: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-30T23:35:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:31 smithi173 ceph-mon[20554]: osdmap e353: 12 total, 12 up, 12 in 2024-08-30T23:35:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:31 smithi173 ceph-mon[20554]: pgmap v843: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-30T23:35:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:31 smithi106 ceph-mon[16603]: osdmap e353: 12 total, 12 up, 12 in 2024-08-30T23:35:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:31 smithi106 ceph-mon[16603]: pgmap v843: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-08-30T23:35:32.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:32.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:32.191 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:32.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:35:32.193 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:32.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:32.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:32.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:32.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:32.468 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.375 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 98239 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 67.9384 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 28.1587 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 138.375 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1087 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 450.552 2024-08-30T23:35:32.469 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2214 2024-08-30T23:35:32.470 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-08-30T23:35:32.470 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0146804 2024-08-30T23:35:32.470 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0610349 2024-08-30T23:35:32.470 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 3.89318 2024-08-30T23:35:32.470 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.0010779 2024-08-30T23:35:32.474 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:35:32.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:32.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:32.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:32.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:32.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:35:32.743 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:32.744 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:32.745 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:32.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:32.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:32.786 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.787 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:32.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:32.824 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:32.828 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:32.855 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:32.856 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:32.880 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:32.880 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:32.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:32.910 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:32.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:32.916 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:32.917 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:32.938 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:32.938 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:32.958 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:32.959 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:32.989 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:32.990 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:33.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:33.022 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:33.022 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:33.022 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1111512802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:33.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:33.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1111512802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:33.066 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:33.067 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1111512802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:33.102 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:33.103 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:33.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:33.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:33.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:33.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:33.350 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:33.351 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:33.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:33.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:33.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:33.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:33.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:33.679 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:33.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:33.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:33.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:33 smithi136 ceph-mon[20207]: pgmap v844: 17 pgs: 17 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:35:34.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:33 smithi173 ceph-mon[20554]: pgmap v844: 17 pgs: 17 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:35:34.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:33 smithi106 ceph-mon[16603]: pgmap v844: 17 pgs: 17 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:35:34.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:34.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:34.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:34.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:34.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:34.462 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:34.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:35:34.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:34.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:34.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:34.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:34.812 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:34.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:34.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:34.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:34.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:34.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:35.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:35.087 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:35.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:35.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:35.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:35.321 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:35.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:35.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:35.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:35.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:35.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:35.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:35.697 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:35.698 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:35.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:35.803 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:35.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:35:35.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:35.870 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:35.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:35.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:36.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:36.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:36.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:36.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:36.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:36.353 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:36.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:36.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:36.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:35:36.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:36.528 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:36 smithi106 ceph-mon[16603]: pgmap v845: 17 pgs: 17 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 151 B/s rd, 50 MiB/s wr, 941 op/s 2024-08-30T23:35:36.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:36.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:36.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:36.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:36 smithi136 ceph-mon[20207]: pgmap v845: 17 pgs: 17 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 151 B/s rd, 50 MiB/s wr, 941 op/s 2024-08-30T23:35:36.790 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:36.811 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:36 smithi173 ceph-mon[20554]: pgmap v845: 17 pgs: 17 active+clean; 5.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 151 B/s rd, 50 MiB/s wr, 941 op/s 2024-08-30T23:35:36.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:36.924 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:36.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:36.977 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:36.977 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:36.977 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:36.977 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:36.978 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:37.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:37.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:37.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:37.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:37.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:37.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:35:37.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:37.362 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:37.362 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:37.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:37.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:37.528 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:35:37.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:37.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3997361047' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:35:37.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:37.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:37.767 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:37.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:35:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3997361047' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:35:37.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:37.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3997361047' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:35:37.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:37.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:37.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:38.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:38.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:38.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:38.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:35:38.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:38.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:38.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:38.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:38.468 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:38.517 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:38.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:38.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:38 smithi106 ceph-mon[16603]: pgmap v846: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 915 op/s 2024-08-30T23:35:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:38 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3997361047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:35:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:38 smithi106 ceph-mon[16603]: osdmap e354: 12 total, 12 up, 12 in 2024-08-30T23:35:38.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:38.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:38 smithi136 ceph-mon[20207]: pgmap v846: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 915 op/s 2024-08-30T23:35:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:38 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3997361047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:35:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:38 smithi136 ceph-mon[20207]: osdmap e354: 12 total, 12 up, 12 in 2024-08-30T23:35:38.799 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:38 smithi173 ceph-mon[20554]: pgmap v846: 17 pgs: 17 active+clean; 5.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 915 op/s 2024-08-30T23:35:38.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:38 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:38.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3997361047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:35:38.800 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:38 smithi173 ceph-mon[20554]: osdmap e354: 12 total, 12 up, 12 in 2024-08-30T23:35:38.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:38.831 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:38.854 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:38.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:38.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:38.991 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:39.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:39.099 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:39.099 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:39.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:39.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:39.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:39.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:39.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:39.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:39.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1645110862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:39 smithi106 ceph-mon[16603]: pgmap v848: 17 pgs: 17 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 25 MiB/s wr, 873 op/s 2024-08-30T23:35:39.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:39.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:39.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1645110862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:39.721 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:39 smithi173 ceph-mon[20554]: pgmap v848: 17 pgs: 17 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 25 MiB/s wr, 873 op/s 2024-08-30T23:35:39.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:39.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:35:39.778 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:39.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1645110862' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:39 smithi136 ceph-mon[20207]: pgmap v848: 17 pgs: 17 active+clean; 5.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 25 MiB/s wr, 873 op/s 2024-08-30T23:35:39.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:39.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:39.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:40.004 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:40.004 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:40.004 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:40.004 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:40.004 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:40.005 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:40.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:40.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:40.118 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:40.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:40.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:40.278 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:40.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:40.307 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:40.307 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:40.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:40.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:40.365 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:40.394 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:40.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:40.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:40.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:40.447 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:40.452 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:40.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:40.496 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:40.496 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:40.522 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:40.522 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:40.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:40.550 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:40.550 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:40.581 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:40.581 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:40.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:40.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.24460.0:16911 2.4 e111/97 2:26b3b856:::benchmark_data_smithi173_47195_object16910:head v 111'1012, mlcod=111'1009)", 2024-08-30T23:35:40.699 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 570.02246198900002, 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225497685, 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672855+0000", 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672856+0000", 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0200000000000003e-07 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.672918+0000", 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1801e-05 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897644+0000", 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22472613699999999 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897651+0000", 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.99e-06 2024-08-30T23:35:40.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897700+0000", 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9373000000000003e-05 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.897721+0000", 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1314e-05 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.898318+0000", 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00059680200000000001 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:10.898353+0000", 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4866000000000001e-05 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:164675 3.1 3:8010289e:::benchmark_data_smithi173_60780_object61221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:35:40.704 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 170.72557819400001, 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27192025399999997, 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 164675 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969741+0000", 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0329999999999998e-06 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969748+0000", 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.241e-06 2024-08-30T23:35:40.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969761+0000", 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3162999999999999e-05 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969768+0000", 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5429999999999996e-06 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.970901+0000", 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011326870000000001 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.971033+0000", 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013183100000000001 2024-08-30T23:35:40.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047341+0000", 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076308670999999995 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047456+0000", 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114108 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-08-30T23:35:40.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047620+0000", 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164504 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.061526+0000", 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013906091000000001 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.065492+0000", 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039661649999999998 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.240280+0000", 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17478812399999999 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.240349+0000", 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8520000000000001e-05 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.241660+0000", 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001310573 2024-08-30T23:35:40.712 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e300,280 3:83685fcd:::benchmark_data_smithi173_60780_object2411:head,257'6623)", 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 116.293154104, 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41143703700000001, 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402164+0000", 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402168+0000", 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0570000000000003e-06 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812764+0000", 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41059553700000001 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812774+0000", 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529e-05 2024-08-30T23:35:40.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812860+0000", 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5934999999999996e-05 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.813600+0000", 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074028899999999999 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e301,280 3:8490e9ac:::benchmark_data_smithi173_60780_object7986:head,258'6976)", 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 113.516339617, 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70969468099999999, 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4199999999999995e-07 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178982+0000", 2024-08-30T23:35:40.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9779999999999999e-06 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887644+0000", 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70866116300000004 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887661+0000", 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6954999999999999e-05 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887778+0000", 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117812 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.888673+0000", 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00089413100000000005 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:24732 4.a e315/306 4:5449864d:::benchmark_data_smithi173_74914_object24731:head v 315'1539, mlcod=315'1535)", 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:35:40.720 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 84.185992627999994, 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24331156100000001, 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509326+0000", 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509403+0000", 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7809e-05 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751493+0000", 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24208919500000001 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751500+0000", 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5190000000000003e-06 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751549+0000", 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8891000000000003e-05 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751566+0000", 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7102999999999999e-05 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.752577+0000", 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001011248 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.752636+0000", 2024-08-30T23:35:40.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9079000000000001e-05 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31037 4.5 4.306472d5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 80.012904512000006, 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0020195190000001, 2024-08-30T23:35:40.725 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31037 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682415+0000", 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6589999999999999e-06 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682488+0000", 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3084999999999996e-05 2024-08-30T23:35:40.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682498+0000", 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0644e-05 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682506+0000", 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0229999999999992e-06 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682702+0000", 2024-08-30T23:35:40.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019531299999999999 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682713+0000", 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1778e-05 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684400+0000", 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016865800000001 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.729 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684432+0000", 2024-08-30T23:35:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2437000000000001e-05 2024-08-30T23:35:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.730 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31040 4.5 4.eb15935 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 80.012746925000002, 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.001902058, 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.731 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31040 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682627+0000", 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5911000000000001e-05 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682637+0000", 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0363999999999999e-05 2024-08-30T23:35:40.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682642+0000", 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8509999999999998e-06 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682765+0000", 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123311 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682799+0000", 2024-08-30T23:35:40.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3145000000000002e-05 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684465+0000", 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016660740000001 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684473+0000", 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9149999999999996e-06 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.735 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31060 4.5 4.c6dc7475 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 80.003341700999997, 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99255960700000001, 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31060 2024-08-30T23:35:40.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691977+0000", 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.066e-06 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691998+0000", 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.158e-05 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692011+0000", 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2848e-05 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692019+0000", 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8990000000000001e-06 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692069+0000", 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0080999999999997e-05 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692076+0000", 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1450000000000002e-06 2024-08-30T23:35:40.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684524+0000", 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99244810000000006 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684535+0000", 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0888e-05 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31063 4.5 4.5c985545 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 79.558835654000006, 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54808229100000005, 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31063 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.741 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136484+0000", 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7069999999999999e-06 2024-08-30T23:35:40.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136509+0000", 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4990999999999998e-05 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136526+0000", 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7178999999999999e-05 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136533+0000", 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8789999999999997e-06 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136588+0000", 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5018000000000001e-05 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136596+0000", 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9820000000000008e-06 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684558+0000", 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54796194799999998 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684564+0000", 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5869999999999998e-06 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.745 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:44161 4.7 4:e6ac4d14:::benchmark_data_smithi173_74914_object44160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:35.888114+0000", 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 64.807203920000006, 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22262327000000001, 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 44161 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888114+0000", 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888114+0000", 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888115+0000", 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.528e-06 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888135+0000", 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0152e-05 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888158+0000", 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2388999999999999e-05 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888166+0000", 2024-08-30T23:35:40.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2169999999999994e-06 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888214+0000", 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7944e-05 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888349+0000", 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013541099999999999 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:35:40.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888505+0000", 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015614900000000001 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.890107+0000", 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001601568 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.905284+0000", 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015176766 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:36.110110+0000", 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20482651499999999 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:36.110148+0000", 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7489e-05 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:36.110737+0000", 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058914199999999996 2024-08-30T23:35:40.751 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:50603 4.a e329/318 4:54bdef61:::benchmark_data_smithi173_74914_object50602:head v 329'3160, mlcod=315'1602)", 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 59.033989869999999, 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25119703700000001, 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.752 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:35:40.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661786+0000", 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045819399999999998 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.910994+0000", 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.249208242 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.910999+0000", 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8640000000000004e-06 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.911049+0000", 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0216000000000002e-05 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.911074+0000", 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4484000000000001e-05 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.912487+0000", 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001413648 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.912525+0000", 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7171e-05 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(4.4 329/318 [PushOp(4:2f097f2f:::benchmark_data_smithi173_74914_object29004:head, version: 316'1834, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:2f097f2f:::benchmark_data_smithi173_74914_object29004:head@316'1834, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 58.627730001000003, 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51130092400000005, 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.758 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:40.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:40.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067588+0000", 2024-08-30T23:35:40.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6199999999999997e-07 2024-08-30T23:35:40.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067620+0000", 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2128999999999997e-05 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577769+0000", 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.510148398 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577780+0000", 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1617000000000001e-05 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577885+0000", 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104666 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577899+0000", 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4568e-05 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.578888+0000", 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098888399999999999 2024-08-30T23:35:40.762 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.5 335/318 [PushReplyOp(4:a5afb7d9:::benchmark_data_smithi173_74914_object29619:head)])", 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 44.436533093999998, 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.227084861, 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258785+0000", 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9500000000000003e-07 2024-08-30T23:35:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258788+0000", 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2160000000000002e-06 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485597+0000", 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.226809118 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.765 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485606+0000", 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1260000000000004e-06 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485684+0000", 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7238999999999997e-05 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485869+0000", 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018566700000000001 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.766 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62091 4.4 e337/331 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head v 337'3874, mlcod=336'3872)", 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.529063430000001, 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.317048422, 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166315+0000", 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0946000000000002e-05 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481856+0000", 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31554066800000002 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481863+0000", 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0960000000000003e-06 2024-08-30T23:35:40.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481922+0000", 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8789000000000002e-05 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481941+0000", 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9307999999999999e-05 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.483268+0000", 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013262339999999999 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.483302+0000", 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4947999999999998e-05 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62105 4.4 e337/331 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head v 337'3875, mlcod=336'3872)", 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.515477214999997, 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34727296499999999, 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179841+0000", 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.68e-07 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179895+0000", 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4670999999999999e-05 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525681+0000", 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34578567199999999 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525688+0000", 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4069999999999998e-06 2024-08-30T23:35:40.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525744+0000", 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6588000000000001e-05 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525761+0000", 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6642000000000001e-05 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.527080+0000", 2024-08-30T23:35:40.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013189460000000001 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.527113+0000", 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3470999999999999e-05 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62174 4.4 e337/331 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head v 337'3877, mlcod=336'3873)", 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:35:40.776 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.169712259999997, 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57728454699999998, 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:35:40.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525606+0000", 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1429999999999999e-06 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525661+0000", 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4715e-05 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101380+0000", 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57571867700000001 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101387+0000", 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.746e-06 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101441+0000", 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3916999999999999e-05 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101460+0000", 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8185999999999998e-05 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.102857+0000", 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013971820000000001 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.102890+0000", 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2981000000000002e-05 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62186 4.4 e337/331 4:257aab52:::benchmark_data_smithi173_74914_object62185:head v 337'3878, mlcod=337'3874)", 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:35:40.781 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 43.141087294000002, 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55065025599999995, 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:35:40.782 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554231+0000", 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999995e-07 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554289+0000", 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8440000000000003e-05 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.783 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103923+0000", 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54963335000000002 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103927+0000", 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7199999999999997e-06 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103985+0000", 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7398999999999997e-05 2024-08-30T23:35:40.784 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103997+0000", 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2617e-05 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.104853+0000", 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085557500000000002 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.104880+0000", 2024-08-30T23:35:40.785 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7444e-05 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98221 4.5 4:a4ee3b1f:::benchmark_data_smithi173_74914_object98220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.2581270320000009, 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225938741, 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.786 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98221 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.787 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437192+0000", 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.598e-06 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437212+0000", 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0387999999999999e-05 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.788 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437225+0000", 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2393e-05 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437230+0000", 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5319999999999997e-06 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437273+0000", 2024-08-30T23:35:40.789 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2778999999999998e-05 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437360+0000", 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6718000000000006e-05 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437446+0000", 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6215e-05 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.790 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.439899+0000", 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024530929999999999 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447609+0000", 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.007709624 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.662941+0000", 2024-08-30T23:35:40.791 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21533202500000001 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.663056+0000", 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115085 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.663129+0000", 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3291e-05 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.792 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98225 4.5 4:ad425262:::benchmark_data_smithi173_74914_object98224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.2508077209999993, 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32404802500000002, 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.793 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98225 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.794 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0200000000000001e-07 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444529+0000", 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8377999999999999e-05 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.795 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444538+0000", 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6719999999999999e-06 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444543+0000", 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9119999999999997e-06 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444596+0000", 2024-08-30T23:35:40.796 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2286e-05 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444722+0000", 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126229 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444864+0000", 2024-08-30T23:35:40.797 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014163000000000001 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447348+0000", 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024847910000000001 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.450736+0000", 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033874059999999999 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.798 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768431+0000", 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31769526799999998 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768502+0000", 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0516e-05 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768558+0000", 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6234999999999999e-05 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.799 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98228 4.7 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.2476843730000002, 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35701674100000003, 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:40.800 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98228 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.801 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447634+0000", 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.035e-06 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447655+0000", 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0735000000000001e-05 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447665+0000", 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0353999999999999e-05 2024-08-30T23:35:40.802 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447679+0000", 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3804e-05 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447729+0000", 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9570000000000001e-05 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447855+0000", 2024-08-30T23:35:40.803 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126453 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.448003+0000", 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014797200000000001 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.450323+0000", 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002320192 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.804 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.604622+0000", 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15429874599999999 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804484+0000", 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.199861699 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804600+0000", 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011622500000000001 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:40.805 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804650+0000", 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9956000000000001e-05 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:40.806 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:40.806 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:35:40.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:40.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:40.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:40.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:41.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:41.046 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:41.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-30T23:35:41.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:41.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:41.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:41.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:41.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:41.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:41.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:41.524 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:41.555 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:41.572 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:41.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:41.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:41.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:35:41.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:41.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:41.836 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:41 smithi106 ceph-mon[16603]: 4.8 scrub starts 2024-08-30T23:35:41.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:41 smithi106 ceph-mon[16603]: pgmap v849: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 14 MiB/s wr, 925 op/s 2024-08-30T23:35:41.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:41.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:41.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:41.997 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:42.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:42.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:41 smithi136 ceph-mon[20207]: 4.8 scrub starts 2024-08-30T23:35:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:41 smithi136 ceph-mon[20207]: pgmap v849: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 14 MiB/s wr, 925 op/s 2024-08-30T23:35:42.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:41 smithi173 ceph-mon[20554]: 4.8 scrub starts 2024-08-30T23:35:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:41 smithi173 ceph-mon[20554]: pgmap v849: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 14 MiB/s wr, 925 op/s 2024-08-30T23:35:42.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:42.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:42.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:42.291 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:42.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:42.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:42.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:42.539 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:35:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:35:42.539 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:42.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:42.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:42.685 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:42.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:42.689 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:42.690 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:42.690 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:42.690 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:42.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:42.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:42 smithi173 ceph-mon[20554]: 4.9 scrub starts 2024-08-30T23:35:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:42 smithi173 ceph-mon[20554]: 4.8 scrub ok 2024-08-30T23:35:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:42 smithi106 ceph-mon[16603]: 4.9 scrub starts 2024-08-30T23:35:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:42 smithi106 ceph-mon[16603]: 4.8 scrub ok 2024-08-30T23:35:42.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:42.942 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:42.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:42 smithi136 ceph-mon[20207]: 4.9 scrub starts 2024-08-30T23:35:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:42 smithi136 ceph-mon[20207]: 4.8 scrub ok 2024-08-30T23:35:43.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:43.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:43.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:43.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:43.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:43.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:43.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:35:43.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:43.469 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:43.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-30T23:35:43.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:43.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:35:43.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:43.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:43.656 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:35:43.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:43.741 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:43.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:43.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/635732889' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:43 smithi106 ceph-mon[16603]: 4.9 scrub ok 2024-08-30T23:35:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:43 smithi106 ceph-mon[16603]: pgmap v850: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.3 MiB/s wr, 956 op/s 2024-08-30T23:35:43.872 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:43.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:43.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:43.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:44.000 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/635732889' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:44.000 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:43 smithi173 ceph-mon[20554]: 4.9 scrub ok 2024-08-30T23:35:44.000 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:43 smithi173 ceph-mon[20554]: pgmap v850: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.3 MiB/s wr, 956 op/s 2024-08-30T23:35:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/635732889' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:43 smithi136 ceph-mon[20207]: 4.9 scrub ok 2024-08-30T23:35:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:43 smithi136 ceph-mon[20207]: pgmap v850: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 5.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 4.3 MiB/s wr, 956 op/s 2024-08-30T23:35:44.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:44.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:44.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:44.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:44.286 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:44.286 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:35:44.286 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:35:44.286 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:35:44.286 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:44.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:44.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:44.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:44.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:44.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.644 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:44.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:44.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:44 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:35:44.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:44 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:35:44.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/635732889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:44.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:44 smithi106 ceph-mon[16603]: osdmap e355: 12 total, 12 up, 12 in 2024-08-30T23:35:44.745 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:44.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:44.783 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:44.827 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:44.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:44.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:44.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:44.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:44 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:35:44.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:44 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:35:44.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/635732889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:44.916 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:44 smithi173 ceph-mon[20554]: osdmap e355: 12 total, 12 up, 12 in 2024-08-30T23:35:44.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:35:45.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:44 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:35:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:44 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:35:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/635732889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:44 smithi136 ceph-mon[20207]: osdmap e355: 12 total, 12 up, 12 in 2024-08-30T23:35:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:45.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:45.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:45.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:45.260 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:35:45.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:45.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:45.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:45.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:45.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:45.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:45.586 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:45.587 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:45.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:45.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:45.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:45.832 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:45.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:62896 2.a 2:5bc7d7b5:::benchmark_data_smithi173_47195_object62895:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 539.924496417, 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.272135707, 2024-08-30T23:35:45.838 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62896 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911874+0000", 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911875+0000", 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.099e-06 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911895+0000", 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9724999999999998e-05 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911907+0000", 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1423000000000001e-05 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911912+0000", 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5160000000000002e-06 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.911972+0000", 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9698999999999999e-05 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912062+0000", 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0204999999999994e-05 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.912153+0000", 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0696e-05 2024-08-30T23:35:45.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.913669+0000", 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001516807 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:45.914495+0000", 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082544200000000002 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183885+0000", 2024-08-30T23:35:45.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26939059300000001 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.183937+0000", 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1586999999999999e-05 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:46.184010+0000", 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2915000000000002e-05 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.844 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:75402 2.c e132/97 ondisk, result = 0)", 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 529.21337027300001, 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.335240237, 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:35:45.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623000+0000", 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623001+0000", 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2099999999999996e-07 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.623007+0000", 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7710000000000003e-06 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.957925+0000", 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33491853599999999 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.957938+0000", 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3142999999999999e-05 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958067+0000", 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012853699999999999 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958082+0000", 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4620000000000001e-05 2024-08-30T23:35:45.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:26:56.958241+0000", 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015900899999999999 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:110499 2.b 2:d409eb3a:::benchmark_data_smithi173_47195_object13491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 479.76156121700001, 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25912935999999998, 2024-08-30T23:35:45.849 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 110499 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074810+0000", 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7499999999999998e-07 2024-08-30T23:35:45.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074815+0000", 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5700000000000003e-06 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074826+0000", 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.101e-05 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074832+0000", 2024-08-30T23:35:45.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.643e-06 2024-08-30T23:35:45.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074884+0000", 2024-08-30T23:35:45.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2614999999999997e-05 2024-08-30T23:35:45.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074986+0000", 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101808 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.075067+0000", 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0661000000000006e-05 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.077720+0000", 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026533749999999999 2024-08-30T23:35:45.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.078416+0000", 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069541700000000004 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333266+0000", 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25485053000000002 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333335+0000", 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8868999999999994e-05 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333939+0000", 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000603887 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:41062 3.f e226/206 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head v 226'2525, mlcod=226'2521)", 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:35:45.857 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 302.63403890000001, 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32836641100000002, 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:35:45.858 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202340+0000", 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9529999999999999e-06 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202390+0000", 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0473999999999998e-05 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.859 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529209+0000", 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32681829099999998 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529215+0000", 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8379999999999996e-06 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529257+0000", 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2036999999999997e-05 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529272+0000", 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4331999999999999e-05 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530655+0000", 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001382961 2024-08-30T23:35:45.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530698+0000", 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3525000000000001e-05 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:199193 3.0 3:0da882ce:::benchmark_data_smithi173_60780_object95739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.478480311, 2024-08-30T23:35:45.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30993041100000002, 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199193 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:35:45.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357892+0000", 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.189e-06 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357896+0000", 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5090000000000004e-06 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357907+0000", 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0607e-05 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357912+0000", 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2190000000000003e-06 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357956+0000", 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3745000000000002e-05 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.358054+0000", 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7940000000000001e-05 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.358129+0000", 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5887e-05 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362642+0000", 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045129510000000003 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.364032+0000", 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00138984 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667719+0000", 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30368694699999998 2024-08-30T23:35:45.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667778+0000", 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8754e-05 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667821+0000", 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2823e-05 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.869 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:199193 3.0 e299/206 ondisk, result = 0)", 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.47401758199999, 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30548378700000001, 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362354+0000", 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1060000000000001e-06 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362361+0000", 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4230000000000002e-06 2024-08-30T23:35:45.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667470+0000", 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30510947599999999 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667476+0000", 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8379999999999998e-06 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.873 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667704+0000", 2024-08-30T23:35:45.873 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022796099999999999 2024-08-30T23:35:45.873 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.873 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.874 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667712+0000", 2024-08-30T23:35:45.874 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4749999999999993e-06 2024-08-30T23:35:45.874 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.874 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.874 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667837+0000", 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012450800000000001 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:199216 3.0 3:0986db6c:::benchmark_data_smithi173_60780_object95762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.437411844, 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28454402699999998, 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.875 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199216 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398960+0000", 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0100000000000001e-06 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398972+0000", 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2381000000000001e-05 2024-08-30T23:35:45.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398982+0000", 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0236e-05 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398988+0000", 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5940000000000001e-06 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399037+0000", 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8613999999999999e-05 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399169+0000", 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132657 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399297+0000", 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127732 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.403718+0000", 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044206580000000001 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.403817+0000", 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9599999999999995e-05 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683416+0000", 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27959865299999997 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683467+0000", 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1248999999999997e-05 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683503+0000", 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5642999999999997e-05 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:199216 3.0 e299/206 ondisk, result = 0)", 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.43220349800001, 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27933989799999998, 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404168+0000", 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7700000000000004e-07 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404173+0000", 2024-08-30T23:35:45.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8230000000000002e-06 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.679627+0000", 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27545433000000002 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.679635+0000", 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9200000000000004e-06 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683406+0000", 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037705299999999998 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683412+0000", 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5640000000000002e-06 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683507+0000", 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4953999999999999e-05 2024-08-30T23:35:45.885 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:206901 3.7 e300/206 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head v 300'13076, mlcod=300'13071)", 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 121.772225862, 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38146454200000002, 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064146+0000", 2024-08-30T23:35:45.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8000000000000005e-07 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064151+0000", 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8270000000000003e-06 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441558+0000", 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37740696499999998 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441573+0000", 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4623e-05 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441672+0000", 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9351999999999993e-05 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441691+0000", 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8134999999999998e-05 2024-08-30T23:35:45.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445570+0000", 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003879511 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445609+0000", 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9348999999999999e-05 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87cb8fd9:::benchmark_data_smithi173_60780_object16345:head,262'7462)", 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 113.890119715, 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72540086000000004, 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946252+0000", 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3399999999999999e-07 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946257+0000", 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349999999999999e-06 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671354+0000", 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72509723199999998 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671379+0000", 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5279e-05 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671555+0000", 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000175203 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671572+0000", 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.774e-05 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671652+0000", 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9536999999999994e-05 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.894 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87ea66dc:::benchmark_data_smithi173_60780_object18546:head,263'7595)", 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 111.95377132500001, 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85628250299999997, 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882600+0000", 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2024-08-30T23:35:45.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882604+0000", 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2289999999999997e-06 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738700+0000", 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85609620099999995 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738719+0000", 2024-08-30T23:35:45.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8343e-05 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738882+0000", 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016310000000000001 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31056 4.8 4.b8866df8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 85.150317295999997, 2024-08-30T23:35:45.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99756266500000002, 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31056 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686055+0000", 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6950000000000001e-06 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.901 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:45 smithi106 ceph-mon[16603]: 4.e scrub starts 2024-08-30T23:35:45.901 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:45 smithi106 ceph-mon[16603]: pgmap v852: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686078+0000", 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3113e-05 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686093+0000", 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4856e-05 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686100+0000", 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3930000000000002e-06 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686163+0000", 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2131000000000005e-05 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686172+0000", 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9849999999999995e-06 2024-08-30T23:35:45.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683581+0000", 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99740900499999996 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683616+0000", 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5487000000000001e-05 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31071 4.8 4.35742af8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 84.540659202000001, 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38794466599999999, 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31071 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.905 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:35:45.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295712+0000", 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1299999999999998e-07 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295733+0000", 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0463e-05 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295744+0000", 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1534e-05 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295750+0000", 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0599999999999996e-06 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295804+0000", 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3983000000000001e-05 2024-08-30T23:35:45.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295812+0000", 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4290000000000004e-06 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683649+0000", 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38783691999999997 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683656+0000", 2024-08-30T23:35:45.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3640000000000001e-06 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62151 4.1 4:8b395779:::benchmark_data_smithi173_74914_object62150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.358063887999997, 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38737673, 2024-08-30T23:35:45.910 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62151 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478309+0000", 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478334+0000", 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5167000000000001e-05 2024-08-30T23:35:45.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478351+0000", 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7391e-05 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478360+0000", 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2579999999999995e-06 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478413+0000", 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3780999999999999e-05 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478541+0000", 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127358 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478645+0000", 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001039 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480555+0000", 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019108790000000001 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480787+0000", 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023142799999999999 2024-08-30T23:35:45.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865599+0000", 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38481200700000001 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865636+0000", 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7416999999999998e-05 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865684+0000", 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7215000000000001e-05 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:62151 4.1 e337/306 ondisk, result = 0)", 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.355650726, 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38498058299999999, 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.917 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480721+0000", 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480728+0000", 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7719999999999994e-06 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865502+0000", 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38477343200000003 2024-08-30T23:35:45.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865508+0000", 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7300000000000002e-06 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865589+0000", 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.119e-05 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865595+0000", 2024-08-30T23:35:45.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4529999999999999e-06 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865701+0000", 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105266 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62151 4.1 4:8b395779:::benchmark_data_smithi173_74914_object62150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.333338980000001, 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.362616665, 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for ondisk", 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62151 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:35:45.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2399999999999998e-07 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503054+0000", 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1647000000000001e-05 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503061+0000", 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503066+0000", 2024-08-30T23:35:45.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6750000000000001e-06 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503115+0000", 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8968999999999998e-05 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for ondisk", 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503174+0000", 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8739000000000001e-05 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865648+0000", 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36247479799999999 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62193 4.8 4:198a939b:::benchmark_data_smithi173_74914_object62192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.604484+0000", 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 48.231886598000003, 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35078139200000003, 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.926 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62193 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604484+0000", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604484+0000", 2024-08-30T23:35:45.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604486+0000", 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9319999999999999e-06 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604509+0000", 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2966e-05 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604524+0000", 2024-08-30T23:35:45.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-05 2024-08-30T23:35:45.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604531+0000", 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.277e-06 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604592+0000", 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0331000000000002e-05 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604730+0000", 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000138897 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604887+0000", 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015684499999999999 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.606731+0000", 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018433290000000001 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.848856+0000", 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24212561399999999 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.955132+0000", 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10627551 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.955238+0000", 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010603900000000001 2024-08-30T23:35:45.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.955265+0000", 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7722e-05 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98220 4.1 4:8d0adb9e:::benchmark_data_smithi173_74914_object98219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.433687+0000", 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.402683822, 2024-08-30T23:35:45.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26153179399999998, 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98220 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433687+0000", 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433687+0000", 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433689+0000", 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1270000000000001e-06 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433711+0000", 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2305000000000001e-05 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433727+0000", 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5455e-05 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433734+0000", 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6320000000000008e-06 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433787+0000", 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349999999999999e-05 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433893+0000", 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106539 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433996+0000", 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102798 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.436497+0000", 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025010290000000001 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437108+0000", 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061081500000000001 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695097+0000", 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25798935299999998 2024-08-30T23:35:45.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695188+0000", 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0463000000000004e-05 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695219+0000", 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0927999999999999e-05 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98222 4.1 4:8f28ee96:::benchmark_data_smithi173_74914_object98221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.440408+0000", 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.395962927999999, 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.368547611, 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98222 2024-08-30T23:35:45.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440408+0000", 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440408+0000", 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440409+0000", 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440451+0000", 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2049999999999999e-05 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440462+0000", 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0725e-05 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440467+0000", 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9109999999999998e-06 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440513+0000", 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5627000000000002e-05 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440603+0000", 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0085000000000002e-05 2024-08-30T23:35:45.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440715+0000", 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011188600000000001 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443186+0000", 2024-08-30T23:35:45.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00247117 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443662+0000", 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047565599999999997 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.808739+0000", 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36507735699999999 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:35:45.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.808908+0000", 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000168581 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.808955+0000", 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.757e-05 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.948 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:98220 4.1 e353/306 ondisk, result = 0)", 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.443385+0000", 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.392985222, 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25184049600000002, 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443385+0000", 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443385+0000", 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443386+0000", 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0299999999999999e-07 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443394+0000", 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1529999999999997e-06 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.694900+0000", 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.251505441 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.694917+0000", 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7257000000000001e-05 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695073+0000", 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015597699999999999 2024-08-30T23:35:45.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695088+0000", 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4806e-05 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695226+0000", 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013825900000000001 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.952 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:35:45.953 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:35:45.953 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:35:45.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:45.981 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:45.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:45 smithi136 ceph-mon[20207]: 4.e scrub starts 2024-08-30T23:35:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:45 smithi136 ceph-mon[20207]: pgmap v852: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-08-30T23:35:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:45 smithi173 ceph-mon[20554]: 4.e scrub starts 2024-08-30T23:35:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:45 smithi173 ceph-mon[20554]: pgmap v852: 17 pgs: 3 active+clean+scrubbing, 14 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s 2024-08-30T23:35:46.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:46.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:46.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:46.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:46.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:46.366 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:46.451 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:46.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-30T23:35:46.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:46.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:46.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:46.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:46.647 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:46.647 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:46.675 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:46.675 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:46.701 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:46.702 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:46.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:46.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:46.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:46.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:46.739 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:46.759 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:46.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:46.785 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:46.785 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:46.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:35:46.814 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:46.814 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:46.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:46.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:35:46.845 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:46.846 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:46.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:46.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:46.879 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:46.879 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:46.898 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/110671598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:46.898 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:46 smithi106 ceph-mon[16603]: 4.e scrub ok 2024-08-30T23:35:46.913 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:46.914 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:46.950 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:46.950 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:46.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:46.984 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:46.984 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:47.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/110671598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:46 smithi136 ceph-mon[20207]: 4.e scrub ok 2024-08-30T23:35:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/110671598' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:47.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:46 smithi173 ceph-mon[20554]: 4.e scrub ok 2024-08-30T23:35:47.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:47.144 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:47.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:47.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:47.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:47.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:47.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:47.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:47.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:47.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:47.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:47.782 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:47 smithi106 ceph-mon[16603]: pgmap v853: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:35:47.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:47.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:47.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:48.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:48.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:48.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:47 smithi136 ceph-mon[20207]: pgmap v853: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:35:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:47 smithi173 ceph-mon[20554]: pgmap v853: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:35:48.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:48.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:48.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:48.279 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:48.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:48.303 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:48.304 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:48.304 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:48.304 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:48.304 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:48.304 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:48.304 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:48.305 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:48.305 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:48.305 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:48.305 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:48.305 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:48.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:48.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:48.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:48.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:48.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:48.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:48.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:48.798 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:48.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:48.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:48.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:48.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:49.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:49.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:49.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:49.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:49.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:49.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:49.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:49.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:49.485 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:35:49.485 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:35:49.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:49.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:49.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:49.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:49.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:49.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:49.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:49.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:49.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:49.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:35:50.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:50.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:50.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:50.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:35:50.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:50.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:50.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:50.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:50.544 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:35:50.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:50.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:50.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:50.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:50.754 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:50.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-30T23:35:50.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:50 smithi136 ceph-mon[20207]: pgmap v854: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-08-30T23:35:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1054590115' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:50 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:50.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:50 smithi106 ceph-mon[16603]: pgmap v854: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-08-30T23:35:50.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1054590115' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:50.822 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:50 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:50 smithi173 ceph-mon[20554]: pgmap v854: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-08-30T23:35:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1054590115' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:50 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:35:50.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.852 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:50.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:50.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:50.974 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:50.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:50.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:51.076 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:51.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:51.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:51.225 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:51.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:51.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:51.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:51.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:51.479 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:51.524 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:51.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:51.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:51.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:51.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:51.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:51 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:51.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:51 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:51.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:51 smithi106 ceph-mon[16603]: osdmap e356: 12 total, 12 up, 12 in 2024-08-30T23:35:51.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:51 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:51.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:51 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:51 smithi136 ceph-mon[20207]: osdmap e356: 12 total, 12 up, 12 in 2024-08-30T23:35:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:51 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:51 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:35:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:51 smithi173 ceph-mon[20554]: osdmap e356: 12 total, 12 up, 12 in 2024-08-30T23:35:51.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:51.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:51.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:51.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:51.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:52.022 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:52.038 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:35:52.039 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:35:52.064 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:35:52.064 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:35:52.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:52.096 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:35:52.097 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:35:52.115 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:52.130 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:35:52.130 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:35:52.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:35:52.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:35:52.194 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:35:52.195 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:35:52.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:52.226 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:35:52.226 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:35:52.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:52.262 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:35:52.262 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:35:52.302 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:35:52.302 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:35:52.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:35:52.338 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:35:52.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:52.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:52.373 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:35:52.374 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:35:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:52.404 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:35:52.405 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:35:52.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:52.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:52.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:52.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:52.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:52.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.767 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:52.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:52.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:52 smithi136 ceph-mon[20207]: pgmap v856: 17 pgs: 17 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-08-30T23:35:52.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2528410840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:52.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:52 smithi106 ceph-mon[16603]: pgmap v856: 17 pgs: 17 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-08-30T23:35:52.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2528410840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:52 smithi173 ceph-mon[20554]: pgmap v856: 17 pgs: 17 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-08-30T23:35:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2528410840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:35:52.863 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:35:52.863 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:35:52.863 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:35:52.863 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:35:52.864 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:35:52.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:52.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:52.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:53.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:53.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:53.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:53.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:53.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:53.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:53.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:53.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:53.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:35:53.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:53.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:53.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:53.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:53.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:53 smithi136 ceph-mon[20207]: pgmap v857: 17 pgs: 17 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 981 op/s 2024-08-30T23:35:53.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:53 smithi106 ceph-mon[16603]: pgmap v857: 17 pgs: 17 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 981 op/s 2024-08-30T23:35:53.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:53 smithi173 ceph-mon[20554]: pgmap v857: 17 pgs: 17 active+clean; 4.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 981 op/s 2024-08-30T23:35:53.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:53.936 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:53.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:54.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:54.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.064 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:54.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:54.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:54.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:54.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:54.379 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:54.379 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:35:54.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:54.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:54.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:54.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:54.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:54.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.701 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:54.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:54.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:54.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:35:54.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:54.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:54.949 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:55.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:35:55.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:35:55.151 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:55.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:55.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:55.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:55.328 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:55.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:55.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:55.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:55.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:55.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:55.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:55.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:55.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:55.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:35:55.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:35:55.953 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:35:55.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-30T23:35:55.970 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:55.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:56.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:56.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:56.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:56.236 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:56.338 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:56.365 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:56.442 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:56.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:56.525 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:56.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:56.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:56.726 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:56 smithi106 ceph-mon[16603]: pgmap v858: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-08-30T23:35:56.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:35:56.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:35:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:56 smithi136 ceph-mon[20207]: pgmap v858: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-08-30T23:35:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:56 smithi173 ceph-mon[20554]: pgmap v858: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 965 op/s 2024-08-30T23:35:56.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:56.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:56.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:56.989 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:35:56.989 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:35:56.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:35:56.989 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:14225 2.5 e109/97 2:a1e6a123:::benchmark_data_smithi173_47195_object14224:head v 109'898, mlcod=109'894)", 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 588.114057215, 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.239755616, 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:35:56.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874010+0000", 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874011+0000", 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:08.874108+0000", 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6961999999999998e-05 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112976+0000", 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23886803600000001 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.112983+0000", 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.951e-06 2024-08-30T23:35:56.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113055+0000", 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1806999999999999e-05 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113079+0000", 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4754000000000001e-05 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113724+0000", 2024-08-30T23:35:56.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00064475700000000003 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:09.113766+0000", 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1664e-05 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:35:56.994 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 508.49030189699999, 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:35:56.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109556 2.5 2:a6b86250:::benchmark_data_smithi173_47195_object12548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 492.13371891200001, 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220264298, 2024-08-30T23:35:56.998 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109556 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:56.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854350+0000", 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8840000000000001e-06 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854356+0000", 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6590000000000004e-06 2024-08-30T23:35:57.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854369+0000", 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.289e-05 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854377+0000", 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4649999999999994e-06 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854454+0000", 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6791000000000004e-05 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854600+0000", 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145864 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854751+0000", 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015046499999999999 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.857557+0000", 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028060419999999999 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.873657+0000", 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016100032 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073675+0000", 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200018474 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073721+0000", 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6072999999999997e-05 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.074613+0000", 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089165900000000005 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.004 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109564 2.4 2:242bfffa:::benchmark_data_smithi173_47195_object12556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 492.12627489200003, 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220047624, 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109564 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.005 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861793+0000", 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861794+0000", 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5889999999999999e-06 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861800+0000", 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861809+0000", 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0599999999999997e-06 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861815+0000", 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0669999999999997e-06 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.861910+0000", 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5278000000000005e-05 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862048+0000", 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137911 2024-08-30T23:35:57.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.862173+0000", 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012423299999999999 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.878997+0000", 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016824741000000001 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879048+0000", 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0890999999999998e-05 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080272+0000", 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20122369600000001 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.080313+0000", 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0785999999999999e-05 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.081840+0000", 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015274679999999999 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109569 2.5 2:a8435ff8:::benchmark_data_smithi173_47195_object12561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 492.11857771400003, 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22083613699999999, 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.011 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109569 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:35:57.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869491+0000", 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4270000000000001e-06 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869498+0000", 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869510+0000", 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1630999999999999e-05 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869519+0000", 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869586+0000", 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6699999999999995e-05 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869741+0000", 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155141 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869888+0000", 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014657099999999999 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879878+0000", 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099902989999999994 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879907+0000", 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8612999999999999e-05 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089168+0000", 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20926119900000001 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089267+0000", 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9097000000000003e-05 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090326+0000", 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010589670000000001 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144821 2.4 2:2415a84c:::benchmark_data_smithi173_47195_object47813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:35:57.017 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 445.42468373899999, 2024-08-30T23:35:57.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:57.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23107453999999999, 2024-08-30T23:35:57.018 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.018 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144821 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:35:57.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563385+0000", 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.59e-06 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563391+0000", 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563404+0000", 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2982e-05 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563411+0000", 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565303+0000", 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001891846 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565431+0000", 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012746400000000001 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565550+0000", 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972900000000001 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.573426+0000", 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078756799999999995 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588851+0000", 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015424679 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793322+0000", 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20447127600000001 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793387+0000", 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4586999999999997e-05 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.794458+0000", 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010715060000000001 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144836 2.4 2:2d7ce86c:::benchmark_data_smithi173_47195_object47828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 445.39995037300002, 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23589892500000001, 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.024 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144836 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588118+0000", 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3829999999999999e-06 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588123+0000", 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588169+0000", 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199999999999998e-05 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588182+0000", 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2748999999999999e-05 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.592926+0000", 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047443779999999996 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595020+0000", 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002094243 2024-08-30T23:35:57.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595429+0000", 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040842299999999999 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603861+0000", 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0084324050000000005 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.604206+0000", 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034452400000000002 2024-08-30T23:35:57.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.822951+0000", 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21874496900000001 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.823010+0000", 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9601000000000001e-05 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824016+0000", 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010057499999999999 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:193978 2.5 2:a2ee5e67:::benchmark_data_smithi173_47195_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.23192092099998, 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.283382829, 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.030 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193978 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756148+0000", 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756154+0000", 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756166+0000", 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041e-05 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756172+0000", 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2349999999999996e-06 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756333+0000", 2024-08-30T23:35:57.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000160767 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756484+0000", 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015133799999999999 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756627+0000", 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014263700000000001 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764342+0000", 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077147439999999999 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.769055+0000", 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047136399999999998 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039012+0000", 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26995667699999998 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039055+0000", 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2567000000000001e-05 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039529+0000", 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047478400000000001 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:193978 2.5 e200/182 ondisk, result = 0)", 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:35:57.036 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.22396565499997, 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27543190299999998, 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764103+0000", 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764108+0000", 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9080000000000003e-06 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038928+0000", 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27481996400000003 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038937+0000", 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7860000000000008e-06 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038998+0000", 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0915999999999997e-05 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039007+0000", 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4319999999999999e-06 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039534+0000", 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052686200000000001 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e200,182 2:a1c9afc1:::benchmark_data_smithi173_47195_object1378:head,146'6135)", 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.205257453, 2024-08-30T23:35:57.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42837196799999999, 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782811+0000", 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782816+0000", 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6969999999999998e-06 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210395+0000", 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42757847500000001 2024-08-30T23:35:57.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210404+0000", 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0000000000000002e-06 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210493+0000", 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9438000000000007e-05 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.211182+0000", 2024-08-30T23:35:57.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068873000000000003 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2685 3.6 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 344.294533726, 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22443937, 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.044 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2685 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.046 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:57.046 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693535+0000", 2024-08-30T23:35:57.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.009e-06 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693573+0000", 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7843999999999999e-05 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693589+0000", 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6359e-05 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693603+0000", 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.412e-05 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693657+0000", 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3757e-05 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693735+0000", 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8444999999999999e-05 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693820+0000", 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4290999999999996e-05 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.694189+0000", 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036947500000000002 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.874073+0000", 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.179884508 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917876+0000", 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043802633000000001 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917931+0000", 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5297999999999998e-05 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917973+0000", 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1631000000000002e-05 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.051 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176646 3.e 3:78ef2980:::benchmark_data_smithi173_60780_object73192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 172.25323374999999, 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22395779099999999, 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:57.052 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176646 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734835+0000", 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7399999999999991e-07 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734839+0000", 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5800000000000002e-06 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734853+0000", 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.375e-05 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734858+0000", 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738711+0000", 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003852847 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738850+0000", 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000138818 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738954+0000", 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001037 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748083+0000", 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091293829999999996 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748375+0000", 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000292427 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748430+0000", 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4191999999999999e-05 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.756943+0000", 2024-08-30T23:35:57.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085137770000000002 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958661+0000", 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.201717853 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958707+0000", 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5840999999999997e-05 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958791+0000", 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4349000000000002e-05 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176647 3.e 3:751af89f:::benchmark_data_smithi173_60780_object73193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 172.25198499199999, 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223004061, 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.059 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176647 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:35:57.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736083+0000", 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8299999999999995e-07 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736088+0000", 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2370000000000003e-06 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736098+0000", 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0835e-05 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736104+0000", 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742826+0000", 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006721505 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742967+0000", 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140977 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.743123+0000", 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015588399999999999 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748383+0000", 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052605059999999999 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.758513+0000", 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010130116 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767529+0000", 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090161789999999992 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771374+0000", 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038445660000000002 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958849+0000", 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18747534399999999 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958930+0000", 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0463000000000005e-05 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.959086+0000", 2024-08-30T23:35:57.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000156701 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:31057 4.b 4:d5378b27:::benchmark_data_smithi173_74914_object31056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 96.300709455000003, 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.641950197, 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:35:57.066 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 31057 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687360+0000", 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0669999999999999e-06 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687397+0000", 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6677999999999997e-05 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687413+0000", 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532999999999999e-05 2024-08-30T23:35:57.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687421+0000", 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9410000000000007e-06 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687486+0000", 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4894000000000006e-05 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687630+0000", 2024-08-30T23:35:57.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014421900000000001 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687780+0000", 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000149748 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689089+0000", 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013092609999999999 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689219+0000", 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013019 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329200+0000", 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63998067999999997 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329262+0000", 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1508999999999997e-05 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329308+0000", 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6477000000000002e-05 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.0 319/318 [PushOp(4:08a755c5:::benchmark_data_smithi173_74914_object390:head, version: 311'18, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:08a755c5:::benchmark_data_smithi173_74914_object390:head@311'18, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 94.064237528000007, 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29680203999999999, 2024-08-30T23:35:57.072 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923831+0000", 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923901+0000", 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0507999999999996e-05 2024-08-30T23:35:57.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219654+0000", 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.295752615 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219681+0000", 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7390999999999999e-05 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219790+0000", 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108904 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219807+0000", 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6871000000000002e-05 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.220632+0000", 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082508799999999997 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.076 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:61123 4.b 4:d896c312:::benchmark_data_smithi173_74914_object61122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 62.774670125, 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93993815199999997, 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.077 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 61123 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213400+0000", 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5000000000000002e-06 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213421+0000", 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1537000000000001e-05 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213434+0000", 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2478e-05 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213440+0000", 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8699999999999997e-06 2024-08-30T23:35:57.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213485+0000", 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5439999999999999e-05 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213576+0000", 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0767999999999996e-05 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:35:57.081 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213667+0000", 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1135000000000001e-05 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215888+0000", 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022207139999999999 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215938+0000", 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0022000000000003e-05 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153264+0000", 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93732649300000004 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153304+0000", 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0139999999999999e-05 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153335+0000", 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1055e-05 2024-08-30T23:35:57.083 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:61123 4.b e335/306 ondisk, result = 0)", 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 62.772029326000002, 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93730281299999996, 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216039+0000", 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4099999999999999e-07 2024-08-30T23:35:57.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216042+0000", 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7529999999999999e-06 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153195+0000", 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93715309700000005 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153202+0000", 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7340000000000002e-06 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153255+0000", 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2420999999999998e-05 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153260+0000", 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7370000000000003e-06 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153341+0000", 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0530000000000003e-05 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.5 335/318 [PushOp(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head, version: 316'1784, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head@316'1784, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 61.813816723000002, 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.83137486000000005, 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.088 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174252+0000", 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174402+0000", 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015002700000000001 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003927+0000", 2024-08-30T23:35:57.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82952564699999998 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003946+0000", 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8678999999999999e-05 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004043+0000", 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6718999999999993e-05 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004054+0000", 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0925e-05 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.005626+0000", 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001571813 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98225 4.5 e353/337 4:ad425262:::benchmark_data_smithi173_74914_object98224:head v 353'6223, mlcod=353'6220)", 2024-08-30T23:35:57.092 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 25.543097883000002, 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32306562100000003, 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444971+0000", 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0699999999999999e-06 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.445067+0000", 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5938000000000002e-05 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765335+0000", 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32026851299999998 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765347+0000", 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.151e-05 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765448+0000", 2024-08-30T23:35:57.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100967 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765468+0000", 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0219000000000001e-05 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.767987+0000", 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025194890000000002 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.768035+0000", 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7914999999999998e-05 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98228 4.7 e353/329 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head v 353'6056, mlcod=353'6052)", 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 25.539961198, 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35600754000000001, 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:35:57.097 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448107+0000", 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3300000000000001e-07 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448202+0000", 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4503e-05 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801211+0000", 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35300926399999999 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801228+0000", 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6694999999999999e-05 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801373+0000", 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014589400000000001 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801409+0000", 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6025999999999997e-05 2024-08-30T23:35:57.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804078+0000", 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026682939999999999 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804114+0000", 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6031000000000001e-05 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:35:57.101 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:35:57.102 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:35:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:57.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:57.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:57.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:57.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:57.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:57.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:35:57.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:35:57.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:57.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:57.658 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:35:57.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:57.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:57.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:57.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:57.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:57.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:57.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:35:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:35:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:57 smithi136 ceph-mon[20207]: osdmap e357: 12 total, 12 up, 12 in 2024-08-30T23:35:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1417069145' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:57 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:57 smithi136 ceph-mon[20207]: pgmap v860: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-08-30T23:35:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:35:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:35:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:57 smithi173 ceph-mon[20554]: osdmap e357: 12 total, 12 up, 12 in 2024-08-30T23:35:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1417069145' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:57 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:57 smithi173 ceph-mon[20554]: pgmap v860: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-08-30T23:35:58.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:57 smithi106 ceph-mon[16603]: osdmap e357: 12 total, 12 up, 12 in 2024-08-30T23:35:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1417069145' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:57 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:35:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:57 smithi106 ceph-mon[16603]: pgmap v860: 17 pgs: 17 active+clean; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 985 op/s 2024-08-30T23:35:58.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:58.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:35:58.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:58.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:58.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:58.380 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:58.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:35:58.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:35:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:35:58.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:58.633 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:58.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:35:58.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:35:58.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:35:58.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:58.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:58.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:35:58.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:59.018 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:58 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:58 smithi136 ceph-mon[20207]: osdmap e358: 12 total, 12 up, 12 in 2024-08-30T23:35:59.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:58 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:59.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:58 smithi173 ceph-mon[20554]: osdmap e358: 12 total, 12 up, 12 in 2024-08-30T23:35:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:58 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:35:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:35:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:58 smithi106 ceph-mon[16603]: osdmap e358: 12 total, 12 up, 12 in 2024-08-30T23:35:59.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:35:59.151 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:35:59.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:35:59.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:35:59.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:59.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:59.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:35:59.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:35:59.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:35:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.670 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:35:59.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:35:59.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:35:59.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:35:59.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:35:59.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:35:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:35:59.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:35:59.944 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:59 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:35:59.944 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:59 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:35:59.945 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:59 smithi106 ceph-mon[16603]: osdmap e359: 12 total, 12 up, 12 in 2024-08-30T23:35:59.945 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:35:59 smithi106 ceph-mon[16603]: pgmap v863: 17 pgs: 17 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-08-30T23:35:59.994 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:35:59.995 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:36:00.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:59 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:59 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:59 smithi136 ceph-mon[20207]: osdmap e359: 12 total, 12 up, 12 in 2024-08-30T23:36:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:35:59 smithi136 ceph-mon[20207]: pgmap v863: 17 pgs: 17 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-08-30T23:36:00.055 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:59 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:59 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:59 smithi173 ceph-mon[20554]: osdmap e359: 12 total, 12 up, 12 in 2024-08-30T23:36:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:35:59 smithi173 ceph-mon[20554]: pgmap v863: 17 pgs: 17 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-08-30T23:36:00.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:00.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:00.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:00.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:00.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:00.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:00.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:00.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:00.627 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:00.628 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:00.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:36:00.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:36:00.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:00.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:00.695 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:00.722 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:00.722 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:00.722 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3674405997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:00.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:00.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:00.778 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:00.778 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:00.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:00.807 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:00.807 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:00.822 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:00.837 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:00.838 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:00.867 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:00.867 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:00.898 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:00.898 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:00.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:00.926 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:00.927 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:00.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:00.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:00.953 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:00.954 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3674405997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:01.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3674405997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:01.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:01.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:01.209 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:01.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:01.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:01.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:01.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:01.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:01.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:01.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:01.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:01.817 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:01.844 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:01.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:01.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:01.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:02.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:01 smithi136 ceph-mon[20207]: pgmap v864: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:36:02.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:01 smithi173 ceph-mon[20554]: pgmap v864: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:36:02.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:02.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:01 smithi106 ceph-mon[16603]: pgmap v864: 17 pgs: 17 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-08-30T23:36:02.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:02.231 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:02.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:02.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:02.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:02.489 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:02.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:02.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:02.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:02.746 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:02.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:02.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:36:02.917 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:36:02.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-30T23:36:02.975 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:02.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:02.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:03.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:03.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:03.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:03.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:03.349 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:03.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:03.356 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:03.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:03.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:03.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:03.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:03.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:03.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:03.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:03.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.764 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:03.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:03.896 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:03.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:03.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:04.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:04.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:04.133 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:36:04.134 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:36:04.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:04.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:04.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:04.336 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:04.336 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:04.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:04.411 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:04.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:04.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:04.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:04.714 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:04 smithi106 ceph-mon[16603]: pgmap v865: 17 pgs: 17 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-08-30T23:36:04.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:04.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:36:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:04 smithi136 ceph-mon[20207]: pgmap v865: 17 pgs: 17 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-08-30T23:36:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:04 smithi173 ceph-mon[20554]: pgmap v865: 17 pgs: 17 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 996 op/s 2024-08-30T23:36:04.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:04.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:04.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:04.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:04.987 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:05.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:05.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:05.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:36:05.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:05.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:05.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:36:05.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:05.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:05.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:05.492 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:05.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:05 smithi106 ceph-mon[16603]: pgmap v866: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-08-30T23:36:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:05.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:05.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:05.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:05.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:05 smithi173 ceph-mon[20554]: pgmap v866: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-08-30T23:36:05.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:05.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:05.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:05.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:06.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:06.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:05 smithi136 ceph-mon[20207]: pgmap v866: 17 pgs: 17 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-08-30T23:36:06.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:06.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:36:06.160 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:36:06.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-08-30T23:36:06.205 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:06.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:06.260 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:06.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:06.287 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:06.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:06.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:06.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:06.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:06.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:06.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:36:06.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:36:06.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:06 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:36:06.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:06 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:36:06.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:06.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:06.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:06.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:36:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:36:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:06 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:36:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:06 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:36:06.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:06.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:06.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:07.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:36:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:36:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:06 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:36:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:06 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:36:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:07.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:07.092 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:07.092 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:07.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:36:07.122 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:36:07.152 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:07.153 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:07.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:07.181 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:07.182 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:07.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:07.210 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:07.211 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:07.243 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:07.243 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:07.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:07.274 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:07.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:07.298 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:36:07.298 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:36:07.298 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:36:07.298 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:36:07.298 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:36:07.307 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:07.307 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:07.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:07.344 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:07.345 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:07.374 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:07.375 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:07.403 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:07.404 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:07.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:07.438 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:07.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:07.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:07.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:07.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:07.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:07.724 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/17872701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:07 smithi173 ceph-mon[20554]: pgmap v867: 17 pgs: 17 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 955 op/s 2024-08-30T23:36:07.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:07.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:07.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/17872701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:07 smithi106 ceph-mon[16603]: pgmap v867: 17 pgs: 17 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 955 op/s 2024-08-30T23:36:07.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:07.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:07.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/17872701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:07 smithi136 ceph-mon[20207]: pgmap v867: 17 pgs: 17 active+clean; 4.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 955 op/s 2024-08-30T23:36:08.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:08.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:08.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:08.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:08.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:08.360 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:08.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:08.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:08.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:08.621 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:08.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:08.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:08 smithi173 ceph-mon[20554]: osdmap e360: 12 total, 12 up, 12 in 2024-08-30T23:36:08.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:08.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:08.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:08.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:08 smithi106 ceph-mon[16603]: osdmap e360: 12 total, 12 up, 12 in 2024-08-30T23:36:08.888 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:08.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:08.940 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:08.940 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:08.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:08 smithi136 ceph-mon[20207]: osdmap e360: 12 total, 12 up, 12 in 2024-08-30T23:36:08.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:09.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:09.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:09.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:09.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:09.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:09.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:09.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:09.464 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:09.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:36:09.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:09.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:09.581 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:36:09.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:09.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:09.726 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/647111984' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:09.726 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:09 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:09.726 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:09 smithi106 ceph-mon[16603]: pgmap v869: 17 pgs: 17 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-08-30T23:36:09.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:09.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:09.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:09.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:09.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:09.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/647111984' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:09.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:09 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:09.957 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:09 smithi136 ceph-mon[20207]: pgmap v869: 17 pgs: 17 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-08-30T23:36:10.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/647111984' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:09 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:09 smithi173 ceph-mon[20554]: pgmap v869: 17 pgs: 17 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-08-30T23:36:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.079 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:10.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:10.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:10.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:10.231 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:36:10.231 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:36:10.231 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:36:10.232 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:36:10.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:10.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:36:10.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.350 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:10.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:10.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:10.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:10 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:10 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:36:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:10 smithi106 ceph-mon[16603]: osdmap e361: 12 total, 12 up, 12 in 2024-08-30T23:36:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:10.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:10.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:10.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:10.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:10 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:10 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:36:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:10 smithi173 ceph-mon[20554]: osdmap e361: 12 total, 12 up, 12 in 2024-08-30T23:36:10.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:10.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:10.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:10.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:11.001 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:11.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:10 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:10 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:36:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:10 smithi136 ceph-mon[20207]: osdmap e361: 12 total, 12 up, 12 in 2024-08-30T23:36:11.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:11.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:11.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:11.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:11.344 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:11.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:36:11.370 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:36:11.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:11.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:11.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:11.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:11.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:11.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:11.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:11 smithi136 ceph-mon[20207]: pgmap v871: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 994 op/s 2024-08-30T23:36:11.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:11.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:11.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:11.909 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:11.919 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:11.920 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:11.920 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:11 smithi106 ceph-mon[16603]: pgmap v871: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 994 op/s 2024-08-30T23:36:11.954 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:36:11.954 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:36:11.980 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:11.981 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:11.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:12.005 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:12.005 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:12.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:12.044 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:12.045 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:12.073 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:12.073 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:11 smithi173 ceph-mon[20554]: pgmap v871: 17 pgs: 17 active+clean; 3.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 994 op/s 2024-08-30T23:36:12.101 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:12.101 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:12.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:12.130 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:12.130 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:12.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:12.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:12.161 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:12.190 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:12.190 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:12.220 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:12.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:12.249 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:12.249 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:12.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:12.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:12.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:12.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:12.503 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:12.503 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:12.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:12.549 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:12.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:36:12.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:12.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:12.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:12.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:12.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:12 smithi106 ceph-mon[16603]: osdmap e362: 12 total, 12 up, 12 in 2024-08-30T23:36:12.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/459823905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:12.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:12.943 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:12 smithi136 ceph-mon[20207]: osdmap e362: 12 total, 12 up, 12 in 2024-08-30T23:36:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/459823905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:13.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:12 smithi173 ceph-mon[20554]: osdmap e362: 12 total, 12 up, 12 in 2024-08-30T23:36:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/459823905' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.076 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:13.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:13.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:13.206 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:13.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:13.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:13.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.469 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:13.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:13.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:13.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:13.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:13.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:13.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:13.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:13 smithi106 ceph-mon[16603]: pgmap v873: 17 pgs: 17 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-08-30T23:36:13.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:13.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:36:13.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:13.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:13 smithi136 ceph-mon[20207]: pgmap v873: 17 pgs: 17 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-08-30T23:36:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:13 smithi173 ceph-mon[20554]: pgmap v873: 17 pgs: 17 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s 2024-08-30T23:36:14.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:14.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:14.113 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:14.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:14.235 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:14.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:14.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:14.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:14.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:14.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:14.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:14.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:14.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:14.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:14 smithi106 ceph-mon[16603]: 4.a scrub starts 2024-08-30T23:36:14.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:14.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:14.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:14.885 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:14.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:15.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:14 smithi136 ceph-mon[20207]: 4.a scrub starts 2024-08-30T23:36:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:14 smithi173 ceph-mon[20554]: 4.a scrub starts 2024-08-30T23:36:15.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:15.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:15.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:15.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:15.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:36:15.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:15.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:15.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:15.459 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:15.459 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:15.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:15.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:15.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:15.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:15.689 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:36:15.717 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:15 smithi106 ceph-mon[16603]: 4.a scrub ok 2024-08-30T23:36:15.717 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1676930761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:15.717 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:15 smithi106 ceph-mon[16603]: pgmap v874: 17 pgs: 17 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 919 op/s 2024-08-30T23:36:15.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:15.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:15.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:15.913 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:15.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:16.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:15 smithi136 ceph-mon[20207]: 4.a scrub ok 2024-08-30T23:36:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1676930761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:15 smithi136 ceph-mon[20207]: pgmap v874: 17 pgs: 17 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 919 op/s 2024-08-30T23:36:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:16.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:15 smithi173 ceph-mon[20554]: 4.a scrub ok 2024-08-30T23:36:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1676930761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:15 smithi173 ceph-mon[20554]: pgmap v874: 17 pgs: 17 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 919 op/s 2024-08-30T23:36:16.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:16.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:16.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:36:16.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:16.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:16.371 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:36:16.371 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:36:16.371 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:36:16.372 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-08-30T23:36:16.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 0 0 2024-08-30T23:36:16.404 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.460 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:16.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:16.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.603 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:16.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:16.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:16.732 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:16.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:16 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:36:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:16 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:36:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1676930761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:16 smithi106 ceph-mon[16603]: osdmap e363: 12 total, 12 up, 12 in 2024-08-30T23:36:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:16.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:16.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:16.990 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:16.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:16 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:36:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:16 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:36:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1676930761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:16 smithi136 ceph-mon[20207]: osdmap e363: 12 total, 12 up, 12 in 2024-08-30T23:36:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:16 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:36:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:16 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:36:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1676930761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:16 smithi173 ceph-mon[20554]: osdmap e363: 12 total, 12 up, 12 in 2024-08-30T23:36:17.074 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:17.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:17.137 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:17.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:17.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:17.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:17.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:17.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:17.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:17.607 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:17.608 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:17.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:17.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:17.684 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:17.685 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:17.721 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:36:17.721 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:36:17.755 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:17.755 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:17.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:17.780 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:17.780 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:17.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:17.814 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:17.814 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:17.841 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:17.841 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:17.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:17.872 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:17.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:17.901 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:17.901 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:17.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:17.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:17.934 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:17.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:17.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:17 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:17.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:17 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:17.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:17 smithi106 ceph-mon[16603]: pgmap v876: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s 2024-08-30T23:36:17.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2488214440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:17.973 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:17.974 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:18.014 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:18.015 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:18.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:17 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:17 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:17 smithi136 ceph-mon[20207]: pgmap v876: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s 2024-08-30T23:36:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2488214440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:18.045 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:18.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:18.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:17 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:17 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:17 smithi173 ceph-mon[20554]: pgmap v876: 17 pgs: 17 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 964 op/s 2024-08-30T23:36:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2488214440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:18.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:18.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.181 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:18.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:18.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:18.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:18.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:18.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:18.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:18.694 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:18.712 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.0 primary-affinity to 0 (802) 2024-08-30T23:36:18.738 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:36:18.739 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:36:18.739 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:36:18.739 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:36:18.739 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:36:18.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:18.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:18 smithi106 ceph-mon[16603]: osdmap e364: 12 total, 12 up, 12 in 2024-08-30T23:36:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1421730296' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-08-30T23:36:18.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:18 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-08-30T23:36:18.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:18.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:18.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:18 smithi136 ceph-mon[20207]: osdmap e364: 12 total, 12 up, 12 in 2024-08-30T23:36:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1421730296' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-08-30T23:36:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:18 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-08-30T23:36:19.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:19.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:36:19.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:18 smithi173 ceph-mon[20554]: osdmap e364: 12 total, 12 up, 12 in 2024-08-30T23:36:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1421730296' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-08-30T23:36:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:18 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} : dispatch 2024-08-30T23:36:19.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:19.192 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:19.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:19.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:19.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:19.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:19.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:19.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:19.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:19.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:19.686 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:19.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:19.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:19 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-08-30T23:36:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:19 smithi106 ceph-mon[16603]: osdmap e365: 12 total, 12 up, 12 in 2024-08-30T23:36:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:19 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:19 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:19 smithi106 ceph-mon[16603]: pgmap v879: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-08-30T23:36:19.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:19.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:19.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:19 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-08-30T23:36:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:19 smithi136 ceph-mon[20207]: osdmap e365: 12 total, 12 up, 12 in 2024-08-30T23:36:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:19 smithi136 ceph-mon[20207]: pgmap v879: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-08-30T23:36:20.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:20.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:19 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-08-30T23:36:20.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:19 smithi173 ceph-mon[20554]: osdmap e365: 12 total, 12 up, 12 in 2024-08-30T23:36:20.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:19 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:20.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:19 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:20.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:19 smithi173 ceph-mon[20554]: pgmap v879: 17 pgs: 17 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-08-30T23:36:20.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:20.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:20.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:20.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:20.338 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:20.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:20.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:20.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:20.589 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:20.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:20.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:20.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:20.854 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:20.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:20.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:20.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:20 smithi136 ceph-mon[20207]: osdmap e366: 12 total, 12 up, 12 in 2024-08-30T23:36:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:20 smithi173 ceph-mon[20554]: osdmap e366: 12 total, 12 up, 12 in 2024-08-30T23:36:21.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:20 smithi106 ceph-mon[16603]: osdmap e366: 12 total, 12 up, 12 in 2024-08-30T23:36:21.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:21.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:36:21.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:21.217 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:21.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:21.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:21.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:21.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:21.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:21.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:21.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:21.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:21.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:21.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:21 smithi106 ceph-mon[16603]: 4.f scrub starts 2024-08-30T23:36:21.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:21 smithi106 ceph-mon[16603]: 4.6 scrub starts 2024-08-30T23:36:21.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:21 smithi106 ceph-mon[16603]: 4.6 scrub ok 2024-08-30T23:36:21.773 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:21 smithi106 ceph-mon[16603]: pgmap v881: 17 pgs: 1 activating, 16 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 980 op/s 2024-08-30T23:36:21.824 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:21.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:21.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:21.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:21.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:22.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:21 smithi136 ceph-mon[20207]: 4.f scrub starts 2024-08-30T23:36:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:21 smithi136 ceph-mon[20207]: 4.6 scrub starts 2024-08-30T23:36:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:21 smithi136 ceph-mon[20207]: 4.6 scrub ok 2024-08-30T23:36:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:21 smithi136 ceph-mon[20207]: pgmap v881: 17 pgs: 1 activating, 16 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 980 op/s 2024-08-30T23:36:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:21 smithi173 ceph-mon[20554]: 4.f scrub starts 2024-08-30T23:36:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:21 smithi173 ceph-mon[20554]: 4.6 scrub starts 2024-08-30T23:36:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:21 smithi173 ceph-mon[20554]: 4.6 scrub ok 2024-08-30T23:36:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:21 smithi173 ceph-mon[20554]: pgmap v881: 17 pgs: 1 activating, 16 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 980 op/s 2024-08-30T23:36:22.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:22.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:22.180 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:36:22.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:22.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:22.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.406 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:22.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:56060 2.e 2:7b4ebd5c:::benchmark_data_smithi173_47195_object56059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 582.66597143399997, 2024-08-30T23:36:22.450 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21409088000000001, 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 56060 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:36:22.451 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779477+0000", 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779478+0000", 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4869999999999999e-06 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.452 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779513+0000", 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4384999999999997e-05 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779529+0000", 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6447999999999999e-05 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779537+0000", 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999993e-06 2024-08-30T23:36:22.453 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779580+0000", 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3254999999999998e-05 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779658+0000", 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7711000000000002e-05 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:36:22.454 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.779738+0000", 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9775000000000001e-05 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781534+0000", 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017964820000000001 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992319+0000", 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21078449599999999 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993464+0000", 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001145294 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993511+0000", 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6344000000000002e-05 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.456 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.993568+0000", 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6993000000000001e-05 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:56060 2.e e124/97 ondisk, result = 0)", 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 582.66363146499998, 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21051657200000001, 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781817+0000", 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9999999999999997e-07 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.781821+0000", 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9480000000000001e-06 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990657+0000", 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20883629200000001 2024-08-30T23:36:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.990664+0000", 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.793e-06 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992305+0000", 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016408480000000001 2024-08-30T23:36:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992313+0000", 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1100000000000003e-06 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:39.992333+0000", 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9981000000000001e-05 2024-08-30T23:36:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:67116 2.e 2:727fcaa1:::benchmark_data_smithi173_47195_object67115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 572.833517648, 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22641219700000001, 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 67116 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611930+0000", 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.611931+0000", 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612007+0000", 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5883999999999994e-05 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612021+0000", 2024-08-30T23:36:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4007e-05 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.612028+0000", 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4099999999999996e-06 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614110+0000", 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020827520000000002 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614270+0000", 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000159235 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.614420+0000", 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015048199999999999 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617176+0000", 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027561619999999999 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.617722+0000", 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054594799999999998 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838237+0000", 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22051536999999999 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838285+0000", 2024-08-30T23:36:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7808e-05 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:49.838343+0000", 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7340000000000003e-05 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:75429 2.1 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:36:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 565.48700963199997, 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24538217700000001, 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 75429 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:36:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958438+0000", 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958440+0000", 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9410000000000001e-06 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958471+0000", 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0919e-05 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958494+0000", 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3078000000000002e-05 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958504+0000", 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1190000000000003e-06 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958565+0000", 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1364000000000004e-05 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958669+0000", 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010414199999999999 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958772+0000", 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102979 2024-08-30T23:36:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.960745+0000", 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019727600000000001 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.066102+0000", 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.105356845 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203744+0000", 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13764249000000001 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203787+0000", 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.286e-05 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203821+0000", 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3680000000000003e-05 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head, version: 114'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head@114'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 517.55329072999996, 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19944226600000001, 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892158+0000", 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892186+0000", 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7710000000000001e-05 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089119+0000", 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19693307299999999 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089137+0000", 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7533e-05 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089255+0000", 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118318 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089275+0000", 2024-08-30T23:36:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9828000000000001e-05 2024-08-30T23:36:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091600+0000", 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023249030000000001 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:126363 2.7 e160/138 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head v 160'7890, mlcod=119'2657)", 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 495.71582542300001, 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.217439837, 2024-08-30T23:36:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729630+0000", 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.674e-06 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940470+0000", 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21084006199999999 2024-08-30T23:36:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940478+0000", 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4390000000000003e-06 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940541+0000", 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3115999999999997e-05 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940571+0000", 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9981e-05 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947012+0000", 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064411269999999996 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947062+0000", 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0708999999999998e-05 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144840 2.1 2:8bdc20ef:::benchmark_data_smithi173_47195_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 470.85196976100002, 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21736361900000001, 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144840 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593481+0000", 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3460000000000001e-06 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593490+0000", 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593512+0000", 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2410000000000001e-05 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593522+0000", 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4129999999999995e-06 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593705+0000", 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018286800000000001 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593993+0000", 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028869300000000002 2024-08-30T23:36:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.594314+0000", 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032096700000000002 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.599306+0000", 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049920049999999999 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603619+0000", 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043124419999999997 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810726+0000", 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20710743000000001 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810784+0000", 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7893999999999998e-05 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810842+0000", 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7791000000000003e-05 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144845 2.1 2:80fd4430:::benchmark_data_smithi173_47195_object47837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 470.84212477900002, 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20768498299999999, 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144845 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603326+0000", 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.373e-06 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603335+0000", 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5449999999999992e-06 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603359+0000", 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3377e-05 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603370+0000", 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1759e-05 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603649+0000", 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027892500000000002 2024-08-30T23:36:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603755+0000", 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010549500000000001 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603833+0000", 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8150000000000002e-05 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.609421+0000", 2024-08-30T23:36:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0055884799999999998 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.640300+0000", 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030878974999999999 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810910+0000", 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.170609233 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810964+0000", 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4549999999999998e-05 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.811008+0000", 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4121e-05 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155294 2.e 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:36:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 456.32156586899998, 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26817000400000002, 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155294 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123884+0000", 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3960000000000001e-06 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123889+0000", 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4820000000000002e-06 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123902+0000", 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3202e-05 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123910+0000", 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8869999999999992e-06 2024-08-30T23:36:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123962+0000", 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1959000000000002e-05 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124045+0000", 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2850999999999997e-05 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124115+0000", 2024-08-30T23:36:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9513999999999999e-05 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.127474+0000", 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033596149999999998 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.128135+0000", 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00066107399999999999 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.391962+0000", 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26382699500000001 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392009+0000", 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6669999999999999e-05 2024-08-30T23:36:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392052+0000", 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3359000000000003e-05 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 182/143 [PushOp(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head, version: 132'4708, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head@132'4708, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:36:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 455.005887105, 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25797871999999999, 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:36:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439562+0000", 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439584+0000", 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2395000000000002e-05 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693170+0000", 2024-08-30T23:36:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25358568300000001 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693191+0000", 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1192999999999999e-05 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693349+0000", 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015837300000000001 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693372+0000", 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2869000000000001e-05 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.697540+0000", 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004167558 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168870 2.e 2:74d7d6f3:::benchmark_data_smithi173_47195_object71862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.82981090800001, 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223579366, 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168870 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615640+0000", 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3590000000000002e-06 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615648+0000", 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8559999999999996e-06 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615665+0000", 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6563999999999999e-05 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615676+0000", 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1075e-05 2024-08-30T23:36:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615800+0000", 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123905 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615963+0000", 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016347900000000001 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:36:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616104+0000", 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140846 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620671+0000", 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045667540000000001 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620686+0000", 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.484e-05 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839134+0000", 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21844793100000001 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839172+0000", 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7737999999999997e-05 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839217+0000", 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5018999999999997e-05 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168873 2.1 2:8ff1cc28:::benchmark_data_smithi173_47195_object71865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.82974299699998, 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22707487400000001, 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168873 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615706+0000", 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2500000000000001e-07 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615708+0000", 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6970000000000002e-06 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615715+0000", 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8499999999999996e-06 2024-08-30T23:36:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615720+0000", 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.395e-06 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616068+0000", 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034794699999999998 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616173+0000", 2024-08-30T23:36:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104145 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616295+0000", 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012253000000000001 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620669+0000", 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043740690000000004 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.621058+0000", 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038845600000000002 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842677+0000", 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22161980100000001 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842735+0000", 2024-08-30T23:36:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7420000000000003e-05 2024-08-30T23:36:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842780+0000", 2024-08-30T23:36:22.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5139000000000003e-05 2024-08-30T23:36:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168892 2.e 2:7a358138:::benchmark_data_smithi173_47195_object71884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.80731869700003, 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208213383, 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168892 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638131+0000", 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638136+0000", 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638147+0000", 2024-08-30T23:36:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1119e-05 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638153+0000", 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8320000000000002e-06 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638212+0000", 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9219999999999999e-05 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638342+0000", 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012929599999999999 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638476+0000", 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013422799999999999 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643016+0000", 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045403120000000003 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643358+0000", 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034153500000000002 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846238+0000", 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202880476 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846298+0000", 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9453000000000002e-05 2024-08-30T23:36:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846343+0000", 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5059999999999999e-05 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168895 2.1 2:802748b8:::benchmark_data_smithi173_47195_object71887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.80308240400001, 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21216137199999999, 2024-08-30T23:36:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168895 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642368+0000", 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3080000000000002e-06 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642376+0000", 2024-08-30T23:36:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6769999999999995e-06 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642392+0000", 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642400+0000", 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6329999999999996e-06 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642458+0000", 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8498000000000001e-05 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642574+0000", 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115822 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642662+0000", 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655709+0000", 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013046791 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655767+0000", 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8464000000000001e-05 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854414+0000", 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19864683999999999 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854479+0000", 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5042999999999994e-05 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854527+0000", 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8195999999999997e-05 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168896 2.1 2:82321f29:::benchmark_data_smithi173_47195_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:36:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.80224938200001, 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211510109, 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168896 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643200+0000", 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6899999999999996e-07 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643204+0000", 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643215+0000", 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.041e-05 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643220+0000", 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:36:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646422+0000", 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032022190000000001 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646565+0000", 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142564 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646696+0000", 2024-08-30T23:36:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130877 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655755+0000", 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090594430000000004 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655773+0000", 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7263999999999999e-05 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854613+0000", 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19883995700000001 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854672+0000", 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9522999999999999e-05 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854709+0000", 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6693000000000003e-05 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168897 2.1 2:8f1372f3:::benchmark_data_smithi173_47195_object71889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.80172428399999, 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21982235999999999, 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168897 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643725+0000", 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643729+0000", 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643740+0000", 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1079e-05 2024-08-30T23:36:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643745+0000", 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2009999999999998e-06 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651404+0000", 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007658916 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651490+0000", 2024-08-30T23:36:22.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5811999999999998e-05 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651572+0000", 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2472999999999995e-05 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656527+0000", 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049544910000000001 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656537+0000", 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0497e-05 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863418+0000", 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20688067199999999 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863501+0000", 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2806000000000006e-05 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863546+0000", 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5430999999999999e-05 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:193992 2.7 e200/182 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head v 200'12142, mlcod=200'12139)", 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 399.669876418, 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60714938399999996, 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.548 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9599999999999999e-07 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775578+0000", 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8089999999999998e-06 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378923+0000", 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60334498999999997 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378934+0000", 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0772e-05 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379004+0000", 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9783000000000006e-05 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379024+0000", 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0044e-05 2024-08-30T23:36:22.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382681+0000", 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036575549999999998 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382721+0000", 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9734999999999998e-05 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:194011 2.1 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 399.648162345, 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82807010000000003, 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194011 2024-08-30T23:36:22.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797287+0000", 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.277e-06 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797292+0000", 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6419999999999998e-06 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797301+0000", 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0070000000000003e-06 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797307+0000", 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2809999999999997e-06 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797355+0000", 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8557000000000003e-05 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797473+0000", 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117904 2024-08-30T23:36:22.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797597+0000", 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012318100000000001 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.802546+0000", 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004948956 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.939024+0000", 2024-08-30T23:36:22.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13647832200000001 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625288+0000", 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.686263866 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625326+0000", 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8028999999999998e-05 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625356+0000", 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0077999999999999e-05 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:1186 4.4 e311/306 4:2bc10e34:::benchmark_data_smithi173_74914_object1185:head v 311'76, mlcod=311'72)", 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 140.58285631499999, 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21510685600000001, 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.559 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862593+0000", 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4200000000000005e-07 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862614+0000", 2024-08-30T23:36:22.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1129e-05 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077250+0000", 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214636047 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077258+0000", 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9789999999999993e-06 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077310+0000", 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2367999999999997e-05 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077334+0000", 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4168e-05 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077669+0000", 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033434399999999998 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077699+0000", 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9978999999999999e-05 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:145042 4.4 e366/331 4:26acdbe8:::benchmark_data_smithi173_74914_object46800:head v 366'9011, mlcod=366'9007)", 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:21.094230+0000", 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.3512178289999999, 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.077658545999999995, 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:22.564 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.094230+0000", 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.094230+0000", 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.094231+0000", 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2600000000000001e-07 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.094237+0000", 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8089999999999998e-06 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.165793+0000", 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.071555742000000006 2024-08-30T23:36:22.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.165800+0000", 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1289999999999998e-06 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.165877+0000", 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7097999999999999e-05 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.165896+0000", 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9417000000000001e-05 2024-08-30T23:36:22.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.171612+0000", 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0057152590000000003 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:21.171889+0000", 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027726600000000003 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:22.569 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:22.569 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:36:22.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:22.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:22.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:22.734 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:36:22.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:22.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:22.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:22.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:22.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:22 smithi136 ceph-mon[20207]: 4.f scrub ok 2024-08-30T23:36:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/394066320' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:22 smithi173 ceph-mon[20554]: 4.f scrub ok 2024-08-30T23:36:23.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/394066320' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:23.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:23.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:22 smithi106 ceph-mon[16603]: 4.f scrub ok 2024-08-30T23:36:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/394066320' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:23.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:23.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:23.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:23.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:23.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.511 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:23.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:23.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:23.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:23.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:23.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:23.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:23.898 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:23.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:23.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:24.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:23 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/394066320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:36:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:23 smithi136 ceph-mon[20207]: osdmap e367: 12 total, 12 up, 12 in 2024-08-30T23:36:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:23 smithi136 ceph-mon[20207]: pgmap v883: 17 pgs: 1 activating, 16 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s 2024-08-30T23:36:24.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:23 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:24.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/394066320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:36:24.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:23 smithi106 ceph-mon[16603]: osdmap e367: 12 total, 12 up, 12 in 2024-08-30T23:36:24.064 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:23 smithi106 ceph-mon[16603]: pgmap v883: 17 pgs: 1 activating, 16 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s 2024-08-30T23:36:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:23 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/394066320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:36:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:23 smithi173 ceph-mon[20554]: osdmap e367: 12 total, 12 up, 12 in 2024-08-30T23:36:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:23 smithi173 ceph-mon[20554]: pgmap v883: 17 pgs: 1 activating, 16 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 822 op/s 2024-08-30T23:36:24.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:36:24.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:24.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:24.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:24.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:24.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.423 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:24.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:24.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:24.550 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:24.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:24.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:24.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:24.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:24.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:24.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:24.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:25.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:25.080 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:25.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:25.218 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:25.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:25.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:25.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:25.453 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/99624646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.484 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:25.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:25.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/99624646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:25.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/99624646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:25.581 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:25.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:25.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:25.585 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:36:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.620 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:25.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:36:25.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:25.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:25.774 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:36:25.774 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:36:25.775 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:36:25.775 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-30T23:36:25.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1 2024-08-30T23:36:25.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:25.879 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:25.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:25.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:26.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:26.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:26.105 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:25 smithi106 systemd[1]: Stopping Ceph osd.1 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:36:26.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:26.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:26.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:26 smithi106 ceph-mon[16603]: pgmap v884: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s 2024-08-30T23:36:26.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:26.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:26.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.384 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:26.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:26.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:26 smithi136 ceph-mon[20207]: pgmap v884: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s 2024-08-30T23:36:26.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:26 smithi173 ceph-mon[20554]: pgmap v884: 17 pgs: 17 active+clean; 3.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s 2024-08-30T23:36:26.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:26.641 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:26.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:26.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:26.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:26.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:26.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:26.892 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:26.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:36:26.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:27.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:27.107 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[35847]: 2024-08-30T23:36:27.028+0000 7f66aafc5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-30T23:36:27.107 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[35847]: 2024-08-30T23:36:27.028+0000 7f66aafc5640 -1 osd.1 368 *** Got signal Terminated *** 2024-08-30T23:36:27.107 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[35847]: 2024-08-30T23:36:27.028+0000 7f66aafc5640 -1 osd.1 368 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-30T23:36:27.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:27.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:27.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:27.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:27.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:27.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:27.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:27.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:27.622 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:27.622 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:27.626 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:27.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:27.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:27.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:27.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-mon[16603]: osdmap e368: 12 total, 12 up, 12 in 2024-08-30T23:36:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-mon[16603]: osd.1 marked itself down and dead 2024-08-30T23:36:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:27 smithi106 ceph-mon[16603]: pgmap v886: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-08-30T23:36:27.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:27.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:27.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:28.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:27 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:27 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:27 smithi136 ceph-mon[20207]: osdmap e368: 12 total, 12 up, 12 in 2024-08-30T23:36:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:27 smithi136 ceph-mon[20207]: osd.1 marked itself down and dead 2024-08-30T23:36:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:27 smithi136 ceph-mon[20207]: pgmap v886: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-08-30T23:36:28.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:27 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:27 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:27 smithi173 ceph-mon[20554]: osdmap e368: 12 total, 12 up, 12 in 2024-08-30T23:36:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:27 smithi173 ceph-mon[20554]: osd.1 marked itself down and dead 2024-08-30T23:36:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:27 smithi173 ceph-mon[20554]: pgmap v886: 17 pgs: 17 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 886 op/s 2024-08-30T23:36:28.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:28.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:28.283 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:28.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:28.428 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:28.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:28.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.555 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:28.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:28.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:28.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:28.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:28.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:36:28.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:28.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:28.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:28 smithi136 ceph-mon[20207]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:36:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:28 smithi136 ceph-mon[20207]: osdmap e369: 12 total, 11 up, 12 in 2024-08-30T23:36:29.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:29.072 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.073 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:28 smithi173 ceph-mon[20554]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:36:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:28 smithi173 ceph-mon[20554]: osdmap e369: 12 total, 11 up, 12 in 2024-08-30T23:36:29.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:28 smithi106 ceph-mon[16603]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:36:29.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:28 smithi106 ceph-mon[16603]: osdmap e369: 12 total, 11 up, 12 in 2024-08-30T23:36:29.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:29.199 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:29.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:29.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:29.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:29.449 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:29.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:29.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:29.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:29.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:29.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:29.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:29.869 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:36:29.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:29.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:29.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:29 smithi136 ceph-mon[20207]: osdmap e370: 12 total, 11 up, 12 in 2024-08-30T23:36:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:29 smithi136 ceph-mon[20207]: pgmap v889: 17 pgs: 1 stale+active+clean, 1 active+undersized+degraded, 15 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2790/141012 objects degraded (1.979%) 2024-08-30T23:36:30.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:30.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:29 smithi173 ceph-mon[20554]: osdmap e370: 12 total, 11 up, 12 in 2024-08-30T23:36:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:29 smithi173 ceph-mon[20554]: pgmap v889: 17 pgs: 1 stale+active+clean, 1 active+undersized+degraded, 15 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2790/141012 objects degraded (1.979%) 2024-08-30T23:36:30.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:29 smithi106 ceph-mon[16603]: osdmap e370: 12 total, 11 up, 12 in 2024-08-30T23:36:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:29 smithi106 ceph-mon[16603]: pgmap v889: 17 pgs: 1 stale+active+clean, 1 active+undersized+degraded, 15 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s; 2790/141012 objects degraded (1.979%) 2024-08-30T23:36:30.107 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:29 smithi106 podman[128973]: 2024-08-30 23:36:29.872498719 +0000 UTC m=+2.974726722 container died 689e033b2b5fa5f8e044e7678e6464e67a16397deb98d6f557d2a592238a565a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-08-30T23:36:30.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:30.212 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:30.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:30.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:30.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:30.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:30.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:30.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:30.607 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:30 smithi106 podman[128973]: 2024-08-30 23:36:30.38229772 +0000 UTC m=+3.484525718 container cleanup 689e033b2b5fa5f8e044e7678e6464e67a16397deb98d6f557d2a592238a565a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-30T23:36:30.607 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:30 smithi106 bash[128973]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1 2024-08-30T23:36:30.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:30.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:30.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:30.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:30.952 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:30 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 2790/141012 objects degraded (1.979%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:36:30.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:30.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:30.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:31.002 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:31.020 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:30 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 2790/141012 objects degraded (1.979%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:36:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:30 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 2790/141012 objects degraded (1.979%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:36:31.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:31.111 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:31.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:31.237 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:31.279 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:30 smithi106 podman[129139]: 2024-08-30 23:36:30.952674626 +0000 UTC m=+2.605647668 container remove 689e033b2b5fa5f8e044e7678e6464e67a16397deb98d6f557d2a592238a565a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:36:31.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:31.363 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:31.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:31.489 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:31.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:31.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:31.684 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:36:31.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:31.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:31.816 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1896181013' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:31.816 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:31 smithi106 ceph-mon[16603]: pgmap v890: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s; 13644/132792 objects degraded (10.275%) 2024-08-30T23:36:31.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:31.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:31.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:31.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:31.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:32.034 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:32.034 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:36:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1896181013' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:31 smithi136 ceph-mon[20207]: pgmap v890: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s; 13644/132792 objects degraded (10.275%) 2024-08-30T23:36:32.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:32.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1896181013' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:31 smithi173 ceph-mon[20554]: pgmap v890: 17 pgs: 5 active+undersized+degraded, 12 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s; 13644/132792 objects degraded (10.275%) 2024-08-30T23:36:32.087 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:32.087 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:32.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:32.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:32.124 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:32.125 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:32.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:32.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:32.190 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:32.190 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:32.213 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:32.214 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:32.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:32.245 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:32.245 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:32.248 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:32.275 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:32.275 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:32.309 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:32.309 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:32.342 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:32.342 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:32.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:32.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:32.455 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:36:32.455 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:36:32.455 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:36:32.455 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:36:32.455 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:36:32.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:32.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:32.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:32.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:32.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:32.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:32.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:32.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:32.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:33.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:33.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:33.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:32 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1896181013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:32 smithi136 ceph-mon[20207]: osdmap e371: 12 total, 11 up, 12 in 2024-08-30T23:36:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1255097292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:32 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1896181013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:32 smithi173 ceph-mon[20554]: osdmap e371: 12 total, 11 up, 12 in 2024-08-30T23:36:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1255097292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:32 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1896181013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:32 smithi106 ceph-mon[16603]: osdmap e371: 12 total, 11 up, 12 in 2024-08-30T23:36:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1255097292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:33.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:33.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:33.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:33.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:33.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:33.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:33.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:36:33.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:33.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:33.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:33.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:33.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:33.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:36:33.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:36:33.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:33.924 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:33.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:34.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:33 smithi136 ceph-mon[20207]: pgmap v892: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 15908/128865 objects degraded (12.345%) 2024-08-30T23:36:34.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:33 smithi173 ceph-mon[20554]: pgmap v892: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 15908/128865 objects degraded (12.345%) 2024-08-30T23:36:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:33 smithi106 ceph-mon[16603]: pgmap v892: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 865 op/s; 15908/128865 objects degraded (12.345%) 2024-08-30T23:36:34.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:34.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:34.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:34.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:34.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:34.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:34.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:34.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:34.615 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:34.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:34.701 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:34.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:34.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:34.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:34.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:34.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:35.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:35.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:35.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:35.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75397 2.1 e132/97 2:8c19b29d:::benchmark_data_smithi173_47195_object75396:head v 132'4751, mlcod=132'4748)", 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.69077721899998, 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22761530499999999, 2024-08-30T23:36:35.316 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609011+0000", 2024-08-30T23:36:35.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609012+0000", 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8199999999999998e-07 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.609036+0000", 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4414999999999999e-05 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834165+0000", 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22512892100000001 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834172+0000", 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2589999999999997e-06 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834226+0000", 2024-08-30T23:36:35.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3919999999999999e-05 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.834249+0000", 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.298e-05 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836584+0000", 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023353750000000002 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.836626+0000", 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2552999999999997e-05 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75405 2.1 e132/97 2:80371289:::benchmark_data_smithi173_47195_object75404:head v 132'4752, mlcod=132'4748)", 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.67055020800001, 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.240592844, 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.321 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629238+0000", 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629239+0000", 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.629261+0000", 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2293000000000002e-05 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867840+0000", 2024-08-30T23:36:35.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23857915299999999 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867849+0000", 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4940000000000003e-06 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867913+0000", 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3818000000000005e-05 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.867935+0000", 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2024000000000001e-05 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869794+0000", 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001859018 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.869831+0000", 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7239e-05 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.326 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:75429 2.1 e132/97 2:8fc7ac90:::benchmark_data_smithi173_47195_object75428:head v 132'4755, mlcod=132'4752)", 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 578.34089019500004, 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24468055399999999, 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958898+0000", 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958899+0000", 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1939999999999999e-06 2024-08-30T23:36:35.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:56.958944+0000", 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4851999999999997e-05 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201529+0000", 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24258502100000001 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201540+0000", 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0744000000000001e-05 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201630+0000", 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0168999999999997e-05 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.201650+0000", 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9477999999999999e-05 2024-08-30T23:36:35.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203542+0000", 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00189236 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:26:57.203579+0000", 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6736000000000002e-05 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.331 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:100667 2.6 e147/142 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head v 147'6317, mlcod=147'6313)", 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 541.95171035099997, 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.273264648, 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.332 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348078+0000", 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348079+0000", 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1459999999999999e-06 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.348086+0000", 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6649999999999997e-06 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615008+0000", 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26692174899999999 2024-08-30T23:36:35.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615016+0000", 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0909999999999993e-06 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615082+0000", 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6359000000000006e-05 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.615102+0000", 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9945999999999998e-05 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621308+0000", 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006205828 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:33.621343+0000", 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4863999999999997e-05 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.336 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:194011 2.1 e200/97 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head v 200'12204, mlcod=200'12201)", 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 412.50211156, 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82744930299999997, 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797678+0000", 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1800000000000005e-07 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797683+0000", 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6529999999999999e-06 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620982+0000", 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82329911700000002 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620996+0000", 2024-08-30T23:36:35.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3995999999999999e-05 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621133+0000", 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013682700000000001 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621198+0000", 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4368000000000004e-05 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625091+0000", 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003893194 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625126+0000", 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5429999999999998e-05 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.341 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:103424 3.2 3:4caab458:::benchmark_data_smithi173_60780_object103423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 294.04016495500002, 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.221222537, 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103424 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259624+0000", 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0800000000000004e-07 2024-08-30T23:36:35.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259682+0000", 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7364999999999997e-05 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259692+0000", 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0686000000000001e-05 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259698+0000", 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.361e-06 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261702+0000", 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020041289999999999 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261778+0000", 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6148000000000003e-05 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261872+0000", 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4598000000000005e-05 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.263845+0000", 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001972419 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.264707+0000", 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086252099999999999 2024-08-30T23:36:35.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480753+0000", 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.216045457 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480795+0000", 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2605000000000003e-05 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480846+0000", 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0540000000000001e-05 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198168 3.2 3:46c77281:::benchmark_data_smithi173_60780_object94714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 182.33757228100001, 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22735555199999999, 2024-08-30T23:36:35.348 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198168 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962217+0000", 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2099999999999995e-07 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962221+0000", 2024-08-30T23:36:35.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999999e-06 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962234+0000", 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2412000000000001e-05 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962241+0000", 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6789999999999999e-06 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962296+0000", 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5347999999999999e-05 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962411+0000", 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011498299999999999 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962524+0000", 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011293700000000001 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.967040+0000", 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045164259999999996 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.968062+0000", 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010217270000000001 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189490+0000", 2024-08-30T23:36:35.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22142835899999999 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189535+0000", 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4338999999999997e-05 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189572+0000", 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6990999999999999e-05 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198178 3.2 3:47f13797:::benchmark_data_smithi173_60780_object94724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 182.326782853, 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21670980100000001, 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198178 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.355 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973007+0000", 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973013+0000", 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1140000000000002e-06 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973023+0000", 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0516000000000001e-05 2024-08-30T23:36:35.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973030+0000", 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8390000000000001e-06 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973072+0000", 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1919000000000003e-05 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973158+0000", 2024-08-30T23:36:35.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6148000000000003e-05 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973234+0000", 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6165e-05 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.976842+0000", 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036077290000000001 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.978153+0000", 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013109930000000001 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189617+0000", 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21146347300000001 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189674+0000", 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7082e-05 2024-08-30T23:36:35.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189715+0000", 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1656999999999998e-05 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206885 3.5 e300/206 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head v 300'12859, mlcod=300'12855)", 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:36:35.361 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.25854511399999, 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23467711799999999, 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.362 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041244+0000", 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041249+0000", 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2360000000000004e-06 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.363 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271945+0000", 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.230695859 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271961+0000", 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5815999999999999e-05 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272088+0000", 2024-08-30T23:36:35.364 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126721 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272127+0000", 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8872000000000003e-05 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275888+0000", 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037616350000000002 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.365 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275920+0000", 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2255000000000002e-05 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206897 3.b 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:36:35.366 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.242704105, 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58358057900000004, 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206897 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.367 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057085+0000", 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3399999999999999e-07 2024-08-30T23:36:35.368 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057088+0000", 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8609999999999998e-06 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057094+0000", 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5429999999999999e-06 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.369 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057098+0000", 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9720000000000003e-06 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057149+0000", 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0683000000000002e-05 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057264+0000", 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114772 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.370 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057387+0000", 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012322799999999999 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.061158+0000", 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037710170000000002 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062218+0000", 2024-08-30T23:36:35.371 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001059948 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640580+0000", 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57836202199999998 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640625+0000", 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5116e-05 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640665+0000", 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9882999999999997e-05 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206901 3.7 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.23702284699999, 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38317995100000002, 2024-08-30T23:36:35.373 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206901 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:36:35.374 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062767+0000", 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.375 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062771+0000", 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6779999999999999e-06 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062785+0000", 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3839000000000001e-05 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.376 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062793+0000", 2024-08-30T23:36:35.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:35.378 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:36:35.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063873+0000", 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001080632 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063965+0000", 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1717000000000003e-05 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:36:35.379 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.064031+0000", 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5729999999999996e-05 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.067745+0000", 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037140430000000002 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.071385+0000", 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036407620000000001 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445862+0000", 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37447608799999998 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445912+0000", 2024-08-30T23:36:35.381 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0059999999999998e-05 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445946+0000", 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3927000000000002e-05 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206900 3.5 e300/206 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head v 300'12860, mlcod=300'12856)", 2024-08-30T23:36:35.382 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 171.23658158000001, 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36104319299999998, 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063208+0000", 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.384 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063213+0000", 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0080000000000002e-06 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420406+0000", 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35719326299999998 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420426+0000", 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0253000000000001e-05 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420597+0000", 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170786 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420630+0000", 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3578999999999998e-05 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424218+0000", 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035875490000000002 2024-08-30T23:36:35.386 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424250+0000", 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2030999999999999e-05 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62091 4.4 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:36:35.387 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 98.134120616000004, 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318614905, 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62091 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.388 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165670+0000", 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.389 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165713+0000", 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3368000000000002e-05 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165732+0000", 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9031000000000001e-05 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165740+0000", 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8050000000000007e-06 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165802+0000", 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2279999999999993e-05 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165946+0000", 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014352899999999999 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.166046+0000", 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100046 2024-08-30T23:36:35.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169317+0000", 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003271267 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169909+0000", 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059201200000000005 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.392 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484178+0000", 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31426850299999998 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484234+0000", 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6678000000000002e-05 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484283+0000", 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8457e-05 2024-08-30T23:36:35.393 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62105 4.4 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 98.120536283999996, 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34846845199999998, 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:36:35.394 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62105 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:36:35.395 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179254+0000", 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179297+0000", 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3108000000000001e-05 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.396 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179311+0000", 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.345e-05 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179316+0000", 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179371+0000", 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5316999999999999e-05 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179508+0000", 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643600000000001 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179665+0000", 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015680499999999999 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182071+0000", 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024068090000000002 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182226+0000", 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015503899999999999 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527606+0000", 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34537996300000001 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527669+0000", 2024-08-30T23:36:35.399 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2966999999999995e-05 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527721+0000", 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1224000000000003e-05 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62174 4.4 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.775324503999997, 2024-08-30T23:36:35.400 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57863658799999995, 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62174 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:36:35.401 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524466+0000", 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0600000000000002e-06 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.402 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524501+0000", 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4635999999999998e-05 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524521+0000", 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0193000000000001e-05 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524531+0000", 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7780000000000002e-06 2024-08-30T23:36:35.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524604+0000", 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3646000000000006e-05 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524763+0000", 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015852999999999999 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.404 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524940+0000", 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017771799999999999 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527733+0000", 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027928789999999998 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.528529+0000", 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000795487 2024-08-30T23:36:35.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103030+0000", 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57450106899999998 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103068+0000", 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8398000000000001e-05 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.406 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103100+0000", 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2193999999999997e-05 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62186 4.4 4:257aab52:::benchmark_data_smithi173_74914_object62185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.746212383, 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55186797300000001, 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.407 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62186 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.408 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553578+0000", 2024-08-30T23:36:35.409 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553612+0000", 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3773999999999999e-05 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553627+0000", 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5410999999999999e-05 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.410 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553637+0000", 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0101e-05 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553709+0000", 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2365000000000003e-05 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553885+0000", 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017522699999999999 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.554065+0000", 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000180101 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556530+0000", 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024650240000000001 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556570+0000", 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9975999999999998e-05 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105343+0000", 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54877291299999997 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105399+0000", 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6648999999999999e-05 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105444+0000", 2024-08-30T23:36:35.413 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4660000000000003e-05 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:71384 4.e e342/327 4:706b20d0:::benchmark_data_smithi173_74914_object71383:head v 342'4468, mlcod=342'4465)", 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:06.472375+0000", 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 88.827413131, 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21530670800000001, 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.414 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.472375+0000", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.472375+0000", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.472376+0000", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.122e-06 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.415 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.472480+0000", 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010395600000000001 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.685017+0000", 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21253661300000001 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.685025+0000", 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5909999999999998e-06 2024-08-30T23:36:35.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.685086+0000", 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1877000000000004e-05 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.685102+0000", 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5869999999999999e-05 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.687624+0000", 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025216499999999998 2024-08-30T23:36:35.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:06.687682+0000", 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8029000000000003e-05 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:93692 4.4 4:24d1e68e:::benchmark_data_smithi173_74914_object93691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.276008732999998, 2024-08-30T23:36:35.418 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28134646200000002, 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 93692 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023781+0000", 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.420 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023840+0000", 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9305999999999998e-05 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023854+0000", 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3773e-05 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023872+0000", 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7708000000000001e-05 2024-08-30T23:36:35.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023930+0000", 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8007999999999998e-05 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024050+0000", 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120209 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.422 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024158+0000", 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010744299999999999 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026599+0000", 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024417430000000001 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026641+0000", 2024-08-30T23:36:35.423 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305020+0000", 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27837906600000001 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305079+0000", 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9080999999999999e-05 2024-08-30T23:36:35.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305126+0000", 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6657999999999999e-05 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:93692 4.4 e350/331 ondisk, result = 0)", 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:36:35.425 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.272732395999995, 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27807699200000002, 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.426 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027057+0000", 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027060+0000", 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.01e-06 2024-08-30T23:36:35.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304915+0000", 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27785516100000002 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304921+0000", 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1800000000000001e-06 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.428 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305007+0000", 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5783000000000002e-05 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305015+0000", 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7659999999999998e-06 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305133+0000", 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118472 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.429 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:107608 4.f e354/306 4:fc26249f:::benchmark_data_smithi173_74914_object9366:head v 354'6690, mlcod=354'6687)", 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 53.801975325000001, 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22329728099999999, 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:36:35.430 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497814+0000", 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3399999999999998e-07 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.431 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497821+0000", 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8900000000000001e-06 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717759+0000", 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21993858999999999 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717767+0000", 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6000000000000001e-06 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717856+0000", 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8516000000000004e-05 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717893+0000", 2024-08-30T23:36:35.433 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7342999999999998e-05 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721075+0000", 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031817579999999998 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721110+0000", 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5750000000000002e-05 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.434 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:36:35.435 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:36:35.435 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:36:35.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:35.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:35.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:35.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:35.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:35.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:35.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:35.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:35.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:35.817 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:35.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:35 smithi106 ceph-mon[16603]: pgmap v893: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 14982/123207 objects degraded (12.160%) 2024-08-30T23:36:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 14982/123207 objects degraded (12.160%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:35.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:36:35.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:35.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:35.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:35 smithi136 ceph-mon[20207]: pgmap v893: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 14982/123207 objects degraded (12.160%) 2024-08-30T23:36:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 14982/123207 objects degraded (12.160%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:36.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:36.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:35 smithi173 ceph-mon[20554]: pgmap v893: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s; 14982/123207 objects degraded (12.160%) 2024-08-30T23:36:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 14982/123207 objects degraded (12.160%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:36.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:36.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:36.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:36:36.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:36.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:36.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:36.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:36.525 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:36 smithi106 podman[129822]: 2024-08-30 23:36:36.251541236 +0000 UTC m=+0.025754131 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:36:36.525 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:36 smithi106 podman[129822]: 2024-08-30 23:36:36.394643855 +0000 UTC m=+0.168856729 container create 114962f35a3057c7a8301c43dae8cac4edc666e9e91a0bdd8cb4f196a0941919 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-08-30T23:36:36.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:36.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:36:36.560 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:36.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:36.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:36.713 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:36.806 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:36 smithi106 podman[129822]: 2024-08-30 23:36:36.697789413 +0000 UTC m=+0.472002296 container init 114962f35a3057c7a8301c43dae8cac4edc666e9e91a0bdd8cb4f196a0941919 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS) 2024-08-30T23:36:36.806 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:36 smithi106 podman[129822]: 2024-08-30 23:36:36.704889393 +0000 UTC m=+0.479102274 container start 114962f35a3057c7a8301c43dae8cac4edc666e9e91a0bdd8cb4f196a0941919 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.license=GPLv2) 2024-08-30T23:36:36.806 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:36 smithi106 podman[129822]: 2024-08-30 23:36:36.806136385 +0000 UTC m=+0.580349267 container attach 114962f35a3057c7a8301c43dae8cac4edc666e9e91a0bdd8cb4f196a0941919 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-30T23:36:36.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:36.819 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:36.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:36.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:36.983 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:36.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:37.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:37.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:159361 4.1 e371/306 4:8201eb06:::benchmark_data_smithi173_74914_object61119:head v 371'10102, mlcod=371'10098)", 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:37.153473+0000", 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012714402, 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012741518, 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153473+0000", 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153473+0000", 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153474+0000", 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.051e-06 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153481+0000", 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8460000000000002e-06 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153513+0000", 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1353999999999998e-05 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153521+0000", 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2989999999999997e-06 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153579+0000", 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7788999999999998e-05 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.153599+0000", 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0530000000000002e-05 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:159363 4.1 e371/306 4:83f27349:::benchmark_data_smithi173_74914_object61121:head v 371'10103, mlcod=371'10099)", 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:37.158462+0000", 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0077258020000000004, 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078053059999999997, 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.158462+0000", 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.158462+0000", 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.158463+0000", 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.079e-06 2024-08-30T23:36:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.158471+0000", 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2840000000000006e-06 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.158514+0000", 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2580999999999998e-05 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.158522+0000", 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1240000000000005e-06 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.159887+0000", 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001365237 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.159930+0000", 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2314000000000002e-05 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:159368 4.1 e371/306 4:89510437:::benchmark_data_smithi173_74914_object61126:head v 371'10104, mlcod=371'10099)", 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:37.163162+0000", 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0030253839999999999, 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031528149999999998, 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.163162+0000", 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.163162+0000", 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.163163+0000", 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.048e-06 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.163171+0000", 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.52e-06 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.163211+0000", 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0111000000000003e-05 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.163219+0000", 2024-08-30T23:36:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.909e-06 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.164654+0000", 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001434608 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:37.164672+0000", 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8712e-05 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:36:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 3 2024-08-30T23:36:37.180 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:36:37.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:37.250 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:37.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:37.357 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:36:37 smithi106 podman[129822]: 2024-08-30 23:36:37.096789462 +0000 UTC m=+0.871002347 container died 114962f35a3057c7a8301c43dae8cac4edc666e9e91a0bdd8cb4f196a0941919 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-08-30T23:36:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:37.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:37.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:37.510 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:37.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:37.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:37.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:37.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:37.770 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:37.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:37.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:37.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:37.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:38.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:38.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:38.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:38.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:36:38.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:38.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:38.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1.service' 2024-08-30T23:36:38.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-30T23:36:38.363 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-30T23:36:38.363 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-08-30T23:36:38.363 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi106.front.sepia.ceph.com 2024-08-30T23:36:38.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-08-30T23:36:38.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:38.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:38.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:38.569 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:38.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:38.612 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:36:38.639 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:38.639 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:38.642 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:36:38.671 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:38.671 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:38.672 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:38.697 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:38.697 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:38.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:38.720 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:38.721 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:38.763 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:38.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:38 smithi136 ceph-mon[20207]: pgmap v894: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 945 op/s; 14668/117603 objects degraded (12.472%) 2024-08-30T23:36:38.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/785895442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:38.794 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:38.794 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:38.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:38 smithi106 ceph-mon[16603]: pgmap v894: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 945 op/s; 14668/117603 objects degraded (12.472%) 2024-08-30T23:36:38.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/785895442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:38.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:38.821 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:38.821 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:38.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:38 smithi173 ceph-mon[20554]: pgmap v894: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 945 op/s; 14668/117603 objects degraded (12.472%) 2024-08-30T23:36:38.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/785895442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:38.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:38.844 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:38.844 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:38.871 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:38.871 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:38.895 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:38.896 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:38.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:38.931 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:38.932 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:38.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:38.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:39.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:39.081 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:39.128 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:39.129 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:39.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:39.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:39.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:39.334 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:39.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:39.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:39.549 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:36:39.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:39.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:39.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:39.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:39.748 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3987041857' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:39.749 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:39 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:39.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:36:39.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3987041857' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:39 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:39.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:39.847 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3987041857' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:39.847 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:39 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:36:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:36:39.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:36:39.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:39.966 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:39.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:40.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:40.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:40.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:40.211 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:40.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:40.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:40.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:40.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:40.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:40.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:40.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:40.665 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:40 smithi136 ceph-mon[20207]: pgmap v895: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 14035/113862 objects degraded (12.326%) 2024-08-30T23:36:40.666 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:40 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:40.666 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:40 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:36:40.666 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:40 smithi136 ceph-mon[20207]: osdmap e372: 12 total, 11 up, 12 in 2024-08-30T23:36:40.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:40.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:40.804 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:40.804 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:40 smithi173 ceph-mon[20554]: pgmap v895: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 14035/113862 objects degraded (12.326%) 2024-08-30T23:36:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:40 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:40 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:36:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:40 smithi173 ceph-mon[20554]: osdmap e372: 12 total, 11 up, 12 in 2024-08-30T23:36:40.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:40 smithi106 ceph-mon[16603]: pgmap v895: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 904 op/s; 14035/113862 objects degraded (12.326%) 2024-08-30T23:36:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:40 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:40 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:36:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:40 smithi106 ceph-mon[16603]: osdmap e372: 12 total, 11 up, 12 in 2024-08-30T23:36:40.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:40.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:40.962 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:36:40.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:41.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:41.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:41.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:41.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:41.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:41.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:41.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:41.494 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:41.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: osdmap e373: 12 total, 11 up, 12 in 2024-08-30T23:36:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 14035/113862 objects degraded (12.326%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:36:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:36:41.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: 4.7 scrub starts 2024-08-30T23:36:41.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:41 smithi106 ceph-mon[16603]: pgmap v898: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 12821/103908 objects degraded (12.339%) 2024-08-30T23:36:41.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:41.653 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:41.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:41.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: osdmap e373: 12 total, 11 up, 12 in 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 14035/113862 objects degraded (12.326%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: 4.7 scrub starts 2024-08-30T23:36:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:41 smithi136 ceph-mon[20207]: pgmap v898: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 12821/103908 objects degraded (12.339%) 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: osdmap e373: 12 total, 11 up, 12 in 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 14035/113862 objects degraded (12.326%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: 4.7 scrub starts 2024-08-30T23:36:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:41 smithi173 ceph-mon[20554]: pgmap v898: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 2.1 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 12821/103908 objects degraded (12.339%) 2024-08-30T23:36:41.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:41.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:36:41.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:41.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:42.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:42.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:42.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:36:42.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:42.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:42.203 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:42.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:42.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:42.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:42.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:42.428 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:42.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:42.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:42.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:42.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:42.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:42.794 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:42.795 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:42.795 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:36:42.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:42.820 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:42 smithi136 ceph-mon[20207]: 4.7 scrub ok 2024-08-30T23:36:42.822 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:42.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:42.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:42.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:42.880 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:42.881 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:42.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:42.917 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:42 smithi106 ceph-mon[16603]: 4.7 scrub ok 2024-08-30T23:36:42.917 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:42.918 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:42.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:42.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:42.951 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:42.951 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:42.959 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:36:42.959 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:36:42.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.959 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:164672 4.9 4:9472e160:::benchmark_data_smithi173_74914_object66430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-08-30T23:36:42.959 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:42.952541+0000", 2024-08-30T23:36:42.959 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059231910000000004, 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005943561, 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 164672 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:42.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952541+0000", 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952541+0000", 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952543+0000", 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6610000000000001e-06 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952550+0000", 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5889999999999997e-06 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:42.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952566+0000", 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5532999999999998e-05 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952574+0000", 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1859999999999992e-06 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952642+0000", 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8262000000000005e-05 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952750+0000", 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107271 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.952830+0000", 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9887000000000003e-05 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:42.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:42.957752+0000", 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049228509999999998 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:36:42.965 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:36:42.978 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:42.978 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:43.000 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:43.000 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:43.027 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:43.027 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:43.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:43.051 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:43.052 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:43.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:43.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:42 smithi173 ceph-mon[20554]: 4.7 scrub ok 2024-08-30T23:36:43.084 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:43.084 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:43.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:43.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:43.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:43.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:43.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:36:43.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:43.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:43.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:43.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:43.625 INFO:teuthology.orchestra.run.smithi106.stdout:4.d 2024-08-30T23:36:43.625 INFO:teuthology.orchestra.run.smithi106.stdout:4.b 2024-08-30T23:36:43.625 INFO:teuthology.orchestra.run.smithi106.stdout:4.f 2024-08-30T23:36:43.625 INFO:teuthology.orchestra.run.smithi106.stdout:4.a 2024-08-30T23:36:43.625 INFO:teuthology.orchestra.run.smithi106.stdout:4.6 2024-08-30T23:36:43.626 INFO:teuthology.orchestra.run.smithi106.stdout:4.3 2024-08-30T23:36:43.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:36:43.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:43.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:43.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:43.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:36:43.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:36:43.922 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:43.923 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1037850774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:43.923 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:43 smithi106 ceph-mon[16603]: pgmap v899: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 2.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s; 12280/99759 objects degraded (12.310%) 2024-08-30T23:36:43.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:43.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:43.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:43.958 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:44.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1037850774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:43 smithi136 ceph-mon[20207]: pgmap v899: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 2.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s; 12280/99759 objects degraded (12.310%) 2024-08-30T23:36:44.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1037850774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:43 smithi173 ceph-mon[20554]: pgmap v899: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 2.0 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s; 12280/99759 objects degraded (12.310%) 2024-08-30T23:36:44.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:44.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:44.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:44.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:44.332 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:44.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:44.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:44.497 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:36:44.497 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:36:44.497 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:36:44.497 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:36:44.497 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:36:44.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:44.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:44.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:44.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:44.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 4.f --file /var/log/ceph/exp.4.f.1 2024-08-30T23:36:44.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:44.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:44.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:44.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:44.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:45.027 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:36:45.038 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:44 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:45.038 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:44 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:45.039 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:44 smithi106 ceph-mon[16603]: 4.5 scrub starts 2024-08-30T23:36:45.039 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:44 smithi106 ceph-mon[16603]: 4.4 scrub starts 2024-08-30T23:36:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:44 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:44 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:44 smithi136 ceph-mon[20207]: 4.5 scrub starts 2024-08-30T23:36:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:44 smithi136 ceph-mon[20207]: 4.4 scrub starts 2024-08-30T23:36:45.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:44 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:44 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:44 smithi173 ceph-mon[20554]: 4.5 scrub starts 2024-08-30T23:36:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:44 smithi173 ceph-mon[20554]: 4.4 scrub starts 2024-08-30T23:36:45.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:45.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:36:45.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:45.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:45.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:45.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:45.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:45.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:45.533 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:45.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:45.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:45.684 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:36:45.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:45.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:45.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:36:45.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:45.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:45.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:45.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: 4.5 scrub ok 2024-08-30T23:36:45.892 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: osdmap e374: 12 total, 11 up, 12 in 2024-08-30T23:36:45.893 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: 4.4 scrub ok 2024-08-30T23:36:45.893 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: pgmap v901: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 1.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 11464/95058 objects degraded (12.060%) 2024-08-30T23:36:45.893 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/127547120' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:45.893 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:45 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 12280/99759 objects degraded (12.310%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:45.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:45.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:45.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:45.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:45.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: 4.5 scrub ok 2024-08-30T23:36:45.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: osdmap e374: 12 total, 11 up, 12 in 2024-08-30T23:36:45.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: 4.4 scrub ok 2024-08-30T23:36:45.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: pgmap v901: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 1.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 11464/95058 objects degraded (12.060%) 2024-08-30T23:36:45.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/127547120' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:45.945 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:45 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 12280/99759 objects degraded (12.310%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:46.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:46.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: 4.5 scrub ok 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: osdmap e374: 12 total, 11 up, 12 in 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: 4.4 scrub ok 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: pgmap v901: 17 pgs: 1 active+clean+scrubbing, 6 active+undersized+degraded, 10 active+clean; 1.9 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s; 11464/95058 objects degraded (12.060%) 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/127547120' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:45 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 12280/99759 objects degraded (12.310%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 146/138 [PushReplyOp(2:ee294341:::benchmark_data_smithi173_47195_object15777:head)])", 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.88083354299999, 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.001463438, 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.076 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191032+0000", 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:28.191037+0000", 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2749999999999997e-06 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181108+0000", 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99007171599999999 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.181115+0000", 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.189300+0000", 2024-08-30T23:36:46.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081848760000000007 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:29.192495+0000", 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031953250000000002 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(2.7 153/138 [PushReplyOp(2:eebfff31:::benchmark_data_smithi173_47195_object26886:head)])", 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:36:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 541.14683463100005, 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21501724999999999, 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925031+0000", 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.09e-07 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:44.925033+0000", 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.419e-06 2024-08-30T23:36:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136148+0000", 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.211115204 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136156+0000", 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1880000000000002e-06 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.136222+0000", 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5320999999999993e-05 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:27:45.140048+0000", 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003826009 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:126363 2.7 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 519.34262873700004, 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221802834, 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 126363 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729236+0000", 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729238+0000", 2024-08-30T23:36:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9360000000000002e-06 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729244+0000", 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8819999999999998e-06 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729258+0000", 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3994e-05 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729265+0000", 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0570000000000004e-06 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729318+0000", 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2410000000000001e-05 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729417+0000", 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9414999999999996e-05 2024-08-30T23:36:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.729508+0000", 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1209e-05 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.732169+0000", 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026607919999999999 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.737250+0000", 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050804370000000001 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949780+0000", 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21253043999999999 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.949815+0000", 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5308000000000001e-05 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:06.951039+0000", 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001223954 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193992 2.7 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 423.29661440899997, 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60785984599999998, 2024-08-30T23:36:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193992 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775252+0000", 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.528e-06 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775258+0000", 2024-08-30T23:36:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.485e-06 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775273+0000", 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5355000000000001e-05 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775281+0000", 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5140000000000004e-06 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775342+0000", 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1334000000000006e-05 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775448+0000", 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010587 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775514+0000", 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5747000000000006e-05 2024-08-30T23:36:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.779105+0000", 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035911110000000001 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.780025+0000", 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092040100000000005 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383045+0000", 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60302017900000005 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383083+0000", 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7670999999999999e-05 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383111+0000", 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7651e-05 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:22811 3.c e215/206 3:362f756c:::benchmark_data_smithi173_60780_object22810:head v 215'1421, mlcod=215'1418)", 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 380.06647727900003, 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32690804400000001, 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005389+0000", 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005420+0000", 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1340000000000001e-05 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331407+0000", 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32598661899999998 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331414+0000", 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7409999999999992e-06 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331466+0000", 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1319000000000001e-05 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331490+0000", 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4544000000000001e-05 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332257+0000", 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076687200000000004 2024-08-30T23:36:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332296+0000", 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8828000000000002e-05 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206871 3.a 3:5d60dc93:::benchmark_data_smithi173_60780_object103417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 182.05131676400001, 2024-08-30T23:36:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45068615499999998, 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206871 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020550+0000", 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.559e-06 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020557+0000", 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5810000000000002e-06 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020573+0000", 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6767999999999999e-05 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020581+0000", 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020645+0000", 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3559000000000006e-05 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020750+0000", 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010555100000000001 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020853+0000", 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103207 2024-08-30T23:36:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.025592+0000", 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047385489999999999 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.053826+0000", 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028234459999999999 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471165+0000", 2024-08-30T23:36:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41733862300000002 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471204+0000", 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8769000000000001e-05 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471235+0000", 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0818000000000002e-05 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24732 4.a 4:5449864d:::benchmark_data_smithi173_74914_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.56297127900001, 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24409322, 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24732 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9800000000000003e-07 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508987+0000", 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2602999999999994e-05 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509002+0000", 2024-08-30T23:36:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509009+0000", 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509048+0000", 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9002999999999999e-05 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509127+0000", 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9573000000000005e-05 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509220+0000", 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2218000000000004e-05 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.510205+0000", 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098522999999999992 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.550346+0000", 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040140995999999998 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752876+0000", 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20253054600000001 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752927+0000", 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0686999999999998e-05 2024-08-30T23:36:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752987+0000", 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9998999999999999e-05 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24732 4.a 4:5449864d:::benchmark_data_smithi173_74914_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.55714977700001, 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23822328400000001, 2024-08-30T23:36:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for ondisk", 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24732 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514716+0000", 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4300000000000003e-07 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514742+0000", 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6389000000000001e-05 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514749+0000", 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0879999999999997e-06 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514755+0000", 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999998e-06 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514806+0000", 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1598999999999998e-05 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for ondisk", 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514835+0000", 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8839999999999998e-05 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752939+0000", 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23810340499999999 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24761 4.5 4:a203c2b9:::benchmark_data_smithi173_74914_object24760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.541700+0000", 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.53016501299999, 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21552817699999999, 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24761 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541700+0000", 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541700+0000", 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541703+0000", 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6259999999999999e-06 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541748+0000", 2024-08-30T23:36:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4728999999999998e-05 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541777+0000", 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9433000000000001e-05 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541796+0000", 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9219999999999999e-05 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541908+0000", 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111381 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542074+0000", 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016683400000000001 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,10", 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542164+0000", 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9829999999999999e-05 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.543031+0000", 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086726299999999995 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752709+0000", 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20967720500000001 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.757122+0000", 2024-08-30T23:36:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044134389999999999 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.757184+0000", 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1823999999999996e-05 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.757228+0000", 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4393e-05 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49429 4.9 4:90c7f456:::benchmark_data_smithi173_74914_object49428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.629894+0000", 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.441971053, 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21828159899999999, 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49429 2024-08-30T23:36:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629894+0000", 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629894+0000", 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629895+0000", 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2600000000000002e-07 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629925+0000", 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0862999999999999e-05 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629932+0000", 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7209999999999997e-06 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629937+0000", 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6210000000000003e-06 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630472+0000", 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053525800000000004 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630479+0000", 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6769999999999998e-06 2024-08-30T23:36:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.840168+0000", 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20968874400000001 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.840273+0000", 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105151 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.840407+0000", 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013429499999999999 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843607+0000", 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031995579999999999 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843793+0000", 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018662400000000001 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848052+0000", 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042584820000000001 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848110+0000", 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8057999999999999e-05 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848176+0000", 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6020999999999997e-05 2024-08-30T23:36:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49435 4.9 4:94174f8f:::benchmark_data_smithi173_74914_object49434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.44192131299999, 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21848274200000001, 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49435 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:36:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9000000000000002e-07 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629978+0000", 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3299000000000001e-05 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629985+0000", 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7209999999999994e-06 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629995+0000", 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630501+0000", 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050542899999999995 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630507+0000", 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5889999999999999e-06 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.841699+0000", 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21119121399999999 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.841812+0000", 2024-08-30T23:36:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113396 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.841980+0000", 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016822300000000001 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843801+0000", 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018204689999999999 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845139+0000", 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013380740000000001 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848343+0000", 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032047180000000001 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848374+0000", 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0738999999999998e-05 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848427+0000", 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2364000000000002e-05 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49437 4.9 4:992ba69c:::benchmark_data_smithi173_74914_object49436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:36:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.441862384, 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21855950099999999, 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49437 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8100000000000003e-07 2024-08-30T23:36:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630037+0000", 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3992000000000003e-05 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630045+0000", 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630050+0000", 2024-08-30T23:36:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7219999999999999e-06 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630529+0000", 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000479073 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630535+0000", 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8880000000000002e-06 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843987+0000", 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21345172600000001 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.844072+0000", 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5761999999999996e-05 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.844162+0000", 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9186999999999998e-05 2024-08-30T23:36:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848231+0000", 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040693170000000002 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848246+0000", 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5461e-05 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848480+0000", 2024-08-30T23:36:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023351099999999999 2024-08-30T23:36:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848511+0000", 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0855000000000002e-05 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848562+0000", 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1625000000000001e-05 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49440 4.9 4:9de739e2:::benchmark_data_smithi173_74914_object49439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.44180814800001, 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22086719499999999, 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49440 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:36:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5999999999999999e-07 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630090+0000", 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2196000000000001e-05 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630096+0000", 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3160000000000002e-06 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630102+0000", 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9050000000000002e-06 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630553+0000", 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045081400000000001 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630557+0000", 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6979999999999997e-06 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845169+0000", 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21461201499999999 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845239+0000", 2024-08-30T23:36:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0062999999999996e-05 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845330+0000", 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0912e-05 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848252+0000", 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002922043 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848274+0000", 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1472999999999999e-05 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850752+0000", 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024776540000000001 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850813+0000", 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1593999999999993e-05 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850924+0000", 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011105200000000001 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49445 4.9 4:915080b3:::benchmark_data_smithi173_74914_object49444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.441756676, 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22553746099999999, 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49445 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630109+0000", 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630137+0000", 2024-08-30T23:36:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8206999999999999e-05 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630143+0000", 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0329999999999997e-06 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630148+0000", 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4769999999999997e-06 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630574+0000", 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042622100000000001 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630579+0000", 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9989999999999999e-06 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846364+0000", 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21578499100000001 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846489+0000", 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012525599999999999 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846624+0000", 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013508199999999999 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848317+0000", 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016928519999999999 2024-08-30T23:36:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848441+0000", 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012401 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855479+0000", 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070380499999999997 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855541+0000", 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1371000000000006e-05 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855646+0000", 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010543300000000001 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62188 4.9 4:9d0786de:::benchmark_data_smithi173_74914_object62187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.559263+0000", 2024-08-30T23:36:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.512602219, 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21637304099999999, 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62188 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559263+0000", 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559263+0000", 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559265+0000", 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5689999999999999e-06 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559316+0000", 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1032999999999997e-05 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559328+0000", 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2824e-05 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559333+0000", 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9529999999999998e-06 2024-08-30T23:36:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559422+0000", 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8231000000000003e-05 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559545+0000", 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123783 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559700+0000", 2024-08-30T23:36:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015455199999999999 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.561474+0000", 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017739710000000001 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.562339+0000", 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086553400000000005 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775566+0000", 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21322675199999999 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775605+0000", 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8424999999999999e-05 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775636+0000", 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1414000000000001e-05 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62189 4.9 4:952cc0c3:::benchmark_data_smithi173_74914_object62188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.508737607, 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.475062977, 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62189 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563190+0000", 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1838e-05 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563201+0000", 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0794e-05 2024-08-30T23:36:46.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563206+0000", 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.429e-06 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563251+0000", 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4363999999999997e-05 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563339+0000", 2024-08-30T23:36:46.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8675000000000001e-05 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563428+0000", 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8559000000000004e-05 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.564850+0000", 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014217069999999999 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.565634+0000", 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078473200000000001 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038076+0000", 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.47244159499999999 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038142+0000", 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6005000000000002e-05 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038191+0000", 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8541000000000002e-05 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62196 4.9 4:91ca1c4e:::benchmark_data_smithi173_74914_object62195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:36:46.180 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.29870014799999, 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310692415, 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62196 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773167+0000", 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5409999999999999e-06 2024-08-30T23:36:46.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773206+0000", 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9888000000000002e-05 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773222+0000", 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5690000000000001e-05 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773228+0000", 2024-08-30T23:36:46.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9819999999999996e-06 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773279+0000", 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0417000000000002e-05 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773374+0000", 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5396000000000006e-05 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773464+0000", 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0315000000000005e-05 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775336+0000", 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018712749999999999 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.779671+0000", 2024-08-30T23:36:46.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043352089999999996 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083790+0000", 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.304119056 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083827+0000", 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6863999999999998e-05 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083857+0000", 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0781999999999998e-05 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:62189 4.9 e337/327 ondisk, result = 0)", 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.296383928, 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26272571900000002, 2024-08-30T23:36:46.187 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775482+0000", 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.64e-07 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775487+0000", 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.037983+0000", 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26249586200000002 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.037988+0000", 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.561e-06 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038060+0000", 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2241000000000002e-05 2024-08-30T23:36:46.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038070+0000", 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038207+0000", 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013733899999999999 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:158582 4.9 4:9776805f:::benchmark_data_smithi173_74914_object60340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:36.429648+0000", 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.6422175750000001, 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.123883388, 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 158582 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.192 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429648+0000", 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429648+0000", 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429649+0000", 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8679999999999999e-06 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429657+0000", 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6869999999999994e-06 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429676+0000", 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8453000000000001e-05 2024-08-30T23:36:46.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429685+0000", 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1239999999999994e-06 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:46.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429759+0000", 2024-08-30T23:36:46.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4677000000000003e-05 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429880+0000", 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012023099999999999 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.429988+0000", 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108244 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.435527+0000", 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005539296 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.437279+0000", 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017519600000000001 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553356+0000", 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.116076949 2024-08-30T23:36:46.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553464+0000", 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108168 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553531+0000", 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6730999999999995e-05 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.199 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:158582 4.9 e371/327 ondisk, result = 0)", 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:36.434691+0000", 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.6371740460000002, 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.118846041, 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.434691+0000", 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.434691+0000", 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.434692+0000", 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-08-30T23:36:46.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.434700+0000", 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9840000000000001e-06 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553288+0000", 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.118587597 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553295+0000", 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0690000000000004e-06 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553341+0000", 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.634e-05 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553349+0000", 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3739999999999998e-06 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:46.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:36.553537+0000", 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000187765 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:46.204 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:36:46.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:46.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:46.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:46.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:46.489 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:36:46.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:46.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:46.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:46.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:46.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:46 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:36:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/127547120' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:46 smithi136 ceph-mon[20207]: osdmap e375: 12 total, 11 up, 12 in 2024-08-30T23:36:46.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:46.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:46.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:46.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:47.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:47.042 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:46 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:36:47.042 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/127547120' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:47.042 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:46 smithi106 ceph-mon[16603]: osdmap e375: 12 total, 11 up, 12 in 2024-08-30T23:36:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:46 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:36:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/127547120' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:46 smithi173 ceph-mon[20554]: osdmap e375: 12 total, 11 up, 12 in 2024-08-30T23:36:47.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:47.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:47.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:47.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:47.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:47.400 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:47.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:47.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:47.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:47.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:47.683 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:47.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:47.777 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:47.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:47 smithi106 ceph-mon[16603]: 4.c scrub starts 2024-08-30T23:36:47.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:47 smithi106 ceph-mon[16603]: 4.c scrub ok 2024-08-30T23:36:47.810 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:47 smithi106 ceph-mon[16603]: pgmap v903: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s; 11213/90036 objects degraded (12.454%) 2024-08-30T23:36:47.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:36:47.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:47.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:47.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:48.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:48.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:47 smithi136 ceph-mon[20207]: 4.c scrub starts 2024-08-30T23:36:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:47 smithi136 ceph-mon[20207]: 4.c scrub ok 2024-08-30T23:36:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:47 smithi136 ceph-mon[20207]: pgmap v903: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s; 11213/90036 objects degraded (12.454%) 2024-08-30T23:36:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:47 smithi173 ceph-mon[20554]: 4.c scrub starts 2024-08-30T23:36:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:47 smithi173 ceph-mon[20554]: 4.c scrub ok 2024-08-30T23:36:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:47 smithi173 ceph-mon[20554]: pgmap v903: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.8 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 877 op/s; 11213/90036 objects degraded (12.454%) 2024-08-30T23:36:48.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:48.166 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:48.171 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:48.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:48.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:48.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:48.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:48.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:48.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:48.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:48.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:48.713 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:36:48.713 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:36:48.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:169732 4.0 4:098a18b1:::benchmark_data_smithi173_74914_object71490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:48.704307+0000", 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0090252849999999992, 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0090546970000000004, 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 169732 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.714 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704307+0000", 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704307+0000", 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704308+0000", 2024-08-30T23:36:48.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.78e-07 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704312+0000", 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0219999999999998e-06 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704321+0000", 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4560000000000006e-06 2024-08-30T23:36:48.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704326+0000", 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6840000000000004e-06 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704360+0000", 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4032000000000002e-05 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:48.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704485+0000", 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012468000000000001 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.704559+0000", 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4460000000000002e-05 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.709321+0000", 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047617980000000002 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.709771+0000", 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0004504 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:169741 4.0 4:018aa22d:::benchmark_data_smithi173_74914_object71499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:48.710659+0000", 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0026726739999999999, 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027787810000000001, 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:36:48.719 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 169741 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710659+0000", 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710659+0000", 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710661+0000", 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4360000000000001e-06 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710666+0000", 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3569999999999997e-06 2024-08-30T23:36:48.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710685+0000", 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8366e-05 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710693+0000", 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6279999999999994e-06 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:36:48.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710750+0000", 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6394000000000003e-05 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.710886+0000", 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013646599999999999 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:48.711011+0000", 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012506799999999999 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:36:48.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:48.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:36:48.724 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:36:48.724 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-08-30T23:36:48.724 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:36:48.780 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:48.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:48.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:48.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:48.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:49.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/208751660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:49.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:49.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/208751660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:49.078 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:49.079 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:49.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/208751660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:49.079 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:36:49.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:36:49.114 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:49.115 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:49.152 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:49.152 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:49.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:49.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:49.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:49.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:49.211 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:49.211 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:49.246 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:49.247 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:49.261 INFO:teuthology.orchestra.run.smithi106.stderr:Exporting 4.f info 4.f( v 368'9353 (0'0,368'9353] local-lis/les=306/307 n=2805 ec=306/306 lis/c=306/306 les/c/f=307/307/0 sis=306) 2024-08-30T23:36:49.275 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:49.275 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:49.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:49.310 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:49.310 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:49.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:49.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f002868a:::benchmark_data_smithi173_74914_object61686:head# 2024-08-30T23:36:49.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f002b325:::benchmark_data_smithi173_74914_object87581:head# 2024-08-30T23:36:49.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f007cb59:::benchmark_data_smithi173_74914_object90155:head# 2024-08-30T23:36:49.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f008c387:::benchmark_data_smithi173_74914_object85988:head# 2024-08-30T23:36:49.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00947d9:::benchmark_data_smithi173_74914_object64042:head# 2024-08-30T23:36:49.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00999f2:::benchmark_data_smithi173_74914_object78429:head# 2024-08-30T23:36:49.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00a3049:::benchmark_data_smithi173_74914_object90740:head# 2024-08-30T23:36:49.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00b8306:::benchmark_data_smithi173_74914_object89563:head# 2024-08-30T23:36:49.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00c0d2d:::benchmark_data_smithi173_74914_object65020:head# 2024-08-30T23:36:49.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00de138:::benchmark_data_smithi173_74914_object94006:head# 2024-08-30T23:36:49.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00eee5d:::benchmark_data_smithi173_74914_object83469:head# 2024-08-30T23:36:49.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00f5dc0:::benchmark_data_smithi173_74914_object53545:head# 2024-08-30T23:36:49.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f00ff2f1:::benchmark_data_smithi173_74914_object74319:head# 2024-08-30T23:36:49.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f010bb39:::benchmark_data_smithi173_74914_object66760:head# 2024-08-30T23:36:49.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f01111e7:::benchmark_data_smithi173_74914_object61430:head# 2024-08-30T23:36:49.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0112e43:::benchmark_data_smithi173_74914_object77849:head# 2024-08-30T23:36:49.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0116a6f:::benchmark_data_smithi173_74914_object81843:head# 2024-08-30T23:36:49.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0124fd9:::benchmark_data_smithi173_74914_object74043:head# 2024-08-30T23:36:49.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f01266e0:::benchmark_data_smithi173_74914_object88367:head# 2024-08-30T23:36:49.341 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:49.341 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:49.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0129921:::benchmark_data_smithi173_74914_object77855:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0143e9b:::benchmark_data_smithi173_74914_object64655:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f01de819:::benchmark_data_smithi173_74914_object91758:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f01e8d34:::benchmark_data_smithi173_74914_object96320:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f01ed92b:::benchmark_data_smithi173_74914_object65085:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f01fee22:::benchmark_data_smithi173_74914_object66027:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0200416:::benchmark_data_smithi173_74914_object68751:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f02109e1:::benchmark_data_smithi173_74914_object78220:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0220a7f:::benchmark_data_smithi173_74914_object58070:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0224bcb:::benchmark_data_smithi173_74914_object95941:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f023a268:::benchmark_data_smithi173_74914_object74241:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0246868:::benchmark_data_smithi173_74914_object83754:head# 2024-08-30T23:36:49.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f024e0d6:::benchmark_data_smithi173_74914_object74226:head# 2024-08-30T23:36:49.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f025c189:::benchmark_data_smithi173_74914_object65838:head# 2024-08-30T23:36:49.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0277a4f:::benchmark_data_smithi173_74914_object78384:head# 2024-08-30T23:36:49.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f02bd4cd:::benchmark_data_smithi173_74914_object95285:head# 2024-08-30T23:36:49.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f02e898d:::benchmark_data_smithi173_74914_object85563:head# 2024-08-30T23:36:49.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f032b7cf:::benchmark_data_smithi173_74914_object90067:head# 2024-08-30T23:36:49.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f033e9f3:::benchmark_data_smithi173_74914_object92300:head# 2024-08-30T23:36:49.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0365d93:::benchmark_data_smithi173_74914_object62560:head# 2024-08-30T23:36:49.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0372d40:::benchmark_data_smithi173_74914_object79766:head# 2024-08-30T23:36:49.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0374ece:::benchmark_data_smithi173_74914_object81818:head# 2024-08-30T23:36:49.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03781a8:::benchmark_data_smithi173_74914_object92683:head# 2024-08-30T23:36:49.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0380e27:::benchmark_data_smithi173_74914_object67791:head# 2024-08-30T23:36:49.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0386dab:::benchmark_data_smithi173_74914_object86974:head# 2024-08-30T23:36:49.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f039070d:::benchmark_data_smithi173_74914_object78471:head# 2024-08-30T23:36:49.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03991c2:::benchmark_data_smithi173_74914_object98156:head# 2024-08-30T23:36:49.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03ae0ca:::benchmark_data_smithi173_74914_object63861:head# 2024-08-30T23:36:49.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03dcb97:::benchmark_data_smithi173_74914_object72412:head# 2024-08-30T23:36:49.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03e470b:::benchmark_data_smithi173_74914_object60063:head# 2024-08-30T23:36:49.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03e963e:::benchmark_data_smithi173_74914_object85974:head# 2024-08-30T23:36:49.360 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f03efb49:::benchmark_data_smithi173_74914_object77756:head# 2024-08-30T23:36:49.360 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f04274e4:::benchmark_data_smithi173_74914_object66728:head# 2024-08-30T23:36:49.360 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0469490:::benchmark_data_smithi173_74914_object73964:head# 2024-08-30T23:36:49.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0491d63:::benchmark_data_smithi173_74914_object75707:head# 2024-08-30T23:36:49.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f04b0b3c:::benchmark_data_smithi173_74914_object90564:head# 2024-08-30T23:36:49.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f04b797d:::benchmark_data_smithi173_74914_object79752:head# 2024-08-30T23:36:49.362 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f04cc918:::benchmark_data_smithi173_74914_object84297:head# 2024-08-30T23:36:49.362 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f04ec094:::benchmark_data_smithi173_74914_object66697:head# 2024-08-30T23:36:49.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f04ede97:::benchmark_data_smithi173_74914_object61237:head# 2024-08-30T23:36:49.366 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f050d553:::benchmark_data_smithi173_74914_object90590:head# 2024-08-30T23:36:49.366 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f052e30f:::benchmark_data_smithi173_74914_object95300:head# 2024-08-30T23:36:49.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0532f02:::benchmark_data_smithi173_74914_object95216:head# 2024-08-30T23:36:49.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0537508:::benchmark_data_smithi173_74914_object62324:head# 2024-08-30T23:36:49.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0553b04:::benchmark_data_smithi173_74914_object91162:head# 2024-08-30T23:36:49.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f056b1b9:::benchmark_data_smithi173_74914_object97399:head# 2024-08-30T23:36:49.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0570a6c:::benchmark_data_smithi173_74914_object63336:head# 2024-08-30T23:36:49.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f058cd57:::benchmark_data_smithi173_74914_object78591:head# 2024-08-30T23:36:49.370 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0599b2f:::benchmark_data_smithi173_74914_object77274:head# 2024-08-30T23:36:49.370 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:49.370 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:49.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f05a8274:::benchmark_data_smithi173_74914_object64420:head# 2024-08-30T23:36:49.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f05e5dae:::benchmark_data_smithi173_74914_object58758:head# 2024-08-30T23:36:49.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f060f0a0:::benchmark_data_smithi173_74914_object55605:head# 2024-08-30T23:36:49.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f063e6d6:::benchmark_data_smithi173_74914_object60697:head# 2024-08-30T23:36:49.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0652abc:::benchmark_data_smithi173_74914_object97798:head# 2024-08-30T23:36:49.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0653a94:::benchmark_data_smithi173_74914_object70147:head# 2024-08-30T23:36:49.373 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0655d49:::benchmark_data_smithi173_74914_object59529:head# 2024-08-30T23:36:49.373 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0659981:::benchmark_data_smithi173_74914_object80460:head# 2024-08-30T23:36:49.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f065a9cd:::benchmark_data_smithi173_74914_object68263:head# 2024-08-30T23:36:49.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0672271:::benchmark_data_smithi173_74914_object79899:head# 2024-08-30T23:36:49.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f06b79e8:::benchmark_data_smithi173_74914_object72634:head# 2024-08-30T23:36:49.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f06ba776:::benchmark_data_smithi173_74914_object79640:head# 2024-08-30T23:36:49.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f06cebf2:::benchmark_data_smithi173_74914_object95406:head# 2024-08-30T23:36:49.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f06d3357:::benchmark_data_smithi173_74914_object94866:head# 2024-08-30T23:36:49.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f06ddd20:::benchmark_data_smithi173_74914_object74747:head# 2024-08-30T23:36:49.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f070e8be:::benchmark_data_smithi173_74914_object64150:head# 2024-08-30T23:36:49.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07462e2:::benchmark_data_smithi173_74914_object80118:head# 2024-08-30T23:36:49.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0757dee:::benchmark_data_smithi173_74914_object57392:head# 2024-08-30T23:36:49.379 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0774496:::benchmark_data_smithi173_74914_object57239:head# 2024-08-30T23:36:49.381 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f077fbef:::benchmark_data_smithi173_74914_object74818:head# 2024-08-30T23:36:49.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0783867:::benchmark_data_smithi173_74914_object64549:head# 2024-08-30T23:36:49.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f078857c:::benchmark_data_smithi173_74914_object78984:head# 2024-08-30T23:36:49.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0788fd7:::benchmark_data_smithi173_74914_object65671:head# 2024-08-30T23:36:49.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0796ae5:::benchmark_data_smithi173_74914_object76417:head# 2024-08-30T23:36:49.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07a6355:::benchmark_data_smithi173_74914_object95648:head# 2024-08-30T23:36:49.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07af1dd:::benchmark_data_smithi173_74914_object96820:head# 2024-08-30T23:36:49.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07b7143:::benchmark_data_smithi173_74914_object59882:head# 2024-08-30T23:36:49.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07b76f7:::benchmark_data_smithi173_74914_object63505:head# 2024-08-30T23:36:49.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07c290f:::benchmark_data_smithi173_74914_object74894:head# 2024-08-30T23:36:49.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f07d5d19:::benchmark_data_smithi173_74914_object76854:head# 2024-08-30T23:36:49.388 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f080b372:::benchmark_data_smithi173_74914_object79090:head# 2024-08-30T23:36:49.393 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0823204:::benchmark_data_smithi173_74914_object73215:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08494b5:::benchmark_data_smithi173_74914_object81489:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f087e628:::benchmark_data_smithi173_74914_object91881:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f087eef7:::benchmark_data_smithi173_74914_object74456:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08858b4:::benchmark_data_smithi173_74914_object53820:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f088903d:::benchmark_data_smithi173_74914_object86139:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f088a7c3:::benchmark_data_smithi173_74914_object73944:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08a5bb2:::benchmark_data_smithi173_74914_object61378:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08b2bc1:::benchmark_data_smithi173_74914_object78002:head# 2024-08-30T23:36:49.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08c9d39:::benchmark_data_smithi173_74914_object64945:head# 2024-08-30T23:36:49.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08d58a1:::benchmark_data_smithi173_74914_object96691:head# 2024-08-30T23:36:49.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08ebead:::benchmark_data_smithi173_74914_object95574:head# 2024-08-30T23:36:49.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f08f5469:::benchmark_data_smithi173_74914_object54965:head# 2024-08-30T23:36:49.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f091d489:::benchmark_data_smithi173_74914_object83817:head# 2024-08-30T23:36:49.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f094c943:::benchmark_data_smithi173_74914_object77339:head# 2024-08-30T23:36:49.400 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f096ff6a:::benchmark_data_smithi173_74914_object75241:head# 2024-08-30T23:36:49.400 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:49.400 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:49.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0978f65:::benchmark_data_smithi173_74914_object97974:head# 2024-08-30T23:36:49.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f097f8fa:::benchmark_data_smithi173_74914_object61133:head# 2024-08-30T23:36:49.401 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:49.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f09821e2:::benchmark_data_smithi173_74914_object73054:head# 2024-08-30T23:36:49.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0985984:::benchmark_data_smithi173_74914_object64499:head# 2024-08-30T23:36:49.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f09955ca:::benchmark_data_smithi173_74914_object80024:head# 2024-08-30T23:36:49.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f09a6d6d:::benchmark_data_smithi173_74914_object80237:head# 2024-08-30T23:36:49.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f09f9272:::benchmark_data_smithi173_74914_object60163:head# 2024-08-30T23:36:49.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0a08bcf:::benchmark_data_smithi173_74914_object78807:head# 2024-08-30T23:36:49.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0a209aa:::benchmark_data_smithi173_74914_object92895:head# 2024-08-30T23:36:49.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0a8f7c1:::benchmark_data_smithi173_74914_object93498:head# 2024-08-30T23:36:49.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0aae2ff:::benchmark_data_smithi173_74914_object92284:head# 2024-08-30T23:36:49.406 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0ad5d60:::benchmark_data_smithi173_74914_object81456:head# 2024-08-30T23:36:49.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0aeadc6:::benchmark_data_smithi173_74914_object88877:head# 2024-08-30T23:36:49.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b26954:::benchmark_data_smithi173_74914_object85746:head# 2024-08-30T23:36:49.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b26b6b:::benchmark_data_smithi173_74914_object88018:head# 2024-08-30T23:36:49.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b291e5:::benchmark_data_smithi173_74914_object75390:head# 2024-08-30T23:36:49.409 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b2b1d5:::benchmark_data_smithi173_74914_object60083:head# 2024-08-30T23:36:49.409 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b4a36f:::benchmark_data_smithi173_74914_object81562:head# 2024-08-30T23:36:49.410 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b4c639:::benchmark_data_smithi173_74914_object58700:head# 2024-08-30T23:36:49.410 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b56c66:::benchmark_data_smithi173_74914_object86067:head# 2024-08-30T23:36:49.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0b9ddbc:::benchmark_data_smithi173_74914_object77746:head# 2024-08-30T23:36:49.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0bc2a61:::benchmark_data_smithi173_74914_object64368:head# 2024-08-30T23:36:49.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0bcb01d:::benchmark_data_smithi173_74914_object69121:head# 2024-08-30T23:36:49.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0bd6511:::benchmark_data_smithi173_74914_object94278:head# 2024-08-30T23:36:49.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c17b41:::benchmark_data_smithi173_74914_object79285:head# 2024-08-30T23:36:49.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c255a6:::benchmark_data_smithi173_74914_object61699:head# 2024-08-30T23:36:49.414 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c640ad:::benchmark_data_smithi173_74914_object70887:head# 2024-08-30T23:36:49.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:49.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c7d792:::benchmark_data_smithi173_74914_object90795:head# 2024-08-30T23:36:49.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c802f5:::benchmark_data_smithi173_74914_object57858:head# 2024-08-30T23:36:49.417 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c80e25:::benchmark_data_smithi173_74914_object57980:head# 2024-08-30T23:36:49.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c9338e:::benchmark_data_smithi173_74914_object90008:head# 2024-08-30T23:36:49.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0c941cc:::benchmark_data_smithi173_74914_object77681:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0cfc79a:::benchmark_data_smithi173_74914_object97430:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d01781:::benchmark_data_smithi173_74914_object59270:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d08e7d:::benchmark_data_smithi173_74914_object85783:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d0cd7e:::benchmark_data_smithi173_74914_object65054:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d19af4:::benchmark_data_smithi173_74914_object55277:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d1b06d:::benchmark_data_smithi173_74914_object92267:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d20ea5:::benchmark_data_smithi173_74914_object72810:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d2450b:::benchmark_data_smithi173_74914_object60079:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d54470:::benchmark_data_smithi173_74914_object54434:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d56a9e:::benchmark_data_smithi173_74914_object64072:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d61f76:::benchmark_data_smithi173_74914_object76022:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d7099e:::benchmark_data_smithi173_74914_object94841:head# 2024-08-30T23:36:49.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0d8a639:::benchmark_data_smithi173_74914_object76937:head# 2024-08-30T23:36:49.429 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0db13f6:::benchmark_data_smithi173_74914_object88056:head# 2024-08-30T23:36:49.429 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0db30b8:::benchmark_data_smithi173_74914_object89118:head# 2024-08-30T23:36:49.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0dc7482:::benchmark_data_smithi173_74914_object54040:head# 2024-08-30T23:36:49.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0dd5876:::benchmark_data_smithi173_74914_object71667:head# 2024-08-30T23:36:49.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0de4ddb:::benchmark_data_smithi173_74914_object66265:head# 2024-08-30T23:36:49.431 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0de4f4f:::benchmark_data_smithi173_74914_object77947:head# 2024-08-30T23:36:49.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0de8a4e:::benchmark_data_smithi173_74914_object65449:head# 2024-08-30T23:36:49.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0de956f:::benchmark_data_smithi173_74914_object88392:head# 2024-08-30T23:36:49.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0e666fb:::benchmark_data_smithi173_74914_object86804:head# 2024-08-30T23:36:49.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0e98f5f:::benchmark_data_smithi173_74914_object82592:head# 2024-08-30T23:36:49.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0ea0c3a:::benchmark_data_smithi173_74914_object63608:head# 2024-08-30T23:36:49.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0eb52b6:::benchmark_data_smithi173_74914_object91331:head# 2024-08-30T23:36:49.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0ecdea9:::benchmark_data_smithi173_74914_object68307:head# 2024-08-30T23:36:49.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f0a8f4:::benchmark_data_smithi173_74914_object95093:head# 2024-08-30T23:36:49.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f1a5a4:::benchmark_data_smithi173_74914_object60084:head# 2024-08-30T23:36:49.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f276a4:::benchmark_data_smithi173_74914_object62151:head# 2024-08-30T23:36:49.438 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f32ed1:::benchmark_data_smithi173_74914_object98186:head# 2024-08-30T23:36:49.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f3dad0:::benchmark_data_smithi173_74914_object62157:head# 2024-08-30T23:36:49.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f73f4c:::benchmark_data_smithi173_74914_object77697:head# 2024-08-30T23:36:49.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f76eb5:::benchmark_data_smithi173_74914_object71529:head# 2024-08-30T23:36:49.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0f8baf2:::benchmark_data_smithi173_74914_object97514:head# 2024-08-30T23:36:49.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0fdb023:::benchmark_data_smithi173_74914_object59345:head# 2024-08-30T23:36:49.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0feb7b2:::benchmark_data_smithi173_74914_object78822:head# 2024-08-30T23:36:49.444 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f0ff8a2a:::benchmark_data_smithi173_74914_object90343:head# 2024-08-30T23:36:49.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f100dc66:::benchmark_data_smithi173_74914_object63371:head# 2024-08-30T23:36:49.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1036ef9:::benchmark_data_smithi173_74914_object61502:head# 2024-08-30T23:36:49.446 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1037b10:::benchmark_data_smithi173_74914_object84962:head# 2024-08-30T23:36:49.447 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f105d454:::benchmark_data_smithi173_74914_object87414:head# 2024-08-30T23:36:49.447 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:49.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f105d8cb:::benchmark_data_smithi173_74914_object86903:head# 2024-08-30T23:36:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:49.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f105f507:::benchmark_data_smithi173_74914_object86035:head# 2024-08-30T23:36:49.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f108ac1e:::benchmark_data_smithi173_74914_object95476:head# 2024-08-30T23:36:49.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1090b7f:::benchmark_data_smithi173_74914_object87325:head# 2024-08-30T23:36:49.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f10b5cc9:::benchmark_data_smithi173_74914_object97196:head# 2024-08-30T23:36:49.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f10bd2e5:::benchmark_data_smithi173_74914_object64451:head# 2024-08-30T23:36:49.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f10c8728:::benchmark_data_smithi173_74914_object90429:head# 2024-08-30T23:36:49.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f10f845a:::benchmark_data_smithi173_74914_object93811:head# 2024-08-30T23:36:49.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f113b5d6:::benchmark_data_smithi173_74914_object53827:head# 2024-08-30T23:36:49.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1151a82:::benchmark_data_smithi173_74914_object59727:head# 2024-08-30T23:36:49.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f115d847:::benchmark_data_smithi173_74914_object88515:head# 2024-08-30T23:36:49.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1166dfa:::benchmark_data_smithi173_74914_object67400:head# 2024-08-30T23:36:49.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1186475:::benchmark_data_smithi173_74914_object80693:head# 2024-08-30T23:36:49.456 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f11bb270:::benchmark_data_smithi173_74914_object93133:head# 2024-08-30T23:36:49.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f11c4ac7:::benchmark_data_smithi173_74914_object61268:head# 2024-08-30T23:36:49.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f11cfcb1:::benchmark_data_smithi173_74914_object89000:head# 2024-08-30T23:36:49.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f120af0d:::benchmark_data_smithi173_74914_object86388:head# 2024-08-30T23:36:49.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f123c944:::benchmark_data_smithi173_74914_object55883:head# 2024-08-30T23:36:49.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1249c8a:::benchmark_data_smithi173_74914_object95569:head# 2024-08-30T23:36:49.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1254f92:::benchmark_data_smithi173_74914_object58838:head# 2024-08-30T23:36:49.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1280692:::benchmark_data_smithi173_74914_object56758:head# 2024-08-30T23:36:49.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f129f8e2:::benchmark_data_smithi173_74914_object61934:head# 2024-08-30T23:36:49.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f12a2fff:::benchmark_data_smithi173_74914_object97017:head# 2024-08-30T23:36:49.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f12aec8e:::benchmark_data_smithi173_74914_object88479:head# 2024-08-30T23:36:49.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f12b7594:::benchmark_data_smithi173_74914_object65397:head# 2024-08-30T23:36:49.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f12d43df:::benchmark_data_smithi173_74914_object71608:head# 2024-08-30T23:36:49.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f12e930b:::benchmark_data_smithi173_74914_object57918:head# 2024-08-30T23:36:49.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f130ee2f:::benchmark_data_smithi173_74914_object69906:head# 2024-08-30T23:36:49.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1311ea3:::benchmark_data_smithi173_74914_object61311:head# 2024-08-30T23:36:49.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1324e98:::benchmark_data_smithi173_74914_object90631:head# 2024-08-30T23:36:49.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f132bdd4:::benchmark_data_smithi173_74914_object80919:head# 2024-08-30T23:36:49.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f137ddd8:::benchmark_data_smithi173_74914_object74897:head# 2024-08-30T23:36:49.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1386878:::benchmark_data_smithi173_74914_object85612:head# 2024-08-30T23:36:49.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f13ad077:::benchmark_data_smithi173_74914_object93301:head# 2024-08-30T23:36:49.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f13ba9e5:::benchmark_data_smithi173_74914_object65362:head# 2024-08-30T23:36:49.465 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f13c0558:::benchmark_data_smithi173_74914_object68727:head# 2024-08-30T23:36:49.465 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f13c8005:::benchmark_data_smithi173_74914_object54775:head# 2024-08-30T23:36:49.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f13d9623:::benchmark_data_smithi173_74914_object93432:head# 2024-08-30T23:36:49.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f13da5fc:::benchmark_data_smithi173_74914_object64519:head# 2024-08-30T23:36:49.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f140115d:::benchmark_data_smithi173_74914_object66964:head# 2024-08-30T23:36:49.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1434dbd:::benchmark_data_smithi173_74914_object82687:head# 2024-08-30T23:36:49.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f14369dc:::benchmark_data_smithi173_74914_object62533:head# 2024-08-30T23:36:49.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1440da0:::benchmark_data_smithi173_74914_object59782:head# 2024-08-30T23:36:49.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f14503f3:::benchmark_data_smithi173_74914_object71150:head# 2024-08-30T23:36:49.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f145495e:::benchmark_data_smithi173_74914_object84358:head# 2024-08-30T23:36:49.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1463792:::benchmark_data_smithi173_74914_object58854:head# 2024-08-30T23:36:49.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1465d6f:::benchmark_data_smithi173_74914_object63327:head# 2024-08-30T23:36:49.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f147929c:::benchmark_data_smithi173_74914_object53771:head# 2024-08-30T23:36:49.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f14e05a3:::benchmark_data_smithi173_74914_object85161:head# 2024-08-30T23:36:49.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f14e321a:::benchmark_data_smithi173_74914_object83824:head# 2024-08-30T23:36:49.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f14e8b36:::benchmark_data_smithi173_74914_object74590:head# 2024-08-30T23:36:49.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f151fe7e:::benchmark_data_smithi173_74914_object63765:head# 2024-08-30T23:36:49.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f153fe14:::benchmark_data_smithi173_74914_object83492:head# 2024-08-30T23:36:49.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f155a4a6:::benchmark_data_smithi173_74914_object97054:head# 2024-08-30T23:36:49.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1569413:::benchmark_data_smithi173_74914_object94571:head# 2024-08-30T23:36:49.480 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f158c4f2:::benchmark_data_smithi173_74914_object55465:head# 2024-08-30T23:36:49.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f15f54c9:::benchmark_data_smithi173_74914_object84640:head# 2024-08-30T23:36:49.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f15f8f99:::benchmark_data_smithi173_74914_object79634:head# 2024-08-30T23:36:49.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1612556:::benchmark_data_smithi173_74914_object82672:head# 2024-08-30T23:36:49.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1619e1c:::benchmark_data_smithi173_74914_object53359:head# 2024-08-30T23:36:49.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f162367d:::benchmark_data_smithi173_74914_object91476:head# 2024-08-30T23:36:49.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1651a60:::benchmark_data_smithi173_74914_object73133:head# 2024-08-30T23:36:49.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f165af19:::benchmark_data_smithi173_74914_object71405:head# 2024-08-30T23:36:49.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1672dd9:::benchmark_data_smithi173_74914_object56113:head# 2024-08-30T23:36:49.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f168ce2f:::benchmark_data_smithi173_74914_object63324:head# 2024-08-30T23:36:49.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f16c8bed:::benchmark_data_smithi173_74914_object77296:head# 2024-08-30T23:36:49.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f16c9187:::benchmark_data_smithi173_74914_object90322:head# 2024-08-30T23:36:49.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f16cc974:::benchmark_data_smithi173_74914_object68355:head# 2024-08-30T23:36:49.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f16d2fcc:::benchmark_data_smithi173_74914_object69400:head# 2024-08-30T23:36:49.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f16e1871:::benchmark_data_smithi173_74914_object83517:head# 2024-08-30T23:36:49.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1709dcf:::benchmark_data_smithi173_74914_object87591:head# 2024-08-30T23:36:49.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1717337:::benchmark_data_smithi173_74914_object78802:head# 2024-08-30T23:36:49.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1731894:::benchmark_data_smithi173_74914_object55921:head# 2024-08-30T23:36:49.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1737b2e:::benchmark_data_smithi173_74914_object66724:head# 2024-08-30T23:36:49.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1744bae:::benchmark_data_smithi173_74914_object92712:head# 2024-08-30T23:36:49.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f174c00c:::benchmark_data_smithi173_74914_object83497:head# 2024-08-30T23:36:49.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1756dd2:::benchmark_data_smithi173_74914_object53872:head# 2024-08-30T23:36:49.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f17667af:::benchmark_data_smithi173_74914_object65979:head# 2024-08-30T23:36:49.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f17ba282:::benchmark_data_smithi173_74914_object93123:head# 2024-08-30T23:36:49.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f17c194e:::benchmark_data_smithi173_74914_object63190:head# 2024-08-30T23:36:49.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f17df36a:::benchmark_data_smithi173_74914_object90884:head# 2024-08-30T23:36:49.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f17f06ad:::benchmark_data_smithi173_74914_object75129:head# 2024-08-30T23:36:49.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1852aa0:::benchmark_data_smithi173_74914_object54828:head# 2024-08-30T23:36:49.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f185816e:::benchmark_data_smithi173_74914_object96948:head# 2024-08-30T23:36:49.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1863d8d:::benchmark_data_smithi173_74914_object61985:head# 2024-08-30T23:36:49.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1869856:::benchmark_data_smithi173_74914_object63642:head# 2024-08-30T23:36:49.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f186b411:::benchmark_data_smithi173_74914_object68922:head# 2024-08-30T23:36:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f187247b:::benchmark_data_smithi173_74914_object65933:head# 2024-08-30T23:36:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18a2a65:::benchmark_data_smithi173_74914_object76276:head# 2024-08-30T23:36:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18a3aa1:::benchmark_data_smithi173_74914_object81388:head# 2024-08-30T23:36:49.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18c3d1e:::benchmark_data_smithi173_74914_object83210:head# 2024-08-30T23:36:49.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18c3df3:::benchmark_data_smithi173_74914_object91498:head# 2024-08-30T23:36:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18c8940:::benchmark_data_smithi173_74914_object97988:head# 2024-08-30T23:36:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18f8d0d:::benchmark_data_smithi173_74914_object93615:head# 2024-08-30T23:36:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f18fe858:::benchmark_data_smithi173_74914_object70881:head# 2024-08-30T23:36:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1905dfe:::benchmark_data_smithi173_74914_object96599:head# 2024-08-30T23:36:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f195a633:::benchmark_data_smithi173_74914_object55620:head# 2024-08-30T23:36:49.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19696e1:::benchmark_data_smithi173_74914_object86403:head# 2024-08-30T23:36:49.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19738b8:::benchmark_data_smithi173_74914_object88728:head# 2024-08-30T23:36:49.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f197cc89:::benchmark_data_smithi173_74914_object91349:head# 2024-08-30T23:36:49.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19a5c09:::benchmark_data_smithi173_74914_object85622:head# 2024-08-30T23:36:49.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19a6f4d:::benchmark_data_smithi173_74914_object82308:head# 2024-08-30T23:36:49.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19a7be2:::benchmark_data_smithi173_74914_object57563:head# 2024-08-30T23:36:49.504 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19beadf:::benchmark_data_smithi173_74914_object64380:head# 2024-08-30T23:36:49.504 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19d07bd:::benchmark_data_smithi173_74914_object84646:head# 2024-08-30T23:36:49.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19d09f1:::benchmark_data_smithi173_74914_object82490:head# 2024-08-30T23:36:49.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19d86d0:::benchmark_data_smithi173_74914_object89060:head# 2024-08-30T23:36:49.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19de880:::benchmark_data_smithi173_74914_object77568:head# 2024-08-30T23:36:49.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f19e2f37:::benchmark_data_smithi173_74914_object88866:head# 2024-08-30T23:36:49.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1a375f7:::benchmark_data_smithi173_74914_object62398:head# 2024-08-30T23:36:49.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:36:49.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1a40e26:::benchmark_data_smithi173_74914_object65847:head# 2024-08-30T23:36:49.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1a654fa:::benchmark_data_smithi173_74914_object62435:head# 2024-08-30T23:36:49.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ab3038:::benchmark_data_smithi173_74914_object78267:head# 2024-08-30T23:36:49.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1aba507:::benchmark_data_smithi173_74914_object72068:head# 2024-08-30T23:36:49.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1abe918:::benchmark_data_smithi173_74914_object88495:head# 2024-08-30T23:36:49.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ace83b:::benchmark_data_smithi173_74914_object53171:head# 2024-08-30T23:36:49.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ae3303:::benchmark_data_smithi173_74914_object70262:head# 2024-08-30T23:36:49.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ae8d61:::benchmark_data_smithi173_74914_object89541:head# 2024-08-30T23:36:49.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1b01d9d:::benchmark_data_smithi173_74914_object66892:head# 2024-08-30T23:36:49.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1b28adb:::benchmark_data_smithi173_74914_object55837:head# 2024-08-30T23:36:49.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1b7a78e:::benchmark_data_smithi173_74914_object77514:head# 2024-08-30T23:36:49.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1b8c966:::benchmark_data_smithi173_74914_object75406:head# 2024-08-30T23:36:49.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1b9a331:::benchmark_data_smithi173_74914_object60195:head# 2024-08-30T23:36:49.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ba36a4:::benchmark_data_smithi173_74914_object63675:head# 2024-08-30T23:36:49.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1bc165f:::benchmark_data_smithi173_74914_object85044:head# 2024-08-30T23:36:49.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1bcf7ba:::benchmark_data_smithi173_74914_object63791:head# 2024-08-30T23:36:49.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1bdb096:::benchmark_data_smithi173_74914_object65683:head# 2024-08-30T23:36:49.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1bebfa1:::benchmark_data_smithi173_74914_object77565:head# 2024-08-30T23:36:49.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1bf0fd9:::benchmark_data_smithi173_74914_object94360:head# 2024-08-30T23:36:49.517 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1bfe928:::benchmark_data_smithi173_74914_object94718:head# 2024-08-30T23:36:49.517 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1c6aa6e:::benchmark_data_smithi173_74914_object67158:head# 2024-08-30T23:36:49.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1c93398:::benchmark_data_smithi173_74914_object80697:head# 2024-08-30T23:36:49.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1c93d72:::benchmark_data_smithi173_74914_object72458:head# 2024-08-30T23:36:49.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1c96c74:::benchmark_data_smithi173_74914_object93744:head# 2024-08-30T23:36:49.519 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ca0741:::benchmark_data_smithi173_74914_object67995:head# 2024-08-30T23:36:49.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ca216f:::benchmark_data_smithi173_74914_object70789:head# 2024-08-30T23:36:49.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1cb3c26:::benchmark_data_smithi173_74914_object56282:head# 2024-08-30T23:36:49.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1cdb9ac:::benchmark_data_smithi173_74914_object80700:head# 2024-08-30T23:36:49.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1cdc46f:::benchmark_data_smithi173_74914_object97775:head# 2024-08-30T23:36:49.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1cde481:::benchmark_data_smithi173_74914_object77706:head# 2024-08-30T23:36:49.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1d0da0c:::benchmark_data_smithi173_74914_object54396:head# 2024-08-30T23:36:49.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1d3c646:::benchmark_data_smithi173_74914_object69705:head# 2024-08-30T23:36:49.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1d4508c:::benchmark_data_smithi173_74914_object93105:head# 2024-08-30T23:36:49.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1d8b675:::benchmark_data_smithi173_74914_object85821:head# 2024-08-30T23:36:49.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1dc104b:::benchmark_data_smithi173_74914_object72546:head# 2024-08-30T23:36:49.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e0ef5c:::benchmark_data_smithi173_74914_object75801:head# 2024-08-30T23:36:49.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e1cf50:::benchmark_data_smithi173_74914_object81795:head# 2024-08-30T23:36:49.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e2dfe1:::benchmark_data_smithi173_74914_object73189:head# 2024-08-30T23:36:49.528 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e34d7e:::benchmark_data_smithi173_74914_object78495:head# 2024-08-30T23:36:49.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e5d7a5:::benchmark_data_smithi173_74914_object70771:head# 2024-08-30T23:36:49.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e6db44:::benchmark_data_smithi173_74914_object75931:head# 2024-08-30T23:36:49.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e81bdf:::benchmark_data_smithi173_74914_object79462:head# 2024-08-30T23:36:49.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e85548:::benchmark_data_smithi173_74914_object89294:head# 2024-08-30T23:36:49.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1e8fe08:::benchmark_data_smithi173_74914_object86535:head# 2024-08-30T23:36:49.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1eab8cd:::benchmark_data_smithi173_74914_object60719:head# 2024-08-30T23:36:49.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1eade43:::benchmark_data_smithi173_74914_object84129:head# 2024-08-30T23:36:49.533 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1eaee6a:::benchmark_data_smithi173_74914_object74513:head# 2024-08-30T23:36:49.533 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ef4372:::benchmark_data_smithi173_74914_object86491:head# 2024-08-30T23:36:49.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1f1c106:::benchmark_data_smithi173_74914_object75284:head# 2024-08-30T23:36:49.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1f4cb29:::benchmark_data_smithi173_74914_object71073:head# 2024-08-30T23:36:49.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1f6cba1:::benchmark_data_smithi173_74914_object69867:head# 2024-08-30T23:36:49.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1f7271e:::benchmark_data_smithi173_74914_object84030:head# 2024-08-30T23:36:49.537 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1fa5e05:::benchmark_data_smithi173_74914_object54783:head# 2024-08-30T23:36:49.538 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1fb2222:::benchmark_data_smithi173_74914_object86101:head# 2024-08-30T23:36:49.538 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1fd69d5:::benchmark_data_smithi173_74914_object96804:head# 2024-08-30T23:36:49.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1fd8071:::benchmark_data_smithi173_74914_object62241:head# 2024-08-30T23:36:49.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1fe13a1:::benchmark_data_smithi173_74914_object91902:head# 2024-08-30T23:36:49.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ff6920:::benchmark_data_smithi173_74914_object93953:head# 2024-08-30T23:36:49.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1ff8c00:::benchmark_data_smithi173_74914_object83618:head# 2024-08-30T23:36:49.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f1fffc37:::benchmark_data_smithi173_74914_object71586:head# 2024-08-30T23:36:49.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2006ddf:::benchmark_data_smithi173_74914_object92863:head# 2024-08-30T23:36:49.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f20105df:::benchmark_data_smithi173_74914_object58654:head# 2024-08-30T23:36:49.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2029ee2:::benchmark_data_smithi173_74914_object75295:head# 2024-08-30T23:36:49.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2031102:::benchmark_data_smithi173_74914_object55292:head# 2024-08-30T23:36:49.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2047682:::benchmark_data_smithi173_74914_object90976:head# 2024-08-30T23:36:49.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2065349:::benchmark_data_smithi173_74914_object87777:head# 2024-08-30T23:36:49.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f20662e6:::benchmark_data_smithi173_74914_object70957:head# 2024-08-30T23:36:49.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f206f32f:::benchmark_data_smithi173_74914_object64188:head# 2024-08-30T23:36:49.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f20722fd:::benchmark_data_smithi173_74914_object81915:head# 2024-08-30T23:36:49.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f20b3702:::benchmark_data_smithi173_74914_object72175:head# 2024-08-30T23:36:49.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f20b9b9d:::benchmark_data_smithi173_74914_object71464:head# 2024-08-30T23:36:49.546 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f21234c8:::benchmark_data_smithi173_74914_object58539:head# 2024-08-30T23:36:49.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2139c54:::benchmark_data_smithi173_74914_object77881:head# 2024-08-30T23:36:49.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f216d928:::benchmark_data_smithi173_74914_object79688:head# 2024-08-30T23:36:49.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2175e34:::benchmark_data_smithi173_74914_object82247:head# 2024-08-30T23:36:49.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f219313e:::benchmark_data_smithi173_74914_object97340:head# 2024-08-30T23:36:49.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2198803:::benchmark_data_smithi173_74914_object56192:head# 2024-08-30T23:36:49.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:49.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f21b3e3b:::benchmark_data_smithi173_74914_object64211:head# 2024-08-30T23:36:49.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f21bb4d5:::benchmark_data_smithi173_74914_object90318:head# 2024-08-30T23:36:49.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f21be7fd:::benchmark_data_smithi173_74914_object64947:head# 2024-08-30T23:36:49.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f21cf402:::benchmark_data_smithi173_74914_object92443:head# 2024-08-30T23:36:49.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f21fb058:::benchmark_data_smithi173_74914_object91921:head# 2024-08-30T23:36:49.552 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f22369a8:::benchmark_data_smithi173_74914_object77963:head# 2024-08-30T23:36:49.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2249769:::benchmark_data_smithi173_74914_object64151:head# 2024-08-30T23:36:49.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f224d6f7:::benchmark_data_smithi173_74914_object80148:head# 2024-08-30T23:36:49.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2264318:::benchmark_data_smithi173_74914_object86556:head# 2024-08-30T23:36:49.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2266ebf:::benchmark_data_smithi173_74914_object92946:head# 2024-08-30T23:36:49.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f22694bb:::benchmark_data_smithi173_74914_object73960:head# 2024-08-30T23:36:49.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f226cf71:::benchmark_data_smithi173_74914_object61080:head# 2024-08-30T23:36:49.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2273acf:::benchmark_data_smithi173_74914_object56653:head# 2024-08-30T23:36:49.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f229f9a4:::benchmark_data_smithi173_74914_object75725:head# 2024-08-30T23:36:49.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f22e225e:::benchmark_data_smithi173_74914_object97457:head# 2024-08-30T23:36:49.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f22e576f:::benchmark_data_smithi173_74914_object84293:head# 2024-08-30T23:36:49.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2300909:::benchmark_data_smithi173_74914_object71090:head# 2024-08-30T23:36:49.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2333c23:::benchmark_data_smithi173_74914_object80520:head# 2024-08-30T23:36:49.559 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2353026:::benchmark_data_smithi173_74914_object79585:head# 2024-08-30T23:36:49.559 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2353ec1:::benchmark_data_smithi173_74914_object88129:head# 2024-08-30T23:36:49.559 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2358996:::benchmark_data_smithi173_74914_object54437:head# 2024-08-30T23:36:49.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f237585a:::benchmark_data_smithi173_74914_object60257:head# 2024-08-30T23:36:49.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2380762:::benchmark_data_smithi173_74914_object79792:head# 2024-08-30T23:36:49.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f239d4f2:::benchmark_data_smithi173_74914_object92147:head# 2024-08-30T23:36:49.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f23d0169:::benchmark_data_smithi173_74914_object69653:head# 2024-08-30T23:36:49.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f23dce8e:::benchmark_data_smithi173_74914_object80712:head# 2024-08-30T23:36:49.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f23fe6f3:::benchmark_data_smithi173_74914_object93026:head# 2024-08-30T23:36:49.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2418e0f:::benchmark_data_smithi173_74914_object94252:head# 2024-08-30T23:36:49.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f24445ec:::benchmark_data_smithi173_74914_object78241:head# 2024-08-30T23:36:49.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f24448cc:::benchmark_data_smithi173_74914_object77696:head# 2024-08-30T23:36:49.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f245008a:::benchmark_data_smithi173_74914_object77030:head# 2024-08-30T23:36:49.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2455d04:::benchmark_data_smithi173_74914_object88772:head# 2024-08-30T23:36:49.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2472fe6:::benchmark_data_smithi173_74914_object93710:head# 2024-08-30T23:36:49.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f247674b:::benchmark_data_smithi173_74914_object55903:head# 2024-08-30T23:36:49.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f247f3fe:::benchmark_data_smithi173_74914_object81393:head# 2024-08-30T23:36:49.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2493c99:::benchmark_data_smithi173_74914_object65763:head# 2024-08-30T23:36:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f24a27dd:::benchmark_data_smithi173_74914_object92307:head# 2024-08-30T23:36:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f24b465a:::benchmark_data_smithi173_74914_object67760:head# 2024-08-30T23:36:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f24c390c:::benchmark_data_smithi173_74914_object57614:head# 2024-08-30T23:36:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f24c7905:::benchmark_data_smithi173_74914_object96128:head# 2024-08-30T23:36:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f25401c3:::benchmark_data_smithi173_74914_object80984:head# 2024-08-30T23:36:49.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f254bdfb:::benchmark_data_smithi173_74914_object94429:head# 2024-08-30T23:36:49.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f254e8e4:::benchmark_data_smithi173_74914_object69808:head# 2024-08-30T23:36:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f256e5b3:::benchmark_data_smithi173_74914_object93220:head# 2024-08-30T23:36:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2574f42:::benchmark_data_smithi173_74914_object66181:head# 2024-08-30T23:36:49.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f25ae307:::benchmark_data_smithi173_74914_object84634:head# 2024-08-30T23:36:49.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f25cca5d:::benchmark_data_smithi173_74914_object97799:head# 2024-08-30T23:36:49.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f25cdc70:::benchmark_data_smithi173_74914_object84928:head# 2024-08-30T23:36:49.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2602c98:::benchmark_data_smithi173_74914_object87356:head# 2024-08-30T23:36:49.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f260e92a:::benchmark_data_smithi173_74914_object96973:head# 2024-08-30T23:36:49.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f262b873:::benchmark_data_smithi173_74914_object88151:head# 2024-08-30T23:36:49.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2638f9f:::benchmark_data_smithi173_74914_object74739:head# 2024-08-30T23:36:49.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f264699c:::benchmark_data_smithi173_74914_object87335:head# 2024-08-30T23:36:49.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2657b76:::benchmark_data_smithi173_74914_object56405:head# 2024-08-30T23:36:49.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f265b482:::benchmark_data_smithi173_74914_object87254:head# 2024-08-30T23:36:49.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f265b7d6:::benchmark_data_smithi173_74914_object66441:head# 2024-08-30T23:36:49.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f267011b:::benchmark_data_smithi173_74914_object65722:head# 2024-08-30T23:36:49.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2687575:::benchmark_data_smithi173_74914_object96741:head# 2024-08-30T23:36:49.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f268b1d5:::benchmark_data_smithi173_74914_object88312:head# 2024-08-30T23:36:49.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f269d078:::benchmark_data_smithi173_74914_object73085:head# 2024-08-30T23:36:49.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f26a0a14:::benchmark_data_smithi173_74914_object90458:head# 2024-08-30T23:36:49.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f26c7070:::benchmark_data_smithi173_74914_object88307:head# 2024-08-30T23:36:49.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f26c9f2d:::benchmark_data_smithi173_74914_object97580:head# 2024-08-30T23:36:49.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f26cec92:::benchmark_data_smithi173_74914_object67319:head# 2024-08-30T23:36:49.580 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f26d1bd9:::benchmark_data_smithi173_74914_object63155:head# 2024-08-30T23:36:49.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f270cb8b:::benchmark_data_smithi173_74914_object59047:head# 2024-08-30T23:36:49.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f271e754:::benchmark_data_smithi173_74914_object61814:head# 2024-08-30T23:36:49.581 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f276b74d:::benchmark_data_smithi173_74914_object93547:head# 2024-08-30T23:36:49.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2773e9e:::benchmark_data_smithi173_74914_object89785:head# 2024-08-30T23:36:49.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2777800:::benchmark_data_smithi173_74914_object66397:head# 2024-08-30T23:36:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:49.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:49.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f277e028:::benchmark_data_smithi173_74914_object69709:head# 2024-08-30T23:36:49.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2780774:::benchmark_data_smithi173_74914_object92290:head# 2024-08-30T23:36:49.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2781217:::benchmark_data_smithi173_74914_object89592:head# 2024-08-30T23:36:49.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f27940bc:::benchmark_data_smithi173_74914_object91897:head# 2024-08-30T23:36:49.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f279fb98:::benchmark_data_smithi173_74914_object74210:head# 2024-08-30T23:36:49.585 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f27b1041:::benchmark_data_smithi173_74914_object78629:head# 2024-08-30T23:36:49.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f27d5092:::benchmark_data_smithi173_74914_object71385:head# 2024-08-30T23:36:49.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f27e5e25:::benchmark_data_smithi173_74914_object90492:head# 2024-08-30T23:36:49.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f27fdd29:::benchmark_data_smithi173_74914_object77099:head# 2024-08-30T23:36:49.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f281225b:::benchmark_data_smithi173_74914_object75909:head# 2024-08-30T23:36:49.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2819077:::benchmark_data_smithi173_74914_object56431:head# 2024-08-30T23:36:49.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2824bc0:::benchmark_data_smithi173_74914_object57107:head# 2024-08-30T23:36:49.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2838565:::benchmark_data_smithi173_74914_object74949:head# 2024-08-30T23:36:49.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f283c1e8:::benchmark_data_smithi173_74914_object79145:head# 2024-08-30T23:36:49.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2840922:::benchmark_data_smithi173_74914_object90132:head# 2024-08-30T23:36:49.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f285d8a1:::benchmark_data_smithi173_74914_object85932:head# 2024-08-30T23:36:49.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2861d84:::benchmark_data_smithi173_74914_object63370:head# 2024-08-30T23:36:49.590 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f287109b:::benchmark_data_smithi173_74914_object53870:head# 2024-08-30T23:36:49.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f28a5e72:::benchmark_data_smithi173_74914_object61331:head# 2024-08-30T23:36:49.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f28a77cd:::benchmark_data_smithi173_74914_object58004:head# 2024-08-30T23:36:49.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f28c6fee:::benchmark_data_smithi173_74914_object75745:head# 2024-08-30T23:36:49.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f28e3eef:::benchmark_data_smithi173_74914_object75278:head# 2024-08-30T23:36:49.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f28f7bc3:::benchmark_data_smithi173_74914_object78124:head# 2024-08-30T23:36:49.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f28fed99:::benchmark_data_smithi173_74914_object89970:head# 2024-08-30T23:36:49.594 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f294e112:::benchmark_data_smithi173_74914_object66229:head# 2024-08-30T23:36:49.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f297fdde:::benchmark_data_smithi173_74914_object61010:head# 2024-08-30T23:36:49.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f298909e:::benchmark_data_smithi173_74914_object57259:head# 2024-08-30T23:36:49.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f29b4418:::benchmark_data_smithi173_74914_object63050:head# 2024-08-30T23:36:49.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f29ebadb:::benchmark_data_smithi173_74914_object73950:head# 2024-08-30T23:36:49.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a021a6:::benchmark_data_smithi173_74914_object84961:head# 2024-08-30T23:36:49.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a1cb7f:::benchmark_data_smithi173_74914_object93297:head# 2024-08-30T23:36:49.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a22022:::benchmark_data_smithi173_74914_object55007:head# 2024-08-30T23:36:49.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a2b5cd:::benchmark_data_smithi173_74914_object61365:head# 2024-08-30T23:36:49.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a50b51:::benchmark_data_smithi173_74914_object55246:head# 2024-08-30T23:36:49.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a5897c:::benchmark_data_smithi173_74914_object77957:head# 2024-08-30T23:36:49.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2a9dc25:::benchmark_data_smithi173_74914_object87714:head# 2024-08-30T23:36:49.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2abf328:::benchmark_data_smithi173_74914_object52824:head# 2024-08-30T23:36:49.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ac9db8:::benchmark_data_smithi173_74914_object54809:head# 2024-08-30T23:36:49.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2adab4c:::benchmark_data_smithi173_74914_object91229:head# 2024-08-30T23:36:49.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2af44a2:::benchmark_data_smithi173_74914_object77326:head# 2024-08-30T23:36:49.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b064c2:::benchmark_data_smithi173_74914_object52825:head# 2024-08-30T23:36:49.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b23cc8:::benchmark_data_smithi173_74914_object75846:head# 2024-08-30T23:36:49.604 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b26fde:::benchmark_data_smithi173_74914_object56293:head# 2024-08-30T23:36:49.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b59b05:::benchmark_data_smithi173_74914_object65267:head# 2024-08-30T23:36:49.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b66538:::benchmark_data_smithi173_74914_object78836:head# 2024-08-30T23:36:49.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b6c5a1:::benchmark_data_smithi173_74914_object60181:head# 2024-08-30T23:36:49.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b9638a:::benchmark_data_smithi173_74914_object94426:head# 2024-08-30T23:36:49.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b993d9:::benchmark_data_smithi173_74914_object69568:head# 2024-08-30T23:36:49.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2b9f3bc:::benchmark_data_smithi173_74914_object67629:head# 2024-08-30T23:36:49.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ba4178:::benchmark_data_smithi173_74914_object66994:head# 2024-08-30T23:36:49.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2bacb01:::benchmark_data_smithi173_74914_object61489:head# 2024-08-30T23:36:49.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2bb7d33:::benchmark_data_smithi173_74914_object85656:head# 2024-08-30T23:36:49.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2bca998:::benchmark_data_smithi173_74914_object74098:head# 2024-08-30T23:36:49.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2bdedf1:::benchmark_data_smithi173_74914_object91034:head# 2024-08-30T23:36:49.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c0c7eb:::benchmark_data_smithi173_74914_object88666:head# 2024-08-30T23:36:49.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c31f4c:::benchmark_data_smithi173_74914_object81858:head# 2024-08-30T23:36:49.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c64b02:::benchmark_data_smithi173_74914_object84989:head# 2024-08-30T23:36:49.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c7c647:::benchmark_data_smithi173_74914_object73884:head# 2024-08-30T23:36:49.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c81aaa:::benchmark_data_smithi173_74914_object95218:head# 2024-08-30T23:36:49.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c91d24:::benchmark_data_smithi173_74914_object56298:head# 2024-08-30T23:36:49.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2c94f3f:::benchmark_data_smithi173_74914_object66866:head# 2024-08-30T23:36:49.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2cb93e7:::benchmark_data_smithi173_74914_object60182:head# 2024-08-30T23:36:49.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2cbd97f:::benchmark_data_smithi173_74914_object62784:head# 2024-08-30T23:36:49.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ccbeb3:::benchmark_data_smithi173_74914_object62417:head# 2024-08-30T23:36:49.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2cde177:::benchmark_data_smithi173_74914_object79056:head# 2024-08-30T23:36:49.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2cf268e:::benchmark_data_smithi173_74914_object78031:head# 2024-08-30T23:36:49.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2d0f7de:::benchmark_data_smithi173_74914_object54235:head# 2024-08-30T23:36:49.615 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2d17dbc:::benchmark_data_smithi173_74914_object67126:head# 2024-08-30T23:36:49.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2d2dd0b:::benchmark_data_smithi173_74914_object68970:head# 2024-08-30T23:36:49.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2d71ff1:::benchmark_data_smithi173_74914_object92970:head# 2024-08-30T23:36:49.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2d8f566:::benchmark_data_smithi173_74914_object54574:head# 2024-08-30T23:36:49.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2d91bfa:::benchmark_data_smithi173_74914_object70115:head# 2024-08-30T23:36:49.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2daa94e:::benchmark_data_smithi173_74914_object66273:head# 2024-08-30T23:36:49.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2db2297:::benchmark_data_smithi173_74914_object85610:head# 2024-08-30T23:36:49.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ded113:::benchmark_data_smithi173_74914_object69164:head# 2024-08-30T23:36:49.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e1738a:::benchmark_data_smithi173_74914_object54880:head# 2024-08-30T23:36:49.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e22f11:::benchmark_data_smithi173_74914_object73388:head# 2024-08-30T23:36:49.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e315b8:::benchmark_data_smithi173_74914_object54742:head# 2024-08-30T23:36:49.621 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e3474d:::benchmark_data_smithi173_74914_object83672:head# 2024-08-30T23:36:49.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e4ec51:::benchmark_data_smithi173_74914_object67077:head# 2024-08-30T23:36:49.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e8692e:::benchmark_data_smithi173_74914_object63602:head# 2024-08-30T23:36:49.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e872ba:::benchmark_data_smithi173_74914_object60947:head# 2024-08-30T23:36:49.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2e8e5c5:::benchmark_data_smithi173_74914_object94219:head# 2024-08-30T23:36:49.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ebea63:::benchmark_data_smithi173_74914_object59363:head# 2024-08-30T23:36:49.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ec09ad:::benchmark_data_smithi173_74914_object76225:head# 2024-08-30T23:36:49.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ed3dcd:::benchmark_data_smithi173_74914_object59234:head# 2024-08-30T23:36:49.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ee73ec:::benchmark_data_smithi173_74914_object86261:head# 2024-08-30T23:36:49.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ee7700:::benchmark_data_smithi173_74914_object68620:head# 2024-08-30T23:36:49.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2eebe4c:::benchmark_data_smithi173_74914_object54451:head# 2024-08-30T23:36:49.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2ef1b7f:::benchmark_data_smithi173_74914_object74523:head# 2024-08-30T23:36:49.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f02481:::benchmark_data_smithi173_74914_object78649:head# 2024-08-30T23:36:49.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f048a6:::benchmark_data_smithi173_74914_object68402:head# 2024-08-30T23:36:49.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f0eccb:::benchmark_data_smithi173_74914_object93110:head# 2024-08-30T23:36:49.629 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f2a9c0:::benchmark_data_smithi173_74914_object88336:head# 2024-08-30T23:36:49.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f46fa2:::benchmark_data_smithi173_74914_object78216:head# 2024-08-30T23:36:49.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f9483e:::benchmark_data_smithi173_74914_object88145:head# 2024-08-30T23:36:49.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2f9d5b8:::benchmark_data_smithi173_74914_object53393:head# 2024-08-30T23:36:49.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2fa2867:::benchmark_data_smithi173_74914_object86865:head# 2024-08-30T23:36:49.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2fa9d9f:::benchmark_data_smithi173_74914_object58935:head# 2024-08-30T23:36:49.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2fc0f98:::benchmark_data_smithi173_74914_object66888:head# 2024-08-30T23:36:49.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f2fc835d:::benchmark_data_smithi173_74914_object85237:head# 2024-08-30T23:36:49.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f304f0c6:::benchmark_data_smithi173_74914_object53226:head# 2024-08-30T23:36:49.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3057325:::benchmark_data_smithi173_74914_object71258:head# 2024-08-30T23:36:49.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f309c4cf:::benchmark_data_smithi173_74914_object71255:head# 2024-08-30T23:36:49.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f30b0ad4:::benchmark_data_smithi173_74914_object74522:head# 2024-08-30T23:36:49.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f30b42d0:::benchmark_data_smithi173_74914_object57952:head# 2024-08-30T23:36:49.637 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f30ba251:::benchmark_data_smithi173_74914_object77612:head# 2024-08-30T23:36:49.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f30d6a14:::benchmark_data_smithi173_74914_object81769:head# 2024-08-30T23:36:49.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f30de516:::benchmark_data_smithi173_74914_object81357:head# 2024-08-30T23:36:49.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31050f3:::benchmark_data_smithi173_74914_object90705:head# 2024-08-30T23:36:49.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3151e72:::benchmark_data_smithi173_74914_object79177:head# 2024-08-30T23:36:49.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f317164a:::benchmark_data_smithi173_74914_object86304:head# 2024-08-30T23:36:49.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3175982:::benchmark_data_smithi173_74914_object69258:head# 2024-08-30T23:36:49.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f319bf25:::benchmark_data_smithi173_74914_object95438:head# 2024-08-30T23:36:49.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31a7772:::benchmark_data_smithi173_74914_object86847:head# 2024-08-30T23:36:49.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31c0cc9:::benchmark_data_smithi173_74914_object60806:head# 2024-08-30T23:36:49.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31cfc96:::benchmark_data_smithi173_74914_object97222:head# 2024-08-30T23:36:49.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31d0d73:::benchmark_data_smithi173_74914_object67677:head# 2024-08-30T23:36:49.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31d5be8:::benchmark_data_smithi173_74914_object81362:head# 2024-08-30T23:36:49.647 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31eea26:::benchmark_data_smithi173_74914_object90143:head# 2024-08-30T23:36:49.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f31f289c:::benchmark_data_smithi173_74914_object69760:head# 2024-08-30T23:36:49.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f320488f:::benchmark_data_smithi173_74914_object91984:head# 2024-08-30T23:36:49.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3219f82:::benchmark_data_smithi173_74914_object60381:head# 2024-08-30T23:36:49.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f321a884:::benchmark_data_smithi173_74914_object65303:head# 2024-08-30T23:36:49.649 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3228bd2:::benchmark_data_smithi173_74914_object88455:head# 2024-08-30T23:36:49.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f324f6b5:::benchmark_data_smithi173_74914_object53540:head# 2024-08-30T23:36:49.650 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3258412:::benchmark_data_smithi173_74914_object96133:head# 2024-08-30T23:36:49.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f32618c4:::benchmark_data_smithi173_74914_object78648:head# 2024-08-30T23:36:49.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f327c793:::benchmark_data_smithi173_74914_object74751:head# 2024-08-30T23:36:49.651 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f327f012:::benchmark_data_smithi173_74914_object87177:head# 2024-08-30T23:36:49.652 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f32ad58f:::benchmark_data_smithi173_74914_object60511:head# 2024-08-30T23:36:49.653 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f32be216:::benchmark_data_smithi173_74914_object67483:head# 2024-08-30T23:36:49.654 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f32c24d7:::benchmark_data_smithi173_74914_object64234:head# 2024-08-30T23:36:49.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f32cdf52:::benchmark_data_smithi173_74914_object62819:head# 2024-08-30T23:36:49.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3318342:::benchmark_data_smithi173_74914_object70649:head# 2024-08-30T23:36:49.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f332d353:::benchmark_data_smithi173_74914_object68924:head# 2024-08-30T23:36:49.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3333cb7:::benchmark_data_smithi173_74914_object94048:head# 2024-08-30T23:36:49.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f333da0b:::benchmark_data_smithi173_74914_object56847:head# 2024-08-30T23:36:49.659 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f33648fa:::benchmark_data_smithi173_74914_object58946:head# 2024-08-30T23:36:49.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3381ff3:::benchmark_data_smithi173_74914_object56861:head# 2024-08-30T23:36:49.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f338934c:::benchmark_data_smithi173_74914_object78709:head# 2024-08-30T23:36:49.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f33b1ca8:::benchmark_data_smithi173_74914_object68800:head# 2024-08-30T23:36:49.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f33c8875:::benchmark_data_smithi173_74914_object53351:head# 2024-08-30T23:36:49.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f33d8967:::benchmark_data_smithi173_74914_object84273:head# 2024-08-30T23:36:49.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f33f9204:::benchmark_data_smithi173_74914_object60505:head# 2024-08-30T23:36:49.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3429b09:::benchmark_data_smithi173_74914_object74691:head# 2024-08-30T23:36:49.665 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3443ee1:::benchmark_data_smithi173_74914_object77896:head# 2024-08-30T23:36:49.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f345ebcc:::benchmark_data_smithi173_74914_object67967:head# 2024-08-30T23:36:49.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f34753f7:::benchmark_data_smithi173_74914_object58232:head# 2024-08-30T23:36:49.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f349b258:::benchmark_data_smithi173_74914_object89784:head# 2024-08-30T23:36:49.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f34aba90:::benchmark_data_smithi173_74914_object64562:head# 2024-08-30T23:36:49.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f34b9457:::benchmark_data_smithi173_74914_object84103:head# 2024-08-30T23:36:49.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f34d971d:::benchmark_data_smithi173_74914_object56229:head# 2024-08-30T23:36:49.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35053b1:::benchmark_data_smithi173_74914_object97020:head# 2024-08-30T23:36:49.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f352b44b:::benchmark_data_smithi173_74914_object59617:head# 2024-08-30T23:36:49.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35540b0:::benchmark_data_smithi173_74914_object90386:head# 2024-08-30T23:36:49.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f356845d:::benchmark_data_smithi173_74914_object85013:head# 2024-08-30T23:36:49.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3580539:::benchmark_data_smithi173_74914_object53099:head# 2024-08-30T23:36:49.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3586d30:::benchmark_data_smithi173_74914_object64914:head# 2024-08-30T23:36:49.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35954df:::benchmark_data_smithi173_74914_object55679:head# 2024-08-30T23:36:49.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35a146a:::benchmark_data_smithi173_74914_object52939:head# 2024-08-30T23:36:49.675 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35af60a:::benchmark_data_smithi173_74914_object91926:head# 2024-08-30T23:36:49.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35bb1ab:::benchmark_data_smithi173_74914_object91927:head# 2024-08-30T23:36:49.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35c02ce:::benchmark_data_smithi173_74914_object88290:head# 2024-08-30T23:36:49.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f35f2467:::benchmark_data_smithi173_74914_object85790:head# 2024-08-30T23:36:49.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f361966f:::benchmark_data_smithi173_74914_object79456:head# 2024-08-30T23:36:49.679 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3625d5b:::benchmark_data_smithi173_74914_object64825:head# 2024-08-30T23:36:49.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f367e5eb:::benchmark_data_smithi173_74914_object61536:head# 2024-08-30T23:36:49.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3687607:::benchmark_data_smithi173_74914_object94437:head# 2024-08-30T23:36:49.681 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f369292a:::benchmark_data_smithi173_74914_object84572:head# 2024-08-30T23:36:49.681 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f369393c:::benchmark_data_smithi173_74914_object74397:head# 2024-08-30T23:36:49.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f369504a:::benchmark_data_smithi173_74914_object70775:head# 2024-08-30T23:36:49.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f36bb988:::benchmark_data_smithi173_74914_object80554:head# 2024-08-30T23:36:49.683 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f36c6a60:::benchmark_data_smithi173_74914_object90660:head# 2024-08-30T23:36:49.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f36e763a:::benchmark_data_smithi173_74914_object77872:head# 2024-08-30T23:36:49.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:49.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3700d44:::benchmark_data_smithi173_74914_object71148:head# 2024-08-30T23:36:49.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3705b86:::benchmark_data_smithi173_74914_object85111:head# 2024-08-30T23:36:49.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3723b10:::benchmark_data_smithi173_74914_object66754:head# 2024-08-30T23:36:49.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f374a31d:::benchmark_data_smithi173_74914_object71111:head# 2024-08-30T23:36:49.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3755a83:::benchmark_data_smithi173_74914_object91997:head# 2024-08-30T23:36:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3778a82:::benchmark_data_smithi173_74914_object72274:head# 2024-08-30T23:36:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f377d3f5:::benchmark_data_smithi173_74914_object58252:head# 2024-08-30T23:36:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f37bf3e4:::benchmark_data_smithi173_74914_object77348:head# 2024-08-30T23:36:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f37ee217:::benchmark_data_smithi173_74914_object58081:head# 2024-08-30T23:36:49.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f37f8d66:::benchmark_data_smithi173_74914_object79941:head# 2024-08-30T23:36:49.689 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f37f9233:::benchmark_data_smithi173_74914_object64689:head# 2024-08-30T23:36:49.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f38036f6:::benchmark_data_smithi173_74914_object68917:head# 2024-08-30T23:36:49.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f381ac2e:::benchmark_data_smithi173_74914_object55072:head# 2024-08-30T23:36:49.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f383061f:::benchmark_data_smithi173_74914_object82313:head# 2024-08-30T23:36:49.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f38733ad:::benchmark_data_smithi173_74914_object59092:head# 2024-08-30T23:36:49.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f387e414:::benchmark_data_smithi173_74914_object83147:head# 2024-08-30T23:36:49.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f387f36a:::benchmark_data_smithi173_74914_object84782:head# 2024-08-30T23:36:49.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f38951a1:::benchmark_data_smithi173_74914_object57224:head# 2024-08-30T23:36:49.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f38d9f57:::benchmark_data_smithi173_74914_object58096:head# 2024-08-30T23:36:49.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f390c3b2:::benchmark_data_smithi173_74914_object78353:head# 2024-08-30T23:36:49.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f39236ba:::benchmark_data_smithi173_74914_object63386:head# 2024-08-30T23:36:49.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f39377db:::benchmark_data_smithi173_74914_object57504:head# 2024-08-30T23:36:49.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3938151:::benchmark_data_smithi173_74914_object79465:head# 2024-08-30T23:36:49.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f394c238:::benchmark_data_smithi173_74914_object82662:head# 2024-08-30T23:36:49.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f398801d:::benchmark_data_smithi173_74914_object95838:head# 2024-08-30T23:36:49.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f39daa3a:::benchmark_data_smithi173_74914_object98230:head# 2024-08-30T23:36:49.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f39ddd3c:::benchmark_data_smithi173_74914_object53704:head# 2024-08-30T23:36:49.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f39f3f7f:::benchmark_data_smithi173_74914_object59989:head# 2024-08-30T23:36:49.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a1cdd0:::benchmark_data_smithi173_74914_object64694:head# 2024-08-30T23:36:49.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a34bc1:::benchmark_data_smithi173_74914_object67414:head# 2024-08-30T23:36:49.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a553a3:::benchmark_data_smithi173_74914_object74330:head# 2024-08-30T23:36:49.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:49 smithi136 ceph-mon[20207]: 4.1 scrub starts 2024-08-30T23:36:49.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:49 smithi136 ceph-mon[20207]: 4.2 scrub starts 2024-08-30T23:36:49.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:49 smithi136 ceph-mon[20207]: pgmap v904: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s; 10693/86757 objects degraded (12.325%) 2024-08-30T23:36:49.705 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:49 smithi136 ceph-mon[20207]: 4.2 scrub ok 2024-08-30T23:36:49.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a6bbeb:::benchmark_data_smithi173_74914_object71504:head# 2024-08-30T23:36:49.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a6edbd:::benchmark_data_smithi173_74914_object89287:head# 2024-08-30T23:36:49.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a77496:::benchmark_data_smithi173_74914_object72727:head# 2024-08-30T23:36:49.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3a9497b:::benchmark_data_smithi173_74914_object77164:head# 2024-08-30T23:36:49.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3aa258b:::benchmark_data_smithi173_74914_object78790:head# 2024-08-30T23:36:49.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3abc660:::benchmark_data_smithi173_74914_object74685:head# 2024-08-30T23:36:49.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3b1d432:::benchmark_data_smithi173_74914_object88409:head# 2024-08-30T23:36:49.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3b7447a:::benchmark_data_smithi173_74914_object90206:head# 2024-08-30T23:36:49.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3bb9755:::benchmark_data_smithi173_74914_object82501:head# 2024-08-30T23:36:49.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3bf3b6b:::benchmark_data_smithi173_74914_object97847:head# 2024-08-30T23:36:49.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3bfdfdc:::benchmark_data_smithi173_74914_object73794:head# 2024-08-30T23:36:49.709 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3c23b30:::benchmark_data_smithi173_74914_object63786:head# 2024-08-30T23:36:49.710 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3c27bab:::benchmark_data_smithi173_74914_object57309:head# 2024-08-30T23:36:49.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3c759c1:::benchmark_data_smithi173_74914_object75264:head# 2024-08-30T23:36:49.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3c91cd7:::benchmark_data_smithi173_74914_object81443:head# 2024-08-30T23:36:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3c96953:::benchmark_data_smithi173_74914_object97469:head# 2024-08-30T23:36:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3cbcb0a:::benchmark_data_smithi173_74914_object63752:head# 2024-08-30T23:36:49.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ccf8c2:::benchmark_data_smithi173_74914_object67840:head# 2024-08-30T23:36:49.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3cddc00:::benchmark_data_smithi173_74914_object70885:head# 2024-08-30T23:36:49.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ceea7d:::benchmark_data_smithi173_74914_object57209:head# 2024-08-30T23:36:49.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3d28edb:::benchmark_data_smithi173_74914_object63462:head# 2024-08-30T23:36:49.719 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3d29497:::benchmark_data_smithi173_74914_object75711:head# 2024-08-30T23:36:49.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3d486bb:::benchmark_data_smithi173_74914_object57171:head# 2024-08-30T23:36:49.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3d7e955:::benchmark_data_smithi173_74914_object58742:head# 2024-08-30T23:36:49.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3d89ebc:::benchmark_data_smithi173_74914_object74700:head# 2024-08-30T23:36:49.721 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3d8f7ce:::benchmark_data_smithi173_74914_object81114:head# 2024-08-30T23:36:49.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3da3b2e:::benchmark_data_smithi173_74914_object67600:head# 2024-08-30T23:36:49.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:49.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3dc09a1:::benchmark_data_smithi173_74914_object54127:head# 2024-08-30T23:36:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:49.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3dc8216:::benchmark_data_smithi173_74914_object98193:head# 2024-08-30T23:36:49.724 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3def60a:::benchmark_data_smithi173_74914_object98065:head# 2024-08-30T23:36:49.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3e15082:::benchmark_data_smithi173_74914_object97466:head# 2024-08-30T23:36:49.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3e2b614:::benchmark_data_smithi173_74914_object84624:head# 2024-08-30T23:36:49.725 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3e4a541:::benchmark_data_smithi173_74914_object66767:head# 2024-08-30T23:36:49.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3e66696:::benchmark_data_smithi173_74914_object67724:head# 2024-08-30T23:36:49.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3e9a4d1:::benchmark_data_smithi173_74914_object71947:head# 2024-08-30T23:36:49.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ea7b05:::benchmark_data_smithi173_74914_object88573:head# 2024-08-30T23:36:49.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3eaa19a:::benchmark_data_smithi173_74914_object56667:head# 2024-08-30T23:36:49.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ec3037:::benchmark_data_smithi173_74914_object97822:head# 2024-08-30T23:36:49.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ec44ea:::benchmark_data_smithi173_74914_object78566:head# 2024-08-30T23:36:49.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ed4cbb:::benchmark_data_smithi173_74914_object89159:head# 2024-08-30T23:36:49.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ee0aba:::benchmark_data_smithi173_74914_object88975:head# 2024-08-30T23:36:49.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3eec947:::benchmark_data_smithi173_74914_object63764:head# 2024-08-30T23:36:49.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3ef8080:::benchmark_data_smithi173_74914_object61960:head# 2024-08-30T23:36:49.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3efcc62:::benchmark_data_smithi173_74914_object57829:head# 2024-08-30T23:36:49.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3f1a002:::benchmark_data_smithi173_74914_object55704:head# 2024-08-30T23:36:49.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3f1f5df:::benchmark_data_smithi173_74914_object62389:head# 2024-08-30T23:36:49.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3f2a88a:::benchmark_data_smithi173_74914_object93295:head# 2024-08-30T23:36:49.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3f69ef6:::benchmark_data_smithi173_74914_object82253:head# 2024-08-30T23:36:49.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3f73dda:::benchmark_data_smithi173_74914_object88552:head# 2024-08-30T23:36:49.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3f8c3b3:::benchmark_data_smithi173_74914_object77805:head# 2024-08-30T23:36:49.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fa6827:::benchmark_data_smithi173_74914_object88206:head# 2024-08-30T23:36:49.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fa9340:::benchmark_data_smithi173_74914_object77804:head# 2024-08-30T23:36:49.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fbfd0a:::benchmark_data_smithi173_74914_object61747:head# 2024-08-30T23:36:49.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fc24c0:::benchmark_data_smithi173_74914_object89655:head# 2024-08-30T23:36:49.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fdfe41:::benchmark_data_smithi173_74914_object66016:head# 2024-08-30T23:36:49.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fe563e:::benchmark_data_smithi173_74914_object87584:head# 2024-08-30T23:36:49.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f3fee8e0:::benchmark_data_smithi173_74914_object90702:head# 2024-08-30T23:36:49.741 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4008663:::benchmark_data_smithi173_74914_object89519:head# 2024-08-30T23:36:49.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f40283a3:::benchmark_data_smithi173_74914_object97513:head# 2024-08-30T23:36:49.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4030238:::benchmark_data_smithi173_74914_object88391:head# 2024-08-30T23:36:49.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f404ed43:::benchmark_data_smithi173_74914_object93828:head# 2024-08-30T23:36:49.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f405f434:::benchmark_data_smithi173_74914_object79780:head# 2024-08-30T23:36:49.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f405f526:::benchmark_data_smithi173_74914_object55836:head# 2024-08-30T23:36:49.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f406539a:::benchmark_data_smithi173_74914_object74961:head# 2024-08-30T23:36:49.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4084e9c:::benchmark_data_smithi173_74914_object92662:head# 2024-08-30T23:36:49.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f408698b:::benchmark_data_smithi173_74914_object70148:head# 2024-08-30T23:36:49.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4095a75:::benchmark_data_smithi173_74914_object70882:head# 2024-08-30T23:36:49.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f40a0db2:::benchmark_data_smithi173_74914_object89827:head# 2024-08-30T23:36:49.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f40abdf1:::benchmark_data_smithi173_74914_object84868:head# 2024-08-30T23:36:49.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f40ac2cf:::benchmark_data_smithi173_74914_object85702:head# 2024-08-30T23:36:49.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f40b2a93:::benchmark_data_smithi173_74914_object71780:head# 2024-08-30T23:36:49.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f40cca28:::benchmark_data_smithi173_74914_object75608:head# 2024-08-30T23:36:49.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4100835:::benchmark_data_smithi173_74914_object56091:head# 2024-08-30T23:36:49.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f410a41a:::benchmark_data_smithi173_74914_object97579:head# 2024-08-30T23:36:49.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4114569:::benchmark_data_smithi173_74914_object56922:head# 2024-08-30T23:36:49.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4136715:::benchmark_data_smithi173_74914_object54372:head# 2024-08-30T23:36:49.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f413c223:::benchmark_data_smithi173_74914_object87585:head# 2024-08-30T23:36:49.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f413d6a8:::benchmark_data_smithi173_74914_object85327:head# 2024-08-30T23:36:49.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f41ba510:::benchmark_data_smithi173_74914_object54673:head# 2024-08-30T23:36:49.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f41d3976:::benchmark_data_smithi173_74914_object58902:head# 2024-08-30T23:36:49.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f41d62be:::benchmark_data_smithi173_74914_object85100:head# 2024-08-30T23:36:49.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f41e6b75:::benchmark_data_smithi173_74914_object84142:head# 2024-08-30T23:36:49.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4208a9b:::benchmark_data_smithi173_74914_object95426:head# 2024-08-30T23:36:49.753 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f420e070:::benchmark_data_smithi173_74914_object71425:head# 2024-08-30T23:36:49.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f422aed1:::benchmark_data_smithi173_74914_object69548:head# 2024-08-30T23:36:49.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f424879a:::benchmark_data_smithi173_74914_object97363:head# 2024-08-30T23:36:49.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f424a2da:::benchmark_data_smithi173_74914_object89215:head# 2024-08-30T23:36:49.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f424bf7c:::benchmark_data_smithi173_74914_object60522:head# 2024-08-30T23:36:49.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f425a930:::benchmark_data_smithi173_74914_object93775:head# 2024-08-30T23:36:49.756 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f427dbee:::benchmark_data_smithi173_74914_object90136:head# 2024-08-30T23:36:49.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4291d7e:::benchmark_data_smithi173_74914_object82614:head# 2024-08-30T23:36:49.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f42c4647:::benchmark_data_smithi173_74914_object88011:head# 2024-08-30T23:36:49.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f42c98ad:::benchmark_data_smithi173_74914_object77905:head# 2024-08-30T23:36:49.758 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f42d6529:::benchmark_data_smithi173_74914_object80272:head# 2024-08-30T23:36:49.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f42e8542:::benchmark_data_smithi173_74914_object69995:head# 2024-08-30T23:36:49.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f42f0ed4:::benchmark_data_smithi173_74914_object73624:head# 2024-08-30T23:36:49.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f42f2c68:::benchmark_data_smithi173_74914_object92680:head# 2024-08-30T23:36:49.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f430fab3:::benchmark_data_smithi173_74914_object65237:head# 2024-08-30T23:36:49.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f431c137:::benchmark_data_smithi173_74914_object79112:head# 2024-08-30T23:36:49.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f432114e:::benchmark_data_smithi173_74914_object97739:head# 2024-08-30T23:36:49.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f435fd71:::benchmark_data_smithi173_74914_object96060:head# 2024-08-30T23:36:49.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f43b3009:::benchmark_data_smithi173_74914_object87726:head# 2024-08-30T23:36:49.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f43b377a:::benchmark_data_smithi173_74914_object66933:head# 2024-08-30T23:36:49.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f43de8c0:::benchmark_data_smithi173_74914_object55720:head# 2024-08-30T23:36:49.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f43f6e20:::benchmark_data_smithi173_74914_object89945:head# 2024-08-30T23:36:49.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f44035d6:::benchmark_data_smithi173_74914_object65279:head# 2024-08-30T23:36:49.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4407ec4:::benchmark_data_smithi173_74914_object56973:head# 2024-08-30T23:36:49.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4418f1d:::benchmark_data_smithi173_74914_object77232:head# 2024-08-30T23:36:49.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f441a34f:::benchmark_data_smithi173_74914_object79221:head# 2024-08-30T23:36:49.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f443725d:::benchmark_data_smithi173_74914_object71049:head# 2024-08-30T23:36:49.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f44528da:::benchmark_data_smithi173_74914_object78555:head# 2024-08-30T23:36:49.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4456d35:::benchmark_data_smithi173_74914_object94728:head# 2024-08-30T23:36:49.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4458847:::benchmark_data_smithi173_74914_object90226:head# 2024-08-30T23:36:49.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4467dd3:::benchmark_data_smithi173_74914_object67670:head# 2024-08-30T23:36:49.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f446b4d7:::benchmark_data_smithi173_74914_object85545:head# 2024-08-30T23:36:49.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f448280f:::benchmark_data_smithi173_74914_object88906:head# 2024-08-30T23:36:49.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4485363:::benchmark_data_smithi173_74914_object61308:head# 2024-08-30T23:36:49.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f449d688:::benchmark_data_smithi173_74914_object78712:head# 2024-08-30T23:36:49.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f454ff26:::benchmark_data_smithi173_74914_object90117:head# 2024-08-30T23:36:49.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f455908f:::benchmark_data_smithi173_74914_object57788:head# 2024-08-30T23:36:49.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f457fa5b:::benchmark_data_smithi173_74914_object56079:head# 2024-08-30T23:36:49.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f459eb1c:::benchmark_data_smithi173_74914_object64939:head# 2024-08-30T23:36:49.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f45b08c1:::benchmark_data_smithi173_74914_object77029:head# 2024-08-30T23:36:49.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f45c3cf0:::benchmark_data_smithi173_74914_object90272:head# 2024-08-30T23:36:49.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f463c9bb:::benchmark_data_smithi173_74914_object53819:head# 2024-08-30T23:36:49.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f463dc91:::benchmark_data_smithi173_74914_object65486:head# 2024-08-30T23:36:49.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4662484:::benchmark_data_smithi173_74914_object73816:head# 2024-08-30T23:36:49.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4662ba9:::benchmark_data_smithi173_74914_object66897:head# 2024-08-30T23:36:49.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f466c551:::benchmark_data_smithi173_74914_object91132:head# 2024-08-30T23:36:49.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4695535:::benchmark_data_smithi173_74914_object84521:head# 2024-08-30T23:36:49.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f46b0448:::benchmark_data_smithi173_74914_object55113:head# 2024-08-30T23:36:49.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f46eec30:::benchmark_data_smithi173_74914_object72963:head# 2024-08-30T23:36:49.778 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f46f85aa:::benchmark_data_smithi173_74914_object92455:head# 2024-08-30T23:36:49.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f472be6b:::benchmark_data_smithi173_74914_object88038:head# 2024-08-30T23:36:49.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f475f8c2:::benchmark_data_smithi173_74914_object79513:head# 2024-08-30T23:36:49.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f477489f:::benchmark_data_smithi173_74914_object76223:head# 2024-08-30T23:36:49.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f47812c3:::benchmark_data_smithi173_74914_object67847:head# 2024-08-30T23:36:49.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f47917f4:::benchmark_data_smithi173_74914_object71560:head# 2024-08-30T23:36:49.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f47ca869:::benchmark_data_smithi173_74914_object91660:head# 2024-08-30T23:36:49.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f47e49ef:::benchmark_data_smithi173_74914_object64748:head# 2024-08-30T23:36:49.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f47ed469:::benchmark_data_smithi173_74914_object98125:head# 2024-08-30T23:36:49.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4800c16:::benchmark_data_smithi173_74914_object84181:head# 2024-08-30T23:36:49.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f48391ce:::benchmark_data_smithi173_74914_object91748:head# 2024-08-30T23:36:49.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4864213:::benchmark_data_smithi173_74914_object82518:head# 2024-08-30T23:36:49.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f488167a:::benchmark_data_smithi173_74914_object54527:head# 2024-08-30T23:36:49.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f488523c:::benchmark_data_smithi173_74914_object68351:head# 2024-08-30T23:36:49.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4885bd7:::benchmark_data_smithi173_74914_object89379:head# 2024-08-30T23:36:49.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f488b350:::benchmark_data_smithi173_74914_object91989:head# 2024-08-30T23:36:49.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f488ba5e:::benchmark_data_smithi173_74914_object87846:head# 2024-08-30T23:36:49.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f489dd8b:::benchmark_data_smithi173_74914_object94103:head# 2024-08-30T23:36:49.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f48a46c2:::benchmark_data_smithi173_74914_object53407:head# 2024-08-30T23:36:49.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f48e7549:::benchmark_data_smithi173_74914_object98061:head# 2024-08-30T23:36:49.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f48eedea:::benchmark_data_smithi173_74914_object78879:head# 2024-08-30T23:36:49.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f48f4ac6:::benchmark_data_smithi173_74914_object58886:head# 2024-08-30T23:36:49.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f490aac0:::benchmark_data_smithi173_74914_object54352:head# 2024-08-30T23:36:49.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f493e4ae:::benchmark_data_smithi173_74914_object63429:head# 2024-08-30T23:36:49.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4994f86:::benchmark_data_smithi173_74914_object56123:head# 2024-08-30T23:36:49.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f49dae3b:::benchmark_data_smithi173_74914_object78941:head# 2024-08-30T23:36:49.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f49ebc6a:::benchmark_data_smithi173_74914_object92523:head# 2024-08-30T23:36:49.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4a394b7:::benchmark_data_smithi173_74914_object53416:head# 2024-08-30T23:36:49.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4a59c2e:::benchmark_data_smithi173_74914_object79545:head# 2024-08-30T23:36:49.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4a688a3:::benchmark_data_smithi173_74914_object52969:head# 2024-08-30T23:36:49.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4a78cc0:::benchmark_data_smithi173_74914_object76066:head# 2024-08-30T23:36:49.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4a9936b:::benchmark_data_smithi173_74914_object64631:head# 2024-08-30T23:36:49.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4aa1cc6:::benchmark_data_smithi173_74914_object60288:head# 2024-08-30T23:36:49.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4aa836a:::benchmark_data_smithi173_74914_object59573:head# 2024-08-30T23:36:49.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4aba72d:::benchmark_data_smithi173_74914_object72608:head# 2024-08-30T23:36:49.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4abc597:::benchmark_data_smithi173_74914_object69757:head# 2024-08-30T23:36:49.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4abfec2:::benchmark_data_smithi173_74914_object84893:head# 2024-08-30T23:36:49.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4aec6f4:::benchmark_data_smithi173_74914_object68981:head# 2024-08-30T23:36:49.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4af5511:::benchmark_data_smithi173_74914_object71216:head# 2024-08-30T23:36:49.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b1e680:::benchmark_data_smithi173_74914_object59542:head# 2024-08-30T23:36:49.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b21283:::benchmark_data_smithi173_74914_object88730:head# 2024-08-30T23:36:49.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b3e689:::benchmark_data_smithi173_74914_object74709:head# 2024-08-30T23:36:49.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b4fcbd:::benchmark_data_smithi173_74914_object67814:head# 2024-08-30T23:36:49.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b63cdd:::benchmark_data_smithi173_74914_object83424:head# 2024-08-30T23:36:49.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b66445:::benchmark_data_smithi173_74914_object96653:head# 2024-08-30T23:36:49.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b75ef4:::benchmark_data_smithi173_74914_object65870:head# 2024-08-30T23:36:49.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b84aaf:::benchmark_data_smithi173_74914_object77768:head# 2024-08-30T23:36:49.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4b867e6:::benchmark_data_smithi173_74914_object96328:head# 2024-08-30T23:36:49.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4baf311:::benchmark_data_smithi173_74914_object97726:head# 2024-08-30T23:36:49.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4be5d55:::benchmark_data_smithi173_74914_object92850:head# 2024-08-30T23:36:49.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c00cde:::benchmark_data_smithi173_74914_object73351:head# 2024-08-30T23:36:49.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c1f80e:::benchmark_data_smithi173_74914_object61094:head# 2024-08-30T23:36:49.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c3ec51:::benchmark_data_smithi173_74914_object90041:head# 2024-08-30T23:36:49.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c42fcb:::benchmark_data_smithi173_74914_object94458:head# 2024-08-30T23:36:49.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c52d98:::benchmark_data_smithi173_74914_object89397:head# 2024-08-30T23:36:49.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c5c6fc:::benchmark_data_smithi173_74914_object54524:head# 2024-08-30T23:36:49.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c60ade:::benchmark_data_smithi173_74914_object56516:head# 2024-08-30T23:36:49.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c60ba6:::benchmark_data_smithi173_74914_object91499:head# 2024-08-30T23:36:49.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c6d836:::benchmark_data_smithi173_74914_object65424:head# 2024-08-30T23:36:49.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c7a594:::benchmark_data_smithi173_74914_object57120:head# 2024-08-30T23:36:49.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4c80c6b:::benchmark_data_smithi173_74914_object84548:head# 2024-08-30T23:36:49.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4caf8dc:::benchmark_data_smithi173_74914_object81071:head# 2024-08-30T23:36:49.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4cb65e0:::benchmark_data_smithi173_74914_object74536:head# 2024-08-30T23:36:49.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4cc8119:::benchmark_data_smithi173_74914_object94528:head# 2024-08-30T23:36:49.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4cd41c9:::benchmark_data_smithi173_74914_object73711:head# 2024-08-30T23:36:49.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4ce3d48:::benchmark_data_smithi173_74914_object80478:head# 2024-08-30T23:36:49.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4ce6a12:::benchmark_data_smithi173_74914_object96805:head# 2024-08-30T23:36:49.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4cf12a4:::benchmark_data_smithi173_74914_object53290:head# 2024-08-30T23:36:49.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4d20164:::benchmark_data_smithi173_74914_object82219:head# 2024-08-30T23:36:49.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4d321ca:::benchmark_data_smithi173_74914_object72480:head# 2024-08-30T23:36:49.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4d3fd94:::benchmark_data_smithi173_74914_object89581:head# 2024-08-30T23:36:49.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4d47b5d:::benchmark_data_smithi173_74914_object72300:head# 2024-08-30T23:36:49.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4d530a4:::benchmark_data_smithi173_74914_object52955:head# 2024-08-30T23:36:49.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4d9f4db:::benchmark_data_smithi173_74914_object54086:head# 2024-08-30T23:36:49.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4daec36:::benchmark_data_smithi173_74914_object59050:head# 2024-08-30T23:36:49.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4dba92c:::benchmark_data_smithi173_74914_object87943:head# 2024-08-30T23:36:49.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4ddac33:::benchmark_data_smithi173_74914_object89821:head# 2024-08-30T23:36:49.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4df25ad:::benchmark_data_smithi173_74914_object76940:head# 2024-08-30T23:36:49.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e040c2:::benchmark_data_smithi173_74914_object94097:head# 2024-08-30T23:36:49.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e0bd7c:::benchmark_data_smithi173_74914_object96713:head# 2024-08-30T23:36:49.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e0dbba:::benchmark_data_smithi173_74914_object96722:head# 2024-08-30T23:36:49.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e146ff:::benchmark_data_smithi173_74914_object69737:head# 2024-08-30T23:36:49.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e1cee0:::benchmark_data_smithi173_74914_object93159:head# 2024-08-30T23:36:49.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e25e09:::benchmark_data_smithi173_74914_object68408:head# 2024-08-30T23:36:49.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e34f0e:::benchmark_data_smithi173_74914_object85099:head# 2024-08-30T23:36:49.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e39018:::benchmark_data_smithi173_74914_object90306:head# 2024-08-30T23:36:49.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e4306d:::benchmark_data_smithi173_74914_object71685:head# 2024-08-30T23:36:49.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e4785d:::benchmark_data_smithi173_74914_object76851:head# 2024-08-30T23:36:49.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e4c6d9:::benchmark_data_smithi173_74914_object67291:head# 2024-08-30T23:36:49.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e502ad:::benchmark_data_smithi173_74914_object92177:head# 2024-08-30T23:36:49.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e5792f:::benchmark_data_smithi173_74914_object73335:head# 2024-08-30T23:36:49.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e5a6e3:::benchmark_data_smithi173_74914_object71662:head# 2024-08-30T23:36:49.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:49.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e755da:::benchmark_data_smithi173_74914_object82418:head# 2024-08-30T23:36:49.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e7919b:::benchmark_data_smithi173_74914_object64734:head# 2024-08-30T23:36:49.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4e92a9b:::benchmark_data_smithi173_74914_object76474:head# 2024-08-30T23:36:49.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4eace47:::benchmark_data_smithi173_74914_object76695:head# 2024-08-30T23:36:49.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4eee5cf:::benchmark_data_smithi173_74914_object63122:head# 2024-08-30T23:36:49.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4f0a7d4:::benchmark_data_smithi173_74914_object80874:head# 2024-08-30T23:36:49.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4f139c9:::benchmark_data_smithi173_74914_object66830:head# 2024-08-30T23:36:49.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4f3caec:::benchmark_data_smithi173_74914_object59588:head# 2024-08-30T23:36:49.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4f41f54:::benchmark_data_smithi173_74914_object63319:head# 2024-08-30T23:36:49.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4f82331:::benchmark_data_smithi173_74914_object98095:head# 2024-08-30T23:36:49.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4fad07b:::benchmark_data_smithi173_74914_object63787:head# 2024-08-30T23:36:49.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4fad0cd:::benchmark_data_smithi173_74914_object76499:head# 2024-08-30T23:36:49.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:49 smithi106 ceph-mon[16603]: 4.1 scrub starts 2024-08-30T23:36:49.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:49 smithi106 ceph-mon[16603]: 4.2 scrub starts 2024-08-30T23:36:49.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:49 smithi106 ceph-mon[16603]: pgmap v904: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s; 10693/86757 objects degraded (12.325%) 2024-08-30T23:36:49.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:49 smithi106 ceph-mon[16603]: 4.2 scrub ok 2024-08-30T23:36:49.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4fb4f61:::benchmark_data_smithi173_74914_object67354:head# 2024-08-30T23:36:49.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4fd76f4:::benchmark_data_smithi173_74914_object68173:head# 2024-08-30T23:36:49.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4fe8916:::benchmark_data_smithi173_74914_object86719:head# 2024-08-30T23:36:49.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4ff99fc:::benchmark_data_smithi173_74914_object97516:head# 2024-08-30T23:36:49.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f4ffa016:::benchmark_data_smithi173_74914_object80995:head# 2024-08-30T23:36:49.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5003fe0:::benchmark_data_smithi173_74914_object87396:head# 2024-08-30T23:36:49.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50045a2:::benchmark_data_smithi173_74914_object96802:head# 2024-08-30T23:36:49.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f500c9bb:::benchmark_data_smithi173_74914_object97087:head# 2024-08-30T23:36:49.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f501dd5f:::benchmark_data_smithi173_74914_object89156:head# 2024-08-30T23:36:49.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f504119e:::benchmark_data_smithi173_74914_object94581:head# 2024-08-30T23:36:49.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50965e7:::benchmark_data_smithi173_74914_object93315:head# 2024-08-30T23:36:49.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50a4fb3:::benchmark_data_smithi173_74914_object86471:head# 2024-08-30T23:36:49.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50c3313:::benchmark_data_smithi173_74914_object73121:head# 2024-08-30T23:36:49.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50c3361:::benchmark_data_smithi173_74914_object90309:head# 2024-08-30T23:36:49.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50d5f67:::benchmark_data_smithi173_74914_object90577:head# 2024-08-30T23:36:49.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50d9e11:::benchmark_data_smithi173_74914_object75958:head# 2024-08-30T23:36:49.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f50db352:::benchmark_data_smithi173_74914_object86650:head# 2024-08-30T23:36:49.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5117cda:::benchmark_data_smithi173_74914_object77941:head# 2024-08-30T23:36:49.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5151c1c:::benchmark_data_smithi173_74914_object96549:head# 2024-08-30T23:36:49.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f516393c:::benchmark_data_smithi173_74914_object62328:head# 2024-08-30T23:36:49.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5181a1f:::benchmark_data_smithi173_74914_object94366:head# 2024-08-30T23:36:49.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f519b723:::benchmark_data_smithi173_74914_object68229:head# 2024-08-30T23:36:49.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f51a0807:::benchmark_data_smithi173_74914_object95952:head# 2024-08-30T23:36:49.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f51b68ea:::benchmark_data_smithi173_74914_object65779:head# 2024-08-30T23:36:49.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f51d9fee:::benchmark_data_smithi173_74914_object72807:head# 2024-08-30T23:36:49.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f51e88ee:::benchmark_data_smithi173_74914_object93351:head# 2024-08-30T23:36:49.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f521fcea:::benchmark_data_smithi173_74914_object65240:head# 2024-08-30T23:36:49.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52271a1:::benchmark_data_smithi173_74914_object87825:head# 2024-08-30T23:36:49.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52545a7:::benchmark_data_smithi173_74914_object91258:head# 2024-08-30T23:36:49.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52bd0cb:::benchmark_data_smithi173_74914_object96614:head# 2024-08-30T23:36:49.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52bd200:::benchmark_data_smithi173_74914_object77672:head# 2024-08-30T23:36:49.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52d033c:::benchmark_data_smithi173_74914_object80188:head# 2024-08-30T23:36:49.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52f5de4:::benchmark_data_smithi173_74914_object62886:head# 2024-08-30T23:36:49.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52fbf9d:::benchmark_data_smithi173_74914_object69965:head# 2024-08-30T23:36:49.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f52fc42f:::benchmark_data_smithi173_74914_object60074:head# 2024-08-30T23:36:49.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f530567a:::benchmark_data_smithi173_74914_object86669:head# 2024-08-30T23:36:49.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f531d4ad:::benchmark_data_smithi173_74914_object80732:head# 2024-08-30T23:36:49.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f53a0f01:::benchmark_data_smithi173_74914_object82353:head# 2024-08-30T23:36:49.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f53d2643:::benchmark_data_smithi173_74914_object67476:head# 2024-08-30T23:36:49.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f54324a4:::benchmark_data_smithi173_74914_object93214:head# 2024-08-30T23:36:49.854 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f544d340:::benchmark_data_smithi173_74914_object57555:head# 2024-08-30T23:36:49.855 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:49.855 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f546fe3e:::benchmark_data_smithi173_74914_object54850:head# 2024-08-30T23:36:49.855 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:49.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5476c58:::benchmark_data_smithi173_74914_object72330:head# 2024-08-30T23:36:49.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f547a1a2:::benchmark_data_smithi173_74914_object57409:head# 2024-08-30T23:36:49.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5487680:::benchmark_data_smithi173_74914_object95876:head# 2024-08-30T23:36:49.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f54bb6fd:::benchmark_data_smithi173_74914_object76711:head# 2024-08-30T23:36:49.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f54cd940:::benchmark_data_smithi173_74914_object90393:head# 2024-08-30T23:36:49.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f54f3c69:::benchmark_data_smithi173_74914_object90855:head# 2024-08-30T23:36:49.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f54fa80d:::benchmark_data_smithi173_74914_object95723:head# 2024-08-30T23:36:49.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5509fd8:::benchmark_data_smithi173_74914_object86890:head# 2024-08-30T23:36:49.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f550a45a:::benchmark_data_smithi173_74914_object75223:head# 2024-08-30T23:36:49.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f552861d:::benchmark_data_smithi173_74914_object83050:head# 2024-08-30T23:36:49.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5551f2c:::benchmark_data_smithi173_74914_object56725:head# 2024-08-30T23:36:49.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f555d438:::benchmark_data_smithi173_74914_object85419:head# 2024-08-30T23:36:49.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5579171:::benchmark_data_smithi173_74914_object58884:head# 2024-08-30T23:36:49.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f557aa12:::benchmark_data_smithi173_74914_object55845:head# 2024-08-30T23:36:49.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f558a3f6:::benchmark_data_smithi173_74914_object93310:head# 2024-08-30T23:36:49.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f55ae382:::benchmark_data_smithi173_74914_object89174:head# 2024-08-30T23:36:49.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f55d7048:::benchmark_data_smithi173_74914_object83628:head# 2024-08-30T23:36:49.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f55f9bdd:::benchmark_data_smithi173_74914_object90112:head# 2024-08-30T23:36:49.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56141e8:::benchmark_data_smithi173_74914_object84124:head# 2024-08-30T23:36:49.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5642bfe:::benchmark_data_smithi173_74914_object90737:head# 2024-08-30T23:36:49.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5648110:::benchmark_data_smithi173_74914_object94289:head# 2024-08-30T23:36:49.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56615ca:::benchmark_data_smithi173_74914_object59393:head# 2024-08-30T23:36:49.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5675ae8:::benchmark_data_smithi173_74914_object75630:head# 2024-08-30T23:36:49.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5688000:::benchmark_data_smithi173_74914_object96290:head# 2024-08-30T23:36:49.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56aed79:::benchmark_data_smithi173_74914_object70301:head# 2024-08-30T23:36:49.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56c15bc:::benchmark_data_smithi173_74914_object70168:head# 2024-08-30T23:36:49.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56d6456:::benchmark_data_smithi173_74914_object71196:head# 2024-08-30T23:36:49.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56daebd:::benchmark_data_smithi173_74914_object56774:head# 2024-08-30T23:36:49.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56f3c9d:::benchmark_data_smithi173_74914_object54735:head# 2024-08-30T23:36:49.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f56fa1bb:::benchmark_data_smithi173_74914_object77844:head# 2024-08-30T23:36:49.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f570c6d1:::benchmark_data_smithi173_74914_object53197:head# 2024-08-30T23:36:49.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5719bc5:::benchmark_data_smithi173_74914_object91720:head# 2024-08-30T23:36:49.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f572e38b:::benchmark_data_smithi173_74914_object82143:head# 2024-08-30T23:36:49.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57382b8:::benchmark_data_smithi173_74914_object90285:head# 2024-08-30T23:36:49.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57477de:::benchmark_data_smithi173_74914_object59163:head# 2024-08-30T23:36:49.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5750064:::benchmark_data_smithi173_74914_object77834:head# 2024-08-30T23:36:49.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5760b60:::benchmark_data_smithi173_74914_object73763:head# 2024-08-30T23:36:49.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f579078a:::benchmark_data_smithi173_74914_object61555:head# 2024-08-30T23:36:49.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57932c2:::benchmark_data_smithi173_74914_object58977:head# 2024-08-30T23:36:49.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57a66e5:::benchmark_data_smithi173_74914_object72206:head# 2024-08-30T23:36:49.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57bbdeb:::benchmark_data_smithi173_74914_object75217:head# 2024-08-30T23:36:49.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57cad02:::benchmark_data_smithi173_74914_object71803:head# 2024-08-30T23:36:49.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57dde52:::benchmark_data_smithi173_74914_object74508:head# 2024-08-30T23:36:49.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f57ee7c1:::benchmark_data_smithi173_74914_object92081:head# 2024-08-30T23:36:49.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5803339:::benchmark_data_smithi173_74914_object56080:head# 2024-08-30T23:36:49.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5842fd1:::benchmark_data_smithi173_74914_object71712:head# 2024-08-30T23:36:49.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f584e33c:::benchmark_data_smithi173_74914_object54318:head# 2024-08-30T23:36:49.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5878bf1:::benchmark_data_smithi173_74914_object77198:head# 2024-08-30T23:36:49.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f58a9e19:::benchmark_data_smithi173_74914_object75470:head# 2024-08-30T23:36:49.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f58af315:::benchmark_data_smithi173_74914_object79048:head# 2024-08-30T23:36:49.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f58bcc09:::benchmark_data_smithi173_74914_object71776:head# 2024-08-30T23:36:49.881 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f58dd3d0:::benchmark_data_smithi173_74914_object85840:head# 2024-08-30T23:36:49.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f59201b4:::benchmark_data_smithi173_74914_object89058:head# 2024-08-30T23:36:49.882 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f592989c:::benchmark_data_smithi173_74914_object53694:head# 2024-08-30T23:36:49.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5943d9d:::benchmark_data_smithi173_74914_object86061:head# 2024-08-30T23:36:49.883 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5949b23:::benchmark_data_smithi173_74914_object71346:head# 2024-08-30T23:36:49.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f595f024:::benchmark_data_smithi173_74914_object83226:head# 2024-08-30T23:36:49.884 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f59611fa:::benchmark_data_smithi173_74914_object89638:head# 2024-08-30T23:36:49.885 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5963730:::benchmark_data_smithi173_74914_object76464:head# 2024-08-30T23:36:49.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5978c91:::benchmark_data_smithi173_74914_object95286:head# 2024-08-30T23:36:49.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5988cd5:::benchmark_data_smithi173_74914_object95225:head# 2024-08-30T23:36:49.886 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f59cd8bc:::benchmark_data_smithi173_74914_object85533:head# 2024-08-30T23:36:49.887 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f59d21a3:::benchmark_data_smithi173_74914_object70840:head# 2024-08-30T23:36:49.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f59f2c42:::benchmark_data_smithi173_74914_object69658:head# 2024-08-30T23:36:49.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a08c82:::benchmark_data_smithi173_74914_object61029:head# 2024-08-30T23:36:49.888 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a0a926:::benchmark_data_smithi173_74914_object76133:head# 2024-08-30T23:36:49.891 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a17508:::benchmark_data_smithi173_74914_object71126:head# 2024-08-30T23:36:49.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a1acaf:::benchmark_data_smithi173_74914_object88375:head# 2024-08-30T23:36:49.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a50f23:::benchmark_data_smithi173_74914_object67782:head# 2024-08-30T23:36:49.892 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a52b54:::benchmark_data_smithi173_74914_object97812:head# 2024-08-30T23:36:49.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a5e962:::benchmark_data_smithi173_74914_object72349:head# 2024-08-30T23:36:49.893 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a615bb:::benchmark_data_smithi173_74914_object76774:head# 2024-08-30T23:36:49.894 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a6848a:::benchmark_data_smithi173_74914_object77106:head# 2024-08-30T23:36:49.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a6ae43:::benchmark_data_smithi173_74914_object82852:head# 2024-08-30T23:36:49.895 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5a7b671:::benchmark_data_smithi173_74914_object82685:head# 2024-08-30T23:36:49.896 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5abd1f7:::benchmark_data_smithi173_74914_object71510:head# 2024-08-30T23:36:49.896 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5abd71e:::benchmark_data_smithi173_74914_object55132:head# 2024-08-30T23:36:49.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5adc75b:::benchmark_data_smithi173_74914_object58170:head# 2024-08-30T23:36:49.897 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b11822:::benchmark_data_smithi173_74914_object72573:head# 2024-08-30T23:36:49.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b3c702:::benchmark_data_smithi173_74914_object80589:head# 2024-08-30T23:36:49.898 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b48251:::benchmark_data_smithi173_74914_object86521:head# 2024-08-30T23:36:49.899 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b5ebd0:::benchmark_data_smithi173_74914_object78293:head# 2024-08-30T23:36:49.899 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b683ea:::benchmark_data_smithi173_74914_object93344:head# 2024-08-30T23:36:49.900 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b68a6d:::benchmark_data_smithi173_74914_object83728:head# 2024-08-30T23:36:49.900 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5b68e62:::benchmark_data_smithi173_74914_object93931:head# 2024-08-30T23:36:49.901 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5bc32a3:::benchmark_data_smithi173_74914_object66197:head# 2024-08-30T23:36:49.902 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5bcaec0:::benchmark_data_smithi173_74914_object92820:head# 2024-08-30T23:36:49.902 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5bdf5cc:::benchmark_data_smithi173_74914_object52954:head# 2024-08-30T23:36:49.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5beef25:::benchmark_data_smithi173_74914_object71448:head# 2024-08-30T23:36:49.905 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c0a035:::benchmark_data_smithi173_74914_object96297:head# 2024-08-30T23:36:49.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c356bc:::benchmark_data_smithi173_74914_object81805:head# 2024-08-30T23:36:49.906 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c383a1:::benchmark_data_smithi173_74914_object96832:head# 2024-08-30T23:36:49.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c45227:::benchmark_data_smithi173_74914_object81486:head# 2024-08-30T23:36:49.907 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c51cd2:::benchmark_data_smithi173_74914_object84354:head# 2024-08-30T23:36:49.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c5b8b3:::benchmark_data_smithi173_74914_object73183:head# 2024-08-30T23:36:49.908 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c64743:::benchmark_data_smithi173_74914_object84904:head# 2024-08-30T23:36:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c75914:::benchmark_data_smithi173_74914_object63134:head# 2024-08-30T23:36:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c88ac3:::benchmark_data_smithi173_74914_object84475:head# 2024-08-30T23:36:49.910 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c8fd46:::benchmark_data_smithi173_74914_object85864:head# 2024-08-30T23:36:49.911 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5c9bb22:::benchmark_data_smithi173_74914_object68282:head# 2024-08-30T23:36:49.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5ca0651:::benchmark_data_smithi173_74914_object70855:head# 2024-08-30T23:36:49.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5cb8351:::benchmark_data_smithi173_74914_object96421:head# 2024-08-30T23:36:49.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5cf41c3:::benchmark_data_smithi173_74914_object60716:head# 2024-08-30T23:36:49.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5d101ec:::benchmark_data_smithi173_74914_object90302:head# 2024-08-30T23:36:49.913 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5d31255:::benchmark_data_smithi173_74914_object64761:head# 2024-08-30T23:36:49.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5d370b4:::benchmark_data_smithi173_74914_object55280:head# 2024-08-30T23:36:49.914 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5d39b44:::benchmark_data_smithi173_74914_object74318:head# 2024-08-30T23:36:49.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5d488e0:::benchmark_data_smithi173_74914_object92440:head# 2024-08-30T23:36:49.915 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5d643f2:::benchmark_data_smithi173_74914_object56300:head# 2024-08-30T23:36:49.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5ddc368:::benchmark_data_smithi173_74914_object68174:head# 2024-08-30T23:36:49.916 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5df7ed6:::benchmark_data_smithi173_74914_object84614:head# 2024-08-30T23:36:49.918 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5dfdbc1:::benchmark_data_smithi173_74914_object96925:head# 2024-08-30T23:36:49.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e1c055:::benchmark_data_smithi173_74914_object88319:head# 2024-08-30T23:36:49.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e23bc3:::benchmark_data_smithi173_74914_object62614:head# 2024-08-30T23:36:49.919 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e2e32a:::benchmark_data_smithi173_74914_object58118:head# 2024-08-30T23:36:49.920 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e3e510:::benchmark_data_smithi173_74914_object97402:head# 2024-08-30T23:36:49.921 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e67ca9:::benchmark_data_smithi173_74914_object79674:head# 2024-08-30T23:36:49.922 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e6a39d:::benchmark_data_smithi173_74914_object66941:head# 2024-08-30T23:36:49.923 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e78125:::benchmark_data_smithi173_74914_object57446:head# 2024-08-30T23:36:49.923 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e86ad0:::benchmark_data_smithi173_74914_object89975:head# 2024-08-30T23:36:49.924 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5e8bc19:::benchmark_data_smithi173_74914_object76320:head# 2024-08-30T23:36:49.924 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5ec126a:::benchmark_data_smithi173_74914_object62176:head# 2024-08-30T23:36:49.925 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5ed34d4:::benchmark_data_smithi173_74914_object83033:head# 2024-08-30T23:36:49.925 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5edab1f:::benchmark_data_smithi173_74914_object53309:head# 2024-08-30T23:36:49.926 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5f4a7ac:::benchmark_data_smithi173_74914_object58630:head# 2024-08-30T23:36:49.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5f7a9cb:::benchmark_data_smithi173_74914_object86850:head# 2024-08-30T23:36:49.927 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5f81d6c:::benchmark_data_smithi173_74914_object53979:head# 2024-08-30T23:36:49.928 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5f89dc3:::benchmark_data_smithi173_74914_object86169:head# 2024-08-30T23:36:49.930 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5fa3fff:::benchmark_data_smithi173_74914_object88511:head# 2024-08-30T23:36:49.931 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5fb761a:::benchmark_data_smithi173_74914_object77152:head# 2024-08-30T23:36:49.931 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5fe9aab:::benchmark_data_smithi173_74914_object72185:head# 2024-08-30T23:36:49.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f5ff06e1:::benchmark_data_smithi173_74914_object96185:head# 2024-08-30T23:36:49.932 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f600dd41:::benchmark_data_smithi173_74914_object63717:head# 2024-08-30T23:36:49.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f602dbc3:::benchmark_data_smithi173_74914_object88028:head# 2024-08-30T23:36:49.933 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f605525b:::benchmark_data_smithi173_74914_object73225:head# 2024-08-30T23:36:49.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60594c5:::benchmark_data_smithi173_74914_object72936:head# 2024-08-30T23:36:49.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6087461:::benchmark_data_smithi173_74914_object92365:head# 2024-08-30T23:36:49.934 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f608c47c:::benchmark_data_smithi173_74914_object89803:head# 2024-08-30T23:36:49.935 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60956e4:::benchmark_data_smithi173_74914_object91367:head# 2024-08-30T23:36:49.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6096fcb:::benchmark_data_smithi173_74914_object77161:head# 2024-08-30T23:36:49.936 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60aa141:::benchmark_data_smithi173_74914_object86479:head# 2024-08-30T23:36:49.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60ae587:::benchmark_data_smithi173_74914_object69857:head# 2024-08-30T23:36:49.938 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60c5f2e:::benchmark_data_smithi173_74914_object91879:head# 2024-08-30T23:36:49.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60ccede:::benchmark_data_smithi173_74914_object89026:head# 2024-08-30T23:36:49.939 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f60d700f:::benchmark_data_smithi173_74914_object71753:head# 2024-08-30T23:36:49.940 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6107ed0:::benchmark_data_smithi173_74914_object81783:head# 2024-08-30T23:36:49.942 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f612aa88:::benchmark_data_smithi173_74914_object94793:head# 2024-08-30T23:36:49.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6135c46:::benchmark_data_smithi173_74914_object65317:head# 2024-08-30T23:36:49.943 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f613699b:::benchmark_data_smithi173_74914_object94321:head# 2024-08-30T23:36:49.946 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f613eac8:::benchmark_data_smithi173_74914_object68106:head# 2024-08-30T23:36:49.947 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f613fdea:::benchmark_data_smithi173_74914_object83978:head# 2024-08-30T23:36:49.948 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f614a1fe:::benchmark_data_smithi173_74914_object76858:head# 2024-08-30T23:36:49.948 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6176977:::benchmark_data_smithi173_74914_object82721:head# 2024-08-30T23:36:49.949 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6194fa8:::benchmark_data_smithi173_74914_object85300:head# 2024-08-30T23:36:49.949 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f61b4932:::benchmark_data_smithi173_74914_object73430:head# 2024-08-30T23:36:49.952 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f61cdab3:::benchmark_data_smithi173_74914_object79298:head# 2024-08-30T23:36:49.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f61dcd5f:::benchmark_data_smithi173_74914_object70680:head# 2024-08-30T23:36:49.953 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6214294:::benchmark_data_smithi173_74914_object53926:head# 2024-08-30T23:36:49.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f623b9b7:::benchmark_data_smithi173_74914_object86361:head# 2024-08-30T23:36:49.954 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6254539:::benchmark_data_smithi173_74914_object93645:head# 2024-08-30T23:36:49.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f62d1b6a:::benchmark_data_smithi173_74914_object79184:head# 2024-08-30T23:36:49.955 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f62f9713:::benchmark_data_smithi173_74914_object71598:head# 2024-08-30T23:36:49.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f62fe668:::benchmark_data_smithi173_74914_object69410:head# 2024-08-30T23:36:49.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f630a0d5:::benchmark_data_smithi173_74914_object66488:head# 2024-08-30T23:36:49.956 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f630d805:::benchmark_data_smithi173_74914_object83964:head# 2024-08-30T23:36:49.957 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f630e73f:::benchmark_data_smithi173_74914_object75579:head# 2024-08-30T23:36:49.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:49.958 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6315780:::benchmark_data_smithi173_74914_object58957:head# 2024-08-30T23:36:49.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f632c85c:::benchmark_data_smithi173_74914_object81457:head# 2024-08-30T23:36:49.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f633dd51:::benchmark_data_smithi173_74914_object94031:head# 2024-08-30T23:36:49.959 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6343d6f:::benchmark_data_smithi173_74914_object54173:head# 2024-08-30T23:36:49.960 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6347b4d:::benchmark_data_smithi173_74914_object74410:head# 2024-08-30T23:36:49.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6367068:::benchmark_data_smithi173_74914_object89720:head# 2024-08-30T23:36:49.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f63753de:::benchmark_data_smithi173_74914_object83898:head# 2024-08-30T23:36:49.961 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6384c24:::benchmark_data_smithi173_74914_object64048:head# 2024-08-30T23:36:49.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f63ced43:::benchmark_data_smithi173_74914_object61027:head# 2024-08-30T23:36:49.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f63eb2bf:::benchmark_data_smithi173_74914_object79594:head# 2024-08-30T23:36:49.962 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6413a56:::benchmark_data_smithi173_74914_object56116:head# 2024-08-30T23:36:49.965 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64259ba:::benchmark_data_smithi173_74914_object94155:head# 2024-08-30T23:36:49.968 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6434b6c:::benchmark_data_smithi173_74914_object93121:head# 2024-08-30T23:36:49.969 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6441b3b:::benchmark_data_smithi173_74914_object73647:head# 2024-08-30T23:36:49.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6447e5b:::benchmark_data_smithi173_74914_object55622:head# 2024-08-30T23:36:49.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f644c5a4:::benchmark_data_smithi173_74914_object58657:head# 2024-08-30T23:36:49.970 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f645d26c:::benchmark_data_smithi173_74914_object54079:head# 2024-08-30T23:36:49.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6496156:::benchmark_data_smithi173_74914_object78870:head# 2024-08-30T23:36:49.971 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64a07a2:::benchmark_data_smithi173_74914_object80521:head# 2024-08-30T23:36:49.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64a1ae9:::benchmark_data_smithi173_74914_object70761:head# 2024-08-30T23:36:49.972 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64b82e8:::benchmark_data_smithi173_74914_object61314:head# 2024-08-30T23:36:49.974 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64befa7:::benchmark_data_smithi173_74914_object96221:head# 2024-08-30T23:36:49.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64d8828:::benchmark_data_smithi173_74914_object70616:head# 2024-08-30T23:36:49.977 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64eef20:::benchmark_data_smithi173_74914_object89114:head# 2024-08-30T23:36:49.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64efce6:::benchmark_data_smithi173_74914_object71198:head# 2024-08-30T23:36:49.978 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f64fef29:::benchmark_data_smithi173_74914_object55475:head# 2024-08-30T23:36:49.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f650e7e4:::benchmark_data_smithi173_74914_object70038:head# 2024-08-30T23:36:49.979 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f652cf69:::benchmark_data_smithi173_74914_object69249:head# 2024-08-30T23:36:49.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6538dc9:::benchmark_data_smithi173_74914_object65041:head# 2024-08-30T23:36:49.980 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f653b801:::benchmark_data_smithi173_74914_object58691:head# 2024-08-30T23:36:49.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f654ef2d:::benchmark_data_smithi173_74914_object78740:head# 2024-08-30T23:36:49.981 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6591611:::benchmark_data_smithi173_74914_object67379:head# 2024-08-30T23:36:49.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f65a836e:::benchmark_data_smithi173_74914_object69064:head# 2024-08-30T23:36:49.982 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f65c35aa:::benchmark_data_smithi173_74914_object56230:head# 2024-08-30T23:36:49.983 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f65dc6c5:::benchmark_data_smithi173_74914_object70374:head# 2024-08-30T23:36:49.983 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f65e4e59:::benchmark_data_smithi173_74914_object59583:head# 2024-08-30T23:36:49.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f65e8ace:::benchmark_data_smithi173_74914_object79111:head# 2024-08-30T23:36:49.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f65e9ec9:::benchmark_data_smithi173_74914_object73903:head# 2024-08-30T23:36:49.984 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6629ad5:::benchmark_data_smithi173_74914_object82075:head# 2024-08-30T23:36:49.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6649d2b:::benchmark_data_smithi173_74914_object62219:head# 2024-08-30T23:36:49.985 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6689c63:::benchmark_data_smithi173_74914_object90537:head# 2024-08-30T23:36:49.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f668bc58:::benchmark_data_smithi173_74914_object90688:head# 2024-08-30T23:36:49.986 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6691e78:::benchmark_data_smithi173_74914_object82523:head# 2024-08-30T23:36:49.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66a6f3a:::benchmark_data_smithi173_74914_object62207:head# 2024-08-30T23:36:49.987 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66a8ba1:::benchmark_data_smithi173_74914_object71708:head# 2024-08-30T23:36:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:49.988 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:49.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:49.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66ab2b5:::benchmark_data_smithi173_74914_object79512:head# 2024-08-30T23:36:49.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66adb29:::benchmark_data_smithi173_74914_object82683:head# 2024-08-30T23:36:49.989 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66af07c:::benchmark_data_smithi173_74914_object57391:head# 2024-08-30T23:36:49.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66ba803:::benchmark_data_smithi173_74914_object98222:head# 2024-08-30T23:36:49.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66d83eb:::benchmark_data_smithi173_74914_object91977:head# 2024-08-30T23:36:49.990 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f66fd410:::benchmark_data_smithi173_74914_object80546:head# 2024-08-30T23:36:49.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6715ae9:::benchmark_data_smithi173_74914_object84085:head# 2024-08-30T23:36:49.991 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67173b2:::benchmark_data_smithi173_74914_object61225:head# 2024-08-30T23:36:49.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f674ecb6:::benchmark_data_smithi173_74914_object81057:head# 2024-08-30T23:36:49.992 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6753abc:::benchmark_data_smithi173_74914_object60138:head# 2024-08-30T23:36:49.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f677f2ab:::benchmark_data_smithi173_74914_object56689:head# 2024-08-30T23:36:49.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67814d2:::benchmark_data_smithi173_74914_object82064:head# 2024-08-30T23:36:49.993 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f678ce3a:::benchmark_data_smithi173_74914_object61230:head# 2024-08-30T23:36:49.994 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6798f42:::benchmark_data_smithi173_74914_object82120:head# 2024-08-30T23:36:49.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67be3e4:::benchmark_data_smithi173_74914_object90832:head# 2024-08-30T23:36:49.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67c0aec:::benchmark_data_smithi173_74914_object69746:head# 2024-08-30T23:36:49.995 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67e4ed9:::benchmark_data_smithi173_74914_object56626:head# 2024-08-30T23:36:49.996 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67e71a5:::benchmark_data_smithi173_74914_object60865:head# 2024-08-30T23:36:49.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67ebb8e:::benchmark_data_smithi173_74914_object58347:head# 2024-08-30T23:36:49.997 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f67ff6a1:::benchmark_data_smithi173_74914_object79878:head# 2024-08-30T23:36:49.998 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6810e5f:::benchmark_data_smithi173_74914_object84014:head# 2024-08-30T23:36:49.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f682b69f:::benchmark_data_smithi173_74914_object94564:head# 2024-08-30T23:36:49.999 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f682d940:::benchmark_data_smithi173_74914_object75732:head# 2024-08-30T23:36:50.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f683b5d3:::benchmark_data_smithi173_74914_object73359:head# 2024-08-30T23:36:50.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6841f18:::benchmark_data_smithi173_74914_object54780:head# 2024-08-30T23:36:50.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68529d6:::benchmark_data_smithi173_74914_object96081:head# 2024-08-30T23:36:50.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6866f83:::benchmark_data_smithi173_74914_object89871:head# 2024-08-30T23:36:50.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f689dc15:::benchmark_data_smithi173_74914_object72578:head# 2024-08-30T23:36:50.004 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68a6b7a:::benchmark_data_smithi173_74914_object73702:head# 2024-08-30T23:36:50.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68aedf2:::benchmark_data_smithi173_74914_object59375:head# 2024-08-30T23:36:50.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68c1aa2:::benchmark_data_smithi173_74914_object83031:head# 2024-08-30T23:36:50.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68d74ea:::benchmark_data_smithi173_74914_object61621:head# 2024-08-30T23:36:50.005 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68f3659:::benchmark_data_smithi173_74914_object57168:head# 2024-08-30T23:36:50.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f68f88b6:::benchmark_data_smithi173_74914_object65286:head# 2024-08-30T23:36:50.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6904b0e:::benchmark_data_smithi173_74914_object93873:head# 2024-08-30T23:36:50.006 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f69144ae:::benchmark_data_smithi173_74914_object57167:head# 2024-08-30T23:36:50.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f691caf9:::benchmark_data_smithi173_74914_object92692:head# 2024-08-30T23:36:50.007 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f693e62f:::benchmark_data_smithi173_74914_object60930:head# 2024-08-30T23:36:50.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f696835d:::benchmark_data_smithi173_74914_object83490:head# 2024-08-30T23:36:50.008 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f697ca0c:::benchmark_data_smithi173_74914_object75158:head# 2024-08-30T23:36:50.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f69845c4:::benchmark_data_smithi173_74914_object54625:head# 2024-08-30T23:36:50.009 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f69adecd:::benchmark_data_smithi173_74914_object60349:head# 2024-08-30T23:36:50.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f69af484:::benchmark_data_smithi173_74914_object88830:head# 2024-08-30T23:36:50.010 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f69e5e7c:::benchmark_data_smithi173_74914_object85473:head# 2024-08-30T23:36:50.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a25a30:::benchmark_data_smithi173_74914_object74945:head# 2024-08-30T23:36:50.011 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a2a0d2:::benchmark_data_smithi173_74914_object58968:head# 2024-08-30T23:36:50.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a39360:::benchmark_data_smithi173_74914_object69217:head# 2024-08-30T23:36:50.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a39b74:::benchmark_data_smithi173_74914_object54704:head# 2024-08-30T23:36:50.012 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a3efa2:::benchmark_data_smithi173_74914_object97947:head# 2024-08-30T23:36:50.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a552f9:::benchmark_data_smithi173_74914_object74205:head# 2024-08-30T23:36:50.013 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6a5ac7e:::benchmark_data_smithi173_74914_object79054:head# 2024-08-30T23:36:50.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6aaf8c7:::benchmark_data_smithi173_74914_object95560:head# 2024-08-30T23:36:50.014 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6aba557:::benchmark_data_smithi173_74914_object80630:head# 2024-08-30T23:36:50.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ae1f90:::benchmark_data_smithi173_74914_object86706:head# 2024-08-30T23:36:50.015 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ae664b:::benchmark_data_smithi173_74914_object62444:head# 2024-08-30T23:36:50.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ae9e3f:::benchmark_data_smithi173_74914_object97504:head# 2024-08-30T23:36:50.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6afb626:::benchmark_data_smithi173_74914_object94970:head# 2024-08-30T23:36:50.016 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6b09027:::benchmark_data_smithi173_74914_object62450:head# 2024-08-30T23:36:50.017 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6b462cd:::benchmark_data_smithi173_74914_object54784:head# 2024-08-30T23:36:50.018 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6b4c155:::benchmark_data_smithi173_74914_object91139:head# 2024-08-30T23:36:50.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6b5174e:::benchmark_data_smithi173_74914_object77935:head# 2024-08-30T23:36:50.019 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6b62ac1:::benchmark_data_smithi173_74914_object76454:head# 2024-08-30T23:36:50.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bb9cd4:::benchmark_data_smithi173_74914_object55584:head# 2024-08-30T23:36:50.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bbb323:::benchmark_data_smithi173_74914_object76146:head# 2024-08-30T23:36:50.020 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bbe29b:::benchmark_data_smithi173_74914_object78448:head# 2024-08-30T23:36:50.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bc5ebb:::benchmark_data_smithi173_74914_object96546:head# 2024-08-30T23:36:50.021 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bce0fa:::benchmark_data_smithi173_74914_object73108:head# 2024-08-30T23:36:50.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bdbcce:::benchmark_data_smithi173_74914_object78700:head# 2024-08-30T23:36:50.022 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6be8fa1:::benchmark_data_smithi173_74914_object63253:head# 2024-08-30T23:36:50.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6bf28f2:::benchmark_data_smithi173_74914_object61711:head# 2024-08-30T23:36:50.023 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c0aad1:::benchmark_data_smithi173_74914_object54999:head# 2024-08-30T23:36:50.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c0c2a6:::benchmark_data_smithi173_74914_object83415:head# 2024-08-30T23:36:50.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c137b7:::benchmark_data_smithi173_74914_object61241:head# 2024-08-30T23:36:50.024 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c169d4:::benchmark_data_smithi173_74914_object84080:head# 2024-08-30T23:36:50.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c76c4b:::benchmark_data_smithi173_74914_object85796:head# 2024-08-30T23:36:50.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c87375:::benchmark_data_smithi173_74914_object84036:head# 2024-08-30T23:36:50.026 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c8846e:::benchmark_data_smithi173_74914_object92603:head# 2024-08-30T23:36:50.027 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c8aa0b:::benchmark_data_smithi173_74914_object83386:head# 2024-08-30T23:36:50.028 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c8d096:::benchmark_data_smithi173_74914_object66824:head# 2024-08-30T23:36:50.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6c93979:::benchmark_data_smithi173_74914_object55794:head# 2024-08-30T23:36:50.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ca0659:::benchmark_data_smithi173_74914_object58831:head# 2024-08-30T23:36:50.029 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ce0636:::benchmark_data_smithi173_74914_object61328:head# 2024-08-30T23:36:50.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ce7574:::benchmark_data_smithi173_74914_object78453:head# 2024-08-30T23:36:50.030 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ce8075:::benchmark_data_smithi173_74914_object67733:head# 2024-08-30T23:36:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ce9854:::benchmark_data_smithi173_74914_object65520:head# 2024-08-30T23:36:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6d005c8:::benchmark_data_smithi173_74914_object53490:head# 2024-08-30T23:36:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6d03c8c:::benchmark_data_smithi173_74914_object88087:head# 2024-08-30T23:36:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6d13e1e:::benchmark_data_smithi173_74914_object77973:head# 2024-08-30T23:36:50.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6d31835:::benchmark_data_smithi173_74914_object57220:head# 2024-08-30T23:36:50.033 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6d4f187:::benchmark_data_smithi173_74914_object98183:head# 2024-08-30T23:36:50.034 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6d88d4b:::benchmark_data_smithi173_74914_object82996:head# 2024-08-30T23:36:50.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6da2b10:::benchmark_data_smithi173_74914_object74981:head# 2024-08-30T23:36:50.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6da5bf5:::benchmark_data_smithi173_74914_object58155:head# 2024-08-30T23:36:50.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6dcb916:::benchmark_data_smithi173_74914_object88112:head# 2024-08-30T23:36:50.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6dddb6e:::benchmark_data_smithi173_74914_object63656:head# 2024-08-30T23:36:50.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e0c62e:::benchmark_data_smithi173_74914_object58495:head# 2024-08-30T23:36:50.037 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e1f408:::benchmark_data_smithi173_74914_object85735:head# 2024-08-30T23:36:50.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e29915:::benchmark_data_smithi173_74914_object89048:head# 2024-08-30T23:36:50.038 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e3791e:::benchmark_data_smithi173_74914_object54932:head# 2024-08-30T23:36:50.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e4a6bb:::benchmark_data_smithi173_74914_object90923:head# 2024-08-30T23:36:50.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e4f964:::benchmark_data_smithi173_74914_object83555:head# 2024-08-30T23:36:50.039 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e6f15d:::benchmark_data_smithi173_74914_object97691:head# 2024-08-30T23:36:50.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e75521:::benchmark_data_smithi173_74914_object91387:head# 2024-08-30T23:36:50.040 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e896a8:::benchmark_data_smithi173_74914_object66043:head# 2024-08-30T23:36:50.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6e98bf9:::benchmark_data_smithi173_74914_object66115:head# 2024-08-30T23:36:50.041 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6ea1f9d:::benchmark_data_smithi173_74914_object82624:head# 2024-08-30T23:36:50.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6eec02c:::benchmark_data_smithi173_74914_object56737:head# 2024-08-30T23:36:50.042 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f0fe0e:::benchmark_data_smithi173_74914_object91812:head# 2024-08-30T23:36:50.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f187ae:::benchmark_data_smithi173_74914_object75949:head# 2024-08-30T23:36:50.043 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f20a7c:::benchmark_data_smithi173_74914_object70575:head# 2024-08-30T23:36:50.044 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f21f33:::benchmark_data_smithi173_74914_object67631:head# 2024-08-30T23:36:50.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f29fcc:::benchmark_data_smithi173_74914_object97574:head# 2024-08-30T23:36:50.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f3164c:::benchmark_data_smithi173_74914_object88119:head# 2024-08-30T23:36:50.045 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f746b5:::benchmark_data_smithi173_74914_object57837:head# 2024-08-30T23:36:50.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f757c3:::benchmark_data_smithi173_74914_object87543:head# 2024-08-30T23:36:50.046 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f7f103:::benchmark_data_smithi173_74914_object72389:head# 2024-08-30T23:36:50.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6f945db:::benchmark_data_smithi173_74914_object81372:head# 2024-08-30T23:36:50.047 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f6fd9c36:::benchmark_data_smithi173_74914_object53370:head# 2024-08-30T23:36:50.048 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f700175b:::benchmark_data_smithi173_74914_object54793:head# 2024-08-30T23:36:50.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7022f2d:::benchmark_data_smithi173_74914_object85368:head# 2024-08-30T23:36:50.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f702bbee:::benchmark_data_smithi173_74914_object60396:head# 2024-08-30T23:36:50.049 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70377fc:::benchmark_data_smithi173_74914_object58932:head# 2024-08-30T23:36:50.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7056527:::benchmark_data_smithi173_74914_object58696:head# 2024-08-30T23:36:50.050 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f705b1ce:::benchmark_data_smithi173_74914_object59702:head# 2024-08-30T23:36:50.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7069c46:::benchmark_data_smithi173_74914_object72616:head# 2024-08-30T23:36:50.051 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70a71e4:::benchmark_data_smithi173_74914_object83568:head# 2024-08-30T23:36:50.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70c31fb:::benchmark_data_smithi173_74914_object75665:head# 2024-08-30T23:36:50.052 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70d102b:::benchmark_data_smithi173_74914_object53456:head# 2024-08-30T23:36:50.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70e4c19:::benchmark_data_smithi173_74914_object90720:head# 2024-08-30T23:36:50.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70fa5ae:::benchmark_data_smithi173_74914_object92218:head# 2024-08-30T23:36:50.053 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f70fb0d4:::benchmark_data_smithi173_74914_object70814:head# 2024-08-30T23:36:50.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f710ee11:::benchmark_data_smithi173_74914_object74378:head# 2024-08-30T23:36:50.054 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f712a60b:::benchmark_data_smithi173_74914_object61954:head# 2024-08-30T23:36:50.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7143f68:::benchmark_data_smithi173_74914_object53924:head# 2024-08-30T23:36:50.055 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f715d529:::benchmark_data_smithi173_74914_object57097:head# 2024-08-30T23:36:50.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7161514:::benchmark_data_smithi173_74914_object54851:head# 2024-08-30T23:36:50.056 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f71883bd:::benchmark_data_smithi173_74914_object54936:head# 2024-08-30T23:36:50.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f71e1c6d:::benchmark_data_smithi173_74914_object62063:head# 2024-08-30T23:36:50.057 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f721bc40:::benchmark_data_smithi173_74914_object62065:head# 2024-08-30T23:36:50.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7256dec:::benchmark_data_smithi173_74914_object64807:head# 2024-08-30T23:36:50.058 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7265ac3:::benchmark_data_smithi173_74914_object57448:head# 2024-08-30T23:36:50.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f726ba97:::benchmark_data_smithi173_74914_object65977:head# 2024-08-30T23:36:50.059 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f727b773:::benchmark_data_smithi173_74914_object58964:head# 2024-08-30T23:36:50.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f72c4be4:::benchmark_data_smithi173_74914_object90273:head# 2024-08-30T23:36:50.060 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f72c5917:::benchmark_data_smithi173_74914_object90903:head# 2024-08-30T23:36:50.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f72e54b2:::benchmark_data_smithi173_74914_object58359:head# 2024-08-30T23:36:50.061 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f72e5508:::benchmark_data_smithi173_74914_object81807:head# 2024-08-30T23:36:50.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f72f160b:::benchmark_data_smithi173_74914_object88297:head# 2024-08-30T23:36:50.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f72fc7af:::benchmark_data_smithi173_74914_object75957:head# 2024-08-30T23:36:50.062 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f730a2b9:::benchmark_data_smithi173_74914_object79945:head# 2024-08-30T23:36:50.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73404f0:::benchmark_data_smithi173_74914_object56057:head# 2024-08-30T23:36:50.063 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7353203:::benchmark_data_smithi173_74914_object78611:head# 2024-08-30T23:36:50.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73769e3:::benchmark_data_smithi173_74914_object64256:head# 2024-08-30T23:36:50.064 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f737af3b:::benchmark_data_smithi173_74914_object65228:head# 2024-08-30T23:36:50.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73b299f:::benchmark_data_smithi173_74914_object55380:head# 2024-08-30T23:36:50.065 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73b8f0b:::benchmark_data_smithi173_74914_object89988:head# 2024-08-30T23:36:50.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73b952b:::benchmark_data_smithi173_74914_object82333:head# 2024-08-30T23:36:50.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73d1d63:::benchmark_data_smithi173_74914_object77949:head# 2024-08-30T23:36:50.066 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73ea1ac:::benchmark_data_smithi173_74914_object92910:head# 2024-08-30T23:36:50.067 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73fb576:::benchmark_data_smithi173_74914_object88240:head# 2024-08-30T23:36:50.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73fb934:::benchmark_data_smithi173_74914_object64165:head# 2024-08-30T23:36:50.068 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f73fd03d:::benchmark_data_smithi173_74914_object72718:head# 2024-08-30T23:36:50.069 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f740944c:::benchmark_data_smithi173_74914_object63267:head# 2024-08-30T23:36:50.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7419444:::benchmark_data_smithi173_74914_object89194:head# 2024-08-30T23:36:50.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f744ddcf:::benchmark_data_smithi173_74914_object78525:head# 2024-08-30T23:36:50.070 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f744ecf4:::benchmark_data_smithi173_74914_object70959:head# 2024-08-30T23:36:50.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f745c42e:::benchmark_data_smithi173_74914_object55119:head# 2024-08-30T23:36:50.071 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f748e396:::benchmark_data_smithi173_74914_object59146:head# 2024-08-30T23:36:50.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f74925c1:::benchmark_data_smithi173_74914_object71278:head# 2024-08-30T23:36:50.073 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f74af8d4:::benchmark_data_smithi173_74914_object72539:head# 2024-08-30T23:36:50.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f74e9741:::benchmark_data_smithi173_74914_object93887:head# 2024-08-30T23:36:50.074 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f750aa76:::benchmark_data_smithi173_74914_object87366:head# 2024-08-30T23:36:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:49 smithi173 ceph-mon[20554]: 4.1 scrub starts 2024-08-30T23:36:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:49 smithi173 ceph-mon[20554]: 4.2 scrub starts 2024-08-30T23:36:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:49 smithi173 ceph-mon[20554]: pgmap v904: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.8 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 714 op/s; 10693/86757 objects degraded (12.325%) 2024-08-30T23:36:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:49 smithi173 ceph-mon[20554]: 4.2 scrub ok 2024-08-30T23:36:50.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f751eb3c:::benchmark_data_smithi173_74914_object55685:head# 2024-08-30T23:36:50.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7541e88:::benchmark_data_smithi173_74914_object87960:head# 2024-08-30T23:36:50.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7561d90:::benchmark_data_smithi173_74914_object80511:head# 2024-08-30T23:36:50.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f757eba2:::benchmark_data_smithi173_74914_object77096:head# 2024-08-30T23:36:50.076 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f75a9d89:::benchmark_data_smithi173_74914_object80598:head# 2024-08-30T23:36:50.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f75db99a:::benchmark_data_smithi173_74914_object66021:head# 2024-08-30T23:36:50.077 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f75de543:::benchmark_data_smithi173_74914_object88199:head# 2024-08-30T23:36:50.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f75eb9fc:::benchmark_data_smithi173_74914_object57374:head# 2024-08-30T23:36:50.078 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f75fcd63:::benchmark_data_smithi173_74914_object65402:head# 2024-08-30T23:36:50.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f76080e3:::benchmark_data_smithi173_74914_object82634:head# 2024-08-30T23:36:50.079 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7615a14:::benchmark_data_smithi173_74914_object76270:head# 2024-08-30T23:36:50.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7644104:::benchmark_data_smithi173_74914_object92727:head# 2024-08-30T23:36:50.080 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f766bd6e:::benchmark_data_smithi173_74914_object78069:head# 2024-08-30T23:36:50.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f76a2795:::benchmark_data_smithi173_74914_object77210:head# 2024-08-30T23:36:50.081 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f76b0fdf:::benchmark_data_smithi173_74914_object88503:head# 2024-08-30T23:36:50.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f76b308e:::benchmark_data_smithi173_74914_object91134:head# 2024-08-30T23:36:50.082 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f76df31a:::benchmark_data_smithi173_74914_object80252:head# 2024-08-30T23:36:50.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7700345:::benchmark_data_smithi173_74914_object94929:head# 2024-08-30T23:36:50.083 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f775dff6:::benchmark_data_smithi173_74914_object95152:head# 2024-08-30T23:36:50.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f776982b:::benchmark_data_smithi173_74914_object81895:head# 2024-08-30T23:36:50.090 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f777de71:::benchmark_data_smithi173_74914_object61084:head# 2024-08-30T23:36:50.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f779c00b:::benchmark_data_smithi173_74914_object80199:head# 2024-08-30T23:36:50.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f77c2d5f:::benchmark_data_smithi173_74914_object54903:head# 2024-08-30T23:36:50.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f77da231:::benchmark_data_smithi173_74914_object63537:head# 2024-08-30T23:36:50.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f77dfdfc:::benchmark_data_smithi173_74914_object73172:head# 2024-08-30T23:36:50.091 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f77f45a1:::benchmark_data_smithi173_74914_object87121:head# 2024-08-30T23:36:50.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:50.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f781f3e1:::benchmark_data_smithi173_74914_object90754:head# 2024-08-30T23:36:50.092 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7823dca:::benchmark_data_smithi173_74914_object77667:head# 2024-08-30T23:36:50.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7882cbe:::benchmark_data_smithi173_74914_object68261:head# 2024-08-30T23:36:50.093 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f78946d1:::benchmark_data_smithi173_74914_object88683:head# 2024-08-30T23:36:50.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f78a5a75:::benchmark_data_smithi173_74914_object73341:head# 2024-08-30T23:36:50.094 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f790c7b5:::benchmark_data_smithi173_74914_object85355:head# 2024-08-30T23:36:50.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7918335:::benchmark_data_smithi173_74914_object87011:head# 2024-08-30T23:36:50.095 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7923eae:::benchmark_data_smithi173_74914_object85139:head# 2024-08-30T23:36:50.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7928a87:::benchmark_data_smithi173_74914_object82632:head# 2024-08-30T23:36:50.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f792e9cf:::benchmark_data_smithi173_74914_object94562:head# 2024-08-30T23:36:50.096 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f795a7af:::benchmark_data_smithi173_74914_object72571:head# 2024-08-30T23:36:50.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f796d4e0:::benchmark_data_smithi173_74914_object54918:head# 2024-08-30T23:36:50.097 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f796eaa3:::benchmark_data_smithi173_74914_object69991:head# 2024-08-30T23:36:50.098 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f799722b:::benchmark_data_smithi173_74914_object74661:head# 2024-08-30T23:36:50.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f79b004c:::benchmark_data_smithi173_74914_object66539:head# 2024-08-30T23:36:50.099 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f79d521c:::benchmark_data_smithi173_74914_object53634:head# 2024-08-30T23:36:50.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f79da34c:::benchmark_data_smithi173_74914_object59342:head# 2024-08-30T23:36:50.100 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7a19ae3:::benchmark_data_smithi173_74914_object83369:head# 2024-08-30T23:36:50.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7a35b68:::benchmark_data_smithi173_74914_object81583:head# 2024-08-30T23:36:50.101 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7a4428a:::benchmark_data_smithi173_74914_object69500:head# 2024-08-30T23:36:50.102 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7a4b370:::benchmark_data_smithi173_74914_object74538:head# 2024-08-30T23:36:50.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7a96855:::benchmark_data_smithi173_74914_object72620:head# 2024-08-30T23:36:50.103 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ac67a2:::benchmark_data_smithi173_74914_object59219:head# 2024-08-30T23:36:50.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ad2409:::benchmark_data_smithi173_74914_object56466:head# 2024-08-30T23:36:50.104 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7aeaeac:::benchmark_data_smithi173_74914_object82194:head# 2024-08-30T23:36:50.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7af016f:::benchmark_data_smithi173_74914_object74049:head# 2024-08-30T23:36:50.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7b0188a:::benchmark_data_smithi173_74914_object91552:head# 2024-08-30T23:36:50.105 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7b53181:::benchmark_data_smithi173_74914_object82979:head# 2024-08-30T23:36:50.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7b5cfc8:::benchmark_data_smithi173_74914_object76448:head# 2024-08-30T23:36:50.108 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7b76403:::benchmark_data_smithi173_74914_object87826:head# 2024-08-30T23:36:50.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ba0e12:::benchmark_data_smithi173_74914_object74486:head# 2024-08-30T23:36:50.109 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7bcf7ec:::benchmark_data_smithi173_74914_object71568:head# 2024-08-30T23:36:50.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7bd2521:::benchmark_data_smithi173_74914_object97138:head# 2024-08-30T23:36:50.112 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7bddd93:::benchmark_data_smithi173_74914_object56855:head# 2024-08-30T23:36:50.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7bf2148:::benchmark_data_smithi173_74914_object58900:head# 2024-08-30T23:36:50.113 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c0ef76:::benchmark_data_smithi173_74914_object87445:head# 2024-08-30T23:36:50.114 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c1e85a:::benchmark_data_smithi173_74914_object54277:head# 2024-08-30T23:36:50.115 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c269dc:::benchmark_data_smithi173_74914_object71404:head# 2024-08-30T23:36:50.117 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c4000b:::benchmark_data_smithi173_74914_object71169:head# 2024-08-30T23:36:50.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:50.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c683f9:::benchmark_data_smithi173_74914_object55855:head# 2024-08-30T23:36:50.121 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c7e238:::benchmark_data_smithi173_74914_object87144:head# 2024-08-30T23:36:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:50.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c8f4ae:::benchmark_data_smithi173_74914_object67201:head# 2024-08-30T23:36:50.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7c96a67:::benchmark_data_smithi173_74914_object88838:head# 2024-08-30T23:36:50.123 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ca96ba:::benchmark_data_smithi173_74914_object74972:head# 2024-08-30T23:36:50.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d0bf14:::benchmark_data_smithi173_74914_object73032:head# 2024-08-30T23:36:50.124 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d0f3a6:::benchmark_data_smithi173_74914_object85360:head# 2024-08-30T23:36:50.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d22cd3:::benchmark_data_smithi173_74914_object66560:head# 2024-08-30T23:36:50.125 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d3af9c:::benchmark_data_smithi173_74914_object64346:head# 2024-08-30T23:36:50.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d3b351:::benchmark_data_smithi173_74914_object76691:head# 2024-08-30T23:36:50.126 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d44a6c:::benchmark_data_smithi173_74914_object63869:head# 2024-08-30T23:36:50.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d5e897:::benchmark_data_smithi173_74914_object67681:head# 2024-08-30T23:36:50.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d60467:::benchmark_data_smithi173_74914_object64566:head# 2024-08-30T23:36:50.127 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7d83f66:::benchmark_data_smithi173_74914_object89068:head# 2024-08-30T23:36:50.128 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7daccd0:::benchmark_data_smithi173_74914_object93509:head# 2024-08-30T23:36:50.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7dc0034:::benchmark_data_smithi173_74914_object73520:head# 2024-08-30T23:36:50.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7dd38d4:::benchmark_data_smithi173_74914_object72585:head# 2024-08-30T23:36:50.129 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7dd8326:::benchmark_data_smithi173_74914_object96117:head# 2024-08-30T23:36:50.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ddd35b:::benchmark_data_smithi173_74914_object53353:head# 2024-08-30T23:36:50.130 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7df3839:::benchmark_data_smithi173_74914_object93962:head# 2024-08-30T23:36:50.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e06e93:::benchmark_data_smithi173_74914_object89248:head# 2024-08-30T23:36:50.131 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e191b1:::benchmark_data_smithi173_74914_object71551:head# 2024-08-30T23:36:50.134 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e21da3:::benchmark_data_smithi173_74914_object57729:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e46ee3:::benchmark_data_smithi173_74914_object55138:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e4770a:::benchmark_data_smithi173_74914_object95884:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e4aeef:::benchmark_data_smithi173_74914_object76479:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e52813:::benchmark_data_smithi173_74914_object88763:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e63522:::benchmark_data_smithi173_74914_object60676:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e65fe7:::benchmark_data_smithi173_74914_object96841:head# 2024-08-30T23:36:50.140 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e7cda1:::benchmark_data_smithi173_74914_object91018:head# 2024-08-30T23:36:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e83913:::benchmark_data_smithi173_74914_object74417:head# 2024-08-30T23:36:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7e948a0:::benchmark_data_smithi173_74914_object94857:head# 2024-08-30T23:36:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ea96fe:::benchmark_data_smithi173_74914_object92132:head# 2024-08-30T23:36:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7eac342:::benchmark_data_smithi173_74914_object90439:head# 2024-08-30T23:36:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ec4b52:::benchmark_data_smithi173_74914_object67273:head# 2024-08-30T23:36:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7eeec5e:::benchmark_data_smithi173_74914_object85090:head# 2024-08-30T23:36:50.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7eefedb:::benchmark_data_smithi173_74914_object55328:head# 2024-08-30T23:36:50.144 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7ef7eef:::benchmark_data_smithi173_74914_object93200:head# 2024-08-30T23:36:50.147 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7efb684:::benchmark_data_smithi173_74914_object68982:head# 2024-08-30T23:36:50.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f0ab1e:::benchmark_data_smithi173_74914_object90007:head# 2024-08-30T23:36:50.148 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f0b60c:::benchmark_data_smithi173_74914_object81848:head# 2024-08-30T23:36:50.151 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f10685:::benchmark_data_smithi173_74914_object82450:head# 2024-08-30T23:36:50.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f28cf2:::benchmark_data_smithi173_74914_object79178:head# 2024-08-30T23:36:50.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f3feba:::benchmark_data_smithi173_74914_object68357:head# 2024-08-30T23:36:50.152 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f67d39:::benchmark_data_smithi173_74914_object83660:head# 2024-08-30T23:36:50.153 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f6b7c6:::benchmark_data_smithi173_74914_object89561:head# 2024-08-30T23:36:50.156 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f6c1fb:::benchmark_data_smithi173_74914_object96922:head# 2024-08-30T23:36:50.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f701ff:::benchmark_data_smithi173_74914_object67284:head# 2024-08-30T23:36:50.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f846a1:::benchmark_data_smithi173_74914_object69245:head# 2024-08-30T23:36:50.157 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7f9beed:::benchmark_data_smithi173_74914_object97754:head# 2024-08-30T23:36:50.158 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7fc5b71:::benchmark_data_smithi173_74914_object71181:head# 2024-08-30T23:36:50.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f7fed6c2:::benchmark_data_smithi173_74914_object81541:head# 2024-08-30T23:36:50.161 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f800c84e:::benchmark_data_smithi173_74914_object67223:head# 2024-08-30T23:36:50.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f800ce2d:::benchmark_data_smithi173_74914_object65957:head# 2024-08-30T23:36:50.162 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8018e6b:::benchmark_data_smithi173_74914_object62420:head# 2024-08-30T23:36:50.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f802b63c:::benchmark_data_smithi173_74914_object75731:head# 2024-08-30T23:36:50.163 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8037fe5:::benchmark_data_smithi173_74914_object62796:head# 2024-08-30T23:36:50.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f805873f:::benchmark_data_smithi173_74914_object73280:head# 2024-08-30T23:36:50.166 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f805b922:::benchmark_data_smithi173_74914_object72409:head# 2024-08-30T23:36:50.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f80668c9:::benchmark_data_smithi173_74914_object93197:head# 2024-08-30T23:36:50.167 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f807a327:::benchmark_data_smithi173_74914_object97379:head# 2024-08-30T23:36:50.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f80ab1c3:::benchmark_data_smithi173_74914_object90667:head# 2024-08-30T23:36:50.168 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f80aebb7:::benchmark_data_smithi173_74914_object90057:head# 2024-08-30T23:36:50.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f80c117a:::benchmark_data_smithi173_74914_object62024:head# 2024-08-30T23:36:50.169 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f80fad36:::benchmark_data_smithi173_74914_object82706:head# 2024-08-30T23:36:50.172 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8105553:::benchmark_data_smithi173_74914_object74682:head# 2024-08-30T23:36:50.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f810cfa5:::benchmark_data_smithi173_74914_object88910:head# 2024-08-30T23:36:50.173 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f810e706:::benchmark_data_smithi173_74914_object67697:head# 2024-08-30T23:36:50.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8131307:::benchmark_data_smithi173_74914_object56012:head# 2024-08-30T23:36:50.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8136808:::benchmark_data_smithi173_74914_object83074:head# 2024-08-30T23:36:50.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8187216:::benchmark_data_smithi173_74914_object88349:head# 2024-08-30T23:36:50.175 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8187597:::benchmark_data_smithi173_74914_object71890:head# 2024-08-30T23:36:50.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f81a2472:::benchmark_data_smithi173_74914_object56528:head# 2024-08-30T23:36:50.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f81b4dcf:::benchmark_data_smithi173_74914_object82802:head# 2024-08-30T23:36:50.176 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f81e2b83:::benchmark_data_smithi173_74914_object81127:head# 2024-08-30T23:36:50.177 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f81e61e6:::benchmark_data_smithi173_74914_object94575:head# 2024-08-30T23:36:50.178 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f820bf2c:::benchmark_data_smithi173_74914_object62384:head# 2024-08-30T23:36:50.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f820c9f2:::benchmark_data_smithi173_74914_object74617:head# 2024-08-30T23:36:50.179 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8217e8a:::benchmark_data_smithi173_74914_object59208:head# 2024-08-30T23:36:50.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8221e39:::benchmark_data_smithi173_74914_object95361:head# 2024-08-30T23:36:50.180 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82289fd:::benchmark_data_smithi173_74914_object92229:head# 2024-08-30T23:36:50.181 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82478f3:::benchmark_data_smithi173_74914_object80557:head# 2024-08-30T23:36:50.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8262950:::benchmark_data_smithi173_74914_object75813:head# 2024-08-30T23:36:50.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82646d1:::benchmark_data_smithi173_74914_object56895:head# 2024-08-30T23:36:50.182 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8268a10:::benchmark_data_smithi173_74914_object89458:head# 2024-08-30T23:36:50.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8275f42:::benchmark_data_smithi173_74914_object65891:head# 2024-08-30T23:36:50.183 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82ae115:::benchmark_data_smithi173_74914_object74320:head# 2024-08-30T23:36:50.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82c052d:::benchmark_data_smithi173_74914_object56916:head# 2024-08-30T23:36:50.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82c5722:::benchmark_data_smithi173_74914_object53026:head# 2024-08-30T23:36:50.184 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82ca11a:::benchmark_data_smithi173_74914_object55152:head# 2024-08-30T23:36:50.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82f74f1:::benchmark_data_smithi173_74914_object75161:head# 2024-08-30T23:36:50.185 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82f914c:::benchmark_data_smithi173_74914_object82389:head# 2024-08-30T23:36:50.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f82f9be3:::benchmark_data_smithi173_74914_object94196:head# 2024-08-30T23:36:50.186 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f833c863:::benchmark_data_smithi173_74914_object55703:head# 2024-08-30T23:36:50.188 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f833ed26:::benchmark_data_smithi173_74914_object54967:head# 2024-08-30T23:36:50.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f833ef71:::benchmark_data_smithi173_74914_object64311:head# 2024-08-30T23:36:50.189 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f835e6d1:::benchmark_data_smithi173_74914_object86979:head# 2024-08-30T23:36:50.190 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f837a114:::benchmark_data_smithi173_74914_object92536:head# 2024-08-30T23:36:50.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f83a2f79:::benchmark_data_smithi173_74914_object92591:head# 2024-08-30T23:36:50.192 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f83bc20a:::benchmark_data_smithi173_74914_object54557:head# 2024-08-30T23:36:50.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f83cd6a8:::benchmark_data_smithi173_74914_object56008:head# 2024-08-30T23:36:50.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f83cdd8e:::benchmark_data_smithi173_74914_object69903:head# 2024-08-30T23:36:50.193 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f84442c7:::benchmark_data_smithi173_74914_object91643:head# 2024-08-30T23:36:50.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f84842c5:::benchmark_data_smithi173_74914_object92729:head# 2024-08-30T23:36:50.194 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f84b54ea:::benchmark_data_smithi173_74914_object82644:head# 2024-08-30T23:36:50.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f84eacf8:::benchmark_data_smithi173_74914_object63779:head# 2024-08-30T23:36:50.195 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8509266:::benchmark_data_smithi173_74914_object62000:head# 2024-08-30T23:36:50.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f85213ec:::benchmark_data_smithi173_74914_object96151:head# 2024-08-30T23:36:50.196 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f853415e:::benchmark_data_smithi173_74914_object82121:head# 2024-08-30T23:36:50.199 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8544e50:::benchmark_data_smithi173_74914_object59594:head# 2024-08-30T23:36:50.199 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f856b785:::benchmark_data_smithi173_74914_object92565:head# 2024-08-30T23:36:50.200 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f857a6e5:::benchmark_data_smithi173_74914_object70208:head# 2024-08-30T23:36:50.200 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f85a2998:::benchmark_data_smithi173_74914_object64782:head# 2024-08-30T23:36:50.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f85b1efd:::benchmark_data_smithi173_74914_object73410:head# 2024-08-30T23:36:50.201 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f85bfd0f:::benchmark_data_smithi173_74914_object57539:head# 2024-08-30T23:36:50.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f85e4692:::benchmark_data_smithi173_74914_object78776:head# 2024-08-30T23:36:50.202 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8606fe9:::benchmark_data_smithi173_74914_object66634:head# 2024-08-30T23:36:50.203 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f862f789:::benchmark_data_smithi173_74914_object53766:head# 2024-08-30T23:36:50.203 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f862f8f8:::benchmark_data_smithi173_74914_object83051:head# 2024-08-30T23:36:50.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f862f95d:::benchmark_data_smithi173_74914_object86536:head# 2024-08-30T23:36:50.204 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8660cf9:::benchmark_data_smithi173_74914_object88784:head# 2024-08-30T23:36:50.205 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f866b866:::benchmark_data_smithi173_74914_object81346:head# 2024-08-30T23:36:50.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8670520:::benchmark_data_smithi173_74914_object69692:head# 2024-08-30T23:36:50.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8673acc:::benchmark_data_smithi173_74914_object93267:head# 2024-08-30T23:36:50.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8678739:::benchmark_data_smithi173_74914_object70939:head# 2024-08-30T23:36:50.206 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f86c962f:::benchmark_data_smithi173_74914_object77964:head# 2024-08-30T23:36:50.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f86f41f7:::benchmark_data_smithi173_74914_object65669:head# 2024-08-30T23:36:50.207 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8711b7d:::benchmark_data_smithi173_74914_object69771:head# 2024-08-30T23:36:50.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f87255d8:::benchmark_data_smithi173_74914_object69227:head# 2024-08-30T23:36:50.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8761e46:::benchmark_data_smithi173_74914_object94488:head# 2024-08-30T23:36:50.208 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f877082f:::benchmark_data_smithi173_74914_object79130:head# 2024-08-30T23:36:50.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8775fb6:::benchmark_data_smithi173_74914_object92372:head# 2024-08-30T23:36:50.209 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8790973:::benchmark_data_smithi173_74914_object85316:head# 2024-08-30T23:36:50.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f87b3d3f:::benchmark_data_smithi173_74914_object67465:head# 2024-08-30T23:36:50.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f87cdab7:::benchmark_data_smithi173_74914_object54293:head# 2024-08-30T23:36:50.210 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f87d2a5d:::benchmark_data_smithi173_74914_object62983:head# 2024-08-30T23:36:50.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f87e0f88:::benchmark_data_smithi173_74914_object90586:head# 2024-08-30T23:36:50.211 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f87e1dd5:::benchmark_data_smithi173_74914_object82162:head# 2024-08-30T23:36:50.212 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f880323c:::benchmark_data_smithi173_74914_object77976:head# 2024-08-30T23:36:50.215 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f88288ac:::benchmark_data_smithi173_74914_object96941:head# 2024-08-30T23:36:50.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8835c16:::benchmark_data_smithi173_74914_object96460:head# 2024-08-30T23:36:50.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f885c061:::benchmark_data_smithi173_74914_object58604:head# 2024-08-30T23:36:50.216 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8865c29:::benchmark_data_smithi173_74914_object59228:head# 2024-08-30T23:36:50.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f88a5f83:::benchmark_data_smithi173_74914_object75356:head# 2024-08-30T23:36:50.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f88ac552:::benchmark_data_smithi173_74914_object89207:head# 2024-08-30T23:36:50.217 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f88c7e8e:::benchmark_data_smithi173_74914_object68991:head# 2024-08-30T23:36:50.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f88d869b:::benchmark_data_smithi173_74914_object88905:head# 2024-08-30T23:36:50.218 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8940940:::benchmark_data_smithi173_74914_object82517:head# 2024-08-30T23:36:50.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8957d9c:::benchmark_data_smithi173_74914_object77040:head# 2024-08-30T23:36:50.219 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f895b376:::benchmark_data_smithi173_74914_object92316:head# 2024-08-30T23:36:50.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f89705e9:::benchmark_data_smithi173_74914_object55708:head# 2024-08-30T23:36:50.220 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8988cbe:::benchmark_data_smithi173_74914_object72649:head# 2024-08-30T23:36:50.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f898a805:::benchmark_data_smithi173_74914_object56972:head# 2024-08-30T23:36:50.221 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f89ccddd:::benchmark_data_smithi173_74914_object95498:head# 2024-08-30T23:36:50.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f89ce1ec:::benchmark_data_smithi173_74914_object88222:head# 2024-08-30T23:36:50.222 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f89df6df:::benchmark_data_smithi173_74914_object58674:head# 2024-08-30T23:36:50.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:50.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8a09de9:::benchmark_data_smithi173_74914_object54874:head# 2024-08-30T23:36:50.223 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8a1f13b:::benchmark_data_smithi173_74914_object57388:head# 2024-08-30T23:36:50.224 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8a490d6:::benchmark_data_smithi173_74914_object91429:head# 2024-08-30T23:36:50.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8a57eb8:::benchmark_data_smithi173_74914_object55563:head# 2024-08-30T23:36:50.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8a6cd97:::benchmark_data_smithi173_74914_object71589:head# 2024-08-30T23:36:50.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8a7e032:::benchmark_data_smithi173_74914_object97794:head# 2024-08-30T23:36:50.225 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8aa681e:::benchmark_data_smithi173_74914_object81969:head# 2024-08-30T23:36:50.227 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8ac4d66:::benchmark_data_smithi173_74914_object66338:head# 2024-08-30T23:36:50.229 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8ad494d:::benchmark_data_smithi173_74914_object65805:head# 2024-08-30T23:36:50.229 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8aef00a:::benchmark_data_smithi173_74914_object74993:head# 2024-08-30T23:36:50.230 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8affa4d:::benchmark_data_smithi173_74914_object55581:head# 2024-08-30T23:36:50.230 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b00c4d:::benchmark_data_smithi173_74914_object89442:head# 2024-08-30T23:36:50.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b0d750:::benchmark_data_smithi173_74914_object82138:head# 2024-08-30T23:36:50.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b11ef0:::benchmark_data_smithi173_74914_object96359:head# 2024-08-30T23:36:50.231 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b3d9a2:::benchmark_data_smithi173_74914_object78128:head# 2024-08-30T23:36:50.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b55fec:::benchmark_data_smithi173_74914_object73808:head# 2024-08-30T23:36:50.232 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b713f6:::benchmark_data_smithi173_74914_object85198:head# 2024-08-30T23:36:50.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b725e2:::benchmark_data_smithi173_74914_object54592:head# 2024-08-30T23:36:50.233 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8b7bd10:::benchmark_data_smithi173_74914_object90622:head# 2024-08-30T23:36:50.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8bc7894:::benchmark_data_smithi173_74914_object59470:head# 2024-08-30T23:36:50.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8bc808a:::benchmark_data_smithi173_74914_object62146:head# 2024-08-30T23:36:50.234 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8bce144:::benchmark_data_smithi173_74914_object88288:head# 2024-08-30T23:36:50.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8bcea3a:::benchmark_data_smithi173_74914_object83042:head# 2024-08-30T23:36:50.235 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8c1607e:::benchmark_data_smithi173_74914_object84315:head# 2024-08-30T23:36:50.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8c2577f:::benchmark_data_smithi173_74914_object83399:head# 2024-08-30T23:36:50.236 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8c391c4:::benchmark_data_smithi173_74914_object65208:head# 2024-08-30T23:36:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8c6d9c6:::benchmark_data_smithi173_74914_object83745:head# 2024-08-30T23:36:50.237 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8c9e781:::benchmark_data_smithi173_74914_object53207:head# 2024-08-30T23:36:50.238 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8c9eafc:::benchmark_data_smithi173_74914_object62911:head# 2024-08-30T23:36:50.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8cbea77:::benchmark_data_smithi173_74914_object80239:head# 2024-08-30T23:36:50.252 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8cd7f28:::benchmark_data_smithi173_74914_object55799:head# 2024-08-30T23:36:50.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8cd8b26:::benchmark_data_smithi173_74914_object66478:head# 2024-08-30T23:36:50.253 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8cec708:::benchmark_data_smithi173_74914_object89213:head# 2024-08-30T23:36:50.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8cee423:::benchmark_data_smithi173_74914_object67452:head# 2024-08-30T23:36:50.254 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8d0cd4c:::benchmark_data_smithi173_74914_object89704:head# 2024-08-30T23:36:50.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8d0f460:::benchmark_data_smithi173_74914_object81933:head# 2024-08-30T23:36:50.255 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8d3a67a:::benchmark_data_smithi173_74914_object73128:head# 2024-08-30T23:36:50.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:50.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8d40da8:::benchmark_data_smithi173_74914_object61963:head# 2024-08-30T23:36:50.256 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8d5b166:::benchmark_data_smithi173_74914_object62996:head# 2024-08-30T23:36:50.257 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8d900a8:::benchmark_data_smithi173_74914_object66546:head# 2024-08-30T23:36:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:50.258 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8da4e5d:::benchmark_data_smithi173_74914_object82712:head# 2024-08-30T23:36:50.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8db4603:::benchmark_data_smithi173_74914_object62392:head# 2024-08-30T23:36:50.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8dc0ccc:::benchmark_data_smithi173_74914_object85920:head# 2024-08-30T23:36:50.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8dd2b5b:::benchmark_data_smithi173_74914_object71908:head# 2024-08-30T23:36:50.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8dd8db0:::benchmark_data_smithi173_74914_object82884:head# 2024-08-30T23:36:50.259 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8dec75a:::benchmark_data_smithi173_74914_object81854:head# 2024-08-30T23:36:50.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e537d0:::benchmark_data_smithi173_74914_object55317:head# 2024-08-30T23:36:50.260 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e730da:::benchmark_data_smithi173_74914_object97295:head# 2024-08-30T23:36:50.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e76450:::benchmark_data_smithi173_74914_object81364:head# 2024-08-30T23:36:50.261 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e76ac6:::benchmark_data_smithi173_74914_object57123:head# 2024-08-30T23:36:50.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e7bf0e:::benchmark_data_smithi173_74914_object61454:head# 2024-08-30T23:36:50.262 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e89559:::benchmark_data_smithi173_74914_object81384:head# 2024-08-30T23:36:50.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8e98df1:::benchmark_data_smithi173_74914_object89360:head# 2024-08-30T23:36:50.263 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8eb6f6b:::benchmark_data_smithi173_74914_object66250:head# 2024-08-30T23:36:50.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8eb8915:::benchmark_data_smithi173_74914_object86802:head# 2024-08-30T23:36:50.264 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8edb0c0:::benchmark_data_smithi173_74914_object91579:head# 2024-08-30T23:36:50.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8f0eef8:::benchmark_data_smithi173_74914_object55011:head# 2024-08-30T23:36:50.265 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8f8eef8:::benchmark_data_smithi173_74914_object64191:head# 2024-08-30T23:36:50.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8fb2e82:::benchmark_data_smithi173_74914_object88591:head# 2024-08-30T23:36:50.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f8fe8ca2:::benchmark_data_smithi173_74914_object57608:head# 2024-08-30T23:36:50.266 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f901411d:::benchmark_data_smithi173_74914_object74956:head# 2024-08-30T23:36:50.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f903829b:::benchmark_data_smithi173_74914_object90310:head# 2024-08-30T23:36:50.267 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9046367:::benchmark_data_smithi173_74914_object63257:head# 2024-08-30T23:36:50.268 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f907b5e4:::benchmark_data_smithi173_74914_object53328:head# 2024-08-30T23:36:50.268 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9088f49:::benchmark_data_smithi173_74914_object65310:head# 2024-08-30T23:36:50.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f909b933:::benchmark_data_smithi173_74914_object90251:head# 2024-08-30T23:36:50.269 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f90cd719:::benchmark_data_smithi173_74914_object65974:head# 2024-08-30T23:36:50.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f90ea3ad:::benchmark_data_smithi173_74914_object79272:head# 2024-08-30T23:36:50.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f90ffba3:::benchmark_data_smithi173_74914_object94621:head# 2024-08-30T23:36:50.270 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9110afe:::benchmark_data_smithi173_74914_object93299:head# 2024-08-30T23:36:50.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f914f5d5:::benchmark_data_smithi173_74914_object90835:head# 2024-08-30T23:36:50.271 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9151ced:::benchmark_data_smithi173_74914_object97245:head# 2024-08-30T23:36:50.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f91711c9:::benchmark_data_smithi173_74914_object79793:head# 2024-08-30T23:36:50.272 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f918a8f6:::benchmark_data_smithi173_74914_object68787:head# 2024-08-30T23:36:50.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f91a4c00:::benchmark_data_smithi173_74914_object96177:head# 2024-08-30T23:36:50.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f91bab48:::benchmark_data_smithi173_74914_object89347:head# 2024-08-30T23:36:50.273 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f91c7c76:::benchmark_data_smithi173_74914_object83082:head# 2024-08-30T23:36:50.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f91cb03e:::benchmark_data_smithi173_74914_object97257:head# 2024-08-30T23:36:50.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f91f3ece:::benchmark_data_smithi173_74914_object92783:head# 2024-08-30T23:36:50.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9222ec3:::benchmark_data_smithi173_74914_object84652:head# 2024-08-30T23:36:50.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f92474cf:::benchmark_data_smithi173_74914_object85547:head# 2024-08-30T23:36:50.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f924f4bf:::benchmark_data_smithi173_74914_object75766:head# 2024-08-30T23:36:50.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9265872:::benchmark_data_smithi173_74914_object97570:head# 2024-08-30T23:36:50.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f926b876:::benchmark_data_smithi173_74914_object80842:head# 2024-08-30T23:36:50.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f92a31be:::benchmark_data_smithi173_74914_object93586:head# 2024-08-30T23:36:50.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f92b1255:::benchmark_data_smithi173_74914_object80347:head# 2024-08-30T23:36:50.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f92c76ff:::benchmark_data_smithi173_74914_object58676:head# 2024-08-30T23:36:50.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9347a48:::benchmark_data_smithi173_74914_object64503:head# 2024-08-30T23:36:50.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f934f219:::benchmark_data_smithi173_74914_object68557:head# 2024-08-30T23:36:50.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f93904ee:::benchmark_data_smithi173_74914_object81467:head# 2024-08-30T23:36:50.280 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f939a416:::benchmark_data_smithi173_74914_object56175:head# 2024-08-30T23:36:50.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f93da155:::benchmark_data_smithi173_74914_object94007:head# 2024-08-30T23:36:50.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94042df:::benchmark_data_smithi173_74914_object55659:head# 2024-08-30T23:36:50.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9432469:::benchmark_data_smithi173_74914_object79521:head# 2024-08-30T23:36:50.282 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f945da64:::benchmark_data_smithi173_74914_object82282:head# 2024-08-30T23:36:50.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9469c68:::benchmark_data_smithi173_74914_object65439:head# 2024-08-30T23:36:50.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9491620:::benchmark_data_smithi173_74914_object84360:head# 2024-08-30T23:36:50.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9498fb7:::benchmark_data_smithi173_74914_object85152:head# 2024-08-30T23:36:50.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f949cc45:::benchmark_data_smithi173_74914_object90918:head# 2024-08-30T23:36:50.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94b79ee:::benchmark_data_smithi173_74914_object59083:head# 2024-08-30T23:36:50.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94bdedd:::benchmark_data_smithi173_74914_object67978:head# 2024-08-30T23:36:50.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94e3550:::benchmark_data_smithi173_74914_object74748:head# 2024-08-30T23:36:50.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94ec713:::benchmark_data_smithi173_74914_object63069:head# 2024-08-30T23:36:50.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94ec944:::benchmark_data_smithi173_74914_object96818:head# 2024-08-30T23:36:50.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94ed40b:::benchmark_data_smithi173_74914_object64415:head# 2024-08-30T23:36:50.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f94fc96d:::benchmark_data_smithi173_74914_object80217:head# 2024-08-30T23:36:50.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f951fed7:::benchmark_data_smithi173_74914_object85186:head# 2024-08-30T23:36:50.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f952bcfc:::benchmark_data_smithi173_74914_object97685:head# 2024-08-30T23:36:50.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f958dc3a:::benchmark_data_smithi173_74914_object74050:head# 2024-08-30T23:36:50.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f95995aa:::benchmark_data_smithi173_74914_object77202:head# 2024-08-30T23:36:50.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f959aebd:::benchmark_data_smithi173_74914_object58637:head# 2024-08-30T23:36:50.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f95b316d:::benchmark_data_smithi173_74914_object64536:head# 2024-08-30T23:36:50.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f95dbfc7:::benchmark_data_smithi173_74914_object74009:head# 2024-08-30T23:36:50.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f95dfa22:::benchmark_data_smithi173_74914_object66442:head# 2024-08-30T23:36:50.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f95ea044:::benchmark_data_smithi173_74914_object81125:head# 2024-08-30T23:36:50.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f962ac69:::benchmark_data_smithi173_74914_object71050:head# 2024-08-30T23:36:50.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f964f02c:::benchmark_data_smithi173_74914_object97123:head# 2024-08-30T23:36:50.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9650345:::benchmark_data_smithi173_74914_object72735:head# 2024-08-30T23:36:50.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9680fd9:::benchmark_data_smithi173_74914_object84729:head# 2024-08-30T23:36:50.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9688b59:::benchmark_data_smithi173_74914_object92609:head# 2024-08-30T23:36:50.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f969917c:::benchmark_data_smithi173_74914_object60574:head# 2024-08-30T23:36:50.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f96a6c2a:::benchmark_data_smithi173_74914_object93934:head# 2024-08-30T23:36:50.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f96a7677:::benchmark_data_smithi173_74914_object52871:head# 2024-08-30T23:36:50.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f96e9a69:::benchmark_data_smithi173_74914_object87950:head# 2024-08-30T23:36:50.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f96f35a8:::benchmark_data_smithi173_74914_object83887:head# 2024-08-30T23:36:50.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f970ce19:::benchmark_data_smithi173_74914_object71558:head# 2024-08-30T23:36:50.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f97121fb:::benchmark_data_smithi173_74914_object60037:head# 2024-08-30T23:36:50.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f97329e5:::benchmark_data_smithi173_74914_object58386:head# 2024-08-30T23:36:50.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9746995:::benchmark_data_smithi173_74914_object97625:head# 2024-08-30T23:36:50.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9748fd5:::benchmark_data_smithi173_74914_object90239:head# 2024-08-30T23:36:50.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f977d4ca:::benchmark_data_smithi173_74914_object94523:head# 2024-08-30T23:36:50.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9788c44:::benchmark_data_smithi173_74914_object58481:head# 2024-08-30T23:36:50.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f97b3d7c:::benchmark_data_smithi173_74914_object71443:head# 2024-08-30T23:36:50.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f97bb3ed:::benchmark_data_smithi173_74914_object54845:head# 2024-08-30T23:36:50.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f97cc49d:::benchmark_data_smithi173_74914_object77440:head# 2024-08-30T23:36:50.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f97cdb55:::benchmark_data_smithi173_74914_object86317:head# 2024-08-30T23:36:50.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f982e3e2:::benchmark_data_smithi173_74914_object86664:head# 2024-08-30T23:36:50.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f987b89c:::benchmark_data_smithi173_74914_object54006:head# 2024-08-30T23:36:50.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9889f23:::benchmark_data_smithi173_74914_object80451:head# 2024-08-30T23:36:50.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9896089:::benchmark_data_smithi173_74914_object53904:head# 2024-08-30T23:36:50.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9899133:::benchmark_data_smithi173_74914_object74325:head# 2024-08-30T23:36:50.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f98cc0e6:::benchmark_data_smithi173_74914_object95501:head# 2024-08-30T23:36:50.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f98ddc08:::benchmark_data_smithi173_74914_object83003:head# 2024-08-30T23:36:50.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f98eaf80:::benchmark_data_smithi173_74914_object59608:head# 2024-08-30T23:36:50.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f98f2f59:::benchmark_data_smithi173_74914_object92574:head# 2024-08-30T23:36:50.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9921833:::benchmark_data_smithi173_74914_object66707:head# 2024-08-30T23:36:50.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f99591bb:::benchmark_data_smithi173_74914_object71506:head# 2024-08-30T23:36:50.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f99764b9:::benchmark_data_smithi173_74914_object71689:head# 2024-08-30T23:36:50.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9983d83:::benchmark_data_smithi173_74914_object60795:head# 2024-08-30T23:36:50.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f99b80d6:::benchmark_data_smithi173_74914_object71846:head# 2024-08-30T23:36:50.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f99e20ae:::benchmark_data_smithi173_74914_object67076:head# 2024-08-30T23:36:50.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f99e7038:::benchmark_data_smithi173_74914_object86363:head# 2024-08-30T23:36:50.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a04229:::benchmark_data_smithi173_74914_object81531:head# 2024-08-30T23:36:50.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a2904f:::benchmark_data_smithi173_74914_object75007:head# 2024-08-30T23:36:50.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a2921c:::benchmark_data_smithi173_74914_object67199:head# 2024-08-30T23:36:50.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a5615c:::benchmark_data_smithi173_74914_object85559:head# 2024-08-30T23:36:50.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a6f6b3:::benchmark_data_smithi173_74914_object89230:head# 2024-08-30T23:36:50.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a7d4cf:::benchmark_data_smithi173_74914_object82171:head# 2024-08-30T23:36:50.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a98eeb:::benchmark_data_smithi173_74914_object83449:head# 2024-08-30T23:36:50.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a9c019:::benchmark_data_smithi173_74914_object95650:head# 2024-08-30T23:36:50.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9a9e853:::benchmark_data_smithi173_74914_object62153:head# 2024-08-30T23:36:50.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ac10e0:::benchmark_data_smithi173_74914_object77564:head# 2024-08-30T23:36:50.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ac8f5d:::benchmark_data_smithi173_74914_object70457:head# 2024-08-30T23:36:50.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ad28bf:::benchmark_data_smithi173_74914_object74046:head# 2024-08-30T23:36:50.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ae8ee4:::benchmark_data_smithi173_74914_object89637:head# 2024-08-30T23:36:50.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9aecf40:::benchmark_data_smithi173_74914_object56708:head# 2024-08-30T23:36:50.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9af20c3:::benchmark_data_smithi173_74914_object74769:head# 2024-08-30T23:36:50.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9affddc:::benchmark_data_smithi173_74914_object58509:head# 2024-08-30T23:36:50.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b0d35f:::benchmark_data_smithi173_74914_object79554:head# 2024-08-30T23:36:50.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b1646a:::benchmark_data_smithi173_74914_object71202:head# 2024-08-30T23:36:50.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b30ee2:::benchmark_data_smithi173_74914_object80442:head# 2024-08-30T23:36:50.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b3a427:::benchmark_data_smithi173_74914_object72484:head# 2024-08-30T23:36:50.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b4540a:::benchmark_data_smithi173_74914_object83682:head# 2024-08-30T23:36:50.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b4c109:::benchmark_data_smithi173_74914_object76238:head# 2024-08-30T23:36:50.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b5181f:::benchmark_data_smithi173_74914_object96780:head# 2024-08-30T23:36:50.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b51cae:::benchmark_data_smithi173_74914_object57069:head# 2024-08-30T23:36:50.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b65ad7:::benchmark_data_smithi173_74914_object85795:head# 2024-08-30T23:36:50.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b6939a:::benchmark_data_smithi173_74914_object84982:head# 2024-08-30T23:36:50.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b76de3:::benchmark_data_smithi173_74914_object63685:head# 2024-08-30T23:36:50.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b88343:::benchmark_data_smithi173_74914_object64778:head# 2024-08-30T23:36:50.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9b920f8:::benchmark_data_smithi173_74914_object60877:head# 2024-08-30T23:36:50.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ba795d:::benchmark_data_smithi173_74914_object66066:head# 2024-08-30T23:36:50.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9bac6f3:::benchmark_data_smithi173_74914_object81067:head# 2024-08-30T23:36:50.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9bb945f:::benchmark_data_smithi173_74914_object73546:head# 2024-08-30T23:36:50.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9bc850c:::benchmark_data_smithi173_74914_object56627:head# 2024-08-30T23:36:50.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9be48bc:::benchmark_data_smithi173_74914_object59404:head# 2024-08-30T23:36:50.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9c01575:::benchmark_data_smithi173_74914_object63410:head# 2024-08-30T23:36:50.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9c437a4:::benchmark_data_smithi173_74914_object87876:head# 2024-08-30T23:36:50.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9c71629:::benchmark_data_smithi173_74914_object60030:head# 2024-08-30T23:36:50.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9c74724:::benchmark_data_smithi173_74914_object83762:head# 2024-08-30T23:36:50.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9c938fc:::benchmark_data_smithi173_74914_object67619:head# 2024-08-30T23:36:50.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ca0450:::benchmark_data_smithi173_74914_object83942:head# 2024-08-30T23:36:50.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9cab1ee:::benchmark_data_smithi173_74914_object73317:head# 2024-08-30T23:36:50.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9cc5beb:::benchmark_data_smithi173_74914_object79005:head# 2024-08-30T23:36:50.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ccff23:::benchmark_data_smithi173_74914_object74673:head# 2024-08-30T23:36:50.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d2db12:::benchmark_data_smithi173_74914_object60041:head# 2024-08-30T23:36:50.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d44c3e:::benchmark_data_smithi173_74914_object59752:head# 2024-08-30T23:36:50.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d4c2af:::benchmark_data_smithi173_74914_object64534:head# 2024-08-30T23:36:50.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d4ff6a:::benchmark_data_smithi173_74914_object78586:head# 2024-08-30T23:36:50.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d5281b:::benchmark_data_smithi173_74914_object55260:head# 2024-08-30T23:36:50.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d6a201:::benchmark_data_smithi173_74914_object79478:head# 2024-08-30T23:36:50.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9d98c72:::benchmark_data_smithi173_74914_object62379:head# 2024-08-30T23:36:50.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9da25d8:::benchmark_data_smithi173_74914_object85191:head# 2024-08-30T23:36:50.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9da3ae0:::benchmark_data_smithi173_74914_object65285:head# 2024-08-30T23:36:50.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9db31c7:::benchmark_data_smithi173_74914_object76705:head# 2024-08-30T23:36:50.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9db892c:::benchmark_data_smithi173_74914_object77477:head# 2024-08-30T23:36:50.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9dbacb2:::benchmark_data_smithi173_74914_object96679:head# 2024-08-30T23:36:50.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9dece86:::benchmark_data_smithi173_74914_object53077:head# 2024-08-30T23:36:50.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9dfc307:::benchmark_data_smithi173_74914_object59614:head# 2024-08-30T23:36:50.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e0dfb0:::benchmark_data_smithi173_74914_object68584:head# 2024-08-30T23:36:50.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e0f179:::benchmark_data_smithi173_74914_object94381:head# 2024-08-30T23:36:50.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e41923:::benchmark_data_smithi173_74914_object72765:head# 2024-08-30T23:36:50.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e5ced0:::benchmark_data_smithi173_74914_object60317:head# 2024-08-30T23:36:50.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e610be:::benchmark_data_smithi173_74914_object73747:head# 2024-08-30T23:36:50.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e7dfde:::benchmark_data_smithi173_74914_object92498:head# 2024-08-30T23:36:50.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e892b3:::benchmark_data_smithi173_74914_object86283:head# 2024-08-30T23:36:50.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e8aa4b:::benchmark_data_smithi173_74914_object97100:head# 2024-08-30T23:36:50.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9e9c941:::benchmark_data_smithi173_74914_object57784:head# 2024-08-30T23:36:50.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9eb0b81:::benchmark_data_smithi173_74914_object73437:head# 2024-08-30T23:36:50.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ecdbbc:::benchmark_data_smithi173_74914_object64539:head# 2024-08-30T23:36:50.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9eddc12:::benchmark_data_smithi173_74914_object53610:head# 2024-08-30T23:36:50.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9f26410:::benchmark_data_smithi173_74914_object67941:head# 2024-08-30T23:36:50.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9f8a5da:::benchmark_data_smithi173_74914_object96785:head# 2024-08-30T23:36:50.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9f8d301:::benchmark_data_smithi173_74914_object78042:head# 2024-08-30T23:36:50.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9f8e0db:::benchmark_data_smithi173_74914_object88922:head# 2024-08-30T23:36:50.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9fa9405:::benchmark_data_smithi173_74914_object92502:head# 2024-08-30T23:36:50.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9fc7b89:::benchmark_data_smithi173_74914_object53895:head# 2024-08-30T23:36:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9fd5f39:::benchmark_data_smithi173_74914_object54871:head# 2024-08-30T23:36:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ffa377:::benchmark_data_smithi173_74914_object57802:head# 2024-08-30T23:36:50.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:f9ffb571:::benchmark_data_smithi173_74914_object62029:head# 2024-08-30T23:36:50.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa010ce2:::benchmark_data_smithi173_74914_object54905:head# 2024-08-30T23:36:50.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa01518b:::benchmark_data_smithi173_74914_object62022:head# 2024-08-30T23:36:50.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa03c80f:::benchmark_data_smithi173_74914_object83288:head# 2024-08-30T23:36:50.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa048538:::benchmark_data_smithi173_74914_object75752:head# 2024-08-30T23:36:50.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa054f08:::benchmark_data_smithi173_74914_object86277:head# 2024-08-30T23:36:50.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa07361b:::benchmark_data_smithi173_74914_object92256:head# 2024-08-30T23:36:50.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa0971ea:::benchmark_data_smithi173_74914_object80305:head# 2024-08-30T23:36:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa0b0c9d:::benchmark_data_smithi173_74914_object80814:head# 2024-08-30T23:36:50.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa0c9f4a:::benchmark_data_smithi173_74914_object82905:head# 2024-08-30T23:36:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa0d4c28:::benchmark_data_smithi173_74914_object53573:head# 2024-08-30T23:36:50.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa0d75e5:::benchmark_data_smithi173_74914_object95738:head# 2024-08-30T23:36:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa0d83d1:::benchmark_data_smithi173_74914_object71771:head# 2024-08-30T23:36:50.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa12b371:::benchmark_data_smithi173_74914_object83493:head# 2024-08-30T23:36:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa1339b4:::benchmark_data_smithi173_74914_object53998:head# 2024-08-30T23:36:50.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa14dcfe:::benchmark_data_smithi173_74914_object77497:head# 2024-08-30T23:36:50.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa15a50f:::benchmark_data_smithi173_74914_object69716:head# 2024-08-30T23:36:50.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa15b4f2:::benchmark_data_smithi173_74914_object75464:head# 2024-08-30T23:36:50.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa166d16:::benchmark_data_smithi173_74914_object75597:head# 2024-08-30T23:36:50.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa16b7e8:::benchmark_data_smithi173_74914_object81170:head# 2024-08-30T23:36:50.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa188760:::benchmark_data_smithi173_74914_object92616:head# 2024-08-30T23:36:50.361 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa193019:::benchmark_data_smithi173_74914_object65070:head# 2024-08-30T23:36:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa1999bd:::benchmark_data_smithi173_74914_object57848:head# 2024-08-30T23:36:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa1bb20a:::benchmark_data_smithi173_74914_object65418:head# 2024-08-30T23:36:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa1cf1e9:::benchmark_data_smithi173_74914_object85679:head# 2024-08-30T23:36:50.363 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa1e4b21:::benchmark_data_smithi173_74914_object75741:head# 2024-08-30T23:36:50.364 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa1f8dde:::benchmark_data_smithi173_74914_object77573:head# 2024-08-30T23:36:50.364 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa23b016:::benchmark_data_smithi173_74914_object92885:head# 2024-08-30T23:36:50.366 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa24731c:::benchmark_data_smithi173_74914_object91297:head# 2024-08-30T23:36:50.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa2a178d:::benchmark_data_smithi173_74914_object57657:head# 2024-08-30T23:36:50.367 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa2b5e69:::benchmark_data_smithi173_74914_object88098:head# 2024-08-30T23:36:50.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa2ddb2e:::benchmark_data_smithi173_74914_object87344:head# 2024-08-30T23:36:50.368 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa311882:::benchmark_data_smithi173_74914_object96116:head# 2024-08-30T23:36:50.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa31f8a9:::benchmark_data_smithi173_74914_object62340:head# 2024-08-30T23:36:50.369 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa35015d:::benchmark_data_smithi173_74914_object59008:head# 2024-08-30T23:36:50.370 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa36d1a6:::benchmark_data_smithi173_74914_object96668:head# 2024-08-30T23:36:50.370 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa376000:::benchmark_data_smithi173_74914_object91717:head# 2024-08-30T23:36:50.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa37b580:::benchmark_data_smithi173_74914_object71305:head# 2024-08-30T23:36:50.371 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa3ab643:::benchmark_data_smithi173_74914_object86308:head# 2024-08-30T23:36:50.372 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa3df68e:::benchmark_data_smithi173_74914_object76456:head# 2024-08-30T23:36:50.373 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa40b031:::benchmark_data_smithi173_74914_object61604:head# 2024-08-30T23:36:50.373 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa41f553:::benchmark_data_smithi173_74914_object53058:head# 2024-08-30T23:36:50.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa41fb48:::benchmark_data_smithi173_74914_object67147:head# 2024-08-30T23:36:50.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa44b260:::benchmark_data_smithi173_74914_object90301:head# 2024-08-30T23:36:50.374 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa484210:::benchmark_data_smithi173_74914_object94269:head# 2024-08-30T23:36:50.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa484bcc:::benchmark_data_smithi173_74914_object54319:head# 2024-08-30T23:36:50.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa4a9588:::benchmark_data_smithi173_74914_object95251:head# 2024-08-30T23:36:50.375 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa4e8873:::benchmark_data_smithi173_74914_object93235:head# 2024-08-30T23:36:50.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa4fd5fb:::benchmark_data_smithi173_74914_object76136:head# 2024-08-30T23:36:50.376 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa51040d:::benchmark_data_smithi173_74914_object70055:head# 2024-08-30T23:36:50.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa52a0f6:::benchmark_data_smithi173_74914_object63952:head# 2024-08-30T23:36:50.377 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa52a2ad:::benchmark_data_smithi173_74914_object72491:head# 2024-08-30T23:36:50.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5504ae:::benchmark_data_smithi173_74914_object64155:head# 2024-08-30T23:36:50.378 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa558767:::benchmark_data_smithi173_74914_object76598:head# 2024-08-30T23:36:50.380 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa55b9f4:::benchmark_data_smithi173_74914_object96945:head# 2024-08-30T23:36:50.381 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa55dda8:::benchmark_data_smithi173_74914_object68359:head# 2024-08-30T23:36:50.381 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa56e6f6:::benchmark_data_smithi173_74914_object68349:head# 2024-08-30T23:36:50.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5752d7:::benchmark_data_smithi173_74914_object71717:head# 2024-08-30T23:36:50.382 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa585891:::benchmark_data_smithi173_74914_object67952:head# 2024-08-30T23:36:50.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5a6092:::benchmark_data_smithi173_74914_object75581:head# 2024-08-30T23:36:50.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5b4cd7:::benchmark_data_smithi173_74914_object62829:head# 2024-08-30T23:36:50.383 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5c0f0a:::benchmark_data_smithi173_74914_object67068:head# 2024-08-30T23:36:50.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5c9d47:::benchmark_data_smithi173_74914_object93428:head# 2024-08-30T23:36:50.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa5d94b6:::benchmark_data_smithi173_74914_object87760:head# 2024-08-30T23:36:50.384 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa61eca6:::benchmark_data_smithi173_74914_object96604:head# 2024-08-30T23:36:50.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa621661:::benchmark_data_smithi173_74914_object54521:head# 2024-08-30T23:36:50.385 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6232e4:::benchmark_data_smithi173_74914_object55642:head# 2024-08-30T23:36:50.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6243b5:::benchmark_data_smithi173_74914_object94059:head# 2024-08-30T23:36:50.386 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6413f5:::benchmark_data_smithi173_74914_object96095:head# 2024-08-30T23:36:50.387 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6544f3:::benchmark_data_smithi173_74914_object78112:head# 2024-08-30T23:36:50.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.389 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa65def8:::benchmark_data_smithi173_74914_object66252:head# 2024-08-30T23:36:50.390 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa65f1c3:::benchmark_data_smithi173_74914_object80155:head# 2024-08-30T23:36:50.390 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa66cb0f:::benchmark_data_smithi173_74914_object62454:head# 2024-08-30T23:36:50.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:50.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa67dd96:::benchmark_data_smithi173_74914_object58044:head# 2024-08-30T23:36:50.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa686b93:::benchmark_data_smithi173_74914_object70785:head# 2024-08-30T23:36:50.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6b183b:::benchmark_data_smithi173_74914_object73750:head# 2024-08-30T23:36:50.391 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6b4de0:::benchmark_data_smithi173_74914_object78385:head# 2024-08-30T23:36:50.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6d8951:::benchmark_data_smithi173_74914_object97249:head# 2024-08-30T23:36:50.394 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa6e3e47:::benchmark_data_smithi173_74914_object89492:head# 2024-08-30T23:36:50.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa709967:::benchmark_data_smithi173_74914_object92355:head# 2024-08-30T23:36:50.395 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa71177a:::benchmark_data_smithi173_74914_object77118:head# 2024-08-30T23:36:50.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa712b9d:::benchmark_data_smithi173_74914_object54766:head# 2024-08-30T23:36:50.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa722b70:::benchmark_data_smithi173_74914_object71146:head# 2024-08-30T23:36:50.396 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa738068:::benchmark_data_smithi173_74914_object73552:head# 2024-08-30T23:36:50.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa73912d:::benchmark_data_smithi173_74914_object87307:head# 2024-08-30T23:36:50.397 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa785e36:::benchmark_data_smithi173_74914_object94924:head# 2024-08-30T23:36:50.398 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa7a33c6:::benchmark_data_smithi173_74914_object69067:head# 2024-08-30T23:36:50.398 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa7bcf1e:::benchmark_data_smithi173_74914_object61052:head# 2024-08-30T23:36:50.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa7cd365:::benchmark_data_smithi173_74914_object66765:head# 2024-08-30T23:36:50.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa8169a4:::benchmark_data_smithi173_74914_object53485:head# 2024-08-30T23:36:50.399 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa85c07d:::benchmark_data_smithi173_74914_object77519:head# 2024-08-30T23:36:50.400 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa871a97:::benchmark_data_smithi173_74914_object83477:head# 2024-08-30T23:36:50.400 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa893404:::benchmark_data_smithi173_74914_object86909:head# 2024-08-30T23:36:50.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa89e0b9:::benchmark_data_smithi173_74914_object85786:head# 2024-08-30T23:36:50.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa8a8e6b:::benchmark_data_smithi173_74914_object98203:head# 2024-08-30T23:36:50.401 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa8abbe9:::benchmark_data_smithi173_74914_object68962:head# 2024-08-30T23:36:50.402 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa8abe96:::benchmark_data_smithi173_74914_object67453:head# 2024-08-30T23:36:50.403 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa901d33:::benchmark_data_smithi173_74914_object82497:head# 2024-08-30T23:36:50.403 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa911756:::benchmark_data_smithi173_74914_object86282:head# 2024-08-30T23:36:50.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa921d37:::benchmark_data_smithi173_74914_object91333:head# 2024-08-30T23:36:50.404 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9261b9:::benchmark_data_smithi173_74914_object93457:head# 2024-08-30T23:36:50.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa92fcd7:::benchmark_data_smithi173_74914_object60019:head# 2024-08-30T23:36:50.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa97afbe:::benchmark_data_smithi173_74914_object68302:head# 2024-08-30T23:36:50.405 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa995538:::benchmark_data_smithi173_74914_object90256:head# 2024-08-30T23:36:50.406 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa999cd1:::benchmark_data_smithi173_74914_object63924:head# 2024-08-30T23:36:50.406 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9a6eb2:::benchmark_data_smithi173_74914_object91739:head# 2024-08-30T23:36:50.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9bb75d:::benchmark_data_smithi173_74914_object76866:head# 2024-08-30T23:36:50.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9c0f52:::benchmark_data_smithi173_74914_object62839:head# 2024-08-30T23:36:50.407 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9d0bb7:::benchmark_data_smithi173_74914_object86248:head# 2024-08-30T23:36:50.408 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9d1bf8:::benchmark_data_smithi173_74914_object53916:head# 2024-08-30T23:36:50.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fa9e520b:::benchmark_data_smithi173_74914_object61429:head# 2024-08-30T23:36:50.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faa00001:::benchmark_data_smithi173_74914_object87369:head# 2024-08-30T23:36:50.411 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faa0b5b8:::benchmark_data_smithi173_74914_object57609:head# 2024-08-30T23:36:50.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faa13900:::benchmark_data_smithi173_74914_object63082:head# 2024-08-30T23:36:50.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faa4ec37:::benchmark_data_smithi173_74914_object65689:head# 2024-08-30T23:36:50.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faa74379:::benchmark_data_smithi173_74914_object68622:head# 2024-08-30T23:36:50.412 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faa9791d:::benchmark_data_smithi173_74914_object89301:head# 2024-08-30T23:36:50.413 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faab70db:::benchmark_data_smithi173_74914_object59778:head# 2024-08-30T23:36:50.415 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faabd22f:::benchmark_data_smithi173_74914_object56588:head# 2024-08-30T23:36:50.415 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faac3e08:::benchmark_data_smithi173_74914_object83258:head# 2024-08-30T23:36:50.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faad6448:::benchmark_data_smithi173_74914_object56273:head# 2024-08-30T23:36:50.416 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faaf512c:::benchmark_data_smithi173_74914_object84311:head# 2024-08-30T23:36:50.417 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab0591f:::benchmark_data_smithi173_74914_object52836:head# 2024-08-30T23:36:50.417 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab16dc6:::benchmark_data_smithi173_74914_object88983:head# 2024-08-30T23:36:50.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab1a67d:::benchmark_data_smithi173_74914_object66710:head# 2024-08-30T23:36:50.418 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab314a9:::benchmark_data_smithi173_74914_object95045:head# 2024-08-30T23:36:50.419 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab38a16:::benchmark_data_smithi173_74914_object68347:head# 2024-08-30T23:36:50.419 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab44a77:::benchmark_data_smithi173_74914_object82475:head# 2024-08-30T23:36:50.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fab75650:::benchmark_data_smithi173_74914_object62287:head# 2024-08-30T23:36:50.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faba316f:::benchmark_data_smithi173_74914_object61423:head# 2024-08-30T23:36:50.420 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fabca71f:::benchmark_data_smithi173_74914_object58497:head# 2024-08-30T23:36:50.421 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac0067c:::benchmark_data_smithi173_74914_object60617:head# 2024-08-30T23:36:50.421 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac0d528:::benchmark_data_smithi173_74914_object87298:head# 2024-08-30T23:36:50.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac13a55:::benchmark_data_smithi173_74914_object76862:head# 2024-08-30T23:36:50.422 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac14540:::benchmark_data_smithi173_74914_object67556:head# 2024-08-30T23:36:50.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac1fe94:::benchmark_data_smithi173_74914_object89841:head# 2024-08-30T23:36:50.423 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac37c9d:::benchmark_data_smithi173_74914_object68613:head# 2024-08-30T23:36:50.424 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac431cf:::benchmark_data_smithi173_74914_object91248:head# 2024-08-30T23:36:50.424 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac4ac11:::benchmark_data_smithi173_74914_object75016:head# 2024-08-30T23:36:50.425 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac95155:::benchmark_data_smithi173_74914_object84086:head# 2024-08-30T23:36:50.425 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fac9dc9a:::benchmark_data_smithi173_74914_object56640:head# 2024-08-30T23:36:50.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faca2ea8:::benchmark_data_smithi173_74914_object94073:head# 2024-08-30T23:36:50.426 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faced844:::benchmark_data_smithi173_74914_object62050:head# 2024-08-30T23:36:50.427 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad0e249:::benchmark_data_smithi173_74914_object68537:head# 2024-08-30T23:36:50.428 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad1472e:::benchmark_data_smithi173_74914_object62211:head# 2024-08-30T23:36:50.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad17643:::benchmark_data_smithi173_74914_object93338:head# 2024-08-30T23:36:50.430 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad1bcc3:::benchmark_data_smithi173_74914_object71064:head# 2024-08-30T23:36:50.432 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad441be:::benchmark_data_smithi173_74914_object97895:head# 2024-08-30T23:36:50.433 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad4e1fa:::benchmark_data_smithi173_74914_object86768:head# 2024-08-30T23:36:50.433 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad62cb4:::benchmark_data_smithi173_74914_object68409:head# 2024-08-30T23:36:50.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fad76bdc:::benchmark_data_smithi173_74914_object85667:head# 2024-08-30T23:36:50.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fadc02c4:::benchmark_data_smithi173_74914_object68366:head# 2024-08-30T23:36:50.434 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fade3233:::benchmark_data_smithi173_74914_object78806:head# 2024-08-30T23:36:50.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fade8c0c:::benchmark_data_smithi173_74914_object87673:head# 2024-08-30T23:36:50.435 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fadfafa7:::benchmark_data_smithi173_74914_object57994:head# 2024-08-30T23:36:50.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae0820a:::benchmark_data_smithi173_74914_object71388:head# 2024-08-30T23:36:50.436 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae0d869:::benchmark_data_smithi173_74914_object54639:head# 2024-08-30T23:36:50.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae378ef:::benchmark_data_smithi173_74914_object63481:head# 2024-08-30T23:36:50.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae59ccd:::benchmark_data_smithi173_74914_object58226:head# 2024-08-30T23:36:50.437 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae635e6:::benchmark_data_smithi173_74914_object89433:head# 2024-08-30T23:36:50.438 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae6e805:::benchmark_data_smithi173_74914_object89811:head# 2024-08-30T23:36:50.439 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fae9d461:::benchmark_data_smithi173_74914_object58107:head# 2024-08-30T23:36:50.439 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faea070f:::benchmark_data_smithi173_74914_object94362:head# 2024-08-30T23:36:50.440 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faed1c63:::benchmark_data_smithi173_74914_object96188:head# 2024-08-30T23:36:50.440 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:50.440 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:50.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faef221a:::benchmark_data_smithi173_74914_object86258:head# 2024-08-30T23:36:50.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faf165a1:::benchmark_data_smithi173_74914_object73158:head# 2024-08-30T23:36:50.441 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faf21504:::benchmark_data_smithi173_74914_object95571:head# 2024-08-30T23:36:50.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faf2fef9:::benchmark_data_smithi173_74914_object62679:head# 2024-08-30T23:36:50.442 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faf45a35:::benchmark_data_smithi173_74914_object79292:head# 2024-08-30T23:36:50.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faf4fc8d:::benchmark_data_smithi173_74914_object67101:head# 2024-08-30T23:36:50.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafa7740:::benchmark_data_smithi173_74914_object69795:head# 2024-08-30T23:36:50.443 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafc2473:::benchmark_data_smithi173_74914_object81585:head# 2024-08-30T23:36:50.444 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafd01ed:::benchmark_data_smithi173_74914_object78544:head# 2024-08-30T23:36:50.444 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafdf3dd:::benchmark_data_smithi173_74914_object53592:head# 2024-08-30T23:36:50.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafe0f27:::benchmark_data_smithi173_74914_object93000:head# 2024-08-30T23:36:50.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafe1959:::benchmark_data_smithi173_74914_object65231:head# 2024-08-30T23:36:50.445 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafe1bdc:::benchmark_data_smithi173_74914_object82349:head# 2024-08-30T23:36:50.447 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:faffb770:::benchmark_data_smithi173_74914_object82956:head# 2024-08-30T23:36:50.447 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fafffbd0:::benchmark_data_smithi173_74914_object72128:head# 2024-08-30T23:36:50.447 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb017c20:::benchmark_data_smithi173_74914_object78137:head# 2024-08-30T23:36:50.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb019ce3:::benchmark_data_smithi173_74914_object76609:head# 2024-08-30T23:36:50.448 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb02202b:::benchmark_data_smithi173_74914_object92847:head# 2024-08-30T23:36:50.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb02a6f4:::benchmark_data_smithi173_74914_object94326:head# 2024-08-30T23:36:50.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb04b5f1:::benchmark_data_smithi173_74914_object56341:head# 2024-08-30T23:36:50.449 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb051c26:::benchmark_data_smithi173_74914_object81360:head# 2024-08-30T23:36:50.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb05b555:::benchmark_data_smithi173_74914_object91435:head# 2024-08-30T23:36:50.450 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb07e730:::benchmark_data_smithi173_74914_object76869:head# 2024-08-30T23:36:50.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb089b40:::benchmark_data_smithi173_74914_object54013:head# 2024-08-30T23:36:50.451 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb08a449:::benchmark_data_smithi173_74914_object93421:head# 2024-08-30T23:36:50.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb0d72e1:::benchmark_data_smithi173_74914_object57453:head# 2024-08-30T23:36:50.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb0dcdce:::benchmark_data_smithi173_74914_object87045:head# 2024-08-30T23:36:50.452 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb0ed265:::benchmark_data_smithi173_74914_object76054:head# 2024-08-30T23:36:50.453 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb108702:::benchmark_data_smithi173_74914_object59647:head# 2024-08-30T23:36:50.453 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb10d68e:::benchmark_data_smithi173_74914_object83119:head# 2024-08-30T23:36:50.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb10f931:::benchmark_data_smithi173_74914_object91522:head# 2024-08-30T23:36:50.454 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb11ccf6:::benchmark_data_smithi173_74914_object68567:head# 2024-08-30T23:36:50.455 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb139c09:::benchmark_data_smithi173_74914_object71102:head# 2024-08-30T23:36:50.455 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb1584cd:::benchmark_data_smithi173_74914_object64104:head# 2024-08-30T23:36:50.456 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb18e283:::benchmark_data_smithi173_74914_object77325:head# 2024-08-30T23:36:50.456 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb19685c:::benchmark_data_smithi173_74914_object95695:head# 2024-08-30T23:36:50.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb1b983f:::benchmark_data_smithi173_74914_object84972:head# 2024-08-30T23:36:50.457 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb1e985d:::benchmark_data_smithi173_74914_object97427:head# 2024-08-30T23:36:50.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb1fc5bb:::benchmark_data_smithi173_74914_object57505:head# 2024-08-30T23:36:50.458 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb2000eb:::benchmark_data_smithi173_74914_object56014:head# 2024-08-30T23:36:50.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb206e54:::benchmark_data_smithi173_74914_object56315:head# 2024-08-30T23:36:50.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb208a64:::benchmark_data_smithi173_74914_object52921:head# 2024-08-30T23:36:50.459 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb223abf:::benchmark_data_smithi173_74914_object54962:head# 2024-08-30T23:36:50.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb242290:::benchmark_data_smithi173_74914_object96749:head# 2024-08-30T23:36:50.460 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb26dca2:::benchmark_data_smithi173_74914_object89057:head# 2024-08-30T23:36:50.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb2ba412:::benchmark_data_smithi173_74914_object78755:head# 2024-08-30T23:36:50.461 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb2e2611:::benchmark_data_smithi173_74914_object53439:head# 2024-08-30T23:36:50.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb2fd4c2:::benchmark_data_smithi173_74914_object58093:head# 2024-08-30T23:36:50.462 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb310be2:::benchmark_data_smithi173_74914_object54691:head# 2024-08-30T23:36:50.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb32adfd:::benchmark_data_smithi173_74914_object77102:head# 2024-08-30T23:36:50.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3395b0:::benchmark_data_smithi173_74914_object53125:head# 2024-08-30T23:36:50.463 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb35d04b:::benchmark_data_smithi173_74914_object72215:head# 2024-08-30T23:36:50.464 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3676c1:::benchmark_data_smithi173_74914_object58769:head# 2024-08-30T23:36:50.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3796d1:::benchmark_data_smithi173_74914_object80226:head# 2024-08-30T23:36:50.466 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3b816d:::benchmark_data_smithi173_74914_object72497:head# 2024-08-30T23:36:50.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3c3e53:::benchmark_data_smithi173_74914_object94017:head# 2024-08-30T23:36:50.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3d02a6:::benchmark_data_smithi173_74914_object83333:head# 2024-08-30T23:36:50.467 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3d97a7:::benchmark_data_smithi173_74914_object86202:head# 2024-08-30T23:36:50.468 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb3eea57:::benchmark_data_smithi173_74914_object98189:head# 2024-08-30T23:36:50.468 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb404476:::benchmark_data_smithi173_74914_object77405:head# 2024-08-30T23:36:50.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb46d1c5:::benchmark_data_smithi173_74914_object55859:head# 2024-08-30T23:36:50.469 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb47f73f:::benchmark_data_smithi173_74914_object82551:head# 2024-08-30T23:36:50.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb48fe1c:::benchmark_data_smithi173_74914_object84689:head# 2024-08-30T23:36:50.470 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4b2ac9:::benchmark_data_smithi173_74914_object62039:head# 2024-08-30T23:36:50.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4b4f89:::benchmark_data_smithi173_74914_object69290:head# 2024-08-30T23:36:50.471 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4be2db:::benchmark_data_smithi173_74914_object54749:head# 2024-08-30T23:36:50.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4c35ad:::benchmark_data_smithi173_74914_object54100:head# 2024-08-30T23:36:50.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4d9252:::benchmark_data_smithi173_74914_object68794:head# 2024-08-30T23:36:50.472 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4def20:::benchmark_data_smithi173_74914_object93818:head# 2024-08-30T23:36:50.473 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4e2163:::benchmark_data_smithi173_74914_object66902:head# 2024-08-30T23:36:50.473 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb4e97bd:::benchmark_data_smithi173_74914_object75093:head# 2024-08-30T23:36:50.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb504782:::benchmark_data_smithi173_74914_object93603:head# 2024-08-30T23:36:50.474 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb50f9b7:::benchmark_data_smithi173_74914_object56171:head# 2024-08-30T23:36:50.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5343bd:::benchmark_data_smithi173_74914_object92311:head# 2024-08-30T23:36:50.475 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb534d3e:::benchmark_data_smithi173_74914_object56502:head# 2024-08-30T23:36:50.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb53ecce:::benchmark_data_smithi173_74914_object97281:head# 2024-08-30T23:36:50.476 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb541383:::benchmark_data_smithi173_74914_object53381:head# 2024-08-30T23:36:50.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb542b3b:::benchmark_data_smithi173_74914_object91275:head# 2024-08-30T23:36:50.477 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb58b9e4:::benchmark_data_smithi173_74914_object79441:head# 2024-08-30T23:36:50.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb597492:::benchmark_data_smithi173_74914_object61573:head# 2024-08-30T23:36:50.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5b291f:::benchmark_data_smithi173_74914_object77021:head# 2024-08-30T23:36:50.478 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5d400c:::benchmark_data_smithi173_74914_object69374:head# 2024-08-30T23:36:50.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5e19dd:::benchmark_data_smithi173_74914_object79927:head# 2024-08-30T23:36:50.479 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5e729f:::benchmark_data_smithi173_74914_object81600:head# 2024-08-30T23:36:50.480 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5eb487:::benchmark_data_smithi173_74914_object80948:head# 2024-08-30T23:36:50.480 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb5f96bc:::benchmark_data_smithi173_74914_object53422:head# 2024-08-30T23:36:50.481 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb602991:::benchmark_data_smithi173_74914_object80149:head# 2024-08-30T23:36:50.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb605d75:::benchmark_data_smithi173_74914_object71013:head# 2024-08-30T23:36:50.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb6348b8:::benchmark_data_smithi173_74914_object85648:head# 2024-08-30T23:36:50.482 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb6353bf:::benchmark_data_smithi173_74914_object79794:head# 2024-08-30T23:36:50.483 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb65c9d6:::benchmark_data_smithi173_74914_object85174:head# 2024-08-30T23:36:50.483 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb661f08:::benchmark_data_smithi173_74914_object72153:head# 2024-08-30T23:36:50.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb676595:::benchmark_data_smithi173_74914_object85589:head# 2024-08-30T23:36:50.484 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb687448:::benchmark_data_smithi173_74914_object65080:head# 2024-08-30T23:36:50.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb694729:::benchmark_data_smithi173_74914_object75530:head# 2024-08-30T23:36:50.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb69d587:::benchmark_data_smithi173_74914_object94214:head# 2024-08-30T23:36:50.485 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb6b7838:::benchmark_data_smithi173_74914_object60113:head# 2024-08-30T23:36:50.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb6ca667:::benchmark_data_smithi173_74914_object76075:head# 2024-08-30T23:36:50.486 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb6d9978:::benchmark_data_smithi173_74914_object60678:head# 2024-08-30T23:36:50.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7135f9:::benchmark_data_smithi173_74914_object89187:head# 2024-08-30T23:36:50.487 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb72cf31:::benchmark_data_smithi173_74914_object55059:head# 2024-08-30T23:36:50.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb735073:::benchmark_data_smithi173_74914_object96430:head# 2024-08-30T23:36:50.488 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb75786d:::benchmark_data_smithi173_74914_object89693:head# 2024-08-30T23:36:50.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb75e4cd:::benchmark_data_smithi173_74914_object94418:head# 2024-08-30T23:36:50.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb77292d:::benchmark_data_smithi173_74914_object88013:head# 2024-08-30T23:36:50.489 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7d2676:::benchmark_data_smithi173_74914_object94365:head# 2024-08-30T23:36:50.490 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7d4b15:::benchmark_data_smithi173_74914_object89549:head# 2024-08-30T23:36:50.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:50.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7d88c2:::benchmark_data_smithi173_74914_object77131:head# 2024-08-30T23:36:50.491 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7e1c76:::benchmark_data_smithi173_74914_object57722:head# 2024-08-30T23:36:50.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7e2d8f:::benchmark_data_smithi173_74914_object64744:head# 2024-08-30T23:36:50.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb7f012c:::benchmark_data_smithi173_74914_object91104:head# 2024-08-30T23:36:50.492 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb803bc7:::benchmark_data_smithi173_74914_object68135:head# 2024-08-30T23:36:50.493 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb8209d1:::benchmark_data_smithi173_74914_object86566:head# 2024-08-30T23:36:50.494 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb824ae4:::benchmark_data_smithi173_74914_object89752:head# 2024-08-30T23:36:50.494 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb830e45:::benchmark_data_smithi173_74914_object97155:head# 2024-08-30T23:36:50.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb83173a:::benchmark_data_smithi173_74914_object59886:head# 2024-08-30T23:36:50.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb852dd3:::benchmark_data_smithi173_74914_object88323:head# 2024-08-30T23:36:50.495 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb88c4bd:::benchmark_data_smithi173_74914_object73525:head# 2024-08-30T23:36:50.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb89907f:::benchmark_data_smithi173_74914_object65496:head# 2024-08-30T23:36:50.496 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb89e5a6:::benchmark_data_smithi173_74914_object71791:head# 2024-08-30T23:36:50.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb8a1703:::benchmark_data_smithi173_74914_object59293:head# 2024-08-30T23:36:50.497 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb8ad9c4:::benchmark_data_smithi173_74914_object64837:head# 2024-08-30T23:36:50.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb8bb2fa:::benchmark_data_smithi173_74914_object68678:head# 2024-08-30T23:36:50.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb8cc3b9:::benchmark_data_smithi173_74914_object81302:head# 2024-08-30T23:36:50.498 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb8ed16d:::benchmark_data_smithi173_74914_object77985:head# 2024-08-30T23:36:50.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb90c6dc:::benchmark_data_smithi173_74914_object52835:head# 2024-08-30T23:36:50.499 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb9202f7:::benchmark_data_smithi173_74914_object74081:head# 2024-08-30T23:36:50.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb9713fb:::benchmark_data_smithi173_74914_object74726:head# 2024-08-30T23:36:50.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fb9d3511:::benchmark_data_smithi173_74914_object67987:head# 2024-08-30T23:36:50.500 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba00a84:::benchmark_data_smithi173_74914_object67243:head# 2024-08-30T23:36:50.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba245c8:::benchmark_data_smithi173_74914_object91787:head# 2024-08-30T23:36:50.501 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba359c2:::benchmark_data_smithi173_74914_object59991:head# 2024-08-30T23:36:50.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba3aeab:::benchmark_data_smithi173_74914_object78073:head# 2024-08-30T23:36:50.502 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba3c3af:::benchmark_data_smithi173_74914_object73740:head# 2024-08-30T23:36:50.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba4d4e9:::benchmark_data_smithi173_74914_object81617:head# 2024-08-30T23:36:50.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba525e3:::benchmark_data_smithi173_74914_object54902:head# 2024-08-30T23:36:50.503 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba64c8a:::benchmark_data_smithi173_74914_object90781:head# 2024-08-30T23:36:50.504 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba78266:::benchmark_data_smithi173_74914_object55604:head# 2024-08-30T23:36:50.504 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba8294f:::benchmark_data_smithi173_74914_object88311:head# 2024-08-30T23:36:50.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fba95bd5:::benchmark_data_smithi173_74914_object79296:head# 2024-08-30T23:36:50.505 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbacfd7e:::benchmark_data_smithi173_74914_object79938:head# 2024-08-30T23:36:50.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbade4bb:::benchmark_data_smithi173_74914_object84032:head# 2024-08-30T23:36:50.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbae6c82:::benchmark_data_smithi173_74914_object94068:head# 2024-08-30T23:36:50.506 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb0a10a:::benchmark_data_smithi173_74914_object76308:head# 2024-08-30T23:36:50.507 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb0b2e1:::benchmark_data_smithi173_74914_object89912:head# 2024-08-30T23:36:50.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb2d881:::benchmark_data_smithi173_74914_object88043:head# 2024-08-30T23:36:50.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb573b8:::benchmark_data_smithi173_74914_object72157:head# 2024-08-30T23:36:50.508 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb5c3b3:::benchmark_data_smithi173_74914_object89428:head# 2024-08-30T23:36:50.509 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb62035:::benchmark_data_smithi173_74914_object78923:head# 2024-08-30T23:36:50.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb672ae:::benchmark_data_smithi173_74914_object63189:head# 2024-08-30T23:36:50.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbb6a1bc:::benchmark_data_smithi173_74914_object94830:head# 2024-08-30T23:36:50.510 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbbbf7ce:::benchmark_data_smithi173_74914_object94886:head# 2024-08-30T23:36:50.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbbdca1c:::benchmark_data_smithi173_74914_object82410:head# 2024-08-30T23:36:50.511 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbbfe5b3:::benchmark_data_smithi173_74914_object94844:head# 2024-08-30T23:36:50.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbc11a5b:::benchmark_data_smithi173_74914_object70401:head# 2024-08-30T23:36:50.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbc7ceaf:::benchmark_data_smithi173_74914_object59144:head# 2024-08-30T23:36:50.512 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbc8703b:::benchmark_data_smithi173_74914_object91468:head# 2024-08-30T23:36:50.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbcf4229:::benchmark_data_smithi173_74914_object97208:head# 2024-08-30T23:36:50.513 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd24904:::benchmark_data_smithi173_74914_object77408:head# 2024-08-30T23:36:50.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd28a38:::benchmark_data_smithi173_74914_object77022:head# 2024-08-30T23:36:50.514 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd2eaab:::benchmark_data_smithi173_74914_object80336:head# 2024-08-30T23:36:50.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd4bc11:::benchmark_data_smithi173_74914_object91556:head# 2024-08-30T23:36:50.515 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd62ccb:::benchmark_data_smithi173_74914_object55577:head# 2024-08-30T23:36:50.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd6f901:::benchmark_data_smithi173_74914_object83204:head# 2024-08-30T23:36:50.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbd8ce4d:::benchmark_data_smithi173_74914_object57939:head# 2024-08-30T23:36:50.516 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbda6562:::benchmark_data_smithi173_74914_object79445:head# 2024-08-30T23:36:50.517 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbdca495:::benchmark_data_smithi173_74914_object81596:head# 2024-08-30T23:36:50.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:50.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbde738a:::benchmark_data_smithi173_74914_object62018:head# 2024-08-30T23:36:50.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe14532:::benchmark_data_smithi173_74914_object54771:head# 2024-08-30T23:36:50.518 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe18b48:::benchmark_data_smithi173_74914_object77128:head# 2024-08-30T23:36:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe1e352:::benchmark_data_smithi173_74914_object65719:head# 2024-08-30T23:36:50.520 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe32051:::benchmark_data_smithi173_74914_object88168:head# 2024-08-30T23:36:50.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:50.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe50436:::benchmark_data_smithi173_74914_object53847:head# 2024-08-30T23:36:50.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe686d7:::benchmark_data_smithi173_74914_object60015:head# 2024-08-30T23:36:50.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe7b4cd:::benchmark_data_smithi173_74914_object63514:head# 2024-08-30T23:36:50.521 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe847ea:::benchmark_data_smithi173_74914_object80737:head# 2024-08-30T23:36:50.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbe87358:::benchmark_data_smithi173_74914_object76387:head# 2024-08-30T23:36:50.522 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbeb9ec6:::benchmark_data_smithi173_74914_object87838:head# 2024-08-30T23:36:50.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbec09ca:::benchmark_data_smithi173_74914_object95519:head# 2024-08-30T23:36:50.523 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbec610e:::benchmark_data_smithi173_74914_object77373:head# 2024-08-30T23:36:50.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbf1dd42:::benchmark_data_smithi173_74914_object72830:head# 2024-08-30T23:36:50.524 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbf2fb1f:::benchmark_data_smithi173_74914_object68102:head# 2024-08-30T23:36:50.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbf61847:::benchmark_data_smithi173_74914_object81784:head# 2024-08-30T23:36:50.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbf8ae97:::benchmark_data_smithi173_74914_object94681:head# 2024-08-30T23:36:50.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbfac3af:::benchmark_data_smithi173_74914_object54387:head# 2024-08-30T23:36:50.525 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbfade79:::benchmark_data_smithi173_74914_object83937:head# 2024-08-30T23:36:50.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbfae5a7:::benchmark_data_smithi173_74914_object90550:head# 2024-08-30T23:36:50.526 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbfb7470:::benchmark_data_smithi173_74914_object53950:head# 2024-08-30T23:36:50.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbfdb9bf:::benchmark_data_smithi173_74914_object71629:head# 2024-08-30T23:36:50.527 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fbfe3bd6:::benchmark_data_smithi173_74914_object98091:head# 2024-08-30T23:36:50.528 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc007d7a:::benchmark_data_smithi173_74914_object83703:head# 2024-08-30T23:36:50.528 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc0282a0:::benchmark_data_smithi173_74914_object61438:head# 2024-08-30T23:36:50.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc02f1a1:::benchmark_data_smithi173_74914_object56013:head# 2024-08-30T23:36:50.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc03bd84:::benchmark_data_smithi173_74914_object80998:head# 2024-08-30T23:36:50.529 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc066610:::benchmark_data_smithi173_74914_object97019:head# 2024-08-30T23:36:50.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc089e34:::benchmark_data_smithi173_74914_object64185:head# 2024-08-30T23:36:50.530 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc09b9eb:::benchmark_data_smithi173_74914_object85941:head# 2024-08-30T23:36:50.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc09ee68:::benchmark_data_smithi173_74914_object57299:head# 2024-08-30T23:36:50.531 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc0d32c2:::benchmark_data_smithi173_74914_object81644:head# 2024-08-30T23:36:50.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc0dddf1:::benchmark_data_smithi173_74914_object79969:head# 2024-08-30T23:36:50.532 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc10cf0f:::benchmark_data_smithi173_74914_object66559:head# 2024-08-30T23:36:50.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc10e2b6:::benchmark_data_smithi173_74914_object55021:head# 2024-08-30T23:36:50.534 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc15a7d2:::benchmark_data_smithi173_74914_object85120:head# 2024-08-30T23:36:50.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc169127:::benchmark_data_smithi173_74914_object68115:head# 2024-08-30T23:36:50.535 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc172063:::benchmark_data_smithi173_74914_object64405:head# 2024-08-30T23:36:50.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc192d93:::benchmark_data_smithi173_74914_object89557:head# 2024-08-30T23:36:50.536 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc1a9220:::benchmark_data_smithi173_74914_object83819:head# 2024-08-30T23:36:50.537 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc1baa1f:::benchmark_data_smithi173_74914_object61828:head# 2024-08-30T23:36:50.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc1e30ea:::benchmark_data_smithi173_74914_object68212:head# 2024-08-30T23:36:50.539 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc22860a:::benchmark_data_smithi173_74914_object55400:head# 2024-08-30T23:36:50.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc22af45:::benchmark_data_smithi173_74914_object61775:head# 2024-08-30T23:36:50.540 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc233f11:::benchmark_data_smithi173_74914_object95460:head# 2024-08-30T23:36:50.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc24046f:::benchmark_data_smithi173_74914_object64732:head# 2024-08-30T23:36:50.541 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc2488a5:::benchmark_data_smithi173_74914_object90006:head# 2024-08-30T23:36:50.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc254796:::benchmark_data_smithi173_74914_object83048:head# 2024-08-30T23:36:50.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc25d788:::benchmark_data_smithi173_74914_object58049:head# 2024-08-30T23:36:50.542 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc27ce13:::benchmark_data_smithi173_74914_object74031:head# 2024-08-30T23:36:50.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc2b5cf0:::benchmark_data_smithi173_74914_object62575:head# 2024-08-30T23:36:50.543 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc2e35e2:::benchmark_data_smithi173_74914_object80823:head# 2024-08-30T23:36:50.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc2e9c42:::benchmark_data_smithi173_74914_object86811:head# 2024-08-30T23:36:50.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc2edfcd:::benchmark_data_smithi173_74914_object93728:head# 2024-08-30T23:36:50.544 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc2fa7c0:::benchmark_data_smithi173_74914_object67024:head# 2024-08-30T23:36:50.545 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc352524:::benchmark_data_smithi173_74914_object82952:head# 2024-08-30T23:36:50.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc37e61a:::benchmark_data_smithi173_74914_object81811:head# 2024-08-30T23:36:50.547 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc3e0df0:::benchmark_data_smithi173_74914_object80567:head# 2024-08-30T23:36:50.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc3ffc48:::benchmark_data_smithi173_74914_object73177:head# 2024-08-30T23:36:50.548 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4333c8:::benchmark_data_smithi173_74914_object57295:head# 2024-08-30T23:36:50.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc439f99:::benchmark_data_smithi173_74914_object75784:head# 2024-08-30T23:36:50.549 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc45310a:::benchmark_data_smithi173_74914_object88816:head# 2024-08-30T23:36:50.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4574bd:::benchmark_data_smithi173_74914_object97640:head# 2024-08-30T23:36:50.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc46b862:::benchmark_data_smithi173_74914_object90830:head# 2024-08-30T23:36:50.550 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc47704d:::benchmark_data_smithi173_74914_object52863:head# 2024-08-30T23:36:50.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc490de8:::benchmark_data_smithi173_74914_object73453:head# 2024-08-30T23:36:50.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc491678:::benchmark_data_smithi173_74914_object78816:head# 2024-08-30T23:36:50.551 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4bc0cf:::benchmark_data_smithi173_74914_object76930:head# 2024-08-30T23:36:50.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4c7db8:::benchmark_data_smithi173_74914_object83580:head# 2024-08-30T23:36:50.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4c9229:::benchmark_data_smithi173_74914_object55858:head# 2024-08-30T23:36:50.553 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4eaa7e:::benchmark_data_smithi173_74914_object66414:head# 2024-08-30T23:36:50.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc4effd3:::benchmark_data_smithi173_74914_object70866:head# 2024-08-30T23:36:50.554 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc5049e9:::benchmark_data_smithi173_74914_object63730:head# 2024-08-30T23:36:50.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc5123c5:::benchmark_data_smithi173_74914_object67239:head# 2024-08-30T23:36:50.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc52bec5:::benchmark_data_smithi173_74914_object96229:head# 2024-08-30T23:36:50.555 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc53e6d9:::benchmark_data_smithi173_74914_object69195:head# 2024-08-30T23:36:50.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc54c03c:::benchmark_data_smithi173_74914_object96942:head# 2024-08-30T23:36:50.556 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc562f42:::benchmark_data_smithi173_74914_object69512:head# 2024-08-30T23:36:50.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc57ae57:::benchmark_data_smithi173_74914_object78775:head# 2024-08-30T23:36:50.557 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc5a2685:::benchmark_data_smithi173_74914_object73629:head# 2024-08-30T23:36:50.558 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc5bbe4c:::benchmark_data_smithi173_74914_object75434:head# 2024-08-30T23:36:50.559 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc5d80a8:::benchmark_data_smithi173_74914_object97761:head# 2024-08-30T23:36:50.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc5db8b5:::benchmark_data_smithi173_74914_object80307:head# 2024-08-30T23:36:50.560 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc602e84:::benchmark_data_smithi173_74914_object56994:head# 2024-08-30T23:36:50.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc60f67e:::benchmark_data_smithi173_74914_object77270:head# 2024-08-30T23:36:50.561 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc6115b8:::benchmark_data_smithi173_74914_object93808:head# 2024-08-30T23:36:50.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc61809b:::benchmark_data_smithi173_74914_object62855:head# 2024-08-30T23:36:50.562 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc62a47f:::benchmark_data_smithi173_74914_object69243:head# 2024-08-30T23:36:50.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc632606:::benchmark_data_smithi173_74914_object56369:head# 2024-08-30T23:36:50.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc63282d:::benchmark_data_smithi173_74914_object56594:head# 2024-08-30T23:36:50.563 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc63d6ff:::benchmark_data_smithi173_74914_object62868:head# 2024-08-30T23:36:50.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc6514d5:::benchmark_data_smithi173_74914_object83071:head# 2024-08-30T23:36:50.564 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc651b59:::benchmark_data_smithi173_74914_object93966:head# 2024-08-30T23:36:50.565 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc662bd6:::benchmark_data_smithi173_74914_object61815:head# 2024-08-30T23:36:50.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc671a9c:::benchmark_data_smithi173_74914_object61857:head# 2024-08-30T23:36:50.567 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc677b13:::benchmark_data_smithi173_74914_object93703:head# 2024-08-30T23:36:50.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc698f2d:::benchmark_data_smithi173_74914_object87722:head# 2024-08-30T23:36:50.568 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc6e7917:::benchmark_data_smithi173_74914_object70254:head# 2024-08-30T23:36:50.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc6ee3aa:::benchmark_data_smithi173_74914_object58202:head# 2024-08-30T23:36:50.569 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc6f965c:::benchmark_data_smithi173_74914_object79094:head# 2024-08-30T23:36:50.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc70b84c:::benchmark_data_smithi173_74914_object65050:head# 2024-08-30T23:36:50.570 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc724878:::benchmark_data_smithi173_74914_object97914:head# 2024-08-30T23:36:50.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc726cb0:::benchmark_data_smithi173_74914_object74057:head# 2024-08-30T23:36:50.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc72d5d4:::benchmark_data_smithi173_74914_object84488:head# 2024-08-30T23:36:50.571 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc779310:::benchmark_data_smithi173_74914_object95644:head# 2024-08-30T23:36:50.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc7c5c55:::benchmark_data_smithi173_74914_object68733:head# 2024-08-30T23:36:50.572 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc7d48cb:::benchmark_data_smithi173_74914_object76847:head# 2024-08-30T23:36:50.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc80aeff:::benchmark_data_smithi173_74914_object96504:head# 2024-08-30T23:36:50.573 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc838ad3:::benchmark_data_smithi173_74914_object83224:head# 2024-08-30T23:36:50.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc86e58d:::benchmark_data_smithi173_74914_object55301:head# 2024-08-30T23:36:50.574 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc8a5b63:::benchmark_data_smithi173_74914_object66875:head# 2024-08-30T23:36:50.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc8ac37c:::benchmark_data_smithi173_74914_object95043:head# 2024-08-30T23:36:50.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc8d27c7:::benchmark_data_smithi173_74914_object98004:head# 2024-08-30T23:36:50.575 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc8d32ed:::benchmark_data_smithi173_74914_object87317:head# 2024-08-30T23:36:50.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc8d4436:::benchmark_data_smithi173_74914_object76511:head# 2024-08-30T23:36:50.576 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc8eb560:::benchmark_data_smithi173_74914_object74669:head# 2024-08-30T23:36:50.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc9053a6:::benchmark_data_smithi173_74914_object76373:head# 2024-08-30T23:36:50.577 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc929eed:::benchmark_data_smithi173_74914_object92610:head# 2024-08-30T23:36:50.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc932922:::benchmark_data_smithi173_74914_object66047:head# 2024-08-30T23:36:50.578 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc963ca1:::benchmark_data_smithi173_74914_object69017:head# 2024-08-30T23:36:50.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc966fc6:::benchmark_data_smithi173_74914_object91313:head# 2024-08-30T23:36:50.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc97c06f:::benchmark_data_smithi173_74914_object57432:head# 2024-08-30T23:36:50.579 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc997007:::benchmark_data_smithi173_74914_object67344:head# 2024-08-30T23:36:50.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc99e165:::benchmark_data_smithi173_74914_object92216:head# 2024-08-30T23:36:50.582 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc9aaebf:::benchmark_data_smithi173_74914_object61973:head# 2024-08-30T23:36:50.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc9af1db:::benchmark_data_smithi173_74914_object93461:head# 2024-08-30T23:36:50.583 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc9c7fe1:::benchmark_data_smithi173_74914_object74358:head# 2024-08-30T23:36:50.584 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fc9d69d9:::benchmark_data_smithi173_74914_object97747:head# 2024-08-30T23:36:50.586 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fca00b10:::benchmark_data_smithi173_74914_object78890:head# 2024-08-30T23:36:50.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fca1fef3:::benchmark_data_smithi173_74914_object80187:head# 2024-08-30T23:36:50.587 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fca4db41:::benchmark_data_smithi173_74914_object94573:head# 2024-08-30T23:36:50.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fca50c58:::benchmark_data_smithi173_74914_object81061:head# 2024-08-30T23:36:50.588 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fca7380a:::benchmark_data_smithi173_74914_object92159:head# 2024-08-30T23:36:50.589 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcaad56f:::benchmark_data_smithi173_74914_object55274:head# 2024-08-30T23:36:50.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcac0c48:::benchmark_data_smithi173_74914_object66600:head# 2024-08-30T23:36:50.591 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcade851:::benchmark_data_smithi173_74914_object86705:head# 2024-08-30T23:36:50.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcae01a5:::benchmark_data_smithi173_74914_object73468:head# 2024-08-30T23:36:50.592 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb126b9:::benchmark_data_smithi173_74914_object82869:head# 2024-08-30T23:36:50.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb15f9b:::benchmark_data_smithi173_74914_object62135:head# 2024-08-30T23:36:50.593 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb1da7e:::benchmark_data_smithi173_74914_object64862:head# 2024-08-30T23:36:50.595 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb21c78:::benchmark_data_smithi173_74914_object55501:head# 2024-08-30T23:36:50.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb2971d:::benchmark_data_smithi173_74914_object66931:head# 2024-08-30T23:36:50.596 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb36809:::benchmark_data_smithi173_74914_object81245:head# 2024-08-30T23:36:50.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcb6a156:::benchmark_data_smithi173_74914_object61912:head# 2024-08-30T23:36:50.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcbb1834:::benchmark_data_smithi173_74914_object86531:head# 2024-08-30T23:36:50.597 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcbfeb9d:::benchmark_data_smithi173_74914_object86716:head# 2024-08-30T23:36:50.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcc0dfe8:::benchmark_data_smithi173_74914_object83245:head# 2024-08-30T23:36:50.598 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcc35967:::benchmark_data_smithi173_74914_object57166:head# 2024-08-30T23:36:50.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcc737bb:::benchmark_data_smithi173_74914_object63596:head# 2024-08-30T23:36:50.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcc9351c:::benchmark_data_smithi173_74914_object92337:head# 2024-08-30T23:36:50.599 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcc9aa83:::benchmark_data_smithi173_74914_object54436:head# 2024-08-30T23:36:50.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fccf1320:::benchmark_data_smithi173_74914_object60626:head# 2024-08-30T23:36:50.600 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd3b056:::benchmark_data_smithi173_74914_object71698:head# 2024-08-30T23:36:50.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd64456:::benchmark_data_smithi173_74914_object63904:head# 2024-08-30T23:36:50.601 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd66ada:::benchmark_data_smithi173_74914_object73282:head# 2024-08-30T23:36:50.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd6ceee:::benchmark_data_smithi173_74914_object58585:head# 2024-08-30T23:36:50.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd8c748:::benchmark_data_smithi173_74914_object89367:head# 2024-08-30T23:36:50.602 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd919e9:::benchmark_data_smithi173_74914_object95989:head# 2024-08-30T23:36:50.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd952a9:::benchmark_data_smithi173_74914_object89612:head# 2024-08-30T23:36:50.603 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcd97510:::benchmark_data_smithi173_74914_object57983:head# 2024-08-30T23:36:50.605 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcdc6bf4:::benchmark_data_smithi173_74914_object75899:head# 2024-08-30T23:36:50.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcdc75b4:::benchmark_data_smithi173_74914_object73034:head# 2024-08-30T23:36:50.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcde5c84:::benchmark_data_smithi173_74914_object68696:head# 2024-08-30T23:36:50.606 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcde7c63:::benchmark_data_smithi173_74914_object86128:head# 2024-08-30T23:36:50.607 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fce488c1:::benchmark_data_smithi173_74914_object97148:head# 2024-08-30T23:36:50.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fce5636f:::benchmark_data_smithi173_74914_object52808:head# 2024-08-30T23:36:50.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fce6842f:::benchmark_data_smithi173_74914_object53254:head# 2024-08-30T23:36:50.608 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fce7e2e9:::benchmark_data_smithi173_74914_object81044:head# 2024-08-30T23:36:50.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fce851e9:::benchmark_data_smithi173_74914_object67922:head# 2024-08-30T23:36:50.609 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fce86a0c:::benchmark_data_smithi173_74914_object84720:head# 2024-08-30T23:36:50.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fceb6bef:::benchmark_data_smithi173_74914_object83968:head# 2024-08-30T23:36:50.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcec4d5d:::benchmark_data_smithi173_74914_object75668:head# 2024-08-30T23:36:50.610 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcec6c69:::benchmark_data_smithi173_74914_object78061:head# 2024-08-30T23:36:50.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fceef0cd:::benchmark_data_smithi173_74914_object54476:head# 2024-08-30T23:36:50.611 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf1a8ec:::benchmark_data_smithi173_74914_object55398:head# 2024-08-30T23:36:50.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf1cdcc:::benchmark_data_smithi173_74914_object67519:head# 2024-08-30T23:36:50.612 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf25eb3:::benchmark_data_smithi173_74914_object83892:head# 2024-08-30T23:36:50.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf5530f:::benchmark_data_smithi173_74914_object56267:head# 2024-08-30T23:36:50.613 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf63337:::benchmark_data_smithi173_74914_object83181:head# 2024-08-30T23:36:50.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf893f8:::benchmark_data_smithi173_74914_object63556:head# 2024-08-30T23:36:50.614 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcf936ce:::benchmark_data_smithi173_74914_object96066:head# 2024-08-30T23:36:50.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcfb287c:::benchmark_data_smithi173_74914_object77171:head# 2024-08-30T23:36:50.617 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcfbd40a:::benchmark_data_smithi173_74914_object79884:head# 2024-08-30T23:36:50.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcfcc105:::benchmark_data_smithi173_74914_object77190:head# 2024-08-30T23:36:50.618 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcfd1207:::benchmark_data_smithi173_74914_object57211:head# 2024-08-30T23:36:50.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fcffdfe4:::benchmark_data_smithi173_74914_object87640:head# 2024-08-30T23:36:50.619 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd007fcc:::benchmark_data_smithi173_74914_object95996:head# 2024-08-30T23:36:50.620 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd01c108:::benchmark_data_smithi173_74914_object78232:head# 2024-08-30T23:36:50.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:50.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd026e58:::benchmark_data_smithi173_74914_object75834:head# 2024-08-30T23:36:50.622 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd04d939:::benchmark_data_smithi173_74914_object75765:head# 2024-08-30T23:36:50.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd08e99b:::benchmark_data_smithi173_74914_object61554:head# 2024-08-30T23:36:50.623 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd0957eb:::benchmark_data_smithi173_74914_object71213:head# 2024-08-30T23:36:50.624 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd0db87d:::benchmark_data_smithi173_74914_object91983:head# 2024-08-30T23:36:50.625 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd10053e:::benchmark_data_smithi173_74914_object58499:head# 2024-08-30T23:36:50.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd10960f:::benchmark_data_smithi173_74914_object62499:head# 2024-08-30T23:36:50.626 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd115d09:::benchmark_data_smithi173_74914_object78950:head# 2024-08-30T23:36:50.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd116c44:::benchmark_data_smithi173_74914_object82651:head# 2024-08-30T23:36:50.627 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd11fbf4:::benchmark_data_smithi173_74914_object64395:head# 2024-08-30T23:36:50.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd12570e:::benchmark_data_smithi173_74914_object53815:head# 2024-08-30T23:36:50.628 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd13f605:::benchmark_data_smithi173_74914_object81957:head# 2024-08-30T23:36:50.630 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd1707a7:::benchmark_data_smithi173_74914_object55196:head# 2024-08-30T23:36:50.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd17806f:::benchmark_data_smithi173_74914_object66130:head# 2024-08-30T23:36:50.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd1879fc:::benchmark_data_smithi173_74914_object75553:head# 2024-08-30T23:36:50.631 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd1964fd:::benchmark_data_smithi173_74914_object83447:head# 2024-08-30T23:36:50.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd19da22:::benchmark_data_smithi173_74914_object66720:head# 2024-08-30T23:36:50.632 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd1cbcf1:::benchmark_data_smithi173_74914_object87911:head# 2024-08-30T23:36:50.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd1ea92b:::benchmark_data_smithi173_74914_object58254:head# 2024-08-30T23:36:50.633 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd1f423c:::benchmark_data_smithi173_74914_object81516:head# 2024-08-30T23:36:50.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd20da7a:::benchmark_data_smithi173_74914_object79644:head# 2024-08-30T23:36:50.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd214a6f:::benchmark_data_smithi173_74914_object87116:head# 2024-08-30T23:36:50.634 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd22bee2:::benchmark_data_smithi173_74914_object58763:head# 2024-08-30T23:36:50.635 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd22ef16:::benchmark_data_smithi173_74914_object87360:head# 2024-08-30T23:36:50.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd235f38:::benchmark_data_smithi173_74914_object63990:head# 2024-08-30T23:36:50.636 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd279bfb:::benchmark_data_smithi173_74914_object85209:head# 2024-08-30T23:36:50.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd29197a:::benchmark_data_smithi173_74914_object87869:head# 2024-08-30T23:36:50.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd2ac3b8:::benchmark_data_smithi173_74914_object76588:head# 2024-08-30T23:36:50.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd2ba837:::benchmark_data_smithi173_74914_object78719:head# 2024-08-30T23:36:50.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd2c525e:::benchmark_data_smithi173_74914_object72540:head# 2024-08-30T23:36:50.638 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd2c7f34:::benchmark_data_smithi173_74914_object60473:head# 2024-08-30T23:36:50.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd2fa872:::benchmark_data_smithi173_74914_object76110:head# 2024-08-30T23:36:50.639 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd30555d:::benchmark_data_smithi173_74914_object68530:head# 2024-08-30T23:36:50.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd325407:::benchmark_data_smithi173_74914_object62851:head# 2024-08-30T23:36:50.640 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd32c84d:::benchmark_data_smithi173_74914_object56004:head# 2024-08-30T23:36:50.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd36991f:::benchmark_data_smithi173_74914_object77761:head# 2024-08-30T23:36:50.641 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd36a04a:::benchmark_data_smithi173_74914_object80053:head# 2024-08-30T23:36:50.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd383348:::benchmark_data_smithi173_74914_object75843:head# 2024-08-30T23:36:50.642 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd3a3d2d:::benchmark_data_smithi173_74914_object58671:head# 2024-08-30T23:36:50.643 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd3a4e79:::benchmark_data_smithi173_74914_object59654:head# 2024-08-30T23:36:50.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd3c5d35:::benchmark_data_smithi173_74914_object79114:head# 2024-08-30T23:36:50.644 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd3dc615:::benchmark_data_smithi173_74914_object65695:head# 2024-08-30T23:36:50.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd3dee1b:::benchmark_data_smithi173_74914_object90960:head# 2024-08-30T23:36:50.645 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd457f1e:::benchmark_data_smithi173_74914_object82938:head# 2024-08-30T23:36:50.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd45988e:::benchmark_data_smithi173_74914_object75550:head# 2024-08-30T23:36:50.646 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd459f71:::benchmark_data_smithi173_74914_object57196:head# 2024-08-30T23:36:50.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd460e50:::benchmark_data_smithi173_74914_object58706:head# 2024-08-30T23:36:50.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd463812:::benchmark_data_smithi173_74914_object72047:head# 2024-08-30T23:36:50.648 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd4660e8:::benchmark_data_smithi173_74914_object76575:head# 2024-08-30T23:36:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:50.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:50.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd47d329:::benchmark_data_smithi173_74914_object90194:head# 2024-08-30T23:36:50.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd4a0c3c:::benchmark_data_smithi173_74914_object83691:head# 2024-08-30T23:36:50.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd4a6799:::benchmark_data_smithi173_74914_object71570:head# 2024-08-30T23:36:50.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd4afdbb:::benchmark_data_smithi173_74914_object66672:head# 2024-08-30T23:36:50.655 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd4da81e:::benchmark_data_smithi173_74914_object61987:head# 2024-08-30T23:36:50.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd4fcaa0:::benchmark_data_smithi173_74914_object80609:head# 2024-08-30T23:36:50.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd52a6d8:::benchmark_data_smithi173_74914_object88546:head# 2024-08-30T23:36:50.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd52e5db:::benchmark_data_smithi173_74914_object68532:head# 2024-08-30T23:36:50.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd53d838:::benchmark_data_smithi173_74914_object56312:head# 2024-08-30T23:36:50.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd54bca1:::benchmark_data_smithi173_74914_object89961:head# 2024-08-30T23:36:50.656 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd552f01:::benchmark_data_smithi173_74914_object76992:head# 2024-08-30T23:36:50.657 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd575c8f:::benchmark_data_smithi173_74914_object53465:head# 2024-08-30T23:36:50.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd5944f6:::benchmark_data_smithi173_74914_object62600:head# 2024-08-30T23:36:50.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd5b1e15:::benchmark_data_smithi173_74914_object54804:head# 2024-08-30T23:36:50.658 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd5f4439:::benchmark_data_smithi173_74914_object69417:head# 2024-08-30T23:36:50.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd623a32:::benchmark_data_smithi173_74914_object72104:head# 2024-08-30T23:36:50.660 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd6311cf:::benchmark_data_smithi173_74914_object60380:head# 2024-08-30T23:36:50.661 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd647813:::benchmark_data_smithi173_74914_object66109:head# 2024-08-30T23:36:50.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd65acea:::benchmark_data_smithi173_74914_object73154:head# 2024-08-30T23:36:50.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd66ba22:::benchmark_data_smithi173_74914_object55214:head# 2024-08-30T23:36:50.662 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd671508:::benchmark_data_smithi173_74914_object88799:head# 2024-08-30T23:36:50.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd69a996:::benchmark_data_smithi173_74914_object94118:head# 2024-08-30T23:36:50.663 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd6a0af3:::benchmark_data_smithi173_74914_object71534:head# 2024-08-30T23:36:50.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd6dff93:::benchmark_data_smithi173_74914_object74599:head# 2024-08-30T23:36:50.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd6e7789:::benchmark_data_smithi173_74914_object78319:head# 2024-08-30T23:36:50.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd6f46f9:::benchmark_data_smithi173_74914_object64204:head# 2024-08-30T23:36:50.664 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd6f623d:::benchmark_data_smithi173_74914_object73212:head# 2024-08-30T23:36:50.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd7020fd:::benchmark_data_smithi173_74914_object53138:head# 2024-08-30T23:36:50.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd736059:::benchmark_data_smithi173_74914_object67666:head# 2024-08-30T23:36:50.666 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd74e092:::benchmark_data_smithi173_74914_object64660:head# 2024-08-30T23:36:50.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd75ecac:::benchmark_data_smithi173_74914_object53696:head# 2024-08-30T23:36:50.667 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd7626e1:::benchmark_data_smithi173_74914_object52967:head# 2024-08-30T23:36:50.668 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd773b5e:::benchmark_data_smithi173_74914_object60358:head# 2024-08-30T23:36:50.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd7a7c2f:::benchmark_data_smithi173_74914_object93083:head# 2024-08-30T23:36:50.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd7d3c90:::benchmark_data_smithi173_74914_object90867:head# 2024-08-30T23:36:50.669 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd8012d1:::benchmark_data_smithi173_74914_object57412:head# 2024-08-30T23:36:50.670 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd82478c:::benchmark_data_smithi173_74914_object53848:head# 2024-08-30T23:36:50.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd82a10e:::benchmark_data_smithi173_74914_object54275:head# 2024-08-30T23:36:50.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd82ea63:::benchmark_data_smithi173_74914_object89144:head# 2024-08-30T23:36:50.671 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd834f8e:::benchmark_data_smithi173_74914_object69472:head# 2024-08-30T23:36:50.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd8441e2:::benchmark_data_smithi173_74914_object79742:head# 2024-08-30T23:36:50.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd855335:::benchmark_data_smithi173_74914_object67879:head# 2024-08-30T23:36:50.673 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd85c773:::benchmark_data_smithi173_74914_object96158:head# 2024-08-30T23:36:50.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd86d084:::benchmark_data_smithi173_74914_object58283:head# 2024-08-30T23:36:50.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd86e5be:::benchmark_data_smithi173_74914_object76801:head# 2024-08-30T23:36:50.674 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd894eed:::benchmark_data_smithi173_74914_object80985:head# 2024-08-30T23:36:50.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd896f20:::benchmark_data_smithi173_74914_object53894:head# 2024-08-30T23:36:50.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd8bebcf:::benchmark_data_smithi173_74914_object54425:head# 2024-08-30T23:36:50.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd8c2b23:::benchmark_data_smithi173_74914_object52989:head# 2024-08-30T23:36:50.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd8c429e:::benchmark_data_smithi173_74914_object62099:head# 2024-08-30T23:36:50.676 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd8d029d:::benchmark_data_smithi173_74914_object80837:head# 2024-08-30T23:36:50.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd906d67:::benchmark_data_smithi173_74914_object93986:head# 2024-08-30T23:36:50.677 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd91d4d2:::benchmark_data_smithi173_74914_object92270:head# 2024-08-30T23:36:50.678 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd92b80f:::benchmark_data_smithi173_74914_object85077:head# 2024-08-30T23:36:50.680 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd932ff4:::benchmark_data_smithi173_74914_object67947:head# 2024-08-30T23:36:50.682 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd935d93:::benchmark_data_smithi173_74914_object96574:head# 2024-08-30T23:36:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:36:50.683 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:36:50.684 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd93f117:::benchmark_data_smithi173_74914_object71031:head# 2024-08-30T23:36:50.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd966b79:::benchmark_data_smithi173_74914_object76916:head# 2024-08-30T23:36:50.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd99a562:::benchmark_data_smithi173_74914_object57816:head# 2024-08-30T23:36:50.685 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd99c65f:::benchmark_data_smithi173_74914_object65503:head# 2024-08-30T23:36:50.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd9a541f:::benchmark_data_smithi173_74914_object57018:head# 2024-08-30T23:36:50.686 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd9a82d9:::benchmark_data_smithi173_74914_object66128:head# 2024-08-30T23:36:50.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd9dacff:::benchmark_data_smithi173_74914_object66946:head# 2024-08-30T23:36:50.687 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd9e2707:::benchmark_data_smithi173_74914_object82248:head# 2024-08-30T23:36:50.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fd9f1e33:::benchmark_data_smithi173_74914_object59584:head# 2024-08-30T23:36:50.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda2075b:::benchmark_data_smithi173_74914_object85437:head# 2024-08-30T23:36:50.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda23411:::benchmark_data_smithi173_74914_object76714:head# 2024-08-30T23:36:50.688 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda36808:::benchmark_data_smithi173_74914_object79758:head# 2024-08-30T23:36:50.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda84b24:::benchmark_data_smithi173_74914_object84188:head# 2024-08-30T23:36:50.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda8cefc:::benchmark_data_smithi173_74914_object75234:head# 2024-08-30T23:36:50.690 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda9645a:::benchmark_data_smithi173_74914_object69396:head# 2024-08-30T23:36:50.691 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda9acfc:::benchmark_data_smithi173_74914_object72335:head# 2024-08-30T23:36:50.692 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fda9b0f5:::benchmark_data_smithi173_74914_object94759:head# 2024-08-30T23:36:50.692 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdab1814:::benchmark_data_smithi173_74914_object88180:head# 2024-08-30T23:36:50.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb0c27d:::benchmark_data_smithi173_74914_object60487:head# 2024-08-30T23:36:50.693 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb1319f:::benchmark_data_smithi173_74914_object85482:head# 2024-08-30T23:36:50.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb5302b:::benchmark_data_smithi173_74914_object80993:head# 2024-08-30T23:36:50.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb58589:::benchmark_data_smithi173_74914_object76807:head# 2024-08-30T23:36:50.694 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb607d3:::benchmark_data_smithi173_74914_object67157:head# 2024-08-30T23:36:50.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb8039d:::benchmark_data_smithi173_74914_object76893:head# 2024-08-30T23:36:50.695 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb87dd1:::benchmark_data_smithi173_74914_object75758:head# 2024-08-30T23:36:50.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdb9aec7:::benchmark_data_smithi173_74914_object56729:head# 2024-08-30T23:36:50.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbb9d13:::benchmark_data_smithi173_74914_object64785:head# 2024-08-30T23:36:50.696 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbc5337:::benchmark_data_smithi173_74914_object92289:head# 2024-08-30T23:36:50.697 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbcdb23:::benchmark_data_smithi173_74914_object80996:head# 2024-08-30T23:36:50.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbd5012:::benchmark_data_smithi173_74914_object78996:head# 2024-08-30T23:36:50.698 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbd5bc3:::benchmark_data_smithi173_74914_object66736:head# 2024-08-30T23:36:50.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbe8718:::benchmark_data_smithi173_74914_object94133:head# 2024-08-30T23:36:50.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdbf997c:::benchmark_data_smithi173_74914_object67956:head# 2024-08-30T23:36:50.699 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdc2edf9:::benchmark_data_smithi173_74914_object92650:head# 2024-08-30T23:36:50.700 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdc32571:::benchmark_data_smithi173_74914_object64740:head# 2024-08-30T23:36:50.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdc71b94:::benchmark_data_smithi173_74914_object62966:head# 2024-08-30T23:36:50.701 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdcb6df2:::benchmark_data_smithi173_74914_object92179:head# 2024-08-30T23:36:50.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdcb7055:::benchmark_data_smithi173_74914_object89833:head# 2024-08-30T23:36:50.702 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdcec965:::benchmark_data_smithi173_74914_object94050:head# 2024-08-30T23:36:50.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd177ce:::benchmark_data_smithi173_74914_object97542:head# 2024-08-30T23:36:50.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd36a23:::benchmark_data_smithi173_74914_object96975:head# 2024-08-30T23:36:50.703 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd513da:::benchmark_data_smithi173_74914_object69123:head# 2024-08-30T23:36:50.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd65dd0:::benchmark_data_smithi173_74914_object57915:head# 2024-08-30T23:36:50.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd8c59c:::benchmark_data_smithi173_74914_object60372:head# 2024-08-30T23:36:50.704 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd8cc50:::benchmark_data_smithi173_74914_object79410:head# 2024-08-30T23:36:50.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd9407c:::benchmark_data_smithi173_74914_object62164:head# 2024-08-30T23:36:50.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdd9e3a7:::benchmark_data_smithi173_74914_object87380:head# 2024-08-30T23:36:50.705 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fddaa823:::benchmark_data_smithi173_74914_object56396:head# 2024-08-30T23:36:50.706 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fddb3cfb:::benchmark_data_smithi173_74914_object67475:head# 2024-08-30T23:36:50.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fddc10d8:::benchmark_data_smithi173_74914_object79836:head# 2024-08-30T23:36:50.707 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fddc7219:::benchmark_data_smithi173_74914_object96772:head# 2024-08-30T23:36:50.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fddf2ddf:::benchmark_data_smithi173_74914_object85352:head# 2024-08-30T23:36:50.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fde16151:::benchmark_data_smithi173_74914_object82586:head# 2024-08-30T23:36:50.708 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fde3de36:::benchmark_data_smithi173_74914_object80616:head# 2024-08-30T23:36:50.711 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fde717df:::benchmark_data_smithi173_74914_object78502:head# 2024-08-30T23:36:50.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fde9991d:::benchmark_data_smithi173_74914_object78827:head# 2024-08-30T23:36:50.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fde9d12e:::benchmark_data_smithi173_74914_object65478:head# 2024-08-30T23:36:50.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdec3904:::benchmark_data_smithi173_74914_object62758:head# 2024-08-30T23:36:50.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdeec741:::benchmark_data_smithi173_74914_object89266:head# 2024-08-30T23:36:50.712 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdef64be:::benchmark_data_smithi173_74914_object78036:head# 2024-08-30T23:36:50.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdefbeb8:::benchmark_data_smithi173_74914_object60243:head# 2024-08-30T23:36:50.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf2fb54:::benchmark_data_smithi173_74914_object71815:head# 2024-08-30T23:36:50.713 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf33d51:::benchmark_data_smithi173_74914_object68840:head# 2024-08-30T23:36:50.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf3aa42:::benchmark_data_smithi173_74914_object70269:head# 2024-08-30T23:36:50.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf3eea8:::benchmark_data_smithi173_74914_object75427:head# 2024-08-30T23:36:50.715 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf5f9a3:::benchmark_data_smithi173_74914_object76968:head# 2024-08-30T23:36:50.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf638e3:::benchmark_data_smithi173_74914_object57141:head# 2024-08-30T23:36:50.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf6ad90:::benchmark_data_smithi173_74914_object72188:head# 2024-08-30T23:36:50.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf6b82b:::benchmark_data_smithi173_74914_object75463:head# 2024-08-30T23:36:50.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdf7ad92:::benchmark_data_smithi173_74914_object73598:head# 2024-08-30T23:36:50.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfabdf6:::benchmark_data_smithi173_74914_object56419:head# 2024-08-30T23:36:50.716 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfb18a3:::benchmark_data_smithi173_74914_object54102:head# 2024-08-30T23:36:50.718 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfb3f2d:::benchmark_data_smithi173_74914_object76072:head# 2024-08-30T23:36:50.718 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfb62dd:::benchmark_data_smithi173_74914_object86254:head# 2024-08-30T23:36:50.719 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfb7fe5:::benchmark_data_smithi173_74914_object80508:head# 2024-08-30T23:36:50.719 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfd8dd6:::benchmark_data_smithi173_74914_object63446:head# 2024-08-30T23:36:50.719 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfe0c97:::benchmark_data_smithi173_74914_object73420:head# 2024-08-30T23:36:50.720 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fdfed93b:::benchmark_data_smithi173_74914_object53727:head# 2024-08-30T23:36:50.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe027a92:::benchmark_data_smithi173_74914_object62402:head# 2024-08-30T23:36:50.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe047e2e:::benchmark_data_smithi173_74914_object92560:head# 2024-08-30T23:36:50.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe051fab:::benchmark_data_smithi173_74914_object75371:head# 2024-08-30T23:36:50.722 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe06d79e:::benchmark_data_smithi173_74914_object67250:head# 2024-08-30T23:36:50.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe079dfc:::benchmark_data_smithi173_74914_object60588:head# 2024-08-30T23:36:50.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe08f8cd:::benchmark_data_smithi173_74914_object68915:head# 2024-08-30T23:36:50.726 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe093672:::benchmark_data_smithi173_74914_object58565:head# 2024-08-30T23:36:50.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe0aa82d:::benchmark_data_smithi173_74914_object92681:head# 2024-08-30T23:36:50.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe0aadfa:::benchmark_data_smithi173_74914_object92152:head# 2024-08-30T23:36:50.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe0aceee:::benchmark_data_smithi173_74914_object95782:head# 2024-08-30T23:36:50.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe0b218c:::benchmark_data_smithi173_74914_object96267:head# 2024-08-30T23:36:50.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe0b91b0:::benchmark_data_smithi173_74914_object83361:head# 2024-08-30T23:36:50.727 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe0c8100:::benchmark_data_smithi173_74914_object67406:head# 2024-08-30T23:36:50.728 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe1001d1:::benchmark_data_smithi173_74914_object76960:head# 2024-08-30T23:36:50.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe117a03:::benchmark_data_smithi173_74914_object71492:head# 2024-08-30T23:36:50.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe11e089:::benchmark_data_smithi173_74914_object72786:head# 2024-08-30T23:36:50.729 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe132759:::benchmark_data_smithi173_74914_object67054:head# 2024-08-30T23:36:50.730 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe13f9cb:::benchmark_data_smithi173_74914_object82727:head# 2024-08-30T23:36:50.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe15770c:::benchmark_data_smithi173_74914_object75538:head# 2024-08-30T23:36:50.731 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe165038:::benchmark_data_smithi173_74914_object83573:head# 2024-08-30T23:36:50.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe18b8e4:::benchmark_data_smithi173_74914_object74784:head# 2024-08-30T23:36:50.732 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe1b51dd:::benchmark_data_smithi173_74914_object82298:head# 2024-08-30T23:36:50.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe1b68b0:::benchmark_data_smithi173_74914_object73767:head# 2024-08-30T23:36:50.733 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe1c429f:::benchmark_data_smithi173_74914_object55946:head# 2024-08-30T23:36:50.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe1cdbfd:::benchmark_data_smithi173_74914_object84064:head# 2024-08-30T23:36:50.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe205a76:::benchmark_data_smithi173_74914_object69213:head# 2024-08-30T23:36:50.734 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe223219:::benchmark_data_smithi173_74914_object71902:head# 2024-08-30T23:36:50.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe239b3f:::benchmark_data_smithi173_74914_object82057:head# 2024-08-30T23:36:50.735 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe23d738:::benchmark_data_smithi173_74914_object95829:head# 2024-08-30T23:36:50.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe23df97:::benchmark_data_smithi173_74914_object92768:head# 2024-08-30T23:36:50.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe278347:::benchmark_data_smithi173_74914_object62731:head# 2024-08-30T23:36:50.736 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe27c905:::benchmark_data_smithi173_74914_object52930:head# 2024-08-30T23:36:50.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe280c10:::benchmark_data_smithi173_74914_object88097:head# 2024-08-30T23:36:50.737 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe281e3a:::benchmark_data_smithi173_74914_object95328:head# 2024-08-30T23:36:50.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe28df59:::benchmark_data_smithi173_74914_object76102:head# 2024-08-30T23:36:50.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe293e0e:::benchmark_data_smithi173_74914_object69353:head# 2024-08-30T23:36:50.738 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe2b38b7:::benchmark_data_smithi173_74914_object53273:head# 2024-08-30T23:36:50.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe2d022c:::benchmark_data_smithi173_74914_object63014:head# 2024-08-30T23:36:50.739 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe302b0c:::benchmark_data_smithi173_74914_object60775:head# 2024-08-30T23:36:50.740 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe33db52:::benchmark_data_smithi173_74914_object65290:head# 2024-08-30T23:36:50.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe3419ed:::benchmark_data_smithi173_74914_object62698:head# 2024-08-30T23:36:50.742 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe374cd5:::benchmark_data_smithi173_74914_object60264:head# 2024-08-30T23:36:50.743 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe39861e:::benchmark_data_smithi173_74914_object54979:head# 2024-08-30T23:36:50.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe3cab97:::benchmark_data_smithi173_74914_object78079:head# 2024-08-30T23:36:50.744 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe3f5800:::benchmark_data_smithi173_74914_object76463:head# 2024-08-30T23:36:50.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe403869:::benchmark_data_smithi173_74914_object71491:head# 2024-08-30T23:36:50.745 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe40a04d:::benchmark_data_smithi173_74914_object85888:head# 2024-08-30T23:36:50.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe40c2da:::benchmark_data_smithi173_74914_object66606:head# 2024-08-30T23:36:50.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe4197a5:::benchmark_data_smithi173_74914_object91209:head# 2024-08-30T23:36:50.746 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe449425:::benchmark_data_smithi173_74914_object66008:head# 2024-08-30T23:36:50.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe45448b:::benchmark_data_smithi173_74914_object61853:head# 2024-08-30T23:36:50.747 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe463096:::benchmark_data_smithi173_74914_object55443:head# 2024-08-30T23:36:50.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe463a43:::benchmark_data_smithi173_74914_object58537:head# 2024-08-30T23:36:50.748 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe464417:::benchmark_data_smithi173_74914_object70961:head# 2024-08-30T23:36:50.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe46fa24:::benchmark_data_smithi173_74914_object83098:head# 2024-08-30T23:36:50.749 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe485e25:::benchmark_data_smithi173_74914_object86919:head# 2024-08-30T23:36:50.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe48a99d:::benchmark_data_smithi173_74914_object70673:head# 2024-08-30T23:36:50.750 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe49991f:::benchmark_data_smithi173_74914_object63548:head# 2024-08-30T23:36:50.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe49d884:::benchmark_data_smithi173_74914_object71572:head# 2024-08-30T23:36:50.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe49f1f0:::benchmark_data_smithi173_74914_object96786:head# 2024-08-30T23:36:50.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe4d8cb5:::benchmark_data_smithi173_74914_object53052:head# 2024-08-30T23:36:50.751 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe4ffceb:::benchmark_data_smithi173_74914_object90735:head# 2024-08-30T23:36:50.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe53f681:::benchmark_data_smithi173_74914_object86070:head# 2024-08-30T23:36:50.752 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe55da61:::benchmark_data_smithi173_74914_object53536:head# 2024-08-30T23:36:50.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:50.754 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe55e4c6:::benchmark_data_smithi173_74914_object93060:head# 2024-08-30T23:36:50.755 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe57181d:::benchmark_data_smithi173_74914_object94335:head# 2024-08-30T23:36:50.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe5970f4:::benchmark_data_smithi173_74914_object94119:head# 2024-08-30T23:36:50.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe59bdc3:::benchmark_data_smithi173_74914_object67287:head# 2024-08-30T23:36:50.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe5b6c0d:::benchmark_data_smithi173_74914_object95623:head# 2024-08-30T23:36:50.757 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe5c53da:::benchmark_data_smithi173_74914_object70974:head# 2024-08-30T23:36:50.759 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe5cd580:::benchmark_data_smithi173_74914_object55509:head# 2024-08-30T23:36:50.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe5d865d:::benchmark_data_smithi173_74914_object72964:head# 2024-08-30T23:36:50.760 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe626600:::benchmark_data_smithi173_74914_object80169:head# 2024-08-30T23:36:50.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe665d71:::benchmark_data_smithi173_74914_object74481:head# 2024-08-30T23:36:50.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe6abd42:::benchmark_data_smithi173_74914_object86387:head# 2024-08-30T23:36:50.761 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe6dd34e:::benchmark_data_smithi173_74914_object95582:head# 2024-08-30T23:36:50.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe6e2a1e:::benchmark_data_smithi173_74914_object54182:head# 2024-08-30T23:36:50.762 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe6eb744:::benchmark_data_smithi173_74914_object73679:head# 2024-08-30T23:36:50.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe6ffa5c:::benchmark_data_smithi173_74914_object57669:head# 2024-08-30T23:36:50.763 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe7238ae:::benchmark_data_smithi173_74914_object87433:head# 2024-08-30T23:36:50.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe728c3b:::benchmark_data_smithi173_74914_object87661:head# 2024-08-30T23:36:50.764 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe73ed9b:::benchmark_data_smithi173_74914_object83471:head# 2024-08-30T23:36:50.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe75b7b0:::benchmark_data_smithi173_74914_object58563:head# 2024-08-30T23:36:50.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe78934a:::benchmark_data_smithi173_74914_object68089:head# 2024-08-30T23:36:50.765 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe79ad98:::benchmark_data_smithi173_74914_object64551:head# 2024-08-30T23:36:50.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe79d185:::benchmark_data_smithi173_74914_object98040:head# 2024-08-30T23:36:50.766 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe79e3f5:::benchmark_data_smithi173_74914_object82564:head# 2024-08-30T23:36:50.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe7a501d:::benchmark_data_smithi173_74914_object87308:head# 2024-08-30T23:36:50.767 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe7e9379:::benchmark_data_smithi173_74914_object87026:head# 2024-08-30T23:36:50.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe7f2d24:::benchmark_data_smithi173_74914_object69092:head# 2024-08-30T23:36:50.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe81b11f:::benchmark_data_smithi173_74914_object60286:head# 2024-08-30T23:36:50.768 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe84b1bc:::benchmark_data_smithi173_74914_object55777:head# 2024-08-30T23:36:50.769 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe856042:::benchmark_data_smithi173_74914_object64263:head# 2024-08-30T23:36:50.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe8807b4:::benchmark_data_smithi173_74914_object84825:head# 2024-08-30T23:36:50.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe8cac9a:::benchmark_data_smithi173_74914_object87104:head# 2024-08-30T23:36:50.770 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe8f2b50:::benchmark_data_smithi173_74914_object97323:head# 2024-08-30T23:36:50.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe92317c:::benchmark_data_smithi173_74914_object59305:head# 2024-08-30T23:36:50.771 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9247b5:::benchmark_data_smithi173_74914_object74714:head# 2024-08-30T23:36:50.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe932876:::benchmark_data_smithi173_74914_object73501:head# 2024-08-30T23:36:50.772 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe932c1c:::benchmark_data_smithi173_74914_object75201:head# 2024-08-30T23:36:50.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe964bd4:::benchmark_data_smithi173_74914_object79370:head# 2024-08-30T23:36:50.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe97204b:::benchmark_data_smithi173_74914_object81126:head# 2024-08-30T23:36:50.773 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9926b3:::benchmark_data_smithi173_74914_object83997:head# 2024-08-30T23:36:50.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe998bef:::benchmark_data_smithi173_74914_object57867:head# 2024-08-30T23:36:50.774 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9b027c:::benchmark_data_smithi173_74914_object63409:head# 2024-08-30T23:36:50.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9b09ca:::benchmark_data_smithi173_74914_object84256:head# 2024-08-30T23:36:50.775 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9b7d4e:::benchmark_data_smithi173_74914_object95573:head# 2024-08-30T23:36:50.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9c70d8:::benchmark_data_smithi173_74914_object67963:head# 2024-08-30T23:36:50.776 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9e82d4:::benchmark_data_smithi173_74914_object80686:head# 2024-08-30T23:36:50.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fe9f805e:::benchmark_data_smithi173_74914_object78262:head# 2024-08-30T23:36:50.777 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:50.777 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fea01f75:::benchmark_data_smithi173_74914_object59354:head# 2024-08-30T23:36:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:50.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fea0b2bc:::benchmark_data_smithi173_74914_object73450:head# 2024-08-30T23:36:50.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fea1a7a7:::benchmark_data_smithi173_74914_object65358:head# 2024-08-30T23:36:50.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fea53c32:::benchmark_data_smithi173_74914_object92199:head# 2024-08-30T23:36:50.779 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fea93451:::benchmark_data_smithi173_74914_object82719:head# 2024-08-30T23:36:50.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fea9d9a2:::benchmark_data_smithi173_74914_object67066:head# 2024-08-30T23:36:50.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feaa1105:::benchmark_data_smithi173_74914_object85897:head# 2024-08-30T23:36:50.780 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feaa1e8a:::benchmark_data_smithi173_74914_object75082:head# 2024-08-30T23:36:50.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feaaed0c:::benchmark_data_smithi173_74914_object83089:head# 2024-08-30T23:36:50.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feae227e:::benchmark_data_smithi173_74914_object59285:head# 2024-08-30T23:36:50.781 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feb295dd:::benchmark_data_smithi173_74914_object72589:head# 2024-08-30T23:36:50.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feb5b1fd:::benchmark_data_smithi173_74914_object88990:head# 2024-08-30T23:36:50.782 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feb7f974:::benchmark_data_smithi173_74914_object66782:head# 2024-08-30T23:36:50.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:febd0ae1:::benchmark_data_smithi173_74914_object64137:head# 2024-08-30T23:36:50.783 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec0aa44:::benchmark_data_smithi173_74914_object90579:head# 2024-08-30T23:36:50.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec0c465:::benchmark_data_smithi173_74914_object60770:head# 2024-08-30T23:36:50.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec20b57:::benchmark_data_smithi173_74914_object72474:head# 2024-08-30T23:36:50.784 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec4303e:::benchmark_data_smithi173_74914_object58122:head# 2024-08-30T23:36:50.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec7d0e5:::benchmark_data_smithi173_74914_object73807:head# 2024-08-30T23:36:50.785 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec97a48:::benchmark_data_smithi173_74914_object70728:head# 2024-08-30T23:36:50.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fec98ee7:::benchmark_data_smithi173_74914_object80734:head# 2024-08-30T23:36:50.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fecda303:::benchmark_data_smithi173_74914_object59086:head# 2024-08-30T23:36:50.786 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fece6dc5:::benchmark_data_smithi173_74914_object94089:head# 2024-08-30T23:36:50.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fed0b121:::benchmark_data_smithi173_74914_object97219:head# 2024-08-30T23:36:50.787 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fed4cdd3:::benchmark_data_smithi173_74914_object63413:head# 2024-08-30T23:36:50.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fed6d0ed:::benchmark_data_smithi173_74914_object83994:head# 2024-08-30T23:36:50.788 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fed6dd10:::benchmark_data_smithi173_74914_object62712:head# 2024-08-30T23:36:50.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fed74290:::benchmark_data_smithi173_74914_object58938:head# 2024-08-30T23:36:50.789 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fedaa12a:::benchmark_data_smithi173_74914_object66141:head# 2024-08-30T23:36:50.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fedb1ebb:::benchmark_data_smithi173_74914_object56036:head# 2024-08-30T23:36:50.790 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fedbc954:::benchmark_data_smithi173_74914_object59926:head# 2024-08-30T23:36:50.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fedc35df:::benchmark_data_smithi173_74914_object95229:head# 2024-08-30T23:36:50.791 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fedc9380:::benchmark_data_smithi173_74914_object76832:head# 2024-08-30T23:36:50.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee0ce0d:::benchmark_data_smithi173_74914_object83401:head# 2024-08-30T23:36:50.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee2003a:::benchmark_data_smithi173_74914_object71579:head# 2024-08-30T23:36:50.792 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee21295:::benchmark_data_smithi173_74914_object63793:head# 2024-08-30T23:36:50.793 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee239cc:::benchmark_data_smithi173_74914_object67491:head# 2024-08-30T23:36:50.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee24f79:::benchmark_data_smithi173_74914_object90079:head# 2024-08-30T23:36:50.794 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee3ce19:::benchmark_data_smithi173_74914_object69868:head# 2024-08-30T23:36:50.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee41ed5:::benchmark_data_smithi173_74914_object57105:head# 2024-08-30T23:36:50.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee65544:::benchmark_data_smithi173_74914_object93177:head# 2024-08-30T23:36:50.795 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee7916e:::benchmark_data_smithi173_74914_object79017:head# 2024-08-30T23:36:50.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee83231:::benchmark_data_smithi173_74914_object62703:head# 2024-08-30T23:36:50.796 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee837b4:::benchmark_data_smithi173_74914_object94393:head# 2024-08-30T23:36:50.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fee9f61d:::benchmark_data_smithi173_74914_object75539:head# 2024-08-30T23:36:50.797 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feea92ec:::benchmark_data_smithi173_74914_object82114:head# 2024-08-30T23:36:50.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feeda329:::benchmark_data_smithi173_74914_object55549:head# 2024-08-30T23:36:50.798 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feee7ae3:::benchmark_data_smithi173_74914_object90294:head# 2024-08-30T23:36:50.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fef03003:::benchmark_data_smithi173_74914_object58393:head# 2024-08-30T23:36:50.799 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fef0b6ea:::benchmark_data_smithi173_74914_object55625:head# 2024-08-30T23:36:50.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fef3f9fd:::benchmark_data_smithi173_74914_object94382:head# 2024-08-30T23:36:50.800 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fef505e8:::benchmark_data_smithi173_74914_object98238:head# 2024-08-30T23:36:50.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fef5cbb7:::benchmark_data_smithi173_74914_object81971:head# 2024-08-30T23:36:50.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fef83cfe:::benchmark_data_smithi173_74914_object95420:head# 2024-08-30T23:36:50.801 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fefa3cf0:::benchmark_data_smithi173_74914_object72972:head# 2024-08-30T23:36:50.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fefa88f7:::benchmark_data_smithi173_74914_object59940:head# 2024-08-30T23:36:50.802 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fefab923:::benchmark_data_smithi173_74914_object54294:head# 2024-08-30T23:36:50.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fefde6d9:::benchmark_data_smithi173_74914_object75632:head# 2024-08-30T23:36:50.803 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:feffc1ca:::benchmark_data_smithi173_74914_object93940:head# 2024-08-30T23:36:50.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff0279e2:::benchmark_data_smithi173_74914_object58731:head# 2024-08-30T23:36:50.804 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff02f067:::benchmark_data_smithi173_74914_object65688:head# 2024-08-30T23:36:50.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff037f68:::benchmark_data_smithi173_74914_object54344:head# 2024-08-30T23:36:50.805 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff058334:::benchmark_data_smithi173_74914_object60312:head# 2024-08-30T23:36:50.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff070511:::benchmark_data_smithi173_74914_object80781:head# 2024-08-30T23:36:50.806 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff079a47:::benchmark_data_smithi173_74914_object78524:head# 2024-08-30T23:36:50.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff07d2e8:::benchmark_data_smithi173_74914_object78135:head# 2024-08-30T23:36:50.807 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff091fa4:::benchmark_data_smithi173_74914_object74234:head# 2024-08-30T23:36:50.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff0a601b:::benchmark_data_smithi173_74914_object86729:head# 2024-08-30T23:36:50.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff0b6051:::benchmark_data_smithi173_74914_object57705:head# 2024-08-30T23:36:50.808 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff0c36aa:::benchmark_data_smithi173_74914_object83512:head# 2024-08-30T23:36:50.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff11fd9a:::benchmark_data_smithi173_74914_object80595:head# 2024-08-30T23:36:50.809 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff17eaf0:::benchmark_data_smithi173_74914_object80266:head# 2024-08-30T23:36:50.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff1879bd:::benchmark_data_smithi173_74914_object81139:head# 2024-08-30T23:36:50.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff189bd8:::benchmark_data_smithi173_74914_object94949:head# 2024-08-30T23:36:50.810 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff18d78f:::benchmark_data_smithi173_74914_object55518:head# 2024-08-30T23:36:50.811 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff1ac7c7:::benchmark_data_smithi173_74914_object89171:head# 2024-08-30T23:36:50.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff1b1728:::benchmark_data_smithi173_74914_object77991:head# 2024-08-30T23:36:50.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff1c0275:::benchmark_data_smithi173_74914_object75401:head# 2024-08-30T23:36:50.812 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff1dbe8e:::benchmark_data_smithi173_74914_object62257:head# 2024-08-30T23:36:50.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff1e07b8:::benchmark_data_smithi173_74914_object70940:head# 2024-08-30T23:36:50.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff21e197:::benchmark_data_smithi173_74914_object88034:head# 2024-08-30T23:36:50.813 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff2273ef:::benchmark_data_smithi173_74914_object91192:head# 2024-08-30T23:36:50.814 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff23efda:::benchmark_data_smithi173_74914_object73669:head# 2024-08-30T23:36:50.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff24cd8a:::benchmark_data_smithi173_74914_object82457:head# 2024-08-30T23:36:50.815 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff2628f3:::benchmark_data_smithi173_74914_object72233:head# 2024-08-30T23:36:50.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff27750b:::benchmark_data_smithi173_74914_object67136:head# 2024-08-30T23:36:50.816 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff2be05f:::benchmark_data_smithi173_74914_object83478:head# 2024-08-30T23:36:50.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff2d07f7:::benchmark_data_smithi173_74914_object81136:head# 2024-08-30T23:36:50.817 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff2e1b1a:::benchmark_data_smithi173_74914_object80020:head# 2024-08-30T23:36:50.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff377dcd:::benchmark_data_smithi173_74914_object89081:head# 2024-08-30T23:36:50.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff3797bf:::benchmark_data_smithi173_74914_object82818:head# 2024-08-30T23:36:50.818 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff3e2596:::benchmark_data_smithi173_74914_object83437:head# 2024-08-30T23:36:50.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff3f5012:::benchmark_data_smithi173_74914_object77156:head# 2024-08-30T23:36:50.819 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff4077d4:::benchmark_data_smithi173_74914_object92602:head# 2024-08-30T23:36:50.820 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff4092cd:::benchmark_data_smithi173_74914_object86440:head# 2024-08-30T23:36:50.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff41f59d:::benchmark_data_smithi173_74914_object93244:head# 2024-08-30T23:36:50.821 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff420711:::benchmark_data_smithi173_74914_object75036:head# 2024-08-30T23:36:50.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff431387:::benchmark_data_smithi173_74914_object72069:head# 2024-08-30T23:36:50.822 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff431aee:::benchmark_data_smithi173_74914_object56559:head# 2024-08-30T23:36:50.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff437a4c:::benchmark_data_smithi173_74914_object66017:head# 2024-08-30T23:36:50.823 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff43a777:::benchmark_data_smithi173_74914_object70572:head# 2024-08-30T23:36:50.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff44eda6:::benchmark_data_smithi173_74914_object64187:head# 2024-08-30T23:36:50.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff476938:::benchmark_data_smithi173_74914_object98081:head# 2024-08-30T23:36:50.824 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff491258:::benchmark_data_smithi173_74914_object56670:head# 2024-08-30T23:36:50.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff493c50:::benchmark_data_smithi173_74914_object94486:head# 2024-08-30T23:36:50.825 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff495124:::benchmark_data_smithi173_74914_object86867:head# 2024-08-30T23:36:50.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff4bc0d0:::benchmark_data_smithi173_74914_object61115:head# 2024-08-30T23:36:50.826 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff4d1a9f:::benchmark_data_smithi173_74914_object72793:head# 2024-08-30T23:36:50.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff4de42b:::benchmark_data_smithi173_74914_object72775:head# 2024-08-30T23:36:50.827 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff504451:::benchmark_data_smithi173_74914_object71450:head# 2024-08-30T23:36:50.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff50886e:::benchmark_data_smithi173_74914_object70714:head# 2024-08-30T23:36:50.828 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff51a43c:::benchmark_data_smithi173_74914_object70648:head# 2024-08-30T23:36:50.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff520c6e:::benchmark_data_smithi173_74914_object67178:head# 2024-08-30T23:36:50.829 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff521a83:::benchmark_data_smithi173_74914_object92242:head# 2024-08-30T23:36:50.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff524ad0:::benchmark_data_smithi173_74914_object70736:head# 2024-08-30T23:36:50.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff52554e:::benchmark_data_smithi173_74914_object90908:head# 2024-08-30T23:36:50.830 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff537860:::benchmark_data_smithi173_74914_object82326:head# 2024-08-30T23:36:50.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff5406bf:::benchmark_data_smithi173_74914_object70760:head# 2024-08-30T23:36:50.831 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff55ebe4:::benchmark_data_smithi173_74914_object74168:head# 2024-08-30T23:36:50.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff59ef2d:::benchmark_data_smithi173_74914_object62631:head# 2024-08-30T23:36:50.832 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff5c5740:::benchmark_data_smithi173_74914_object59445:head# 2024-08-30T23:36:50.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:50 smithi106 ceph-mon[16603]: 4.1 scrub ok 2024-08-30T23:36:50.833 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:50 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10693/86757 objects degraded (12.325%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:50.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff5d08e9:::benchmark_data_smithi173_74914_object72646:head# 2024-08-30T23:36:50.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff601459:::benchmark_data_smithi173_74914_object78324:head# 2024-08-30T23:36:50.833 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff60ced1:::benchmark_data_smithi173_74914_object53398:head# 2024-08-30T23:36:50.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff6317c2:::benchmark_data_smithi173_74914_object90669:head# 2024-08-30T23:36:50.834 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff645874:::benchmark_data_smithi173_74914_object95611:head# 2024-08-30T23:36:50.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff65d1a1:::benchmark_data_smithi173_74914_object60046:head# 2024-08-30T23:36:50.835 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff67e4f5:::benchmark_data_smithi173_74914_object93923:head# 2024-08-30T23:36:50.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff68aea6:::benchmark_data_smithi173_74914_object77860:head# 2024-08-30T23:36:50.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff698475:::benchmark_data_smithi173_74914_object71990:head# 2024-08-30T23:36:50.836 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff6bd63e:::benchmark_data_smithi173_74914_object78381:head# 2024-08-30T23:36:50.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff6cea18:::benchmark_data_smithi173_74914_object72340:head# 2024-08-30T23:36:50.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff6d0416:::benchmark_data_smithi173_74914_object73736:head# 2024-08-30T23:36:50.837 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff6f1878:::benchmark_data_smithi173_74914_object80134:head# 2024-08-30T23:36:50.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff729524:::benchmark_data_smithi173_74914_object70270:head# 2024-08-30T23:36:50.838 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff74064e:::benchmark_data_smithi173_74914_object92812:head# 2024-08-30T23:36:50.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff74f80f:::benchmark_data_smithi173_74914_object55912:head# 2024-08-30T23:36:50.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff785b9c:::benchmark_data_smithi173_74914_object90799:head# 2024-08-30T23:36:50.839 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff7aada1:::benchmark_data_smithi173_74914_object86873:head# 2024-08-30T23:36:50.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff7b86be:::benchmark_data_smithi173_74914_object78016:head# 2024-08-30T23:36:50.840 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff7d8058:::benchmark_data_smithi173_74914_object60722:head# 2024-08-30T23:36:50.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8157f6:::benchmark_data_smithi173_74914_object82199:head# 2024-08-30T23:36:50.841 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8252f6:::benchmark_data_smithi173_74914_object66812:head# 2024-08-30T23:36:50.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8586f3:::benchmark_data_smithi173_74914_object61085:head# 2024-08-30T23:36:50.842 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff85b45a:::benchmark_data_smithi173_74914_object82460:head# 2024-08-30T23:36:50.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8662b8:::benchmark_data_smithi173_74914_object96191:head# 2024-08-30T23:36:50.843 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff86643b:::benchmark_data_smithi173_74914_object59275:head# 2024-08-30T23:36:50.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff86e9d2:::benchmark_data_smithi173_74914_object83018:head# 2024-08-30T23:36:50.844 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff878698:::benchmark_data_smithi173_74914_object80629:head# 2024-08-30T23:36:50.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8932e8:::benchmark_data_smithi173_74914_object66835:head# 2024-08-30T23:36:50.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8a202c:::benchmark_data_smithi173_74914_object66647:head# 2024-08-30T23:36:50.845 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8b59d2:::benchmark_data_smithi173_74914_object89910:head# 2024-08-30T23:36:50.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8b8302:::benchmark_data_smithi173_74914_object71400:head# 2024-08-30T23:36:50.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff8fc29b:::benchmark_data_smithi173_74914_object59487:head# 2024-08-30T23:36:50.846 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff90188c:::benchmark_data_smithi173_74914_object70891:head# 2024-08-30T23:36:50.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff939f3d:::benchmark_data_smithi173_74914_object57265:head# 2024-08-30T23:36:50.847 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff94d9a6:::benchmark_data_smithi173_74914_object90163:head# 2024-08-30T23:36:50.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff94f53a:::benchmark_data_smithi173_74914_object53920:head# 2024-08-30T23:36:50.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff9739c7:::benchmark_data_smithi173_74914_object56436:head# 2024-08-30T23:36:50.848 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff97a9e2:::benchmark_data_smithi173_74914_object97850:head# 2024-08-30T23:36:50.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff980bfe:::benchmark_data_smithi173_74914_object77763:head# 2024-08-30T23:36:50.849 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff9ad1e1:::benchmark_data_smithi173_74914_object90662:head# 2024-08-30T23:36:50.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff9be41d:::benchmark_data_smithi173_74914_object82927:head# 2024-08-30T23:36:50.850 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff9c46ec:::benchmark_data_smithi173_74914_object79551:head# 2024-08-30T23:36:50.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff9f5f7c:::benchmark_data_smithi173_74914_object75526:head# 2024-08-30T23:36:50.851 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ff9fae67:::benchmark_data_smithi173_74914_object58713:head# 2024-08-30T23:36:50.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa015b0:::benchmark_data_smithi173_74914_object92345:head# 2024-08-30T23:36:50.852 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa1b5fd:::benchmark_data_smithi173_74914_object96162:head# 2024-08-30T23:36:50.853 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa2d026:::benchmark_data_smithi173_74914_object73310:head# 2024-08-30T23:36:50.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa586db:::benchmark_data_smithi173_74914_object53263:head# 2024-08-30T23:36:50.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa720a5:::benchmark_data_smithi173_74914_object98237:head# 2024-08-30T23:36:50.856 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa79112:::benchmark_data_smithi173_74914_object57592:head# 2024-08-30T23:36:50.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffa90558:::benchmark_data_smithi173_74914_object86730:head# 2024-08-30T23:36:50.857 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffab797e:::benchmark_data_smithi173_74914_object65029:head# 2024-08-30T23:36:50.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffacf267:::benchmark_data_smithi173_74914_object63615:head# 2024-08-30T23:36:50.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffad08fa:::benchmark_data_smithi173_74914_object65171:head# 2024-08-30T23:36:50.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffae2a11:::benchmark_data_smithi173_74914_object97326:head# 2024-08-30T23:36:50.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffaffd08:::benchmark_data_smithi173_74914_object58598:head# 2024-08-30T23:36:50.858 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb0ad8a:::benchmark_data_smithi173_74914_object61277:head# 2024-08-30T23:36:50.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb0b0ae:::benchmark_data_smithi173_74914_object58844:head# 2024-08-30T23:36:50.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb35d5a:::benchmark_data_smithi173_74914_object64339:head# 2024-08-30T23:36:50.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb41d34:::benchmark_data_smithi173_74914_object73471:head# 2024-08-30T23:36:50.859 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb6075e:::benchmark_data_smithi173_74914_object60512:head# 2024-08-30T23:36:50.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb70584:::benchmark_data_smithi173_74914_object67768:head# 2024-08-30T23:36:50.860 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb7a804:::benchmark_data_smithi173_74914_object88147:head# 2024-08-30T23:36:50.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb8213f:::benchmark_data_smithi173_74914_object55872:head# 2024-08-30T23:36:50.861 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb9ac09:::benchmark_data_smithi173_74914_object52946:head# 2024-08-30T23:36:50.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffb9c6c9:::benchmark_data_smithi173_74914_object66856:head# 2024-08-30T23:36:50.862 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffbb013e:::benchmark_data_smithi173_74914_object82800:head# 2024-08-30T23:36:50.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffbc9bba:::benchmark_data_smithi173_74914_object90019:head# 2024-08-30T23:36:50.863 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffbcfe8b:::benchmark_data_smithi173_74914_object76749:head# 2024-08-30T23:36:50.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffbe53ff:::benchmark_data_smithi173_74914_object59504:head# 2024-08-30T23:36:50.864 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffc11248:::benchmark_data_smithi173_74914_object71733:head# 2024-08-30T23:36:50.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffc18022:::benchmark_data_smithi173_74914_object57819:head# 2024-08-30T23:36:50.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffc4024f:::benchmark_data_smithi173_74914_object90936:head# 2024-08-30T23:36:50.865 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffc99d89:::benchmark_data_smithi173_74914_object70451:head# 2024-08-30T23:36:50.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffc9e1e6:::benchmark_data_smithi173_74914_object76213:head# 2024-08-30T23:36:50.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffcd7c18:::benchmark_data_smithi173_74914_object82561:head# 2024-08-30T23:36:50.866 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffcf2df5:::benchmark_data_smithi173_74914_object64429:head# 2024-08-30T23:36:50.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffcf5ebd:::benchmark_data_smithi173_74914_object58457:head# 2024-08-30T23:36:50.867 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd02607:::benchmark_data_smithi173_74914_object97382:head# 2024-08-30T23:36:50.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd066e9:::benchmark_data_smithi173_74914_object93716:head# 2024-08-30T23:36:50.868 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd125da:::benchmark_data_smithi173_74914_object57099:head# 2024-08-30T23:36:50.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd51fc1:::benchmark_data_smithi173_74914_object85660:head# 2024-08-30T23:36:50.869 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd73f63:::benchmark_data_smithi173_74914_object76176:head# 2024-08-30T23:36:50.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd7dd6f:::benchmark_data_smithi173_74914_object73770:head# 2024-08-30T23:36:50.870 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffd8df6d:::benchmark_data_smithi173_74914_object55160:head# 2024-08-30T23:36:50.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffdaabe7:::benchmark_data_smithi173_74914_object57003:head# 2024-08-30T23:36:50.871 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe0df6e:::benchmark_data_smithi173_74914_object60827:head# 2024-08-30T23:36:50.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe1a186:::benchmark_data_smithi173_74914_object55944:head# 2024-08-30T23:36:50.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe37aab:::benchmark_data_smithi173_74914_object88718:head# 2024-08-30T23:36:50.872 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe3873a:::benchmark_data_smithi173_74914_object82316:head# 2024-08-30T23:36:50.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe48c5c:::benchmark_data_smithi173_74914_object88881:head# 2024-08-30T23:36:50.873 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe5bbcc:::benchmark_data_smithi173_74914_object96351:head# 2024-08-30T23:36:50.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe6cf62:::benchmark_data_smithi173_74914_object65102:head# 2024-08-30T23:36:50.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffe8c04b:::benchmark_data_smithi173_74914_object72526:head# 2024-08-30T23:36:50.874 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffea5099:::benchmark_data_smithi173_74914_object75774:head# 2024-08-30T23:36:50.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffed5813:::benchmark_data_smithi173_74914_object68974:head# 2024-08-30T23:36:50.875 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffedb42a:::benchmark_data_smithi173_74914_object95553:head# 2024-08-30T23:36:50.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffee0921:::benchmark_data_smithi173_74914_object65728:head# 2024-08-30T23:36:50.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:ffef12ef:::benchmark_data_smithi173_74914_object56209:head# 2024-08-30T23:36:50.876 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff044f3:::benchmark_data_smithi173_74914_object67131:head# 2024-08-30T23:36:50.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff0a487:::benchmark_data_smithi173_74914_object84421:head# 2024-08-30T23:36:50.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff19876:::benchmark_data_smithi173_74914_object54098:head# 2024-08-30T23:36:50.877 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff7011e:::benchmark_data_smithi173_74914_object67821:head# 2024-08-30T23:36:50.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff907d7:::benchmark_data_smithi173_74914_object62020:head# 2024-08-30T23:36:50.878 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff955fa:::benchmark_data_smithi173_74914_object68921:head# 2024-08-30T23:36:50.879 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:50.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fff9c6fa:::benchmark_data_smithi173_74914_object69489:head# 2024-08-30T23:36:50.879 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fffd9315:::benchmark_data_smithi173_74914_object73236:head# 2024-08-30T23:36:50.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fffe2766:::benchmark_data_smithi173_74914_object81819:head# 2024-08-30T23:36:50.880 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:fffe6b19:::benchmark_data_smithi173_74914_object71179:head# 2024-08-30T23:36:50.884 INFO:teuthology.orchestra.run.smithi106.stderr:Export successful 2024-08-30T23:36:50.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:50.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:51.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:51.030 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:50 smithi136 ceph-mon[20207]: 4.1 scrub ok 2024-08-30T23:36:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:50 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10693/86757 objects degraded (12.325%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:51.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:50 smithi173 ceph-mon[20554]: 4.1 scrub ok 2024-08-30T23:36:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:50 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10693/86757 objects degraded (12.325%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:51.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:51.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:51.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:51.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:51.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:51.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:51.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:51.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:51.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:51.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:36:51.711 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:36:51.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:51.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:51.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:51.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:51.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:51.996 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:36:52.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:36:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:36:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:51 smithi136 ceph-mon[20207]: pgmap v905: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 9579/77553 objects degraded (12.352%) 2024-08-30T23:36:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3782788329' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:51 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:52.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:36:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:36:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:51 smithi173 ceph-mon[20554]: pgmap v905: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 9579/77553 objects degraded (12.352%) 2024-08-30T23:36:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3782788329' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:51 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:36:52.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:36:52.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:51 smithi106 ceph-mon[16603]: pgmap v905: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.6 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 9579/77553 objects degraded (12.352%) 2024-08-30T23:36:52.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3782788329' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:52.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:51 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:36:52.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:52.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:52.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:52.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:52.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:52.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:52.532 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:52.532 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:52.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:52.579 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:52.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:52.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:36:52.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:52.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:52.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:52 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:52 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:36:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:52 smithi106 ceph-mon[16603]: osdmap e376: 12 total, 11 up, 12 in 2024-08-30T23:36:52.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:52.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:52.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:52 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:52 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:36:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:52 smithi136 ceph-mon[20207]: osdmap e376: 12 total, 11 up, 12 in 2024-08-30T23:36:53.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:52 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:52 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:36:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:52 smithi173 ceph-mon[20554]: osdmap e376: 12 total, 11 up, 12 in 2024-08-30T23:36:53.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:53.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:53.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:53.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:53.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:53.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:53.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:53.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:53.601 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:53.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:36:53.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.741 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:53.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:53.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 4.f 2024-08-30T23:36:53.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:53.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:53 smithi106 ceph-mon[16603]: pgmap v907: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 9069/73530 objects degraded (12.334%) 2024-08-30T23:36:53.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:53.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:53.958 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:36:53.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:54.010 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:54.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:53 smithi136 ceph-mon[20207]: pgmap v907: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 9069/73530 objects degraded (12.334%) 2024-08-30T23:36:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:53 smithi173 ceph-mon[20554]: pgmap v907: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 896 op/s; 9069/73530 objects degraded (12.334%) 2024-08-30T23:36:54.110 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:36:54.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:54.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:54.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:54.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:54.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:54.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:54.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:54.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:54.567 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:36:54.568 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:36:54.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:36:54.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:36:54.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:36:54.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:36:54.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:36:54.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:36:54.632 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:36:54.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:36:54.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:36:54.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:36:54.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:54.658 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:36:54.659 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:36:54.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:54.690 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:36:54.690 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:36:54.729 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:36:54.729 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:36:54.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:54.767 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:36:54.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:36:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:54.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:54.811 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:36:54.811 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:36:54.811 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1198856775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:54.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:36:54.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:36:54.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:36:54.891 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:36:54.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:54.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:54.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:55.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1198856775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:55.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1198856775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:36:55.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:55.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:55.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:55.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:55.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:55.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:55.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:55.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:55.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:55.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:55.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:55.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:55.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:36:55.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:55.822 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:36:55.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:36:55.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:55.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:55.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:56.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:55 smithi106 ceph-mon[16603]: pgmap v908: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s; 8239/68433 objects degraded (12.040%) 2024-08-30T23:36:56.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:55 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9069/73530 objects degraded (12.334%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:56.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:55 smithi136 ceph-mon[20207]: pgmap v908: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s; 8239/68433 objects degraded (12.040%) 2024-08-30T23:36:56.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:55 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9069/73530 objects degraded (12.334%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:56.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:36:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:55 smithi173 ceph-mon[20554]: pgmap v908: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s; 8239/68433 objects degraded (12.040%) 2024-08-30T23:36:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:55 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9069/73530 objects degraded (12.334%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:36:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:56.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:36:56.127 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:36:56.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:36:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.220 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:56.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:36:56.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:56.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:56.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:56.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:56.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:56.495 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:56.518 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:36:56.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:56.601 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:56.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:56.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:56.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:56.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:56.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:56.976 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:36:56.976 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:36:56.976 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:36:56.976 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:36:56.976 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:36:56.981 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:56.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:36:57.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:57.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:57.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:57.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:57.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:57.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:57.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:57.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:57.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:57.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:57.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:57.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:36:57.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:36:57.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:36:57.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:57.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:36:57.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:57.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:57.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:57.996 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:36:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:58.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:58.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:58.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:58.175 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:36:58.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:58.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:58.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:58.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:58.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:58.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:58.549 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:36:58.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:58 smithi106 ceph-mon[16603]: pgmap v909: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 7959/63252 objects degraded (12.583%) 2024-08-30T23:36:58.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/173492902' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:58.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:96989 2.f 2:f906d66e:::benchmark_data_smithi173_47195_object96988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:36:58.564 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 570.41918516099997, 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84153836199999998, 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 96989 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142691+0000", 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142692+0000", 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:36:58.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142718+0000", 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5548999999999999e-05 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142729+0000", 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1126e-05 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.142736+0000", 2024-08-30T23:36:58.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1840000000000002e-06 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148635+0000", 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058993520000000001 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148767+0000", 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013146699999999999 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.148916+0000", 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149083 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152326+0000", 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034105860000000002 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.152337+0000", 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1100999999999999e-05 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984109+0000", 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83177168400000001 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984180+0000", 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0542000000000003e-05 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984229+0000", 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9808000000000002e-05 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.570 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.24460.0:96989 2.f e146/97 ondisk, result = 0)", 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 570.41057780200003, 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83293943299999995, 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:36:58.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151298+0000", 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151299+0000", 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5600000000000005e-07 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.151308+0000", 2024-08-30T23:36:58.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984021+0000", 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83271241100000004 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984029+0000", 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9379999999999992e-06 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984091+0000", 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.211e-05 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984101+0000", 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0368e-05 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:28.984238+0000", 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013670199999999999 2024-08-30T23:36:58.574 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100665 2.6 2:6a73c426:::benchmark_data_smithi173_47195_object3657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 565.218061315, 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21497229600000001, 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:58.575 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100665 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343815+0000", 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343816+0000", 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.494e-06 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343824+0000", 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2570000000000001e-06 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343841+0000", 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7107000000000001e-05 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343850+0000", 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4299999999999995e-06 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.343921+0000", 2024-08-30T23:36:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0264000000000004e-05 2024-08-30T23:36:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344020+0000", 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9296999999999995e-05 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.344100+0000", 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0465999999999998e-05 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.346504+0000", 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002403723 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347743+0000", 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001238549 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558717+0000", 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21097406299999999 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558756+0000", 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9728000000000003e-05 2024-08-30T23:36:58.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.558787+0000", 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0917999999999998e-05 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:100667 2.6 2:6833bcab:::benchmark_data_smithi173_47195_object3659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 565.21409482900003, 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27402247099999999, 2024-08-30T23:36:58.582 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 100667 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347781+0000", 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347782+0000", 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347787+0000", 2024-08-30T23:36:58.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3039999999999998e-06 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347798+0000", 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0834e-05 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347803+0000", 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.242e-06 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347840+0000", 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7630999999999999e-05 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347918+0000", 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7778e-05 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.347989+0000", 2024-08-30T23:36:58.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0988999999999994e-05 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.349336+0000", 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013472670000000001 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.351224+0000", 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018879960000000001 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621737+0000", 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27051204000000001 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621775+0000", 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.875e-05 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:33.621804+0000", 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8620999999999999e-05 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.588 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:127040 2.6 2:6091191f:::benchmark_data_smithi173_47195_object30032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 530.71072940399995, 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18230701299999999, 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:58.589 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127040 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851147+0000", 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851148+0000", 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851155+0000", 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8990000000000004e-06 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851175+0000", 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9711000000000002e-05 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851184+0000", 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6780000000000006e-06 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851247+0000", 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2780000000000005e-05 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851360+0000", 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112946 2024-08-30T23:36:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.851485+0000", 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012559 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.854952+0000", 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034665049999999999 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:07.862420+0000", 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0074685050000000003 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033372+0000", 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17095152199999999 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033407+0000", 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5567e-05 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:08.033454+0000", 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6681000000000002e-05 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.595 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:144898 2.6 2:692f2db4:::benchmark_data_smithi173_47195_object47890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 506.88270864600003, 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18135380100000001, 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144898 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.596 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679169+0000", 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679174+0000", 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.198e-06 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679187+0000", 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.31e-05 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679194+0000", 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679236+0000", 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1828e-05 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679316+0000", 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0026999999999998e-05 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679383+0000", 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6925e-05 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.682228+0000", 2024-08-30T23:36:58.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028449149999999999 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.829406+0000", 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14717798100000001 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860431+0000", 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031025377 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860482+0000", 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0779000000000003e-05 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860521+0000", 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9307999999999997e-05 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.602 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:d5896595:::benchmark_data_smithi173_47195_object12946:head,153'6861)", 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 434.82442909100001, 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59714269600000003, 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737449+0000", 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737471+0000", 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2631999999999999e-05 2024-08-30T23:36:58.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334475+0000", 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59700367600000004 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334480+0000", 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2719999999999997e-06 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334536+0000", 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5430999999999998e-05 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334546+0000", 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9389999999999992e-06 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334590+0000", 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4265999999999999e-05 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.606 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:dc4db0cb:::benchmark_data_smithi173_47195_object13755:head,153'6902)", 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 434.01261994200001, 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85400190600000003, 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549257+0000", 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:36:58.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549260+0000", 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.45e-06 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403003+0000", 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85374268200000003 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403016+0000", 2024-08-30T23:36:58.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3033e-05 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403163+0000", 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014668400000000001 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403180+0000", 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6813e-05 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403258+0000", 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8710999999999999e-05 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.5 e202,182 2:ac5fcd3c:::benchmark_data_smithi173_47195_object8560:head,150'6576)", 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 432.573387615, 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91766887600000002, 2024-08-30T23:36:58.611 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1099999999999996e-07 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988496+0000", 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4989999999999999e-06 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906006+0000", 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91750992200000003 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906016+0000", 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0199e-05 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906087+0000", 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1316999999999995e-05 2024-08-30T23:36:58.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906098+0000", 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0808000000000001e-05 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906158+0000", 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9620000000000002e-05 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.615 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103438 3.9 3:9093f9df:::benchmark_data_smithi173_60780_object103437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.28701011599998, 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38848340100000001, 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:58.616 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103438 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:36:58.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274868+0000", 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274894+0000", 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5692000000000001e-05 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274909+0000", 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5378000000000001e-05 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274918+0000", 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6489999999999997e-06 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274990+0000", 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2335000000000005e-05 2024-08-30T23:36:58.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275112+0000", 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012186 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275235+0000", 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123585 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278451+0000", 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003215954 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.279271+0000", 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081947200000000002 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663147+0000", 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38387581399999998 2024-08-30T23:36:58.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663253+0000", 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106125 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663350+0000", 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6832999999999999e-05 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:103438 3.9 e255/206 ondisk, result = 0)", 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 317.28376084199999, 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38524248500000002, 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278116+0000", 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278121+0000", 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7609999999999998e-06 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662960+0000", 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38483874699999998 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662985+0000", 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663120+0000", 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013537999999999999 2024-08-30T23:36:58.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663136+0000", 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6220000000000001e-05 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663358+0000", 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022142900000000001 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:163721 3.9 3:9f4ef878:::benchmark_data_smithi173_60780_object60267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 249.908637072, 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.179110358, 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.630 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 163721 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:36:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653241+0000", 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.013e-06 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653250+0000", 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653272+0000", 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2860000000000001e-05 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653281+0000", 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8289999999999997e-06 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654712+0000", 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014306620000000001 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654854+0000", 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141755 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654980+0000", 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012682000000000001 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.658949+0000", 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039685739999999999 2024-08-30T23:36:58.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.714803+0000", 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055854251000000001 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832230+0000", 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117426525 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832301+0000", 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0921999999999996e-05 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832350+0000", 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8835000000000003e-05 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:26909 4.0 e315/306 4:05316899:::benchmark_data_smithi173_74914_object26908:head v 315'1674, mlcod=315'1671)", 2024-08-30T23:36:58.636 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 160.365390717, 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.194211306, 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196487+0000", 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196605+0000", 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011812299999999999 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389338+0000", 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19273368799999999 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389343+0000", 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6589999999999998e-06 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389405+0000", 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1645999999999999e-05 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389423+0000", 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8641e-05 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390666+0000", 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001242993 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390697+0000", 2024-08-30T23:36:58.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0565000000000003e-05 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:49431 4.2 e327/306 4:49678b2f:::benchmark_data_smithi173_74914_object49430:head v 327'3104, mlcod=326'3101)", 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 138.01980820099999, 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.203368459, 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.641 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:36:58.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0900000000000002e-07 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542091+0000", 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2452000000000001e-05 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743809+0000", 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20171845999999999 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743817+0000", 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0960000000000001e-06 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743892+0000", 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4893000000000003e-05 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743911+0000", 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8479999999999999e-05 2024-08-30T23:36:58.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745394+0000", 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001483617 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745437+0000", 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2052000000000003e-05 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:84204 4.0 e346/329 4:023d35bd:::benchmark_data_smithi173_74914_object84203:head v 346'5351, mlcod=346'5348)", 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 101.160944384, 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20227883999999999, 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400933+0000", 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1990000000000001e-06 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400986+0000", 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3118000000000002e-05 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601032+0000", 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200045414 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601038+0000", 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4330000000000001e-06 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601086+0000", 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7720999999999999e-05 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601104+0000", 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8267999999999999e-05 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603163+0000", 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020593569999999999 2024-08-30T23:36:58.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603211+0000", 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7330000000000003e-05 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:142950 4.2 e364/306 4:4b8e0a77:::benchmark_data_smithi173_74914_object44708:head v 364'8978, mlcod=364'8974)", 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:18.072738+0000", 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 40.489138652999998, 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119578422, 2024-08-30T23:36:58.650 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.072738+0000", 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.072738+0000", 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.072738+0000", 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4199999999999995e-07 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.072744+0000", 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9159999999999998e-06 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.187931+0000", 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.115187083 2024-08-30T23:36:58.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.187939+0000", 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0590000000000003e-06 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.188005+0000", 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5859999999999996e-05 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.188029+0000", 2024-08-30T23:36:58.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3730000000000001e-05 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.192236+0000", 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042067709999999998 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.192316+0000", 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0260999999999996e-05 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:144897 4.0 e366/329 4:095cbd3d:::benchmark_data_smithi173_74914_object46655:head v 366'9227, mlcod=366'9223)", 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:20.918496+0000", 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 37.643380090999997, 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.104885165, 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:20.918496+0000", 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:20.918496+0000", 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:20.918497+0000", 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6299999999999993e-07 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:20.918504+0000", 2024-08-30T23:36:58.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7660000000000001e-06 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:21.018890+0000", 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.100386183 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:21.018898+0000", 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7349999999999996e-06 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:21.018972+0000", 2024-08-30T23:36:58.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4433999999999999e-05 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:21.018996+0000", 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4117999999999998e-05 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:21.023329+0000", 2024-08-30T23:36:58.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043329320000000003 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:21.023381+0000", 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2033999999999997e-05 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:165168 4.0 e373/329 4:0c243000:::benchmark_data_smithi173_74914_object66926:head v 373'10494, mlcod=373'10490)", 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:43.509373+0000", 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.052502933, 2024-08-30T23:36:58.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12675467100000001, 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509373+0000", 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509373+0000", 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509374+0000", 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1399999999999995e-07 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509380+0000", 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9610000000000002e-06 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632491+0000", 2024-08-30T23:36:58.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.123110441 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632498+0000", 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8129999999999996e-06 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632552+0000", 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3498999999999997e-05 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632569+0000", 2024-08-30T23:36:58.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6973000000000002e-05 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.636080+0000", 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035110139999999998 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.636128+0000", 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8056000000000002e-05 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:165186 4.0 e373/329 4:04c5fbe1:::benchmark_data_smithi173_74914_object66944:head v 373'10495, mlcod=373'10491)", 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:43.535823+0000", 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 15.026052928, 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.121679222, 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.535823+0000", 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.535823+0000", 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.535825+0000", 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.331e-06 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.535831+0000", 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3520000000000003e-06 2024-08-30T23:36:58.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.653942+0000", 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.118111411 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.653952+0000", 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1679999999999993e-06 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.654010+0000", 2024-08-30T23:36:58.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8440000000000003e-05 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.654026+0000", 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6259000000000001e-05 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.657471+0000", 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003445151 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.657503+0000", 2024-08-30T23:36:58.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1109999999999999e-05 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 4.2,from:0'0,to:375'10705,epoch:375/306,start:4:47243d77:::benchmark_data_smithi173_74914_object84691:0,end:4:47ff4719:::benchmark_data_smithi173_74914_object80101:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:49.162596+0000", 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.3992802930000003, 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12535713500000001, 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:36:58.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162596+0000", 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162596+0000", 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162597+0000", 2024-08-30T23:36:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.389e-06 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162606+0000", 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2260000000000005e-06 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287807+0000", 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.125201072 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287817+0000", 2024-08-30T23:36:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0295000000000001e-05 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287894+0000", 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7205000000000004e-05 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287911+0000", 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7031e-05 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:36:58.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287953+0000", 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1916999999999999e-05 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:36:58.672 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:36:58.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:58 smithi136 ceph-mon[20207]: pgmap v909: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 7959/63252 objects degraded (12.583%) 2024-08-30T23:36:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/173492902' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:58.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:58 smithi173 ceph-mon[20554]: pgmap v909: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.3 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 7959/63252 objects degraded (12.583%) 2024-08-30T23:36:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/173492902' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:36:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:58.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:58.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:36:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:58.967 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:58.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:36:59.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:36:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:59.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:36:59.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:59.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:59.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:36:59.351 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:36:59.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:36:59.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:36:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:36:59.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:36:59.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:36:59.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:36:59.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:36:59.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:36:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:59 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:59 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:36:59 smithi173 ceph-mon[20554]: osdmap e377: 12 total, 11 up, 12 in 2024-08-30T23:36:59.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:36:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:59 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:36:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:59 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:36:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:36:59 smithi106 ceph-mon[16603]: osdmap e377: 12 total, 11 up, 12 in 2024-08-30T23:36:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:36:59.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:36:59.891 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:36:59.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:36:59.998 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:36:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:36:59.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:59 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:59 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:36:59 smithi136 ceph-mon[20207]: osdmap e377: 12 total, 11 up, 12 in 2024-08-30T23:37:00.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:00.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:00.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:00.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:00.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:00.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:00.446 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:00.447 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:00.447 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:00.450 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:37:00.450 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:37:00.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.450 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:180265 4.d e377/369)", 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.440127+0000", 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.009595035, 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0096157740000000005, 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440127+0000", 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440127+0000", 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440128+0000", 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1800000000000005e-07 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440132+0000", 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2300000000000002e-06 2024-08-30T23:37:00.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446449+0000", 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0063171410000000001 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446457+0000", 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5240000000000003e-06 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.453 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:180272 4.6 e377/369)", 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.448668+0000", 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0010543639999999999, 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001114468, 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.448668+0000", 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.448668+0000", 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.448669+0000", 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.448673+0000", 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.746e-06 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.448705+0000", 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1640999999999997e-05 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.448713+0000", 2024-08-30T23:37:00.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4730000000000003e-06 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:180265 4.d 4:b0aa67c4:::benchmark_data_smithi173_74914_object82023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.435729+0000", 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.013993209, 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014096917, 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:37:00.458 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 180265 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435729+0000", 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435729+0000", 2024-08-30T23:37:00.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435730+0000", 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9000000000000005e-07 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435736+0000", 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5289999999999999e-06 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435744+0000", 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3089999999999996e-06 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435749+0000", 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9189999999999998e-06 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435814+0000", 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5279999999999998e-05 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.435967+0000", 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000153294 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 11", 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.436112+0000", 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144738 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.462 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:180268 4.d 4:b342e400:::benchmark_data_smithi173_74914_object82026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.439478+0000", 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.010244174999999999, 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010409532000000001, 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:00.463 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 180268 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.439478+0000", 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.439478+0000", 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.439480+0000", 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.401e-06 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.439485+0000", 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5790000000000003e-06 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.439500+0000", 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4738e-05 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.439508+0000", 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0309999999999998e-06 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.441007+0000", 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001498753 2024-08-30T23:37:00.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:00.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.441140+0000", 2024-08-30T23:37:00.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013298300000000001 2024-08-30T23:37:00.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 11", 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.441289+0000", 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014972100000000001 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:180269 4.d 4:b3571541:::benchmark_data_smithi173_74914_object82027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.440455+0000", 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0092669329999999998, 2024-08-30T23:37:00.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0094897720000000005, 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:00.469 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 180269 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440455+0000", 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440455+0000", 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440456+0000", 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9000000000000005e-07 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440462+0000", 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1580000000000004e-06 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440473+0000", 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1782e-05 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.440480+0000", 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6479999999999997e-06 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.447333+0000", 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068533279999999997 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.447460+0000", 2024-08-30T23:37:00.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012637799999999999 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 11", 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.447598+0000", 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013877299999999999 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:180272 4.6 4:66f76757:::benchmark_data_smithi173_74914_object82030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.444169+0000", 2024-08-30T23:37:00.473 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0055536470000000001, 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058433749999999996, 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 180272 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444169+0000", 2024-08-30T23:37:00.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444169+0000", 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444170+0000", 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7570000000000001e-06 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444178+0000", 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1119999999999998e-06 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444196+0000", 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8787000000000002e-05 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444205+0000", 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7010000000000002e-06 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444286+0000", 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0649000000000007e-05 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444428+0000", 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014206200000000001 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2", 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.444563+0000", 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013505400000000001 2024-08-30T23:37:00.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:180275 4.d 4.1f63117d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:00.446054+0000", 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0036678750000000001, 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040110880000000003, 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:00.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 180275 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446054+0000", 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446054+0000", 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:00.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446056+0000", 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3239999999999999e-06 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446061+0000", 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0039999999999999e-06 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446078+0000", 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7374e-05 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:00.446087+0000", 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8740000000000001e-06 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:00.481 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:37:00.483 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 7 2024-08-30T23:37:00.484 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:37:00.485 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:00.485 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:00.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:00.513 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:00.513 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:00.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:00.543 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:00.544 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:00.573 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:00.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:00 smithi106 ceph-mon[16603]: pgmap v911: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 7371/59694 objects degraded (12.348%) 2024-08-30T23:37:00.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3862669654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:00 smithi173 ceph-mon[20554]: pgmap v911: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 7371/59694 objects degraded (12.348%) 2024-08-30T23:37:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3862669654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:00.601 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:00.601 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:00.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:00.633 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:00.633 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:00.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:00.668 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:00.668 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:00.699 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:00.699 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:00.729 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:00.730 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:00.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:00.758 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:00.758 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:00.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:00.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:00.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:00.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:01.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:01.023 INFO:teuthology.orchestra.run.smithi106.stdout: marking collection for removal 2024-08-30T23:37:01.036 INFO:teuthology.orchestra.run.smithi106.stdout:setting '_remove' omap key 2024-08-30T23:37:01.036 INFO:teuthology.orchestra.run.smithi106.stdout:finish_remove_pgs 4.f_head removing 4.f 2024-08-30T23:37:01.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:00 smithi136 ceph-mon[20207]: pgmap v911: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.2 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s; 7371/59694 objects degraded (12.348%) 2024-08-30T23:37:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3862669654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:01.105 INFO:teuthology.orchestra.run.smithi106.stdout:Remove successful 2024-08-30T23:37:01.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:01.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:37:01.163 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:01.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:01.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:01.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:01.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:01.503 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:01.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:01.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7371/59694 objects degraded (12.348%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:01 smithi106 ceph-mon[16603]: pgmap v912: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 6273/50403 objects degraded (12.446%) 2024-08-30T23:37:01.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:01.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:01.692 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:01.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:01.804 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7371/59694 objects degraded (12.348%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:01.804 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:01 smithi173 ceph-mon[20554]: pgmap v912: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 6273/50403 objects degraded (12.446%) 2024-08-30T23:37:01.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:01.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:01.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:01.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:02.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7371/59694 objects degraded (12.348%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:01 smithi136 ceph-mon[20207]: pgmap v912: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 6273/50403 objects degraded (12.446%) 2024-08-30T23:37:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:02.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:02.097 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:02.098 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:02.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:02.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:02.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:02.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:02.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:02.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.f.1 2024-08-30T23:37:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:02.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:02.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:02.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:02.687 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:37:02.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:02.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:02.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:02.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:37:02.852 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:02.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:02.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:37:02.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:02.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:03.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:03.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:03.201 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:03.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:03.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:03.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:03.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:03.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:03.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:03.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:03.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:03.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:03.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:37:03.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:03.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:03.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:03.785 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:03.785 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:03.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:03.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:03.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:04.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:04.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:04.061 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:04.068 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:37:04.116 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:04.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:04.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:04.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:04.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:04.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:04.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:04.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:04.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:04.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:37:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:04 smithi106 ceph-mon[16603]: pgmap v913: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 960 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s; 5696/46053 objects degraded (12.368%) 2024-08-30T23:37:04.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.670 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:04.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:04.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:04.798 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:04.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:04 smithi173 ceph-mon[20554]: pgmap v913: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 960 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s; 5696/46053 objects degraded (12.368%) 2024-08-30T23:37:04.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:04.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:04.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:04.934 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:05.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:05.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:04 smithi136 ceph-mon[20207]: pgmap v913: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 960 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 915 op/s; 5696/46053 objects degraded (12.368%) 2024-08-30T23:37:05.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:05.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:05.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:05.281 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:05.291 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:05.292 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:05.292 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:05.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:05.323 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:05.323 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:05.352 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:05.352 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:05.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:05.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:05.403 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:05.404 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:05.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:05.433 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:05.433 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:05.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:05.468 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:05.468 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:05.486 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:05.487 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:05.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:05.499 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:05.529 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:05.529 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:05.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:05.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:05.562 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:05.562 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:05.590 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:05.591 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:05.592 INFO:teuthology.orchestra.run.smithi106.stdout:get_pg_num_history pg_num_history pg_num_history(e368 pg_nums {1={20=1},4={306=16}} deleted_pools ) 2024-08-30T23:37:05.593 INFO:teuthology.orchestra.run.smithi106.stdout:Importing pgid 4.f 2024-08-30T23:37:05.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f002868a:::benchmark_data_smithi173_74914_object61686:head# 2024-08-30T23:37:05.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f002b325:::benchmark_data_smithi173_74914_object87581:head# 2024-08-30T23:37:05.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f007cb59:::benchmark_data_smithi173_74914_object90155:head# 2024-08-30T23:37:05.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f008c387:::benchmark_data_smithi173_74914_object85988:head# 2024-08-30T23:37:05.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00947d9:::benchmark_data_smithi173_74914_object64042:head# 2024-08-30T23:37:05.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00999f2:::benchmark_data_smithi173_74914_object78429:head# 2024-08-30T23:37:05.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00a3049:::benchmark_data_smithi173_74914_object90740:head# 2024-08-30T23:37:05.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00b8306:::benchmark_data_smithi173_74914_object89563:head# 2024-08-30T23:37:05.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00c0d2d:::benchmark_data_smithi173_74914_object65020:head# 2024-08-30T23:37:05.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00de138:::benchmark_data_smithi173_74914_object94006:head# 2024-08-30T23:37:05.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00eee5d:::benchmark_data_smithi173_74914_object83469:head# 2024-08-30T23:37:05.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00f5dc0:::benchmark_data_smithi173_74914_object53545:head# 2024-08-30T23:37:05.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f00ff2f1:::benchmark_data_smithi173_74914_object74319:head# 2024-08-30T23:37:05.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f010bb39:::benchmark_data_smithi173_74914_object66760:head# 2024-08-30T23:37:05.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f01111e7:::benchmark_data_smithi173_74914_object61430:head# 2024-08-30T23:37:05.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0112e43:::benchmark_data_smithi173_74914_object77849:head# 2024-08-30T23:37:05.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0116a6f:::benchmark_data_smithi173_74914_object81843:head# 2024-08-30T23:37:05.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0124fd9:::benchmark_data_smithi173_74914_object74043:head# 2024-08-30T23:37:05.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f01266e0:::benchmark_data_smithi173_74914_object88367:head# 2024-08-30T23:37:05.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0129921:::benchmark_data_smithi173_74914_object77855:head# 2024-08-30T23:37:05.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0143e9b:::benchmark_data_smithi173_74914_object64655:head# 2024-08-30T23:37:05.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f01de819:::benchmark_data_smithi173_74914_object91758:head# 2024-08-30T23:37:05.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f01e8d34:::benchmark_data_smithi173_74914_object96320:head# 2024-08-30T23:37:05.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f01ed92b:::benchmark_data_smithi173_74914_object65085:head# 2024-08-30T23:37:05.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f01fee22:::benchmark_data_smithi173_74914_object66027:head# 2024-08-30T23:37:05.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0200416:::benchmark_data_smithi173_74914_object68751:head# 2024-08-30T23:37:05.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.634 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f02109e1:::benchmark_data_smithi173_74914_object78220:head# 2024-08-30T23:37:05.634 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.635 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0220a7f:::benchmark_data_smithi173_74914_object58070:head# 2024-08-30T23:37:05.635 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0224bcb:::benchmark_data_smithi173_74914_object95941:head# 2024-08-30T23:37:05.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.636 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f023a268:::benchmark_data_smithi173_74914_object74241:head# 2024-08-30T23:37:05.636 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0246868:::benchmark_data_smithi173_74914_object83754:head# 2024-08-30T23:37:05.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f024e0d6:::benchmark_data_smithi173_74914_object74226:head# 2024-08-30T23:37:05.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f025c189:::benchmark_data_smithi173_74914_object65838:head# 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0277a4f:::benchmark_data_smithi173_74914_object78384:head# 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f02bd4cd:::benchmark_data_smithi173_74914_object95285:head# 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f02e898d:::benchmark_data_smithi173_74914_object85563:head# 2024-08-30T23:37:05.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f032b7cf:::benchmark_data_smithi173_74914_object90067:head# 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f033e9f3:::benchmark_data_smithi173_74914_object92300:head# 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0365d93:::benchmark_data_smithi173_74914_object62560:head# 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0372d40:::benchmark_data_smithi173_74914_object79766:head# 2024-08-30T23:37:05.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0374ece:::benchmark_data_smithi173_74914_object81818:head# 2024-08-30T23:37:05.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03781a8:::benchmark_data_smithi173_74914_object92683:head# 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0380e27:::benchmark_data_smithi173_74914_object67791:head# 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0386dab:::benchmark_data_smithi173_74914_object86974:head# 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f039070d:::benchmark_data_smithi173_74914_object78471:head# 2024-08-30T23:37:05.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03991c2:::benchmark_data_smithi173_74914_object98156:head# 2024-08-30T23:37:05.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03ae0ca:::benchmark_data_smithi173_74914_object63861:head# 2024-08-30T23:37:05.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03dcb97:::benchmark_data_smithi173_74914_object72412:head# 2024-08-30T23:37:05.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03e470b:::benchmark_data_smithi173_74914_object60063:head# 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03e963e:::benchmark_data_smithi173_74914_object85974:head# 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f03efb49:::benchmark_data_smithi173_74914_object77756:head# 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f04274e4:::benchmark_data_smithi173_74914_object66728:head# 2024-08-30T23:37:05.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0469490:::benchmark_data_smithi173_74914_object73964:head# 2024-08-30T23:37:05.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0491d63:::benchmark_data_smithi173_74914_object75707:head# 2024-08-30T23:37:05.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f04b0b3c:::benchmark_data_smithi173_74914_object90564:head# 2024-08-30T23:37:05.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f04b797d:::benchmark_data_smithi173_74914_object79752:head# 2024-08-30T23:37:05.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f04cc918:::benchmark_data_smithi173_74914_object84297:head# 2024-08-30T23:37:05.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f04ec094:::benchmark_data_smithi173_74914_object66697:head# 2024-08-30T23:37:05.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f04ede97:::benchmark_data_smithi173_74914_object61237:head# 2024-08-30T23:37:05.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f050d553:::benchmark_data_smithi173_74914_object90590:head# 2024-08-30T23:37:05.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f052e30f:::benchmark_data_smithi173_74914_object95300:head# 2024-08-30T23:37:05.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0532f02:::benchmark_data_smithi173_74914_object95216:head# 2024-08-30T23:37:05.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0537508:::benchmark_data_smithi173_74914_object62324:head# 2024-08-30T23:37:05.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0553b04:::benchmark_data_smithi173_74914_object91162:head# 2024-08-30T23:37:05.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f056b1b9:::benchmark_data_smithi173_74914_object97399:head# 2024-08-30T23:37:05.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0570a6c:::benchmark_data_smithi173_74914_object63336:head# 2024-08-30T23:37:05.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f058cd57:::benchmark_data_smithi173_74914_object78591:head# 2024-08-30T23:37:05.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0599b2f:::benchmark_data_smithi173_74914_object77274:head# 2024-08-30T23:37:05.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f05a8274:::benchmark_data_smithi173_74914_object64420:head# 2024-08-30T23:37:05.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f05e5dae:::benchmark_data_smithi173_74914_object58758:head# 2024-08-30T23:37:05.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f060f0a0:::benchmark_data_smithi173_74914_object55605:head# 2024-08-30T23:37:05.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f063e6d6:::benchmark_data_smithi173_74914_object60697:head# 2024-08-30T23:37:05.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0652abc:::benchmark_data_smithi173_74914_object97798:head# 2024-08-30T23:37:05.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0653a94:::benchmark_data_smithi173_74914_object70147:head# 2024-08-30T23:37:05.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0655d49:::benchmark_data_smithi173_74914_object59529:head# 2024-08-30T23:37:05.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0659981:::benchmark_data_smithi173_74914_object80460:head# 2024-08-30T23:37:05.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f065a9cd:::benchmark_data_smithi173_74914_object68263:head# 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0672271:::benchmark_data_smithi173_74914_object79899:head# 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f06b79e8:::benchmark_data_smithi173_74914_object72634:head# 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f06ba776:::benchmark_data_smithi173_74914_object79640:head# 2024-08-30T23:37:05.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f06cebf2:::benchmark_data_smithi173_74914_object95406:head# 2024-08-30T23:37:05.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:05.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f06d3357:::benchmark_data_smithi173_74914_object94866:head# 2024-08-30T23:37:05.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f06ddd20:::benchmark_data_smithi173_74914_object74747:head# 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f070e8be:::benchmark_data_smithi173_74914_object64150:head# 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07462e2:::benchmark_data_smithi173_74914_object80118:head# 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0757dee:::benchmark_data_smithi173_74914_object57392:head# 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0774496:::benchmark_data_smithi173_74914_object57239:head# 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f077fbef:::benchmark_data_smithi173_74914_object74818:head# 2024-08-30T23:37:05.665 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0783867:::benchmark_data_smithi173_74914_object64549:head# 2024-08-30T23:37:05.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f078857c:::benchmark_data_smithi173_74914_object78984:head# 2024-08-30T23:37:05.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.666 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0788fd7:::benchmark_data_smithi173_74914_object65671:head# 2024-08-30T23:37:05.666 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0796ae5:::benchmark_data_smithi173_74914_object76417:head# 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07a6355:::benchmark_data_smithi173_74914_object95648:head# 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07af1dd:::benchmark_data_smithi173_74914_object96820:head# 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07b7143:::benchmark_data_smithi173_74914_object59882:head# 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.668 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07b76f7:::benchmark_data_smithi173_74914_object63505:head# 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07c290f:::benchmark_data_smithi173_74914_object74894:head# 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f07d5d19:::benchmark_data_smithi173_74914_object76854:head# 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f080b372:::benchmark_data_smithi173_74914_object79090:head# 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0823204:::benchmark_data_smithi173_74914_object73215:head# 2024-08-30T23:37:05.669 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08494b5:::benchmark_data_smithi173_74914_object81489:head# 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f087e628:::benchmark_data_smithi173_74914_object91881:head# 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f087eef7:::benchmark_data_smithi173_74914_object74456:head# 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.670 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08858b4:::benchmark_data_smithi173_74914_object53820:head# 2024-08-30T23:37:05.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f088903d:::benchmark_data_smithi173_74914_object86139:head# 2024-08-30T23:37:05.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.671 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f088a7c3:::benchmark_data_smithi173_74914_object73944:head# 2024-08-30T23:37:05.671 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08a5bb2:::benchmark_data_smithi173_74914_object61378:head# 2024-08-30T23:37:05.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08b2bc1:::benchmark_data_smithi173_74914_object78002:head# 2024-08-30T23:37:05.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.672 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08c9d39:::benchmark_data_smithi173_74914_object64945:head# 2024-08-30T23:37:05.672 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08d58a1:::benchmark_data_smithi173_74914_object96691:head# 2024-08-30T23:37:05.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08ebead:::benchmark_data_smithi173_74914_object95574:head# 2024-08-30T23:37:05.673 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.673 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f08f5469:::benchmark_data_smithi173_74914_object54965:head# 2024-08-30T23:37:05.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f091d489:::benchmark_data_smithi173_74914_object83817:head# 2024-08-30T23:37:05.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f094c943:::benchmark_data_smithi173_74914_object77339:head# 2024-08-30T23:37:05.674 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.674 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f096ff6a:::benchmark_data_smithi173_74914_object75241:head# 2024-08-30T23:37:05.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0978f65:::benchmark_data_smithi173_74914_object97974:head# 2024-08-30T23:37:05.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.675 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f097f8fa:::benchmark_data_smithi173_74914_object61133:head# 2024-08-30T23:37:05.675 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f09821e2:::benchmark_data_smithi173_74914_object73054:head# 2024-08-30T23:37:05.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0985984:::benchmark_data_smithi173_74914_object64499:head# 2024-08-30T23:37:05.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.676 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f09955ca:::benchmark_data_smithi173_74914_object80024:head# 2024-08-30T23:37:05.676 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f09a6d6d:::benchmark_data_smithi173_74914_object80237:head# 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f09f9272:::benchmark_data_smithi173_74914_object60163:head# 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0a08bcf:::benchmark_data_smithi173_74914_object78807:head# 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0a209aa:::benchmark_data_smithi173_74914_object92895:head# 2024-08-30T23:37:05.677 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0a8f7c1:::benchmark_data_smithi173_74914_object93498:head# 2024-08-30T23:37:05.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0aae2ff:::benchmark_data_smithi173_74914_object92284:head# 2024-08-30T23:37:05.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.678 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0ad5d60:::benchmark_data_smithi173_74914_object81456:head# 2024-08-30T23:37:05.678 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0aeadc6:::benchmark_data_smithi173_74914_object88877:head# 2024-08-30T23:37:05.679 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.679 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b26954:::benchmark_data_smithi173_74914_object85746:head# 2024-08-30T23:37:05.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b26b6b:::benchmark_data_smithi173_74914_object88018:head# 2024-08-30T23:37:05.680 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.680 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b291e5:::benchmark_data_smithi173_74914_object75390:head# 2024-08-30T23:37:05.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.681 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b2b1d5:::benchmark_data_smithi173_74914_object60083:head# 2024-08-30T23:37:05.681 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b4a36f:::benchmark_data_smithi173_74914_object81562:head# 2024-08-30T23:37:05.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.682 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b4c639:::benchmark_data_smithi173_74914_object58700:head# 2024-08-30T23:37:05.682 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b56c66:::benchmark_data_smithi173_74914_object86067:head# 2024-08-30T23:37:05.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.683 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0b9ddbc:::benchmark_data_smithi173_74914_object77746:head# 2024-08-30T23:37:05.683 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0bc2a61:::benchmark_data_smithi173_74914_object64368:head# 2024-08-30T23:37:05.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.684 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0bcb01d:::benchmark_data_smithi173_74914_object69121:head# 2024-08-30T23:37:05.684 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.685 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0bd6511:::benchmark_data_smithi173_74914_object94278:head# 2024-08-30T23:37:05.685 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c17b41:::benchmark_data_smithi173_74914_object79285:head# 2024-08-30T23:37:05.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.689 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c255a6:::benchmark_data_smithi173_74914_object61699:head# 2024-08-30T23:37:05.689 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c640ad:::benchmark_data_smithi173_74914_object70887:head# 2024-08-30T23:37:05.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.690 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c7d792:::benchmark_data_smithi173_74914_object90795:head# 2024-08-30T23:37:05.690 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c802f5:::benchmark_data_smithi173_74914_object57858:head# 2024-08-30T23:37:05.691 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.691 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c80e25:::benchmark_data_smithi173_74914_object57980:head# 2024-08-30T23:37:05.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.692 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c9338e:::benchmark_data_smithi173_74914_object90008:head# 2024-08-30T23:37:05.692 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:05.693 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0c941cc:::benchmark_data_smithi173_74914_object77681:head# 2024-08-30T23:37:05.693 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0cfc79a:::benchmark_data_smithi173_74914_object97430:head# 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d01781:::benchmark_data_smithi173_74914_object59270:head# 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d08e7d:::benchmark_data_smithi173_74914_object85783:head# 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d0cd7e:::benchmark_data_smithi173_74914_object65054:head# 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.695 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d19af4:::benchmark_data_smithi173_74914_object55277:head# 2024-08-30T23:37:05.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.696 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d1b06d:::benchmark_data_smithi173_74914_object92267:head# 2024-08-30T23:37:05.696 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d20ea5:::benchmark_data_smithi173_74914_object72810:head# 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d2450b:::benchmark_data_smithi173_74914_object60079:head# 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d54470:::benchmark_data_smithi173_74914_object54434:head# 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.697 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d56a9e:::benchmark_data_smithi173_74914_object64072:head# 2024-08-30T23:37:05.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d61f76:::benchmark_data_smithi173_74914_object76022:head# 2024-08-30T23:37:05.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.698 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d7099e:::benchmark_data_smithi173_74914_object94841:head# 2024-08-30T23:37:05.698 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0d8a639:::benchmark_data_smithi173_74914_object76937:head# 2024-08-30T23:37:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0db13f6:::benchmark_data_smithi173_74914_object88056:head# 2024-08-30T23:37:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0db30b8:::benchmark_data_smithi173_74914_object89118:head# 2024-08-30T23:37:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0dc7482:::benchmark_data_smithi173_74914_object54040:head# 2024-08-30T23:37:05.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.700 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0dd5876:::benchmark_data_smithi173_74914_object71667:head# 2024-08-30T23:37:05.700 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0de4ddb:::benchmark_data_smithi173_74914_object66265:head# 2024-08-30T23:37:05.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.701 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0de4f4f:::benchmark_data_smithi173_74914_object77947:head# 2024-08-30T23:37:05.701 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0de8a4e:::benchmark_data_smithi173_74914_object65449:head# 2024-08-30T23:37:05.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.702 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0de956f:::benchmark_data_smithi173_74914_object88392:head# 2024-08-30T23:37:05.702 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0e666fb:::benchmark_data_smithi173_74914_object86804:head# 2024-08-30T23:37:05.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0e98f5f:::benchmark_data_smithi173_74914_object82592:head# 2024-08-30T23:37:05.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.703 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0ea0c3a:::benchmark_data_smithi173_74914_object63608:head# 2024-08-30T23:37:05.703 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.704 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0eb52b6:::benchmark_data_smithi173_74914_object91331:head# 2024-08-30T23:37:05.704 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0ecdea9:::benchmark_data_smithi173_74914_object68307:head# 2024-08-30T23:37:05.709 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.709 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f0a8f4:::benchmark_data_smithi173_74914_object95093:head# 2024-08-30T23:37:05.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f1a5a4:::benchmark_data_smithi173_74914_object60084:head# 2024-08-30T23:37:05.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f276a4:::benchmark_data_smithi173_74914_object62151:head# 2024-08-30T23:37:05.710 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.710 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f32ed1:::benchmark_data_smithi173_74914_object98186:head# 2024-08-30T23:37:05.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f3dad0:::benchmark_data_smithi173_74914_object62157:head# 2024-08-30T23:37:05.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.711 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f73f4c:::benchmark_data_smithi173_74914_object77697:head# 2024-08-30T23:37:05.711 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f76eb5:::benchmark_data_smithi173_74914_object71529:head# 2024-08-30T23:37:05.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0f8baf2:::benchmark_data_smithi173_74914_object97514:head# 2024-08-30T23:37:05.712 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.712 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0fdb023:::benchmark_data_smithi173_74914_object59345:head# 2024-08-30T23:37:05.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0feb7b2:::benchmark_data_smithi173_74914_object78822:head# 2024-08-30T23:37:05.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.713 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f0ff8a2a:::benchmark_data_smithi173_74914_object90343:head# 2024-08-30T23:37:05.713 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.714 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f100dc66:::benchmark_data_smithi173_74914_object63371:head# 2024-08-30T23:37:05.714 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1036ef9:::benchmark_data_smithi173_74914_object61502:head# 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1037b10:::benchmark_data_smithi173_74914_object84962:head# 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f105d454:::benchmark_data_smithi173_74914_object87414:head# 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f105d8cb:::benchmark_data_smithi173_74914_object86903:head# 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f105f507:::benchmark_data_smithi173_74914_object86035:head# 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f108ac1e:::benchmark_data_smithi173_74914_object95476:head# 2024-08-30T23:37:05.715 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1090b7f:::benchmark_data_smithi173_74914_object87325:head# 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f10b5cc9:::benchmark_data_smithi173_74914_object97196:head# 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f10bd2e5:::benchmark_data_smithi173_74914_object64451:head# 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f10c8728:::benchmark_data_smithi173_74914_object90429:head# 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f10f845a:::benchmark_data_smithi173_74914_object93811:head# 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f113b5d6:::benchmark_data_smithi173_74914_object53827:head# 2024-08-30T23:37:05.718 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1151a82:::benchmark_data_smithi173_74914_object59727:head# 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f115d847:::benchmark_data_smithi173_74914_object88515:head# 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1166dfa:::benchmark_data_smithi173_74914_object67400:head# 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1186475:::benchmark_data_smithi173_74914_object80693:head# 2024-08-30T23:37:05.719 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.720 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f11bb270:::benchmark_data_smithi173_74914_object93133:head# 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f11c4ac7:::benchmark_data_smithi173_74914_object61268:head# 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f11cfcb1:::benchmark_data_smithi173_74914_object89000:head# 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f120af0d:::benchmark_data_smithi173_74914_object86388:head# 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f123c944:::benchmark_data_smithi173_74914_object55883:head# 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1249c8a:::benchmark_data_smithi173_74914_object95569:head# 2024-08-30T23:37:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1254f92:::benchmark_data_smithi173_74914_object58838:head# 2024-08-30T23:37:05.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.724 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1280692:::benchmark_data_smithi173_74914_object56758:head# 2024-08-30T23:37:05.724 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f129f8e2:::benchmark_data_smithi173_74914_object61934:head# 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f12a2fff:::benchmark_data_smithi173_74914_object97017:head# 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f12aec8e:::benchmark_data_smithi173_74914_object88479:head# 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f12b7594:::benchmark_data_smithi173_74914_object65397:head# 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f12d43df:::benchmark_data_smithi173_74914_object71608:head# 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f12e930b:::benchmark_data_smithi173_74914_object57918:head# 2024-08-30T23:37:05.725 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f130ee2f:::benchmark_data_smithi173_74914_object69906:head# 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1311ea3:::benchmark_data_smithi173_74914_object61311:head# 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1324e98:::benchmark_data_smithi173_74914_object90631:head# 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f132bdd4:::benchmark_data_smithi173_74914_object80919:head# 2024-08-30T23:37:05.726 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f137ddd8:::benchmark_data_smithi173_74914_object74897:head# 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1386878:::benchmark_data_smithi173_74914_object85612:head# 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f13ad077:::benchmark_data_smithi173_74914_object93301:head# 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f13ba9e5:::benchmark_data_smithi173_74914_object65362:head# 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.728 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f13c0558:::benchmark_data_smithi173_74914_object68727:head# 2024-08-30T23:37:05.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f13c8005:::benchmark_data_smithi173_74914_object54775:head# 2024-08-30T23:37:05.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.729 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f13d9623:::benchmark_data_smithi173_74914_object93432:head# 2024-08-30T23:37:05.729 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f13da5fc:::benchmark_data_smithi173_74914_object64519:head# 2024-08-30T23:37:05.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f140115d:::benchmark_data_smithi173_74914_object66964:head# 2024-08-30T23:37:05.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.730 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1434dbd:::benchmark_data_smithi173_74914_object82687:head# 2024-08-30T23:37:05.730 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f14369dc:::benchmark_data_smithi173_74914_object62533:head# 2024-08-30T23:37:05.732 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.732 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1440da0:::benchmark_data_smithi173_74914_object59782:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f14503f3:::benchmark_data_smithi173_74914_object71150:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f145495e:::benchmark_data_smithi173_74914_object84358:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1463792:::benchmark_data_smithi173_74914_object58854:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1465d6f:::benchmark_data_smithi173_74914_object63327:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f147929c:::benchmark_data_smithi173_74914_object53771:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f14e05a3:::benchmark_data_smithi173_74914_object85161:head# 2024-08-30T23:37:05.733 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.735 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f14e321a:::benchmark_data_smithi173_74914_object83824:head# 2024-08-30T23:37:05.735 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f14e8b36:::benchmark_data_smithi173_74914_object74590:head# 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f151fe7e:::benchmark_data_smithi173_74914_object63765:head# 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f153fe14:::benchmark_data_smithi173_74914_object83492:head# 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f155a4a6:::benchmark_data_smithi173_74914_object97054:head# 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1569413:::benchmark_data_smithi173_74914_object94571:head# 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.736 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f158c4f2:::benchmark_data_smithi173_74914_object55465:head# 2024-08-30T23:37:05.737 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.737 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f15f54c9:::benchmark_data_smithi173_74914_object84640:head# 2024-08-30T23:37:05.737 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.737 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f15f8f99:::benchmark_data_smithi173_74914_object79634:head# 2024-08-30T23:37:05.737 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.740 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1612556:::benchmark_data_smithi173_74914_object82672:head# 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1619e1c:::benchmark_data_smithi173_74914_object53359:head# 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f162367d:::benchmark_data_smithi173_74914_object91476:head# 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1651a60:::benchmark_data_smithi173_74914_object73133:head# 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f165af19:::benchmark_data_smithi173_74914_object71405:head# 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1672dd9:::benchmark_data_smithi173_74914_object56113:head# 2024-08-30T23:37:05.741 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f168ce2f:::benchmark_data_smithi173_74914_object63324:head# 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f16c8bed:::benchmark_data_smithi173_74914_object77296:head# 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f16c9187:::benchmark_data_smithi173_74914_object90322:head# 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f16cc974:::benchmark_data_smithi173_74914_object68355:head# 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f16d2fcc:::benchmark_data_smithi173_74914_object69400:head# 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f16e1871:::benchmark_data_smithi173_74914_object83517:head# 2024-08-30T23:37:05.742 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1709dcf:::benchmark_data_smithi173_74914_object87591:head# 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1717337:::benchmark_data_smithi173_74914_object78802:head# 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1731894:::benchmark_data_smithi173_74914_object55921:head# 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1737b2e:::benchmark_data_smithi173_74914_object66724:head# 2024-08-30T23:37:05.743 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1744bae:::benchmark_data_smithi173_74914_object92712:head# 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f174c00c:::benchmark_data_smithi173_74914_object83497:head# 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1756dd2:::benchmark_data_smithi173_74914_object53872:head# 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f17667af:::benchmark_data_smithi173_74914_object65979:head# 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.746 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f17ba282:::benchmark_data_smithi173_74914_object93123:head# 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f17c194e:::benchmark_data_smithi173_74914_object63190:head# 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f17df36a:::benchmark_data_smithi173_74914_object90884:head# 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f17f06ad:::benchmark_data_smithi173_74914_object75129:head# 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1852aa0:::benchmark_data_smithi173_74914_object54828:head# 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f185816e:::benchmark_data_smithi173_74914_object96948:head# 2024-08-30T23:37:05.747 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.748 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1863d8d:::benchmark_data_smithi173_74914_object61985:head# 2024-08-30T23:37:05.748 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.748 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1869856:::benchmark_data_smithi173_74914_object63642:head# 2024-08-30T23:37:05.748 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.748 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f186b411:::benchmark_data_smithi173_74914_object68922:head# 2024-08-30T23:37:05.748 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.749 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f187247b:::benchmark_data_smithi173_74914_object65933:head# 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18a2a65:::benchmark_data_smithi173_74914_object76276:head# 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18a3aa1:::benchmark_data_smithi173_74914_object81388:head# 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18c3d1e:::benchmark_data_smithi173_74914_object83210:head# 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18c3df3:::benchmark_data_smithi173_74914_object91498:head# 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18c8940:::benchmark_data_smithi173_74914_object97988:head# 2024-08-30T23:37:05.750 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.751 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18f8d0d:::benchmark_data_smithi173_74914_object93615:head# 2024-08-30T23:37:05.751 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.752 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f18fe858:::benchmark_data_smithi173_74914_object70881:head# 2024-08-30T23:37:05.752 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.752 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1905dfe:::benchmark_data_smithi173_74914_object96599:head# 2024-08-30T23:37:05.752 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.752 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f195a633:::benchmark_data_smithi173_74914_object55620:head# 2024-08-30T23:37:05.752 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.753 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19696e1:::benchmark_data_smithi173_74914_object86403:head# 2024-08-30T23:37:05.753 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.753 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19738b8:::benchmark_data_smithi173_74914_object88728:head# 2024-08-30T23:37:05.753 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.753 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f197cc89:::benchmark_data_smithi173_74914_object91349:head# 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19a5c09:::benchmark_data_smithi173_74914_object85622:head# 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19a6f4d:::benchmark_data_smithi173_74914_object82308:head# 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19a7be2:::benchmark_data_smithi173_74914_object57563:head# 2024-08-30T23:37:05.754 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.755 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19beadf:::benchmark_data_smithi173_74914_object64380:head# 2024-08-30T23:37:05.755 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.755 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19d07bd:::benchmark_data_smithi173_74914_object84646:head# 2024-08-30T23:37:05.755 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.756 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19d09f1:::benchmark_data_smithi173_74914_object82490:head# 2024-08-30T23:37:05.756 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.756 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19d86d0:::benchmark_data_smithi173_74914_object89060:head# 2024-08-30T23:37:05.756 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.758 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19de880:::benchmark_data_smithi173_74914_object77568:head# 2024-08-30T23:37:05.758 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.758 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f19e2f37:::benchmark_data_smithi173_74914_object88866:head# 2024-08-30T23:37:05.758 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.758 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1a375f7:::benchmark_data_smithi173_74914_object62398:head# 2024-08-30T23:37:05.758 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.759 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1a40e26:::benchmark_data_smithi173_74914_object65847:head# 2024-08-30T23:37:05.759 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1a654fa:::benchmark_data_smithi173_74914_object62435:head# 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ab3038:::benchmark_data_smithi173_74914_object78267:head# 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1aba507:::benchmark_data_smithi173_74914_object72068:head# 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1abe918:::benchmark_data_smithi173_74914_object88495:head# 2024-08-30T23:37:05.760 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.761 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ace83b:::benchmark_data_smithi173_74914_object53171:head# 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ae3303:::benchmark_data_smithi173_74914_object70262:head# 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ae8d61:::benchmark_data_smithi173_74914_object89541:head# 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1b01d9d:::benchmark_data_smithi173_74914_object66892:head# 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1b28adb:::benchmark_data_smithi173_74914_object55837:head# 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1b7a78e:::benchmark_data_smithi173_74914_object77514:head# 2024-08-30T23:37:05.762 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1b8c966:::benchmark_data_smithi173_74914_object75406:head# 2024-08-30T23:37:05.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1b9a331:::benchmark_data_smithi173_74914_object60195:head# 2024-08-30T23:37:05.763 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.763 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ba36a4:::benchmark_data_smithi173_74914_object63675:head# 2024-08-30T23:37:05.764 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.769 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1bc165f:::benchmark_data_smithi173_74914_object85044:head# 2024-08-30T23:37:05.769 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/186663640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:05.769 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:05 smithi106 ceph-mon[16603]: pgmap v914: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 845 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 4773/40548 objects degraded (11.771%) 2024-08-30T23:37:05.769 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.770 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1bcf7ba:::benchmark_data_smithi173_74914_object63791:head# 2024-08-30T23:37:05.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.771 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1bdb096:::benchmark_data_smithi173_74914_object65683:head# 2024-08-30T23:37:05.771 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1bebfa1:::benchmark_data_smithi173_74914_object77565:head# 2024-08-30T23:37:05.772 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.772 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1bf0fd9:::benchmark_data_smithi173_74914_object94360:head# 2024-08-30T23:37:05.773 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1bfe928:::benchmark_data_smithi173_74914_object94718:head# 2024-08-30T23:37:05.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1c6aa6e:::benchmark_data_smithi173_74914_object67158:head# 2024-08-30T23:37:05.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.774 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1c93398:::benchmark_data_smithi173_74914_object80697:head# 2024-08-30T23:37:05.774 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1c93d72:::benchmark_data_smithi173_74914_object72458:head# 2024-08-30T23:37:05.775 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.775 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1c96c74:::benchmark_data_smithi173_74914_object93744:head# 2024-08-30T23:37:05.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.776 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ca0741:::benchmark_data_smithi173_74914_object67995:head# 2024-08-30T23:37:05.776 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.777 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ca216f:::benchmark_data_smithi173_74914_object70789:head# 2024-08-30T23:37:05.777 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1cb3c26:::benchmark_data_smithi173_74914_object56282:head# 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1cdb9ac:::benchmark_data_smithi173_74914_object80700:head# 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1cdc46f:::benchmark_data_smithi173_74914_object97775:head# 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1cde481:::benchmark_data_smithi173_74914_object77706:head# 2024-08-30T23:37:05.778 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1d0da0c:::benchmark_data_smithi173_74914_object54396:head# 2024-08-30T23:37:05.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1d3c646:::benchmark_data_smithi173_74914_object69705:head# 2024-08-30T23:37:05.781 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.781 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1d4508c:::benchmark_data_smithi173_74914_object93105:head# 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1d8b675:::benchmark_data_smithi173_74914_object85821:head# 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1dc104b:::benchmark_data_smithi173_74914_object72546:head# 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e0ef5c:::benchmark_data_smithi173_74914_object75801:head# 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e1cf50:::benchmark_data_smithi173_74914_object81795:head# 2024-08-30T23:37:05.782 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e2dfe1:::benchmark_data_smithi173_74914_object73189:head# 2024-08-30T23:37:05.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e34d7e:::benchmark_data_smithi173_74914_object78495:head# 2024-08-30T23:37:05.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.783 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e5d7a5:::benchmark_data_smithi173_74914_object70771:head# 2024-08-30T23:37:05.783 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e6db44:::benchmark_data_smithi173_74914_object75931:head# 2024-08-30T23:37:05.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.789 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e81bdf:::benchmark_data_smithi173_74914_object79462:head# 2024-08-30T23:37:05.789 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e85548:::benchmark_data_smithi173_74914_object89294:head# 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1e8fe08:::benchmark_data_smithi173_74914_object86535:head# 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1eab8cd:::benchmark_data_smithi173_74914_object60719:head# 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1eade43:::benchmark_data_smithi173_74914_object84129:head# 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1eaee6a:::benchmark_data_smithi173_74914_object74513:head# 2024-08-30T23:37:05.790 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ef4372:::benchmark_data_smithi173_74914_object86491:head# 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1f1c106:::benchmark_data_smithi173_74914_object75284:head# 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1f4cb29:::benchmark_data_smithi173_74914_object71073:head# 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1f6cba1:::benchmark_data_smithi173_74914_object69867:head# 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1f7271e:::benchmark_data_smithi173_74914_object84030:head# 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1fa5e05:::benchmark_data_smithi173_74914_object54783:head# 2024-08-30T23:37:05.791 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1fb2222:::benchmark_data_smithi173_74914_object86101:head# 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1fd69d5:::benchmark_data_smithi173_74914_object96804:head# 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1fd8071:::benchmark_data_smithi173_74914_object62241:head# 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1fe13a1:::benchmark_data_smithi173_74914_object91902:head# 2024-08-30T23:37:05.792 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ff6920:::benchmark_data_smithi173_74914_object93953:head# 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1ff8c00:::benchmark_data_smithi173_74914_object83618:head# 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f1fffc37:::benchmark_data_smithi173_74914_object71586:head# 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2006ddf:::benchmark_data_smithi173_74914_object92863:head# 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f20105df:::benchmark_data_smithi173_74914_object58654:head# 2024-08-30T23:37:05.793 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.794 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2029ee2:::benchmark_data_smithi173_74914_object75295:head# 2024-08-30T23:37:05.794 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.796 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2031102:::benchmark_data_smithi173_74914_object55292:head# 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2047682:::benchmark_data_smithi173_74914_object90976:head# 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2065349:::benchmark_data_smithi173_74914_object87777:head# 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f20662e6:::benchmark_data_smithi173_74914_object70957:head# 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f206f32f:::benchmark_data_smithi173_74914_object64188:head# 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f20722fd:::benchmark_data_smithi173_74914_object81915:head# 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.797 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f20b3702:::benchmark_data_smithi173_74914_object72175:head# 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f20b9b9d:::benchmark_data_smithi173_74914_object71464:head# 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f21234c8:::benchmark_data_smithi173_74914_object58539:head# 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2139c54:::benchmark_data_smithi173_74914_object77881:head# 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f216d928:::benchmark_data_smithi173_74914_object79688:head# 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2175e34:::benchmark_data_smithi173_74914_object82247:head# 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.798 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f219313e:::benchmark_data_smithi173_74914_object97340:head# 2024-08-30T23:37:05.799 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:05.799 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2198803:::benchmark_data_smithi173_74914_object56192:head# 2024-08-30T23:37:05.800 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.800 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f21b3e3b:::benchmark_data_smithi173_74914_object64211:head# 2024-08-30T23:37:05.800 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.801 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f21bb4d5:::benchmark_data_smithi173_74914_object90318:head# 2024-08-30T23:37:05.801 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.801 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f21be7fd:::benchmark_data_smithi173_74914_object64947:head# 2024-08-30T23:37:05.802 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.802 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f21cf402:::benchmark_data_smithi173_74914_object92443:head# 2024-08-30T23:37:05.802 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.805 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f21fb058:::benchmark_data_smithi173_74914_object91921:head# 2024-08-30T23:37:05.805 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.805 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f22369a8:::benchmark_data_smithi173_74914_object77963:head# 2024-08-30T23:37:05.805 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.805 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2249769:::benchmark_data_smithi173_74914_object64151:head# 2024-08-30T23:37:05.805 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f224d6f7:::benchmark_data_smithi173_74914_object80148:head# 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2264318:::benchmark_data_smithi173_74914_object86556:head# 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2266ebf:::benchmark_data_smithi173_74914_object92946:head# 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f22694bb:::benchmark_data_smithi173_74914_object73960:head# 2024-08-30T23:37:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f226cf71:::benchmark_data_smithi173_74914_object61080:head# 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2273acf:::benchmark_data_smithi173_74914_object56653:head# 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f229f9a4:::benchmark_data_smithi173_74914_object75725:head# 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.809 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f22e225e:::benchmark_data_smithi173_74914_object97457:head# 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f22e576f:::benchmark_data_smithi173_74914_object84293:head# 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2300909:::benchmark_data_smithi173_74914_object71090:head# 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2333c23:::benchmark_data_smithi173_74914_object80520:head# 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2353026:::benchmark_data_smithi173_74914_object79585:head# 2024-08-30T23:37:05.810 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2353ec1:::benchmark_data_smithi173_74914_object88129:head# 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2358996:::benchmark_data_smithi173_74914_object54437:head# 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f237585a:::benchmark_data_smithi173_74914_object60257:head# 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2380762:::benchmark_data_smithi173_74914_object79792:head# 2024-08-30T23:37:05.811 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.812 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f239d4f2:::benchmark_data_smithi173_74914_object92147:head# 2024-08-30T23:37:05.812 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.812 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f23d0169:::benchmark_data_smithi173_74914_object69653:head# 2024-08-30T23:37:05.812 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.821 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f23dce8e:::benchmark_data_smithi173_74914_object80712:head# 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f23fe6f3:::benchmark_data_smithi173_74914_object93026:head# 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2418e0f:::benchmark_data_smithi173_74914_object94252:head# 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f24445ec:::benchmark_data_smithi173_74914_object78241:head# 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f24448cc:::benchmark_data_smithi173_74914_object77696:head# 2024-08-30T23:37:05.822 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f245008a:::benchmark_data_smithi173_74914_object77030:head# 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2455d04:::benchmark_data_smithi173_74914_object88772:head# 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2472fe6:::benchmark_data_smithi173_74914_object93710:head# 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f247674b:::benchmark_data_smithi173_74914_object55903:head# 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f247f3fe:::benchmark_data_smithi173_74914_object81393:head# 2024-08-30T23:37:05.823 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2493c99:::benchmark_data_smithi173_74914_object65763:head# 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f24a27dd:::benchmark_data_smithi173_74914_object92307:head# 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f24b465a:::benchmark_data_smithi173_74914_object67760:head# 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f24c390c:::benchmark_data_smithi173_74914_object57614:head# 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.824 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f24c7905:::benchmark_data_smithi173_74914_object96128:head# 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f25401c3:::benchmark_data_smithi173_74914_object80984:head# 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f254bdfb:::benchmark_data_smithi173_74914_object94429:head# 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f254e8e4:::benchmark_data_smithi173_74914_object69808:head# 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f256e5b3:::benchmark_data_smithi173_74914_object93220:head# 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2574f42:::benchmark_data_smithi173_74914_object66181:head# 2024-08-30T23:37:05.825 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f25ae307:::benchmark_data_smithi173_74914_object84634:head# 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f25cca5d:::benchmark_data_smithi173_74914_object97799:head# 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f25cdc70:::benchmark_data_smithi173_74914_object84928:head# 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2602c98:::benchmark_data_smithi173_74914_object87356:head# 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f260e92a:::benchmark_data_smithi173_74914_object96973:head# 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f262b873:::benchmark_data_smithi173_74914_object88151:head# 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.826 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2638f9f:::benchmark_data_smithi173_74914_object74739:head# 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f264699c:::benchmark_data_smithi173_74914_object87335:head# 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2657b76:::benchmark_data_smithi173_74914_object56405:head# 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f265b482:::benchmark_data_smithi173_74914_object87254:head# 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f265b7d6:::benchmark_data_smithi173_74914_object66441:head# 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f267011b:::benchmark_data_smithi173_74914_object65722:head# 2024-08-30T23:37:05.827 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2687575:::benchmark_data_smithi173_74914_object96741:head# 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f268b1d5:::benchmark_data_smithi173_74914_object88312:head# 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f269d078:::benchmark_data_smithi173_74914_object73085:head# 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f26a0a14:::benchmark_data_smithi173_74914_object90458:head# 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f26c7070:::benchmark_data_smithi173_74914_object88307:head# 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.828 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f26c9f2d:::benchmark_data_smithi173_74914_object97580:head# 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f26cec92:::benchmark_data_smithi173_74914_object67319:head# 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f26d1bd9:::benchmark_data_smithi173_74914_object63155:head# 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f270cb8b:::benchmark_data_smithi173_74914_object59047:head# 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f271e754:::benchmark_data_smithi173_74914_object61814:head# 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f276b74d:::benchmark_data_smithi173_74914_object93547:head# 2024-08-30T23:37:05.829 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/186663640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:05.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:05 smithi173 ceph-mon[20554]: pgmap v914: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 845 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 4773/40548 objects degraded (11.771%) 2024-08-30T23:37:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:05.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:05.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:05.839 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2773e9e:::benchmark_data_smithi173_74914_object89785:head# 2024-08-30T23:37:05.839 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.839 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2777800:::benchmark_data_smithi173_74914_object66397:head# 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f277e028:::benchmark_data_smithi173_74914_object69709:head# 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2780774:::benchmark_data_smithi173_74914_object92290:head# 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2781217:::benchmark_data_smithi173_74914_object89592:head# 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f27940bc:::benchmark_data_smithi173_74914_object91897:head# 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f279fb98:::benchmark_data_smithi173_74914_object74210:head# 2024-08-30T23:37:05.840 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f27b1041:::benchmark_data_smithi173_74914_object78629:head# 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f27d5092:::benchmark_data_smithi173_74914_object71385:head# 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f27e5e25:::benchmark_data_smithi173_74914_object90492:head# 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f27fdd29:::benchmark_data_smithi173_74914_object77099:head# 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f281225b:::benchmark_data_smithi173_74914_object75909:head# 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2819077:::benchmark_data_smithi173_74914_object56431:head# 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.841 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2824bc0:::benchmark_data_smithi173_74914_object57107:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2838565:::benchmark_data_smithi173_74914_object74949:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f283c1e8:::benchmark_data_smithi173_74914_object79145:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2840922:::benchmark_data_smithi173_74914_object90132:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f285d8a1:::benchmark_data_smithi173_74914_object85932:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2861d84:::benchmark_data_smithi173_74914_object63370:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f287109b:::benchmark_data_smithi173_74914_object53870:head# 2024-08-30T23:37:05.842 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f28a5e72:::benchmark_data_smithi173_74914_object61331:head# 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f28a77cd:::benchmark_data_smithi173_74914_object58004:head# 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f28c6fee:::benchmark_data_smithi173_74914_object75745:head# 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f28e3eef:::benchmark_data_smithi173_74914_object75278:head# 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f28f7bc3:::benchmark_data_smithi173_74914_object78124:head# 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.843 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f28fed99:::benchmark_data_smithi173_74914_object89970:head# 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f294e112:::benchmark_data_smithi173_74914_object66229:head# 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f297fdde:::benchmark_data_smithi173_74914_object61010:head# 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f298909e:::benchmark_data_smithi173_74914_object57259:head# 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f29b4418:::benchmark_data_smithi173_74914_object63050:head# 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f29ebadb:::benchmark_data_smithi173_74914_object73950:head# 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.844 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a021a6:::benchmark_data_smithi173_74914_object84961:head# 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a1cb7f:::benchmark_data_smithi173_74914_object93297:head# 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a22022:::benchmark_data_smithi173_74914_object55007:head# 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a2b5cd:::benchmark_data_smithi173_74914_object61365:head# 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a50b51:::benchmark_data_smithi173_74914_object55246:head# 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a5897c:::benchmark_data_smithi173_74914_object77957:head# 2024-08-30T23:37:05.845 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2a9dc25:::benchmark_data_smithi173_74914_object87714:head# 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2abf328:::benchmark_data_smithi173_74914_object52824:head# 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ac9db8:::benchmark_data_smithi173_74914_object54809:head# 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2adab4c:::benchmark_data_smithi173_74914_object91229:head# 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.846 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2af44a2:::benchmark_data_smithi173_74914_object77326:head# 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b064c2:::benchmark_data_smithi173_74914_object52825:head# 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b23cc8:::benchmark_data_smithi173_74914_object75846:head# 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b26fde:::benchmark_data_smithi173_74914_object56293:head# 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b59b05:::benchmark_data_smithi173_74914_object65267:head# 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.847 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b66538:::benchmark_data_smithi173_74914_object78836:head# 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b6c5a1:::benchmark_data_smithi173_74914_object60181:head# 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b9638a:::benchmark_data_smithi173_74914_object94426:head# 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b993d9:::benchmark_data_smithi173_74914_object69568:head# 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2b9f3bc:::benchmark_data_smithi173_74914_object67629:head# 2024-08-30T23:37:05.848 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.849 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ba4178:::benchmark_data_smithi173_74914_object66994:head# 2024-08-30T23:37:05.849 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.849 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2bacb01:::benchmark_data_smithi173_74914_object61489:head# 2024-08-30T23:37:05.849 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.858 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2bb7d33:::benchmark_data_smithi173_74914_object85656:head# 2024-08-30T23:37:05.858 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:05.858 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2bca998:::benchmark_data_smithi173_74914_object74098:head# 2024-08-30T23:37:06.012 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2bdedf1:::benchmark_data_smithi173_74914_object91034:head# 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c0c7eb:::benchmark_data_smithi173_74914_object88666:head# 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c31f4c:::benchmark_data_smithi173_74914_object81858:head# 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c64b02:::benchmark_data_smithi173_74914_object84989:head# 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.013 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c7c647:::benchmark_data_smithi173_74914_object73884:head# 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c81aaa:::benchmark_data_smithi173_74914_object95218:head# 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c91d24:::benchmark_data_smithi173_74914_object56298:head# 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2c94f3f:::benchmark_data_smithi173_74914_object66866:head# 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2cb93e7:::benchmark_data_smithi173_74914_object60182:head# 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2cbd97f:::benchmark_data_smithi173_74914_object62784:head# 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.014 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ccbeb3:::benchmark_data_smithi173_74914_object62417:head# 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2cde177:::benchmark_data_smithi173_74914_object79056:head# 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2cf268e:::benchmark_data_smithi173_74914_object78031:head# 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2d0f7de:::benchmark_data_smithi173_74914_object54235:head# 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2d17dbc:::benchmark_data_smithi173_74914_object67126:head# 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2d2dd0b:::benchmark_data_smithi173_74914_object68970:head# 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.015 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2d71ff1:::benchmark_data_smithi173_74914_object92970:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2d8f566:::benchmark_data_smithi173_74914_object54574:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2d91bfa:::benchmark_data_smithi173_74914_object70115:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2daa94e:::benchmark_data_smithi173_74914_object66273:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2db2297:::benchmark_data_smithi173_74914_object85610:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ded113:::benchmark_data_smithi173_74914_object69164:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e1738a:::benchmark_data_smithi173_74914_object54880:head# 2024-08-30T23:37:06.016 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e22f11:::benchmark_data_smithi173_74914_object73388:head# 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e315b8:::benchmark_data_smithi173_74914_object54742:head# 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e3474d:::benchmark_data_smithi173_74914_object83672:head# 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e4ec51:::benchmark_data_smithi173_74914_object67077:head# 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e8692e:::benchmark_data_smithi173_74914_object63602:head# 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e872ba:::benchmark_data_smithi173_74914_object60947:head# 2024-08-30T23:37:06.017 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2e8e5c5:::benchmark_data_smithi173_74914_object94219:head# 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ebea63:::benchmark_data_smithi173_74914_object59363:head# 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ec09ad:::benchmark_data_smithi173_74914_object76225:head# 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ed3dcd:::benchmark_data_smithi173_74914_object59234:head# 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ee73ec:::benchmark_data_smithi173_74914_object86261:head# 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ee7700:::benchmark_data_smithi173_74914_object68620:head# 2024-08-30T23:37:06.018 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2eebe4c:::benchmark_data_smithi173_74914_object54451:head# 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2ef1b7f:::benchmark_data_smithi173_74914_object74523:head# 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f02481:::benchmark_data_smithi173_74914_object78649:head# 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f048a6:::benchmark_data_smithi173_74914_object68402:head# 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f0eccb:::benchmark_data_smithi173_74914_object93110:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f2a9c0:::benchmark_data_smithi173_74914_object88336:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f46fa2:::benchmark_data_smithi173_74914_object78216:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f9483e:::benchmark_data_smithi173_74914_object88145:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2f9d5b8:::benchmark_data_smithi173_74914_object53393:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2fa2867:::benchmark_data_smithi173_74914_object86865:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2fa9d9f:::benchmark_data_smithi173_74914_object58935:head# 2024-08-30T23:37:06.020 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2fc0f98:::benchmark_data_smithi173_74914_object66888:head# 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f2fc835d:::benchmark_data_smithi173_74914_object85237:head# 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f304f0c6:::benchmark_data_smithi173_74914_object53226:head# 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3057325:::benchmark_data_smithi173_74914_object71258:head# 2024-08-30T23:37:06.021 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:06.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/186663640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:06.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:05 smithi136 ceph-mon[20207]: pgmap v914: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 845 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 929 op/s; 4773/40548 objects degraded (11.771%) 2024-08-30T23:37:06.124 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f309c4cf:::benchmark_data_smithi173_74914_object71255:head# 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f30b0ad4:::benchmark_data_smithi173_74914_object74522:head# 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f30b42d0:::benchmark_data_smithi173_74914_object57952:head# 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f30ba251:::benchmark_data_smithi173_74914_object77612:head# 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f30d6a14:::benchmark_data_smithi173_74914_object81769:head# 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f30de516:::benchmark_data_smithi173_74914_object81357:head# 2024-08-30T23:37:06.125 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31050f3:::benchmark_data_smithi173_74914_object90705:head# 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3151e72:::benchmark_data_smithi173_74914_object79177:head# 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f317164a:::benchmark_data_smithi173_74914_object86304:head# 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3175982:::benchmark_data_smithi173_74914_object69258:head# 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f319bf25:::benchmark_data_smithi173_74914_object95438:head# 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31a7772:::benchmark_data_smithi173_74914_object86847:head# 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31c0cc9:::benchmark_data_smithi173_74914_object60806:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31cfc96:::benchmark_data_smithi173_74914_object97222:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31d0d73:::benchmark_data_smithi173_74914_object67677:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31d5be8:::benchmark_data_smithi173_74914_object81362:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31eea26:::benchmark_data_smithi173_74914_object90143:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f31f289c:::benchmark_data_smithi173_74914_object69760:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f320488f:::benchmark_data_smithi173_74914_object91984:head# 2024-08-30T23:37:06.127 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3219f82:::benchmark_data_smithi173_74914_object60381:head# 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f321a884:::benchmark_data_smithi173_74914_object65303:head# 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3228bd2:::benchmark_data_smithi173_74914_object88455:head# 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f324f6b5:::benchmark_data_smithi173_74914_object53540:head# 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3258412:::benchmark_data_smithi173_74914_object96133:head# 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f32618c4:::benchmark_data_smithi173_74914_object78648:head# 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f327c793:::benchmark_data_smithi173_74914_object74751:head# 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f327f012:::benchmark_data_smithi173_74914_object87177:head# 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f32ad58f:::benchmark_data_smithi173_74914_object60511:head# 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f32be216:::benchmark_data_smithi173_74914_object67483:head# 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f32c24d7:::benchmark_data_smithi173_74914_object64234:head# 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.129 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f32cdf52:::benchmark_data_smithi173_74914_object62819:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3318342:::benchmark_data_smithi173_74914_object70649:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f332d353:::benchmark_data_smithi173_74914_object68924:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3333cb7:::benchmark_data_smithi173_74914_object94048:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f333da0b:::benchmark_data_smithi173_74914_object56847:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f33648fa:::benchmark_data_smithi173_74914_object58946:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3381ff3:::benchmark_data_smithi173_74914_object56861:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f338934c:::benchmark_data_smithi173_74914_object78709:head# 2024-08-30T23:37:06.130 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f33b1ca8:::benchmark_data_smithi173_74914_object68800:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f33c8875:::benchmark_data_smithi173_74914_object53351:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f33d8967:::benchmark_data_smithi173_74914_object84273:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f33f9204:::benchmark_data_smithi173_74914_object60505:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3429b09:::benchmark_data_smithi173_74914_object74691:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3443ee1:::benchmark_data_smithi173_74914_object77896:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f345ebcc:::benchmark_data_smithi173_74914_object67967:head# 2024-08-30T23:37:06.131 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f34753f7:::benchmark_data_smithi173_74914_object58232:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f349b258:::benchmark_data_smithi173_74914_object89784:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f34aba90:::benchmark_data_smithi173_74914_object64562:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f34b9457:::benchmark_data_smithi173_74914_object84103:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f34d971d:::benchmark_data_smithi173_74914_object56229:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35053b1:::benchmark_data_smithi173_74914_object97020:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f352b44b:::benchmark_data_smithi173_74914_object59617:head# 2024-08-30T23:37:06.132 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35540b0:::benchmark_data_smithi173_74914_object90386:head# 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f356845d:::benchmark_data_smithi173_74914_object85013:head# 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3580539:::benchmark_data_smithi173_74914_object53099:head# 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3586d30:::benchmark_data_smithi173_74914_object64914:head# 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35954df:::benchmark_data_smithi173_74914_object55679:head# 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35a146a:::benchmark_data_smithi173_74914_object52939:head# 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.133 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35af60a:::benchmark_data_smithi173_74914_object91926:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35bb1ab:::benchmark_data_smithi173_74914_object91927:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35c02ce:::benchmark_data_smithi173_74914_object88290:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f35f2467:::benchmark_data_smithi173_74914_object85790:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f361966f:::benchmark_data_smithi173_74914_object79456:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3625d5b:::benchmark_data_smithi173_74914_object64825:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f367e5eb:::benchmark_data_smithi173_74914_object61536:head# 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.134 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3687607:::benchmark_data_smithi173_74914_object94437:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f369292a:::benchmark_data_smithi173_74914_object84572:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f369393c:::benchmark_data_smithi173_74914_object74397:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f369504a:::benchmark_data_smithi173_74914_object70775:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f36bb988:::benchmark_data_smithi173_74914_object80554:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f36c6a60:::benchmark_data_smithi173_74914_object90660:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f36e763a:::benchmark_data_smithi173_74914_object77872:head# 2024-08-30T23:37:06.135 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3700d44:::benchmark_data_smithi173_74914_object71148:head# 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3705b86:::benchmark_data_smithi173_74914_object85111:head# 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3723b10:::benchmark_data_smithi173_74914_object66754:head# 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f374a31d:::benchmark_data_smithi173_74914_object71111:head# 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3755a83:::benchmark_data_smithi173_74914_object91997:head# 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3778a82:::benchmark_data_smithi173_74914_object72274:head# 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.136 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f377d3f5:::benchmark_data_smithi173_74914_object58252:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f37bf3e4:::benchmark_data_smithi173_74914_object77348:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f37ee217:::benchmark_data_smithi173_74914_object58081:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f37f8d66:::benchmark_data_smithi173_74914_object79941:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f37f9233:::benchmark_data_smithi173_74914_object64689:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f38036f6:::benchmark_data_smithi173_74914_object68917:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f381ac2e:::benchmark_data_smithi173_74914_object55072:head# 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.137 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f383061f:::benchmark_data_smithi173_74914_object82313:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f38733ad:::benchmark_data_smithi173_74914_object59092:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f387e414:::benchmark_data_smithi173_74914_object83147:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f387f36a:::benchmark_data_smithi173_74914_object84782:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f38951a1:::benchmark_data_smithi173_74914_object57224:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f38d9f57:::benchmark_data_smithi173_74914_object58096:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f390c3b2:::benchmark_data_smithi173_74914_object78353:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f39236ba:::benchmark_data_smithi173_74914_object63386:head# 2024-08-30T23:37:06.138 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f39377db:::benchmark_data_smithi173_74914_object57504:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3938151:::benchmark_data_smithi173_74914_object79465:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f394c238:::benchmark_data_smithi173_74914_object82662:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f398801d:::benchmark_data_smithi173_74914_object95838:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f39daa3a:::benchmark_data_smithi173_74914_object98230:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f39ddd3c:::benchmark_data_smithi173_74914_object53704:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f39f3f7f:::benchmark_data_smithi173_74914_object59989:head# 2024-08-30T23:37:06.139 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a1cdd0:::benchmark_data_smithi173_74914_object64694:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a34bc1:::benchmark_data_smithi173_74914_object67414:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a553a3:::benchmark_data_smithi173_74914_object74330:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a6bbeb:::benchmark_data_smithi173_74914_object71504:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a6edbd:::benchmark_data_smithi173_74914_object89287:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a77496:::benchmark_data_smithi173_74914_object72727:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3a9497b:::benchmark_data_smithi173_74914_object77164:head# 2024-08-30T23:37:06.140 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3aa258b:::benchmark_data_smithi173_74914_object78790:head# 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3abc660:::benchmark_data_smithi173_74914_object74685:head# 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3b1d432:::benchmark_data_smithi173_74914_object88409:head# 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3b7447a:::benchmark_data_smithi173_74914_object90206:head# 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3bb9755:::benchmark_data_smithi173_74914_object82501:head# 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3bf3b6b:::benchmark_data_smithi173_74914_object97847:head# 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.141 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3bfdfdc:::benchmark_data_smithi173_74914_object73794:head# 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3c23b30:::benchmark_data_smithi173_74914_object63786:head# 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3c27bab:::benchmark_data_smithi173_74914_object57309:head# 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3c759c1:::benchmark_data_smithi173_74914_object75264:head# 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3c91cd7:::benchmark_data_smithi173_74914_object81443:head# 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3c96953:::benchmark_data_smithi173_74914_object97469:head# 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.142 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3cbcb0a:::benchmark_data_smithi173_74914_object63752:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ccf8c2:::benchmark_data_smithi173_74914_object67840:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3cddc00:::benchmark_data_smithi173_74914_object70885:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ceea7d:::benchmark_data_smithi173_74914_object57209:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3d28edb:::benchmark_data_smithi173_74914_object63462:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3d29497:::benchmark_data_smithi173_74914_object75711:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3d486bb:::benchmark_data_smithi173_74914_object57171:head# 2024-08-30T23:37:06.143 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3d7e955:::benchmark_data_smithi173_74914_object58742:head# 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3d89ebc:::benchmark_data_smithi173_74914_object74700:head# 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3d8f7ce:::benchmark_data_smithi173_74914_object81114:head# 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3da3b2e:::benchmark_data_smithi173_74914_object67600:head# 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3dc09a1:::benchmark_data_smithi173_74914_object54127:head# 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3dc8216:::benchmark_data_smithi173_74914_object98193:head# 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.144 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3def60a:::benchmark_data_smithi173_74914_object98065:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3e15082:::benchmark_data_smithi173_74914_object97466:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3e2b614:::benchmark_data_smithi173_74914_object84624:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3e4a541:::benchmark_data_smithi173_74914_object66767:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3e66696:::benchmark_data_smithi173_74914_object67724:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3e9a4d1:::benchmark_data_smithi173_74914_object71947:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ea7b05:::benchmark_data_smithi173_74914_object88573:head# 2024-08-30T23:37:06.145 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3eaa19a:::benchmark_data_smithi173_74914_object56667:head# 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ec3037:::benchmark_data_smithi173_74914_object97822:head# 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ec44ea:::benchmark_data_smithi173_74914_object78566:head# 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ed4cbb:::benchmark_data_smithi173_74914_object89159:head# 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ee0aba:::benchmark_data_smithi173_74914_object88975:head# 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3eec947:::benchmark_data_smithi173_74914_object63764:head# 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.146 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3ef8080:::benchmark_data_smithi173_74914_object61960:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3efcc62:::benchmark_data_smithi173_74914_object57829:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3f1a002:::benchmark_data_smithi173_74914_object55704:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3f1f5df:::benchmark_data_smithi173_74914_object62389:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3f2a88a:::benchmark_data_smithi173_74914_object93295:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3f69ef6:::benchmark_data_smithi173_74914_object82253:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3f73dda:::benchmark_data_smithi173_74914_object88552:head# 2024-08-30T23:37:06.147 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3f8c3b3:::benchmark_data_smithi173_74914_object77805:head# 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fa6827:::benchmark_data_smithi173_74914_object88206:head# 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fa9340:::benchmark_data_smithi173_74914_object77804:head# 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fbfd0a:::benchmark_data_smithi173_74914_object61747:head# 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fc24c0:::benchmark_data_smithi173_74914_object89655:head# 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fdfe41:::benchmark_data_smithi173_74914_object66016:head# 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.148 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fe563e:::benchmark_data_smithi173_74914_object87584:head# 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f3fee8e0:::benchmark_data_smithi173_74914_object90702:head# 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4008663:::benchmark_data_smithi173_74914_object89519:head# 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f40283a3:::benchmark_data_smithi173_74914_object97513:head# 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4030238:::benchmark_data_smithi173_74914_object88391:head# 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f404ed43:::benchmark_data_smithi173_74914_object93828:head# 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.149 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f405f434:::benchmark_data_smithi173_74914_object79780:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f405f526:::benchmark_data_smithi173_74914_object55836:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f406539a:::benchmark_data_smithi173_74914_object74961:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4084e9c:::benchmark_data_smithi173_74914_object92662:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f408698b:::benchmark_data_smithi173_74914_object70148:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4095a75:::benchmark_data_smithi173_74914_object70882:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f40a0db2:::benchmark_data_smithi173_74914_object89827:head# 2024-08-30T23:37:06.150 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f40abdf1:::benchmark_data_smithi173_74914_object84868:head# 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f40ac2cf:::benchmark_data_smithi173_74914_object85702:head# 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f40b2a93:::benchmark_data_smithi173_74914_object71780:head# 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f40cca28:::benchmark_data_smithi173_74914_object75608:head# 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4100835:::benchmark_data_smithi173_74914_object56091:head# 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f410a41a:::benchmark_data_smithi173_74914_object97579:head# 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.151 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4114569:::benchmark_data_smithi173_74914_object56922:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4136715:::benchmark_data_smithi173_74914_object54372:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f413c223:::benchmark_data_smithi173_74914_object87585:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f413d6a8:::benchmark_data_smithi173_74914_object85327:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f41ba510:::benchmark_data_smithi173_74914_object54673:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f41d3976:::benchmark_data_smithi173_74914_object58902:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f41d62be:::benchmark_data_smithi173_74914_object85100:head# 2024-08-30T23:37:06.152 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f41e6b75:::benchmark_data_smithi173_74914_object84142:head# 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4208a9b:::benchmark_data_smithi173_74914_object95426:head# 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f420e070:::benchmark_data_smithi173_74914_object71425:head# 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f422aed1:::benchmark_data_smithi173_74914_object69548:head# 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f424879a:::benchmark_data_smithi173_74914_object97363:head# 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f424a2da:::benchmark_data_smithi173_74914_object89215:head# 2024-08-30T23:37:06.153 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f424bf7c:::benchmark_data_smithi173_74914_object60522:head# 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f425a930:::benchmark_data_smithi173_74914_object93775:head# 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f427dbee:::benchmark_data_smithi173_74914_object90136:head# 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4291d7e:::benchmark_data_smithi173_74914_object82614:head# 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f42c4647:::benchmark_data_smithi173_74914_object88011:head# 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f42c98ad:::benchmark_data_smithi173_74914_object77905:head# 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.154 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f42d6529:::benchmark_data_smithi173_74914_object80272:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f42e8542:::benchmark_data_smithi173_74914_object69995:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f42f0ed4:::benchmark_data_smithi173_74914_object73624:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f42f2c68:::benchmark_data_smithi173_74914_object92680:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f430fab3:::benchmark_data_smithi173_74914_object65237:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f431c137:::benchmark_data_smithi173_74914_object79112:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f432114e:::benchmark_data_smithi173_74914_object97739:head# 2024-08-30T23:37:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f435fd71:::benchmark_data_smithi173_74914_object96060:head# 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f43b3009:::benchmark_data_smithi173_74914_object87726:head# 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f43b377a:::benchmark_data_smithi173_74914_object66933:head# 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f43de8c0:::benchmark_data_smithi173_74914_object55720:head# 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f43f6e20:::benchmark_data_smithi173_74914_object89945:head# 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f44035d6:::benchmark_data_smithi173_74914_object65279:head# 2024-08-30T23:37:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4407ec4:::benchmark_data_smithi173_74914_object56973:head# 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4418f1d:::benchmark_data_smithi173_74914_object77232:head# 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f441a34f:::benchmark_data_smithi173_74914_object79221:head# 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f443725d:::benchmark_data_smithi173_74914_object71049:head# 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f44528da:::benchmark_data_smithi173_74914_object78555:head# 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4456d35:::benchmark_data_smithi173_74914_object94728:head# 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.157 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4458847:::benchmark_data_smithi173_74914_object90226:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4467dd3:::benchmark_data_smithi173_74914_object67670:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f446b4d7:::benchmark_data_smithi173_74914_object85545:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f448280f:::benchmark_data_smithi173_74914_object88906:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4485363:::benchmark_data_smithi173_74914_object61308:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f449d688:::benchmark_data_smithi173_74914_object78712:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f454ff26:::benchmark_data_smithi173_74914_object90117:head# 2024-08-30T23:37:06.158 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f455908f:::benchmark_data_smithi173_74914_object57788:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f457fa5b:::benchmark_data_smithi173_74914_object56079:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f459eb1c:::benchmark_data_smithi173_74914_object64939:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f45b08c1:::benchmark_data_smithi173_74914_object77029:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f45c3cf0:::benchmark_data_smithi173_74914_object90272:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f463c9bb:::benchmark_data_smithi173_74914_object53819:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f463dc91:::benchmark_data_smithi173_74914_object65486:head# 2024-08-30T23:37:06.159 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4662484:::benchmark_data_smithi173_74914_object73816:head# 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4662ba9:::benchmark_data_smithi173_74914_object66897:head# 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f466c551:::benchmark_data_smithi173_74914_object91132:head# 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4695535:::benchmark_data_smithi173_74914_object84521:head# 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f46b0448:::benchmark_data_smithi173_74914_object55113:head# 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f46eec30:::benchmark_data_smithi173_74914_object72963:head# 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.160 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f46f85aa:::benchmark_data_smithi173_74914_object92455:head# 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f472be6b:::benchmark_data_smithi173_74914_object88038:head# 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f475f8c2:::benchmark_data_smithi173_74914_object79513:head# 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f477489f:::benchmark_data_smithi173_74914_object76223:head# 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f47812c3:::benchmark_data_smithi173_74914_object67847:head# 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f47917f4:::benchmark_data_smithi173_74914_object71560:head# 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.161 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f47ca869:::benchmark_data_smithi173_74914_object91660:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f47e49ef:::benchmark_data_smithi173_74914_object64748:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f47ed469:::benchmark_data_smithi173_74914_object98125:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4800c16:::benchmark_data_smithi173_74914_object84181:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f48391ce:::benchmark_data_smithi173_74914_object91748:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4864213:::benchmark_data_smithi173_74914_object82518:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f488167a:::benchmark_data_smithi173_74914_object54527:head# 2024-08-30T23:37:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f488523c:::benchmark_data_smithi173_74914_object68351:head# 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4885bd7:::benchmark_data_smithi173_74914_object89379:head# 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f488b350:::benchmark_data_smithi173_74914_object91989:head# 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f488ba5e:::benchmark_data_smithi173_74914_object87846:head# 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f489dd8b:::benchmark_data_smithi173_74914_object94103:head# 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f48a46c2:::benchmark_data_smithi173_74914_object53407:head# 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.163 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f48e7549:::benchmark_data_smithi173_74914_object98061:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f48eedea:::benchmark_data_smithi173_74914_object78879:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f48f4ac6:::benchmark_data_smithi173_74914_object58886:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f490aac0:::benchmark_data_smithi173_74914_object54352:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f493e4ae:::benchmark_data_smithi173_74914_object63429:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4994f86:::benchmark_data_smithi173_74914_object56123:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f49dae3b:::benchmark_data_smithi173_74914_object78941:head# 2024-08-30T23:37:06.164 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f49ebc6a:::benchmark_data_smithi173_74914_object92523:head# 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4a394b7:::benchmark_data_smithi173_74914_object53416:head# 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4a59c2e:::benchmark_data_smithi173_74914_object79545:head# 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4a688a3:::benchmark_data_smithi173_74914_object52969:head# 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4a78cc0:::benchmark_data_smithi173_74914_object76066:head# 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4a9936b:::benchmark_data_smithi173_74914_object64631:head# 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4aa1cc6:::benchmark_data_smithi173_74914_object60288:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4aa836a:::benchmark_data_smithi173_74914_object59573:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4aba72d:::benchmark_data_smithi173_74914_object72608:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4abc597:::benchmark_data_smithi173_74914_object69757:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4abfec2:::benchmark_data_smithi173_74914_object84893:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4aec6f4:::benchmark_data_smithi173_74914_object68981:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4af5511:::benchmark_data_smithi173_74914_object71216:head# 2024-08-30T23:37:06.166 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b1e680:::benchmark_data_smithi173_74914_object59542:head# 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b21283:::benchmark_data_smithi173_74914_object88730:head# 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b3e689:::benchmark_data_smithi173_74914_object74709:head# 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b4fcbd:::benchmark_data_smithi173_74914_object67814:head# 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b63cdd:::benchmark_data_smithi173_74914_object83424:head# 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b66445:::benchmark_data_smithi173_74914_object96653:head# 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.167 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b75ef4:::benchmark_data_smithi173_74914_object65870:head# 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b84aaf:::benchmark_data_smithi173_74914_object77768:head# 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4b867e6:::benchmark_data_smithi173_74914_object96328:head# 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4baf311:::benchmark_data_smithi173_74914_object97726:head# 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4be5d55:::benchmark_data_smithi173_74914_object92850:head# 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c00cde:::benchmark_data_smithi173_74914_object73351:head# 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.168 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c1f80e:::benchmark_data_smithi173_74914_object61094:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c3ec51:::benchmark_data_smithi173_74914_object90041:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c42fcb:::benchmark_data_smithi173_74914_object94458:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c52d98:::benchmark_data_smithi173_74914_object89397:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c5c6fc:::benchmark_data_smithi173_74914_object54524:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c60ade:::benchmark_data_smithi173_74914_object56516:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c60ba6:::benchmark_data_smithi173_74914_object91499:head# 2024-08-30T23:37:06.169 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c6d836:::benchmark_data_smithi173_74914_object65424:head# 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c7a594:::benchmark_data_smithi173_74914_object57120:head# 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4c80c6b:::benchmark_data_smithi173_74914_object84548:head# 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4caf8dc:::benchmark_data_smithi173_74914_object81071:head# 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4cb65e0:::benchmark_data_smithi173_74914_object74536:head# 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4cc8119:::benchmark_data_smithi173_74914_object94528:head# 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.170 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4cd41c9:::benchmark_data_smithi173_74914_object73711:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4ce3d48:::benchmark_data_smithi173_74914_object80478:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4ce6a12:::benchmark_data_smithi173_74914_object96805:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4cf12a4:::benchmark_data_smithi173_74914_object53290:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4d20164:::benchmark_data_smithi173_74914_object82219:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4d321ca:::benchmark_data_smithi173_74914_object72480:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4d3fd94:::benchmark_data_smithi173_74914_object89581:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4d47b5d:::benchmark_data_smithi173_74914_object72300:head# 2024-08-30T23:37:06.171 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4d530a4:::benchmark_data_smithi173_74914_object52955:head# 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4d9f4db:::benchmark_data_smithi173_74914_object54086:head# 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4daec36:::benchmark_data_smithi173_74914_object59050:head# 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4dba92c:::benchmark_data_smithi173_74914_object87943:head# 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4ddac33:::benchmark_data_smithi173_74914_object89821:head# 2024-08-30T23:37:06.172 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4df25ad:::benchmark_data_smithi173_74914_object76940:head# 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e040c2:::benchmark_data_smithi173_74914_object94097:head# 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e0bd7c:::benchmark_data_smithi173_74914_object96713:head# 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e0dbba:::benchmark_data_smithi173_74914_object96722:head# 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e146ff:::benchmark_data_smithi173_74914_object69737:head# 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e1cee0:::benchmark_data_smithi173_74914_object93159:head# 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.173 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e25e09:::benchmark_data_smithi173_74914_object68408:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e34f0e:::benchmark_data_smithi173_74914_object85099:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e39018:::benchmark_data_smithi173_74914_object90306:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e4306d:::benchmark_data_smithi173_74914_object71685:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e4785d:::benchmark_data_smithi173_74914_object76851:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e4c6d9:::benchmark_data_smithi173_74914_object67291:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e502ad:::benchmark_data_smithi173_74914_object92177:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e5792f:::benchmark_data_smithi173_74914_object73335:head# 2024-08-30T23:37:06.174 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e5a6e3:::benchmark_data_smithi173_74914_object71662:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e755da:::benchmark_data_smithi173_74914_object82418:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e7919b:::benchmark_data_smithi173_74914_object64734:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4e92a9b:::benchmark_data_smithi173_74914_object76474:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4eace47:::benchmark_data_smithi173_74914_object76695:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4eee5cf:::benchmark_data_smithi173_74914_object63122:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4f0a7d4:::benchmark_data_smithi173_74914_object80874:head# 2024-08-30T23:37:06.175 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4f139c9:::benchmark_data_smithi173_74914_object66830:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4f3caec:::benchmark_data_smithi173_74914_object59588:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4f41f54:::benchmark_data_smithi173_74914_object63319:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4f82331:::benchmark_data_smithi173_74914_object98095:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4fad07b:::benchmark_data_smithi173_74914_object63787:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4fad0cd:::benchmark_data_smithi173_74914_object76499:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4fb4f61:::benchmark_data_smithi173_74914_object67354:head# 2024-08-30T23:37:06.176 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4fd76f4:::benchmark_data_smithi173_74914_object68173:head# 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4fe8916:::benchmark_data_smithi173_74914_object86719:head# 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4ff99fc:::benchmark_data_smithi173_74914_object97516:head# 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f4ffa016:::benchmark_data_smithi173_74914_object80995:head# 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5003fe0:::benchmark_data_smithi173_74914_object87396:head# 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50045a2:::benchmark_data_smithi173_74914_object96802:head# 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.177 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f500c9bb:::benchmark_data_smithi173_74914_object97087:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f501dd5f:::benchmark_data_smithi173_74914_object89156:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f504119e:::benchmark_data_smithi173_74914_object94581:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50965e7:::benchmark_data_smithi173_74914_object93315:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50a4fb3:::benchmark_data_smithi173_74914_object86471:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50c3313:::benchmark_data_smithi173_74914_object73121:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50c3361:::benchmark_data_smithi173_74914_object90309:head# 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.178 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50d5f67:::benchmark_data_smithi173_74914_object90577:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50d9e11:::benchmark_data_smithi173_74914_object75958:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f50db352:::benchmark_data_smithi173_74914_object86650:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5117cda:::benchmark_data_smithi173_74914_object77941:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5151c1c:::benchmark_data_smithi173_74914_object96549:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f516393c:::benchmark_data_smithi173_74914_object62328:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5181a1f:::benchmark_data_smithi173_74914_object94366:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f519b723:::benchmark_data_smithi173_74914_object68229:head# 2024-08-30T23:37:06.179 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f51a0807:::benchmark_data_smithi173_74914_object95952:head# 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f51b68ea:::benchmark_data_smithi173_74914_object65779:head# 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f51d9fee:::benchmark_data_smithi173_74914_object72807:head# 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f51e88ee:::benchmark_data_smithi173_74914_object93351:head# 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f521fcea:::benchmark_data_smithi173_74914_object65240:head# 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52271a1:::benchmark_data_smithi173_74914_object87825:head# 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52545a7:::benchmark_data_smithi173_74914_object91258:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52bd0cb:::benchmark_data_smithi173_74914_object96614:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52bd200:::benchmark_data_smithi173_74914_object77672:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52d033c:::benchmark_data_smithi173_74914_object80188:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52f5de4:::benchmark_data_smithi173_74914_object62886:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52fbf9d:::benchmark_data_smithi173_74914_object69965:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f52fc42f:::benchmark_data_smithi173_74914_object60074:head# 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.181 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f530567a:::benchmark_data_smithi173_74914_object86669:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f531d4ad:::benchmark_data_smithi173_74914_object80732:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f53a0f01:::benchmark_data_smithi173_74914_object82353:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f53d2643:::benchmark_data_smithi173_74914_object67476:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f54324a4:::benchmark_data_smithi173_74914_object93214:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f544d340:::benchmark_data_smithi173_74914_object57555:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f546fe3e:::benchmark_data_smithi173_74914_object54850:head# 2024-08-30T23:37:06.182 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5476c58:::benchmark_data_smithi173_74914_object72330:head# 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f547a1a2:::benchmark_data_smithi173_74914_object57409:head# 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5487680:::benchmark_data_smithi173_74914_object95876:head# 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f54bb6fd:::benchmark_data_smithi173_74914_object76711:head# 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f54cd940:::benchmark_data_smithi173_74914_object90393:head# 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f54f3c69:::benchmark_data_smithi173_74914_object90855:head# 2024-08-30T23:37:06.183 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f54fa80d:::benchmark_data_smithi173_74914_object95723:head# 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5509fd8:::benchmark_data_smithi173_74914_object86890:head# 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f550a45a:::benchmark_data_smithi173_74914_object75223:head# 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f552861d:::benchmark_data_smithi173_74914_object83050:head# 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5551f2c:::benchmark_data_smithi173_74914_object56725:head# 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f555d438:::benchmark_data_smithi173_74914_object85419:head# 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.184 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5579171:::benchmark_data_smithi173_74914_object58884:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f557aa12:::benchmark_data_smithi173_74914_object55845:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f558a3f6:::benchmark_data_smithi173_74914_object93310:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f55ae382:::benchmark_data_smithi173_74914_object89174:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f55d7048:::benchmark_data_smithi173_74914_object83628:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f55f9bdd:::benchmark_data_smithi173_74914_object90112:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56141e8:::benchmark_data_smithi173_74914_object84124:head# 2024-08-30T23:37:06.185 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5642bfe:::benchmark_data_smithi173_74914_object90737:head# 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5648110:::benchmark_data_smithi173_74914_object94289:head# 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56615ca:::benchmark_data_smithi173_74914_object59393:head# 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5675ae8:::benchmark_data_smithi173_74914_object75630:head# 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5688000:::benchmark_data_smithi173_74914_object96290:head# 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56aed79:::benchmark_data_smithi173_74914_object70301:head# 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.186 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56c15bc:::benchmark_data_smithi173_74914_object70168:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56d6456:::benchmark_data_smithi173_74914_object71196:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56daebd:::benchmark_data_smithi173_74914_object56774:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56f3c9d:::benchmark_data_smithi173_74914_object54735:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f56fa1bb:::benchmark_data_smithi173_74914_object77844:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f570c6d1:::benchmark_data_smithi173_74914_object53197:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5719bc5:::benchmark_data_smithi173_74914_object91720:head# 2024-08-30T23:37:06.187 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f572e38b:::benchmark_data_smithi173_74914_object82143:head# 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57382b8:::benchmark_data_smithi173_74914_object90285:head# 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57477de:::benchmark_data_smithi173_74914_object59163:head# 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5750064:::benchmark_data_smithi173_74914_object77834:head# 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5760b60:::benchmark_data_smithi173_74914_object73763:head# 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f579078a:::benchmark_data_smithi173_74914_object61555:head# 2024-08-30T23:37:06.188 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57932c2:::benchmark_data_smithi173_74914_object58977:head# 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57a66e5:::benchmark_data_smithi173_74914_object72206:head# 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57bbdeb:::benchmark_data_smithi173_74914_object75217:head# 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57cad02:::benchmark_data_smithi173_74914_object71803:head# 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57dde52:::benchmark_data_smithi173_74914_object74508:head# 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f57ee7c1:::benchmark_data_smithi173_74914_object92081:head# 2024-08-30T23:37:06.189 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5803339:::benchmark_data_smithi173_74914_object56080:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5842fd1:::benchmark_data_smithi173_74914_object71712:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f584e33c:::benchmark_data_smithi173_74914_object54318:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5878bf1:::benchmark_data_smithi173_74914_object77198:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f58a9e19:::benchmark_data_smithi173_74914_object75470:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f58af315:::benchmark_data_smithi173_74914_object79048:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f58bcc09:::benchmark_data_smithi173_74914_object71776:head# 2024-08-30T23:37:06.190 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f58dd3d0:::benchmark_data_smithi173_74914_object85840:head# 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f59201b4:::benchmark_data_smithi173_74914_object89058:head# 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f592989c:::benchmark_data_smithi173_74914_object53694:head# 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5943d9d:::benchmark_data_smithi173_74914_object86061:head# 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5949b23:::benchmark_data_smithi173_74914_object71346:head# 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f595f024:::benchmark_data_smithi173_74914_object83226:head# 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.191 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f59611fa:::benchmark_data_smithi173_74914_object89638:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5963730:::benchmark_data_smithi173_74914_object76464:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5978c91:::benchmark_data_smithi173_74914_object95286:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5988cd5:::benchmark_data_smithi173_74914_object95225:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f59cd8bc:::benchmark_data_smithi173_74914_object85533:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f59d21a3:::benchmark_data_smithi173_74914_object70840:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f59f2c42:::benchmark_data_smithi173_74914_object69658:head# 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.192 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a08c82:::benchmark_data_smithi173_74914_object61029:head# 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a0a926:::benchmark_data_smithi173_74914_object76133:head# 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a17508:::benchmark_data_smithi173_74914_object71126:head# 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a1acaf:::benchmark_data_smithi173_74914_object88375:head# 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a50f23:::benchmark_data_smithi173_74914_object67782:head# 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a52b54:::benchmark_data_smithi173_74914_object97812:head# 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.193 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a5e962:::benchmark_data_smithi173_74914_object72349:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a615bb:::benchmark_data_smithi173_74914_object76774:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a6848a:::benchmark_data_smithi173_74914_object77106:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a6ae43:::benchmark_data_smithi173_74914_object82852:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5a7b671:::benchmark_data_smithi173_74914_object82685:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5abd1f7:::benchmark_data_smithi173_74914_object71510:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5abd71e:::benchmark_data_smithi173_74914_object55132:head# 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.194 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5adc75b:::benchmark_data_smithi173_74914_object58170:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b11822:::benchmark_data_smithi173_74914_object72573:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b3c702:::benchmark_data_smithi173_74914_object80589:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b48251:::benchmark_data_smithi173_74914_object86521:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b5ebd0:::benchmark_data_smithi173_74914_object78293:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b683ea:::benchmark_data_smithi173_74914_object93344:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b68a6d:::benchmark_data_smithi173_74914_object83728:head# 2024-08-30T23:37:06.195 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5b68e62:::benchmark_data_smithi173_74914_object93931:head# 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5bc32a3:::benchmark_data_smithi173_74914_object66197:head# 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5bcaec0:::benchmark_data_smithi173_74914_object92820:head# 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5bdf5cc:::benchmark_data_smithi173_74914_object52954:head# 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5beef25:::benchmark_data_smithi173_74914_object71448:head# 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c0a035:::benchmark_data_smithi173_74914_object96297:head# 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.196 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c356bc:::benchmark_data_smithi173_74914_object81805:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c383a1:::benchmark_data_smithi173_74914_object96832:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c45227:::benchmark_data_smithi173_74914_object81486:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c51cd2:::benchmark_data_smithi173_74914_object84354:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c5b8b3:::benchmark_data_smithi173_74914_object73183:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c64743:::benchmark_data_smithi173_74914_object84904:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c75914:::benchmark_data_smithi173_74914_object63134:head# 2024-08-30T23:37:06.197 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c88ac3:::benchmark_data_smithi173_74914_object84475:head# 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c8fd46:::benchmark_data_smithi173_74914_object85864:head# 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5c9bb22:::benchmark_data_smithi173_74914_object68282:head# 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5ca0651:::benchmark_data_smithi173_74914_object70855:head# 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5cb8351:::benchmark_data_smithi173_74914_object96421:head# 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5cf41c3:::benchmark_data_smithi173_74914_object60716:head# 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.198 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5d101ec:::benchmark_data_smithi173_74914_object90302:head# 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5d31255:::benchmark_data_smithi173_74914_object64761:head# 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5d370b4:::benchmark_data_smithi173_74914_object55280:head# 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5d39b44:::benchmark_data_smithi173_74914_object74318:head# 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5d488e0:::benchmark_data_smithi173_74914_object92440:head# 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5d643f2:::benchmark_data_smithi173_74914_object56300:head# 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5ddc368:::benchmark_data_smithi173_74914_object68174:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5df7ed6:::benchmark_data_smithi173_74914_object84614:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5dfdbc1:::benchmark_data_smithi173_74914_object96925:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e1c055:::benchmark_data_smithi173_74914_object88319:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e23bc3:::benchmark_data_smithi173_74914_object62614:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e2e32a:::benchmark_data_smithi173_74914_object58118:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e3e510:::benchmark_data_smithi173_74914_object97402:head# 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.200 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e67ca9:::benchmark_data_smithi173_74914_object79674:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e6a39d:::benchmark_data_smithi173_74914_object66941:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e78125:::benchmark_data_smithi173_74914_object57446:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e86ad0:::benchmark_data_smithi173_74914_object89975:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5e8bc19:::benchmark_data_smithi173_74914_object76320:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5ec126a:::benchmark_data_smithi173_74914_object62176:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5ed34d4:::benchmark_data_smithi173_74914_object83033:head# 2024-08-30T23:37:06.201 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5edab1f:::benchmark_data_smithi173_74914_object53309:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5f4a7ac:::benchmark_data_smithi173_74914_object58630:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5f7a9cb:::benchmark_data_smithi173_74914_object86850:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5f81d6c:::benchmark_data_smithi173_74914_object53979:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5f89dc3:::benchmark_data_smithi173_74914_object86169:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5fa3fff:::benchmark_data_smithi173_74914_object88511:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5fb761a:::benchmark_data_smithi173_74914_object77152:head# 2024-08-30T23:37:06.202 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5fe9aab:::benchmark_data_smithi173_74914_object72185:head# 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f5ff06e1:::benchmark_data_smithi173_74914_object96185:head# 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f600dd41:::benchmark_data_smithi173_74914_object63717:head# 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f602dbc3:::benchmark_data_smithi173_74914_object88028:head# 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f605525b:::benchmark_data_smithi173_74914_object73225:head# 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60594c5:::benchmark_data_smithi173_74914_object72936:head# 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.203 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6087461:::benchmark_data_smithi173_74914_object92365:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f608c47c:::benchmark_data_smithi173_74914_object89803:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60956e4:::benchmark_data_smithi173_74914_object91367:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6096fcb:::benchmark_data_smithi173_74914_object77161:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60aa141:::benchmark_data_smithi173_74914_object86479:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60ae587:::benchmark_data_smithi173_74914_object69857:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60c5f2e:::benchmark_data_smithi173_74914_object91879:head# 2024-08-30T23:37:06.204 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60ccede:::benchmark_data_smithi173_74914_object89026:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f60d700f:::benchmark_data_smithi173_74914_object71753:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6107ed0:::benchmark_data_smithi173_74914_object81783:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f612aa88:::benchmark_data_smithi173_74914_object94793:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6135c46:::benchmark_data_smithi173_74914_object65317:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f613699b:::benchmark_data_smithi173_74914_object94321:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f613eac8:::benchmark_data_smithi173_74914_object68106:head# 2024-08-30T23:37:06.205 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f613fdea:::benchmark_data_smithi173_74914_object83978:head# 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f614a1fe:::benchmark_data_smithi173_74914_object76858:head# 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6176977:::benchmark_data_smithi173_74914_object82721:head# 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6194fa8:::benchmark_data_smithi173_74914_object85300:head# 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f61b4932:::benchmark_data_smithi173_74914_object73430:head# 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f61cdab3:::benchmark_data_smithi173_74914_object79298:head# 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.206 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f61dcd5f:::benchmark_data_smithi173_74914_object70680:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6214294:::benchmark_data_smithi173_74914_object53926:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f623b9b7:::benchmark_data_smithi173_74914_object86361:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6254539:::benchmark_data_smithi173_74914_object93645:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f62d1b6a:::benchmark_data_smithi173_74914_object79184:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f62f9713:::benchmark_data_smithi173_74914_object71598:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f62fe668:::benchmark_data_smithi173_74914_object69410:head# 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.207 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f630a0d5:::benchmark_data_smithi173_74914_object66488:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f630d805:::benchmark_data_smithi173_74914_object83964:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f630e73f:::benchmark_data_smithi173_74914_object75579:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6315780:::benchmark_data_smithi173_74914_object58957:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f632c85c:::benchmark_data_smithi173_74914_object81457:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f633dd51:::benchmark_data_smithi173_74914_object94031:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6343d6f:::benchmark_data_smithi173_74914_object54173:head# 2024-08-30T23:37:06.208 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6347b4d:::benchmark_data_smithi173_74914_object74410:head# 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6367068:::benchmark_data_smithi173_74914_object89720:head# 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f63753de:::benchmark_data_smithi173_74914_object83898:head# 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6384c24:::benchmark_data_smithi173_74914_object64048:head# 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f63ced43:::benchmark_data_smithi173_74914_object61027:head# 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f63eb2bf:::benchmark_data_smithi173_74914_object79594:head# 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.209 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6413a56:::benchmark_data_smithi173_74914_object56116:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64259ba:::benchmark_data_smithi173_74914_object94155:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6434b6c:::benchmark_data_smithi173_74914_object93121:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6441b3b:::benchmark_data_smithi173_74914_object73647:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6447e5b:::benchmark_data_smithi173_74914_object55622:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f644c5a4:::benchmark_data_smithi173_74914_object58657:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f645d26c:::benchmark_data_smithi173_74914_object54079:head# 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.210 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6496156:::benchmark_data_smithi173_74914_object78870:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64a07a2:::benchmark_data_smithi173_74914_object80521:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64a1ae9:::benchmark_data_smithi173_74914_object70761:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64b82e8:::benchmark_data_smithi173_74914_object61314:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64befa7:::benchmark_data_smithi173_74914_object96221:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64d8828:::benchmark_data_smithi173_74914_object70616:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64eef20:::benchmark_data_smithi173_74914_object89114:head# 2024-08-30T23:37:06.211 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64efce6:::benchmark_data_smithi173_74914_object71198:head# 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f64fef29:::benchmark_data_smithi173_74914_object55475:head# 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f650e7e4:::benchmark_data_smithi173_74914_object70038:head# 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f652cf69:::benchmark_data_smithi173_74914_object69249:head# 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6538dc9:::benchmark_data_smithi173_74914_object65041:head# 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f653b801:::benchmark_data_smithi173_74914_object58691:head# 2024-08-30T23:37:06.212 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f654ef2d:::benchmark_data_smithi173_74914_object78740:head# 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6591611:::benchmark_data_smithi173_74914_object67379:head# 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f65a836e:::benchmark_data_smithi173_74914_object69064:head# 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f65c35aa:::benchmark_data_smithi173_74914_object56230:head# 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f65dc6c5:::benchmark_data_smithi173_74914_object70374:head# 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f65e4e59:::benchmark_data_smithi173_74914_object59583:head# 2024-08-30T23:37:06.213 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f65e8ace:::benchmark_data_smithi173_74914_object79111:head# 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f65e9ec9:::benchmark_data_smithi173_74914_object73903:head# 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6629ad5:::benchmark_data_smithi173_74914_object82075:head# 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6649d2b:::benchmark_data_smithi173_74914_object62219:head# 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6689c63:::benchmark_data_smithi173_74914_object90537:head# 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f668bc58:::benchmark_data_smithi173_74914_object90688:head# 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6691e78:::benchmark_data_smithi173_74914_object82523:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66a6f3a:::benchmark_data_smithi173_74914_object62207:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66a8ba1:::benchmark_data_smithi173_74914_object71708:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66ab2b5:::benchmark_data_smithi173_74914_object79512:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66adb29:::benchmark_data_smithi173_74914_object82683:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66af07c:::benchmark_data_smithi173_74914_object57391:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66ba803:::benchmark_data_smithi173_74914_object98222:head# 2024-08-30T23:37:06.215 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66d83eb:::benchmark_data_smithi173_74914_object91977:head# 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f66fd410:::benchmark_data_smithi173_74914_object80546:head# 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6715ae9:::benchmark_data_smithi173_74914_object84085:head# 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67173b2:::benchmark_data_smithi173_74914_object61225:head# 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f674ecb6:::benchmark_data_smithi173_74914_object81057:head# 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6753abc:::benchmark_data_smithi173_74914_object60138:head# 2024-08-30T23:37:06.216 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f677f2ab:::benchmark_data_smithi173_74914_object56689:head# 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67814d2:::benchmark_data_smithi173_74914_object82064:head# 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f678ce3a:::benchmark_data_smithi173_74914_object61230:head# 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6798f42:::benchmark_data_smithi173_74914_object82120:head# 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67be3e4:::benchmark_data_smithi173_74914_object90832:head# 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67c0aec:::benchmark_data_smithi173_74914_object69746:head# 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67e4ed9:::benchmark_data_smithi173_74914_object56626:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67e71a5:::benchmark_data_smithi173_74914_object60865:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67ebb8e:::benchmark_data_smithi173_74914_object58347:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f67ff6a1:::benchmark_data_smithi173_74914_object79878:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6810e5f:::benchmark_data_smithi173_74914_object84014:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f682b69f:::benchmark_data_smithi173_74914_object94564:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f682d940:::benchmark_data_smithi173_74914_object75732:head# 2024-08-30T23:37:06.218 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f683b5d3:::benchmark_data_smithi173_74914_object73359:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6841f18:::benchmark_data_smithi173_74914_object54780:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68529d6:::benchmark_data_smithi173_74914_object96081:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6866f83:::benchmark_data_smithi173_74914_object89871:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f689dc15:::benchmark_data_smithi173_74914_object72578:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68a6b7a:::benchmark_data_smithi173_74914_object73702:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68aedf2:::benchmark_data_smithi173_74914_object59375:head# 2024-08-30T23:37:06.219 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68c1aa2:::benchmark_data_smithi173_74914_object83031:head# 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68d74ea:::benchmark_data_smithi173_74914_object61621:head# 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68f3659:::benchmark_data_smithi173_74914_object57168:head# 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f68f88b6:::benchmark_data_smithi173_74914_object65286:head# 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6904b0e:::benchmark_data_smithi173_74914_object93873:head# 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f69144ae:::benchmark_data_smithi173_74914_object57167:head# 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.220 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f691caf9:::benchmark_data_smithi173_74914_object92692:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f693e62f:::benchmark_data_smithi173_74914_object60930:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f696835d:::benchmark_data_smithi173_74914_object83490:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f697ca0c:::benchmark_data_smithi173_74914_object75158:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f69845c4:::benchmark_data_smithi173_74914_object54625:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f69adecd:::benchmark_data_smithi173_74914_object60349:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f69af484:::benchmark_data_smithi173_74914_object88830:head# 2024-08-30T23:37:06.221 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f69e5e7c:::benchmark_data_smithi173_74914_object85473:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a25a30:::benchmark_data_smithi173_74914_object74945:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a2a0d2:::benchmark_data_smithi173_74914_object58968:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a39360:::benchmark_data_smithi173_74914_object69217:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a39b74:::benchmark_data_smithi173_74914_object54704:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a3efa2:::benchmark_data_smithi173_74914_object97947:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a552f9:::benchmark_data_smithi173_74914_object74205:head# 2024-08-30T23:37:06.222 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6a5ac7e:::benchmark_data_smithi173_74914_object79054:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6aaf8c7:::benchmark_data_smithi173_74914_object95560:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6aba557:::benchmark_data_smithi173_74914_object80630:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ae1f90:::benchmark_data_smithi173_74914_object86706:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ae664b:::benchmark_data_smithi173_74914_object62444:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ae9e3f:::benchmark_data_smithi173_74914_object97504:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6afb626:::benchmark_data_smithi173_74914_object94970:head# 2024-08-30T23:37:06.223 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6b09027:::benchmark_data_smithi173_74914_object62450:head# 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6b462cd:::benchmark_data_smithi173_74914_object54784:head# 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6b4c155:::benchmark_data_smithi173_74914_object91139:head# 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6b5174e:::benchmark_data_smithi173_74914_object77935:head# 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6b62ac1:::benchmark_data_smithi173_74914_object76454:head# 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bb9cd4:::benchmark_data_smithi173_74914_object55584:head# 2024-08-30T23:37:06.224 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bbb323:::benchmark_data_smithi173_74914_object76146:head# 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bbe29b:::benchmark_data_smithi173_74914_object78448:head# 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bc5ebb:::benchmark_data_smithi173_74914_object96546:head# 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bce0fa:::benchmark_data_smithi173_74914_object73108:head# 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bdbcce:::benchmark_data_smithi173_74914_object78700:head# 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6be8fa1:::benchmark_data_smithi173_74914_object63253:head# 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.225 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6bf28f2:::benchmark_data_smithi173_74914_object61711:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c0aad1:::benchmark_data_smithi173_74914_object54999:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c0c2a6:::benchmark_data_smithi173_74914_object83415:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c137b7:::benchmark_data_smithi173_74914_object61241:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c169d4:::benchmark_data_smithi173_74914_object84080:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c76c4b:::benchmark_data_smithi173_74914_object85796:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c87375:::benchmark_data_smithi173_74914_object84036:head# 2024-08-30T23:37:06.226 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c8846e:::benchmark_data_smithi173_74914_object92603:head# 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c8aa0b:::benchmark_data_smithi173_74914_object83386:head# 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c8d096:::benchmark_data_smithi173_74914_object66824:head# 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6c93979:::benchmark_data_smithi173_74914_object55794:head# 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ca0659:::benchmark_data_smithi173_74914_object58831:head# 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ce0636:::benchmark_data_smithi173_74914_object61328:head# 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.227 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ce7574:::benchmark_data_smithi173_74914_object78453:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ce8075:::benchmark_data_smithi173_74914_object67733:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ce9854:::benchmark_data_smithi173_74914_object65520:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6d005c8:::benchmark_data_smithi173_74914_object53490:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6d03c8c:::benchmark_data_smithi173_74914_object88087:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6d13e1e:::benchmark_data_smithi173_74914_object77973:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6d31835:::benchmark_data_smithi173_74914_object57220:head# 2024-08-30T23:37:06.228 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6d4f187:::benchmark_data_smithi173_74914_object98183:head# 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6d88d4b:::benchmark_data_smithi173_74914_object82996:head# 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6da2b10:::benchmark_data_smithi173_74914_object74981:head# 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6da5bf5:::benchmark_data_smithi173_74914_object58155:head# 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6dcb916:::benchmark_data_smithi173_74914_object88112:head# 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6dddb6e:::benchmark_data_smithi173_74914_object63656:head# 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.229 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e0c62e:::benchmark_data_smithi173_74914_object58495:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e1f408:::benchmark_data_smithi173_74914_object85735:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e29915:::benchmark_data_smithi173_74914_object89048:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e3791e:::benchmark_data_smithi173_74914_object54932:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e4a6bb:::benchmark_data_smithi173_74914_object90923:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e4f964:::benchmark_data_smithi173_74914_object83555:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e6f15d:::benchmark_data_smithi173_74914_object97691:head# 2024-08-30T23:37:06.230 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e75521:::benchmark_data_smithi173_74914_object91387:head# 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e896a8:::benchmark_data_smithi173_74914_object66043:head# 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6e98bf9:::benchmark_data_smithi173_74914_object66115:head# 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6ea1f9d:::benchmark_data_smithi173_74914_object82624:head# 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6eec02c:::benchmark_data_smithi173_74914_object56737:head# 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f0fe0e:::benchmark_data_smithi173_74914_object91812:head# 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.231 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f187ae:::benchmark_data_smithi173_74914_object75949:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f20a7c:::benchmark_data_smithi173_74914_object70575:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f21f33:::benchmark_data_smithi173_74914_object67631:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f29fcc:::benchmark_data_smithi173_74914_object97574:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f3164c:::benchmark_data_smithi173_74914_object88119:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f746b5:::benchmark_data_smithi173_74914_object57837:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f757c3:::benchmark_data_smithi173_74914_object87543:head# 2024-08-30T23:37:06.232 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f7f103:::benchmark_data_smithi173_74914_object72389:head# 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6f945db:::benchmark_data_smithi173_74914_object81372:head# 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f6fd9c36:::benchmark_data_smithi173_74914_object53370:head# 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f700175b:::benchmark_data_smithi173_74914_object54793:head# 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7022f2d:::benchmark_data_smithi173_74914_object85368:head# 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f702bbee:::benchmark_data_smithi173_74914_object60396:head# 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.233 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70377fc:::benchmark_data_smithi173_74914_object58932:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7056527:::benchmark_data_smithi173_74914_object58696:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f705b1ce:::benchmark_data_smithi173_74914_object59702:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7069c46:::benchmark_data_smithi173_74914_object72616:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70a71e4:::benchmark_data_smithi173_74914_object83568:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70c31fb:::benchmark_data_smithi173_74914_object75665:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70d102b:::benchmark_data_smithi173_74914_object53456:head# 2024-08-30T23:37:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70e4c19:::benchmark_data_smithi173_74914_object90720:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70fa5ae:::benchmark_data_smithi173_74914_object92218:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f70fb0d4:::benchmark_data_smithi173_74914_object70814:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f710ee11:::benchmark_data_smithi173_74914_object74378:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f712a60b:::benchmark_data_smithi173_74914_object61954:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7143f68:::benchmark_data_smithi173_74914_object53924:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f715d529:::benchmark_data_smithi173_74914_object57097:head# 2024-08-30T23:37:06.235 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7161514:::benchmark_data_smithi173_74914_object54851:head# 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f71883bd:::benchmark_data_smithi173_74914_object54936:head# 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f71e1c6d:::benchmark_data_smithi173_74914_object62063:head# 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f721bc40:::benchmark_data_smithi173_74914_object62065:head# 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7256dec:::benchmark_data_smithi173_74914_object64807:head# 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7265ac3:::benchmark_data_smithi173_74914_object57448:head# 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.236 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f726ba97:::benchmark_data_smithi173_74914_object65977:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f727b773:::benchmark_data_smithi173_74914_object58964:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f72c4be4:::benchmark_data_smithi173_74914_object90273:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f72c5917:::benchmark_data_smithi173_74914_object90903:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f72e54b2:::benchmark_data_smithi173_74914_object58359:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f72e5508:::benchmark_data_smithi173_74914_object81807:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f72f160b:::benchmark_data_smithi173_74914_object88297:head# 2024-08-30T23:37:06.237 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f72fc7af:::benchmark_data_smithi173_74914_object75957:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f730a2b9:::benchmark_data_smithi173_74914_object79945:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73404f0:::benchmark_data_smithi173_74914_object56057:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7353203:::benchmark_data_smithi173_74914_object78611:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73769e3:::benchmark_data_smithi173_74914_object64256:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f737af3b:::benchmark_data_smithi173_74914_object65228:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73b299f:::benchmark_data_smithi173_74914_object55380:head# 2024-08-30T23:37:06.238 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73b8f0b:::benchmark_data_smithi173_74914_object89988:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73b952b:::benchmark_data_smithi173_74914_object82333:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73d1d63:::benchmark_data_smithi173_74914_object77949:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73ea1ac:::benchmark_data_smithi173_74914_object92910:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73fb576:::benchmark_data_smithi173_74914_object88240:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73fb934:::benchmark_data_smithi173_74914_object64165:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f73fd03d:::benchmark_data_smithi173_74914_object72718:head# 2024-08-30T23:37:06.239 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f740944c:::benchmark_data_smithi173_74914_object63267:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7419444:::benchmark_data_smithi173_74914_object89194:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f744ddcf:::benchmark_data_smithi173_74914_object78525:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f744ecf4:::benchmark_data_smithi173_74914_object70959:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f745c42e:::benchmark_data_smithi173_74914_object55119:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f748e396:::benchmark_data_smithi173_74914_object59146:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f74925c1:::benchmark_data_smithi173_74914_object71278:head# 2024-08-30T23:37:06.240 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f74af8d4:::benchmark_data_smithi173_74914_object72539:head# 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f74e9741:::benchmark_data_smithi173_74914_object93887:head# 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f750aa76:::benchmark_data_smithi173_74914_object87366:head# 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f751eb3c:::benchmark_data_smithi173_74914_object55685:head# 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7541e88:::benchmark_data_smithi173_74914_object87960:head# 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7561d90:::benchmark_data_smithi173_74914_object80511:head# 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.241 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f757eba2:::benchmark_data_smithi173_74914_object77096:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f75a9d89:::benchmark_data_smithi173_74914_object80598:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f75db99a:::benchmark_data_smithi173_74914_object66021:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f75de543:::benchmark_data_smithi173_74914_object88199:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f75eb9fc:::benchmark_data_smithi173_74914_object57374:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f75fcd63:::benchmark_data_smithi173_74914_object65402:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f76080e3:::benchmark_data_smithi173_74914_object82634:head# 2024-08-30T23:37:06.242 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7615a14:::benchmark_data_smithi173_74914_object76270:head# 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7644104:::benchmark_data_smithi173_74914_object92727:head# 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f766bd6e:::benchmark_data_smithi173_74914_object78069:head# 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f76a2795:::benchmark_data_smithi173_74914_object77210:head# 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f76b0fdf:::benchmark_data_smithi173_74914_object88503:head# 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f76b308e:::benchmark_data_smithi173_74914_object91134:head# 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.243 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f76df31a:::benchmark_data_smithi173_74914_object80252:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7700345:::benchmark_data_smithi173_74914_object94929:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f775dff6:::benchmark_data_smithi173_74914_object95152:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f776982b:::benchmark_data_smithi173_74914_object81895:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f777de71:::benchmark_data_smithi173_74914_object61084:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f779c00b:::benchmark_data_smithi173_74914_object80199:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f77c2d5f:::benchmark_data_smithi173_74914_object54903:head# 2024-08-30T23:37:06.244 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f77da231:::benchmark_data_smithi173_74914_object63537:head# 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f77dfdfc:::benchmark_data_smithi173_74914_object73172:head# 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f77f45a1:::benchmark_data_smithi173_74914_object87121:head# 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f781f3e1:::benchmark_data_smithi173_74914_object90754:head# 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7823dca:::benchmark_data_smithi173_74914_object77667:head# 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7882cbe:::benchmark_data_smithi173_74914_object68261:head# 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.245 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f78946d1:::benchmark_data_smithi173_74914_object88683:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f78a5a75:::benchmark_data_smithi173_74914_object73341:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f790c7b5:::benchmark_data_smithi173_74914_object85355:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7918335:::benchmark_data_smithi173_74914_object87011:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7923eae:::benchmark_data_smithi173_74914_object85139:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7928a87:::benchmark_data_smithi173_74914_object82632:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f792e9cf:::benchmark_data_smithi173_74914_object94562:head# 2024-08-30T23:37:06.246 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f795a7af:::benchmark_data_smithi173_74914_object72571:head# 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f796d4e0:::benchmark_data_smithi173_74914_object54918:head# 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f796eaa3:::benchmark_data_smithi173_74914_object69991:head# 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f799722b:::benchmark_data_smithi173_74914_object74661:head# 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f79b004c:::benchmark_data_smithi173_74914_object66539:head# 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f79d521c:::benchmark_data_smithi173_74914_object53634:head# 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.247 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f79da34c:::benchmark_data_smithi173_74914_object59342:head# 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7a19ae3:::benchmark_data_smithi173_74914_object83369:head# 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7a35b68:::benchmark_data_smithi173_74914_object81583:head# 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7a4428a:::benchmark_data_smithi173_74914_object69500:head# 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7a4b370:::benchmark_data_smithi173_74914_object74538:head# 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7a96855:::benchmark_data_smithi173_74914_object72620:head# 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.248 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ac67a2:::benchmark_data_smithi173_74914_object59219:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ad2409:::benchmark_data_smithi173_74914_object56466:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7aeaeac:::benchmark_data_smithi173_74914_object82194:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7af016f:::benchmark_data_smithi173_74914_object74049:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7b0188a:::benchmark_data_smithi173_74914_object91552:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7b53181:::benchmark_data_smithi173_74914_object82979:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7b5cfc8:::benchmark_data_smithi173_74914_object76448:head# 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.249 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7b76403:::benchmark_data_smithi173_74914_object87826:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ba0e12:::benchmark_data_smithi173_74914_object74486:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7bcf7ec:::benchmark_data_smithi173_74914_object71568:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7bd2521:::benchmark_data_smithi173_74914_object97138:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7bddd93:::benchmark_data_smithi173_74914_object56855:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7bf2148:::benchmark_data_smithi173_74914_object58900:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c0ef76:::benchmark_data_smithi173_74914_object87445:head# 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.250 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c1e85a:::benchmark_data_smithi173_74914_object54277:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c269dc:::benchmark_data_smithi173_74914_object71404:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c4000b:::benchmark_data_smithi173_74914_object71169:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c683f9:::benchmark_data_smithi173_74914_object55855:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c7e238:::benchmark_data_smithi173_74914_object87144:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c8f4ae:::benchmark_data_smithi173_74914_object67201:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7c96a67:::benchmark_data_smithi173_74914_object88838:head# 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.251 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ca96ba:::benchmark_data_smithi173_74914_object74972:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d0bf14:::benchmark_data_smithi173_74914_object73032:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d0f3a6:::benchmark_data_smithi173_74914_object85360:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d22cd3:::benchmark_data_smithi173_74914_object66560:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d3af9c:::benchmark_data_smithi173_74914_object64346:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d3b351:::benchmark_data_smithi173_74914_object76691:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d44a6c:::benchmark_data_smithi173_74914_object63869:head# 2024-08-30T23:37:06.252 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d5e897:::benchmark_data_smithi173_74914_object67681:head# 2024-08-30T23:37:06.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d60467:::benchmark_data_smithi173_74914_object64566:head# 2024-08-30T23:37:06.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.253 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7d83f66:::benchmark_data_smithi173_74914_object89068:head# 2024-08-30T23:37:06.253 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7daccd0:::benchmark_data_smithi173_74914_object93509:head# 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7dc0034:::benchmark_data_smithi173_74914_object73520:head# 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7dd38d4:::benchmark_data_smithi173_74914_object72585:head# 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7dd8326:::benchmark_data_smithi173_74914_object96117:head# 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ddd35b:::benchmark_data_smithi173_74914_object53353:head# 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7df3839:::benchmark_data_smithi173_74914_object93962:head# 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.264 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e06e93:::benchmark_data_smithi173_74914_object89248:head# 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e191b1:::benchmark_data_smithi173_74914_object71551:head# 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e21da3:::benchmark_data_smithi173_74914_object57729:head# 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e46ee3:::benchmark_data_smithi173_74914_object55138:head# 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e4770a:::benchmark_data_smithi173_74914_object95884:head# 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e4aeef:::benchmark_data_smithi173_74914_object76479:head# 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.265 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e52813:::benchmark_data_smithi173_74914_object88763:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e63522:::benchmark_data_smithi173_74914_object60676:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e65fe7:::benchmark_data_smithi173_74914_object96841:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e7cda1:::benchmark_data_smithi173_74914_object91018:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e83913:::benchmark_data_smithi173_74914_object74417:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7e948a0:::benchmark_data_smithi173_74914_object94857:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ea96fe:::benchmark_data_smithi173_74914_object92132:head# 2024-08-30T23:37:06.266 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7eac342:::benchmark_data_smithi173_74914_object90439:head# 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ec4b52:::benchmark_data_smithi173_74914_object67273:head# 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7eeec5e:::benchmark_data_smithi173_74914_object85090:head# 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7eefedb:::benchmark_data_smithi173_74914_object55328:head# 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7ef7eef:::benchmark_data_smithi173_74914_object93200:head# 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7efb684:::benchmark_data_smithi173_74914_object68982:head# 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.267 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f0ab1e:::benchmark_data_smithi173_74914_object90007:head# 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f0b60c:::benchmark_data_smithi173_74914_object81848:head# 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f10685:::benchmark_data_smithi173_74914_object82450:head# 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f28cf2:::benchmark_data_smithi173_74914_object79178:head# 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f3feba:::benchmark_data_smithi173_74914_object68357:head# 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f67d39:::benchmark_data_smithi173_74914_object83660:head# 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.268 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f6b7c6:::benchmark_data_smithi173_74914_object89561:head# 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f6c1fb:::benchmark_data_smithi173_74914_object96922:head# 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f701ff:::benchmark_data_smithi173_74914_object67284:head# 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f846a1:::benchmark_data_smithi173_74914_object69245:head# 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7f9beed:::benchmark_data_smithi173_74914_object97754:head# 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7fc5b71:::benchmark_data_smithi173_74914_object71181:head# 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.269 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f7fed6c2:::benchmark_data_smithi173_74914_object81541:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f800c84e:::benchmark_data_smithi173_74914_object67223:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f800ce2d:::benchmark_data_smithi173_74914_object65957:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8018e6b:::benchmark_data_smithi173_74914_object62420:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f802b63c:::benchmark_data_smithi173_74914_object75731:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8037fe5:::benchmark_data_smithi173_74914_object62796:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f805873f:::benchmark_data_smithi173_74914_object73280:head# 2024-08-30T23:37:06.270 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f805b922:::benchmark_data_smithi173_74914_object72409:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f80668c9:::benchmark_data_smithi173_74914_object93197:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f807a327:::benchmark_data_smithi173_74914_object97379:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f80ab1c3:::benchmark_data_smithi173_74914_object90667:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f80aebb7:::benchmark_data_smithi173_74914_object90057:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f80c117a:::benchmark_data_smithi173_74914_object62024:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f80fad36:::benchmark_data_smithi173_74914_object82706:head# 2024-08-30T23:37:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8105553:::benchmark_data_smithi173_74914_object74682:head# 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f810cfa5:::benchmark_data_smithi173_74914_object88910:head# 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f810e706:::benchmark_data_smithi173_74914_object67697:head# 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8131307:::benchmark_data_smithi173_74914_object56012:head# 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8136808:::benchmark_data_smithi173_74914_object83074:head# 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8187216:::benchmark_data_smithi173_74914_object88349:head# 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.272 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8187597:::benchmark_data_smithi173_74914_object71890:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f81a2472:::benchmark_data_smithi173_74914_object56528:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f81b4dcf:::benchmark_data_smithi173_74914_object82802:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f81e2b83:::benchmark_data_smithi173_74914_object81127:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f81e61e6:::benchmark_data_smithi173_74914_object94575:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f820bf2c:::benchmark_data_smithi173_74914_object62384:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f820c9f2:::benchmark_data_smithi173_74914_object74617:head# 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.273 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8217e8a:::benchmark_data_smithi173_74914_object59208:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8221e39:::benchmark_data_smithi173_74914_object95361:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82289fd:::benchmark_data_smithi173_74914_object92229:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82478f3:::benchmark_data_smithi173_74914_object80557:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8262950:::benchmark_data_smithi173_74914_object75813:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82646d1:::benchmark_data_smithi173_74914_object56895:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8268a10:::benchmark_data_smithi173_74914_object89458:head# 2024-08-30T23:37:06.274 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8275f42:::benchmark_data_smithi173_74914_object65891:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82ae115:::benchmark_data_smithi173_74914_object74320:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82c052d:::benchmark_data_smithi173_74914_object56916:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82c5722:::benchmark_data_smithi173_74914_object53026:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82ca11a:::benchmark_data_smithi173_74914_object55152:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82f74f1:::benchmark_data_smithi173_74914_object75161:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82f914c:::benchmark_data_smithi173_74914_object82389:head# 2024-08-30T23:37:06.275 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f82f9be3:::benchmark_data_smithi173_74914_object94196:head# 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f833c863:::benchmark_data_smithi173_74914_object55703:head# 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f833ed26:::benchmark_data_smithi173_74914_object54967:head# 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f833ef71:::benchmark_data_smithi173_74914_object64311:head# 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f835e6d1:::benchmark_data_smithi173_74914_object86979:head# 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f837a114:::benchmark_data_smithi173_74914_object92536:head# 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.276 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f83a2f79:::benchmark_data_smithi173_74914_object92591:head# 2024-08-30T23:37:06.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.277 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f83bc20a:::benchmark_data_smithi173_74914_object54557:head# 2024-08-30T23:37:06.277 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:06.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f83cd6a8:::benchmark_data_smithi173_74914_object56008:head# 2024-08-30T23:37:06.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.278 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f83cdd8e:::benchmark_data_smithi173_74914_object69903:head# 2024-08-30T23:37:06.278 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:06.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:06.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:37:06.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f84442c7:::benchmark_data_smithi173_74914_object91643:head# 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f84842c5:::benchmark_data_smithi173_74914_object92729:head# 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f84b54ea:::benchmark_data_smithi173_74914_object82644:head# 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f84eacf8:::benchmark_data_smithi173_74914_object63779:head# 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8509266:::benchmark_data_smithi173_74914_object62000:head# 2024-08-30T23:37:06.390 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f85213ec:::benchmark_data_smithi173_74914_object96151:head# 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f853415e:::benchmark_data_smithi173_74914_object82121:head# 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8544e50:::benchmark_data_smithi173_74914_object59594:head# 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f856b785:::benchmark_data_smithi173_74914_object92565:head# 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f857a6e5:::benchmark_data_smithi173_74914_object70208:head# 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f85a2998:::benchmark_data_smithi173_74914_object64782:head# 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.391 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f85b1efd:::benchmark_data_smithi173_74914_object73410:head# 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f85bfd0f:::benchmark_data_smithi173_74914_object57539:head# 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f85e4692:::benchmark_data_smithi173_74914_object78776:head# 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8606fe9:::benchmark_data_smithi173_74914_object66634:head# 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f862f789:::benchmark_data_smithi173_74914_object53766:head# 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f862f8f8:::benchmark_data_smithi173_74914_object83051:head# 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.392 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f862f95d:::benchmark_data_smithi173_74914_object86536:head# 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8660cf9:::benchmark_data_smithi173_74914_object88784:head# 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f866b866:::benchmark_data_smithi173_74914_object81346:head# 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8670520:::benchmark_data_smithi173_74914_object69692:head# 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8673acc:::benchmark_data_smithi173_74914_object93267:head# 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8678739:::benchmark_data_smithi173_74914_object70939:head# 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.393 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f86c962f:::benchmark_data_smithi173_74914_object77964:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f86f41f7:::benchmark_data_smithi173_74914_object65669:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8711b7d:::benchmark_data_smithi173_74914_object69771:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f87255d8:::benchmark_data_smithi173_74914_object69227:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8761e46:::benchmark_data_smithi173_74914_object94488:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f877082f:::benchmark_data_smithi173_74914_object79130:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8775fb6:::benchmark_data_smithi173_74914_object92372:head# 2024-08-30T23:37:06.394 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8790973:::benchmark_data_smithi173_74914_object85316:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f87b3d3f:::benchmark_data_smithi173_74914_object67465:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f87cdab7:::benchmark_data_smithi173_74914_object54293:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f87d2a5d:::benchmark_data_smithi173_74914_object62983:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f87e0f88:::benchmark_data_smithi173_74914_object90586:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f87e1dd5:::benchmark_data_smithi173_74914_object82162:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f880323c:::benchmark_data_smithi173_74914_object77976:head# 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.395 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f88288ac:::benchmark_data_smithi173_74914_object96941:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8835c16:::benchmark_data_smithi173_74914_object96460:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f885c061:::benchmark_data_smithi173_74914_object58604:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8865c29:::benchmark_data_smithi173_74914_object59228:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f88a5f83:::benchmark_data_smithi173_74914_object75356:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f88ac552:::benchmark_data_smithi173_74914_object89207:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f88c7e8e:::benchmark_data_smithi173_74914_object68991:head# 2024-08-30T23:37:06.396 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f88d869b:::benchmark_data_smithi173_74914_object88905:head# 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8940940:::benchmark_data_smithi173_74914_object82517:head# 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8957d9c:::benchmark_data_smithi173_74914_object77040:head# 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f895b376:::benchmark_data_smithi173_74914_object92316:head# 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f89705e9:::benchmark_data_smithi173_74914_object55708:head# 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8988cbe:::benchmark_data_smithi173_74914_object72649:head# 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.397 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f898a805:::benchmark_data_smithi173_74914_object56972:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f89ccddd:::benchmark_data_smithi173_74914_object95498:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f89ce1ec:::benchmark_data_smithi173_74914_object88222:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f89df6df:::benchmark_data_smithi173_74914_object58674:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8a09de9:::benchmark_data_smithi173_74914_object54874:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8a1f13b:::benchmark_data_smithi173_74914_object57388:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8a490d6:::benchmark_data_smithi173_74914_object91429:head# 2024-08-30T23:37:06.398 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8a57eb8:::benchmark_data_smithi173_74914_object55563:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8a6cd97:::benchmark_data_smithi173_74914_object71589:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8a7e032:::benchmark_data_smithi173_74914_object97794:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8aa681e:::benchmark_data_smithi173_74914_object81969:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8ac4d66:::benchmark_data_smithi173_74914_object66338:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8ad494d:::benchmark_data_smithi173_74914_object65805:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8aef00a:::benchmark_data_smithi173_74914_object74993:head# 2024-08-30T23:37:06.399 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8affa4d:::benchmark_data_smithi173_74914_object55581:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b00c4d:::benchmark_data_smithi173_74914_object89442:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b0d750:::benchmark_data_smithi173_74914_object82138:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b11ef0:::benchmark_data_smithi173_74914_object96359:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b3d9a2:::benchmark_data_smithi173_74914_object78128:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b55fec:::benchmark_data_smithi173_74914_object73808:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b713f6:::benchmark_data_smithi173_74914_object85198:head# 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.400 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b725e2:::benchmark_data_smithi173_74914_object54592:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8b7bd10:::benchmark_data_smithi173_74914_object90622:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8bc7894:::benchmark_data_smithi173_74914_object59470:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8bc808a:::benchmark_data_smithi173_74914_object62146:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8bce144:::benchmark_data_smithi173_74914_object88288:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8bcea3a:::benchmark_data_smithi173_74914_object83042:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8c1607e:::benchmark_data_smithi173_74914_object84315:head# 2024-08-30T23:37:06.401 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8c2577f:::benchmark_data_smithi173_74914_object83399:head# 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8c391c4:::benchmark_data_smithi173_74914_object65208:head# 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8c6d9c6:::benchmark_data_smithi173_74914_object83745:head# 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8c9e781:::benchmark_data_smithi173_74914_object53207:head# 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8c9eafc:::benchmark_data_smithi173_74914_object62911:head# 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8cbea77:::benchmark_data_smithi173_74914_object80239:head# 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.402 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8cd7f28:::benchmark_data_smithi173_74914_object55799:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8cd8b26:::benchmark_data_smithi173_74914_object66478:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8cec708:::benchmark_data_smithi173_74914_object89213:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8cee423:::benchmark_data_smithi173_74914_object67452:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8d0cd4c:::benchmark_data_smithi173_74914_object89704:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8d0f460:::benchmark_data_smithi173_74914_object81933:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8d3a67a:::benchmark_data_smithi173_74914_object73128:head# 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.403 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8d40da8:::benchmark_data_smithi173_74914_object61963:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8d5b166:::benchmark_data_smithi173_74914_object62996:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8d900a8:::benchmark_data_smithi173_74914_object66546:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8da4e5d:::benchmark_data_smithi173_74914_object82712:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8db4603:::benchmark_data_smithi173_74914_object62392:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8dc0ccc:::benchmark_data_smithi173_74914_object85920:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8dd2b5b:::benchmark_data_smithi173_74914_object71908:head# 2024-08-30T23:37:06.404 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8dd8db0:::benchmark_data_smithi173_74914_object82884:head# 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8dec75a:::benchmark_data_smithi173_74914_object81854:head# 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e537d0:::benchmark_data_smithi173_74914_object55317:head# 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e730da:::benchmark_data_smithi173_74914_object97295:head# 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e76450:::benchmark_data_smithi173_74914_object81364:head# 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e76ac6:::benchmark_data_smithi173_74914_object57123:head# 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.405 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e7bf0e:::benchmark_data_smithi173_74914_object61454:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e89559:::benchmark_data_smithi173_74914_object81384:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8e98df1:::benchmark_data_smithi173_74914_object89360:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8eb6f6b:::benchmark_data_smithi173_74914_object66250:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8eb8915:::benchmark_data_smithi173_74914_object86802:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8edb0c0:::benchmark_data_smithi173_74914_object91579:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8f0eef8:::benchmark_data_smithi173_74914_object55011:head# 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.406 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8f8eef8:::benchmark_data_smithi173_74914_object64191:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8fb2e82:::benchmark_data_smithi173_74914_object88591:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f8fe8ca2:::benchmark_data_smithi173_74914_object57608:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f901411d:::benchmark_data_smithi173_74914_object74956:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f903829b:::benchmark_data_smithi173_74914_object90310:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9046367:::benchmark_data_smithi173_74914_object63257:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f907b5e4:::benchmark_data_smithi173_74914_object53328:head# 2024-08-30T23:37:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9088f49:::benchmark_data_smithi173_74914_object65310:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f909b933:::benchmark_data_smithi173_74914_object90251:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f90cd719:::benchmark_data_smithi173_74914_object65974:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f90ea3ad:::benchmark_data_smithi173_74914_object79272:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f90ffba3:::benchmark_data_smithi173_74914_object94621:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9110afe:::benchmark_data_smithi173_74914_object93299:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f914f5d5:::benchmark_data_smithi173_74914_object90835:head# 2024-08-30T23:37:06.408 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9151ced:::benchmark_data_smithi173_74914_object97245:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f91711c9:::benchmark_data_smithi173_74914_object79793:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f918a8f6:::benchmark_data_smithi173_74914_object68787:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f91a4c00:::benchmark_data_smithi173_74914_object96177:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f91bab48:::benchmark_data_smithi173_74914_object89347:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f91c7c76:::benchmark_data_smithi173_74914_object83082:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f91cb03e:::benchmark_data_smithi173_74914_object97257:head# 2024-08-30T23:37:06.409 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f91f3ece:::benchmark_data_smithi173_74914_object92783:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9222ec3:::benchmark_data_smithi173_74914_object84652:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f92474cf:::benchmark_data_smithi173_74914_object85547:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f924f4bf:::benchmark_data_smithi173_74914_object75766:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9265872:::benchmark_data_smithi173_74914_object97570:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f926b876:::benchmark_data_smithi173_74914_object80842:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f92a31be:::benchmark_data_smithi173_74914_object93586:head# 2024-08-30T23:37:06.410 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f92b1255:::benchmark_data_smithi173_74914_object80347:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f92c76ff:::benchmark_data_smithi173_74914_object58676:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9347a48:::benchmark_data_smithi173_74914_object64503:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f934f219:::benchmark_data_smithi173_74914_object68557:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f93904ee:::benchmark_data_smithi173_74914_object81467:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f939a416:::benchmark_data_smithi173_74914_object56175:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f93da155:::benchmark_data_smithi173_74914_object94007:head# 2024-08-30T23:37:06.411 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94042df:::benchmark_data_smithi173_74914_object55659:head# 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9432469:::benchmark_data_smithi173_74914_object79521:head# 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f945da64:::benchmark_data_smithi173_74914_object82282:head# 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9469c68:::benchmark_data_smithi173_74914_object65439:head# 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9491620:::benchmark_data_smithi173_74914_object84360:head# 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9498fb7:::benchmark_data_smithi173_74914_object85152:head# 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.412 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f949cc45:::benchmark_data_smithi173_74914_object90918:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94b79ee:::benchmark_data_smithi173_74914_object59083:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94bdedd:::benchmark_data_smithi173_74914_object67978:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94e3550:::benchmark_data_smithi173_74914_object74748:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94ec713:::benchmark_data_smithi173_74914_object63069:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94ec944:::benchmark_data_smithi173_74914_object96818:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94ed40b:::benchmark_data_smithi173_74914_object64415:head# 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.413 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f94fc96d:::benchmark_data_smithi173_74914_object80217:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f951fed7:::benchmark_data_smithi173_74914_object85186:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f952bcfc:::benchmark_data_smithi173_74914_object97685:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f958dc3a:::benchmark_data_smithi173_74914_object74050:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f95995aa:::benchmark_data_smithi173_74914_object77202:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f959aebd:::benchmark_data_smithi173_74914_object58637:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f95b316d:::benchmark_data_smithi173_74914_object64536:head# 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.414 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f95dbfc7:::benchmark_data_smithi173_74914_object74009:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f95dfa22:::benchmark_data_smithi173_74914_object66442:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f95ea044:::benchmark_data_smithi173_74914_object81125:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f962ac69:::benchmark_data_smithi173_74914_object71050:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f964f02c:::benchmark_data_smithi173_74914_object97123:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9650345:::benchmark_data_smithi173_74914_object72735:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9680fd9:::benchmark_data_smithi173_74914_object84729:head# 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.415 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9688b59:::benchmark_data_smithi173_74914_object92609:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f969917c:::benchmark_data_smithi173_74914_object60574:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f96a6c2a:::benchmark_data_smithi173_74914_object93934:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f96a7677:::benchmark_data_smithi173_74914_object52871:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f96e9a69:::benchmark_data_smithi173_74914_object87950:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f96f35a8:::benchmark_data_smithi173_74914_object83887:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f970ce19:::benchmark_data_smithi173_74914_object71558:head# 2024-08-30T23:37:06.416 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f97121fb:::benchmark_data_smithi173_74914_object60037:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f97329e5:::benchmark_data_smithi173_74914_object58386:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9746995:::benchmark_data_smithi173_74914_object97625:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9748fd5:::benchmark_data_smithi173_74914_object90239:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f977d4ca:::benchmark_data_smithi173_74914_object94523:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9788c44:::benchmark_data_smithi173_74914_object58481:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f97b3d7c:::benchmark_data_smithi173_74914_object71443:head# 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.417 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f97bb3ed:::benchmark_data_smithi173_74914_object54845:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f97cc49d:::benchmark_data_smithi173_74914_object77440:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f97cdb55:::benchmark_data_smithi173_74914_object86317:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f982e3e2:::benchmark_data_smithi173_74914_object86664:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f987b89c:::benchmark_data_smithi173_74914_object54006:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9889f23:::benchmark_data_smithi173_74914_object80451:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9896089:::benchmark_data_smithi173_74914_object53904:head# 2024-08-30T23:37:06.418 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9899133:::benchmark_data_smithi173_74914_object74325:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f98cc0e6:::benchmark_data_smithi173_74914_object95501:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f98ddc08:::benchmark_data_smithi173_74914_object83003:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f98eaf80:::benchmark_data_smithi173_74914_object59608:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f98f2f59:::benchmark_data_smithi173_74914_object92574:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9921833:::benchmark_data_smithi173_74914_object66707:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f99591bb:::benchmark_data_smithi173_74914_object71506:head# 2024-08-30T23:37:06.419 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f99764b9:::benchmark_data_smithi173_74914_object71689:head# 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9983d83:::benchmark_data_smithi173_74914_object60795:head# 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f99b80d6:::benchmark_data_smithi173_74914_object71846:head# 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f99e20ae:::benchmark_data_smithi173_74914_object67076:head# 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f99e7038:::benchmark_data_smithi173_74914_object86363:head# 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a04229:::benchmark_data_smithi173_74914_object81531:head# 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a2904f:::benchmark_data_smithi173_74914_object75007:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a2921c:::benchmark_data_smithi173_74914_object67199:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a5615c:::benchmark_data_smithi173_74914_object85559:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a6f6b3:::benchmark_data_smithi173_74914_object89230:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a7d4cf:::benchmark_data_smithi173_74914_object82171:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a98eeb:::benchmark_data_smithi173_74914_object83449:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a9c019:::benchmark_data_smithi173_74914_object95650:head# 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.421 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9a9e853:::benchmark_data_smithi173_74914_object62153:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ac10e0:::benchmark_data_smithi173_74914_object77564:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ac8f5d:::benchmark_data_smithi173_74914_object70457:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ad28bf:::benchmark_data_smithi173_74914_object74046:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ae8ee4:::benchmark_data_smithi173_74914_object89637:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9aecf40:::benchmark_data_smithi173_74914_object56708:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9af20c3:::benchmark_data_smithi173_74914_object74769:head# 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.422 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9affddc:::benchmark_data_smithi173_74914_object58509:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b0d35f:::benchmark_data_smithi173_74914_object79554:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b1646a:::benchmark_data_smithi173_74914_object71202:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b30ee2:::benchmark_data_smithi173_74914_object80442:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b3a427:::benchmark_data_smithi173_74914_object72484:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b4540a:::benchmark_data_smithi173_74914_object83682:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b4c109:::benchmark_data_smithi173_74914_object76238:head# 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.423 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b5181f:::benchmark_data_smithi173_74914_object96780:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b51cae:::benchmark_data_smithi173_74914_object57069:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b65ad7:::benchmark_data_smithi173_74914_object85795:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b6939a:::benchmark_data_smithi173_74914_object84982:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b76de3:::benchmark_data_smithi173_74914_object63685:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b88343:::benchmark_data_smithi173_74914_object64778:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9b920f8:::benchmark_data_smithi173_74914_object60877:head# 2024-08-30T23:37:06.424 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ba795d:::benchmark_data_smithi173_74914_object66066:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9bac6f3:::benchmark_data_smithi173_74914_object81067:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9bb945f:::benchmark_data_smithi173_74914_object73546:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9bc850c:::benchmark_data_smithi173_74914_object56627:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9be48bc:::benchmark_data_smithi173_74914_object59404:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9c01575:::benchmark_data_smithi173_74914_object63410:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9c437a4:::benchmark_data_smithi173_74914_object87876:head# 2024-08-30T23:37:06.425 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9c71629:::benchmark_data_smithi173_74914_object60030:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9c74724:::benchmark_data_smithi173_74914_object83762:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9c938fc:::benchmark_data_smithi173_74914_object67619:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ca0450:::benchmark_data_smithi173_74914_object83942:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9cab1ee:::benchmark_data_smithi173_74914_object73317:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9cc5beb:::benchmark_data_smithi173_74914_object79005:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ccff23:::benchmark_data_smithi173_74914_object74673:head# 2024-08-30T23:37:06.426 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d2db12:::benchmark_data_smithi173_74914_object60041:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d44c3e:::benchmark_data_smithi173_74914_object59752:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d4c2af:::benchmark_data_smithi173_74914_object64534:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d4ff6a:::benchmark_data_smithi173_74914_object78586:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d5281b:::benchmark_data_smithi173_74914_object55260:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d6a201:::benchmark_data_smithi173_74914_object79478:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9d98c72:::benchmark_data_smithi173_74914_object62379:head# 2024-08-30T23:37:06.427 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9da25d8:::benchmark_data_smithi173_74914_object85191:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9da3ae0:::benchmark_data_smithi173_74914_object65285:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9db31c7:::benchmark_data_smithi173_74914_object76705:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9db892c:::benchmark_data_smithi173_74914_object77477:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9dbacb2:::benchmark_data_smithi173_74914_object96679:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9dece86:::benchmark_data_smithi173_74914_object53077:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9dfc307:::benchmark_data_smithi173_74914_object59614:head# 2024-08-30T23:37:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e0dfb0:::benchmark_data_smithi173_74914_object68584:head# 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e0f179:::benchmark_data_smithi173_74914_object94381:head# 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e41923:::benchmark_data_smithi173_74914_object72765:head# 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e5ced0:::benchmark_data_smithi173_74914_object60317:head# 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e610be:::benchmark_data_smithi173_74914_object73747:head# 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e7dfde:::benchmark_data_smithi173_74914_object92498:head# 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.429 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e892b3:::benchmark_data_smithi173_74914_object86283:head# 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e8aa4b:::benchmark_data_smithi173_74914_object97100:head# 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9e9c941:::benchmark_data_smithi173_74914_object57784:head# 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9eb0b81:::benchmark_data_smithi173_74914_object73437:head# 2024-08-30T23:37:06.430 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ecdbbc:::benchmark_data_smithi173_74914_object64539:head# 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9eddc12:::benchmark_data_smithi173_74914_object53610:head# 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9f26410:::benchmark_data_smithi173_74914_object67941:head# 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9f8a5da:::benchmark_data_smithi173_74914_object96785:head# 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9f8d301:::benchmark_data_smithi173_74914_object78042:head# 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9f8e0db:::benchmark_data_smithi173_74914_object88922:head# 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.432 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9fa9405:::benchmark_data_smithi173_74914_object92502:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9fc7b89:::benchmark_data_smithi173_74914_object53895:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9fd5f39:::benchmark_data_smithi173_74914_object54871:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ffa377:::benchmark_data_smithi173_74914_object57802:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:f9ffb571:::benchmark_data_smithi173_74914_object62029:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa010ce2:::benchmark_data_smithi173_74914_object54905:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa01518b:::benchmark_data_smithi173_74914_object62022:head# 2024-08-30T23:37:06.433 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa03c80f:::benchmark_data_smithi173_74914_object83288:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa048538:::benchmark_data_smithi173_74914_object75752:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa054f08:::benchmark_data_smithi173_74914_object86277:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa07361b:::benchmark_data_smithi173_74914_object92256:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa0971ea:::benchmark_data_smithi173_74914_object80305:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa0b0c9d:::benchmark_data_smithi173_74914_object80814:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa0c9f4a:::benchmark_data_smithi173_74914_object82905:head# 2024-08-30T23:37:06.434 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa0d4c28:::benchmark_data_smithi173_74914_object53573:head# 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa0d75e5:::benchmark_data_smithi173_74914_object95738:head# 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa0d83d1:::benchmark_data_smithi173_74914_object71771:head# 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa12b371:::benchmark_data_smithi173_74914_object83493:head# 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa1339b4:::benchmark_data_smithi173_74914_object53998:head# 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa14dcfe:::benchmark_data_smithi173_74914_object77497:head# 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.435 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa15a50f:::benchmark_data_smithi173_74914_object69716:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa15b4f2:::benchmark_data_smithi173_74914_object75464:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa166d16:::benchmark_data_smithi173_74914_object75597:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa16b7e8:::benchmark_data_smithi173_74914_object81170:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa188760:::benchmark_data_smithi173_74914_object92616:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa193019:::benchmark_data_smithi173_74914_object65070:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa1999bd:::benchmark_data_smithi173_74914_object57848:head# 2024-08-30T23:37:06.436 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa1bb20a:::benchmark_data_smithi173_74914_object65418:head# 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa1cf1e9:::benchmark_data_smithi173_74914_object85679:head# 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa1e4b21:::benchmark_data_smithi173_74914_object75741:head# 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa1f8dde:::benchmark_data_smithi173_74914_object77573:head# 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa23b016:::benchmark_data_smithi173_74914_object92885:head# 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa24731c:::benchmark_data_smithi173_74914_object91297:head# 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.437 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa2a178d:::benchmark_data_smithi173_74914_object57657:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa2b5e69:::benchmark_data_smithi173_74914_object88098:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa2ddb2e:::benchmark_data_smithi173_74914_object87344:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa311882:::benchmark_data_smithi173_74914_object96116:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa31f8a9:::benchmark_data_smithi173_74914_object62340:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa35015d:::benchmark_data_smithi173_74914_object59008:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa36d1a6:::benchmark_data_smithi173_74914_object96668:head# 2024-08-30T23:37:06.438 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa376000:::benchmark_data_smithi173_74914_object91717:head# 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa37b580:::benchmark_data_smithi173_74914_object71305:head# 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa3ab643:::benchmark_data_smithi173_74914_object86308:head# 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa3df68e:::benchmark_data_smithi173_74914_object76456:head# 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa40b031:::benchmark_data_smithi173_74914_object61604:head# 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa41f553:::benchmark_data_smithi173_74914_object53058:head# 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.439 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa41fb48:::benchmark_data_smithi173_74914_object67147:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa44b260:::benchmark_data_smithi173_74914_object90301:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa484210:::benchmark_data_smithi173_74914_object94269:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa484bcc:::benchmark_data_smithi173_74914_object54319:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa4a9588:::benchmark_data_smithi173_74914_object95251:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa4e8873:::benchmark_data_smithi173_74914_object93235:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa4fd5fb:::benchmark_data_smithi173_74914_object76136:head# 2024-08-30T23:37:06.440 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa51040d:::benchmark_data_smithi173_74914_object70055:head# 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa52a0f6:::benchmark_data_smithi173_74914_object63952:head# 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa52a2ad:::benchmark_data_smithi173_74914_object72491:head# 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5504ae:::benchmark_data_smithi173_74914_object64155:head# 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa558767:::benchmark_data_smithi173_74914_object76598:head# 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa55b9f4:::benchmark_data_smithi173_74914_object96945:head# 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.441 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa55dda8:::benchmark_data_smithi173_74914_object68359:head# 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa56e6f6:::benchmark_data_smithi173_74914_object68349:head# 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5752d7:::benchmark_data_smithi173_74914_object71717:head# 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa585891:::benchmark_data_smithi173_74914_object67952:head# 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5a6092:::benchmark_data_smithi173_74914_object75581:head# 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5b4cd7:::benchmark_data_smithi173_74914_object62829:head# 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.442 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5c0f0a:::benchmark_data_smithi173_74914_object67068:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5c9d47:::benchmark_data_smithi173_74914_object93428:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa5d94b6:::benchmark_data_smithi173_74914_object87760:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa61eca6:::benchmark_data_smithi173_74914_object96604:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa621661:::benchmark_data_smithi173_74914_object54521:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6232e4:::benchmark_data_smithi173_74914_object55642:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6243b5:::benchmark_data_smithi173_74914_object94059:head# 2024-08-30T23:37:06.443 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6413f5:::benchmark_data_smithi173_74914_object96095:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6544f3:::benchmark_data_smithi173_74914_object78112:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa65def8:::benchmark_data_smithi173_74914_object66252:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa65f1c3:::benchmark_data_smithi173_74914_object80155:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa66cb0f:::benchmark_data_smithi173_74914_object62454:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa67dd96:::benchmark_data_smithi173_74914_object58044:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa686b93:::benchmark_data_smithi173_74914_object70785:head# 2024-08-30T23:37:06.444 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6b183b:::benchmark_data_smithi173_74914_object73750:head# 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6b4de0:::benchmark_data_smithi173_74914_object78385:head# 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6d8951:::benchmark_data_smithi173_74914_object97249:head# 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa6e3e47:::benchmark_data_smithi173_74914_object89492:head# 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa709967:::benchmark_data_smithi173_74914_object92355:head# 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa71177a:::benchmark_data_smithi173_74914_object77118:head# 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.445 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa712b9d:::benchmark_data_smithi173_74914_object54766:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa722b70:::benchmark_data_smithi173_74914_object71146:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa738068:::benchmark_data_smithi173_74914_object73552:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa73912d:::benchmark_data_smithi173_74914_object87307:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa785e36:::benchmark_data_smithi173_74914_object94924:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa7a33c6:::benchmark_data_smithi173_74914_object69067:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa7bcf1e:::benchmark_data_smithi173_74914_object61052:head# 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.446 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa7cd365:::benchmark_data_smithi173_74914_object66765:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa8169a4:::benchmark_data_smithi173_74914_object53485:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa85c07d:::benchmark_data_smithi173_74914_object77519:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa871a97:::benchmark_data_smithi173_74914_object83477:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa893404:::benchmark_data_smithi173_74914_object86909:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa89e0b9:::benchmark_data_smithi173_74914_object85786:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa8a8e6b:::benchmark_data_smithi173_74914_object98203:head# 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.447 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa8abbe9:::benchmark_data_smithi173_74914_object68962:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa8abe96:::benchmark_data_smithi173_74914_object67453:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa901d33:::benchmark_data_smithi173_74914_object82497:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa911756:::benchmark_data_smithi173_74914_object86282:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa921d37:::benchmark_data_smithi173_74914_object91333:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9261b9:::benchmark_data_smithi173_74914_object93457:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa92fcd7:::benchmark_data_smithi173_74914_object60019:head# 2024-08-30T23:37:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa97afbe:::benchmark_data_smithi173_74914_object68302:head# 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa995538:::benchmark_data_smithi173_74914_object90256:head# 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa999cd1:::benchmark_data_smithi173_74914_object63924:head# 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9a6eb2:::benchmark_data_smithi173_74914_object91739:head# 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9bb75d:::benchmark_data_smithi173_74914_object76866:head# 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9c0f52:::benchmark_data_smithi173_74914_object62839:head# 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.449 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9d0bb7:::benchmark_data_smithi173_74914_object86248:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9d1bf8:::benchmark_data_smithi173_74914_object53916:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fa9e520b:::benchmark_data_smithi173_74914_object61429:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faa00001:::benchmark_data_smithi173_74914_object87369:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faa0b5b8:::benchmark_data_smithi173_74914_object57609:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faa13900:::benchmark_data_smithi173_74914_object63082:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faa4ec37:::benchmark_data_smithi173_74914_object65689:head# 2024-08-30T23:37:06.450 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faa74379:::benchmark_data_smithi173_74914_object68622:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faa9791d:::benchmark_data_smithi173_74914_object89301:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faab70db:::benchmark_data_smithi173_74914_object59778:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faabd22f:::benchmark_data_smithi173_74914_object56588:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faac3e08:::benchmark_data_smithi173_74914_object83258:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faad6448:::benchmark_data_smithi173_74914_object56273:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faaf512c:::benchmark_data_smithi173_74914_object84311:head# 2024-08-30T23:37:06.451 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab0591f:::benchmark_data_smithi173_74914_object52836:head# 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab16dc6:::benchmark_data_smithi173_74914_object88983:head# 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab1a67d:::benchmark_data_smithi173_74914_object66710:head# 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab314a9:::benchmark_data_smithi173_74914_object95045:head# 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab38a16:::benchmark_data_smithi173_74914_object68347:head# 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab44a77:::benchmark_data_smithi173_74914_object82475:head# 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.452 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fab75650:::benchmark_data_smithi173_74914_object62287:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faba316f:::benchmark_data_smithi173_74914_object61423:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fabca71f:::benchmark_data_smithi173_74914_object58497:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac0067c:::benchmark_data_smithi173_74914_object60617:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac0d528:::benchmark_data_smithi173_74914_object87298:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac13a55:::benchmark_data_smithi173_74914_object76862:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac14540:::benchmark_data_smithi173_74914_object67556:head# 2024-08-30T23:37:06.453 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac1fe94:::benchmark_data_smithi173_74914_object89841:head# 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac37c9d:::benchmark_data_smithi173_74914_object68613:head# 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac431cf:::benchmark_data_smithi173_74914_object91248:head# 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac4ac11:::benchmark_data_smithi173_74914_object75016:head# 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac95155:::benchmark_data_smithi173_74914_object84086:head# 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fac9dc9a:::benchmark_data_smithi173_74914_object56640:head# 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.454 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faca2ea8:::benchmark_data_smithi173_74914_object94073:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faced844:::benchmark_data_smithi173_74914_object62050:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad0e249:::benchmark_data_smithi173_74914_object68537:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad1472e:::benchmark_data_smithi173_74914_object62211:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad17643:::benchmark_data_smithi173_74914_object93338:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad1bcc3:::benchmark_data_smithi173_74914_object71064:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad441be:::benchmark_data_smithi173_74914_object97895:head# 2024-08-30T23:37:06.455 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad4e1fa:::benchmark_data_smithi173_74914_object86768:head# 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad62cb4:::benchmark_data_smithi173_74914_object68409:head# 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fad76bdc:::benchmark_data_smithi173_74914_object85667:head# 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fadc02c4:::benchmark_data_smithi173_74914_object68366:head# 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fade3233:::benchmark_data_smithi173_74914_object78806:head# 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fade8c0c:::benchmark_data_smithi173_74914_object87673:head# 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.456 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fadfafa7:::benchmark_data_smithi173_74914_object57994:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae0820a:::benchmark_data_smithi173_74914_object71388:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae0d869:::benchmark_data_smithi173_74914_object54639:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae378ef:::benchmark_data_smithi173_74914_object63481:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae59ccd:::benchmark_data_smithi173_74914_object58226:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae635e6:::benchmark_data_smithi173_74914_object89433:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae6e805:::benchmark_data_smithi173_74914_object89811:head# 2024-08-30T23:37:06.457 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fae9d461:::benchmark_data_smithi173_74914_object58107:head# 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faea070f:::benchmark_data_smithi173_74914_object94362:head# 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faed1c63:::benchmark_data_smithi173_74914_object96188:head# 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faef221a:::benchmark_data_smithi173_74914_object86258:head# 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faf165a1:::benchmark_data_smithi173_74914_object73158:head# 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faf21504:::benchmark_data_smithi173_74914_object95571:head# 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faf2fef9:::benchmark_data_smithi173_74914_object62679:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faf45a35:::benchmark_data_smithi173_74914_object79292:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faf4fc8d:::benchmark_data_smithi173_74914_object67101:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafa7740:::benchmark_data_smithi173_74914_object69795:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafc2473:::benchmark_data_smithi173_74914_object81585:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafd01ed:::benchmark_data_smithi173_74914_object78544:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafdf3dd:::benchmark_data_smithi173_74914_object53592:head# 2024-08-30T23:37:06.459 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafe0f27:::benchmark_data_smithi173_74914_object93000:head# 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafe1959:::benchmark_data_smithi173_74914_object65231:head# 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafe1bdc:::benchmark_data_smithi173_74914_object82349:head# 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:faffb770:::benchmark_data_smithi173_74914_object82956:head# 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fafffbd0:::benchmark_data_smithi173_74914_object72128:head# 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb017c20:::benchmark_data_smithi173_74914_object78137:head# 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.460 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb019ce3:::benchmark_data_smithi173_74914_object76609:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb02202b:::benchmark_data_smithi173_74914_object92847:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb02a6f4:::benchmark_data_smithi173_74914_object94326:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb04b5f1:::benchmark_data_smithi173_74914_object56341:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb051c26:::benchmark_data_smithi173_74914_object81360:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb05b555:::benchmark_data_smithi173_74914_object91435:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb07e730:::benchmark_data_smithi173_74914_object76869:head# 2024-08-30T23:37:06.461 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb089b40:::benchmark_data_smithi173_74914_object54013:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb08a449:::benchmark_data_smithi173_74914_object93421:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb0d72e1:::benchmark_data_smithi173_74914_object57453:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb0dcdce:::benchmark_data_smithi173_74914_object87045:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb0ed265:::benchmark_data_smithi173_74914_object76054:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb108702:::benchmark_data_smithi173_74914_object59647:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb10d68e:::benchmark_data_smithi173_74914_object83119:head# 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.462 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb10f931:::benchmark_data_smithi173_74914_object91522:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb11ccf6:::benchmark_data_smithi173_74914_object68567:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb139c09:::benchmark_data_smithi173_74914_object71102:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb1584cd:::benchmark_data_smithi173_74914_object64104:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb18e283:::benchmark_data_smithi173_74914_object77325:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb19685c:::benchmark_data_smithi173_74914_object95695:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb1b983f:::benchmark_data_smithi173_74914_object84972:head# 2024-08-30T23:37:06.463 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb1e985d:::benchmark_data_smithi173_74914_object97427:head# 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb1fc5bb:::benchmark_data_smithi173_74914_object57505:head# 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb2000eb:::benchmark_data_smithi173_74914_object56014:head# 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb206e54:::benchmark_data_smithi173_74914_object56315:head# 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb208a64:::benchmark_data_smithi173_74914_object52921:head# 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb223abf:::benchmark_data_smithi173_74914_object54962:head# 2024-08-30T23:37:06.464 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb242290:::benchmark_data_smithi173_74914_object96749:head# 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb26dca2:::benchmark_data_smithi173_74914_object89057:head# 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb2ba412:::benchmark_data_smithi173_74914_object78755:head# 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb2e2611:::benchmark_data_smithi173_74914_object53439:head# 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb2fd4c2:::benchmark_data_smithi173_74914_object58093:head# 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb310be2:::benchmark_data_smithi173_74914_object54691:head# 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.465 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb32adfd:::benchmark_data_smithi173_74914_object77102:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3395b0:::benchmark_data_smithi173_74914_object53125:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb35d04b:::benchmark_data_smithi173_74914_object72215:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3676c1:::benchmark_data_smithi173_74914_object58769:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3796d1:::benchmark_data_smithi173_74914_object80226:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3b816d:::benchmark_data_smithi173_74914_object72497:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3c3e53:::benchmark_data_smithi173_74914_object94017:head# 2024-08-30T23:37:06.466 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3d02a6:::benchmark_data_smithi173_74914_object83333:head# 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3d97a7:::benchmark_data_smithi173_74914_object86202:head# 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb3eea57:::benchmark_data_smithi173_74914_object98189:head# 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb404476:::benchmark_data_smithi173_74914_object77405:head# 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb46d1c5:::benchmark_data_smithi173_74914_object55859:head# 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb47f73f:::benchmark_data_smithi173_74914_object82551:head# 2024-08-30T23:37:06.467 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb48fe1c:::benchmark_data_smithi173_74914_object84689:head# 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4b2ac9:::benchmark_data_smithi173_74914_object62039:head# 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4b4f89:::benchmark_data_smithi173_74914_object69290:head# 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4be2db:::benchmark_data_smithi173_74914_object54749:head# 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4c35ad:::benchmark_data_smithi173_74914_object54100:head# 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4d9252:::benchmark_data_smithi173_74914_object68794:head# 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.468 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4def20:::benchmark_data_smithi173_74914_object93818:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4e2163:::benchmark_data_smithi173_74914_object66902:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb4e97bd:::benchmark_data_smithi173_74914_object75093:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb504782:::benchmark_data_smithi173_74914_object93603:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb50f9b7:::benchmark_data_smithi173_74914_object56171:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5343bd:::benchmark_data_smithi173_74914_object92311:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb534d3e:::benchmark_data_smithi173_74914_object56502:head# 2024-08-30T23:37:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb53ecce:::benchmark_data_smithi173_74914_object97281:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb541383:::benchmark_data_smithi173_74914_object53381:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb542b3b:::benchmark_data_smithi173_74914_object91275:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb58b9e4:::benchmark_data_smithi173_74914_object79441:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb597492:::benchmark_data_smithi173_74914_object61573:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5b291f:::benchmark_data_smithi173_74914_object77021:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5d400c:::benchmark_data_smithi173_74914_object69374:head# 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.470 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5e19dd:::benchmark_data_smithi173_74914_object79927:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5e729f:::benchmark_data_smithi173_74914_object81600:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5eb487:::benchmark_data_smithi173_74914_object80948:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb5f96bc:::benchmark_data_smithi173_74914_object53422:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb602991:::benchmark_data_smithi173_74914_object80149:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb605d75:::benchmark_data_smithi173_74914_object71013:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb6348b8:::benchmark_data_smithi173_74914_object85648:head# 2024-08-30T23:37:06.471 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb6353bf:::benchmark_data_smithi173_74914_object79794:head# 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb65c9d6:::benchmark_data_smithi173_74914_object85174:head# 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb661f08:::benchmark_data_smithi173_74914_object72153:head# 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb676595:::benchmark_data_smithi173_74914_object85589:head# 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb687448:::benchmark_data_smithi173_74914_object65080:head# 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb694729:::benchmark_data_smithi173_74914_object75530:head# 2024-08-30T23:37:06.472 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb69d587:::benchmark_data_smithi173_74914_object94214:head# 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb6b7838:::benchmark_data_smithi173_74914_object60113:head# 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb6ca667:::benchmark_data_smithi173_74914_object76075:head# 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb6d9978:::benchmark_data_smithi173_74914_object60678:head# 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7135f9:::benchmark_data_smithi173_74914_object89187:head# 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb72cf31:::benchmark_data_smithi173_74914_object55059:head# 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.473 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb735073:::benchmark_data_smithi173_74914_object96430:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb75786d:::benchmark_data_smithi173_74914_object89693:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb75e4cd:::benchmark_data_smithi173_74914_object94418:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb77292d:::benchmark_data_smithi173_74914_object88013:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7d2676:::benchmark_data_smithi173_74914_object94365:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7d4b15:::benchmark_data_smithi173_74914_object89549:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7d88c2:::benchmark_data_smithi173_74914_object77131:head# 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.474 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7e1c76:::benchmark_data_smithi173_74914_object57722:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7e2d8f:::benchmark_data_smithi173_74914_object64744:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb7f012c:::benchmark_data_smithi173_74914_object91104:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb803bc7:::benchmark_data_smithi173_74914_object68135:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb8209d1:::benchmark_data_smithi173_74914_object86566:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb824ae4:::benchmark_data_smithi173_74914_object89752:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb830e45:::benchmark_data_smithi173_74914_object97155:head# 2024-08-30T23:37:06.475 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb83173a:::benchmark_data_smithi173_74914_object59886:head# 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb852dd3:::benchmark_data_smithi173_74914_object88323:head# 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb88c4bd:::benchmark_data_smithi173_74914_object73525:head# 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb89907f:::benchmark_data_smithi173_74914_object65496:head# 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb89e5a6:::benchmark_data_smithi173_74914_object71791:head# 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb8a1703:::benchmark_data_smithi173_74914_object59293:head# 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.476 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb8ad9c4:::benchmark_data_smithi173_74914_object64837:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb8bb2fa:::benchmark_data_smithi173_74914_object68678:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb8cc3b9:::benchmark_data_smithi173_74914_object81302:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb8ed16d:::benchmark_data_smithi173_74914_object77985:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb90c6dc:::benchmark_data_smithi173_74914_object52835:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb9202f7:::benchmark_data_smithi173_74914_object74081:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb9713fb:::benchmark_data_smithi173_74914_object74726:head# 2024-08-30T23:37:06.477 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fb9d3511:::benchmark_data_smithi173_74914_object67987:head# 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba00a84:::benchmark_data_smithi173_74914_object67243:head# 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba245c8:::benchmark_data_smithi173_74914_object91787:head# 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba359c2:::benchmark_data_smithi173_74914_object59991:head# 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba3aeab:::benchmark_data_smithi173_74914_object78073:head# 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba3c3af:::benchmark_data_smithi173_74914_object73740:head# 2024-08-30T23:37:06.478 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba4d4e9:::benchmark_data_smithi173_74914_object81617:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba525e3:::benchmark_data_smithi173_74914_object54902:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba64c8a:::benchmark_data_smithi173_74914_object90781:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba78266:::benchmark_data_smithi173_74914_object55604:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba8294f:::benchmark_data_smithi173_74914_object88311:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fba95bd5:::benchmark_data_smithi173_74914_object79296:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbacfd7e:::benchmark_data_smithi173_74914_object79938:head# 2024-08-30T23:37:06.479 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbade4bb:::benchmark_data_smithi173_74914_object84032:head# 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbae6c82:::benchmark_data_smithi173_74914_object94068:head# 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb0a10a:::benchmark_data_smithi173_74914_object76308:head# 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb0b2e1:::benchmark_data_smithi173_74914_object89912:head# 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb2d881:::benchmark_data_smithi173_74914_object88043:head# 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb573b8:::benchmark_data_smithi173_74914_object72157:head# 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.480 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb5c3b3:::benchmark_data_smithi173_74914_object89428:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb62035:::benchmark_data_smithi173_74914_object78923:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb672ae:::benchmark_data_smithi173_74914_object63189:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbb6a1bc:::benchmark_data_smithi173_74914_object94830:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbbbf7ce:::benchmark_data_smithi173_74914_object94886:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbbdca1c:::benchmark_data_smithi173_74914_object82410:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbbfe5b3:::benchmark_data_smithi173_74914_object94844:head# 2024-08-30T23:37:06.481 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbc11a5b:::benchmark_data_smithi173_74914_object70401:head# 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbc7ceaf:::benchmark_data_smithi173_74914_object59144:head# 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbc8703b:::benchmark_data_smithi173_74914_object91468:head# 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbcf4229:::benchmark_data_smithi173_74914_object97208:head# 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd24904:::benchmark_data_smithi173_74914_object77408:head# 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd28a38:::benchmark_data_smithi173_74914_object77022:head# 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.482 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd2eaab:::benchmark_data_smithi173_74914_object80336:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd4bc11:::benchmark_data_smithi173_74914_object91556:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd62ccb:::benchmark_data_smithi173_74914_object55577:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd6f901:::benchmark_data_smithi173_74914_object83204:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbd8ce4d:::benchmark_data_smithi173_74914_object57939:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbda6562:::benchmark_data_smithi173_74914_object79445:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbdca495:::benchmark_data_smithi173_74914_object81596:head# 2024-08-30T23:37:06.483 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbde738a:::benchmark_data_smithi173_74914_object62018:head# 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe14532:::benchmark_data_smithi173_74914_object54771:head# 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe18b48:::benchmark_data_smithi173_74914_object77128:head# 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe1e352:::benchmark_data_smithi173_74914_object65719:head# 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe32051:::benchmark_data_smithi173_74914_object88168:head# 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe50436:::benchmark_data_smithi173_74914_object53847:head# 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.484 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe686d7:::benchmark_data_smithi173_74914_object60015:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe7b4cd:::benchmark_data_smithi173_74914_object63514:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe847ea:::benchmark_data_smithi173_74914_object80737:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbe87358:::benchmark_data_smithi173_74914_object76387:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbeb9ec6:::benchmark_data_smithi173_74914_object87838:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbec09ca:::benchmark_data_smithi173_74914_object95519:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbec610e:::benchmark_data_smithi173_74914_object77373:head# 2024-08-30T23:37:06.485 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbf1dd42:::benchmark_data_smithi173_74914_object72830:head# 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbf2fb1f:::benchmark_data_smithi173_74914_object68102:head# 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbf61847:::benchmark_data_smithi173_74914_object81784:head# 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbf8ae97:::benchmark_data_smithi173_74914_object94681:head# 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbfac3af:::benchmark_data_smithi173_74914_object54387:head# 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbfade79:::benchmark_data_smithi173_74914_object83937:head# 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.486 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbfae5a7:::benchmark_data_smithi173_74914_object90550:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbfb7470:::benchmark_data_smithi173_74914_object53950:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbfdb9bf:::benchmark_data_smithi173_74914_object71629:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fbfe3bd6:::benchmark_data_smithi173_74914_object98091:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc007d7a:::benchmark_data_smithi173_74914_object83703:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc0282a0:::benchmark_data_smithi173_74914_object61438:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc02f1a1:::benchmark_data_smithi173_74914_object56013:head# 2024-08-30T23:37:06.487 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc03bd84:::benchmark_data_smithi173_74914_object80998:head# 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc066610:::benchmark_data_smithi173_74914_object97019:head# 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc089e34:::benchmark_data_smithi173_74914_object64185:head# 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc09b9eb:::benchmark_data_smithi173_74914_object85941:head# 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc09ee68:::benchmark_data_smithi173_74914_object57299:head# 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc0d32c2:::benchmark_data_smithi173_74914_object81644:head# 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.488 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc0dddf1:::benchmark_data_smithi173_74914_object79969:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc10cf0f:::benchmark_data_smithi173_74914_object66559:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc10e2b6:::benchmark_data_smithi173_74914_object55021:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc15a7d2:::benchmark_data_smithi173_74914_object85120:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc169127:::benchmark_data_smithi173_74914_object68115:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc172063:::benchmark_data_smithi173_74914_object64405:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc192d93:::benchmark_data_smithi173_74914_object89557:head# 2024-08-30T23:37:06.489 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc1a9220:::benchmark_data_smithi173_74914_object83819:head# 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc1baa1f:::benchmark_data_smithi173_74914_object61828:head# 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc1e30ea:::benchmark_data_smithi173_74914_object68212:head# 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc22860a:::benchmark_data_smithi173_74914_object55400:head# 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc22af45:::benchmark_data_smithi173_74914_object61775:head# 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc233f11:::benchmark_data_smithi173_74914_object95460:head# 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.490 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc24046f:::benchmark_data_smithi173_74914_object64732:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc2488a5:::benchmark_data_smithi173_74914_object90006:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc254796:::benchmark_data_smithi173_74914_object83048:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc25d788:::benchmark_data_smithi173_74914_object58049:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc27ce13:::benchmark_data_smithi173_74914_object74031:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc2b5cf0:::benchmark_data_smithi173_74914_object62575:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc2e35e2:::benchmark_data_smithi173_74914_object80823:head# 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.491 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc2e9c42:::benchmark_data_smithi173_74914_object86811:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc2edfcd:::benchmark_data_smithi173_74914_object93728:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc2fa7c0:::benchmark_data_smithi173_74914_object67024:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc352524:::benchmark_data_smithi173_74914_object82952:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc37e61a:::benchmark_data_smithi173_74914_object81811:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc3e0df0:::benchmark_data_smithi173_74914_object80567:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc3ffc48:::benchmark_data_smithi173_74914_object73177:head# 2024-08-30T23:37:06.492 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4333c8:::benchmark_data_smithi173_74914_object57295:head# 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc439f99:::benchmark_data_smithi173_74914_object75784:head# 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc45310a:::benchmark_data_smithi173_74914_object88816:head# 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4574bd:::benchmark_data_smithi173_74914_object97640:head# 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc46b862:::benchmark_data_smithi173_74914_object90830:head# 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc47704d:::benchmark_data_smithi173_74914_object52863:head# 2024-08-30T23:37:06.493 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc490de8:::benchmark_data_smithi173_74914_object73453:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc491678:::benchmark_data_smithi173_74914_object78816:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4bc0cf:::benchmark_data_smithi173_74914_object76930:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4c7db8:::benchmark_data_smithi173_74914_object83580:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4c9229:::benchmark_data_smithi173_74914_object55858:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4eaa7e:::benchmark_data_smithi173_74914_object66414:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc4effd3:::benchmark_data_smithi173_74914_object70866:head# 2024-08-30T23:37:06.494 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc5049e9:::benchmark_data_smithi173_74914_object63730:head# 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc5123c5:::benchmark_data_smithi173_74914_object67239:head# 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc52bec5:::benchmark_data_smithi173_74914_object96229:head# 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc53e6d9:::benchmark_data_smithi173_74914_object69195:head# 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc54c03c:::benchmark_data_smithi173_74914_object96942:head# 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc562f42:::benchmark_data_smithi173_74914_object69512:head# 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.495 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc57ae57:::benchmark_data_smithi173_74914_object78775:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc5a2685:::benchmark_data_smithi173_74914_object73629:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc5bbe4c:::benchmark_data_smithi173_74914_object75434:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc5d80a8:::benchmark_data_smithi173_74914_object97761:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc5db8b5:::benchmark_data_smithi173_74914_object80307:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc602e84:::benchmark_data_smithi173_74914_object56994:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc60f67e:::benchmark_data_smithi173_74914_object77270:head# 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.496 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc6115b8:::benchmark_data_smithi173_74914_object93808:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc61809b:::benchmark_data_smithi173_74914_object62855:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc62a47f:::benchmark_data_smithi173_74914_object69243:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc632606:::benchmark_data_smithi173_74914_object56369:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc63282d:::benchmark_data_smithi173_74914_object56594:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc63d6ff:::benchmark_data_smithi173_74914_object62868:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc6514d5:::benchmark_data_smithi173_74914_object83071:head# 2024-08-30T23:37:06.497 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc651b59:::benchmark_data_smithi173_74914_object93966:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc662bd6:::benchmark_data_smithi173_74914_object61815:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc671a9c:::benchmark_data_smithi173_74914_object61857:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc677b13:::benchmark_data_smithi173_74914_object93703:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc698f2d:::benchmark_data_smithi173_74914_object87722:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc6e7917:::benchmark_data_smithi173_74914_object70254:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc6ee3aa:::benchmark_data_smithi173_74914_object58202:head# 2024-08-30T23:37:06.498 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc6f965c:::benchmark_data_smithi173_74914_object79094:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc70b84c:::benchmark_data_smithi173_74914_object65050:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc724878:::benchmark_data_smithi173_74914_object97914:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc726cb0:::benchmark_data_smithi173_74914_object74057:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc72d5d4:::benchmark_data_smithi173_74914_object84488:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc779310:::benchmark_data_smithi173_74914_object95644:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc7c5c55:::benchmark_data_smithi173_74914_object68733:head# 2024-08-30T23:37:06.499 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc7d48cb:::benchmark_data_smithi173_74914_object76847:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc80aeff:::benchmark_data_smithi173_74914_object96504:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc838ad3:::benchmark_data_smithi173_74914_object83224:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc86e58d:::benchmark_data_smithi173_74914_object55301:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc8a5b63:::benchmark_data_smithi173_74914_object66875:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc8ac37c:::benchmark_data_smithi173_74914_object95043:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc8d27c7:::benchmark_data_smithi173_74914_object98004:head# 2024-08-30T23:37:06.500 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc8d32ed:::benchmark_data_smithi173_74914_object87317:head# 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc8d4436:::benchmark_data_smithi173_74914_object76511:head# 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc8eb560:::benchmark_data_smithi173_74914_object74669:head# 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc9053a6:::benchmark_data_smithi173_74914_object76373:head# 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc929eed:::benchmark_data_smithi173_74914_object92610:head# 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc932922:::benchmark_data_smithi173_74914_object66047:head# 2024-08-30T23:37:06.501 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc963ca1:::benchmark_data_smithi173_74914_object69017:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc966fc6:::benchmark_data_smithi173_74914_object91313:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc97c06f:::benchmark_data_smithi173_74914_object57432:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc997007:::benchmark_data_smithi173_74914_object67344:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc99e165:::benchmark_data_smithi173_74914_object92216:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc9aaebf:::benchmark_data_smithi173_74914_object61973:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc9af1db:::benchmark_data_smithi173_74914_object93461:head# 2024-08-30T23:37:06.502 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc9c7fe1:::benchmark_data_smithi173_74914_object74358:head# 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fc9d69d9:::benchmark_data_smithi173_74914_object97747:head# 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fca00b10:::benchmark_data_smithi173_74914_object78890:head# 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fca1fef3:::benchmark_data_smithi173_74914_object80187:head# 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fca4db41:::benchmark_data_smithi173_74914_object94573:head# 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fca50c58:::benchmark_data_smithi173_74914_object81061:head# 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.503 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fca7380a:::benchmark_data_smithi173_74914_object92159:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcaad56f:::benchmark_data_smithi173_74914_object55274:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcac0c48:::benchmark_data_smithi173_74914_object66600:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcade851:::benchmark_data_smithi173_74914_object86705:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcae01a5:::benchmark_data_smithi173_74914_object73468:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb126b9:::benchmark_data_smithi173_74914_object82869:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb15f9b:::benchmark_data_smithi173_74914_object62135:head# 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.504 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb1da7e:::benchmark_data_smithi173_74914_object64862:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb21c78:::benchmark_data_smithi173_74914_object55501:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb2971d:::benchmark_data_smithi173_74914_object66931:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb36809:::benchmark_data_smithi173_74914_object81245:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcb6a156:::benchmark_data_smithi173_74914_object61912:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcbb1834:::benchmark_data_smithi173_74914_object86531:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcbfeb9d:::benchmark_data_smithi173_74914_object86716:head# 2024-08-30T23:37:06.505 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcc0dfe8:::benchmark_data_smithi173_74914_object83245:head# 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcc35967:::benchmark_data_smithi173_74914_object57166:head# 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcc737bb:::benchmark_data_smithi173_74914_object63596:head# 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcc9351c:::benchmark_data_smithi173_74914_object92337:head# 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcc9aa83:::benchmark_data_smithi173_74914_object54436:head# 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fccf1320:::benchmark_data_smithi173_74914_object60626:head# 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.506 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd3b056:::benchmark_data_smithi173_74914_object71698:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd64456:::benchmark_data_smithi173_74914_object63904:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd66ada:::benchmark_data_smithi173_74914_object73282:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd6ceee:::benchmark_data_smithi173_74914_object58585:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd8c748:::benchmark_data_smithi173_74914_object89367:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd919e9:::benchmark_data_smithi173_74914_object95989:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd952a9:::benchmark_data_smithi173_74914_object89612:head# 2024-08-30T23:37:06.507 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcd97510:::benchmark_data_smithi173_74914_object57983:head# 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcdc6bf4:::benchmark_data_smithi173_74914_object75899:head# 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcdc75b4:::benchmark_data_smithi173_74914_object73034:head# 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcde5c84:::benchmark_data_smithi173_74914_object68696:head# 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcde7c63:::benchmark_data_smithi173_74914_object86128:head# 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fce488c1:::benchmark_data_smithi173_74914_object97148:head# 2024-08-30T23:37:06.508 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fce5636f:::benchmark_data_smithi173_74914_object52808:head# 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fce6842f:::benchmark_data_smithi173_74914_object53254:head# 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fce7e2e9:::benchmark_data_smithi173_74914_object81044:head# 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fce851e9:::benchmark_data_smithi173_74914_object67922:head# 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fce86a0c:::benchmark_data_smithi173_74914_object84720:head# 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fceb6bef:::benchmark_data_smithi173_74914_object83968:head# 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.509 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcec4d5d:::benchmark_data_smithi173_74914_object75668:head# 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcec6c69:::benchmark_data_smithi173_74914_object78061:head# 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fceef0cd:::benchmark_data_smithi173_74914_object54476:head# 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf1a8ec:::benchmark_data_smithi173_74914_object55398:head# 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf1cdcc:::benchmark_data_smithi173_74914_object67519:head# 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf25eb3:::benchmark_data_smithi173_74914_object83892:head# 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.510 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf5530f:::benchmark_data_smithi173_74914_object56267:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf63337:::benchmark_data_smithi173_74914_object83181:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf893f8:::benchmark_data_smithi173_74914_object63556:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcf936ce:::benchmark_data_smithi173_74914_object96066:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcfb287c:::benchmark_data_smithi173_74914_object77171:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcfbd40a:::benchmark_data_smithi173_74914_object79884:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcfcc105:::benchmark_data_smithi173_74914_object77190:head# 2024-08-30T23:37:06.511 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcfd1207:::benchmark_data_smithi173_74914_object57211:head# 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fcffdfe4:::benchmark_data_smithi173_74914_object87640:head# 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd007fcc:::benchmark_data_smithi173_74914_object95996:head# 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd01c108:::benchmark_data_smithi173_74914_object78232:head# 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd026e58:::benchmark_data_smithi173_74914_object75834:head# 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd04d939:::benchmark_data_smithi173_74914_object75765:head# 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.512 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd08e99b:::benchmark_data_smithi173_74914_object61554:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd0957eb:::benchmark_data_smithi173_74914_object71213:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd0db87d:::benchmark_data_smithi173_74914_object91983:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd10053e:::benchmark_data_smithi173_74914_object58499:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd10960f:::benchmark_data_smithi173_74914_object62499:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd115d09:::benchmark_data_smithi173_74914_object78950:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd116c44:::benchmark_data_smithi173_74914_object82651:head# 2024-08-30T23:37:06.513 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd11fbf4:::benchmark_data_smithi173_74914_object64395:head# 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd12570e:::benchmark_data_smithi173_74914_object53815:head# 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd13f605:::benchmark_data_smithi173_74914_object81957:head# 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd1707a7:::benchmark_data_smithi173_74914_object55196:head# 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd17806f:::benchmark_data_smithi173_74914_object66130:head# 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd1879fc:::benchmark_data_smithi173_74914_object75553:head# 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.514 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd1964fd:::benchmark_data_smithi173_74914_object83447:head# 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd19da22:::benchmark_data_smithi173_74914_object66720:head# 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd1cbcf1:::benchmark_data_smithi173_74914_object87911:head# 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd1ea92b:::benchmark_data_smithi173_74914_object58254:head# 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd1f423c:::benchmark_data_smithi173_74914_object81516:head# 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd20da7a:::benchmark_data_smithi173_74914_object79644:head# 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.515 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd214a6f:::benchmark_data_smithi173_74914_object87116:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd22bee2:::benchmark_data_smithi173_74914_object58763:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd22ef16:::benchmark_data_smithi173_74914_object87360:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd235f38:::benchmark_data_smithi173_74914_object63990:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd279bfb:::benchmark_data_smithi173_74914_object85209:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd29197a:::benchmark_data_smithi173_74914_object87869:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd2ac3b8:::benchmark_data_smithi173_74914_object76588:head# 2024-08-30T23:37:06.516 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd2ba837:::benchmark_data_smithi173_74914_object78719:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd2c525e:::benchmark_data_smithi173_74914_object72540:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd2c7f34:::benchmark_data_smithi173_74914_object60473:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd2fa872:::benchmark_data_smithi173_74914_object76110:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd30555d:::benchmark_data_smithi173_74914_object68530:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd325407:::benchmark_data_smithi173_74914_object62851:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd32c84d:::benchmark_data_smithi173_74914_object56004:head# 2024-08-30T23:37:06.517 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd36991f:::benchmark_data_smithi173_74914_object77761:head# 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd36a04a:::benchmark_data_smithi173_74914_object80053:head# 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd383348:::benchmark_data_smithi173_74914_object75843:head# 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd3a3d2d:::benchmark_data_smithi173_74914_object58671:head# 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd3a4e79:::benchmark_data_smithi173_74914_object59654:head# 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd3c5d35:::benchmark_data_smithi173_74914_object79114:head# 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.518 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd3dc615:::benchmark_data_smithi173_74914_object65695:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd3dee1b:::benchmark_data_smithi173_74914_object90960:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd457f1e:::benchmark_data_smithi173_74914_object82938:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd45988e:::benchmark_data_smithi173_74914_object75550:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd459f71:::benchmark_data_smithi173_74914_object57196:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd460e50:::benchmark_data_smithi173_74914_object58706:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd463812:::benchmark_data_smithi173_74914_object72047:head# 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.519 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd4660e8:::benchmark_data_smithi173_74914_object76575:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd47d329:::benchmark_data_smithi173_74914_object90194:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd4a0c3c:::benchmark_data_smithi173_74914_object83691:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd4a6799:::benchmark_data_smithi173_74914_object71570:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd4afdbb:::benchmark_data_smithi173_74914_object66672:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd4da81e:::benchmark_data_smithi173_74914_object61987:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd4fcaa0:::benchmark_data_smithi173_74914_object80609:head# 2024-08-30T23:37:06.520 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd52a6d8:::benchmark_data_smithi173_74914_object88546:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd52e5db:::benchmark_data_smithi173_74914_object68532:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd53d838:::benchmark_data_smithi173_74914_object56312:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd54bca1:::benchmark_data_smithi173_74914_object89961:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd552f01:::benchmark_data_smithi173_74914_object76992:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd575c8f:::benchmark_data_smithi173_74914_object53465:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd5944f6:::benchmark_data_smithi173_74914_object62600:head# 2024-08-30T23:37:06.521 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd5b1e15:::benchmark_data_smithi173_74914_object54804:head# 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd5f4439:::benchmark_data_smithi173_74914_object69417:head# 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd623a32:::benchmark_data_smithi173_74914_object72104:head# 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd6311cf:::benchmark_data_smithi173_74914_object60380:head# 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd647813:::benchmark_data_smithi173_74914_object66109:head# 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd65acea:::benchmark_data_smithi173_74914_object73154:head# 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.522 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd66ba22:::benchmark_data_smithi173_74914_object55214:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd671508:::benchmark_data_smithi173_74914_object88799:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd69a996:::benchmark_data_smithi173_74914_object94118:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd6a0af3:::benchmark_data_smithi173_74914_object71534:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd6dff93:::benchmark_data_smithi173_74914_object74599:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd6e7789:::benchmark_data_smithi173_74914_object78319:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd6f46f9:::benchmark_data_smithi173_74914_object64204:head# 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.523 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd6f623d:::benchmark_data_smithi173_74914_object73212:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd7020fd:::benchmark_data_smithi173_74914_object53138:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd736059:::benchmark_data_smithi173_74914_object67666:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd74e092:::benchmark_data_smithi173_74914_object64660:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd75ecac:::benchmark_data_smithi173_74914_object53696:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd7626e1:::benchmark_data_smithi173_74914_object52967:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd773b5e:::benchmark_data_smithi173_74914_object60358:head# 2024-08-30T23:37:06.524 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd7a7c2f:::benchmark_data_smithi173_74914_object93083:head# 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd7d3c90:::benchmark_data_smithi173_74914_object90867:head# 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd8012d1:::benchmark_data_smithi173_74914_object57412:head# 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd82478c:::benchmark_data_smithi173_74914_object53848:head# 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd82a10e:::benchmark_data_smithi173_74914_object54275:head# 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd82ea63:::benchmark_data_smithi173_74914_object89144:head# 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.525 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd834f8e:::benchmark_data_smithi173_74914_object69472:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd8441e2:::benchmark_data_smithi173_74914_object79742:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd855335:::benchmark_data_smithi173_74914_object67879:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd85c773:::benchmark_data_smithi173_74914_object96158:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd86d084:::benchmark_data_smithi173_74914_object58283:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd86e5be:::benchmark_data_smithi173_74914_object76801:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd894eed:::benchmark_data_smithi173_74914_object80985:head# 2024-08-30T23:37:06.526 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd896f20:::benchmark_data_smithi173_74914_object53894:head# 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd8bebcf:::benchmark_data_smithi173_74914_object54425:head# 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd8c2b23:::benchmark_data_smithi173_74914_object52989:head# 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd8c429e:::benchmark_data_smithi173_74914_object62099:head# 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd8d029d:::benchmark_data_smithi173_74914_object80837:head# 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd906d67:::benchmark_data_smithi173_74914_object93986:head# 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.527 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd91d4d2:::benchmark_data_smithi173_74914_object92270:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd92b80f:::benchmark_data_smithi173_74914_object85077:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd932ff4:::benchmark_data_smithi173_74914_object67947:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd935d93:::benchmark_data_smithi173_74914_object96574:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd93f117:::benchmark_data_smithi173_74914_object71031:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd966b79:::benchmark_data_smithi173_74914_object76916:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd99a562:::benchmark_data_smithi173_74914_object57816:head# 2024-08-30T23:37:06.528 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd99c65f:::benchmark_data_smithi173_74914_object65503:head# 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd9a541f:::benchmark_data_smithi173_74914_object57018:head# 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd9a82d9:::benchmark_data_smithi173_74914_object66128:head# 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd9dacff:::benchmark_data_smithi173_74914_object66946:head# 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd9e2707:::benchmark_data_smithi173_74914_object82248:head# 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.529 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fd9f1e33:::benchmark_data_smithi173_74914_object59584:head# 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda2075b:::benchmark_data_smithi173_74914_object85437:head# 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda23411:::benchmark_data_smithi173_74914_object76714:head# 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda36808:::benchmark_data_smithi173_74914_object79758:head# 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda84b24:::benchmark_data_smithi173_74914_object84188:head# 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda8cefc:::benchmark_data_smithi173_74914_object75234:head# 2024-08-30T23:37:06.530 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda9645a:::benchmark_data_smithi173_74914_object69396:head# 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda9acfc:::benchmark_data_smithi173_74914_object72335:head# 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fda9b0f5:::benchmark_data_smithi173_74914_object94759:head# 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdab1814:::benchmark_data_smithi173_74914_object88180:head# 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb0c27d:::benchmark_data_smithi173_74914_object60487:head# 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.531 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb1319f:::benchmark_data_smithi173_74914_object85482:head# 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb5302b:::benchmark_data_smithi173_74914_object80993:head# 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb58589:::benchmark_data_smithi173_74914_object76807:head# 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb607d3:::benchmark_data_smithi173_74914_object67157:head# 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb8039d:::benchmark_data_smithi173_74914_object76893:head# 2024-08-30T23:37:06.532 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.533 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb87dd1:::benchmark_data_smithi173_74914_object75758:head# 2024-08-30T23:37:06.533 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdb9aec7:::benchmark_data_smithi173_74914_object56729:head# 2024-08-30T23:37:06.546 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.546 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbb9d13:::benchmark_data_smithi173_74914_object64785:head# 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbc5337:::benchmark_data_smithi173_74914_object92289:head# 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbcdb23:::benchmark_data_smithi173_74914_object80996:head# 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbd5012:::benchmark_data_smithi173_74914_object78996:head# 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbd5bc3:::benchmark_data_smithi173_74914_object66736:head# 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.547 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbe8718:::benchmark_data_smithi173_74914_object94133:head# 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdbf997c:::benchmark_data_smithi173_74914_object67956:head# 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdc2edf9:::benchmark_data_smithi173_74914_object92650:head# 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdc32571:::benchmark_data_smithi173_74914_object64740:head# 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdc71b94:::benchmark_data_smithi173_74914_object62966:head# 2024-08-30T23:37:06.548 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdcb6df2:::benchmark_data_smithi173_74914_object92179:head# 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdcb7055:::benchmark_data_smithi173_74914_object89833:head# 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdcec965:::benchmark_data_smithi173_74914_object94050:head# 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd177ce:::benchmark_data_smithi173_74914_object97542:head# 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd36a23:::benchmark_data_smithi173_74914_object96975:head# 2024-08-30T23:37:06.549 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd513da:::benchmark_data_smithi173_74914_object69123:head# 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd65dd0:::benchmark_data_smithi173_74914_object57915:head# 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd8c59c:::benchmark_data_smithi173_74914_object60372:head# 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd8cc50:::benchmark_data_smithi173_74914_object79410:head# 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd9407c:::benchmark_data_smithi173_74914_object62164:head# 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.550 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdd9e3a7:::benchmark_data_smithi173_74914_object87380:head# 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fddaa823:::benchmark_data_smithi173_74914_object56396:head# 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fddb3cfb:::benchmark_data_smithi173_74914_object67475:head# 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fddc10d8:::benchmark_data_smithi173_74914_object79836:head# 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fddc7219:::benchmark_data_smithi173_74914_object96772:head# 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fddf2ddf:::benchmark_data_smithi173_74914_object85352:head# 2024-08-30T23:37:06.551 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fde16151:::benchmark_data_smithi173_74914_object82586:head# 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fde3de36:::benchmark_data_smithi173_74914_object80616:head# 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fde717df:::benchmark_data_smithi173_74914_object78502:head# 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fde9991d:::benchmark_data_smithi173_74914_object78827:head# 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fde9d12e:::benchmark_data_smithi173_74914_object65478:head# 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.552 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdec3904:::benchmark_data_smithi173_74914_object62758:head# 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdeec741:::benchmark_data_smithi173_74914_object89266:head# 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdef64be:::benchmark_data_smithi173_74914_object78036:head# 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdefbeb8:::benchmark_data_smithi173_74914_object60243:head# 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf2fb54:::benchmark_data_smithi173_74914_object71815:head# 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf33d51:::benchmark_data_smithi173_74914_object68840:head# 2024-08-30T23:37:06.553 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf3aa42:::benchmark_data_smithi173_74914_object70269:head# 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf3eea8:::benchmark_data_smithi173_74914_object75427:head# 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf5f9a3:::benchmark_data_smithi173_74914_object76968:head# 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf638e3:::benchmark_data_smithi173_74914_object57141:head# 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf6ad90:::benchmark_data_smithi173_74914_object72188:head# 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf6b82b:::benchmark_data_smithi173_74914_object75463:head# 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.554 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdf7ad92:::benchmark_data_smithi173_74914_object73598:head# 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfabdf6:::benchmark_data_smithi173_74914_object56419:head# 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfb18a3:::benchmark_data_smithi173_74914_object54102:head# 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfb3f2d:::benchmark_data_smithi173_74914_object76072:head# 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfb62dd:::benchmark_data_smithi173_74914_object86254:head# 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfb7fe5:::benchmark_data_smithi173_74914_object80508:head# 2024-08-30T23:37:06.555 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfd8dd6:::benchmark_data_smithi173_74914_object63446:head# 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfe0c97:::benchmark_data_smithi173_74914_object73420:head# 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fdfed93b:::benchmark_data_smithi173_74914_object53727:head# 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe027a92:::benchmark_data_smithi173_74914_object62402:head# 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe047e2e:::benchmark_data_smithi173_74914_object92560:head# 2024-08-30T23:37:06.556 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe051fab:::benchmark_data_smithi173_74914_object75371:head# 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe06d79e:::benchmark_data_smithi173_74914_object67250:head# 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe079dfc:::benchmark_data_smithi173_74914_object60588:head# 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe08f8cd:::benchmark_data_smithi173_74914_object68915:head# 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe093672:::benchmark_data_smithi173_74914_object58565:head# 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.557 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe0aa82d:::benchmark_data_smithi173_74914_object92681:head# 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe0aadfa:::benchmark_data_smithi173_74914_object92152:head# 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe0aceee:::benchmark_data_smithi173_74914_object95782:head# 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe0b218c:::benchmark_data_smithi173_74914_object96267:head# 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe0b91b0:::benchmark_data_smithi173_74914_object83361:head# 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe0c8100:::benchmark_data_smithi173_74914_object67406:head# 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.558 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe1001d1:::benchmark_data_smithi173_74914_object76960:head# 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe117a03:::benchmark_data_smithi173_74914_object71492:head# 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe11e089:::benchmark_data_smithi173_74914_object72786:head# 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe132759:::benchmark_data_smithi173_74914_object67054:head# 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe13f9cb:::benchmark_data_smithi173_74914_object82727:head# 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe15770c:::benchmark_data_smithi173_74914_object75538:head# 2024-08-30T23:37:06.559 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe165038:::benchmark_data_smithi173_74914_object83573:head# 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe18b8e4:::benchmark_data_smithi173_74914_object74784:head# 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe1b51dd:::benchmark_data_smithi173_74914_object82298:head# 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe1b68b0:::benchmark_data_smithi173_74914_object73767:head# 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe1c429f:::benchmark_data_smithi173_74914_object55946:head# 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe1cdbfd:::benchmark_data_smithi173_74914_object84064:head# 2024-08-30T23:37:06.560 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe205a76:::benchmark_data_smithi173_74914_object69213:head# 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe223219:::benchmark_data_smithi173_74914_object71902:head# 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe239b3f:::benchmark_data_smithi173_74914_object82057:head# 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe23d738:::benchmark_data_smithi173_74914_object95829:head# 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe23df97:::benchmark_data_smithi173_74914_object92768:head# 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe278347:::benchmark_data_smithi173_74914_object62731:head# 2024-08-30T23:37:06.561 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe27c905:::benchmark_data_smithi173_74914_object52930:head# 2024-08-30T23:37:06.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.562 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe280c10:::benchmark_data_smithi173_74914_object88097:head# 2024-08-30T23:37:06.562 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:06.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe281e3a:::benchmark_data_smithi173_74914_object95328:head# 2024-08-30T23:37:06.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.563 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe28df59:::benchmark_data_smithi173_74914_object76102:head# 2024-08-30T23:37:06.563 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:06.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:06.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe293e0e:::benchmark_data_smithi173_74914_object69353:head# 2024-08-30T23:37:06.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe2b38b7:::benchmark_data_smithi173_74914_object53273:head# 2024-08-30T23:37:06.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.604 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe2d022c:::benchmark_data_smithi173_74914_object63014:head# 2024-08-30T23:37:06.604 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe302b0c:::benchmark_data_smithi173_74914_object60775:head# 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe33db52:::benchmark_data_smithi173_74914_object65290:head# 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe3419ed:::benchmark_data_smithi173_74914_object62698:head# 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe374cd5:::benchmark_data_smithi173_74914_object60264:head# 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.605 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe39861e:::benchmark_data_smithi173_74914_object54979:head# 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe3cab97:::benchmark_data_smithi173_74914_object78079:head# 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe3f5800:::benchmark_data_smithi173_74914_object76463:head# 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe403869:::benchmark_data_smithi173_74914_object71491:head# 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe40a04d:::benchmark_data_smithi173_74914_object85888:head# 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.606 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe40c2da:::benchmark_data_smithi173_74914_object66606:head# 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe4197a5:::benchmark_data_smithi173_74914_object91209:head# 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe449425:::benchmark_data_smithi173_74914_object66008:head# 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe45448b:::benchmark_data_smithi173_74914_object61853:head# 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe463096:::benchmark_data_smithi173_74914_object55443:head# 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe463a43:::benchmark_data_smithi173_74914_object58537:head# 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.607 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe464417:::benchmark_data_smithi173_74914_object70961:head# 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe46fa24:::benchmark_data_smithi173_74914_object83098:head# 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe485e25:::benchmark_data_smithi173_74914_object86919:head# 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe48a99d:::benchmark_data_smithi173_74914_object70673:head# 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe49991f:::benchmark_data_smithi173_74914_object63548:head# 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.608 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe49d884:::benchmark_data_smithi173_74914_object71572:head# 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe49f1f0:::benchmark_data_smithi173_74914_object96786:head# 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe4d8cb5:::benchmark_data_smithi173_74914_object53052:head# 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe4ffceb:::benchmark_data_smithi173_74914_object90735:head# 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe53f681:::benchmark_data_smithi173_74914_object86070:head# 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.609 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe55da61:::benchmark_data_smithi173_74914_object53536:head# 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe55e4c6:::benchmark_data_smithi173_74914_object93060:head# 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe57181d:::benchmark_data_smithi173_74914_object94335:head# 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe5970f4:::benchmark_data_smithi173_74914_object94119:head# 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe59bdc3:::benchmark_data_smithi173_74914_object67287:head# 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe5b6c0d:::benchmark_data_smithi173_74914_object95623:head# 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.610 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe5c53da:::benchmark_data_smithi173_74914_object70974:head# 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe5cd580:::benchmark_data_smithi173_74914_object55509:head# 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe5d865d:::benchmark_data_smithi173_74914_object72964:head# 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe626600:::benchmark_data_smithi173_74914_object80169:head# 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe665d71:::benchmark_data_smithi173_74914_object74481:head# 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe6abd42:::benchmark_data_smithi173_74914_object86387:head# 2024-08-30T23:37:06.611 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe6dd34e:::benchmark_data_smithi173_74914_object95582:head# 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe6e2a1e:::benchmark_data_smithi173_74914_object54182:head# 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe6eb744:::benchmark_data_smithi173_74914_object73679:head# 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe6ffa5c:::benchmark_data_smithi173_74914_object57669:head# 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe7238ae:::benchmark_data_smithi173_74914_object87433:head# 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe728c3b:::benchmark_data_smithi173_74914_object87661:head# 2024-08-30T23:37:06.612 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe73ed9b:::benchmark_data_smithi173_74914_object83471:head# 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe75b7b0:::benchmark_data_smithi173_74914_object58563:head# 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe78934a:::benchmark_data_smithi173_74914_object68089:head# 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe79ad98:::benchmark_data_smithi173_74914_object64551:head# 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe79d185:::benchmark_data_smithi173_74914_object98040:head# 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.613 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe79e3f5:::benchmark_data_smithi173_74914_object82564:head# 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe7a501d:::benchmark_data_smithi173_74914_object87308:head# 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe7e9379:::benchmark_data_smithi173_74914_object87026:head# 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe7f2d24:::benchmark_data_smithi173_74914_object69092:head# 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe81b11f:::benchmark_data_smithi173_74914_object60286:head# 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe84b1bc:::benchmark_data_smithi173_74914_object55777:head# 2024-08-30T23:37:06.614 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe856042:::benchmark_data_smithi173_74914_object64263:head# 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe8807b4:::benchmark_data_smithi173_74914_object84825:head# 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe8cac9a:::benchmark_data_smithi173_74914_object87104:head# 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe8f2b50:::benchmark_data_smithi173_74914_object97323:head# 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe92317c:::benchmark_data_smithi173_74914_object59305:head# 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.615 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9247b5:::benchmark_data_smithi173_74914_object74714:head# 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe932876:::benchmark_data_smithi173_74914_object73501:head# 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe932c1c:::benchmark_data_smithi173_74914_object75201:head# 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe964bd4:::benchmark_data_smithi173_74914_object79370:head# 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe97204b:::benchmark_data_smithi173_74914_object81126:head# 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9926b3:::benchmark_data_smithi173_74914_object83997:head# 2024-08-30T23:37:06.616 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe998bef:::benchmark_data_smithi173_74914_object57867:head# 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9b027c:::benchmark_data_smithi173_74914_object63409:head# 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9b09ca:::benchmark_data_smithi173_74914_object84256:head# 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9b7d4e:::benchmark_data_smithi173_74914_object95573:head# 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9c70d8:::benchmark_data_smithi173_74914_object67963:head# 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9e82d4:::benchmark_data_smithi173_74914_object80686:head# 2024-08-30T23:37:06.617 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fe9f805e:::benchmark_data_smithi173_74914_object78262:head# 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fea01f75:::benchmark_data_smithi173_74914_object59354:head# 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fea0b2bc:::benchmark_data_smithi173_74914_object73450:head# 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fea1a7a7:::benchmark_data_smithi173_74914_object65358:head# 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fea53c32:::benchmark_data_smithi173_74914_object92199:head# 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fea93451:::benchmark_data_smithi173_74914_object82719:head# 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.618 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fea9d9a2:::benchmark_data_smithi173_74914_object67066:head# 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feaa1105:::benchmark_data_smithi173_74914_object85897:head# 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feaa1e8a:::benchmark_data_smithi173_74914_object75082:head# 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feaaed0c:::benchmark_data_smithi173_74914_object83089:head# 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feae227e:::benchmark_data_smithi173_74914_object59285:head# 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feb295dd:::benchmark_data_smithi173_74914_object72589:head# 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.619 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feb5b1fd:::benchmark_data_smithi173_74914_object88990:head# 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feb7f974:::benchmark_data_smithi173_74914_object66782:head# 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:febd0ae1:::benchmark_data_smithi173_74914_object64137:head# 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec0aa44:::benchmark_data_smithi173_74914_object90579:head# 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec0c465:::benchmark_data_smithi173_74914_object60770:head# 2024-08-30T23:37:06.620 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec20b57:::benchmark_data_smithi173_74914_object72474:head# 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec4303e:::benchmark_data_smithi173_74914_object58122:head# 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec7d0e5:::benchmark_data_smithi173_74914_object73807:head# 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec97a48:::benchmark_data_smithi173_74914_object70728:head# 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fec98ee7:::benchmark_data_smithi173_74914_object80734:head# 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.621 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fecda303:::benchmark_data_smithi173_74914_object59086:head# 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fece6dc5:::benchmark_data_smithi173_74914_object94089:head# 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fed0b121:::benchmark_data_smithi173_74914_object97219:head# 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fed4cdd3:::benchmark_data_smithi173_74914_object63413:head# 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fed6d0ed:::benchmark_data_smithi173_74914_object83994:head# 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fed6dd10:::benchmark_data_smithi173_74914_object62712:head# 2024-08-30T23:37:06.622 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fed74290:::benchmark_data_smithi173_74914_object58938:head# 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fedaa12a:::benchmark_data_smithi173_74914_object66141:head# 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fedb1ebb:::benchmark_data_smithi173_74914_object56036:head# 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fedbc954:::benchmark_data_smithi173_74914_object59926:head# 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fedc35df:::benchmark_data_smithi173_74914_object95229:head# 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.623 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fedc9380:::benchmark_data_smithi173_74914_object76832:head# 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee0ce0d:::benchmark_data_smithi173_74914_object83401:head# 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee2003a:::benchmark_data_smithi173_74914_object71579:head# 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee21295:::benchmark_data_smithi173_74914_object63793:head# 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee239cc:::benchmark_data_smithi173_74914_object67491:head# 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee24f79:::benchmark_data_smithi173_74914_object90079:head# 2024-08-30T23:37:06.624 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee3ce19:::benchmark_data_smithi173_74914_object69868:head# 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee41ed5:::benchmark_data_smithi173_74914_object57105:head# 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee65544:::benchmark_data_smithi173_74914_object93177:head# 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee7916e:::benchmark_data_smithi173_74914_object79017:head# 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee83231:::benchmark_data_smithi173_74914_object62703:head# 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.625 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee837b4:::benchmark_data_smithi173_74914_object94393:head# 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fee9f61d:::benchmark_data_smithi173_74914_object75539:head# 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feea92ec:::benchmark_data_smithi173_74914_object82114:head# 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feeda329:::benchmark_data_smithi173_74914_object55549:head# 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feee7ae3:::benchmark_data_smithi173_74914_object90294:head# 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.626 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fef03003:::benchmark_data_smithi173_74914_object58393:head# 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fef0b6ea:::benchmark_data_smithi173_74914_object55625:head# 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fef3f9fd:::benchmark_data_smithi173_74914_object94382:head# 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fef505e8:::benchmark_data_smithi173_74914_object98238:head# 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fef5cbb7:::benchmark_data_smithi173_74914_object81971:head# 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fef83cfe:::benchmark_data_smithi173_74914_object95420:head# 2024-08-30T23:37:06.627 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fefa3cf0:::benchmark_data_smithi173_74914_object72972:head# 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fefa88f7:::benchmark_data_smithi173_74914_object59940:head# 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fefab923:::benchmark_data_smithi173_74914_object54294:head# 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fefde6d9:::benchmark_data_smithi173_74914_object75632:head# 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:feffc1ca:::benchmark_data_smithi173_74914_object93940:head# 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.628 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff0279e2:::benchmark_data_smithi173_74914_object58731:head# 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff02f067:::benchmark_data_smithi173_74914_object65688:head# 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff037f68:::benchmark_data_smithi173_74914_object54344:head# 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff058334:::benchmark_data_smithi173_74914_object60312:head# 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff070511:::benchmark_data_smithi173_74914_object80781:head# 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.629 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff079a47:::benchmark_data_smithi173_74914_object78524:head# 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff07d2e8:::benchmark_data_smithi173_74914_object78135:head# 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff091fa4:::benchmark_data_smithi173_74914_object74234:head# 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff0a601b:::benchmark_data_smithi173_74914_object86729:head# 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff0b6051:::benchmark_data_smithi173_74914_object57705:head# 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff0c36aa:::benchmark_data_smithi173_74914_object83512:head# 2024-08-30T23:37:06.630 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff11fd9a:::benchmark_data_smithi173_74914_object80595:head# 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff17eaf0:::benchmark_data_smithi173_74914_object80266:head# 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff1879bd:::benchmark_data_smithi173_74914_object81139:head# 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff189bd8:::benchmark_data_smithi173_74914_object94949:head# 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff18d78f:::benchmark_data_smithi173_74914_object55518:head# 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.631 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff1ac7c7:::benchmark_data_smithi173_74914_object89171:head# 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff1b1728:::benchmark_data_smithi173_74914_object77991:head# 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff1c0275:::benchmark_data_smithi173_74914_object75401:head# 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff1dbe8e:::benchmark_data_smithi173_74914_object62257:head# 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff1e07b8:::benchmark_data_smithi173_74914_object70940:head# 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.632 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff21e197:::benchmark_data_smithi173_74914_object88034:head# 2024-08-30T23:37:06.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff2273ef:::benchmark_data_smithi173_74914_object91192:head# 2024-08-30T23:37:06.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.633 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff23efda:::benchmark_data_smithi173_74914_object73669:head# 2024-08-30T23:37:06.633 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff24cd8a:::benchmark_data_smithi173_74914_object82457:head# 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff2628f3:::benchmark_data_smithi173_74914_object72233:head# 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff27750b:::benchmark_data_smithi173_74914_object67136:head# 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff2be05f:::benchmark_data_smithi173_74914_object83478:head# 2024-08-30T23:37:06.637 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff2d07f7:::benchmark_data_smithi173_74914_object81136:head# 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff2e1b1a:::benchmark_data_smithi173_74914_object80020:head# 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff377dcd:::benchmark_data_smithi173_74914_object89081:head# 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff3797bf:::benchmark_data_smithi173_74914_object82818:head# 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff3e2596:::benchmark_data_smithi173_74914_object83437:head# 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.638 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff3f5012:::benchmark_data_smithi173_74914_object77156:head# 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff4077d4:::benchmark_data_smithi173_74914_object92602:head# 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff4092cd:::benchmark_data_smithi173_74914_object86440:head# 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff41f59d:::benchmark_data_smithi173_74914_object93244:head# 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff420711:::benchmark_data_smithi173_74914_object75036:head# 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff431387:::benchmark_data_smithi173_74914_object72069:head# 2024-08-30T23:37:06.639 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff431aee:::benchmark_data_smithi173_74914_object56559:head# 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff437a4c:::benchmark_data_smithi173_74914_object66017:head# 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff43a777:::benchmark_data_smithi173_74914_object70572:head# 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff44eda6:::benchmark_data_smithi173_74914_object64187:head# 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff476938:::benchmark_data_smithi173_74914_object98081:head# 2024-08-30T23:37:06.640 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff491258:::benchmark_data_smithi173_74914_object56670:head# 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff493c50:::benchmark_data_smithi173_74914_object94486:head# 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff495124:::benchmark_data_smithi173_74914_object86867:head# 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff4bc0d0:::benchmark_data_smithi173_74914_object61115:head# 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.641 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff4d1a9f:::benchmark_data_smithi173_74914_object72793:head# 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff4de42b:::benchmark_data_smithi173_74914_object72775:head# 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff504451:::benchmark_data_smithi173_74914_object71450:head# 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff50886e:::benchmark_data_smithi173_74914_object70714:head# 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff51a43c:::benchmark_data_smithi173_74914_object70648:head# 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff520c6e:::benchmark_data_smithi173_74914_object67178:head# 2024-08-30T23:37:06.642 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff521a83:::benchmark_data_smithi173_74914_object92242:head# 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff524ad0:::benchmark_data_smithi173_74914_object70736:head# 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff52554e:::benchmark_data_smithi173_74914_object90908:head# 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff537860:::benchmark_data_smithi173_74914_object82326:head# 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff5406bf:::benchmark_data_smithi173_74914_object70760:head# 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.643 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff55ebe4:::benchmark_data_smithi173_74914_object74168:head# 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff59ef2d:::benchmark_data_smithi173_74914_object62631:head# 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff5c5740:::benchmark_data_smithi173_74914_object59445:head# 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff5d08e9:::benchmark_data_smithi173_74914_object72646:head# 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff601459:::benchmark_data_smithi173_74914_object78324:head# 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff60ced1:::benchmark_data_smithi173_74914_object53398:head# 2024-08-30T23:37:06.644 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff6317c2:::benchmark_data_smithi173_74914_object90669:head# 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff645874:::benchmark_data_smithi173_74914_object95611:head# 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff65d1a1:::benchmark_data_smithi173_74914_object60046:head# 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff67e4f5:::benchmark_data_smithi173_74914_object93923:head# 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff68aea6:::benchmark_data_smithi173_74914_object77860:head# 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.645 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff698475:::benchmark_data_smithi173_74914_object71990:head# 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff6bd63e:::benchmark_data_smithi173_74914_object78381:head# 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff6cea18:::benchmark_data_smithi173_74914_object72340:head# 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff6d0416:::benchmark_data_smithi173_74914_object73736:head# 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff6f1878:::benchmark_data_smithi173_74914_object80134:head# 2024-08-30T23:37:06.646 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff729524:::benchmark_data_smithi173_74914_object70270:head# 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff74064e:::benchmark_data_smithi173_74914_object92812:head# 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff74f80f:::benchmark_data_smithi173_74914_object55912:head# 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff785b9c:::benchmark_data_smithi173_74914_object90799:head# 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff7aada1:::benchmark_data_smithi173_74914_object86873:head# 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.647 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff7b86be:::benchmark_data_smithi173_74914_object78016:head# 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff7d8058:::benchmark_data_smithi173_74914_object60722:head# 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8157f6:::benchmark_data_smithi173_74914_object82199:head# 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8252f6:::benchmark_data_smithi173_74914_object66812:head# 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8586f3:::benchmark_data_smithi173_74914_object61085:head# 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff85b45a:::benchmark_data_smithi173_74914_object82460:head# 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8662b8:::benchmark_data_smithi173_74914_object96191:head# 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff86643b:::benchmark_data_smithi173_74914_object59275:head# 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff86e9d2:::benchmark_data_smithi173_74914_object83018:head# 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff878698:::benchmark_data_smithi173_74914_object80629:head# 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8932e8:::benchmark_data_smithi173_74914_object66835:head# 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.649 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8a202c:::benchmark_data_smithi173_74914_object66647:head# 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8b59d2:::benchmark_data_smithi173_74914_object89910:head# 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8b8302:::benchmark_data_smithi173_74914_object71400:head# 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff8fc29b:::benchmark_data_smithi173_74914_object59487:head# 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff90188c:::benchmark_data_smithi173_74914_object70891:head# 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff939f3d:::benchmark_data_smithi173_74914_object57265:head# 2024-08-30T23:37:06.650 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff94d9a6:::benchmark_data_smithi173_74914_object90163:head# 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff94f53a:::benchmark_data_smithi173_74914_object53920:head# 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff9739c7:::benchmark_data_smithi173_74914_object56436:head# 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff97a9e2:::benchmark_data_smithi173_74914_object97850:head# 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff980bfe:::benchmark_data_smithi173_74914_object77763:head# 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff9ad1e1:::benchmark_data_smithi173_74914_object90662:head# 2024-08-30T23:37:06.651 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff9be41d:::benchmark_data_smithi173_74914_object82927:head# 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff9c46ec:::benchmark_data_smithi173_74914_object79551:head# 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff9f5f7c:::benchmark_data_smithi173_74914_object75526:head# 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ff9fae67:::benchmark_data_smithi173_74914_object58713:head# 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa015b0:::benchmark_data_smithi173_74914_object92345:head# 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.652 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa1b5fd:::benchmark_data_smithi173_74914_object96162:head# 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa2d026:::benchmark_data_smithi173_74914_object73310:head# 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa586db:::benchmark_data_smithi173_74914_object53263:head# 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa720a5:::benchmark_data_smithi173_74914_object98237:head# 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa79112:::benchmark_data_smithi173_74914_object57592:head# 2024-08-30T23:37:06.653 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffa90558:::benchmark_data_smithi173_74914_object86730:head# 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffab797e:::benchmark_data_smithi173_74914_object65029:head# 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffacf267:::benchmark_data_smithi173_74914_object63615:head# 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffad08fa:::benchmark_data_smithi173_74914_object65171:head# 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffae2a11:::benchmark_data_smithi173_74914_object97326:head# 2024-08-30T23:37:06.654 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffaffd08:::benchmark_data_smithi173_74914_object58598:head# 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb0ad8a:::benchmark_data_smithi173_74914_object61277:head# 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb0b0ae:::benchmark_data_smithi173_74914_object58844:head# 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb35d5a:::benchmark_data_smithi173_74914_object64339:head# 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb41d34:::benchmark_data_smithi173_74914_object73471:head# 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.655 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb6075e:::benchmark_data_smithi173_74914_object60512:head# 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb70584:::benchmark_data_smithi173_74914_object67768:head# 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb7a804:::benchmark_data_smithi173_74914_object88147:head# 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb8213f:::benchmark_data_smithi173_74914_object55872:head# 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb9ac09:::benchmark_data_smithi173_74914_object52946:head# 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffb9c6c9:::benchmark_data_smithi173_74914_object66856:head# 2024-08-30T23:37:06.656 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffbb013e:::benchmark_data_smithi173_74914_object82800:head# 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffbc9bba:::benchmark_data_smithi173_74914_object90019:head# 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffbcfe8b:::benchmark_data_smithi173_74914_object76749:head# 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffbe53ff:::benchmark_data_smithi173_74914_object59504:head# 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffc11248:::benchmark_data_smithi173_74914_object71733:head# 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.657 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffc18022:::benchmark_data_smithi173_74914_object57819:head# 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffc4024f:::benchmark_data_smithi173_74914_object90936:head# 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffc99d89:::benchmark_data_smithi173_74914_object70451:head# 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffc9e1e6:::benchmark_data_smithi173_74914_object76213:head# 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffcd7c18:::benchmark_data_smithi173_74914_object82561:head# 2024-08-30T23:37:06.658 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffcf2df5:::benchmark_data_smithi173_74914_object64429:head# 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffcf5ebd:::benchmark_data_smithi173_74914_object58457:head# 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd02607:::benchmark_data_smithi173_74914_object97382:head# 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd066e9:::benchmark_data_smithi173_74914_object93716:head# 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd125da:::benchmark_data_smithi173_74914_object57099:head# 2024-08-30T23:37:06.659 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd51fc1:::benchmark_data_smithi173_74914_object85660:head# 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd73f63:::benchmark_data_smithi173_74914_object76176:head# 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd7dd6f:::benchmark_data_smithi173_74914_object73770:head# 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffd8df6d:::benchmark_data_smithi173_74914_object55160:head# 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffdaabe7:::benchmark_data_smithi173_74914_object57003:head# 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.660 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe0df6e:::benchmark_data_smithi173_74914_object60827:head# 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe1a186:::benchmark_data_smithi173_74914_object55944:head# 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe37aab:::benchmark_data_smithi173_74914_object88718:head# 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe3873a:::benchmark_data_smithi173_74914_object82316:head# 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe48c5c:::benchmark_data_smithi173_74914_object88881:head# 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe5bbcc:::benchmark_data_smithi173_74914_object96351:head# 2024-08-30T23:37:06.661 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe6cf62:::benchmark_data_smithi173_74914_object65102:head# 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffe8c04b:::benchmark_data_smithi173_74914_object72526:head# 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffea5099:::benchmark_data_smithi173_74914_object75774:head# 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffed5813:::benchmark_data_smithi173_74914_object68974:head# 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffedb42a:::benchmark_data_smithi173_74914_object95553:head# 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffee0921:::benchmark_data_smithi173_74914_object65728:head# 2024-08-30T23:37:06.662 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:ffef12ef:::benchmark_data_smithi173_74914_object56209:head# 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff044f3:::benchmark_data_smithi173_74914_object67131:head# 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff0a487:::benchmark_data_smithi173_74914_object84421:head# 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff19876:::benchmark_data_smithi173_74914_object54098:head# 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff7011e:::benchmark_data_smithi173_74914_object67821:head# 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff907d7:::benchmark_data_smithi173_74914_object62020:head# 2024-08-30T23:37:06.663 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff955fa:::benchmark_data_smithi173_74914_object68921:head# 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fff9c6fa:::benchmark_data_smithi173_74914_object69489:head# 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fffd9315:::benchmark_data_smithi173_74914_object73236:head# 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fffe2766:::benchmark_data_smithi173_74914_object81819:head# 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:Write #4:fffe6b19:::benchmark_data_smithi173_74914_object71179:head# 2024-08-30T23:37:06.664 INFO:teuthology.orchestra.run.smithi106.stdout:snapset 0=[]:{} 2024-08-30T23:37:06.665 INFO:teuthology.orchestra.run.smithi106.stdout:write_pg epoch 368 info 4.f( v 368'9353 (0'0,368'9353] local-lis/les=306/307 n=2805 ec=306/306 lis/c=306/306 les/c/f=307/307/0 sis=306) 2024-08-30T23:37:06.665 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:37:06.665 INFO:teuthology.orchestra.run.smithi106.stdout:Import successful 2024-08-30T23:37:06.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:06.667 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:06.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:06.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:06.713 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4773/40548 objects degraded (11.771%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:06.713 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2760882793' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:06.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:06.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:06.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4773/40548 objects degraded (11.771%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:06.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2760882793' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:06.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:06.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:06.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:06.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:06.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4773/40548 objects degraded (11.771%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2760882793' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:07.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:07.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:07.184 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:37:07.184 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:37:07.185 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:37:07.185 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:37:07.185 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:37:07.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:07.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:07.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:07.353 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:07.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:07.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:07.585 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.608 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:07.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:07.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:07.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:07.810 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:07.810 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2760882793' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:07.810 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: osdmap e378: 12 total, 11 up, 12 in 2024-08-30T23:37:07.810 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:37:07.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: 4.e scrub starts 2024-08-30T23:37:07.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: 4.e scrub ok 2024-08-30T23:37:07.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:07 smithi173 ceph-mon[20554]: pgmap v916: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 733 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 4484/35172 objects degraded (12.749%) 2024-08-30T23:37:07.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2760882793' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: osdmap e378: 12 total, 11 up, 12 in 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: 4.e scrub starts 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: 4.e scrub ok 2024-08-30T23:37:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:07 smithi106 ceph-mon[16603]: pgmap v916: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 733 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 4484/35172 objects degraded (12.749%) 2024-08-30T23:37:07.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:07.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:07.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:07.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:37:08.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2760882793' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: osdmap e378: 12 total, 11 up, 12 in 2024-08-30T23:37:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:37:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: 4.e scrub starts 2024-08-30T23:37:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: 4.e scrub ok 2024-08-30T23:37:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:07 smithi136 ceph-mon[20207]: pgmap v916: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 733 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s; 4484/35172 objects degraded (12.749%) 2024-08-30T23:37:08.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:08.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:08.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:08.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:08.357 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:08.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:08.391 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:08.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:08.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:08.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:08.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:08.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:08.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:08.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:37:08.770 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:08.782 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:37:08.782 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:37:08.782 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:08 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:37:08.793 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:37:08.793 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:37:08.793 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:08 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:37:08.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:08.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:08.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:164675 3.1 3:8010289e:::benchmark_data_smithi173_60780_object61221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:37:08.899 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 258.92815862399999, 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27192025399999997, 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 164675 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969741+0000", 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0329999999999998e-06 2024-08-30T23:37:08.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969748+0000", 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.241e-06 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969761+0000", 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3162999999999999e-05 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969768+0000", 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5429999999999996e-06 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.970901+0000", 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011326870000000001 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.971033+0000", 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013183100000000001 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047341+0000", 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076308670999999995 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047456+0000", 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114108 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047620+0000", 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164504 2024-08-30T23:37:08.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.061526+0000", 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013906091000000001 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.065492+0000", 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039661649999999998 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.240280+0000", 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17478812399999999 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.240349+0000", 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8520000000000001e-05 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.241660+0000", 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001310573 2024-08-30T23:37:08.906 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e300,280 3:83685fcd:::benchmark_data_smithi173_60780_object2411:head,257'6623)", 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 204.49573453400001, 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41143703700000001, 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402164+0000", 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:37:08.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402168+0000", 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0570000000000003e-06 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812764+0000", 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41059553700000001 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812774+0000", 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529e-05 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812860+0000", 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5934999999999996e-05 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.813600+0000", 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074028899999999999 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.910 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e301,280 3:8490e9ac:::benchmark_data_smithi173_60780_object7986:head,258'6976)", 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 201.71892004700001, 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70969468099999999, 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:37:08.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4199999999999995e-07 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178982+0000", 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9779999999999999e-06 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887644+0000", 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70866116300000004 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887661+0000", 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6954999999999999e-05 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887778+0000", 2024-08-30T23:37:08.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117812 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.888673+0000", 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00089413100000000005 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:24732 4.a e315/306 4:5449864d:::benchmark_data_smithi173_74914_object24731:head v 315'1539, mlcod=315'1535)", 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:37:08.915 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 172.38857305799999, 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24331156100000001, 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509326+0000", 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509403+0000", 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7809e-05 2024-08-30T23:37:08.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751493+0000", 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24208919500000001 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751500+0000", 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5190000000000003e-06 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751549+0000", 2024-08-30T23:37:08.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8891000000000003e-05 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751566+0000", 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7102999999999999e-05 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.752577+0000", 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001011248 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.752636+0000", 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9079000000000001e-05 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31037 4.5 4.306472d5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.21548494199999, 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0020195190000001, 2024-08-30T23:37:08.920 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31037 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682415+0000", 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6589999999999999e-06 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682488+0000", 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3084999999999996e-05 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682498+0000", 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0644e-05 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682506+0000", 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0229999999999992e-06 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682702+0000", 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019531299999999999 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682713+0000", 2024-08-30T23:37:08.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1778e-05 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684400+0000", 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016865800000001 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684432+0000", 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2437000000000001e-05 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.925 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31040 4.5 4.eb15935 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.215327355, 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.001902058, 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31040 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.926 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:37:08.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682627+0000", 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5911000000000001e-05 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682637+0000", 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0363999999999999e-05 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682642+0000", 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8509999999999998e-06 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682765+0000", 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123311 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682799+0000", 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3145000000000002e-05 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684465+0000", 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016660740000001 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684473+0000", 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9149999999999996e-06 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31060 4.5 4.c6dc7475 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.20592213099999, 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99255960700000001, 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31060 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.931 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691977+0000", 2024-08-30T23:37:08.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.066e-06 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691998+0000", 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.158e-05 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692011+0000", 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2848e-05 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692019+0000", 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8990000000000001e-06 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692069+0000", 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0080999999999997e-05 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692076+0000", 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1450000000000002e-06 2024-08-30T23:37:08.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684524+0000", 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99244810000000006 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684535+0000", 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0888e-05 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31063 4.5 4.5c985545 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 167.76141608399999, 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54808229100000005, 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.936 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31063 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136484+0000", 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7069999999999999e-06 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136509+0000", 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4990999999999998e-05 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136526+0000", 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7178999999999999e-05 2024-08-30T23:37:08.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136533+0000", 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8789999999999997e-06 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136588+0000", 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5018000000000001e-05 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:37:08.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136596+0000", 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9820000000000008e-06 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684558+0000", 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54796194799999998 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684564+0000", 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5869999999999998e-06 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.941 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:44161 4.7 4:e6ac4d14:::benchmark_data_smithi173_74914_object44160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:35.888114+0000", 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 153.00978434999999, 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22262327000000001, 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.942 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 44161 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888114+0000", 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888114+0000", 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888115+0000", 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.528e-06 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888135+0000", 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0152e-05 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888158+0000", 2024-08-30T23:37:08.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2388999999999999e-05 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888166+0000", 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2169999999999994e-06 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888214+0000", 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7944e-05 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888349+0000", 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013541099999999999 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.888505+0000", 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015614900000000001 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.890107+0000", 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001601568 2024-08-30T23:37:08.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:35.905284+0000", 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015176766 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:36.110110+0000", 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20482651499999999 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:36.110148+0000", 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7489e-05 2024-08-30T23:37:08.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:36.110737+0000", 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058914199999999996 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:50603 4.a e329/318 4:54bdef61:::benchmark_data_smithi173_74914_object50602:head v 329'3160, mlcod=315'1602)", 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 147.23657030000001, 2024-08-30T23:37:08.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25119703700000001, 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:37:08.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661786+0000", 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045819399999999998 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.910994+0000", 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.249208242 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.910999+0000", 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8640000000000004e-06 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.911049+0000", 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0216000000000002e-05 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.911074+0000", 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4484000000000001e-05 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.912487+0000", 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001413648 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.912525+0000", 2024-08-30T23:37:08.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7171e-05 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(4.4 329/318 [PushOp(4:2f097f2f:::benchmark_data_smithi173_74914_object29004:head, version: 316'1834, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:2f097f2f:::benchmark_data_smithi173_74914_object29004:head@316'1834, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 146.83031043099999, 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51130092400000005, 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067588+0000", 2024-08-30T23:37:08.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6199999999999997e-07 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067620+0000", 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2128999999999997e-05 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577769+0000", 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.510148398 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577780+0000", 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1617000000000001e-05 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577885+0000", 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104666 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577899+0000", 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4568e-05 2024-08-30T23:37:08.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.578888+0000", 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098888399999999999 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.5 335/318 [PushReplyOp(4:a5afb7d9:::benchmark_data_smithi173_74914_object29619:head)])", 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 132.63911352400001, 2024-08-30T23:37:08.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.227084861, 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258785+0000", 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9500000000000003e-07 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258788+0000", 2024-08-30T23:37:08.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:37:08.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:37:08.960 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:08 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2160000000000002e-06 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485597+0000", 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.226809118 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485606+0000", 2024-08-30T23:37:08.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1260000000000004e-06 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485684+0000", 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7238999999999997e-05 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485869+0000", 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018566700000000001 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62091 4.4 e337/331 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head v 337'3874, mlcod=336'3872)", 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.73164385999999, 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.317048422, 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166315+0000", 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0946000000000002e-05 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481856+0000", 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31554066800000002 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481863+0000", 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0960000000000003e-06 2024-08-30T23:37:08.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481922+0000", 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8789000000000002e-05 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481941+0000", 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9307999999999999e-05 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.483268+0000", 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013262339999999999 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.483302+0000", 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4947999999999998e-05 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62105 4.4 e337/331 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head v 337'3875, mlcod=336'3872)", 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.71805764499999, 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34727296499999999, 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179841+0000", 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.68e-07 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179895+0000", 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4670999999999999e-05 2024-08-30T23:37:08.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525681+0000", 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34578567199999999 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525688+0000", 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4069999999999998e-06 2024-08-30T23:37:08.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525744+0000", 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6588000000000001e-05 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525761+0000", 2024-08-30T23:37:08.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6642000000000001e-05 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.527080+0000", 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013189460000000001 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.527113+0000", 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3470999999999999e-05 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.972 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62174 4.4 e337/331 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head v 337'3877, mlcod=336'3873)", 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.37229268999999, 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57728454699999998, 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:37:08.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525606+0000", 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1429999999999999e-06 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525661+0000", 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4715e-05 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101380+0000", 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57571867700000001 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101387+0000", 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.746e-06 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101441+0000", 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3916999999999999e-05 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101460+0000", 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8185999999999998e-05 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.102857+0000", 2024-08-30T23:37:08.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013971820000000001 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.102890+0000", 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2981000000000002e-05 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62186 4.4 e337/331 4:257aab52:::benchmark_data_smithi173_74914_object62185:head v 337'3878, mlcod=337'3874)", 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:37:08.977 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 131.343667724, 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55065025599999995, 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554231+0000", 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999995e-07 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554289+0000", 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8440000000000003e-05 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103923+0000", 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54963335000000002 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103927+0000", 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7199999999999997e-06 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103985+0000", 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7398999999999997e-05 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103997+0000", 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2617e-05 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.104853+0000", 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085557500000000002 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.104880+0000", 2024-08-30T23:37:08.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7444e-05 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98221 4.5 4:a4ee3b1f:::benchmark_data_smithi173_74914_object98220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 97.460707462000002, 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225938741, 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.982 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98221 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:37:08.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437192+0000", 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.598e-06 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437212+0000", 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0387999999999999e-05 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437225+0000", 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2393e-05 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437230+0000", 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5319999999999997e-06 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437273+0000", 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2778999999999998e-05 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437360+0000", 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6718000000000006e-05 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437446+0000", 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6215e-05 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.439899+0000", 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024530929999999999 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447609+0000", 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.007709624 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.662941+0000", 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21533202500000001 2024-08-30T23:37:08.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.663056+0000", 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115085 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.663129+0000", 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3291e-05 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98225 4.5 4:ad425262:::benchmark_data_smithi173_74914_object98224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 97.453388150999999, 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32404802500000002, 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98225 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:37:08.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0200000000000001e-07 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444529+0000", 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8377999999999999e-05 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444538+0000", 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6719999999999999e-06 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444543+0000", 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9119999999999997e-06 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444596+0000", 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2286e-05 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444722+0000", 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126229 2024-08-30T23:37:08.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444864+0000", 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014163000000000001 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447348+0000", 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024847910000000001 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.450736+0000", 2024-08-30T23:37:08.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033874059999999999 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768431+0000", 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31769526799999998 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768502+0000", 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0516e-05 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768558+0000", 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6234999999999999e-05 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98228 4.7 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:37:08.995 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 97.450264802999996, 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35701674100000003, 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98228 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447634+0000", 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.035e-06 2024-08-30T23:37:08.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447655+0000", 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0735000000000001e-05 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447665+0000", 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0353999999999999e-05 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:08.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447679+0000", 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3804e-05 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447729+0000", 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9570000000000001e-05 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447855+0000", 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126453 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:08.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.448003+0000", 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014797200000000001 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.450323+0000", 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002320192 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.604622+0000", 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15429874599999999 2024-08-30T23:37:09.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804484+0000", 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.199861699 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804600+0000", 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011622500000000001 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:09.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804650+0000", 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9956000000000001e-05 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:164796 4.4 e373/331 4:262b5c71:::benchmark_data_smithi173_74914_object66554:head v 373'10257, mlcod=373'10254)", 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:43.103223+0000", 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 25.794674824000001, 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13095714999999999, 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:09.002 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.103223+0000", 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.103223+0000", 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:09.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.103224+0000", 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0299999999999998e-07 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.103230+0000", 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8189999999999997e-06 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.229399+0000", 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.126169315 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.229406+0000", 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8449999999999997e-06 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.229457+0000", 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1128999999999998e-05 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.229486+0000", 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8792000000000001e-05 2024-08-30T23:37:09.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.234051+0000", 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045649059999999997 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.234180+0000", 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000129541 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:09.006 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:09.007 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:37:09.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:09.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.4.f.1 2024-08-30T23:37:09.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:09.067 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:09.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:09.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:09.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:09.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:09.398 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:09.428 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:09.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:09.557 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:09 smithi106 ceph-mon[16603]: pgmap v917: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s; 3913/31653 objects degraded (12.362%) 2024-08-30T23:37:09.643 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:09.643 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:09.644 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:09.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:09.672 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:09.672 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:09.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:09.698 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:09.698 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:09.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:09.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:09.735 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:09 smithi173 ceph-mon[20554]: pgmap v917: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s; 3913/31653 objects degraded (12.362%) 2024-08-30T23:37:09.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:09.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:09.776 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:09.776 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:09 smithi136 ceph-mon[20207]: pgmap v917: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 660 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s; 3913/31653 objects degraded (12.362%) 2024-08-30T23:37:09.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:09.801 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:09.801 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:09.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:09.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:37:09.926 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:09.927 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:09.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:09.953 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:09.954 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:09.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:09.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:09.978 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:09.978 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:10.006 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:10.007 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:10.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:10.081 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:10.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:10.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:10.279 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:10.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:37:10.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:10.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:10.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:10.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:10.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:10.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:10.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:10 smithi173 ceph-mon[20554]: 4.8 scrub starts 2024-08-30T23:37:10.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1995476223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:10.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:10 smithi173 ceph-mon[20554]: 4.8 scrub ok 2024-08-30T23:37:10.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:10 smithi173 ceph-mon[20554]: 4.0 scrub starts 2024-08-30T23:37:10.706 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:10 smithi173 ceph-mon[20554]: 4.0 scrub ok 2024-08-30T23:37:10.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:10.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:10.845 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:10.845 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:10.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:10 smithi106 ceph-mon[16603]: 4.8 scrub starts 2024-08-30T23:37:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1995476223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:10 smithi106 ceph-mon[16603]: 4.8 scrub ok 2024-08-30T23:37:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:10 smithi106 ceph-mon[16603]: 4.0 scrub starts 2024-08-30T23:37:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:10 smithi106 ceph-mon[16603]: 4.0 scrub ok 2024-08-30T23:37:10.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:10.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:10.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:11.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.009 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:11.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:10 smithi136 ceph-mon[20207]: 4.8 scrub starts 2024-08-30T23:37:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1995476223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:10 smithi136 ceph-mon[20207]: 4.8 scrub ok 2024-08-30T23:37:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:10 smithi136 ceph-mon[20207]: 4.0 scrub starts 2024-08-30T23:37:11.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:10 smithi136 ceph-mon[20207]: 4.0 scrub ok 2024-08-30T23:37:11.047 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:11.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:11.136 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:11.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:11.263 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:11.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:11.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:11.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:11.518 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:11.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:11.641 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:11.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:11.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.774 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3913/31653 objects degraded (12.362%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:11 smithi173 ceph-mon[20554]: pgmap v918: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 481 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 910 op/s; 2866/23073 objects degraded (12.421%) 2024-08-30T23:37:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3913/31653 objects degraded (12.362%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:11 smithi106 ceph-mon[16603]: pgmap v918: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 481 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 910 op/s; 2866/23073 objects degraded (12.421%) 2024-08-30T23:37:11.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:11.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:11.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:11.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:37:12.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:12.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3913/31653 objects degraded (12.362%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:11 smithi136 ceph-mon[20207]: pgmap v918: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 481 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 910 op/s; 2866/23073 objects degraded (12.421%) 2024-08-30T23:37:12.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:12.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:12.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:12.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:12.303 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:12.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:12.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:12.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:12.539 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:12.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:12.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:12.683 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:37:12.752 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1287362018' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:12.752 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:12 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:12.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:12.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:12.905 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:12.906 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:12.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:12.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:12.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:13.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:37:13.022 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1287362018' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:13.022 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:12 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1287362018' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:12 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:13.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:13.068 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:13.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.201 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:13.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:13.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:13.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:13.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:13.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:13.515 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:13.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:13.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:13.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:13.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:37:13.724 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:13.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:13 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:37:13.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:13 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:13.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:13 smithi173 ceph-mon[20554]: osdmap e379: 12 total, 11 up, 12 in 2024-08-30T23:37:13.736 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:13 smithi173 ceph-mon[20554]: pgmap v920: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 395 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 2315/18936 objects degraded (12.225%) 2024-08-30T23:37:13.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:13.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:13.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:13.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:13 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:37:13.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:13 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:13.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:13 smithi106 ceph-mon[16603]: osdmap e379: 12 total, 11 up, 12 in 2024-08-30T23:37:13.978 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:13 smithi106 ceph-mon[16603]: pgmap v920: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 395 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 2315/18936 objects degraded (12.225%) 2024-08-30T23:37:13.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:13.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:13.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:13 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:37:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:13 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:13 smithi136 ceph-mon[20207]: osdmap e379: 12 total, 11 up, 12 in 2024-08-30T23:37:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:13 smithi136 ceph-mon[20207]: pgmap v920: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 395 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s; 2315/18936 objects degraded (12.225%) 2024-08-30T23:37:14.070 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:37:14.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:37:14.075 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:37:14.075 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-30T23:37:14.075 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:37:14.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:14.119 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:14.119 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:14.120 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:14.145 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:14.150 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:14.150 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:14.175 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:14.175 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:14.203 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:14.203 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:14.240 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:14.240 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:14.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:14.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:14.268 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:14.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:14.296 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:14.297 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:14.328 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:14.328 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:14.363 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:14.364 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:14.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:14.396 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:14.396 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:14.402 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:14.404 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:14.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:14.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:14.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:14.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:14.602 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:110499 2.b 2:d409eb3a:::benchmark_data_smithi173_47195_object13491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 568.52595880900003, 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25912935999999998, 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.603 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 110499 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074809+0000", 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074810+0000", 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7499999999999998e-07 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074815+0000", 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5700000000000003e-06 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074826+0000", 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.101e-05 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074832+0000", 2024-08-30T23:37:14.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.643e-06 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074884+0000", 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2614999999999997e-05 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.074986+0000", 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101808 2024-08-30T23:37:14.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,7", 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.075067+0000", 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0661000000000006e-05 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.077720+0000", 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026533749999999999 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.078416+0000", 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069541700000000004 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333266+0000", 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25485053000000002 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333335+0000", 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8868999999999994e-05 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:27:46.333939+0000", 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000603887 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:41062 3.f e226/206 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head v 226'2525, mlcod=226'2521)", 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 391.39843649199997, 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32836641100000002, 2024-08-30T23:37:14.610 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202332+0000", 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202340+0000", 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9529999999999999e-06 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.202390+0000", 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0473999999999998e-05 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529209+0000", 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32681829099999998 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529215+0000", 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8379999999999996e-06 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529257+0000", 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2036999999999997e-05 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.529272+0000", 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4331999999999999e-05 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530655+0000", 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001382961 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:30:43.530698+0000", 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3525000000000001e-05 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:199193 3.0 3:0da882ce:::benchmark_data_smithi173_60780_object95739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 220.24287790299999, 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30993041100000002, 2024-08-30T23:37:14.615 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.616 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.616 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.616 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199193 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357890+0000", 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357892+0000", 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.189e-06 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357896+0000", 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5090000000000004e-06 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357907+0000", 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0607e-05 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357912+0000", 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2190000000000003e-06 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.357956+0000", 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3745000000000002e-05 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.358054+0000", 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7940000000000001e-05 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.358129+0000", 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5887e-05 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362642+0000", 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045129510000000003 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.364032+0000", 2024-08-30T23:37:14.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00138984 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667719+0000", 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30368694699999998 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667778+0000", 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8754e-05 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667821+0000", 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2823e-05 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:199193 3.0 e299/206 ondisk, result = 0)", 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 220.23841517400001, 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30548378700000001, 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.623 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362353+0000", 2024-08-30T23:37:14.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362354+0000", 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1060000000000001e-06 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.362361+0000", 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4230000000000002e-06 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667470+0000", 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30510947599999999 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667476+0000", 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8379999999999998e-06 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667704+0000", 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022796099999999999 2024-08-30T23:37:14.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667712+0000", 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4749999999999993e-06 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.667837+0000", 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012450800000000001 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:199216 3.0 3:0986db6c:::benchmark_data_smithi173_60780_object95762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 220.20180943599999, 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28454402699999998, 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:14.628 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199216 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398959+0000", 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398960+0000", 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0100000000000001e-06 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398972+0000", 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2381000000000001e-05 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398982+0000", 2024-08-30T23:37:14.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0236e-05 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.398988+0000", 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5940000000000001e-06 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399037+0000", 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8613999999999999e-05 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399169+0000", 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000132657 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.399297+0000", 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127732 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.403718+0000", 2024-08-30T23:37:14.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044206580000000001 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.403817+0000", 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9599999999999995e-05 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683416+0000", 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27959865299999997 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683467+0000", 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1248999999999997e-05 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683503+0000", 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5642999999999997e-05 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.634 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:199216 3.0 e299/206 ondisk, result = 0)", 2024-08-30T23:37:14.635 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:37:14.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:14.636 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 220.19660109, 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27933989799999998, 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404167+0000", 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404168+0000", 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7700000000000004e-07 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.404173+0000", 2024-08-30T23:37:14.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8230000000000002e-06 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.679627+0000", 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27545433000000002 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.679635+0000", 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9200000000000004e-06 2024-08-30T23:37:14.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683406+0000", 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037705299999999998 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683412+0000", 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5640000000000002e-06 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:34.683507+0000", 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4953999999999999e-05 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:206901 3.7 e300/206 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head v 300'13076, mlcod=300'13071)", 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 210.53662345399999, 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38146454200000002, 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.641 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064146+0000", 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8000000000000005e-07 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064151+0000", 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8270000000000003e-06 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441558+0000", 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37740696499999998 2024-08-30T23:37:14.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441573+0000", 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4623e-05 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441672+0000", 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9351999999999993e-05 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441691+0000", 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8134999999999998e-05 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445570+0000", 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003879511 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445609+0000", 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9348999999999999e-05 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87cb8fd9:::benchmark_data_smithi173_60780_object16345:head,262'7462)", 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 202.65451730699999, 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72540086000000004, 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946252+0000", 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3399999999999999e-07 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946257+0000", 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349999999999999e-06 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671354+0000", 2024-08-30T23:37:14.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72509723199999998 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671379+0000", 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5279e-05 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671555+0000", 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000175203 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671572+0000", 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.774e-05 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671652+0000", 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9536999999999994e-05 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87ea66dc:::benchmark_data_smithi173_60780_object18546:head,263'7595)", 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.71816891700001, 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85628250299999997, 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882600+0000", 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882604+0000", 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2289999999999997e-06 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738700+0000", 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85609620099999995 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738719+0000", 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8343e-05 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738882+0000", 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016310000000000001 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31056 4.8 4.b8866df8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:14.654 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 173.91471488799999, 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99756266500000002, 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31056 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686055+0000", 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6950000000000001e-06 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686078+0000", 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3113e-05 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686093+0000", 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4856e-05 2024-08-30T23:37:14.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686100+0000", 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3930000000000002e-06 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686163+0000", 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2131000000000005e-05 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686172+0000", 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9849999999999995e-06 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683581+0000", 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99740900499999996 2024-08-30T23:37:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683616+0000", 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5487000000000001e-05 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31071 4.8 4.35742af8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:14.660 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 173.305056794, 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38794466599999999, 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31071 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295712+0000", 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1299999999999998e-07 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295733+0000", 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0463e-05 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295744+0000", 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1534e-05 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295750+0000", 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0599999999999996e-06 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295804+0000", 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3983000000000001e-05 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295812+0000", 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4290000000000004e-06 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683649+0000", 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38783691999999997 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683656+0000", 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3640000000000001e-06 2024-08-30T23:37:14.665 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62151 4.1 4:8b395779:::benchmark_data_smithi173_74914_object62150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.12246148, 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38737673, 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.666 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62151 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478309+0000", 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478334+0000", 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5167000000000001e-05 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478351+0000", 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7391e-05 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478360+0000", 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2579999999999995e-06 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478413+0000", 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3780999999999999e-05 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478541+0000", 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127358 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478645+0000", 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001039 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480555+0000", 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019108790000000001 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480787+0000", 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023142799999999999 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865599+0000", 2024-08-30T23:37:14.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38481200700000001 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865636+0000", 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7416999999999998e-05 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865684+0000", 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7215000000000001e-05 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:62151 4.1 e337/306 ondisk, result = 0)", 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.12004831799999, 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38498058299999999, 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480721+0000", 2024-08-30T23:37:14.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480728+0000", 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7719999999999994e-06 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865502+0000", 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38477343200000003 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865508+0000", 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7300000000000002e-06 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865589+0000", 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.119e-05 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865595+0000", 2024-08-30T23:37:14.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4529999999999999e-06 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865701+0000", 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105266 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62151 4.1 4:8b395779:::benchmark_data_smithi173_74914_object62150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.097736572, 2024-08-30T23:37:14.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.362616665, 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for ondisk", 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62151 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503032+0000", 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2399999999999998e-07 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503054+0000", 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1647000000000001e-05 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503061+0000", 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503066+0000", 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6750000000000001e-06 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503115+0000", 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8968999999999998e-05 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for ondisk", 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.503174+0000", 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8739000000000001e-05 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865648+0000", 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36247479799999999 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.681 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62193 4.8 4:198a939b:::benchmark_data_smithi173_74914_object62192:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.604484+0000", 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 136.99628419000001, 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35078139200000003, 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62193 2024-08-30T23:37:14.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604484+0000", 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604484+0000", 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604486+0000", 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9319999999999999e-06 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604509+0000", 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2966e-05 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604524+0000", 2024-08-30T23:37:14.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4929999999999999e-05 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604531+0000", 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.277e-06 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604592+0000", 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0331000000000002e-05 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604730+0000", 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000138897 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.604887+0000", 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015684499999999999 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.606731+0000", 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018433290000000001 2024-08-30T23:37:14.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.848856+0000", 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24212561399999999 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.955132+0000", 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10627551 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.955238+0000", 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010603900000000001 2024-08-30T23:37:14.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.955265+0000", 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7722e-05 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98220 4.1 4:8d0adb9e:::benchmark_data_smithi173_74914_object98219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.433687+0000", 2024-08-30T23:37:14.688 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 103.16708141399999, 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26153179399999998, 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98220 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433687+0000", 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433687+0000", 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433689+0000", 2024-08-30T23:37:14.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1270000000000001e-06 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433711+0000", 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2305000000000001e-05 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433727+0000", 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5455e-05 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433734+0000", 2024-08-30T23:37:14.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6320000000000008e-06 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433787+0000", 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349999999999999e-05 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433893+0000", 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106539 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.433996+0000", 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102798 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.436497+0000", 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025010290000000001 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437108+0000", 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00061081500000000001 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695097+0000", 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25798935299999998 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695188+0000", 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0463000000000004e-05 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695219+0000", 2024-08-30T23:37:14.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0927999999999999e-05 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98222 4.1 4:8f28ee96:::benchmark_data_smithi173_74914_object98221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.440408+0000", 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 103.16036052, 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.368547611, 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.695 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98222 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440408+0000", 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440408+0000", 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440409+0000", 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9300000000000006e-07 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440451+0000", 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2049999999999999e-05 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440462+0000", 2024-08-30T23:37:14.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0725e-05 2024-08-30T23:37:14.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440467+0000", 2024-08-30T23:37:14.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9109999999999998e-06 2024-08-30T23:37:14.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440513+0000", 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5627000000000002e-05 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440603+0000", 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0085000000000002e-05 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.440715+0000", 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011188600000000001 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443186+0000", 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00247117 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443662+0000", 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047565599999999997 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.808739+0000", 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36507735699999999 2024-08-30T23:37:14.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.808908+0000", 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000168581 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.808955+0000", 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.757e-05 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:98220 4.1 e353/306 ondisk, result = 0)", 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.443385+0000", 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 103.157382814, 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25184049600000002, 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443385+0000", 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443385+0000", 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443386+0000", 2024-08-30T23:37:14.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0299999999999999e-07 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.443394+0000", 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1529999999999997e-06 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.694900+0000", 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.251505441 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.694917+0000", 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7257000000000001e-05 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695073+0000", 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015597699999999999 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695088+0000", 2024-08-30T23:37:14.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4806e-05 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.695226+0000", 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013825900000000001 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:177711 4.1 4:8fbfc29b:::benchmark_data_smithi173_74914_object79469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:57.483141+0000", 2024-08-30T23:37:14.707 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.117627710000001, 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.118970548, 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 177711 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483141+0000", 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483141+0000", 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483142+0000", 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.804e-06 2024-08-30T23:37:14.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483151+0000", 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3780000000000001e-06 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483172+0000", 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1120000000000001e-05 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483180+0000", 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0779999999999996e-06 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483249+0000", 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9541000000000004e-05 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483383+0000", 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013353400000000001 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.483508+0000", 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124967 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.487386+0000", 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038778929999999999 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.599532+0000", 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11214635100000001 2024-08-30T23:37:14.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.601991+0000", 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024590219999999999 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.602059+0000", 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7379000000000007e-05 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.602111+0000", 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2481000000000001e-05 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:177913 4.8 4:1af1c92b:::benchmark_data_smithi173_74914_object79671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:57.747171+0000", 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.853597418, 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.119868417, 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:37:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 177913 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747171+0000", 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747171+0000", 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747173+0000", 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0449999999999999e-06 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747181+0000", 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8080000000000005e-06 2024-08-30T23:37:14.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747200+0000", 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9703000000000001e-05 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747209+0000", 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0399999999999998e-06 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:37:14.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747291+0000", 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1166000000000002e-05 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747402+0000", 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011182 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.747509+0000", 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107036 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.752332+0000", 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048226620000000001 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.754133+0000", 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018012869999999999 2024-08-30T23:37:14.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.866926+0000", 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112792082 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.866991+0000", 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5432999999999995e-05 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:37:14.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:57.867039+0000", 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8334999999999997e-05 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:37:14.721 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:37:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.722 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:14.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:14.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:14 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:14 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:14 smithi173 ceph-mon[20554]: 4.9 scrub starts 2024-08-30T23:37:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:14 smithi173 ceph-mon[20554]: osdmap e380: 12 total, 11 up, 12 in 2024-08-30T23:37:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:14 smithi173 ceph-mon[20554]: 4.9 scrub ok 2024-08-30T23:37:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3980485507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:14.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:14.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:14.979 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:14.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:15.001 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:14 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:15.001 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:14 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:15.002 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:14 smithi106 ceph-mon[16603]: 4.9 scrub starts 2024-08-30T23:37:15.002 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:14 smithi106 ceph-mon[16603]: osdmap e380: 12 total, 11 up, 12 in 2024-08-30T23:37:15.002 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:14 smithi106 ceph-mon[16603]: 4.9 scrub ok 2024-08-30T23:37:15.002 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3980485507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:14 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:14 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:14 smithi136 ceph-mon[20207]: 4.9 scrub starts 2024-08-30T23:37:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:14 smithi136 ceph-mon[20207]: osdmap e380: 12 total, 11 up, 12 in 2024-08-30T23:37:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:14 smithi136 ceph-mon[20207]: 4.9 scrub ok 2024-08-30T23:37:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3980485507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:15.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:15.104 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:15.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:15.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:15.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:15.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:15.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:15.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:37:15.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:15.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:15.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:37:15.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:15.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:15.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:15.739 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:15 smithi106 ceph-mon[16603]: pgmap v922: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 287 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 1464/13746 objects degraded (10.650%) 2024-08-30T23:37:15.739 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2315/18936 objects degraded (12.225%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:15.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:15 smithi173 ceph-mon[20554]: pgmap v922: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 287 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 1464/13746 objects degraded (10.650%) 2024-08-30T23:37:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2315/18936 objects degraded (12.225%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:15.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:15.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:15.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:15.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:15.980 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:15.982 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:16.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:15 smithi136 ceph-mon[20207]: pgmap v922: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 287 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 892 op/s; 1464/13746 objects degraded (10.650%) 2024-08-30T23:37:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2315/18936 objects degraded (12.225%), 6 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:16.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:16.139 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:16.154 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:37:16.154 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 380, 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:37:13.681101+0000", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:36:27.655854+0000", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:30:29.115133+0000", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5799936, 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:37:16.155 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 26, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 4, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "jewel", 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:37:16.156 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:37:16.157 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 1, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 1, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 1, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 1, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 1, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:37:16.158 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "343", 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:37:16.159 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:37:16.160 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:37:16.161 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.162 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 10, 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 10, 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 0.30000001192092896, 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3, 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3, 2024-08-30T23:37:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.77478784322738647, 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.93000000715255737, 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 4, 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_2", 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:33:55.332254+0000", 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:37:16.164 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:37:16.165 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "336", 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:37:16.166 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:37:16.167 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:37:16.168 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:37:16.169 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.8700000047683716, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.8700000047683716, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.059999942779541, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.059999942779541, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.77478784322738647, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.93000000715255737, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:37:16.170 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.923797607421875, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 306, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.171 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.172 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.173 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.173 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:37:16.173 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:37:16.175 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:37:16.176 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.5043182373046875, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.177 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 13, 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 332, 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.178 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.179 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/1861847252", 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/1861847252", 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/1861847252", 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/1861847252", 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.182 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 97, 2024-08-30T23:37:16.183 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:37:16.184 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.185 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.188 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.8072052001953125, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 369, 2024-08-30T23:37:16.189 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.194 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.527435302734375, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 369, 2024-08-30T23:37:16.195 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.197 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:37:16.198 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:37:16.199 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 39, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 206, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.204 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.205 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2282248833", 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2282248833", 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2282248833", 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2282248833", 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.206 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7101593017578125, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 327, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:37:16.208 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.209 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:37:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.212 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.213 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 369, 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.218 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:37:16.219 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 206, 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.220 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:37:16.221 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:37:16.222 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:37:16.223 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 369, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.225 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:37:16.229 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.230 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 369, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.231 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.232 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.235 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.236 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.589813232421875, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 329, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:37:16.237 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.238 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.240 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.241 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.242 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:44.272903+0000", 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.243 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:35:29.938964+0000", 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:35:58.418787+0000", 2024-08-30T23:37:16.244 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:16.714089+0000", 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.245 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:40.042870+0000", 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.246 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:07.357076+0000", 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:25.979646+0000", 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:50.752004+0000", 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:37:16.248 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:13.499413+0000", 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.249 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:10.984667+0000", 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:44.082758+0000", 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:36:18.770479+0000", 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [], 2024-08-30T23:37:16.251 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:37:16.252 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:37:16.253 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:16.254 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:37:16.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:16.286 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:16.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:16.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:16.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:16.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:16.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:16.663 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:16.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:16.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:16.839 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3022770077' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:37:16.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:16.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:16.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:17.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3022770077' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:37:17.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3022770077' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:37:17.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:17.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:17.191 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:195631 4.b e380/369 4:de595efd:::benchmark_data_smithi173_74914_object97389:head v 380'12361, mlcod=380'12359)", 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:17.145180+0000", 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.010607771, 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010626391000000001, 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145180+0000", 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145180+0000", 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145181+0000", 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6800000000000009e-07 2024-08-30T23:37:17.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145188+0000", 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4699999999999999e-06 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145217+0000", 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9311000000000001e-05 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145225+0000", 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145283+0000", 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8174999999999998e-05 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.145308+0000", 2024-08-30T23:37:17.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4913e-05 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:195632 4.3 e380/369 4:c3d462d6:::benchmark_data_smithi173_74914_object97390:head v 380'12049, mlcod=380'12047)", 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:17.147449+0000", 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0083388790000000004, 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0083893040000000002, 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:17.196 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147449+0000", 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147449+0000", 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:17.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147450+0000", 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2999999999999999e-07 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147457+0000", 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1049999999999997e-06 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147479+0000", 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1472999999999999e-05 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147486+0000", 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8569999999999997e-06 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147551+0000", 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5446999999999999e-05 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.147574+0000", 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2912e-05 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.199 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:195635 4.b e380/369 4:dadb16fd:::benchmark_data_smithi173_74914_object97393:head v 380'12362, mlcod=380'12359)", 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:17.150148+0000", 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0056405120000000003, 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0057302309999999997, 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150148+0000", 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150148+0000", 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150149+0000", 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1039999999999999e-06 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150156+0000", 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3069999999999997e-06 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150187+0000", 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1069999999999999e-05 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150194+0000", 2024-08-30T23:37:17.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1300000000000003e-06 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150935+0000", 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00074098700000000003 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.150956+0000", 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0738999999999999e-05 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:195639 4.3 e380/369 4:c42ab591:::benchmark_data_smithi173_74914_object97397:head v 380'12050, mlcod=380'12047)", 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:17.152480+0000", 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0033080039999999998, 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034239330000000001, 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152480+0000", 2024-08-30T23:37:17.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152480+0000", 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152481+0000", 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8699999999999995e-07 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152486+0000", 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6849999999999998e-06 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152506+0000", 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9581000000000001e-05 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152513+0000", 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9920000000000001e-06 2024-08-30T23:37:17.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152854+0000", 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034117700000000002 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.152884+0000", 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9495000000000001e-05 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:195638 4.b e380/369 4:df608921:::benchmark_data_smithi173_74914_object97396:head v 380'12363, mlcod=380'12359)", 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:17.154874+0000", 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.000914262, 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010563370000000001, 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154874+0000", 2024-08-30T23:37:17.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154874+0000", 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154875+0000", 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7700000000000004e-07 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154881+0000", 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6209999999999998e-06 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154910+0000", 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8861999999999999e-05 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154915+0000", 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8779999999999997e-06 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154948+0000", 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999998e-05 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:17.154961+0000", 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3023999999999999e-05 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:37:17.211 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 5 2024-08-30T23:37:17.212 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:37:17.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:17.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:17.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:17.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:17.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:17.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:17.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:37:17.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:17.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:17.709 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:37:17.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:17.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:17.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/632605781' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:37:17.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:17 smithi106 ceph-mon[16603]: pgmap v923: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 173 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 973 op/s; 1166/8274 objects degraded (14.092%) 2024-08-30T23:37:17.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:17.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:17.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/632605781' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:37:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:17 smithi136 ceph-mon[20207]: pgmap v923: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 173 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 973 op/s; 1166/8274 objects degraded (14.092%) 2024-08-30T23:37:18.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/632605781' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:37:18.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:17 smithi173 ceph-mon[20554]: pgmap v923: 17 pgs: 6 active+undersized+degraded, 11 active+clean; 173 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 973 op/s; 1166/8274 objects degraded (14.092%) 2024-08-30T23:37:18.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:18.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:18.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:18.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:18.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:18.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:18.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:18.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:18.569 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 98239 objects 2024-08-30T23:37:18.569 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :106.095 2024-08-30T23:37:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.586 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:18.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:18.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:18.710 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:18.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:18.840 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:18.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:18 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:18.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/632605781' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:37:18.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:18 smithi106 ceph-mon[16603]: osdmap e381: 12 total, 11 up, 12 in 2024-08-30T23:37:18.934 INFO:tasks.thrashosds.thrasher:j is {'epoch': 380, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:37:13.681101+0000', 'last_up_change': '2024-08-30T23:36:27.655854+0000', 'last_in_change': '2024-08-30T23:30:29.115133+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '343', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.7747878432273865, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-08-30T23:33:55.332254+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '336', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8700000047683716, 'score_stable': 1.8700000047683716, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.7747878432273865, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 0.923797607421875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 0, 'in': 1, 'weight': 0.5043182373046875, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 332, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 1861847252}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1861847252}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 1861847252}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1861847252}]}, 'public_addr': '172.21.15.106:6811/1861847252', 'cluster_addr': '172.21.15.106:6813/1861847252', 'heartbeat_back_addr': '172.21.15.106:6817/1861847252', 'heartbeat_front_addr': '172.21.15.106:6815/1861847252', 'state': ['exists']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 0.8072052001953125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 0.527435302734375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2282248833}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2282248833}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2282248833}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2282248833}]}, 'public_addr': '172.21.15.136:6809/2282248833', 'cluster_addr': '172.21.15.136:6811/2282248833', 'heartbeat_back_addr': '172.21.15.136:6815/2282248833', 'heartbeat_front_addr': '172.21.15.136:6813/2282248833', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 0.7101593017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 327, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 0.589813232421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 329, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:44.272903+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:35:29.938964+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:35:58.418787+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:16.714089+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:40.042870+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:07.357076+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:25.979646+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:50.752004+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:13.499413+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:10.984667+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:44.082758+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:36:18.770479+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:37:18.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:37:18.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:18.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:18.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:18 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/632605781' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:37:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:18 smithi136 ceph-mon[20207]: osdmap e381: 12 total, 11 up, 12 in 2024-08-30T23:37:19.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:19.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:18 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:19.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/632605781' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:37:19.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:18 smithi173 ceph-mon[20554]: osdmap e381: 12 total, 11 up, 12 in 2024-08-30T23:37:19.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:19.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:19.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:19.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:37:19.259 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:19.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:19.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:19.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:19.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.472 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:19.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:19.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:19.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:19.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:19.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:19.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:19.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:19.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:20.001 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:19 smithi106 ceph-mon[16603]: pgmap v925: 17 pgs: 1 active+undersized, 5 active+undersized+degraded, 11 active+clean; 104 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 629/4989 objects degraded (12.608%) 2024-08-30T23:37:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:19 smithi136 ceph-mon[20207]: pgmap v925: 17 pgs: 1 active+undersized, 5 active+undersized+degraded, 11 active+clean; 104 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 629/4989 objects degraded (12.608%) 2024-08-30T23:37:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:19 smithi173 ceph-mon[20554]: pgmap v925: 17 pgs: 1 active+undersized, 5 active+undersized+degraded, 11 active+clean; 104 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 761 op/s; 629/4989 objects degraded (12.608%) 2024-08-30T23:37:20.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:20.104 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:20.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:20.224 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:20.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:20.269 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:20.269 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:20.269 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:20.291 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:20.292 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:20.311 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:20.311 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:20.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:20.334 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:20.334 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:20.355 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:20.355 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:20.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:20.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:20.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:20.399 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:20.400 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:20.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:20.420 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:20.442 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:20.442 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:20.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:20.463 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:20.464 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:20.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:20.506 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:20.506 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:20.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:20.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:20.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:20.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:20.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:20.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2568275848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:20 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 629/4989 objects degraded (12.608%), 5 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:20.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:20.948 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:37:20.949 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:37:20.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:20.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:20.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2568275848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:20 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 629/4989 objects degraded (12.608%), 5 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2568275848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:20 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 629/4989 objects degraded (12.608%), 5 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:21.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:21.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:21.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:21.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:21.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:21.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:21.472 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:21.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:21.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:21.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:21.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:21.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:21.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:21.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:37:21.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:21.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:21.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:21.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:22.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:21 smithi136 ceph-mon[20207]: osdmap e382: 12 total, 11 up, 12 in 2024-08-30T23:37:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:21 smithi136 ceph-mon[20207]: from='client.16551 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:37:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:21 smithi136 ceph-mon[20207]: pgmap v927: 17 pgs: 3 active+undersized, 3 active+undersized+degraded, 11 active+clean; 7.8 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 26/354 objects degraded (7.345%) 2024-08-30T23:37:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:21 smithi173 ceph-mon[20554]: osdmap e382: 12 total, 11 up, 12 in 2024-08-30T23:37:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:21 smithi173 ceph-mon[20554]: from='client.16551 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:37:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:21 smithi173 ceph-mon[20554]: pgmap v927: 17 pgs: 3 active+undersized, 3 active+undersized+degraded, 11 active+clean; 7.8 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 26/354 objects degraded (7.345%) 2024-08-30T23:37:22.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:21 smithi106 ceph-mon[16603]: osdmap e382: 12 total, 11 up, 12 in 2024-08-30T23:37:22.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:21 smithi106 ceph-mon[16603]: from='client.16551 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:37:22.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:21 smithi106 ceph-mon[16603]: pgmap v927: 17 pgs: 3 active+undersized, 3 active+undersized+degraded, 11 active+clean; 7.8 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s; 26/354 objects degraded (7.345%) 2024-08-30T23:37:22.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:22.139 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":926,"stamp":"2024-08-30T23:37:20.716433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38535712,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":77,"num_read_kb":65,"num_write":196024,"num_write_kb":6289447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15532,"num_bytes_recovered":1019098720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160893,"ondisk_log_size":160893,"up":45,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4549852,"kb_used_data":1356044,"kb_used_omap":18,"kb_used_meta":3193581,"kb_avail":1120391972,"statfs":{"total":1151940427776,"available":1147281379328,"internally_reserved":0,"allocated":1388589056,"data_stored":1385270922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":3270227257},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,8,8,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-375521304,"num_objects":-5731,"num_object_clones":0,"num_object_copies":-17193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.226509"},"pg_stats":[{"pgid":"4.f","version":"381'12158","reported_seq":12428,"reported_epoch":381,"state":"active+undersized","last_fresh":"2024-08-30T23:37:18.035835+0000","last_change":"2024-08-30T23:37:18.037182+0000","last_active":"2024-08-30T23:37:18.035835+0000","last_peered":"2024-08-30T23:37:18.035835+0000","last_clean":"2024-08-30T23:36:27.644419+0000","last_became_active":"2024-08-30T23:36:28.672049+0000","last_became_peered":"2024-08-30T23:36:28.672049+0000","last_unstale":"2024-08-30T23:37:18.035835+0000","last_undegraded":"2024-08-30T23:36:28.670589+0000","last_fullsized":"2024-08-30T23:36:28.670425+0000","mapping_epoch":369,"log_start":"320'2100","ondisk_log_start":"320'2100","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9004","last_scrub_stamp":"2024-08-30T23:36:21.442931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:21.442931+0000","objects_scrubbed":3193,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:37:44.401812+0000","scrub_duration":1035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12158,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9],"acting":[3,9],"avail_no_missing":["3","9"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"381'12350","reported_seq":16469,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-30T23:37:18.010972+0000","last_change":"2024-08-30T23:37:07.217490+0000","last_active":"2024-08-30T23:37:18.010972+0000","last_peered":"2024-08-30T23:37:18.010972+0000","last_clean":"2024-08-30T23:37:18.010972+0000","last_became_active":"2024-08-30T23:34:40.719708+0000","last_became_peered":"2024-08-30T23:34:40.719708+0000","last_unstale":"2024-08-30T23:37:18.010972+0000","last_undegraded":"2024-08-30T23:37:18.010972+0000","last_fullsized":"2024-08-30T23:37:18.010972+0000","mapping_epoch":327,"log_start":"320'2300","ondisk_log_start":"320'2300","created":306,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'11733","last_scrub_stamp":"2024-08-30T23:37:07.217265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:37:07.217265+0000","objects_scrubbed":621,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:24.741074+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":395200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":125632512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"381'12146","reported_seq":12376,"reported_epoch":381,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:37:18.190218+0000","last_change":"2024-08-30T23:36:28.671862+0000","last_active":"2024-08-30T23:37:18.190218+0000","last_peered":"2024-08-30T23:37:18.190218+0000","last_clean":"2024-08-30T23:36:27.645353+0000","last_became_active":"2024-08-30T23:36:28.671862+0000","last_became_peered":"2024-08-30T23:36:28.671862+0000","last_unstale":"2024-08-30T23:37:18.190218+0000","last_undegraded":"2024-08-30T23:36:28.669487+0000","last_fullsized":"2024-08-30T23:36:28.669314+0000","mapping_epoch":369,"log_start":"320'2100","ondisk_log_start":"320'2100","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'5413","last_scrub_stamp":"2024-08-30T23:35:20.802921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:35:20.802921+0000","objects_scrubbed":5389,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:37:20.815181+0000","scrub_duration":1200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12146,"num_write_kb":388609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11],"acting":[9,11],"avail_no_missing":["9","11"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.c","version":"381'11966","reported_seq":12215,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-30T23:37:18.013664+0000","last_change":"2024-08-30T23:36:47.471892+0000","last_active":"2024-08-30T23:37:18.013664+0000","last_peered":"2024-08-30T23:37:18.013664+0000","last_clean":"2024-08-30T23:37:18.013664+0000","last_became_active":"2024-08-30T23:33:56.657074+0000","last_became_peered":"2024-08-30T23:33:56.657074+0000","last_unstale":"2024-08-30T23:37:18.013664+0000","last_undegraded":"2024-08-30T23:37:18.013664+0000","last_fullsized":"2024-08-30T23:37:18.013664+0000","mapping_epoch":306,"log_start":"319'1900","ondisk_log_start":"319'1900","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'10227","last_scrub_stamp":"2024-08-30T23:36:47.471695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:47.471695+0000","objects_scrubbed":1755,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:37:52.423746+0000","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11966,"num_write_kb":382912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"381'12412","reported_seq":12645,"reported_epoch":381,"state":"active+undersized","last_fresh":"2024-08-30T23:37:18.031764+0000","last_change":"2024-08-30T23:37:18.032219+0000","last_active":"2024-08-30T23:37:18.031764+0000","last_peered":"2024-08-30T23:37:18.031764+0000","last_clean":"2024-08-30T23:36:27.651623+0000","last_became_active":"2024-08-30T23:36:28.673139+0000","last_became_peered":"2024-08-30T23:36:28.673139+0000","last_unstale":"2024-08-30T23:37:18.031764+0000","last_undegraded":"2024-08-30T23:36:28.671436+0000","last_fullsized":"2024-08-30T23:36:28.671298+0000","mapping_epoch":369,"log_start":"320'2400","ondisk_log_start":"320'2400","created":306,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'5286","last_scrub_stamp":"2024-08-30T23:35:17.042928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:35:17.042928+0000","objects_scrubbed":5250,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12412,"num_write_kb":397184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"380'12048","reported_seq":16147,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:37:16.419564+0000","last_change":"2024-08-30T23:36:28.673761+0000","last_active":"2024-08-30T23:37:16.419564+0000","last_peered":"2024-08-30T23:37:16.419564+0000","last_clean":"2024-08-30T23:36:27.617005+0000","last_became_active":"2024-08-30T23:36:28.673761+0000","last_became_peered":"2024-08-30T23:36:28.673761+0000","last_unstale":"2024-08-30T23:37:16.419564+0000","last_undegraded":"2024-08-30T23:36:28.672645+0000","last_fullsized":"2024-08-30T23:36:28.672469+0000","mapping_epoch":369,"log_start":"319'2000","ondisk_log_start":"319'2000","created":306,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"362'8609","last_scrub_stamp":"2024-08-30T23:36:14.315285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:14.315285+0000","objects_scrubbed":3545,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":388224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":125435904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"7,10","objects":84}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.9","version":"380'12232","reported_seq":16391,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:17.003590+0000","last_change":"2024-08-30T23:37:13.771431+0000","last_active":"2024-08-30T23:37:17.003590+0000","last_peered":"2024-08-30T23:37:17.003590+0000","last_clean":"2024-08-30T23:37:17.003590+0000","last_became_active":"2024-08-30T23:34:40.834281+0000","last_became_peered":"2024-08-30T23:34:40.834281+0000","last_unstale":"2024-08-30T23:37:17.003590+0000","last_undegraded":"2024-08-30T23:37:17.003590+0000","last_fullsized":"2024-08-30T23:37:17.003590+0000","mapping_epoch":327,"log_start":"320'2200","ondisk_log_start":"320'2200","created":306,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'12036","last_scrub_stamp":"2024-08-30T23:37:13.771245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:37:13.771245+0000","objects_scrubbed":274,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:39.958543+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4325376,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12232,"num_write_kb":393536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1939,"num_bytes_recovered":127074304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"380'12353","reported_seq":16577,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:16.430950+0000","last_change":"2024-08-30T23:37:09.767321+0000","last_active":"2024-08-30T23:37:16.430950+0000","last_peered":"2024-08-30T23:37:16.430950+0000","last_clean":"2024-08-30T23:37:16.430950+0000","last_became_active":"2024-08-30T23:34:42.641060+0000","last_became_peered":"2024-08-30T23:34:42.641060+0000","last_unstale":"2024-08-30T23:37:16.430950+0000","last_undegraded":"2024-08-30T23:37:16.430950+0000","last_fullsized":"2024-08-30T23:37:16.430950+0000","mapping_epoch":329,"log_start":"320'2300","ondisk_log_start":"320'2300","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'11987","last_scrub_stamp":"2024-08-30T23:37:09.767024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:37:09.767024+0000","objects_scrubbed":465,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:39.268021+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12353,"num_write_kb":398336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1986,"num_bytes_recovered":130154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"380'12017","reported_seq":16069,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:16.386046+0000","last_change":"2024-08-30T23:36:41.505755+0000","last_active":"2024-08-30T23:37:16.386046+0000","last_peered":"2024-08-30T23:37:16.386046+0000","last_clean":"2024-08-30T23:37:16.386046+0000","last_became_active":"2024-08-30T23:34:42.642239+0000","last_became_peered":"2024-08-30T23:34:42.642239+0000","last_unstale":"2024-08-30T23:37:16.386046+0000","last_undegraded":"2024-08-30T23:37:16.386046+0000","last_fullsized":"2024-08-30T23:37:16.386046+0000","mapping_epoch":329,"log_start":"319'2000","ondisk_log_start":"319'2000","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'10057","last_scrub_stamp":"2024-08-30T23:36:41.505585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:41.505585+0000","objects_scrubbed":2065,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:08.609798+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":387584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1895,"num_bytes_recovered":124190720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"381'12314","reported_seq":12567,"reported_epoch":381,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:37:18.015934+0000","last_change":"2024-08-30T23:36:28.671224+0000","last_active":"2024-08-30T23:37:18.015934+0000","last_peered":"2024-08-30T23:37:18.015934+0000","last_clean":"2024-08-30T23:36:27.661270+0000","last_became_active":"2024-08-30T23:36:28.671224+0000","last_became_peered":"2024-08-30T23:36:28.671224+0000","last_unstale":"2024-08-30T23:37:18.015934+0000","last_undegraded":"2024-08-30T23:36:28.669417+0000","last_fullsized":"2024-08-30T23:36:28.669259+0000","mapping_epoch":369,"log_start":"320'2300","ondisk_log_start":"320'2300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9172","last_scrub_stamp":"2024-08-30T23:36:21.341368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:21.341368+0000","objects_scrubbed":3156,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:37:33.816987+0000","scrub_duration":623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":394048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"381'12200","reported_seq":16370,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-30T23:37:18.021297+0000","last_change":"2024-08-30T23:36:44.727705+0000","last_active":"2024-08-30T23:37:18.021297+0000","last_peered":"2024-08-30T23:37:18.021297+0000","last_clean":"2024-08-30T23:37:18.021297+0000","last_became_active":"2024-08-30T23:34:45.636628+0000","last_became_peered":"2024-08-30T23:34:45.636628+0000","last_unstale":"2024-08-30T23:37:18.021297+0000","last_undegraded":"2024-08-30T23:37:18.021297+0000","last_fullsized":"2024-08-30T23:37:18.021297+0000","mapping_epoch":331,"log_start":"319'2100","ondisk_log_start":"319'2100","created":306,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'10319","last_scrub_stamp":"2024-08-30T23:36:44.727506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:44.727506+0000","objects_scrubbed":1890,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:06.187218+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12200,"num_write_kb":390400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1953,"num_bytes_recovered":127991808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,10],"acting":[3,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"380'12076","reported_seq":12304,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:37:17.606985+0000","last_change":"2024-08-30T23:36:28.673327+0000","last_active":"2024-08-30T23:37:17.606985+0000","last_peered":"2024-08-30T23:37:17.606985+0000","last_clean":"2024-08-30T23:36:27.663333+0000","last_became_active":"2024-08-30T23:36:28.673327+0000","last_became_peered":"2024-08-30T23:36:28.673327+0000","last_unstale":"2024-08-30T23:37:17.606985+0000","last_undegraded":"2024-08-30T23:36:28.671250+0000","last_fullsized":"2024-08-30T23:36:28.671073+0000","mapping_epoch":369,"log_start":"319'2000","ondisk_log_start":"319'2000","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'5414","last_scrub_stamp":"2024-08-30T23:35:21.114328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:35:21.114328+0000","objects_scrubbed":5371,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:37:20.940197+0000","scrub_duration":1301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12076,"num_write_kb":387264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":26}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"381'12378","reported_seq":12628,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-08-30T23:37:18.008036+0000","last_change":"2024-08-30T23:36:49.556875+0000","last_active":"2024-08-30T23:37:18.008036+0000","last_peered":"2024-08-30T23:37:18.008036+0000","last_clean":"2024-08-30T23:37:18.008036+0000","last_became_active":"2024-08-30T23:33:57.274974+0000","last_became_peered":"2024-08-30T23:33:57.274974+0000","last_unstale":"2024-08-30T23:37:18.008036+0000","last_undegraded":"2024-08-30T23:37:18.008036+0000","last_fullsized":"2024-08-30T23:37:18.008036+0000","mapping_epoch":306,"log_start":"320'2300","ondisk_log_start":"320'2300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'10738","last_scrub_stamp":"2024-08-30T23:36:49.556615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:49.556615+0000","objects_scrubbed":1655,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:10.000611+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12378,"num_write_kb":396096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,8],"acting":[9,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"380'12368","reported_seq":12616,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:16.425238+0000","last_change":"2024-08-30T23:36:49.293364+0000","last_active":"2024-08-30T23:37:16.425238+0000","last_peered":"2024-08-30T23:37:16.425238+0000","last_clean":"2024-08-30T23:37:16.425238+0000","last_became_active":"2024-08-30T23:33:57.274888+0000","last_became_peered":"2024-08-30T23:33:57.274888+0000","last_unstale":"2024-08-30T23:37:16.425238+0000","last_undegraded":"2024-08-30T23:37:16.425238+0000","last_fullsized":"2024-08-30T23:37:16.425238+0000","mapping_epoch":306,"log_start":"320'2300","ondisk_log_start":"320'2300","created":306,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"375'10790","last_scrub_stamp":"2024-08-30T23:36:49.293076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:49.293076+0000","objects_scrubbed":1689,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:12.854206+0000","scrub_duration":761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12368,"num_write_kb":398720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"380'12534","reported_seq":16669,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:17.596165+0000","last_change":"2024-08-30T23:37:10.013353+0000","last_active":"2024-08-30T23:37:17.596165+0000","last_peered":"2024-08-30T23:37:17.596165+0000","last_clean":"2024-08-30T23:37:17.596165+0000","last_became_active":"2024-08-30T23:34:42.642044+0000","last_became_peered":"2024-08-30T23:34:42.642044+0000","last_unstale":"2024-08-30T23:37:17.596165+0000","last_undegraded":"2024-08-30T23:37:17.596165+0000","last_fullsized":"2024-08-30T23:37:17.596165+0000","mapping_epoch":329,"log_start":"321'2500","ondisk_log_start":"321'2500","created":306,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'12073","last_scrub_stamp":"2024-08-30T23:37:10.013226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:37:10.013226+0000","objects_scrubbed":486,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:27.912447+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12534,"num_write_kb":401856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1944,"num_bytes_recovered":127401984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"380'12349","reported_seq":16565,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:16.424549+0000","last_change":"2024-08-30T23:36:44.619459+0000","last_active":"2024-08-30T23:37:16.424549+0000","last_peered":"2024-08-30T23:37:16.424549+0000","last_clean":"2024-08-30T23:37:16.424549+0000","last_became_active":"2024-08-30T23:34:58.152061+0000","last_became_peered":"2024-08-30T23:34:58.152061+0000","last_unstale":"2024-08-30T23:37:16.424549+0000","last_undegraded":"2024-08-30T23:37:16.424549+0000","last_fullsized":"2024-08-30T23:37:16.424549+0000","mapping_epoch":337,"log_start":"320'2300","ondisk_log_start":"320'2300","created":306,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'10537","last_scrub_stamp":"2024-08-30T23:36:44.619351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:33:55.643119+0000","last_clean_scrub_stamp":"2024-08-30T23:36:44.619351+0000","objects_scrubbed":1921,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:07.718516+0000","scrub_duration":468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12349,"num_write_kb":398272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1978,"num_bytes_recovered":129630208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0],"acting":[10,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":739,"reported_epoch":380,"state":"active+clean","last_fresh":"2024-08-30T23:37:13.692721+0000","last_change":"2024-08-30T23:36:40.921610+0000","last_active":"2024-08-30T23:37:13.692721+0000","last_peered":"2024-08-30T23:37:13.692721+0000","last_clean":"2024-08-30T23:37:13.692721+0000","last_became_active":"2024-08-30T23:34:35.630884+0000","last_became_peered":"2024-08-30T23:34:35.630884+0000","last_unstale":"2024-08-30T23:37:13.692721+0000","last_undegraded":"2024-08-30T23:37:13.692721+0000","last_fullsized":"2024-08-30T23:37:13.692721+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:36:40.921559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:36:40.921559+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:37:59.750895+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195901,"num_write_kb":6287297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15526,"num_bytes_recovered":1017511936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308041216,"data_stored":1308033072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160801,"ondisk_log_size":160801,"up":42,"acting":42,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187609,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376868,"kb_used_data":30548,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93368284,"statfs":{"total":95995035648,"available":95609122816,"internally_reserved":0,"allocated":31281152,"data_stored":30998825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":354613700},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449248,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":191196,"kb_used_data":43400,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93553956,"statfs":{"total":95995035648,"available":95799250944,"internally_reserved":0,"allocated":44441600,"data_stored":44166917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":151321024},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710884,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338728,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406424,"statfs":{"total":95995035648,"available":95648178176,"internally_reserved":0,"allocated":741376,"data_stored":462145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":346094016},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182364,"kb_used_data":20680,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93562788,"statfs":{"total":95995035648,"available":95808294912,"internally_reserved":0,"allocated":21176320,"data_stored":20901637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234156,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":364832,"kb_used_data":26832,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93380320,"statfs":{"total":95995035648,"available":95621447680,"internally_reserved":0,"allocated":27475968,"data_stored":27197733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528498,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219040,"kb_used_data":21264,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93526112,"statfs":{"total":95995035648,"available":95770738688,"internally_reserved":0,"allocated":21774336,"data_stored":21496101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":202504646},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724726,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180452,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564700,"statfs":{"total":95995035648,"available":95810252800,"internally_reserved":0,"allocated":741376,"data_stored":462145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953660,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292900,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93452252,"statfs":{"total":95995035648,"available":95695106048,"internally_reserved":0,"allocated":741376,"data_stored":462145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149889,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334568,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410584,"statfs":{"total":95995035648,"available":95652438016,"internally_reserved":0,"allocated":741376,"data_stored":462145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169432,"kb_used_data":3592,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93575720,"statfs":{"total":95995035648,"available":95821537280,"internally_reserved":0,"allocated":3678208,"data_stored":3403525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526392,"kb_used_data":1198120,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92218760,"statfs":{"total":95995035648,"available":94432010240,"internally_reserved":0,"allocated":1226874880,"data_stored":1226611199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738576,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":373080,"kb_used_data":8712,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93372072,"statfs":{"total":95995035648,"available":95613001728,"internally_reserved":0,"allocated":8921088,"data_stored":8646405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1218842624,"data_stored":1218838552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12124160,"data_stored":12124160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9175040,"data_stored":9175040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":18350080,"data_stored":18350080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":31522816,"data_stored":31522816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18026496,"data_stored":18022424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:37:22.141 INFO:tasks.thrashosds.thrasher:Setting 4.5 to [2, 4, 9, 7, 8, 6] 2024-08-30T23:37:22.141 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.5', '2', '4', '9', '7', '8', '6'] 2024-08-30T23:37:22.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 4.5 2 4 9 7 8 6 2024-08-30T23:37:22.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.144 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:22.176 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:22.181 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:22.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:22.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:22.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:22.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.396 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:22.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:22.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:22.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:22.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.652 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:22.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:22.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:22.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:22.803 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-08-30T23:37:22.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-08-30T23:37:22.863 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:22.863 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:22.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:22.903 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:22.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:23.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:23.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:23.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:23.152 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:23.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:23.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:23.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:23.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:23.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:37:23.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:23.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:23.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:23.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:23.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:37:23.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:23.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:23.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:23.910 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:23.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:24.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:24.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:24.072 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:24.075 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:24.075 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:24.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:24.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:24.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:24.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: osdmap e383: 12 total, 11 up, 12 in 2024-08-30T23:37:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: pgmap v929: 17 pgs: 4 active+undersized, 2 active+undersized+degraded, 11 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 459 op/s 2024-08-30T23:37:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:24.313 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:24.313 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:24.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:23 smithi106 ceph-mon[16603]: osdmap e383: 12 total, 11 up, 12 in 2024-08-30T23:37:24.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:23 smithi106 ceph-mon[16603]: pgmap v929: 17 pgs: 4 active+undersized, 2 active+undersized+degraded, 11 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 459 op/s 2024-08-30T23:37:24.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:24.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: osdmap e383: 12 total, 11 up, 12 in 2024-08-30T23:37:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: pgmap v929: 17 pgs: 4 active+undersized, 2 active+undersized+degraded, 11 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 459 op/s 2024-08-30T23:37:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:24.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:24.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:24.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:24.534 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:24.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:24.658 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:24.734 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:24.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:24.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:24.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:24.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:24.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:25.003 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:37:25.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:25.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:25.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:25.137 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:24 smithi106 ceph-mon[16603]: osdmap e384: 12 total, 11 up, 12 in 2024-08-30T23:37:25.137 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3009853971' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:25.137 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/560299293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:25.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:25.161 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:25.162 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:25.162 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:25.162 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:25.162 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:25.162 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:25.162 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:25.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:25.180 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:25.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:25.201 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:25.201 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:25.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:25.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: osdmap e384: 12 total, 11 up, 12 in 2024-08-30T23:37:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3009853971' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/560299293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:25.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: osdmap e384: 12 total, 11 up, 12 in 2024-08-30T23:37:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3009853971' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/560299293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:25.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:25.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:25.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:25.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:25.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:25.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:25.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:25.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:25.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:25.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:25.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:26.008 INFO:teuthology.orchestra.run.smithi106.stderr:set 4.5 pg_upmap_items mapping to [2->4,9->7,8->6] 2024-08-30T23:37:26.008 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_2' removed 2024-08-30T23:37:26.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:26.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:26.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:26.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:26.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:26.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3009853971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: osdmap e385: 12 total, 11 up, 12 in 2024-08-30T23:37:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1060591149' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [2, 4, 9, 7, 8, 6]} : dispatch 2024-08-30T23:37:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: pgmap v932: 17 pgs: 6 active+undersized, 11 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s 2024-08-30T23:37:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2279080583' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:37:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:26.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3009853971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: osdmap e385: 12 total, 11 up, 12 in 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1060591149' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [2, 4, 9, 7, 8, 6]} : dispatch 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: pgmap v932: 17 pgs: 6 active+undersized, 11 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2279080583' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:37:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3009853971' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: osdmap e385: 12 total, 11 up, 12 in 2024-08-30T23:37:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1060591149' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [2, 4, 9, 7, 8, 6]} : dispatch 2024-08-30T23:37:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: pgmap v932: 17 pgs: 6 active+undersized, 11 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s 2024-08-30T23:37:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2279080583' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:37:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:26.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:26.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:26.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:26.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:26.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:26.663 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:26.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:26.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:26.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:26.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:26.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:26.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:27.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:27.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:27.083 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:37:27.083 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:37:27.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:37:27.083 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:37:27.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(2.4 146/143 [PushReplyOp(2:2b2a4365:::benchmark_data_smithi173_47195_object14072:head)])", 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 598.58391283000003, 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69312322000000004, 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:37:27.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497766+0000", 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000003e-07 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.085 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:28.497771+0000", 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8180000000000003e-06 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183665+0000", 2024-08-30T23:37:27.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.68589385199999997 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.183680+0000", 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5021000000000001e-05 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.186942+0000", 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00326166 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:29.190889+0000", 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039470779999999997 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109556 2.5 2:a6b86250:::benchmark_data_smithi173_47195_object12548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 582.22732984499999, 2024-08-30T23:37:27.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.220264298, 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109556 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854348+0000", 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854350+0000", 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8840000000000001e-06 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854356+0000", 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6590000000000004e-06 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854369+0000", 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.289e-05 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854377+0000", 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4649999999999994e-06 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854454+0000", 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6791000000000004e-05 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854600+0000", 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145864 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.854751+0000", 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015046499999999999 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.857557+0000", 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028060419999999999 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.873657+0000", 2024-08-30T23:37:27.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016100032 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073675+0000", 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.200018474 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.073721+0000", 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6072999999999997e-05 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.074613+0000", 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00089165900000000005 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.096 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:109569 2.5 2:a8435ff8:::benchmark_data_smithi173_47195_object12561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 582.21218864699995, 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22083613699999999, 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 109569 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.097 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869490+0000", 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869491+0000", 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4270000000000001e-06 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869498+0000", 2024-08-30T23:37:27.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869510+0000", 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1630999999999999e-05 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869519+0000", 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.251e-06 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869586+0000", 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6699999999999995e-05 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869741+0000", 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155141 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:37:27.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.869888+0000", 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014657099999999999 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879878+0000", 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099902989999999994 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.879907+0000", 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8612999999999999e-05 2024-08-30T23:37:27.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089168+0000", 2024-08-30T23:37:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20926119900000001 2024-08-30T23:37:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089267+0000", 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9097000000000003e-05 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.090326+0000", 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010589670000000001 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.104 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144821 2.4 2:2415a84c:::benchmark_data_smithi173_47195_object47813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 535.51829467200002, 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23107453999999999, 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.105 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144821 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563384+0000", 2024-08-30T23:37:27.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563385+0000", 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.59e-06 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563391+0000", 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563404+0000", 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2982e-05 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.563411+0000", 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6599999999999995e-06 2024-08-30T23:37:27.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565303+0000", 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001891846 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565431+0000", 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012746400000000001 2024-08-30T23:37:27.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.565550+0000", 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011972900000000001 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.573426+0000", 2024-08-30T23:37:27.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0078756799999999995 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588851+0000", 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015424679 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793322+0000", 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20447127600000001 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.111 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.793387+0000", 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4586999999999997e-05 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.794458+0000", 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010715060000000001 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144836 2.4 2:2d7ce86c:::benchmark_data_smithi173_47195_object47828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:37:27.112 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 535.49356130599995, 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23589892500000001, 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144836 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.113 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588117+0000", 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588118+0000", 2024-08-30T23:37:27.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3829999999999999e-06 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588123+0000", 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3000000000000003e-06 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588169+0000", 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6199999999999998e-05 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.588182+0000", 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2748999999999999e-05 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.592926+0000", 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047443779999999996 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595020+0000", 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002094243 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.595429+0000", 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040842299999999999 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603861+0000", 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0084324050000000005 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.604206+0000", 2024-08-30T23:37:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034452400000000002 2024-08-30T23:37:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.822951+0000", 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21874496900000001 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.823010+0000", 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9601000000000001e-05 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.824016+0000", 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010057499999999999 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:193978 2.5 2:a2ee5e67:::benchmark_data_smithi173_47195_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 464.32553185400002, 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.283382829, 2024-08-30T23:37:27.121 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193978 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.122 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756148+0000", 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.123 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756154+0000", 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756166+0000", 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041e-05 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756172+0000", 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2349999999999996e-06 2024-08-30T23:37:27.124 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756333+0000", 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000160767 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756484+0000", 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015133799999999999 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:37:27.125 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756627+0000", 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014263700000000001 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764342+0000", 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077147439999999999 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.769055+0000", 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047136399999999998 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.126 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039012+0000", 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26995667699999998 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039055+0000", 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2567000000000001e-05 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039529+0000", 2024-08-30T23:37:27.127 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047478400000000001 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:193978 2.5 e200/182 ondisk, result = 0)", 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 464.31757658800001, 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27543190299999998, 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:27.128 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.129 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764103+0000", 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764108+0000", 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9080000000000003e-06 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038928+0000", 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27481996400000003 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.130 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038937+0000", 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7860000000000008e-06 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038998+0000", 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0915999999999997e-05 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039007+0000", 2024-08-30T23:37:27.131 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4319999999999999e-06 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039534+0000", 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052686200000000001 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e200,182 2:a1c9afc1:::benchmark_data_smithi173_47195_object1378:head,146'6135)", 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 464.29886838599998, 2024-08-30T23:37:27.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42837196799999999, 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:37:27.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782811+0000", 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782816+0000", 2024-08-30T23:37:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6969999999999998e-06 2024-08-30T23:37:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210395+0000", 2024-08-30T23:37:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42757847500000001 2024-08-30T23:37:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210404+0000", 2024-08-30T23:37:27.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0000000000000002e-06 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210493+0000", 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9438000000000007e-05 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.211182+0000", 2024-08-30T23:37:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068873000000000003 2024-08-30T23:37:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.138 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2685 3.6 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.38814465899998, 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22443937, 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.139 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2685 2024-08-30T23:37:27.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:37:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:37:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693535+0000", 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.009e-06 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693573+0000", 2024-08-30T23:37:27.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7843999999999999e-05 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693589+0000", 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6359e-05 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693603+0000", 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.412e-05 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693657+0000", 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3757e-05 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693735+0000", 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8444999999999999e-05 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693820+0000", 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4290999999999996e-05 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.694189+0000", 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036947500000000002 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.874073+0000", 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.179884508 2024-08-30T23:37:27.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917876+0000", 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043802633000000001 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917931+0000", 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5297999999999998e-05 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917973+0000", 2024-08-30T23:37:27.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1631000000000002e-05 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176646 3.e 3:78ef2980:::benchmark_data_smithi173_60780_object73192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 262.34684468299997, 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22395779099999999, 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.148 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176646 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:37:27.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734835+0000", 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7399999999999991e-07 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734839+0000", 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5800000000000002e-06 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734853+0000", 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.375e-05 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734858+0000", 2024-08-30T23:37:27.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738711+0000", 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003852847 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738850+0000", 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000138818 2024-08-30T23:37:27.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738954+0000", 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001037 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748083+0000", 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091293829999999996 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748375+0000", 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000292427 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748430+0000", 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4191999999999999e-05 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.756943+0000", 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085137770000000002 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958661+0000", 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.201717853 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958707+0000", 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5840999999999997e-05 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958791+0000", 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4349000000000002e-05 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.156 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176647 3.e 3:751af89f:::benchmark_data_smithi173_60780_object73193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 262.345595925, 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223004061, 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176647 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.157 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736083+0000", 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8299999999999995e-07 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736088+0000", 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2370000000000003e-06 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736098+0000", 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0835e-05 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736104+0000", 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742826+0000", 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006721505 2024-08-30T23:37:27.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742967+0000", 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140977 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.743123+0000", 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015588399999999999 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748383+0000", 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052605059999999999 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.758513+0000", 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010130116 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767529+0000", 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090161789999999992 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771374+0000", 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038445660000000002 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958849+0000", 2024-08-30T23:37:27.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18747534399999999 2024-08-30T23:37:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958930+0000", 2024-08-30T23:37:27.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0463000000000005e-05 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.959086+0000", 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000156701 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:31057 4.b 4:d5378b27:::benchmark_data_smithi173_74914_object31056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.39432038800001, 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.641950197, 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:27.167 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 31057 2024-08-30T23:37:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:37:27.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:37:27.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687360+0000", 2024-08-30T23:37:27.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0669999999999999e-06 2024-08-30T23:37:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687397+0000", 2024-08-30T23:37:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6677999999999997e-05 2024-08-30T23:37:27.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687413+0000", 2024-08-30T23:37:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532999999999999e-05 2024-08-30T23:37:27.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687421+0000", 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9410000000000007e-06 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687486+0000", 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4894000000000006e-05 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687630+0000", 2024-08-30T23:37:27.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014421900000000001 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687780+0000", 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000149748 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689089+0000", 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013092609999999999 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689219+0000", 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013019 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329200+0000", 2024-08-30T23:37:27.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63998067999999997 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329262+0000", 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1508999999999997e-05 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329308+0000", 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6477000000000002e-05 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.176 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.0 319/318 [PushOp(4:08a755c5:::benchmark_data_smithi173_74914_object390:head, version: 311'18, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:08a755c5:::benchmark_data_smithi173_74914_object390:head@311'18, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 184.15784846099999, 2024-08-30T23:37:27.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29680203999999999, 2024-08-30T23:37:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:37:27.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923831+0000", 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923901+0000", 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0507999999999996e-05 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219654+0000", 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.295752615 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219681+0000", 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7390999999999999e-05 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219790+0000", 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108904 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219807+0000", 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6871000000000002e-05 2024-08-30T23:37:27.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.220632+0000", 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082508799999999997 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:61123 4.b 4:d896c312:::benchmark_data_smithi173_74914_object61122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 152.86828105800001, 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93993815199999997, 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 61123 2024-08-30T23:37:27.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213400+0000", 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5000000000000002e-06 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213421+0000", 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1537000000000001e-05 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213434+0000", 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2478e-05 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213440+0000", 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8699999999999997e-06 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213485+0000", 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5439999999999999e-05 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213576+0000", 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0767999999999996e-05 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213667+0000", 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1135000000000001e-05 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215888+0000", 2024-08-30T23:37:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022207139999999999 2024-08-30T23:37:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215938+0000", 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0022000000000003e-05 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153264+0000", 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93732649300000004 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153304+0000", 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0139999999999999e-05 2024-08-30T23:37:27.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153335+0000", 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1055e-05 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:61123 4.b e335/306 ondisk, result = 0)", 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 152.865640259, 2024-08-30T23:37:27.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93730281299999996, 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:37:27.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216039+0000", 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4099999999999999e-07 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216042+0000", 2024-08-30T23:37:27.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7529999999999999e-06 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153195+0000", 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93715309700000005 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153202+0000", 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7340000000000002e-06 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153255+0000", 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2420999999999998e-05 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153260+0000", 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7370000000000003e-06 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153341+0000", 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0530000000000003e-05 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.5 335/318 [PushOp(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head, version: 316'1784, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head@316'1784, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.90742765600001, 2024-08-30T23:37:27.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.83137486000000005, 2024-08-30T23:37:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:37:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:37:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174252+0000", 2024-08-30T23:37:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:37:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.202 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174402+0000", 2024-08-30T23:37:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015002700000000001 2024-08-30T23:37:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003927+0000", 2024-08-30T23:37:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82952564699999998 2024-08-30T23:37:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003946+0000", 2024-08-30T23:37:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8678999999999999e-05 2024-08-30T23:37:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004043+0000", 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6718999999999993e-05 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004054+0000", 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0925e-05 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.005626+0000", 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001571813 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98225 4.5 e353/337 4:ad425262:::benchmark_data_smithi173_74914_object98224:head v 353'6223, mlcod=353'6220)", 2024-08-30T23:37:27.207 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 115.636708816, 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32306562100000003, 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444971+0000", 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0699999999999999e-06 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.445067+0000", 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5938000000000002e-05 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765335+0000", 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32026851299999998 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765347+0000", 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.151e-05 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765448+0000", 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100967 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765468+0000", 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0219000000000001e-05 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.767987+0000", 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025194890000000002 2024-08-30T23:37:27.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.768035+0000", 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7914999999999998e-05 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98228 4.7 e353/329 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head v 353'6056, mlcod=353'6052)", 2024-08-30T23:37:27.213 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:37:27.214 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 115.63357213099999, 2024-08-30T23:37:27.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35600754000000001, 2024-08-30T23:37:27.214 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.214 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.214 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:37:27.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:37:27.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448107+0000", 2024-08-30T23:37:27.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3300000000000001e-07 2024-08-30T23:37:27.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448202+0000", 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4503e-05 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801211+0000", 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35300926399999999 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801228+0000", 2024-08-30T23:37:27.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6694999999999999e-05 2024-08-30T23:37:27.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801373+0000", 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014589400000000001 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801409+0000", 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6025999999999997e-05 2024-08-30T23:37:27.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804078+0000", 2024-08-30T23:37:27.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026682939999999999 2024-08-30T23:37:27.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804114+0000", 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6031000000000001e-05 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:143754 4.0 e365/329 4:0c834e11:::benchmark_data_smithi173_74914_object45512:head v 365'9153, mlcod=365'9150)", 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:19.176383+0000", 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.905295229999993, 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.225303222, 2024-08-30T23:37:27.222 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176383+0000", 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176383+0000", 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176385+0000", 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4309999999999999e-06 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176394+0000", 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8230000000000006e-06 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398124+0000", 2024-08-30T23:37:27.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22172945599999999 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398137+0000", 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3519999999999999e-05 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398255+0000", 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011801100000000001 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398297+0000", 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1956000000000003e-05 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.401655+0000", 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003358069 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.401686+0000", 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0956e-05 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:143768 4.3 e365/306 4:c93e37ef:::benchmark_data_smithi173_74914_object45526:head v 365'8833, mlcod=365'8830)", 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.649712148000006, 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.374659609, 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:27.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431967+0000", 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5000000000000001e-07 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431975+0000", 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4139999999999997e-06 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.793620+0000", 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36164507800000001 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:27.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.793628+0000", 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000005e-06 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.794714+0000", 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010868989999999999 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.794732+0000", 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7396000000000001e-05 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:27.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.806585+0000", 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011852713000000001 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.806626+0000", 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1248999999999998e-05 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:27.231 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:37:27.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:27.253 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:27.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:27 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs degraded) 2024-08-30T23:37:27.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1060591149' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [2, 4, 9, 7, 8, 6]}]': finished 2024-08-30T23:37:27.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2279080583' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:37:27.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:27 smithi136 ceph-mon[20207]: osdmap e386: 12 total, 11 up, 12 in 2024-08-30T23:37:27.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:27 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs degraded) 2024-08-30T23:37:27.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1060591149' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [2, 4, 9, 7, 8, 6]}]': finished 2024-08-30T23:37:27.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2279080583' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:37:27.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:27 smithi173 ceph-mon[20554]: osdmap e386: 12 total, 11 up, 12 in 2024-08-30T23:37:27.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:27 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs degraded) 2024-08-30T23:37:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1060591149' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.5", "id": [2, 4, 9, 7, 8, 6]}]': finished 2024-08-30T23:37:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2279080583' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:37:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:27 smithi106 ceph-mon[16603]: osdmap e386: 12 total, 11 up, 12 in 2024-08-30T23:37:27.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:27.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:27.503 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:27.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:27.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:27.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:27.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:27.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:27.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:27.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:27.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:37:27.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:28.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:28.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:28.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:28.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:28.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:28.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:28 smithi136 ceph-mon[20207]: pgmap v934: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:28.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:28 smithi173 ceph-mon[20554]: pgmap v934: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:28 smithi106 ceph-mon[16603]: pgmap v934: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:28.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:28.389 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:28.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:37:28.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:28.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:28.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:28.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:28.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:28.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:28.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:28.894 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:28.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:28.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:37:28.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:29.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:29.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:29.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:29.187 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:37:29.188 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:37:29.188 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-08-30T23:37:29.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-08-30T23:37:29.193 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:29.218 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:29.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:29.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:29.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:29.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:29.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:29.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:29.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:29.630 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:29.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:29.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:29.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:29.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:29.906 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:29.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:30.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:30.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:30.104 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:30.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:30.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:30 smithi106 ceph-mon[16603]: osdmap e387: 12 total, 11 up, 12 in 2024-08-30T23:37:30.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:30 smithi106 ceph-mon[16603]: pgmap v936: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:30.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:30.121 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:30.122 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:30.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:30.142 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:30.143 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:30.149 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:30.149 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:30.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:30.177 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:30.177 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:30.198 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:30.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:30.219 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:30.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:30.242 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:30.242 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:30.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:30.263 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:30.263 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:30.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:30.284 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:30.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:30.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:30.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:30.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:30 smithi136 ceph-mon[20207]: osdmap e387: 12 total, 11 up, 12 in 2024-08-30T23:37:30.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:30 smithi136 ceph-mon[20207]: pgmap v936: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:30.306 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:30.306 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:30.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:30 smithi173 ceph-mon[20554]: osdmap e387: 12 total, 11 up, 12 in 2024-08-30T23:37:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:30 smithi173 ceph-mon[20554]: pgmap v936: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:30.327 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:30.327 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:30.348 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:30.348 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:30.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:30.418 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:30.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:30.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:30.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:30.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:30.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:30.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:30.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:30.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:30.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:31.018 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:37:31.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:31.053 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:31.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:31.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:31.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3228270555' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:31.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3298691894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:31.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:31.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3228270555' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:31.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3298691894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:31.305 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:31.311 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3228270555' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3298691894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:31.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:31.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:31.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:31.558 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:31.566 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:31.566 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:31.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:31.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:31.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:31.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:31.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:31.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:31.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:32.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:32.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:32.143 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:32 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:32.143 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3228270555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:32.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:32 smithi106 ceph-mon[16603]: osdmap e388: 12 total, 11 up, 12 in 2024-08-30T23:37:32.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:32 smithi106 ceph-mon[16603]: pgmap v938: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:32.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:32.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:32 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3228270555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:32 smithi136 ceph-mon[20207]: osdmap e388: 12 total, 11 up, 12 in 2024-08-30T23:37:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:32 smithi136 ceph-mon[20207]: pgmap v938: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:32.298 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:32.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:32.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:32 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:32.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3228270555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:32.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:32 smithi173 ceph-mon[20554]: osdmap e388: 12 total, 11 up, 12 in 2024-08-30T23:37:32.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:32 smithi173 ceph-mon[20554]: pgmap v938: 1 pgs: 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:32.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:32.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:32.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:32.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:32.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:32.698 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:32.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:32.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:32.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:32.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:32.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:33.026 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_3' created 2024-08-30T23:37:33.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:37:33.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:33.079 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:33.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:33.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1386742210' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-30T23:37:33.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:33.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:37:33.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1386742210' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-30T23:37:33.332 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1386742210' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-30T23:37:33.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:33.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:33.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:33.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.586 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:33.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:33.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:33.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:33.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:33.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-08-30T23:37:33.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:33.820 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:37:33.820 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:37:33.820 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-30T23:37:33.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd out 10 2024-08-30T23:37:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:33.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:33.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:33.964 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:33.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:33.967 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:33.968 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:34.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:34.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:34.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:34.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:34.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1386742210' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-30T23:37:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:34 smithi136 ceph-mon[20207]: osdmap e389: 12 total, 11 up, 12 in 2024-08-30T23:37:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:34 smithi136 ceph-mon[20207]: pgmap v940: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:34.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1386742210' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-30T23:37:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:34 smithi173 ceph-mon[20554]: osdmap e389: 12 total, 11 up, 12 in 2024-08-30T23:37:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:34 smithi173 ceph-mon[20554]: pgmap v940: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:34.343 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1386742210' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-30T23:37:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:34 smithi106 ceph-mon[16603]: osdmap e389: 12 total, 11 up, 12 in 2024-08-30T23:37:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:34 smithi106 ceph-mon[16603]: pgmap v940: 17 pgs: 1 creating+peering, 15 unknown, 1 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:34.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:34.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:34.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.594 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:34.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:34.690 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:34.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:34.699 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:34.699 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:34.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:37:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:34.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:34.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.845 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:34.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:34.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:34.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:34.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:35.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:35.097 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:35.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:35.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:35.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:35 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:37:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:35 smithi136 ceph-mon[20207]: osdmap e390: 12 total, 11 up, 12 in 2024-08-30T23:37:35.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:35 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:37:35.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:35 smithi173 ceph-mon[20554]: osdmap e390: 12 total, 11 up, 12 in 2024-08-30T23:37:35.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:35.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:35 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:37:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:35 smithi106 ceph-mon[16603]: osdmap e390: 12 total, 11 up, 12 in 2024-08-30T23:37:35.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:37:35.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:35.481 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:35.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:35.607 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:35.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:35.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:35.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:35.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:35.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:35.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:35.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:36.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:36.112 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:36.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:36.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:36.235 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:36.236 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:36.238 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:36.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:36.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:36.245 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:36.246 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:36.276 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:36.277 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:36.296 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:36.297 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:36.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:36 smithi106 ceph-mon[16603]: osdmap e391: 12 total, 11 up, 12 in 2024-08-30T23:37:36.297 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:36 smithi106 ceph-mon[16603]: pgmap v943: 17 pgs: 4 creating+peering, 9 unknown, 4 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:36.318 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:36.318 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:36 smithi173 ceph-mon[20554]: osdmap e391: 12 total, 11 up, 12 in 2024-08-30T23:37:36.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:36 smithi173 ceph-mon[20554]: pgmap v943: 17 pgs: 4 creating+peering, 9 unknown, 4 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:36.340 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:36.340 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:36.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:36.361 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:36.361 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:36.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:36.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:36.383 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:36.383 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:36.405 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:36.405 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:36.427 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:36.427 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:36.449 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:36.449 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:36.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:36.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:36.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:36.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:36.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:36 smithi136 ceph-mon[20207]: osdmap e391: 12 total, 11 up, 12 in 2024-08-30T23:37:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:36 smithi136 ceph-mon[20207]: pgmap v943: 17 pgs: 4 creating+peering, 9 unknown, 4 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:36.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:36.723 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:36.724 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:36.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:36.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:36.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:36.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:36.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:36.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:37.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:37.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:37.052 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-08-30T23:37:37.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:37.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:37.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:37.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:37.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2639996957' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:37:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1570849997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2639996957' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:37:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1570849997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:37.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:37.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:37.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:37.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2639996957' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:37:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1570849997' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:37.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:37.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:37.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:37.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:37.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:37.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:37.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:38.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:38.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:38.059 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:37:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:38.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:38.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:38.152 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:38.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:38.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2639996957' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:37:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:38 smithi173 ceph-mon[20554]: osdmap e392: 12 total, 11 up, 12 in 2024-08-30T23:37:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:38 smithi173 ceph-mon[20554]: pgmap v945: 17 pgs: 2 active+undersized, 4 creating+peering, 1 unknown, 10 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3366750831' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:38.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2639996957' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:37:38.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:38 smithi106 ceph-mon[16603]: osdmap e392: 12 total, 11 up, 12 in 2024-08-30T23:37:38.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:38 smithi106 ceph-mon[16603]: pgmap v945: 17 pgs: 2 active+undersized, 4 creating+peering, 1 unknown, 10 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:38.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3366750831' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:38.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:38 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:37:38.168+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e393 definitely_dead 0 2024-08-30T23:37:38.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:38.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:38.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2639996957' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:37:38.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:38 smithi136 ceph-mon[20207]: osdmap e392: 12 total, 11 up, 12 in 2024-08-30T23:37:38.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:38 smithi136 ceph-mon[20207]: pgmap v945: 17 pgs: 2 active+undersized, 4 creating+peering, 1 unknown, 10 active+clean; 577 KiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:37:38.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3366750831' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:38.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:38.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:38.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:38.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:38.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:38.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:38.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:38.879 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 153/143 [PushOp(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head, version: 114'1494, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2b2caff7:::benchmark_data_smithi173_47195_object24444:head@114'1494, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 593.98642101200005, 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19944226600000001, 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892157+0000", 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892158+0000", 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:44.892186+0000", 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7710000000000001e-05 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089119+0000", 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19693307299999999 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089137+0000", 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7533e-05 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089255+0000", 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118318 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.089275+0000", 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9828000000000001e-05 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:27:45.091600+0000", 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023249030000000001 2024-08-30T23:37:38.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:126363 2.7 e160/138 2:e820acf1:::benchmark_data_smithi173_47195_object29355:head v 160'7890, mlcod=119'2657)", 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 572.14895570500005, 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.217439837, 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.885 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729623+0000", 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.729630+0000", 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.674e-06 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940470+0000", 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21084006199999999 2024-08-30T23:37:38.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940478+0000", 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4390000000000003e-06 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940541+0000", 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3115999999999997e-05 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.940571+0000", 2024-08-30T23:37:38.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9981e-05 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947012+0000", 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0064411269999999996 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:06.947062+0000", 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0708999999999998e-05 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.889 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144840 2.1 2:8bdc20ef:::benchmark_data_smithi173_47195_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 547.28510004300006, 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21736361900000001, 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.890 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144840 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593478+0000", 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593481+0000", 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3460000000000001e-06 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593490+0000", 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593512+0000", 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2410000000000001e-05 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593522+0000", 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4129999999999995e-06 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593705+0000", 2024-08-30T23:37:38.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018286800000000001 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.593993+0000", 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028869300000000002 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.594314+0000", 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032096700000000002 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.599306+0000", 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049920049999999999 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603619+0000", 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043124419999999997 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810726+0000", 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20710743000000001 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810784+0000", 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7893999999999998e-05 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810842+0000", 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7791000000000003e-05 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:144845 2.1 2:80fd4430:::benchmark_data_smithi173_47195_object47837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 547.27525506100005, 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20768498299999999, 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.897 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 144845 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603323+0000", 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603326+0000", 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.373e-06 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603335+0000", 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5449999999999992e-06 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603359+0000", 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3377e-05 2024-08-30T23:37:38.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603370+0000", 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1759e-05 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603649+0000", 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027892500000000002 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603755+0000", 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010549500000000001 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.603833+0000", 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8150000000000002e-05 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.609421+0000", 2024-08-30T23:37:38.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0055884799999999998 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.640300+0000", 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.030878974999999999 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810910+0000", 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.170609233 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.810964+0000", 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4549999999999998e-05 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:31.811008+0000", 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4121e-05 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:155294 2.e 2:78ac910c:::benchmark_data_smithi173_47195_object58286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 532.75469615099996, 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26817000400000002, 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 155294 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123882+0000", 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123884+0000", 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3960000000000001e-06 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123889+0000", 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4820000000000002e-06 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123902+0000", 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3202e-05 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123910+0000", 2024-08-30T23:37:38.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8869999999999992e-06 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.123962+0000", 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1959000000000002e-05 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124045+0000", 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2850999999999997e-05 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:37:38.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.124115+0000", 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9513999999999999e-05 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.127474+0000", 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033596149999999998 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.128135+0000", 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00066107399999999999 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.391962+0000", 2024-08-30T23:37:38.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26382699500000001 2024-08-30T23:37:38.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392009+0000", 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6669999999999999e-05 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:46.392052+0000", 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3359000000000003e-05 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(2.4 182/143 [PushOp(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head, version: 132'4708, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:2524c11d:::benchmark_data_smithi173_47195_object75416:head@132'4708, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 531.43901738700004, 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25797871999999999, 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439561+0000", 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439562+0000", 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.439584+0000", 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2395000000000002e-05 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693170+0000", 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25358568300000001 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693191+0000", 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1192999999999999e-05 2024-08-30T23:37:38.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693349+0000", 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015837300000000001 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.693372+0000", 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2869000000000001e-05 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:28:47.697540+0000", 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004167558 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168870 2.e 2:74d7d6f3:::benchmark_data_smithi173_47195_object71862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 511.26294118999999, 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223579366, 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.916 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168870 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615637+0000", 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615640+0000", 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3590000000000002e-06 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615648+0000", 2024-08-30T23:37:38.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8559999999999996e-06 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615665+0000", 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6563999999999999e-05 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615676+0000", 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1075e-05 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615800+0000", 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123905 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615963+0000", 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016347900000000001 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616104+0000", 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140846 2024-08-30T23:37:38.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620671+0000", 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045667540000000001 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620686+0000", 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.484e-05 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839134+0000", 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21844793100000001 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839172+0000", 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7737999999999997e-05 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.839217+0000", 2024-08-30T23:37:38.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5018999999999997e-05 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168873 2.1 2:8ff1cc28:::benchmark_data_smithi173_47195_object71865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 511.26287327900002, 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22707487400000001, 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.923 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168873 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615705+0000", 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615706+0000", 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2500000000000001e-07 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615708+0000", 2024-08-30T23:37:38.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6970000000000002e-06 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615715+0000", 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8499999999999996e-06 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.615720+0000", 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.395e-06 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616068+0000", 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034794699999999998 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616173+0000", 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104145 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.616295+0000", 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012253000000000001 2024-08-30T23:37:38.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.620669+0000", 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043740690000000004 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.621058+0000", 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00038845600000000002 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842677+0000", 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22161980100000001 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842735+0000", 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7420000000000003e-05 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.842780+0000", 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5139000000000003e-05 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.929 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168892 2.e 2:7a358138:::benchmark_data_smithi173_47195_object71884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 511.24044897900001, 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208213383, 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.930 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168892 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638129+0000", 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638131+0000", 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638136+0000", 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:37:38.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638147+0000", 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1119e-05 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638153+0000", 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8320000000000002e-06 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638212+0000", 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9219999999999999e-05 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638342+0000", 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012929599999999999 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.638476+0000", 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013422799999999999 2024-08-30T23:37:38.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643016+0000", 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045403120000000003 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643358+0000", 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034153500000000002 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846238+0000", 2024-08-30T23:37:38.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.202880476 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846298+0000", 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9453000000000002e-05 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.846343+0000", 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5059999999999999e-05 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168895 2.1 2:802748b8:::benchmark_data_smithi173_47195_object71887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 511.23621268599999, 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21216137199999999, 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168895 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.937 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642366+0000", 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642368+0000", 2024-08-30T23:37:38.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3080000000000002e-06 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642376+0000", 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6769999999999995e-06 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642392+0000", 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642400+0000", 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6329999999999996e-06 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642458+0000", 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8498000000000001e-05 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642574+0000", 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115822 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.642662+0000", 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655709+0000", 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013046791 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655767+0000", 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8464000000000001e-05 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854414+0000", 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19864683999999999 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854479+0000", 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5042999999999994e-05 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854527+0000", 2024-08-30T23:37:38.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8195999999999997e-05 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168896 2.1 2:82321f29:::benchmark_data_smithi173_47195_object71888:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 511.23537966399999, 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.211510109, 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.943 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168896 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643199+0000", 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643200+0000", 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6899999999999996e-07 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643204+0000", 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643215+0000", 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.041e-05 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643220+0000", 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646422+0000", 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032022190000000001 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646565+0000", 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000142564 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.646696+0000", 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000130877 2024-08-30T23:37:38.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655755+0000", 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090594430000000004 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.655773+0000", 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7263999999999999e-05 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854613+0000", 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19883995700000001 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854672+0000", 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9522999999999999e-05 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.854709+0000", 2024-08-30T23:37:38.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6693000000000003e-05 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:168897 2.1 2:8f1372f3:::benchmark_data_smithi173_47195_object71889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 511.23485456600002, 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21982235999999999, 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.950 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168897 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643724+0000", 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643725+0000", 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643729+0000", 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643740+0000", 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1079e-05 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.643745+0000", 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2009999999999998e-06 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651404+0000", 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.007658916 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651490+0000", 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5811999999999998e-05 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.651572+0000", 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2472999999999995e-05 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656527+0000", 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049544910000000001 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.656537+0000", 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0497e-05 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863418+0000", 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20688067199999999 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863501+0000", 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2806000000000006e-05 2024-08-30T23:37:38.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:07.863546+0000", 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5430999999999999e-05 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:193992 2.7 e200/182 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head v 200'12142, mlcod=200'12139)", 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:37:38.956 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 476.10300669999998, 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60714938399999996, 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775572+0000", 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9599999999999999e-07 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.775578+0000", 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8089999999999998e-06 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378923+0000", 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60334498999999997 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.378934+0000", 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0772e-05 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379004+0000", 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9783000000000006e-05 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.379024+0000", 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0044e-05 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382681+0000", 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036575549999999998 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.382721+0000", 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9734999999999998e-05 2024-08-30T23:37:38.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:194011 2.1 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 476.08129262699998, 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82807010000000003, 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.961 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 194011 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797286+0000", 2024-08-30T23:37:38.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797287+0000", 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.277e-06 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797292+0000", 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6419999999999998e-06 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797301+0000", 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0070000000000003e-06 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797307+0000", 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2809999999999997e-06 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797355+0000", 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8557000000000003e-05 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797473+0000", 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117904 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,3", 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797597+0000", 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012318100000000001 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.802546+0000", 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.004948956 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.939024+0000", 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.13647832200000001 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625288+0000", 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.686263866 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625326+0000", 2024-08-30T23:37:38.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8028999999999998e-05 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625356+0000", 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0077999999999999e-05 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:1186 4.4 e311/306 4:2bc10e34:::benchmark_data_smithi173_74914_object1185:head v 311'76, mlcod=311'72)", 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:37:38.967 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 217.01598659699999, 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21510685600000001, 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862593+0000", 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4200000000000005e-07 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862614+0000", 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1129e-05 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077250+0000", 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214636047 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077258+0000", 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9789999999999993e-06 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077310+0000", 2024-08-30T23:37:38.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2367999999999997e-05 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077334+0000", 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4168e-05 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077669+0000", 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033434399999999998 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077699+0000", 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9978999999999999e-05 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:158473 4.4 e371/331 4:22dbe5ff:::benchmark_data_smithi173_74914_object60231:head v 371'9866, mlcod=371'9863)", 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:36.283652+0000", 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 62.594925891999999, 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.093542276999999993, 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.972 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.283652+0000", 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.283652+0000", 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.283653+0000", 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.031e-06 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.283662+0000", 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9619999999999999e-06 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.373660+0000", 2024-08-30T23:37:38.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.089997714000000006 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.373668+0000", 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999993e-06 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.373730+0000", 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1748000000000006e-05 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.373746+0000", 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6220000000000001e-05 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.377160+0000", 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034137199999999999 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:36.377195+0000", 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4672000000000003e-05 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:168798 4.6 e375/369 4:68592339:::benchmark_data_smithi173_74914_object70556:head v 375'10605, mlcod=375'10602)", 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.351469711999997, 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208871322, 2024-08-30T23:37:38.977 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:37:38.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527110+0000", 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5499999999999997e-07 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527116+0000", 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0209999999999996e-06 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732384+0000", 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205268487 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732390+0000", 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0419999999999999e-06 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732440+0000", 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9663000000000002e-05 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732470+0000", 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0412e-05 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735945+0000", 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034745869999999999 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735980+0000", 2024-08-30T23:37:38.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5255e-05 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:169046 4.6 e375/369 4:6cb7fed4:::benchmark_data_smithi173_74914_object70804:head v 375'10614, mlcod=375'10611)", 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.868543+0000", 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 51.010035748, 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.111472528, 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.868543+0000", 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.868543+0000", 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.868544+0000", 2024-08-30T23:37:38.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.13e-06 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.868552+0000", 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4479999999999994e-06 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.974612+0000", 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.106059472 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.974618+0000", 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0449999999999997e-06 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.974683+0000", 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4950000000000007e-05 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.974706+0000", 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3288000000000001e-05 2024-08-30T23:37:38.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.979969+0000", 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052628600000000003 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.980015+0000", 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6335000000000003e-05 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:187945 4.6 e378/369 4:6de5cb2f:::benchmark_data_smithi173_74914_object89703:head v 378'11795, mlcod=378'11791)", 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:08.587630+0000", 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 30.290947907, 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.096484103000000002, 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.587630+0000", 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.587630+0000", 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.587632+0000", 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1939999999999999e-06 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.587640+0000", 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4460000000000001e-06 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.676179+0000", 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.088538678999999995 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.676191+0000", 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2106999999999999e-05 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.676328+0000", 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000137186 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.676355+0000", 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6872e-05 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.684078+0000", 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077226029999999998 2024-08-30T23:37:38.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:08.684115+0000", 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7015999999999999e-05 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:188471 4.4 e378/331 4:2d131ded:::benchmark_data_smithi173_74914_object90229:head v 378'11726, mlcod=378'11723)", 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:09.304375+0000", 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 29.574203752999999, 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.11579566099999999, 2024-08-30T23:37:38.991 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.304375+0000", 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.304375+0000", 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.304376+0000", 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6309999999999999e-06 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.304384+0000", 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0679999999999997e-06 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:38.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.416402+0000", 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.112017846 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.416408+0000", 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9240000000000004e-06 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.416476+0000", 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7879999999999994e-05 2024-08-30T23:37:38.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.416497+0000", 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0877e-05 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.420125+0000", 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036280560000000002 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:38.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:09.420170+0000", 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5379e-05 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:38.996 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:37:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:38.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:38.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:39.066 INFO:teuthology.orchestra.run.smithi106.stderr:marked out osd.10. 2024-08-30T23:37:39.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:39.119 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:39.208 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:39 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:37:39.208 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:39 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:37:39.208 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3366750831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:39.208 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:39 smithi106 ceph-mon[16603]: osdmap e393: 12 total, 11 up, 12 in 2024-08-30T23:37:39.208 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4274685559' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:37:39.208 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:39 smithi106 ceph-mon[16603]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:37:39.222 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.243 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:39.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:39 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:37:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:39 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:37:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3366750831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:39 smithi173 ceph-mon[20554]: osdmap e393: 12 total, 11 up, 12 in 2024-08-30T23:37:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4274685559' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:37:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:39 smithi173 ceph-mon[20554]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:37:39.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:39.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.367 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:39.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:39.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:39 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:37:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:39 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:37:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3366750831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:39 smithi136 ceph-mon[20207]: osdmap e393: 12 total, 11 up, 12 in 2024-08-30T23:37:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4274685559' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:37:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:39 smithi136 ceph-mon[20207]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:37:39.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:39.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:39.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:39.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:39.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:39.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:39.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:39.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:39.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:40.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:37:40.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:40.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:37:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.118 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:40.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:40.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:40.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:40.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4274685559' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:37:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:40 smithi136 ceph-mon[20207]: osdmap e394: 12 total, 11 up, 11 in 2024-08-30T23:37:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:40 smithi136 ceph-mon[20207]: pgmap v948: 17 pgs: 3 active+undersized, 14 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-08-30T23:37:40.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4274685559' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:37:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:40 smithi106 ceph-mon[16603]: osdmap e394: 12 total, 11 up, 11 in 2024-08-30T23:37:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:40 smithi106 ceph-mon[16603]: pgmap v948: 17 pgs: 3 active+undersized, 14 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-08-30T23:37:40.365 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:40.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:40.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4274685559' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:37:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:40 smithi173 ceph-mon[20554]: osdmap e394: 12 total, 11 up, 11 in 2024-08-30T23:37:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:40 smithi173 ceph-mon[20554]: pgmap v948: 17 pgs: 3 active+undersized, 14 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-08-30T23:37:40.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:40.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:40.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:40.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:37:40.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:40.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:40.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:40.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:40.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:40.980 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:40.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:40.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:40.999 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:41.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:41.113 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:41.209 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:41 smithi106 ceph-mon[16603]: osdmap e395: 12 total, 11 up, 11 in 2024-08-30T23:37:41.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:41.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:41.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:41.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:41.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:41.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:41 smithi136 ceph-mon[20207]: osdmap e395: 12 total, 11 up, 11 in 2024-08-30T23:37:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:41 smithi173 ceph-mon[20554]: osdmap e395: 12 total, 11 up, 11 in 2024-08-30T23:37:41.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.621 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:41.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:41.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:41.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:41.840 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:41.840 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:41.841 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:41.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:41.861 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:41.861 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:41.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:41.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:41.882 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:41.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:41.903 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:41.904 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:41.924 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:41.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:41.932 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-08-30T23:37:41.935 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:37:41.935 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:37:41.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:41.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:41.966 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:41.966 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:41.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:41.987 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:41.988 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:41.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:37:41.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:37:41.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_87639 2024-08-30T23:37:41.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:37:41.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:37:41.999 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:42.012 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:42.012 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:42.038 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:42.038 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:42.064 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:42.064 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:42.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:42.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:42.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:42.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:42.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:42 smithi106 ceph-mon[16603]: osdmap e396: 12 total, 11 up, 11 in 2024-08-30T23:37:42.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:42 smithi106 ceph-mon[16603]: pgmap v951: 17 pgs: 1 activating, 1 unknown, 2 active+undersized, 13 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-08-30T23:37:42.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4222546777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:42.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:42.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:42.405 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:42.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:42.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:42 smithi136 ceph-mon[20207]: osdmap e396: 12 total, 11 up, 11 in 2024-08-30T23:37:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:42 smithi136 ceph-mon[20207]: pgmap v951: 17 pgs: 1 activating, 1 unknown, 2 active+undersized, 13 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-08-30T23:37:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4222546777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:42 smithi173 ceph-mon[20554]: osdmap e396: 12 total, 11 up, 11 in 2024-08-30T23:37:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:42 smithi173 ceph-mon[20554]: pgmap v951: 17 pgs: 1 activating, 1 unknown, 2 active+undersized, 13 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-08-30T23:37:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4222546777' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:42.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:42.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:42.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:42.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:42.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:42.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:42.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:42.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:42.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 2427 2411 150.679 150.688 0.0025092 0.0065447 2024-08-30T23:37:43.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:43.044 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:43.098 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:37:43.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:43.167 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:43.199 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1399113596' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:37:43.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:43.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:43.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:43.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:43.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1399113596' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:37:43.546 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1399113596' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:37:43.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:43.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:43.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:43.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:43.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:43.929 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:43.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:43.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 4045 4029 125.898 101.125 0.00500942 0.00788423 2024-08-30T23:37:44.000 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:44.000 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:44.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:44.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:44.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:44.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:44.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:44.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:44.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:44 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:44.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1399113596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:37:44.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:44 smithi106 ceph-mon[16603]: osdmap e397: 12 total, 11 up, 11 in 2024-08-30T23:37:44.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:44 smithi106 ceph-mon[16603]: pgmap v953: 17 pgs: 1 activating, 1 unknown, 1 active+undersized, 1 active+undersized+degraded, 13 active+clean; 16 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 3.6 MiB/s wr, 57 op/s; 5/765 objects degraded (0.654%) 2024-08-30T23:37:44.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:44 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:44.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:44 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:44.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:44.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:44.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:44 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1399113596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:37:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:44 smithi136 ceph-mon[20207]: osdmap e397: 12 total, 11 up, 11 in 2024-08-30T23:37:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:44 smithi136 ceph-mon[20207]: pgmap v953: 17 pgs: 1 activating, 1 unknown, 1 active+undersized, 1 active+undersized+degraded, 13 active+clean; 16 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 3.6 MiB/s wr, 57 op/s; 5/765 objects degraded (0.654%) 2024-08-30T23:37:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:44 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:44 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:44.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:44.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:44 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1399113596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:37:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:44 smithi173 ceph-mon[20554]: osdmap e397: 12 total, 11 up, 11 in 2024-08-30T23:37:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:44 smithi173 ceph-mon[20554]: pgmap v953: 17 pgs: 1 activating, 1 unknown, 1 active+undersized, 1 active+undersized+degraded, 13 active+clean; 16 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 3.6 MiB/s wr, 57 op/s; 5/765 objects degraded (0.654%) 2024-08-30T23:37:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:44 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:44 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:44.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:44.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:44.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:44.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:44.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:37:44.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:44.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:44.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:44.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:37:44.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5552 5536 115.325 94.1875 0.00272658 0.00865483 2024-08-30T23:37:45.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:45.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:45.133 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:45.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:45.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:45.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.320 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:45.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:45 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 5/765 objects degraded (0.654%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:37:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:45 smithi106 ceph-mon[16603]: osdmap e398: 12 total, 11 up, 11 in 2024-08-30T23:37:45.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:45.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:45.527 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:37:45.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:37:45.527 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:37:45.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:37:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:45 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 5/765 objects degraded (0.654%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:37:45.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:45 smithi136 ceph-mon[20207]: osdmap e398: 12 total, 11 up, 11 in 2024-08-30T23:37:45.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:45 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 5/765 objects degraded (0.654%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:37:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:45 smithi173 ceph-mon[20554]: osdmap e398: 12 total, 11 up, 11 in 2024-08-30T23:37:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.582 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:45.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:45.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:45.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:45.771 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:45.772 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:45.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:45.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:45.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:37:45.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:45.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:45.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:45.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 7309 7293 113.946 109.812 0.0176965 0.00875201 2024-08-30T23:37:46.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:46.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:46.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:46.216 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:46.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:46.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:46.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:46.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:46.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:46.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:46.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:46.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:46.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:46 smithi106 ceph-mon[16603]: osdmap e399: 12 total, 11 up, 11 in 2024-08-30T23:37:46.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:46 smithi106 ceph-mon[16603]: pgmap v956: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 181 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 41 MiB/s wr, 654 op/s; 233/8679 objects degraded (2.685%); 325/8679 objects misplaced (3.745%) 2024-08-30T23:37:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:46 smithi136 ceph-mon[20207]: osdmap e399: 12 total, 11 up, 11 in 2024-08-30T23:37:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:46 smithi136 ceph-mon[20207]: pgmap v956: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 181 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 41 MiB/s wr, 654 op/s; 233/8679 objects degraded (2.685%); 325/8679 objects misplaced (3.745%) 2024-08-30T23:37:46.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:37:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:46 smithi173 ceph-mon[20554]: osdmap e399: 12 total, 11 up, 11 in 2024-08-30T23:37:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:46 smithi173 ceph-mon[20554]: pgmap v956: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 181 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 41 MiB/s wr, 654 op/s; 233/8679 objects degraded (2.685%); 325/8679 objects misplaced (3.745%) 2024-08-30T23:37:46.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:46.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:46.644 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:46.658 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:46.682 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:46.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:46.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:46.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:46.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:46.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:46.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:46.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:46.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 8955 8939 111.73 102.875 0.00652588 0.00894043 2024-08-30T23:37:47.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:47.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:47.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:47.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:47.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:47.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:47 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:47 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:47 smithi106 ceph-mon[16603]: osdmap e400: 12 total, 11 up, 11 in 2024-08-30T23:37:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:47.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:47.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:47.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:47 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:47 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:47 smithi136 ceph-mon[20207]: osdmap e400: 12 total, 11 up, 11 in 2024-08-30T23:37:47.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:47 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:47 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:47 smithi173 ceph-mon[20554]: osdmap e400: 12 total, 11 up, 11 in 2024-08-30T23:37:47.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:47.614 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:47.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:47.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:47.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:47.872 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:47.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:37:47.905 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:37:47.906 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:37:47.906 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:37:47.906 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:37:47.906 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:37:47.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:47.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 10735 10719 111.649 111.25 0.00320639 0.00894693 2024-08-30T23:37:48.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:48.101 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:48.101 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:48.101 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:48.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:48.114 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:37:48.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:48.127 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:48.128 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:48.152 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:48.153 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:48.175 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:48.175 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:48.175 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:48 smithi106 ceph-mon[16603]: pgmap v958: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 350 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s; 794/16758 objects degraded (4.738%); 325/16758 objects misplaced (1.939%) 2024-08-30T23:37:48.175 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1248687422' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:48.175 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2534365223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:48.197 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:48.197 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:48.220 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:48.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:48.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:48.243 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:48.243 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:48.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:48.264 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:48.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:48.285 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:48.285 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:48.308 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:48.308 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:48.329 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:48.329 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:48.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:48.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:48.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:48.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:48 smithi136 ceph-mon[20207]: pgmap v958: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 350 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s; 794/16758 objects degraded (4.738%); 325/16758 objects misplaced (1.939%) 2024-08-30T23:37:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1248687422' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2534365223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:48.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:48 smithi173 ceph-mon[20554]: pgmap v958: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 350 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s; 794/16758 objects degraded (4.738%); 325/16758 objects misplaced (1.939%) 2024-08-30T23:37:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1248687422' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:37:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2534365223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:48.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:48.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:48.705 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:48.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:48.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:48.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:48.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:48.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:48.979 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:48.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 11925 11909 106.323 74.375 0.00222042 0.00938542 2024-08-30T23:37:49.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:49.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:49.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:49.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:49.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:49.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:49 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1248687422' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:49 smithi173 ceph-mon[20554]: osdmap e401: 12 total, 11 up, 11 in 2024-08-30T23:37:49.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:49.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:49.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:49.489 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:49 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:49.489 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1248687422' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:49.489 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:49 smithi136 ceph-mon[20207]: osdmap e401: 12 total, 11 up, 11 in 2024-08-30T23:37:49.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:49.519 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:49 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:49.519 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1248687422' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:37:49.519 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:49 smithi106 ceph-mon[16603]: osdmap e401: 12 total, 11 up, 11 in 2024-08-30T23:37:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:49.553 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:49.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:49.644 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:49.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:49.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:49.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:49.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:49.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:49.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 13810 13794 107.758 117.812 0.00156988 0.00926867 2024-08-30T23:37:50.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:50.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:50.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:50.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:50.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:50.288 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:50 smithi106 ceph-mon[16603]: pgmap v960: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 540 MiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s; 1286/25890 objects degraded (4.967%); 325/25890 objects misplaced (1.255%) 2024-08-30T23:37:50.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:50.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:50.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:50.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:50 smithi136 ceph-mon[20207]: pgmap v960: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 540 MiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s; 1286/25890 objects degraded (4.967%); 325/25890 objects misplaced (1.255%) 2024-08-30T23:37:50.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:50.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:50 smithi173 ceph-mon[20554]: pgmap v960: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 540 MiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s; 1286/25890 objects degraded (4.967%); 325/25890 objects misplaced (1.255%) 2024-08-30T23:37:50.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:50.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:194011 2.1 e200/97 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head v 200'12204, mlcod=200'12201)", 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:37:50.689 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 487.88900607699998, 2024-08-30T23:37:50.690 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82744930299999997, 2024-08-30T23:37:50.690 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.690 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.690 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.690 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.690 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.692 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797678+0000", 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1800000000000005e-07 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797683+0000", 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6529999999999999e-06 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620982+0000", 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82329911700000002 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.693 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620996+0000", 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3995999999999999e-05 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621133+0000", 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013682700000000001 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621198+0000", 2024-08-30T23:37:50.694 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4368000000000004e-05 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625091+0000", 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003893194 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.695 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625126+0000", 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5429999999999998e-05 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:103424 3.2 3:4caab458:::benchmark_data_smithi173_60780_object103423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 369.427059472, 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.221222537, 2024-08-30T23:37:50.696 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103424 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259624+0000", 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0800000000000004e-07 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259682+0000", 2024-08-30T23:37:50.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7364999999999997e-05 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259692+0000", 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0686000000000001e-05 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259698+0000", 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.361e-06 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261702+0000", 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020041289999999999 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261778+0000", 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6148000000000003e-05 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261872+0000", 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4598000000000005e-05 2024-08-30T23:37:50.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.263845+0000", 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001972419 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.264707+0000", 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086252099999999999 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.701 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480753+0000", 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.216045457 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480795+0000", 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2605000000000003e-05 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480846+0000", 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0540000000000001e-05 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.702 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198168 3.2 3:46c77281:::benchmark_data_smithi173_60780_object94714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 257.72446679799998, 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22735555199999999, 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:50.703 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198168 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:37:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962217+0000", 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2099999999999995e-07 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962221+0000", 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999999e-06 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962234+0000", 2024-08-30T23:37:50.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2412000000000001e-05 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962241+0000", 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6789999999999999e-06 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962296+0000", 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5347999999999999e-05 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962411+0000", 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011498299999999999 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962524+0000", 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011293700000000001 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.708 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.967040+0000", 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045164259999999996 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.968062+0000", 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010217270000000001 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189490+0000", 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22142835899999999 2024-08-30T23:37:50.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189535+0000", 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4338999999999997e-05 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189572+0000", 2024-08-30T23:37:50.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6990999999999999e-05 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198178 3.2 3:47f13797:::benchmark_data_smithi173_60780_object94724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 257.71367737000003, 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21670980100000001, 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.711 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198178 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:37:50.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973007+0000", 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973013+0000", 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1140000000000002e-06 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973023+0000", 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0516000000000001e-05 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973030+0000", 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8390000000000001e-06 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973072+0000", 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1919000000000003e-05 2024-08-30T23:37:50.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973158+0000", 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6148000000000003e-05 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973234+0000", 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6165e-05 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.976842+0000", 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036077290000000001 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.978153+0000", 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013109930000000001 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189617+0000", 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21146347300000001 2024-08-30T23:37:50.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189674+0000", 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7082e-05 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189715+0000", 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1656999999999998e-05 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206885 3.5 e300/206 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head v 300'12859, mlcod=300'12855)", 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.64543963099999, 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23467711799999999, 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041244+0000", 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041249+0000", 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2360000000000004e-06 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271945+0000", 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.230695859 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271961+0000", 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5815999999999999e-05 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272088+0000", 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126721 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272127+0000", 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8872000000000003e-05 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275888+0000", 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037616350000000002 2024-08-30T23:37:50.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275920+0000", 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2255000000000002e-05 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206897 3.b 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.629598622, 2024-08-30T23:37:50.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58358057900000004, 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206897 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:37:50.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057085+0000", 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3399999999999999e-07 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057088+0000", 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8609999999999998e-06 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057094+0000", 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5429999999999999e-06 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057098+0000", 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9720000000000003e-06 2024-08-30T23:37:50.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057149+0000", 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0683000000000002e-05 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057264+0000", 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114772 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057387+0000", 2024-08-30T23:37:50.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012322799999999999 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.061158+0000", 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037710170000000002 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062218+0000", 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001059948 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640580+0000", 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57836202199999998 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640625+0000", 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5116e-05 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640665+0000", 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9882999999999997e-05 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.728 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206901 3.7 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.62391736399999, 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38317995100000002, 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:37:50.729 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206901 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062767+0000", 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062771+0000", 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6779999999999999e-06 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062785+0000", 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3839000000000001e-05 2024-08-30T23:37:50.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062793+0000", 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063873+0000", 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001080632 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063965+0000", 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1717000000000003e-05 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.064031+0000", 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5729999999999996e-05 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.067745+0000", 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037140430000000002 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.071385+0000", 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036407620000000001 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445862+0000", 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37447608799999998 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445912+0000", 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0059999999999998e-05 2024-08-30T23:37:50.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445946+0000", 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3927000000000002e-05 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206900 3.5 e300/206 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head v 300'12860, mlcod=300'12856)", 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.62347609700001, 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36104319299999998, 2024-08-30T23:37:50.735 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:37:50.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063208+0000", 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063213+0000", 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0080000000000002e-06 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420406+0000", 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35719326299999998 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420426+0000", 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0253000000000001e-05 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420597+0000", 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170786 2024-08-30T23:37:50.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420630+0000", 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3578999999999998e-05 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424218+0000", 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035875490000000002 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424250+0000", 2024-08-30T23:37:50.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2030999999999999e-05 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62091 4.4 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 173.52101513299999, 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318614905, 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.740 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62091 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165670+0000", 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165713+0000", 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3368000000000002e-05 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165732+0000", 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9031000000000001e-05 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165740+0000", 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8050000000000007e-06 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165802+0000", 2024-08-30T23:37:50.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2279999999999993e-05 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165946+0000", 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014352899999999999 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.166046+0000", 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100046 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169317+0000", 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003271267 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169909+0000", 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059201200000000005 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484178+0000", 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31426850299999998 2024-08-30T23:37:50.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484234+0000", 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6678000000000002e-05 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484283+0000", 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8457e-05 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62105 4.4 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 173.507430801, 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34846845199999998, 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62105 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179254+0000", 2024-08-30T23:37:50.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179297+0000", 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3108000000000001e-05 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179311+0000", 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.345e-05 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179316+0000", 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179371+0000", 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5316999999999999e-05 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179508+0000", 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643600000000001 2024-08-30T23:37:50.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179665+0000", 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015680499999999999 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182071+0000", 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024068090000000002 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182226+0000", 2024-08-30T23:37:50.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015503899999999999 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527606+0000", 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34537996300000001 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527669+0000", 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2966999999999995e-05 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527721+0000", 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1224000000000003e-05 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62174 4.4 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 173.162219021, 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57863658799999995, 2024-08-30T23:37:50.753 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62174 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524466+0000", 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0600000000000002e-06 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524501+0000", 2024-08-30T23:37:50.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4635999999999998e-05 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524521+0000", 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0193000000000001e-05 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524531+0000", 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7780000000000002e-06 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524604+0000", 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3646000000000006e-05 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524763+0000", 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015852999999999999 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524940+0000", 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017771799999999999 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527733+0000", 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027928789999999998 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.528529+0000", 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000795487 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103030+0000", 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57450106899999998 2024-08-30T23:37:50.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103068+0000", 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8398000000000001e-05 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103100+0000", 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2193999999999997e-05 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62186 4.4 4:257aab52:::benchmark_data_smithi173_74914_object62185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 173.1331069, 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55186797300000001, 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62186 2024-08-30T23:37:50.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553578+0000", 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553612+0000", 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3773999999999999e-05 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553627+0000", 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5410999999999999e-05 2024-08-30T23:37:50.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553637+0000", 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0101e-05 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553709+0000", 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2365000000000003e-05 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553885+0000", 2024-08-30T23:37:50.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017522699999999999 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.554065+0000", 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000180101 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556530+0000", 2024-08-30T23:37:50.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024650240000000001 2024-08-30T23:37:50.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556570+0000", 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9975999999999998e-05 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105343+0000", 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54877291299999997 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105399+0000", 2024-08-30T23:37:50.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6648999999999999e-05 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105444+0000", 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4660000000000003e-05 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:93692 4.4 4:24d1e68e:::benchmark_data_smithi173_74914_object93691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 143.66290325, 2024-08-30T23:37:50.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28134646200000002, 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 93692 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:37:50.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023781+0000", 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023840+0000", 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9305999999999998e-05 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023854+0000", 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3773e-05 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023872+0000", 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7708000000000001e-05 2024-08-30T23:37:50.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023930+0000", 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8007999999999998e-05 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024050+0000", 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120209 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:37:50.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024158+0000", 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010744299999999999 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026599+0000", 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024417430000000001 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026641+0000", 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305020+0000", 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27837906600000001 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305079+0000", 2024-08-30T23:37:50.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9080999999999999e-05 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305126+0000", 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6657999999999999e-05 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:93692 4.4 e350/331 ondisk, result = 0)", 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 143.65962691300001, 2024-08-30T23:37:50.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27807699200000002, 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:37:50.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027057+0000", 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027060+0000", 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.01e-06 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304915+0000", 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27785516100000002 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304921+0000", 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1800000000000001e-06 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305007+0000", 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5783000000000002e-05 2024-08-30T23:37:50.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305015+0000", 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7659999999999998e-06 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305133+0000", 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118472 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.778 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:107608 4.f e354/306 4:fc26249f:::benchmark_data_smithi173_74914_object9366:head v 354'6690, mlcod=354'6687)", 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 129.188869842, 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22329728099999999, 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:37:50.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497814+0000", 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3399999999999998e-07 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497821+0000", 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8900000000000001e-06 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717759+0000", 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21993858999999999 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717767+0000", 2024-08-30T23:37:50.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6000000000000001e-06 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717856+0000", 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8516000000000004e-05 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717893+0000", 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7342999999999998e-05 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721075+0000", 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031817579999999998 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721110+0000", 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5750000000000002e-05 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:168699 4.f 4:f9ac8f5d:::benchmark_data_smithi173_74914_object70457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-08-30T23:37:50.783 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 63.279091676999997, 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23153700799999999, 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168699 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407593+0000", 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.308e-06 2024-08-30T23:37:50.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407598+0000", 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5899999999999998e-06 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407609+0000", 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0975000000000001e-05 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407616+0000", 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0910000000000004e-06 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407664+0000", 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7513999999999999e-05 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407761+0000", 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6817999999999994e-05 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407838+0000", 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6971000000000006e-05 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.411723+0000", 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038854430000000001 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639022+0000", 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.227298575 2024-08-30T23:37:50.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639056+0000", 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4623999999999999e-05 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639128+0000", 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2099000000000002e-05 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:193335 4.f 4:f0f0a8f4:::benchmark_data_smithi173_74914_object95093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 36.058110575999997, 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17749384200000001, 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193335 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.790 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628573+0000", 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.119e-06 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628580+0000", 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1569999999999997e-06 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628592+0000", 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2866e-05 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628599+0000", 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4030000000000004e-06 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628655+0000", 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6322e-05 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628791+0000", 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013567700000000001 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628934+0000", 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014291299999999999 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.633290+0000", 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043565920000000003 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.805971+0000", 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17268071500000001 2024-08-30T23:37:50.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.806014+0000", 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3257999999999998e-05 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.806066+0000", 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1820000000000002e-05 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:196465 4.4 4:242a1b19:::benchmark_data_smithi173_74914_object98223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 32.670572348, 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45786220799999999, 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.796 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196465 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016114+0000", 2024-08-30T23:37:50.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999999e-06 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016121+0000", 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9240000000000001e-06 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016125+0000", 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.704e-06 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016162+0000", 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6528e-05 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016238+0000", 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6246000000000001e-05 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016310+0000", 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2475e-05 2024-08-30T23:37:50.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020389+0000", 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040785120000000003 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020517+0000", 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012816600000000001 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473901+0000", 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45338352300000001 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473940+0000", 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9224000000000003e-05 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473973+0000", 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3034000000000003e-05 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:11167 5.1 5:8495e337:::benchmark_data_smithi173_87639_object11166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:48.303904+0000", 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.3827793000000002, 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15185422400000001, 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 11167 2024-08-30T23:37:50.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.303904+0000", 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.303904+0000", 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.303905+0000", 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1820000000000001e-06 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.303957+0000", 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1987999999999998e-05 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.303972+0000", 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5014000000000001e-05 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.303977+0000", 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5929999999999996e-06 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.304022+0000", 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5055000000000001e-05 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.304114+0000", 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1242000000000005e-05 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.304197+0000", 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3763000000000004e-05 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.304704+0000", 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00050677600000000001 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.305006+0000", 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030172499999999997 2024-08-30T23:37:50.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.455678+0000", 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15067171400000001 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.455708+0000", 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0800000000000003e-05 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:48.455758+0000", 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9372000000000001e-05 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:12194 5.1 5:800234eb:::benchmark_data_smithi173_87639_object12193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:49.214899+0000", 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.471784322, 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15146870400000001, 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:37:50.809 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 12194 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214899+0000", 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214899+0000", 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214900+0000", 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.114e-06 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214926+0000", 2024-08-30T23:37:50.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6625e-05 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214939+0000", 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2731999999999999e-05 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214945+0000", 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.682e-06 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.214984+0000", 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9644999999999997e-05 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.215078+0000", 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3138000000000002e-05 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.215164+0000", 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6867999999999996e-05 2024-08-30T23:37:50.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.215689+0000", 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052412800000000001 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.216114+0000", 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042594300000000002 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:37:50.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.366303+0000", 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15018810499999999 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.366337+0000", 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4453999999999999e-05 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:49.366367+0000", 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.027e-05 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.815 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.816 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:37:50.816 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:37:50.816 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:37:50.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:50.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:50.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:50.963 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:50.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:50.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 15401 15385 106.833 99.4375 0.00336393 0.00934441 2024-08-30T23:37:51.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:51.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:51.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:51.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:51.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:51.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1286/25890 objects degraded (4.967%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:51 smithi106 ceph-mon[16603]: osdmap e402: 12 total, 11 up, 11 in 2024-08-30T23:37:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:51 smithi106 ceph-mon[16603]: osdmap e403: 12 total, 11 up, 11 in 2024-08-30T23:37:51.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:51.477 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1286/25890 objects degraded (4.967%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:51 smithi136 ceph-mon[20207]: osdmap e402: 12 total, 11 up, 11 in 2024-08-30T23:37:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:51 smithi136 ceph-mon[20207]: osdmap e403: 12 total, 11 up, 11 in 2024-08-30T23:37:51.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:37:51.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:51.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1286/25890 objects degraded (4.967%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:51 smithi173 ceph-mon[20554]: osdmap e402: 12 total, 11 up, 11 in 2024-08-30T23:37:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:51 smithi173 ceph-mon[20554]: osdmap e403: 12 total, 11 up, 11 in 2024-08-30T23:37:51.579 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:51.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:51.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:37:51.725 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:51.745 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:51.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:51.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:51.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:51.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:51.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:51.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 17156 17140 107.117 109.688 0.0123668 0.00929642 2024-08-30T23:37:52.026 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:37:52.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:52.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:52.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:52.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:52.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:52 smithi106 ceph-mon[16603]: pgmap v963: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 769 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 109 MiB/s wr, 1.75k op/s; 1286/36891 objects degraded (3.486%); 848/36891 objects misplaced (2.299%) 2024-08-30T23:37:52.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:52.416 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:52 smithi136 ceph-mon[20207]: pgmap v963: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 769 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 109 MiB/s wr, 1.75k op/s; 1286/36891 objects degraded (3.486%); 848/36891 objects misplaced (2.299%) 2024-08-30T23:37:52.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:37:52.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:52.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:52.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:52 smithi173 ceph-mon[20554]: pgmap v963: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 769 MiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 109 MiB/s wr, 1.75k op/s; 1286/36891 objects degraded (3.486%); 848/36891 objects misplaced (2.299%) 2024-08-30T23:37:52.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:52.630 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:52.633 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:37:52.635 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:37:52.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:37:52.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:52.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:52.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:52.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:52.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:52.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:52.997 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 18755 18739 106.463 99.9375 0.00235869 0.00938658 2024-08-30T23:37:53.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:53.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:53.130 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:53.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:53.154 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:53.154 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:53.154 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:53.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:53.188 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:53.215 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:53.215 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:53.243 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:53.243 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:53.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:53.270 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:53.271 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:53.271 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3295738418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:53.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:53.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:53.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:53.428 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3295738418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:53.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:53.500 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:53.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:53.527 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:53.527 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:53.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:53.551 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:53.551 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:53.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3295738418' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:53.579 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:53.580 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:53.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:53.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:53.724 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:53.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:53.778 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:53.779 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:53.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:53.801 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:53.801 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:53.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:53.827 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:53.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:53.917 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:53.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:53.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 20665 20649 107.538 119.375 0.00361245 0.00929084 2024-08-30T23:37:54.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:54.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:54.138 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:37:54.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:54.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:54.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:54 smithi136 ceph-mon[20207]: pgmap v964: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 930 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s; 1816/44628 objects degraded (4.069%); 848/44628 objects misplaced (1.900%) 2024-08-30T23:37:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1482009861' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:54.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:54.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:54.420 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:54 smithi106 ceph-mon[16603]: pgmap v964: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 930 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s; 1816/44628 objects degraded (4.069%); 848/44628 objects misplaced (1.900%) 2024-08-30T23:37:54.421 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1482009861' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:54.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:54.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:37:54.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:54.550 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:54.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:54 smithi173 ceph-mon[20554]: pgmap v964: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 930 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s; 1816/44628 objects degraded (4.069%); 848/44628 objects misplaced (1.900%) 2024-08-30T23:37:54.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1482009861' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:37:54.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:54.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:54.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:54.787 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:37:54.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:54.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:37:54.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:54.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:54.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:54.957 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:54.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 22005 21989 105.708 83.75 0.0111709 0.00945226 2024-08-30T23:37:55.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:55.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:55.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:55.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:55.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:55 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1482009861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:55 smithi136 ceph-mon[20207]: osdmap e404: 12 total, 11 up, 11 in 2024-08-30T23:37:55.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:55.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:55.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:55.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:55.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:55.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:55 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1482009861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:55 smithi173 ceph-mon[20554]: osdmap e404: 12 total, 11 up, 11 in 2024-08-30T23:37:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:55 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:37:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1482009861' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:37:55.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:55 smithi106 ceph-mon[16603]: osdmap e404: 12 total, 11 up, 11 in 2024-08-30T23:37:55.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:55.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:55.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:37:55.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:37:55.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:55.947 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:55.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:55.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 23574 23558 105.161 98.0625 0.00131515 0.00947161 2024-08-30T23:37:56.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:56.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:37:56.140 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:37:56.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:37:56.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:56.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:37:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:37:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:56 smithi136 ceph-mon[20207]: osdmap e405: 12 total, 11 up, 11 in 2024-08-30T23:37:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:56 smithi136 ceph-mon[20207]: pgmap v967: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 2393/59004 objects degraded (4.056%); 1395/59004 objects misplaced (2.364%) 2024-08-30T23:37:56.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1816/44628 objects degraded (4.069%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:56.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:56 smithi136 ceph-mon[20207]: osdmap e406: 12 total, 11 up, 11 in 2024-08-30T23:37:56.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:56.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:56.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:56.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:56.501 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:37:56.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:37:56.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:56 smithi106 ceph-mon[16603]: osdmap e405: 12 total, 11 up, 11 in 2024-08-30T23:37:56.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:56 smithi106 ceph-mon[16603]: pgmap v967: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 2393/59004 objects degraded (4.056%); 1395/59004 objects misplaced (2.364%) 2024-08-30T23:37:56.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1816/44628 objects degraded (4.069%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:56.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:56 smithi106 ceph-mon[16603]: osdmap e406: 12 total, 11 up, 11 in 2024-08-30T23:37:56.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:56.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:37:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:37:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:56 smithi173 ceph-mon[20554]: osdmap e405: 12 total, 11 up, 11 in 2024-08-30T23:37:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:56 smithi173 ceph-mon[20554]: pgmap v967: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.2 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 116 MiB/s wr, 1.86k op/s; 2393/59004 objects degraded (4.056%); 1395/59004 objects misplaced (2.364%) 2024-08-30T23:37:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1816/44628 objects degraded (4.069%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:37:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:56 smithi173 ceph-mon[20554]: osdmap e406: 12 total, 11 up, 11 in 2024-08-30T23:37:56.582 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:56.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:56.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:56.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:56.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:56.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:37:56.887 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:56.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:56.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:56.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:56.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 25151 25135 104.721 98.5625 0.00330639 0.00954313 2024-08-30T23:37:57.031 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:57.031 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:57.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:57.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:57.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:57.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:57.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:57.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:57.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:57.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:57.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:57.614 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:57.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:37:57.736 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:57 smithi136 ceph-mon[20207]: pgmap v969: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.4 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s; 2921/66477 objects degraded (4.394%); 1395/66477 objects misplaced (2.098%) 2024-08-30T23:37:57.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:37:57.751 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:37:57.752 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:37:57.752 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:37:57.752 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:37:57.752 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:37:57.752 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:37:57.754 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:57.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:37:57.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:37:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:57 smithi106 ceph-mon[16603]: pgmap v969: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.4 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s; 2921/66477 objects degraded (4.394%); 1395/66477 objects misplaced (2.098%) 2024-08-30T23:37:57.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.864 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:37:57.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:57.990 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:57.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:57.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 26587 26571 103.785 89.75 0.0202086 0.00962859 2024-08-30T23:37:58.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:57 smithi173 ceph-mon[20554]: pgmap v969: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.4 GiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s; 2921/66477 objects degraded (4.394%); 1395/66477 objects misplaced (2.098%) 2024-08-30T23:37:58.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:37:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.117 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:58.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:37:58.119 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:37:58.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:58.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.247 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:58.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:37:58.371 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:58.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:37:58.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:37:58.499 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:37:58.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:58.625 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:58.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:37:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.757 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:58.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:37:58.792 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:37:58.792 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:37:58.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:37:58.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:58.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:58.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:58.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:58.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 15 28459 28444 104.565 117.062 0.00842819 0.00955633 2024-08-30T23:37:59.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:37:59.092 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:37:59.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:37:59.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:59.160 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:59.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:37:59.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:37:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:37:59.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:37:59.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:37:59.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:37:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:37:59.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:59.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:59.562 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:37:59.562 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:37:59.562 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:37:59.584 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:37:59.584 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:37:59.605 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:59.606 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:37:59.607 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:37:59.609 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:37:59.629 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:37:59.630 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:37:59.631 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:37:59.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:37:59.653 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:37:59.654 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:37:59.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:37:59.675 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:37:59.675 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:37:59.697 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:37:59.698 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:37:59.719 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:37:59.719 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:37:59.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:59 smithi106 ceph-mon[16603]: pgmap v970: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s; 3414/75117 objects degraded (4.545%); 1395/75117 objects misplaced (1.857%) 2024-08-30T23:37:59.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:37:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3145121333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:59.742 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:37:59.742 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:37:59.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:59.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:37:59.766 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:37:59.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:59.789 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:37:59.789 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:37:59.874 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:59 smithi136 ceph-mon[20207]: pgmap v970: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s; 3414/75117 objects degraded (4.545%); 1395/75117 objects misplaced (1.857%) 2024-08-30T23:37:59.875 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:37:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3145121333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:37:59.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:37:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:37:59.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:37:59.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:37:59.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 15 29420 29405 102.093 60.0625 0.00387262 0.00978923 2024-08-30T23:38:00.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:00.050 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:59 smithi173 ceph-mon[20554]: pgmap v970: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 106 MiB/s wr, 1.69k op/s; 3414/75117 objects degraded (4.545%); 1395/75117 objects misplaced (1.857%) 2024-08-30T23:38:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:37:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3145121333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:00.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:29716 5.6 5:630d3fee:::benchmark_data_smithi173_87639_object29715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:00.160714+0000", 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0013316280000000001, 2024-08-30T23:38:00.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001360993, 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 29716 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160714+0000", 2024-08-30T23:38:00.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160714+0000", 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160715+0000", 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1349999999999999e-06 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:00.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160763+0000", 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7834000000000003e-05 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160781+0000", 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8170999999999999e-05 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160790+0000", 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2050000000000002e-06 2024-08-30T23:38:00.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160842+0000", 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2763000000000003e-05 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.160978+0000", 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000136057 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161141+0000", 2024-08-30T23:38:00.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016290799999999999 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:29719 5.6 5.748c4636 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:00.161638+0000", 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000407663, 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00049608300000000003, 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-30T23:38:00.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 29719 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161638+0000", 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161638+0000", 2024-08-30T23:38:00.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161640+0000", 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6670000000000001e-06 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161670+0000", 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0247999999999999e-05 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:00.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161785+0000", 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011480399999999999 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161793+0000", 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1790000000000008e-06 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.26173.0:29717 5.d e406/389 5:bdfe1889:::benchmark_data_smithi173_87639_object29716:head v 406'1820, mlcod=406'1818)", 2024-08-30T23:38:00.171 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:00.161826+0000", 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00021936899999999999, 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000353667, 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161826+0000", 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:00.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161826+0000", 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161827+0000", 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4900000000000005e-07 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.161858+0000", 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0395e-05 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.162072+0000", 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000213852 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.162078+0000", 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2990000000000002e-06 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.162128+0000", 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0000000000000002e-05 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.162147+0000", 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9281999999999999e-05 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:29705 5.6 5:6baf05d5:::benchmark_data_smithi173_87639_object29704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:00.155738+0000", 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063077130000000004, 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0065091339999999998, 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:00.175 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 29705 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155738+0000", 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155738+0000", 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155739+0000", 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4499999999999995e-07 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155797+0000", 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7472999999999997e-05 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155806+0000", 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000008e-06 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155812+0000", 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7610000000000002e-06 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155864+0000", 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1144999999999999e-05 2024-08-30T23:38:00.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.155987+0000", 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012321899999999999 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.156129+0000", 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014260799999999999 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.159163+0000", 2024-08-30T23:38:00.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030332089999999998 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:00.159201+0000", 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8111999999999997e-05 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 4 2024-08-30T23:38:00.180 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:00.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:00.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:00.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:00.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:00.313 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:00.314 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:00.314 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:00.314 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:00.314 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:00.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:00.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:00.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:00.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:00.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:00.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:00.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:00.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:00.855 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:00 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:38:00.856 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:00 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:38:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:00 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:38:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:00 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:38:00.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:00.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:00.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:00.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 30933 30917 101.693 94.5 0.00486533 0.00980265 2024-08-30T23:38:01.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:01.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:00 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:38:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:00 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:38:01.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:01.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:01.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:38:01.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:01.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:01.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:01.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:01.526 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:01.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:01.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:01.597 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:38:01.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:01.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:01.778 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:01.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:01.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3414/75117 objects degraded (4.545%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/10941676' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:01 smithi106 ceph-mon[16603]: pgmap v971: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.7 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s; 3414/84741 objects degraded (4.029%); 1838/84741 objects misplaced (2.169%) 2024-08-30T23:38:01.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:01.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:01.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:01.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:38:01.999291+0000 min lat: 0.00108473 max lat: 0.264809 avg lat: 0.0099372 2024-08-30T23:38:01.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:38:01.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 15 32198 32183 100.564 79.125 0.00207168 0.0099372 2024-08-30T23:38:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3414/75117 objects degraded (4.545%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/10941676' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:01 smithi136 ceph-mon[20207]: pgmap v971: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.7 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s; 3414/84741 objects degraded (4.029%); 1838/84741 objects misplaced (2.169%) 2024-08-30T23:38:02.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3414/75117 objects degraded (4.545%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/10941676' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:01 smithi173 ceph-mon[20554]: pgmap v971: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.7 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 95 MiB/s wr, 1.52k op/s; 3414/84741 objects degraded (4.029%); 1838/84741 objects misplaced (2.169%) 2024-08-30T23:38:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:02.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:02.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:02.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:02.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:02.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:02.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:02.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:02.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:02.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:02.628 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:02.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:02.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:02.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:02.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:02 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/10941676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:02 smithi106 ceph-mon[16603]: osdmap e407: 12 total, 11 up, 11 in 2024-08-30T23:38:02.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:02.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:02.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:02.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:02 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:02.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/10941676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:02.944 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:02 smithi136 ceph-mon[20207]: osdmap e407: 12 total, 11 up, 11 in 2024-08-30T23:38:02.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:02.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 33608 33592 99.9683 88.0625 0.00358582 0.00999527 2024-08-30T23:38:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:03.010 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:03.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:38:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:03.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:02 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/10941676' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:02 smithi173 ceph-mon[20554]: osdmap e407: 12 total, 11 up, 11 in 2024-08-30T23:38:03.082 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:03.082 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:38:03.082 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:38:03.082 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:38:03.083 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:03.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:03.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:03.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:03.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:03.286 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:03.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:03.401 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:03.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:03.528 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:03.630 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:03.658 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:03.681 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:03 smithi106 ceph-mon[16603]: pgmap v973: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s; 3878/91257 objects degraded (4.250%); 1838/91257 objects misplaced (2.014%) 2024-08-30T23:38:03.750 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:03 smithi136 ceph-mon[20207]: pgmap v973: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s; 3878/91257 objects degraded (4.250%); 1838/91257 objects misplaced (2.014%) 2024-08-30T23:38:03.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:03.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:38:03.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:38:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:03.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:03.816 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:03.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:03.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:03.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:03.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:03.817 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:03.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:03.909 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:03.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:03.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 34906 34890 99.1114 81.125 0.0094043 0.0100838 2024-08-30T23:38:04.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:04.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:03 smithi173 ceph-mon[20554]: pgmap v973: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 1.9 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 84 MiB/s wr, 1.34k op/s; 3878/91257 objects degraded (4.250%); 1838/91257 objects misplaced (2.014%) 2024-08-30T23:38:04.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:04.153 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:04.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:04.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:04.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:04.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:04.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:04.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:38:04.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:04.542 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:04.617 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:04.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-30T23:38:04.636 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:04.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:04.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:04.731 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:04.731 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:38:04.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:38:04.731 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:38:04.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.731 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155144 2.b 2:dec0e07e:::benchmark_data_smithi173_47195_object58136:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.00561630799996, 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24062692899999999, 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155144 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721257+0000", 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721258+0000", 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.688e-06 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721264+0000", 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5060000000000003e-06 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721277+0000", 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3345e-05 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721295+0000", 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7598e-05 2024-08-30T23:38:04.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721850+0000", 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055514699999999998 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721861+0000", 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1009000000000001e-05 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.932731+0000", 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.210870364 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.932895+0000", 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016375999999999999 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.933065+0000", 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000170104 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943014+0000", 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099494240000000001 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943076+0000", 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1239000000000001e-05 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943195+0000", 2024-08-30T23:38:04.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118923 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.960869+0000", 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017674390000000002 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.960910+0000", 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1467999999999997e-05 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.961883+0000", 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00097296399999999999 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155158 2.b 2:dcb1d3fb:::benchmark_data_smithi173_47195_object58150:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.00555986999996, 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24126403199999999, 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.739 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155158 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721313+0000", 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4999999999999998e-07 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721316+0000", 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1900000000000002e-06 2024-08-30T23:38:04.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721321+0000", 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4450000000000004e-06 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721324+0000", 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721883+0000", 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055905600000000005 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721891+0000", 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1509999999999998e-06 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937082+0000", 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.215191454 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937182+0000", 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9567000000000004e-05 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.937273+0000", 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1305999999999996e-05 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943145+0000", 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058718049999999999 2024-08-30T23:38:04.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948639+0000", 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054946880000000002 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948721+0000", 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1936999999999998e-05 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.961943+0000", 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013221568 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.962004+0000", 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0990999999999999e-05 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.962577+0000", 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057307400000000002 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155229 2.b 2:ddba2b2a:::benchmark_data_smithi173_47195_object58221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.00554275699994, 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.248255843, 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155229 2024-08-30T23:38:04.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:38:04.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721330+0000", 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2000000000000001e-07 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721332+0000", 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3260000000000001e-06 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721336+0000", 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0759999999999996e-06 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721339+0000", 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.569e-06 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721916+0000", 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057626300000000004 2024-08-30T23:38:04.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721923+0000", 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5530000000000004e-06 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940166+0000", 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21824272 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940264+0000", 2024-08-30T23:38:04.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8392000000000003e-05 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.940350+0000", 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5668999999999996e-05 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.945936+0000", 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055864249999999999 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948776+0000", 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028393670000000002 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957244+0000", 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084679639999999997 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.968623+0000", 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011379146999999999 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.968683+0000", 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0170999999999997e-05 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969586+0000", 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00090298099999999997 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155284 2.b 2:d661bc27:::benchmark_data_smithi173_47195_object58276:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.00553020400002, 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24904695199999999, 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155284 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721343+0000", 2024-08-30T23:38:04.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0200000000000001e-07 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721344+0000", 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.519e-06 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721347+0000", 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.118e-06 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721351+0000", 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5429999999999998e-06 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721957+0000", 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060615599999999995 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721963+0000", 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3619999999999997e-06 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943241+0000", 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221278947 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943333+0000", 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1094000000000006e-05 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.943419+0000", 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6588999999999994e-05 2024-08-30T23:38:04.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948733+0000", 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053133649999999996 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951543+0000", 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002810819 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957285+0000", 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057415169999999998 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969633+0000", 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012348181999999999 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.969670+0000", 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6854000000000003e-05 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970390+0000", 2024-08-30T23:38:04.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071968499999999996 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155286 2.b 2:da5a53b4:::benchmark_data_smithi173_47195_object58278:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.00551818600002, 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24979741499999999, 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155286 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:38:04.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721355+0000", 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1199999999999999e-07 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721356+0000", 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2079999999999999e-06 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721360+0000", 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8750000000000002e-06 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721363+0000", 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7389999999999999e-06 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721987+0000", 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00062452099999999995 2024-08-30T23:38:04.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721998+0000", 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277000000000001e-05 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.945970+0000", 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.223971225 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.946066+0000", 2024-08-30T23:38:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6781999999999996e-05 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.946148+0000", 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1522999999999999e-05 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951509+0000", 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053612979999999996 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.954537+0000", 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030273959999999999 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957323+0000", 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002785839 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970435+0000", 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013112342000000001 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.970469+0000", 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4140000000000002e-05 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.971152+0000", 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068303799999999996 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.770 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:155298 2.b 2:dd53a97e:::benchmark_data_smithi173_47195_object58290:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e183)", 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 557.00550713300004, 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25834940200000001, 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155298 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721366+0000", 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2000000000000001e-07 2024-08-30T23:38:04.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721367+0000", 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0950000000000001e-06 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721371+0000", 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.783e-06 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.721373+0000", 2024-08-30T23:38:04.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6280000000000001e-06 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.722015+0000", 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064132200000000005 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.722020+0000", 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.028e-06 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948797+0000", 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22677699400000001 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948892+0000", 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5272000000000005e-05 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.948978+0000", 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6365000000000004e-05 2024-08-30T23:38:04.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.951655+0000", 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026768669999999999 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.954636+0000", 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029808159999999998 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.957360+0000", 2024-08-30T23:38:04.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002723821 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.978592+0000", 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021232183000000002 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.978633+0000", 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0460000000000002e-05 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:28:47.979715+0000", 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001082548 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193943 2.b 2:d4b90c03:::benchmark_data_smithi173_47195_object96935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 502.01875084400001, 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42676301799999999, 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.778 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193943 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708122+0000", 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708123+0000", 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.09e-07 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708129+0000", 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5119999999999997e-06 2024-08-30T23:38:04.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708141+0000", 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243999999999999e-05 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.708147+0000", 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1920000000000002e-06 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709291+0000", 2024-08-30T23:38:04.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011439899999999999 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709428+0000", 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013711600000000001 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,8,11", 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.709522+0000", 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3832000000000006e-05 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.714473+0000", 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049515779999999999 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.714764+0000", 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029112000000000002 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.718006+0000", 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032420859999999999 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134491+0000", 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41648495699999999 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134534+0000", 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2568000000000003e-05 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134885+0000", 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00035101400000000002 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.784 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.24460.0:193943 2.b e200/182 ondisk, result = 0)", 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 502.01292867799998, 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42094626899999998, 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713944+0000", 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713945+0000", 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9500000000000003e-07 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.713948+0000", 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134314+0000", 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42036561300000003 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134335+0000", 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1803000000000001e-05 2024-08-30T23:38:04.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134478+0000", 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014253700000000001 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134487+0000", 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5469999999999997e-06 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.134890+0000", 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040388000000000002 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(2.b e200,182 2:d6a96fd5:::benchmark_data_smithi173_47195_object12191:head,153'6821)", 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 501.866632514, 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66937705700000005, 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.789 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860240+0000", 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860247+0000", 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3439999999999997e-06 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.860257+0000", 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0582e-05 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529079+0000", 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66882161500000004 2024-08-30T23:38:04.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529086+0000", 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8430000000000004e-06 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529174+0000", 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8320999999999997e-05 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.529617+0000", 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044335199999999999 2024-08-30T23:38:04.792 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:22811 3.c 3:362f756c:::benchmark_data_smithi173_60780_object22810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 458.7218264, 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32742973199999997, 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22811 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005046+0000", 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005047+0000", 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8500000000000001e-07 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005105+0000", 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7797999999999998e-05 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005114+0000", 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0319999999999992e-06 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005119+0000", 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5820000000000003e-06 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005151+0000", 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2404999999999999e-05 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005230+0000", 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9212e-05 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005314+0000", 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3540000000000003e-05 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005956+0000", 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064215699999999997 2024-08-30T23:38:04.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.006179+0000", 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022317300000000001 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332412+0000", 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.326233314 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332449+0000", 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6671999999999997e-05 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332476+0000", 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7161999999999998e-05 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:198842 3.5 3:ab155887:::benchmark_data_smithi173_60780_object95388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:33.844424+0000", 2024-08-30T23:38:04.799 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 270.882448372, 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23186731199999999, 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 198842 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844424+0000", 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844424+0000", 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844426+0000", 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3e-06 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844431+0000", 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844444+0000", 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3559e-05 2024-08-30T23:38:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.844452+0000", 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3259999999999998e-06 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.847658+0000", 2024-08-30T23:38:04.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032057539999999999 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.847800+0000", 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142497 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.847942+0000", 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141485 2024-08-30T23:38:04.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.851731+0000", 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037893169999999999 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:33.939608+0000", 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.087877409000000004 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076162+0000", 2024-08-30T23:38:04.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13655384300000001 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076242+0000", 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0294000000000001e-05 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:34.076292+0000", 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9394000000000002e-05 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206885 3.5 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.040796+0000", 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 260.68607648599999, 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.235763583, 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206885 2024-08-30T23:38:04.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040796+0000", 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040796+0000", 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040798+0000", 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.536e-06 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040803+0000", 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9200000000000003e-06 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040816+0000", 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3443e-05 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040823+0000", 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2400000000000001e-06 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.040886+0000", 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2333000000000001e-05 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.041010+0000", 2024-08-30T23:38:04.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124573 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.041133+0000", 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012286299999999999 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.045228+0000", 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040943810000000002 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.045331+0000", 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103051 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.276401+0000", 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.231070047 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.276493+0000", 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1821000000000001e-05 2024-08-30T23:38:04.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.276560+0000", 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7374999999999998e-05 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206900 3.5 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.062814+0000", 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 260.66405914299997, 2024-08-30T23:38:04.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36188622999999998, 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206900 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062814+0000", 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062814+0000", 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062815+0000", 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0640000000000001e-06 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062818+0000", 2024-08-30T23:38:04.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2610000000000002e-06 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062828+0000", 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7159999999999998e-06 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062833+0000", 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0669999999999999e-06 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.062883+0000", 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9716000000000003e-05 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.063029+0000", 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146766 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.063159+0000", 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012947100000000001 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.067813+0000", 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046544960000000002 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.068092+0000", 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027919499999999999 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.424625+0000", 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.356532126 2024-08-30T23:38:04.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.424667+0000", 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2113000000000002e-05 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.424700+0000", 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3238999999999998e-05 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:31015 4.2 e316/306 4:4a9291e8:::benchmark_data_smithi173_74914_object31014:head v 316'1935, mlcod=316'1932)", 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:20.657893+0000", 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 224.06897981899999, 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27371718299999998, 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657893+0000", 2024-08-30T23:38:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657893+0000", 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657894+0000", 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1900000000000002e-07 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.657996+0000", 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102334 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930224+0000", 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27222813099999998 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930248+0000", 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3320999999999999e-05 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930347+0000", 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9932999999999993e-05 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.930371+0000", 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3311000000000001e-05 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.931568+0000", 2024-08-30T23:38:04.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001196927 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.931610+0000", 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2506999999999998e-05 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:98222 4.1 e353/306 4:8f28ee96:::benchmark_data_smithi173_74914_object98221:head v 353'6230, mlcod=353'6225)", 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:31.440878+0000", 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.285994655, 2024-08-30T23:38:04.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25389183399999998, 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.440878+0000", 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.440878+0000", 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.440879+0000", 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8100000000000001e-07 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.440964+0000", 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4691999999999995e-05 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.691949+0000", 2024-08-30T23:38:04.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25098489400000001 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.691964+0000", 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5588999999999999e-05 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.692135+0000", 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017075400000000001 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.692180+0000", 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4709000000000002e-05 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.694736+0000", 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025558970000000001 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:31.694770+0000", 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4317999999999999e-05 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:15236 5.4 5:275cc9e1:::benchmark_data_smithi173_87639_object15235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:50.728335+0000", 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.998538124, 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20055541700000001, 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:04.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 15236 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728335+0000", 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728335+0000", 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728336+0000", 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2500000000000004e-07 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728409+0000", 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3379000000000003e-05 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728419+0000", 2024-08-30T23:38:04.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0362e-05 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728426+0000", 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0180000000000004e-06 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728467+0000", 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0955999999999999e-05 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728579+0000", 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111391 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,8", 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.728721+0000", 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014252000000000001 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.729384+0000", 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066283499999999997 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928431+0000", 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19904660699999999 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928792+0000", 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000360949 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928850+0000", 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8835999999999997e-05 2024-08-30T23:38:04.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928890+0000", 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9739e-05 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:15236 5.4 e403/394 ondisk, result = 0)", 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:50.729643+0000", 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.997230065, 2024-08-30T23:38:04.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.199255141, 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.729643+0000", 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.729643+0000", 2024-08-30T23:38:04.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.729643+0000", 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8400000000000005e-07 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.729646+0000", 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-06 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928729+0000", 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.199082917 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928735+0000", 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3789999999999997e-06 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928779+0000", 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3915000000000002e-05 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928786+0000", 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9209999999999992e-06 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.928898+0000", 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111455 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:29003 5.f e406/389 ondisk, result = 0)", 2024-08-30T23:38:04.840 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:59.478271+0000", 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.2486017230000002, 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18402199999999999, 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.478271+0000", 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.478271+0000", 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.478272+0000", 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0900000000000002e-07 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.478279+0000", 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6240000000000002e-06 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.662094+0000", 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183814489 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.662100+0000", 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.652e-06 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.662152+0000", 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1385999999999999e-05 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.662160+0000", 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9359999999999999e-06 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.662293+0000", 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000133404 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.844 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:31255 5.7 5:e6f05d18:::benchmark_data_smithi173_87639_object31254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:01.243980+0000", 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 3.4828931980000002, 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.205075126, 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 31255 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.243980+0000", 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.243980+0000", 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.243980+0000", 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5799999999999998e-07 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244042+0000", 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1462000000000002e-05 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244052+0000", 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0558e-05 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244057+0000", 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8019999999999999e-06 2024-08-30T23:38:04.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244097+0000", 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9288000000000001e-05 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244178+0000", 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1607999999999995e-05 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9", 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244259+0000", 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.119e-05 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.244966+0000", 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00070645400000000002 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.448899+0000", 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203933271 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.448964+0000", 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4807999999999993e-05 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:01.449055+0000", 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0927000000000005e-05 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.850 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:32219 5.6 5:66c91781:::benchmark_data_smithi173_87639_object32218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:02.011959+0000", 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 2.7149140919999999, 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17960818200000001, 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:04.851 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32219 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.011959+0000", 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.011959+0000", 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.011959+0000", 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8100000000000003e-07 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.012044+0000", 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4685000000000007e-05 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.012054+0000", 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0482999999999999e-05 2024-08-30T23:38:04.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.012059+0000", 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4020000000000004e-06 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.012100+0000", 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0927000000000003e-05 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:04.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.012191+0000", 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1334000000000003e-05 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.012279+0000", 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8103999999999995e-05 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.013276+0000", 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00099658800000000007 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.013361+0000", 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5680999999999995e-05 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.191505+0000", 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17814371300000001 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.191541+0000", 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5868999999999998e-05 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:04.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:02.191567+0000", 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5914999999999998e-05 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:04.857 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:04.879 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:04.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:04.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:04.998 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 36071 36055 97.9678 72.8125 0.0663556 0.0101848 2024-08-30T23:38:05.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:05.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:05.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:05.192 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:05.196 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:05.196 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:05.196 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:05.230 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:05.231 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:05.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:05.259 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:05.260 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:05.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:05.286 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:05.286 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:05.314 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:05.314 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:05.314 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2127994428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:05.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:05.339 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:05.364 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:05.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:05.387 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:05.388 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:05.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:05.409 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:05.409 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:05.430 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:05.430 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:05.451 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:05.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:05.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:05.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:05.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2127994428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2127994428' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:05.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:05.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:05.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:38:05.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:05.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:05.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:05.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:05.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:05.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:05.931 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:05.931 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:05.931 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:05.931 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:05.932 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:05.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 37305 37289 97.0991 77.125 0.00210774 0.0102898 2024-08-30T23:38:06.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:06.001 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:06.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:06.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:06.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:06.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:06.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:06 smithi106 ceph-mon[16603]: pgmap v974: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.1 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s; 4320/102639 objects degraded (4.209%); 2277/102639 objects misplaced (2.218%) 2024-08-30T23:38:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3878/91257 objects degraded (4.250%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:06 smithi106 ceph-mon[16603]: osdmap e408: 12 total, 11 up, 11 in 2024-08-30T23:38:06.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:06.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:06.454 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:06.467 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:06 smithi136 ceph-mon[20207]: pgmap v974: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.1 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s; 4320/102639 objects degraded (4.209%); 2277/102639 objects misplaced (2.218%) 2024-08-30T23:38:06.468 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3878/91257 objects degraded (4.250%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:06.468 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:06 smithi136 ceph-mon[20207]: osdmap e408: 12 total, 11 up, 11 in 2024-08-30T23:38:06.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:06.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:06.568 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:06.568 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:06 smithi173 ceph-mon[20554]: pgmap v974: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.1 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 93 MiB/s wr, 1.48k op/s; 4320/102639 objects degraded (4.209%); 2277/102639 objects misplaced (2.218%) 2024-08-30T23:38:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3878/91257 objects degraded (4.250%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:06 smithi173 ceph-mon[20554]: osdmap e408: 12 total, 11 up, 11 in 2024-08-30T23:38:06.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:06.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:06.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:06.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:06.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:06.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:06.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:06.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 15 38796 38781 96.9448 93.25 0.00556752 0.0103103 2024-08-30T23:38:07.031 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:07.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:07.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:07.069 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:07.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-30T23:38:07.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:07.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:07.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:07.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:38:07.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:07.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:07.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:07.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:07.575 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:07.578 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:07.660 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:07.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:07.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:07.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:07.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:07 smithi106 ceph-mon[16603]: pgmap v976: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 4727/110031 objects degraded (4.296%); 2277/110031 objects misplaced (2.069%) 2024-08-30T23:38:07.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:07.951 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:07 smithi136 ceph-mon[20207]: pgmap v976: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 4727/110031 objects degraded (4.296%); 2277/110031 objects misplaced (2.069%) 2024-08-30T23:38:07.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:07.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:07.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 40021 40005 96.1582 76.5 0.0023128 0.0103589 2024-08-30T23:38:08.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:07 smithi173 ceph-mon[20554]: pgmap v976: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.2 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 91 MiB/s wr, 1.45k op/s; 4727/110031 objects degraded (4.296%); 2277/110031 objects misplaced (2.069%) 2024-08-30T23:38:08.100 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:08.113 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:08.114 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:08.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:08.226 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:08.226 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:08.231 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:08.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:08.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:38:08.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:08.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:08.608 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:38:08.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:08.660 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:08.693 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:08.694 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4232130336' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:08.694 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:08.694 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:08.694 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:08 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:38:08.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:08.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:08.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:08.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4232130336' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:08.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:08.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:08.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:08 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:38:08.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:08.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4232130336' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:08.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:08.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:08.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:08 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:38:08.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:08.921 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:08.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-30T23:38:08.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:08.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:08.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:38:08.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 41613 41597 96.2817 99.5 0.00874624 0.0103793 2024-08-30T23:38:09.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:09.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:09.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:09.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.189 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:09.189 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:09.285 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:09.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:09.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:09.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:09.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:09.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:09.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:09.656 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:09.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:09.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:09.736 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:09 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:09.736 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4232130336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:09.736 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:09 smithi136 ceph-mon[20207]: osdmap e409: 12 total, 11 up, 11 in 2024-08-30T23:38:09.737 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:09 smithi136 ceph-mon[20207]: pgmap v978: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 5137/115824 objects degraded (4.435%); 2277/115824 objects misplaced (1.966%) 2024-08-30T23:38:09.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:09.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:09 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4232130336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:09 smithi106 ceph-mon[16603]: osdmap e409: 12 total, 11 up, 11 in 2024-08-30T23:38:09.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:09 smithi106 ceph-mon[16603]: pgmap v978: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 5137/115824 objects degraded (4.435%); 2277/115824 objects misplaced (1.966%) 2024-08-30T23:38:09.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:09.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:09.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:09.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 42232 42216 94.2246 38.6875 0.00212528 0.0104375 2024-08-30T23:38:10.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:09 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4232130336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:09 smithi173 ceph-mon[20554]: osdmap e409: 12 total, 11 up, 11 in 2024-08-30T23:38:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:09 smithi173 ceph-mon[20554]: pgmap v978: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s; 5137/115824 objects degraded (4.435%); 2277/115824 objects misplaced (1.966%) 2024-08-30T23:38:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:10.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:10.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:10.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:10.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:10.315 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:10.315 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:10.315 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:10.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:10.348 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:10.348 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:10.375 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:10.375 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:10.406 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:10.406 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:10.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:10.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:10.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:10.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:10.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:10.469 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:10.498 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:10.499 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:10.525 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:10.525 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:10.553 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:10.554 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:10.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:10.559 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:10.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:10.585 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:10.586 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:10.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:10.607 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:10.607 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:10.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:10.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:10.789 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/273636081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/273636081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:10.798 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:38:10.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193992 2.7 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 508.02163920700002, 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60785984599999998, 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:38:10.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193992 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:38:10.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775252+0000", 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.528e-06 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775258+0000", 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.485e-06 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775273+0000", 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5355000000000001e-05 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775281+0000", 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5140000000000004e-06 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775342+0000", 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1334000000000006e-05 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775448+0000", 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010587 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775514+0000", 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5747000000000006e-05 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.779105+0000", 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035911110000000001 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.780025+0000", 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092040100000000005 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383045+0000", 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60302017900000005 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383083+0000", 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7670999999999999e-05 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383111+0000", 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7651e-05 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:22811 3.c e215/206 3:362f756c:::benchmark_data_smithi173_60780_object22810:head v 215'1421, mlcod=215'1418)", 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 464.79150207700002, 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32690804400000001, 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.807 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005389+0000", 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005420+0000", 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1340000000000001e-05 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331407+0000", 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32598661899999998 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331414+0000", 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7409999999999992e-06 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331466+0000", 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1319000000000001e-05 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331490+0000", 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4544000000000001e-05 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332257+0000", 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076687200000000004 2024-08-30T23:38:10.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332296+0000", 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8828000000000002e-05 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206871 3.a 3:5d60dc93:::benchmark_data_smithi173_60780_object103417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:38:10.812 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 266.77634156200003, 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45068615499999998, 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206871 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020550+0000", 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.559e-06 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020557+0000", 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5810000000000002e-06 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020573+0000", 2024-08-30T23:38:10.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6767999999999999e-05 2024-08-30T23:38:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020581+0000", 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020645+0000", 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3559000000000006e-05 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020750+0000", 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010555100000000001 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020853+0000", 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103207 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.025592+0000", 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047385489999999999 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.053826+0000", 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028234459999999999 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471165+0000", 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41733862300000002 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471204+0000", 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8769000000000001e-05 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471235+0000", 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0818000000000002e-05 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24732 4.a 4:5449864d:::benchmark_data_smithi173_74914_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:38:10.820 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 234.287996077, 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24409322, 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24732 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9800000000000003e-07 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508987+0000", 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2602999999999994e-05 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509002+0000", 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509009+0000", 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509048+0000", 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9002999999999999e-05 2024-08-30T23:38:10.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509127+0000", 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9573000000000005e-05 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509220+0000", 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2218000000000004e-05 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.510205+0000", 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098522999999999992 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.550346+0000", 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040140995999999998 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752876+0000", 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20253054600000001 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752927+0000", 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0686999999999998e-05 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752987+0000", 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9998999999999999e-05 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24732 4.a 4:5449864d:::benchmark_data_smithi173_74914_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 234.282174575, 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23822328400000001, 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for ondisk", 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24732 2024-08-30T23:38:10.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514716+0000", 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4300000000000003e-07 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514742+0000", 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6389000000000001e-05 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514749+0000", 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0879999999999997e-06 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514755+0000", 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999998e-06 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514806+0000", 2024-08-30T23:38:10.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1598999999999998e-05 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for ondisk", 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514835+0000", 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8839999999999998e-05 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752939+0000", 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23810340499999999 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24761 4.5 4:a203c2b9:::benchmark_data_smithi173_74914_object24760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.541700+0000", 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 234.25518981100001, 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21552817699999999, 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.833 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24761 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541700+0000", 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541700+0000", 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541703+0000", 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6259999999999999e-06 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541748+0000", 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4728999999999998e-05 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541777+0000", 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9433000000000001e-05 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541796+0000", 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9219999999999999e-05 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.541908+0000", 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111381 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542074+0000", 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016683400000000001 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,10", 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.542164+0000", 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9829999999999999e-05 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.543031+0000", 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086726299999999995 2024-08-30T23:38:10.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752709+0000", 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20967720500000001 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.757122+0000", 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044134389999999999 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.757184+0000", 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1823999999999996e-05 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.757228+0000", 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4393e-05 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.839 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49429 4.9 4:90c7f456:::benchmark_data_smithi173_74914_object49428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.629894+0000", 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.166995851, 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21828159899999999, 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49429 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629894+0000", 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629894+0000", 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629895+0000", 2024-08-30T23:38:10.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2600000000000002e-07 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629925+0000", 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0862999999999999e-05 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629932+0000", 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7209999999999997e-06 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629937+0000", 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6210000000000003e-06 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630472+0000", 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053525800000000004 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630479+0000", 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6769999999999998e-06 2024-08-30T23:38:10.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.840168+0000", 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20968874400000001 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.840273+0000", 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105151 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.840407+0000", 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013429499999999999 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843607+0000", 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031995579999999999 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843793+0000", 2024-08-30T23:38:10.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018662400000000001 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848052+0000", 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042584820000000001 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848110+0000", 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8057999999999999e-05 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848176+0000", 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6020999999999997e-05 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49435 4.9 4:94174f8f:::benchmark_data_smithi173_74914_object49434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.16694611099999, 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21848274200000001, 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.847 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49435 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629944+0000", 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9000000000000002e-07 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629978+0000", 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3299000000000001e-05 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629985+0000", 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7209999999999994e-06 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.629995+0000", 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630501+0000", 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050542899999999995 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630507+0000", 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5889999999999999e-06 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.841699+0000", 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21119121399999999 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.841812+0000", 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113396 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.841980+0000", 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016822300000000001 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843801+0000", 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018204689999999999 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845139+0000", 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013380740000000001 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848343+0000", 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032047180000000001 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848374+0000", 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0738999999999998e-05 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848427+0000", 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2364000000000002e-05 2024-08-30T23:38:10.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49437 4.9 4:992ba69c:::benchmark_data_smithi173_74914_object49436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.16688718200001, 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21855950099999999, 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49437 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:38:10.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630003+0000", 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8100000000000003e-07 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630037+0000", 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3992000000000003e-05 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630045+0000", 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630050+0000", 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7219999999999999e-06 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630529+0000", 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000479073 2024-08-30T23:38:10.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630535+0000", 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8880000000000002e-06 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.843987+0000", 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21345172600000001 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.844072+0000", 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5761999999999996e-05 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.844162+0000", 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9186999999999998e-05 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848231+0000", 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040693170000000002 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848246+0000", 2024-08-30T23:38:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5461e-05 2024-08-30T23:38:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848480+0000", 2024-08-30T23:38:10.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023351099999999999 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848511+0000", 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0855000000000002e-05 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848562+0000", 2024-08-30T23:38:10.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1625000000000001e-05 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49440 4.9 4:9de739e2:::benchmark_data_smithi173_74914_object49439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.166832946, 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22086719499999999, 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.864 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49440 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5999999999999999e-07 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630090+0000", 2024-08-30T23:38:10.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2196000000000001e-05 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630096+0000", 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3160000000000002e-06 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630102+0000", 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9050000000000002e-06 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630553+0000", 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045081400000000001 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630557+0000", 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6979999999999997e-06 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845169+0000", 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21461201499999999 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845239+0000", 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0062999999999996e-05 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845330+0000", 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0912e-05 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848252+0000", 2024-08-30T23:38:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002922043 2024-08-30T23:38:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848274+0000", 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1472999999999999e-05 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850752+0000", 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024776540000000001 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850813+0000", 2024-08-30T23:38:10.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1593999999999993e-05 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850924+0000", 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011105200000000001 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49445 4.9 4:915080b3:::benchmark_data_smithi173_74914_object49444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:38:10.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.166781474, 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22553746099999999, 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49445 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630109+0000", 2024-08-30T23:38:10.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630137+0000", 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8206999999999999e-05 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630143+0000", 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0329999999999997e-06 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630148+0000", 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4769999999999997e-06 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630574+0000", 2024-08-30T23:38:10.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042622100000000001 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630579+0000", 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9989999999999999e-06 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846364+0000", 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21578499100000001 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846489+0000", 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012525599999999999 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846624+0000", 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013508199999999999 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848317+0000", 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016928519999999999 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848441+0000", 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012401 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855479+0000", 2024-08-30T23:38:10.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070380499999999997 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855541+0000", 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1371000000000006e-05 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855646+0000", 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010543300000000001 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.880 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62188 4.9 4:9d0786de:::benchmark_data_smithi173_74914_object62187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.559263+0000", 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.23762701699999, 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21637304099999999, 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62188 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559263+0000", 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559263+0000", 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559265+0000", 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5689999999999999e-06 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559316+0000", 2024-08-30T23:38:10.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1032999999999997e-05 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559328+0000", 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2824e-05 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559333+0000", 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9529999999999998e-06 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559422+0000", 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8231000000000003e-05 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559545+0000", 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123783 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.559700+0000", 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015455199999999999 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.561474+0000", 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017739710000000001 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.562339+0000", 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086553400000000005 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775566+0000", 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21322675199999999 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775605+0000", 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8424999999999999e-05 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775636+0000", 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1414000000000001e-05 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62189 4.9 4:952cc0c3:::benchmark_data_smithi173_74914_object62188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.23376240499999, 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.475062977, 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62189 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563190+0000", 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1838e-05 2024-08-30T23:38:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563201+0000", 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0794e-05 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563206+0000", 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.429e-06 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563251+0000", 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4363999999999997e-05 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563339+0000", 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8675000000000001e-05 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563428+0000", 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8559000000000004e-05 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.564850+0000", 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014217069999999999 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.565634+0000", 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078473200000000001 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038076+0000", 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.47244159499999999 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038142+0000", 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6005000000000002e-05 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038191+0000", 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8541000000000002e-05 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62196 4.9 4:91ca1c4e:::benchmark_data_smithi173_74914_object62195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.02372494599999, 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310692415, 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62196 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.896 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773167+0000", 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5409999999999999e-06 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773206+0000", 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9888000000000002e-05 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773222+0000", 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5690000000000001e-05 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773228+0000", 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9819999999999996e-06 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773279+0000", 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0417000000000002e-05 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773374+0000", 2024-08-30T23:38:10.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5396000000000006e-05 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773464+0000", 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0315000000000005e-05 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775336+0000", 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018712749999999999 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.779671+0000", 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043352089999999996 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083790+0000", 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.304119056 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083827+0000", 2024-08-30T23:38:10.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6863999999999998e-05 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083857+0000", 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0781999999999998e-05 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.902 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:62189 4.9 e337/327 ondisk, result = 0)", 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 193.021408726, 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26272571900000002, 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:38:10.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775482+0000", 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.64e-07 2024-08-30T23:38:10.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775487+0000", 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.037983+0000", 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26249586200000002 2024-08-30T23:38:10.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.037988+0000", 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.561e-06 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038060+0000", 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2241000000000002e-05 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038070+0000", 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038207+0000", 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013733899999999999 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:177604 4.9 4:92803af5:::benchmark_data_smithi173_74914_object79362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:57.352010+0000", 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.444880029000004, 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.203482317, 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 177604 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352010+0000", 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352010+0000", 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352011+0000", 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.091e-06 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352016+0000", 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.673e-06 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352026+0000", 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0416e-05 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352034+0000", 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5290000000000003e-06 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352088+0000", 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3971999999999998e-05 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352202+0000", 2024-08-30T23:38:10.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011469200000000001 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.352291+0000", 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8571000000000003e-05 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.357246+0000", 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049549939999999999 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.372754+0000", 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015508251000000001 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555428+0000", 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.182674118 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555463+0000", 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4316000000000001e-05 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555492+0000", 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9694000000000001e-05 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:177604 4.9 e376/327 ondisk, result = 0)", 2024-08-30T23:38:10.915 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:57.357707+0000", 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.439183243000002, 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19779346, 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.357707+0000", 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.357707+0000", 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.357707+0000", 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.357710+0000", 2024-08-30T23:38:10.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.565e-06 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555338+0000", 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19762868 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555364+0000", 2024-08-30T23:38:10.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5395999999999999e-05 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555417+0000", 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3488e-05 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555423+0000", 2024-08-30T23:38:10.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.161e-06 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:57.555500+0000", 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6743999999999996e-05 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:15243 5.2 5:4bef17d2:::benchmark_data_smithi173_87639_object15242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:50.731363+0000", 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 20.065527195000001, 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20142660700000001, 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:10.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 15243 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731363+0000", 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731363+0000", 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731364+0000", 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.23e-07 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731421+0000", 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7114000000000002e-05 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731430+0000", 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6709999999999994e-06 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731435+0000", 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8879999999999996e-06 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731475+0000", 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9413999999999999e-05 2024-08-30T23:38:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731555+0000", 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0235999999999995e-05 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.731651+0000", 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5977000000000006e-05 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.732207+0000", 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00055650800000000005 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.732686+0000", 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00047815899999999997 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.932724+0000", 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200038308 2024-08-30T23:38:10.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.932763+0000", 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9483000000000001e-05 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.932789+0000", 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6026000000000001e-05 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:17939 5.3 5:cf308ff0:::benchmark_data_smithi173_87639_object17938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:52.428104+0000", 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.368785968000001, 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17575774, 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17939 2024-08-30T23:38:10.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428104+0000", 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428104+0000", 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428105+0000", 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428217+0000", 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112324 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428229+0000", 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2265e-05 2024-08-30T23:38:10.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428235+0000", 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1749999999999996e-06 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428276+0000", 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1396000000000001e-05 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428382+0000", 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105955 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.428500+0000", 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118433 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.429217+0000", 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071617600000000001 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.429548+0000", 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033150399999999999 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.603797+0000", 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17424854100000001 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.603833+0000", 2024-08-30T23:38:10.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6786999999999999e-05 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:52.603862+0000", 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8350000000000001e-05 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:43638 5.3 5:cdb2de2f:::benchmark_data_smithi173_87639_object43637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:10.742532+0000", 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.054357720999999998, 2024-08-30T23:38:10.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017375504, 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 43638 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:10.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.742532+0000", 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.742532+0000", 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.742533+0000", 2024-08-30T23:38:10.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7300000000000004e-07 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.742634+0000", 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010107699999999999 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.742643+0000", 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2070000000000001e-06 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.742650+0000", 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9739999999999996e-06 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.744208+0000", 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015576489999999999 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:10.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.744366+0000", 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000157725 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.744531+0000", 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016524899999999999 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.746417+0000", 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001885512 2024-08-30T23:38:10.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.747093+0000", 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00067649500000000005 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.759849+0000", 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012755953 2024-08-30T23:38:10.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.759879+0000", 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0346e-05 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:10.759908+0000", 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8343999999999998e-05 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.941 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.942 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.942 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:10.942 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:10.942 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:10.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:10.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:10.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 43920 43904 94.6131 105.5 0.0024376 0.0105631 2024-08-30T23:38:11.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/273636081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:11.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:11.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:11.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:11.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:11.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.355 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:11.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:11.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:11.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:11.535 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:11.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-30T23:38:11.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:11.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:11.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:38:11.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:11.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:38:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:11.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:11.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:11.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5137/115824 objects degraded (4.435%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:11 smithi106 ceph-mon[16603]: osdmap e410: 12 total, 11 up, 11 in 2024-08-30T23:38:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:11 smithi106 ceph-mon[16603]: pgmap v980: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 5137/124314 objects degraded (4.132%); 2682/124314 objects misplaced (2.157%) 2024-08-30T23:38:11.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:11.968 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:11.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:11.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:11.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 44486 44470 92.6384 35.375 0.00225002 0.0106361 2024-08-30T23:38:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:12.018 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5137/115824 objects degraded (4.435%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:12.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:11 smithi136 ceph-mon[20207]: osdmap e410: 12 total, 11 up, 11 in 2024-08-30T23:38:12.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:11 smithi136 ceph-mon[20207]: pgmap v980: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 5137/124314 objects degraded (4.132%); 2682/124314 objects misplaced (2.157%) 2024-08-30T23:38:12.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5137/115824 objects degraded (4.435%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:11 smithi173 ceph-mon[20554]: osdmap e410: 12 total, 11 up, 11 in 2024-08-30T23:38:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:11 smithi173 ceph-mon[20554]: pgmap v980: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s; 5137/124314 objects degraded (4.132%); 2682/124314 objects misplaced (2.157%) 2024-08-30T23:38:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:12.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:12.098 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:12.134 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:12.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:12.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:12.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:12.342 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:12.437 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:38:12.438 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:38:12.438 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:12.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:12.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:12.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:12.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:12.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:12.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:12.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:12.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:12.957 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:12.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:12.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 44553 44537 89.7849 4.1875 0.00282834 0.0106324 2024-08-30T23:38:13.007 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:13.015 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:13.016 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:13.016 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:13.016 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:13.016 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:13.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:13.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:38:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:13.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:13.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:13.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/367380553' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:13.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/367380553' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:13.325 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:13.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/367380553' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:13.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:13.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:13.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:13.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:13.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:13.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.702 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:13.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:13.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:13.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:13.834 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:13.834 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:38:13.834 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:38:13.834 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:38:13.834 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:13.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:13.954 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:13.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:13.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 44555 44539 86.9829 0.125 0.00390835 0.0106508 2024-08-30T23:38:14.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:38:14.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.085 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:14.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:14.154 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/367380553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:14.155 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:14 smithi136 ceph-mon[20207]: osdmap e411: 12 total, 11 up, 11 in 2024-08-30T23:38:14.155 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:14 smithi136 ceph-mon[20207]: pgmap v982: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 5504/129285 objects degraded (4.257%); 2682/129285 objects misplaced (2.074%) 2024-08-30T23:38:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:14.162 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:14.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-30T23:38:14.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:14.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:14.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/367380553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:14.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:14 smithi106 ceph-mon[16603]: osdmap e411: 12 total, 11 up, 11 in 2024-08-30T23:38:14.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:14 smithi106 ceph-mon[16603]: pgmap v982: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 5504/129285 objects degraded (4.257%); 2682/129285 objects misplaced (2.074%) 2024-08-30T23:38:14.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/367380553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:14 smithi173 ceph-mon[20554]: osdmap e411: 12 total, 11 up, 11 in 2024-08-30T23:38:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:14 smithi173 ceph-mon[20554]: pgmap v982: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 5504/129285 objects degraded (4.257%); 2682/129285 objects misplaced (2.074%) 2024-08-30T23:38:14.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:14.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:14.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:14.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:38:14.562 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:14.585 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:14.635 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:14.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:14.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:14.754 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:14.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:14.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:14.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:14.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:15.000 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 44555 44539 84.3469 0 - 0.0106508 2024-08-30T23:38:15.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:15.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:15.125 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:15.136 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:15.137 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:15.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:15.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:15.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:15.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41039 3.f 3:fd8786e1:::benchmark_data_smithi173_60780_object41038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:38:15.360 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 452.172877734, 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26531318500000001, 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41039 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186003+0000", 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186005+0000", 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.815e-06 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186051+0000", 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5473999999999999e-05 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186067+0000", 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6861999999999999e-05 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186076+0000", 2024-08-30T23:38:15.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8880000000000003e-06 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186133+0000", 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7101000000000001e-05 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186258+0000", 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012487800000000001 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.186377+0000", 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118268 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.187721+0000", 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013439979999999999 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.188919+0000", 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001198574 2024-08-30T23:38:15.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451218+0000", 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26229850199999999 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451272+0000", 2024-08-30T23:38:15.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4500999999999999e-05 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451316+0000", 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4323999999999998e-05 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:41062 3.f 3:fccf9ae1:::benchmark_data_smithi173_60780_object41061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-08-30T23:38:15.367 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 452.15692869399999, 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32901242400000003, 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 41062 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201952+0000", 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.201954+0000", 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202013+0000", 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9793999999999997e-05 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202024+0000", 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0502999999999999e-05 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202030+0000", 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6210000000000001e-06 2024-08-30T23:38:15.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202072+0000", 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2011999999999997e-05 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202154+0000", 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2306999999999996e-05 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 8,11", 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.202236+0000", 2024-08-30T23:38:15.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2520000000000003e-05 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203586+0000", 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00134931 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.204090+0000", 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050403899999999996 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530901+0000", 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32681120000000002 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530936+0000", 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4857000000000002e-05 2024-08-30T23:38:15.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.530965+0000", 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8999000000000001e-05 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:41039 3.f e226/206 ondisk, result = 0)", 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 452.15500810999998, 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24745020100000001, 2024-08-30T23:38:15.374 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.375 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.375 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:15.375 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203873+0000", 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203874+0000", 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1699999999999997e-07 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.203878+0000", 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.941e-06 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451055+0000", 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24717699200000001 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451067+0000", 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.186e-05 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451186+0000", 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118997 2024-08-30T23:38:15.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451210+0000", 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4219e-05 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:43.451323+0000", 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113375 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74450 3.d 3:b854981d:::benchmark_data_smithi173_60780_object74449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 423.39295006999998, 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29260630999999998, 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:15.380 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74450 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965932+0000", 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.71e-07 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965987+0000", 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5448000000000002e-05 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966002+0000", 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4419999999999999e-05 2024-08-30T23:38:15.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966007+0000", 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0119999999999996e-06 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967853+0000", 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001846547 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967977+0000", 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123404 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.968112+0000", 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013568200000000001 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970457+0000", 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023443460000000002 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.013531+0000", 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043074440999999998 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258300+0000", 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.244768449 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258497+0000", 2024-08-30T23:38:15.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000197209 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258537+0000", 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0481000000000001e-05 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:74450 3.d e243/206 ondisk, result = 0)", 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:38:15.386 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 423.38863614799999, 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28829785299999999, 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:38:15.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970246+0000", 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970250+0000", 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0330000000000002e-06 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258240+0000", 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28799039900000001 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258244+0000", 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1309999999999999e-06 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258287+0000", 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2347e-05 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258294+0000", 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2250000000000003e-06 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258543+0000", 2024-08-30T23:38:15.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000249001 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61137 4.0 4:05184379:::benchmark_data_smithi173_74914_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 200.945814824, 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0987384589999998, 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61137 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413067+0000", 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2640000000000001e-06 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413084+0000", 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6407999999999999e-05 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413095+0000", 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277999999999999e-05 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413117+0000", 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1498e-05 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413168+0000", 2024-08-30T23:38:15.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1266e-05 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413265+0000", 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6620000000000007e-05 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413359+0000", 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4003000000000001e-05 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415078+0000", 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719478 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415468+0000", 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038999199999999998 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511679+0000", 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0962105000000002 2024-08-30T23:38:15.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511758+0000", 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9426999999999997e-05 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511805+0000", 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6724999999999997e-05 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61138 4.0 4:0601b2bd:::benchmark_data_smithi173_74914_object61137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 200.205160065, 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582538660000001, 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61138 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.398 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153722+0000", 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-06 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153826+0000", 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103683 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153839+0000", 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3169e-05 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153863+0000", 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4326000000000001e-05 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153921+0000", 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7283e-05 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154051+0000", 2024-08-30T23:38:15.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012986399999999999 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154154+0000", 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103188 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156034+0000", 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018801009999999999 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156113+0000", 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8807000000000006e-05 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511881+0000", 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3557679949999999 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511932+0000", 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1134999999999997e-05 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511975+0000", 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2973999999999999e-05 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:62193 4.8 e337/329 4:198a939b:::benchmark_data_smithi173_74914_object62192:head v 337'3972, mlcod=337'3968)", 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 197.75375566899999, 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.349666123, 2024-08-30T23:38:15.404 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605126+0000", 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.13e-06 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605148+0000", 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1759e-05 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952851+0000", 2024-08-30T23:38:15.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34770309199999999 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952856+0000", 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1220000000000003e-06 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952891+0000", 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4292999999999998e-05 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952908+0000", 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7277000000000001e-05 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954758+0000", 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018501889999999999 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954791+0000", 2024-08-30T23:38:15.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3260999999999999e-05 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:93721 4.0 4:0b282d24:::benchmark_data_smithi173_74914_object93720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:27.048527+0000", 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 168.31035417800001, 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21384740099999999, 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93721 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.048527+0000", 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.048527+0000", 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.048528+0000", 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1999999999999998e-07 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.048574+0000", 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6019000000000001e-05 2024-08-30T23:38:15.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.048585+0000", 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1123e-05 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.048590+0000", 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6419999999999998e-06 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.050011+0000", 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001421388 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.050124+0000", 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112755 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.050260+0000", 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000135938 2024-08-30T23:38:15.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.055177+0000", 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049172520000000004 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.056977+0000", 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018004659999999999 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.262267+0000", 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.205289164 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.262325+0000", 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8156999999999999e-05 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.262374+0000", 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9577000000000003e-05 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.415 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:93723 4.0 4:04c656f0:::benchmark_data_smithi173_74914_object93722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:27.049008+0000", 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 168.30987317699999, 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21861377400000001, 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.416 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93723 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.049008+0000", 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.049008+0000", 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.049009+0000", 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7400000000000002e-07 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.049077+0000", 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8752999999999997e-05 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.049092+0000", 2024-08-30T23:38:15.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4382999999999999e-05 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.049097+0000", 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0730000000000004e-06 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052830+0000", 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037327929999999999 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052926+0000", 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6318999999999997e-05 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.053023+0000", 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6799000000000006e-05 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.056935+0000", 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039120329999999997 2024-08-30T23:38:15.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.057815+0000", 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00087983500000000004 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.267551+0000", 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.209735955 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.267584+0000", 2024-08-30T23:38:15.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2937e-05 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.267622+0000", 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8019999999999999e-05 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:93726 4.0 4:0dbc7619:::benchmark_data_smithi173_74914_object93725:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:27.052739+0000", 2024-08-30T23:38:15.422 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 168.30614158399999, 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21906863400000001, 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93726 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052739+0000", 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052739+0000", 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052741+0000", 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0750000000000001e-06 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052812+0000", 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1439000000000005e-05 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052824+0000", 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2065000000000001e-05 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.052831+0000", 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3839999999999999e-06 2024-08-30T23:38:15.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.055303+0000", 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024714379999999998 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.055382+0000", 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9207000000000002e-05 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.055499+0000", 2024-08-30T23:38:15.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116481 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.057090+0000", 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015915289999999999 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.066039+0000", 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0089494220000000003 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.271725+0000", 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20568569 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.271767+0000", 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1801000000000002e-05 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:27.271808+0000", 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1103000000000003e-05 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.428 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:95914 4.0 4:00b79c69:::benchmark_data_smithi173_74914_object95913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:29.125366+0000", 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 166.23351514399999, 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.217369273, 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.429 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 95914 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125366+0000", 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125366+0000", 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125367+0000", 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.161e-06 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125418+0000", 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0785999999999998e-05 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125428+0000", 2024-08-30T23:38:15.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0455000000000001e-05 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125433+0000", 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1460000000000004e-06 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125481+0000", 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7154999999999998e-05 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125618+0000", 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013782700000000001 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.125769+0000", 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015101599999999999 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.128138+0000", 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002368972 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.337822+0000", 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.209683482 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.342657+0000", 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0048349489999999998 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.342692+0000", 2024-08-30T23:38:15.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5367999999999997e-05 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:29.342735+0000", 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2956e-05 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:107569 4.0 4:06be00e7:::benchmark_data_smithi173_74914_object9327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:41.457255+0000", 2024-08-30T23:38:15.436 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.90162631499999, 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23871579300000001, 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 107569 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457255+0000", 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457255+0000", 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457256+0000", 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.173e-06 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457260+0000", 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5199999999999999e-06 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457274+0000", 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3309000000000001e-05 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457281+0000", 2024-08-30T23:38:15.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.798e-06 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457331+0000", 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0538000000000003e-05 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457461+0000", 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013044099999999999 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457579+0000", 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011773300000000001 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.460634+0000", 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030552209999999999 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.510843+0000", 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050208424000000001 2024-08-30T23:38:15.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695905+0000", 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18506230500000001 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695942+0000", 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7061999999999999e-05 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695971+0000", 2024-08-30T23:38:15.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8269e-05 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:107569 4.0 e354/329 ondisk, result = 0)", 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.89729024100001, 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.234384061, 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:38:15.443 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000003e-07 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461596+0000", 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695857+0000", 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23426132899999999 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695861+0000", 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4880000000000001e-06 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695895+0000", 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3485999999999998e-05 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695901+0000", 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1410000000000002e-06 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695975+0000", 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3751000000000006e-05 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:143754 4.0 4:0c834e11:::benchmark_data_smithi173_74914_object45512:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:19.175921+0000", 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.182960325, 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22610887700000001, 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 143754 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.448 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.175921+0000", 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.175921+0000", 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.175921+0000", 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9999999999999996e-07 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.175925+0000", 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8349999999999997e-06 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.175936+0000", 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.175943+0000", 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5209999999999999e-06 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.176016+0000", 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3628999999999996e-05 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.176133+0000", 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116181 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.176261+0000", 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012828199999999999 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.179103+0000", 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028424280000000001 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.179698+0000", 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059504999999999996 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.401901+0000", 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.222202971 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.401963+0000", 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1267999999999997e-05 2024-08-30T23:38:15.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.402030+0000", 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6948999999999999e-05 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:143768 4.3 4:c93e37ef:::benchmark_data_smithi173_74914_object45526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:38:15.454 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.927261708, 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37550649899999999, 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 143768 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431621+0000", 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431627+0000", 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431639+0000", 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2404e-05 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431646+0000", 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431724+0000", 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7655000000000001e-05 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431814+0000", 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0143999999999996e-05 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431885+0000", 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1421999999999995e-05 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.435174+0000", 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003288394 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.435587+0000", 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041313500000000002 2024-08-30T23:38:15.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807016+0000", 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37142968700000001 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807076+0000", 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9561000000000001e-05 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807126+0000", 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9864999999999998e-05 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:196473 4.0 4:0415f716:::benchmark_data_smithi173_74914_object98231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.334620424000001, 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49994438299999999, 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.461 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 196473 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024266+0000", 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1779999999999996e-06 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024277+0000", 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1567e-05 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024284+0000", 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4389999999999997e-06 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024336+0000", 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2145000000000003e-05 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024443+0000", 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107633 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024556+0000", 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011263 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029319+0000", 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047626949999999999 2024-08-30T23:38:15.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029471+0000", 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015204800000000001 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524119+0000", 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49464797199999999 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524156+0000", 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7716000000000003e-05 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524205+0000", 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8591000000000003e-05 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.467 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:196478 4.a e381/369 4:550417b9:::benchmark_data_smithi173_74914_object98236:head v 381'12132, mlcod=381'12130)", 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.329525640999996, 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.406346965, 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029356+0000", 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5400000000000001e-07 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029361+0000", 2024-08-30T23:38:15.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6739999999999996e-06 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431719+0000", 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.402358571 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431734+0000", 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4963000000000001e-05 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431842+0000", 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107765 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431888+0000", 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6269e-05 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.435671+0000", 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037823639999999999 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.435702+0000", 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1705000000000002e-05 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:12185 5.9 5:9cdde145:::benchmark_data_smithi173_87639_object12184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:49.205129+0000", 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.153751709000002, 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.184488289, 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.472 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12185 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205129+0000", 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205129+0000", 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205131+0000", 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2219999999999999e-06 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205184+0000", 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3285999999999998e-05 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205196+0000", 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2238e-05 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205201+0000", 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205244+0000", 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2901999999999997e-05 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205325+0000", 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0895000000000004e-05 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,11", 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.205402+0000", 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7171999999999999e-05 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.206006+0000", 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060330099999999999 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.206398+0000", 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039225300000000001 2024-08-30T23:38:15.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:15.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.389520+0000", 2024-08-30T23:38:15.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18312225800000001 2024-08-30T23:38:15.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.389575+0000", 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4518000000000002e-05 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:49.389618+0000", 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3025999999999998e-05 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.26173.0:15231 5.1 e403/389 5:8ac1186e:::benchmark_data_smithi173_87639_object15230:head v 403'920, mlcod=403'917)", 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:50.725538+0000", 2024-08-30T23:38:15.481 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.633343145000001, 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21369076100000001, 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.725538+0000", 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:38:15.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.725538+0000", 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.725539+0000", 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3099999999999997e-07 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.725645+0000", 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106587 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.938292+0000", 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.212647066 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.938298+0000", 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5090000000000001e-06 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.938347+0000", 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9585e-05 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.938390+0000", 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2698999999999999e-05 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.939179+0000", 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078917099999999997 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:50.939229+0000", 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9512999999999998e-05 2024-08-30T23:38:15.485 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.486 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.486 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:38:15.486 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:38:15.486 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:15.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:15.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:15.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:15.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:15.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:15.745 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:15 smithi106 ceph-mon[16603]: pgmap v983: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s; 5697/133575 objects degraded (4.265%); 2817/133575 objects misplaced (2.109%) 2024-08-30T23:38:15.759 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:15.775 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:15.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-30T23:38:15.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:15.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:15.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:15.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:15.999 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 44555 44539 81.866 0 - 0.0106508 2024-08-30T23:38:16.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:15 smithi136 ceph-mon[20207]: pgmap v983: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s; 5697/133575 objects degraded (4.265%); 2817/133575 objects misplaced (2.109%) 2024-08-30T23:38:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:15 smithi173 ceph-mon[20554]: pgmap v983: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s; 5697/133575 objects degraded (4.265%); 2817/133575 objects misplaced (2.109%) 2024-08-30T23:38:16.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:16.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:16.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:16.248 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:16.249 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:16.250 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:16.250 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:16.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:16.271 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:16.272 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:16.293 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:16.293 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:16.314 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:16.314 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:16.335 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:16.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:16.356 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:16.356 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:16.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:16.377 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:16.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:16.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:16.399 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:16.399 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:16.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:16.420 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:16.441 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:16.441 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:16.463 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:16.463 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:16.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:16.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:16.585 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:38:16.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:16.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:16.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:38:16.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:16.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:16 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5697/133575 objects degraded (4.265%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:16 smithi106 ceph-mon[16603]: osdmap e412: 12 total, 11 up, 11 in 2024-08-30T23:38:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/162607479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:16.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:16.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:16.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:16.938 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:16.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:17.000 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 44555 44539 79.5268 0 - 0.0106508 2024-08-30T23:38:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.017 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:17.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:16 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5697/133575 objects degraded (4.265%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:16 smithi136 ceph-mon[20207]: osdmap e412: 12 total, 11 up, 11 in 2024-08-30T23:38:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/162607479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:17.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:16 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:17.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5697/133575 objects degraded (4.265%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:17.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:16 smithi173 ceph-mon[20554]: osdmap e412: 12 total, 11 up, 11 in 2024-08-30T23:38:17.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/162607479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:17.107 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:17.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:17.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:17.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:17.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:17.348 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:17.360 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:38:17.360 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:38:17.360 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:38:17.361 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:38:17.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:17.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:17.440 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:17.440 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:17.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:17.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:17.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:17.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:17.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:17.764 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:17.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:17.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:17.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:17.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:17.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:18.000 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 44555 44539 77.3176 0 - 0.0106508 2024-08-30T23:38:18.016 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:17 smithi136 ceph-mon[20207]: pgmap v985: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 36 MiB/s wr, 579 op/s; 5703/133668 objects degraded (4.267%); 2817/133668 objects misplaced (2.107%) 2024-08-30T23:38:18.053 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:17 smithi106 ceph-mon[16603]: pgmap v985: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 36 MiB/s wr, 579 op/s; 5703/133668 objects degraded (4.267%); 2817/133668 objects misplaced (2.107%) 2024-08-30T23:38:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:17 smithi173 ceph-mon[20554]: pgmap v985: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 36 MiB/s wr, 579 op/s; 5703/133668 objects degraded (4.267%); 2817/133668 objects misplaced (2.107%) 2024-08-30T23:38:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:38:18.080 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:38:18.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:18.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:18.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:38:18.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:18.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:18.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:18.392 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:18.427 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:18.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:18.515 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:18.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:18.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:18.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:18.747 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:38:18.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:18.791 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3409998950' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:38:18.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:18.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:18.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:18.898 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:18.898 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:18.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:19.000 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 44555 44539 75.2275 0 - 0.0106508 2024-08-30T23:38:19.014 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:19.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3409998950' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:38:19.042 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3409998950' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:38:19.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:38:19.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.174 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:19.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:19.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:19.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:19.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:19.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:19.432 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:19.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:19.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:38:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:19.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:19.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:19.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:19.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:19.794 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:19.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3409998950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:38:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:19 smithi173 ceph-mon[20554]: osdmap e413: 12 total, 11 up, 11 in 2024-08-30T23:38:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:19 smithi173 ceph-mon[20554]: pgmap v987: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 16 MiB/s wr, 263 op/s; 5703/133671 objects degraded (4.266%); 2817/133671 objects misplaced (2.107%) 2024-08-30T23:38:19.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:19.911 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:19.911 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:19.911 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3409998950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:38:19.911 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:19 smithi106 ceph-mon[16603]: osdmap e413: 12 total, 11 up, 11 in 2024-08-30T23:38:19.911 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:19 smithi106 ceph-mon[16603]: pgmap v987: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 16 MiB/s wr, 263 op/s; 5703/133671 objects degraded (4.266%); 2817/133671 objects misplaced (2.107%) 2024-08-30T23:38:19.912 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:19.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:19.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:19.932 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:19.932 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:19.952 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:19.952 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:19.976 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:19.977 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:19.993 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:19.994 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:20.000 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 44555 44539 73.2478 0 - 0.0106508 2024-08-30T23:38:20.015 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:20.015 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:20.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:20.038 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:20.038 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3409998950' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:38:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:19 smithi136 ceph-mon[20207]: osdmap e413: 12 total, 11 up, 11 in 2024-08-30T23:38:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:19 smithi136 ceph-mon[20207]: pgmap v987: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 16 MiB/s wr, 263 op/s; 5703/133671 objects degraded (4.266%); 2817/133671 objects misplaced (2.107%) 2024-08-30T23:38:20.044 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:20.059 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:20.059 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:20.080 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:20.080 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:20.101 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:20.101 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:20.119 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:20.120 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:20.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:20.167 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:20.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:20.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:20.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:20.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:20.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:20.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:20.622 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:38:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:38:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:38:20.622 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.24460.0:144898 2.6 2:692f2db4:::benchmark_data_smithi173_47195_object47890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 588.94178651599998, 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18135380100000001, 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.24460", 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144898 2024-08-30T23:38:20.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679168+0000", 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679169+0000", 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679174+0000", 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.198e-06 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679187+0000", 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.31e-05 2024-08-30T23:38:20.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679194+0000", 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679236+0000", 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1828e-05 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679316+0000", 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0026999999999998e-05 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.679383+0000", 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6925e-05 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.682228+0000", 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028449149999999999 2024-08-30T23:38:20.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.829406+0000", 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14717798100000001 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860431+0000", 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.031025377 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860482+0000", 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0779000000000003e-05 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:28:31.860521+0000", 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9307999999999997e-05 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:d5896595:::benchmark_data_smithi173_47195_object12946:head,153'6861)", 2024-08-30T23:38:20.629 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 516.88350696099997, 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59714269600000003, 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737447+0000", 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737449+0000", 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:43.737471+0000", 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2631999999999999e-05 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334475+0000", 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59700367600000004 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334480+0000", 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2719999999999997e-06 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334536+0000", 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5430999999999998e-05 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334546+0000", 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9389999999999992e-06 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.334590+0000", 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4265999999999999e-05 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.b e200,182 2:dc4db0cb:::benchmark_data_smithi173_47195_object13755:head,153'6902)", 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 516.07169781200002, 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85400190600000003, 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.634 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549256+0000", 2024-08-30T23:38:20.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549257+0000", 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:44.549260+0000", 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.45e-06 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403003+0000", 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85374268200000003 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403016+0000", 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3033e-05 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403163+0000", 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014668400000000001 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403180+0000", 2024-08-30T23:38:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6813e-05 2024-08-30T23:38:20.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.403258+0000", 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8710999999999999e-05 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(2.5 e202,182 2:ac5fcd3c:::benchmark_data_smithi173_47195_object8560:head,150'6576)", 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 514.63246548500001, 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91766887600000002, 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988489+0000", 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1099999999999996e-07 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:45.988496+0000", 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4989999999999999e-06 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906006+0000", 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91750992200000003 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906016+0000", 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0199e-05 2024-08-30T23:38:20.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906087+0000", 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1316999999999995e-05 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906098+0000", 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0808000000000001e-05 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:29:46.906158+0000", 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9620000000000002e-05 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103438 3.9 3:9093f9df:::benchmark_data_smithi173_60780_object103437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 399.34608798599999, 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38848340100000001, 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.644 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103438 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274868+0000", 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274894+0000", 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5692000000000001e-05 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274909+0000", 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5378000000000001e-05 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274918+0000", 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6489999999999997e-06 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274990+0000", 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2335000000000005e-05 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275112+0000", 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012186 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:38:20.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275235+0000", 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123585 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278451+0000", 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003215954 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.279271+0000", 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081947200000000002 2024-08-30T23:38:20.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663147+0000", 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38387581399999998 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663253+0000", 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106125 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663350+0000", 2024-08-30T23:38:20.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6832999999999999e-05 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:103438 3.9 e255/206 ondisk, result = 0)", 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 399.342838712, 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38524248500000002, 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:38:20.651 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278116+0000", 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278121+0000", 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7609999999999998e-06 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662960+0000", 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38483874699999998 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662985+0000", 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663120+0000", 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013537999999999999 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663136+0000", 2024-08-30T23:38:20.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6220000000000001e-05 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663358+0000", 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022142900000000001 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:163721 3.9 3:9f4ef878:::benchmark_data_smithi173_60780_object60267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:38:20.655 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 331.96771494199999, 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.179110358, 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 163721 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653241+0000", 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.013e-06 2024-08-30T23:38:20.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653250+0000", 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653272+0000", 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2860000000000001e-05 2024-08-30T23:38:20.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653281+0000", 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8289999999999997e-06 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654712+0000", 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014306620000000001 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654854+0000", 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141755 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654980+0000", 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012682000000000001 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.658949+0000", 2024-08-30T23:38:20.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039685739999999999 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.714803+0000", 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055854251000000001 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832230+0000", 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117426525 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832301+0000", 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0921999999999996e-05 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832350+0000", 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8835000000000003e-05 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:26909 4.0 e315/306 4:05316899:::benchmark_data_smithi173_74914_object26908:head v 315'1674, mlcod=315'1671)", 2024-08-30T23:38:20.662 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.42446858700001, 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.194211306, 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196487+0000", 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196605+0000", 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011812299999999999 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389338+0000", 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19273368799999999 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389343+0000", 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6589999999999998e-06 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389405+0000", 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1645999999999999e-05 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389423+0000", 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8641e-05 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390666+0000", 2024-08-30T23:38:20.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001242993 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390697+0000", 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0565000000000003e-05 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:49431 4.2 e327/306 4:49678b2f:::benchmark_data_smithi173_74914_object49430:head v 327'3104, mlcod=326'3101)", 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 220.078886071, 2024-08-30T23:38:20.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.203368459, 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:38:20.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0900000000000002e-07 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542091+0000", 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2452000000000001e-05 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743809+0000", 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20171845999999999 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743817+0000", 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0960000000000001e-06 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743892+0000", 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4893000000000003e-05 2024-08-30T23:38:20.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743911+0000", 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8479999999999999e-05 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745394+0000", 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001483617 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745437+0000", 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2052000000000003e-05 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:84204 4.0 e346/329 4:023d35bd:::benchmark_data_smithi173_74914_object84203:head v 346'5351, mlcod=346'5348)", 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.22002225400001, 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20227883999999999, 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.672 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400933+0000", 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1990000000000001e-06 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400986+0000", 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3118000000000002e-05 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601032+0000", 2024-08-30T23:38:20.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200045414 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601038+0000", 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4330000000000001e-06 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601086+0000", 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7720999999999999e-05 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601104+0000", 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8267999999999999e-05 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603163+0000", 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020593569999999999 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603211+0000", 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7330000000000003e-05 2024-08-30T23:38:20.676 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:165168 4.0 e373/329 4:0c243000:::benchmark_data_smithi173_74914_object66926:head v 373'10494, mlcod=373'10490)", 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:43.509373+0000", 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 97.111580802999995, 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12675467100000001, 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.677 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509373+0000", 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509373+0000", 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509374+0000", 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1399999999999995e-07 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.509380+0000", 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9610000000000002e-06 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632491+0000", 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.123110441 2024-08-30T23:38:20.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632498+0000", 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8129999999999996e-06 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632552+0000", 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3498999999999997e-05 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.632569+0000", 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6973000000000002e-05 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.636080+0000", 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035110139999999998 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:43.636128+0000", 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8056000000000002e-05 2024-08-30T23:38:20.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 4.2,from:0'0,to:375'10705,epoch:375/306,start:4:47243d77:::benchmark_data_smithi173_74914_object84691:0,end:4:47ff4719:::benchmark_data_smithi173_74914_object80101:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:49.162596+0000", 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 91.458358163, 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.12535713500000001, 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:38:20.682 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162596+0000", 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162596+0000", 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162597+0000", 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.389e-06 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.162606+0000", 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2260000000000005e-06 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287807+0000", 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.125201072 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287817+0000", 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0295000000000001e-05 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287894+0000", 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7205000000000004e-05 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287911+0000", 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7031e-05 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:49.287953+0000", 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1916999999999999e-05 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:196473 4.0 e381/329 4:0415f716:::benchmark_data_smithi173_74914_object98231:head v 381'12558, mlcod=381'12555)", 2024-08-30T23:38:20.686 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.024645+0000", 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 62.596309183999999, 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49918435300000003, 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024645+0000", 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024645+0000", 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024646+0000", 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4160000000000001e-06 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024656+0000", 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1430000000000004e-06 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519446+0000", 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.494790018 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519459+0000", 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3637e-05 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519611+0000", 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000151867 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519654+0000", 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2935000000000001e-05 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.523788+0000", 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041342030000000004 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.523829+0000", 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1134000000000003e-05 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:12179 5.4 e401/394 5:20ccc99b:::benchmark_data_smithi173_87639_object12178:head v 401'786, mlcod=401'783)", 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:49.201090+0000", 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 31.419864048000001, 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.166900032, 2024-08-30T23:38:20.691 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201090+0000", 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201090+0000", 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201091+0000", 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0699999999999996e-07 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201149+0000", 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8604000000000003e-05 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367308+0000", 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16615836 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367313+0000", 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4949999999999999e-06 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367362+0000", 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9107000000000003e-05 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367377+0000", 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5007999999999999e-05 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367957+0000", 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058009700000000001 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367990+0000", 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2654000000000003e-05 2024-08-30T23:38:20.695 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:17934 5.4 e403/394 5:2f9197b6:::benchmark_data_smithi173_87639_object17933:head v 403'1150, mlcod=403'1148)", 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:52.425218+0000", 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.195735707000001, 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15083201399999999, 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.425218+0000", 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.425218+0000", 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.425219+0000", 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999991e-07 2024-08-30T23:38:20.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.425278+0000", 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8381000000000002e-05 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.575247+0000", 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14996924 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.575252+0000", 2024-08-30T23:38:20.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2850000000000002e-06 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.575305+0000", 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2417999999999999e-05 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.575326+0000", 2024-08-30T23:38:20.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1433999999999998e-05 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.576017+0000", 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069125299999999996 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:52.576050+0000", 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3028999999999998e-05 2024-08-30T23:38:20.701 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:34695 5.a e407/394 5:5ba104af:::benchmark_data_smithi173_87639_object34694:head v 407'2145, mlcod=407'2142)", 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:03.756488+0000", 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.864465900999999, 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15478784600000001, 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756488+0000", 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756488+0000", 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756489+0000", 2024-08-30T23:38:20.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756513+0000", 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3717999999999999e-05 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910265+0000", 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15375180799999999 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910269+0000", 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6920000000000001e-06 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910313+0000", 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3556000000000001e-05 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910325+0000", 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2661e-05 2024-08-30T23:38:20.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.911242+0000", 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091632699999999998 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.911276+0000", 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4303e-05 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:34702 5.a e407/394 5:558a094b:::benchmark_data_smithi173_87639_object34701:head v 407'2146, mlcod=407'2142)", 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:03.772866+0000", 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.848087675999999, 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15472281299999999, 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.772866+0000", 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.772866+0000", 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.772868+0000", 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.051e-06 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.772889+0000", 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1642999999999999e-05 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.926477+0000", 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15358806999999999 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.926485+0000", 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4479999999999997e-06 2024-08-30T23:38:20.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.926544+0000", 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9417999999999999e-05 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.926560+0000", 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6359999999999999e-05 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.927557+0000", 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099674500000000005 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.927589+0000", 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2078e-05 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.711 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:44429 5.4 e410/394 5:260bc407:::benchmark_data_smithi173_87639_object44428:head v 410'2763, mlcod=410'2760)", 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:11.291420+0000", 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.3295339580000007, 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.128089536, 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.291420+0000", 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.291420+0000", 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.291421+0000", 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.291451+0000", 2024-08-30T23:38:20.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9889000000000001e-05 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418206+0000", 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.126755585 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418213+0000", 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7900000000000002e-06 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.714 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418261+0000", 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8232999999999997e-05 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418275+0000", 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3988999999999999e-05 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.419462+0000", 2024-08-30T23:38:20.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011863900000000001 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.419510+0000", 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7922e-05 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:44435 5.4 e410/394 5:21107bab:::benchmark_data_smithi173_87639_object44434:head v 410'2764, mlcod=410'2761)", 2024-08-30T23:38:20.716 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:11.418128+0000", 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.2028257169999996, 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16557548499999999, 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418128+0000", 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418128+0000", 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418129+0000", 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0279999999999999e-06 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418188+0000", 2024-08-30T23:38:20.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8851999999999998e-05 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582325+0000", 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16413667500000001 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582331+0000", 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8730000000000004e-06 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582383+0000", 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2444999999999997e-05 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582404+0000", 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0818999999999998e-05 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.583669+0000", 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012650890000000001 2024-08-30T23:38:20.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.583704+0000", 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4703999999999998e-05 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:44461 5.4 e410/394 5:2bfa2ed6:::benchmark_data_smithi173_87639_object44460:head v 410'2765, mlcod=410'2762)", 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:11.582282+0000", 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 9.0386717329999993, 2024-08-30T23:38:20.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41074240899999998, 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582282+0000", 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582282+0000", 2024-08-30T23:38:20.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582283+0000", 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4099999999999998e-07 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582308+0000", 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4638e-05 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:20.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991587+0000", 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40927872999999998 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991605+0000", 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8839999999999999e-05 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991745+0000", 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014014 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991760+0000", 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4893e-05 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.992996+0000", 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012353889999999999 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.993025+0000", 2024-08-30T23:38:20.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9037999999999999e-05 2024-08-30T23:38:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:20.726 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:20.726 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:38:20.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:20.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:20.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:20.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:20.934 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:20.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:21.000 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 44637 44621 71.5011 0.732143 0.00418499 0.0139815 2024-08-30T23:38:21.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2316001933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:20 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:20 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5703/133671 objects degraded (4.266%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:20 smithi136 ceph-mon[20207]: osdmap e414: 12 total, 11 up, 11 in 2024-08-30T23:38:21.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2316001933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:21.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:20 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:21.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:20 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5703/133671 objects degraded (4.266%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:21.052 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:20 smithi173 ceph-mon[20554]: osdmap e414: 12 total, 11 up, 11 in 2024-08-30T23:38:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2316001933' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:20 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:20 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5703/133671 objects degraded (4.266%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:20 smithi106 ceph-mon[16603]: osdmap e414: 12 total, 11 up, 11 in 2024-08-30T23:38:21.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:21.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:21.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:21.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:21.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:21.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:21.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:38:21.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:21.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:21.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:21.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:38:21.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:21.657 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:21.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:21.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:21.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:21.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:21 smithi173 ceph-mon[20554]: pgmap v989: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 1.4 MiB/s wr, 22 op/s; 5703/133977 objects degraded (4.257%); 2817/133977 objects misplaced (2.103%) 2024-08-30T23:38:21.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:21.933 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:21.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:21.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:21.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:22.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:38:22.001557+0000 min lat: 0.00108473 max lat: 9.72704 avg lat: 0.0138255 2024-08-30T23:38:22.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:38:22.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 46295 46279 72.304 103.625 0.00403228 0.0138255 2024-08-30T23:38:22.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:21 smithi136 ceph-mon[20207]: pgmap v989: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 1.4 MiB/s wr, 22 op/s; 5703/133977 objects degraded (4.257%); 2817/133977 objects misplaced (2.103%) 2024-08-30T23:38:22.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:22.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:22.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:21 smithi106 ceph-mon[16603]: pgmap v989: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 1.4 MiB/s wr, 22 op/s; 5703/133977 objects degraded (4.257%); 2817/133977 objects misplaced (2.103%) 2024-08-30T23:38:22.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:22.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:22.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:22.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:22.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:22.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:22.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:22.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:22.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:22.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:22.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:22.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:22.645 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:38:22.645 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:38:22.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:47251 5.b 5:de017344:::benchmark_data_smithi173_87639_object47250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.635610+0000", 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0072356599999999997, 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0072644850000000002, 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47251 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635610+0000", 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635610+0000", 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635612+0000", 2024-08-30T23:38:22.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.477e-06 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635632+0000", 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0775e-05 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635643+0000", 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1153e-05 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635649+0000", 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4349999999999996e-06 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635695+0000", 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6594000000000001e-05 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:22.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635799+0000", 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103875 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.635889+0000", 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9954e-05 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.637346+0000", 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014563110000000001 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.637881+0000", 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053527800000000003 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:47254 5.b 5:dde512e0:::benchmark_data_smithi173_87639_object47253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.639708+0000", 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0031374010000000002, 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003249437, 2024-08-30T23:38:22.651 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47254 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639708+0000", 2024-08-30T23:38:22.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639708+0000", 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639710+0000", 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.426e-06 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639734+0000", 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4573e-05 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639750+0000", 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5237e-05 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639758+0000", 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2330000000000006e-06 2024-08-30T23:38:22.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639813+0000", 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4750999999999998e-05 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639915+0000", 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102688 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:38:22.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640017+0000", 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101718 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:47255 5.b 5:d583cb60:::benchmark_data_smithi173_87639_object47254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.640090+0000", 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0027554070000000001, 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029463779999999999, 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:22.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:22.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:38:22.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:22.657 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47255 2024-08-30T23:38:22.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.809 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.809 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.809 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.809 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640090+0000", 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640090+0000", 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.810 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640091+0000", 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9899999999999999e-07 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.811 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640124+0000", 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2651000000000003e-05 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640136+0000", 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1719000000000001e-05 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640143+0000", 2024-08-30T23:38:22.812 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7100000000000007e-06 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641530+0000", 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013869679999999999 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641652+0000", 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.813 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641800+0000", 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014786500000000001 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:47256 5.e 5:74f2533f:::benchmark_data_smithi173_87639_object47255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.640226+0000", 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.002619564, 2024-08-30T23:38:22.814 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028882830000000002, 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47256 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640226+0000", 2024-08-30T23:38:22.815 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640226+0000", 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640227+0000", 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640248+0000", 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1236000000000001e-05 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640260+0000", 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.219e-05 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640269+0000", 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4430000000000003e-06 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640320+0000", 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0958999999999998e-05 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640438+0000", 2024-08-30T23:38:22.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118668 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.640570+0000", 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013175400000000001 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642181+0000", 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016105259999999999 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642433+0000", 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025234299999999998 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:47254 5.b e414/389)", 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.641552+0000", 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0012938100000000001, 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016270779999999999, 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.820 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641552+0000", 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641552+0000", 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641553+0000", 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641559+0000", 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9580000000000001e-06 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641586+0000", 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6466e-05 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641592+0000", 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8560000000000003e-06 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:47251 5.b e414/389)", 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.639738+0000", 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0031078220000000001, 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034781389999999999, 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639738+0000", 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639738+0000", 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639739+0000", 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8299999999999996e-07 2024-08-30T23:38:22.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.639742+0000", 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.579e-06 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641718+0000", 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019760060000000002 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641731+0000", 2024-08-30T23:38:22.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2388999999999999e-05 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:47254 5.b e414/389)", 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.641695+0000", 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0011510310000000001, 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015554970000000001, 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:38:22.827 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641695+0000", 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641695+0000", 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641695+0000", 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5899999999999996e-07 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.641702+0000", 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1189999999999999e-06 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642498+0000", 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00079540299999999997 2024-08-30T23:38:22.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642513+0000", 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5255999999999999e-05 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:47258 5.e 5:7ed8c4ee:::benchmark_data_smithi173_87639_object47257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.642602+0000", 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00024381199999999999, 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00069094999999999996, 2024-08-30T23:38:22.830 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47258 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642602+0000", 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642602+0000", 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642603+0000", 2024-08-30T23:38:22.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.344e-06 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642627+0000", 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3348000000000001e-05 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642641+0000", 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4144e-05 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642648+0000", 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9349999999999997e-06 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642705+0000", 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7117000000000002e-05 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642826+0000", 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120934 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642975+0000", 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000149276 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:47256 5.e e414/389)", 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:22.642473+0000", 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00037275900000000001, 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00087197399999999997, 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:38:22.836 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642473+0000", 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642473+0000", 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642474+0000", 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7199999999999998e-07 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.642480+0000", 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5239999999999997e-06 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.643034+0000", 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000554219 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:22.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:22.643040+0000", 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5670000000000002e-06 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 9 2024-08-30T23:38:22.839 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:38:22.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:22.841 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:38:22.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:22.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:22.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:38:22.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:23.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 47820 47804 72.865 95.3125 0.00991462 0.013719 2024-08-30T23:38:23.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2034899630' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2034899630' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:23.081 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:38:23.081 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:38:23.081 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-08-30T23:38:23.082 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:38:23.082 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:38:23.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-08-30T23:38:23.104 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2034899630' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:23.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:23.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:23.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:23.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:23.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:23.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:23.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:38:23.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:23.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:23.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:23.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:23.641 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:23.666 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:23.669 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:23.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:23.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:23.771 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:23.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2034899630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:23.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: osdmap e415: 12 total, 11 up, 11 in 2024-08-30T23:38:23.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:38:23.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:38:23.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:38:23.772 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:23.773 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:23.773 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:23.773 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:23 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2034899630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: osdmap e415: 12 total, 11 up, 11 in 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:23.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:23 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:38:23.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:23.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:23.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:23.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:24.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 48700 48684 72.4395 55 0.0156797 0.0137979 2024-08-30T23:38:24.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.021 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2034899630' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: osdmap e415: 12 total, 11 up, 11 in 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:24.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:23 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:38:24.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:24.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:24.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:24.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:24.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:24.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:24.400 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:38:24.400 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:38:24.400 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:38:24.400 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:38:24.400 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:38:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:24.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:24.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:24.583 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:24.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:24.583 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:24.606 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:24.606 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:24.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:24.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:24.631 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:24.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:24.656 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:24.657 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:24.685 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:24.686 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:24.724 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:24.724 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:24.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:24.755 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:24.785 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:24.785 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:24.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:24.816 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:24.816 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:24.818 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:38:24.819 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:38:24.819 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:38:24.846 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:24.847 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:24.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2010269020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:24.874 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:24.874 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:24.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:24.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:24.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:25.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 49963 49947 72.5905 78.9375 0.0298656 0.0137665 2024-08-30T23:38:25.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2010269020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:25.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:25.072 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2010269020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:25.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:38:25.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:25.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:25.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:25.293 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:25.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:25.413 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:25.427 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:25.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:25.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:25.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:38:25.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:25.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:25.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:25.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:25 smithi173 ceph-mon[20554]: pgmap v992: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s; 6074/144186 objects degraded (4.213%); 3150/144186 objects misplaced (2.185%) 2024-08-30T23:38:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:25 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5703/133977 objects degraded (4.257%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:25 smithi173 ceph-mon[20554]: osdmap e416: 12 total, 11 up, 11 in 2024-08-30T23:38:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:25 smithi106 ceph-mon[16603]: pgmap v992: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s; 6074/144186 objects degraded (4.213%); 3150/144186 objects misplaced (2.185%) 2024-08-30T23:38:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:25 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5703/133977 objects degraded (4.257%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:25 smithi106 ceph-mon[16603]: osdmap e416: 12 total, 11 up, 11 in 2024-08-30T23:38:25.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:25.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:25.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:26.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 51291 51275 72.8267 83 0.0099967 0.013696 2024-08-30T23:38:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:25 smithi136 ceph-mon[20207]: pgmap v992: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 2.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s; 6074/144186 objects degraded (4.213%); 3150/144186 objects misplaced (2.185%) 2024-08-30T23:38:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:25 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5703/133977 objects degraded (4.257%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:25 smithi136 ceph-mon[20207]: osdmap e416: 12 total, 11 up, 11 in 2024-08-30T23:38:26.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.066 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:26.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:26.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:26.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:26.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:26.222 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:26.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:26.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:26.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:26.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:26.477 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:26.477 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:26.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:26.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:26.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:26.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:26.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:26.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:26.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:26.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:26.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:27.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 52743 52727 73.2248 90.75 0.00491649 0.0136504 2024-08-30T23:38:27.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:27.082 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:38:27.091 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:27.169 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3366634789' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:27.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:27.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:38:27.217 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:27.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:27.342 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3366634789' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:27.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3366634789' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:27.552 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:27.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:27.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:27.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:27.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:27.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:27.847 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:27.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:27.977 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:27.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:28.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 54050 54034 73.4087 81.6875 0.0169372 0.0136168 2024-08-30T23:38:28.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:28.090 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:28.090 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:28.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:28.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:38:28.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.274 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:28.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:28 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3366634789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:28 smithi173 ceph-mon[20554]: osdmap e417: 12 total, 11 up, 11 in 2024-08-30T23:38:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:28 smithi173 ceph-mon[20554]: pgmap v995: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 971 op/s; 6485/151473 objects degraded (4.281%); 3150/151473 objects misplaced (2.080%) 2024-08-30T23:38:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:28 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3366634789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:28 smithi106 ceph-mon[16603]: osdmap e417: 12 total, 11 up, 11 in 2024-08-30T23:38:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:28 smithi106 ceph-mon[16603]: pgmap v995: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 971 op/s; 6485/151473 objects degraded (4.281%); 3150/151473 objects misplaced (2.080%) 2024-08-30T23:38:28.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:28.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:28.483 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:28.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.528 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:28.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:28.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:28 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:28.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3366634789' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:28 smithi136 ceph-mon[20207]: osdmap e417: 12 total, 11 up, 11 in 2024-08-30T23:38:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:28 smithi136 ceph-mon[20207]: pgmap v995: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 61 MiB/s wr, 971 op/s; 6485/151473 objects degraded (4.281%); 3150/151473 objects misplaced (2.080%) 2024-08-30T23:38:28.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:28.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:28.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:28.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:38:28.817 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:28.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:28.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:28.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:29.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 55276 55260 73.477 76.625 0.0104652 0.0136039 2024-08-30T23:38:29.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:29.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:29.026 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:29.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:29.053 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:29.053 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:29.079 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:29.081 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:29.081 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:29.108 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:29.108 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:29.109 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:29.133 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:29.133 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:29.158 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:29.159 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:29.159 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1908776948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:29.180 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:29.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:29.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:29.201 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:29.201 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:29.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:29.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:29.222 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:29.243 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:29.243 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:29.264 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:29.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:29.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:29.332 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:29.379 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1908776948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:29.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:29.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:29.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1908776948' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:29.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:29.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:29.638 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:38:29.638 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:38:29.638 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:38:29.638 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:38:29.638 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:38:29.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:29.710 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:29.812 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:29.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:29.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:29.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:29.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:30.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 56562 56546 73.6205 80.375 0.00851488 0.0135778 2024-08-30T23:38:30.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:30.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:30.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:30.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:30.271 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:30 smithi173 ceph-mon[20554]: pgmap v996: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 6882/157632 objects degraded (4.366%); 3150/157632 objects misplaced (1.998%) 2024-08-30T23:38:30.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:38:30.318 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:38:30.335 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:30.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:30 smithi106 ceph-mon[16603]: pgmap v996: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 6882/157632 objects degraded (4.366%); 3150/157632 objects misplaced (1.998%) 2024-08-30T23:38:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:30.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:30.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:30.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:30 smithi136 ceph-mon[20207]: pgmap v996: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 6882/157632 objects degraded (4.366%); 3150/157632 objects misplaced (1.998%) 2024-08-30T23:38:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:30.557 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:30.598 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:30.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:30.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:38:30.633 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:38:30.633 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-08-30T23:38:30.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd in 10 2024-08-30T23:38:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:30.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:30.672 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:30.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:30.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:30.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:30.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:30.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:30.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:31.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 57885 57869 73.8054 82.6875 0.00373772 0.0135428 2024-08-30T23:38:31.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:31.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:31.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:31.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:31.202 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:164675 3.1 3:8010289e:::benchmark_data_smithi173_60780_object61221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:38:31.234 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 341.26315170599997, 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27192025399999997, 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 164675 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969739+0000", 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969741+0000", 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0329999999999998e-06 2024-08-30T23:38:31.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969748+0000", 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.241e-06 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969761+0000", 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3162999999999999e-05 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.969768+0000", 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5429999999999996e-06 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.970901+0000", 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011326870000000001 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:49.971033+0000", 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013183100000000001 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047341+0000", 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.076308670999999995 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047456+0000", 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114108 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,11", 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.047620+0000", 2024-08-30T23:38:31.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000164504 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.061526+0000", 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013906091000000001 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.065492+0000", 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039661649999999998 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.240280+0000", 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.17478812399999999 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.240349+0000", 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8520000000000001e-05 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:50.241660+0000", 2024-08-30T23:38:31.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001310573 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e300,280 3:83685fcd:::benchmark_data_smithi173_60780_object2411:head,257'6623)", 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 286.83072761599999, 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41143703700000001, 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.242 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402163+0000", 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402164+0000", 2024-08-30T23:38:31.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.402168+0000", 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0570000000000003e-06 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812764+0000", 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41059553700000001 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812774+0000", 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0529e-05 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.812860+0000", 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5934999999999996e-05 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.813600+0000", 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074028899999999999 2024-08-30T23:38:31.245 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e301,280 3:8490e9ac:::benchmark_data_smithi173_60780_object7986:head,258'6976)", 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 284.05391312900002, 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70969468099999999, 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4199999999999995e-07 2024-08-30T23:38:31.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178982+0000", 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9779999999999999e-06 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887644+0000", 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70866116300000004 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887661+0000", 2024-08-30T23:38:31.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6954999999999999e-05 2024-08-30T23:38:31.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887778+0000", 2024-08-30T23:38:31.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117812 2024-08-30T23:38:31.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.888673+0000", 2024-08-30T23:38:31.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00089413100000000005 2024-08-30T23:38:31.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.252 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:24732 4.a e315/306 4:5449864d:::benchmark_data_smithi173_74914_object24731:head v 315'1539, mlcod=315'1535)", 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 254.72356614, 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24331156100000001, 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509325+0000", 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509326+0000", 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.509403+0000", 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7809e-05 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751493+0000", 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24208919500000001 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751500+0000", 2024-08-30T23:38:31.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5190000000000003e-06 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751549+0000", 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8891000000000003e-05 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.751566+0000", 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7102999999999999e-05 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.752577+0000", 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001011248 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:16.752636+0000", 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9079000000000001e-05 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.257 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31037 4.5 4.306472d5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.550478024, 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0020195190000001, 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31037 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682415+0000", 2024-08-30T23:38:31.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6589999999999999e-06 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682488+0000", 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3084999999999996e-05 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682498+0000", 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0644e-05 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682506+0000", 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0229999999999992e-06 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682702+0000", 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019531299999999999 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682713+0000", 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1778e-05 2024-08-30T23:38:31.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684400+0000", 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016865800000001 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684432+0000", 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2437000000000001e-05 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31040 4.5 4.eb15935 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.55032043700001, 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.001902058, 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31040 2024-08-30T23:38:31.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682627+0000", 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5911000000000001e-05 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682637+0000", 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0363999999999999e-05 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682642+0000", 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8509999999999998e-06 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682765+0000", 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123311 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:38:31.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682799+0000", 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3145000000000002e-05 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684465+0000", 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016660740000001 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684473+0000", 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9149999999999996e-06 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.267 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31060 4.5 4.c6dc7475 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.54091521300001, 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99255960700000001, 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.268 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31060 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691977+0000", 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.066e-06 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691998+0000", 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.158e-05 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692011+0000", 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2848e-05 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692019+0000", 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8990000000000001e-06 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692069+0000", 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0080999999999997e-05 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692076+0000", 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1450000000000002e-06 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684524+0000", 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99244810000000006 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684535+0000", 2024-08-30T23:38:31.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0888e-05 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31063 4.5 4.5c985545 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.096409166, 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54808229100000005, 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31063 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136482+0000", 2024-08-30T23:38:31.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136484+0000", 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7069999999999999e-06 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136509+0000", 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4990999999999998e-05 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136526+0000", 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7178999999999999e-05 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136533+0000", 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8789999999999997e-06 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136588+0000", 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5018000000000001e-05 2024-08-30T23:38:31.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.136596+0000", 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9820000000000008e-06 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684558+0000", 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54796194799999998 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684564+0000", 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5869999999999998e-06 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:50603 4.a e329/318 4:54bdef61:::benchmark_data_smithi173_74914_object50602:head v 329'3160, mlcod=315'1602)", 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 229.57156338199999, 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25119703700000001, 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.278 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661328+0000", 2024-08-30T23:38:31.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.661786+0000", 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00045819399999999998 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.910994+0000", 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.249208242 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.910999+0000", 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8640000000000004e-06 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.911049+0000", 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0216000000000002e-05 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.911074+0000", 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4484000000000001e-05 2024-08-30T23:38:31.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.912487+0000", 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001413648 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:41.912525+0000", 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7171e-05 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(4.4 329/318 [PushOp(4:2f097f2f:::benchmark_data_smithi173_74914_object29004:head, version: 316'1834, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:2f097f2f:::benchmark_data_smithi173_74914_object29004:head@316'1834, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 229.165303513, 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.51130092400000005, 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067587+0000", 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067588+0000", 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6199999999999997e-07 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.067620+0000", 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2128999999999997e-05 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577769+0000", 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.510148398 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577780+0000", 2024-08-30T23:38:31.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1617000000000001e-05 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577885+0000", 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104666 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.577899+0000", 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4568e-05 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:42.578888+0000", 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098888399999999999 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(4.5 335/318 [PushReplyOp(4:a5afb7d9:::benchmark_data_smithi173_74914_object29619:head)])", 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 214.97410660599999, 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.227084861, 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.287 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258784+0000", 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258785+0000", 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9500000000000003e-07 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.258788+0000", 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2160000000000002e-06 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485597+0000", 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.226809118 2024-08-30T23:38:31.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485606+0000", 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1260000000000004e-06 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485684+0000", 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7238999999999997e-05 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:56.485869+0000", 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018566700000000001 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62091 4.4 e337/331 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head v 337'3874, mlcod=336'3872)", 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 214.066636942, 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.317048422, 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.291 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166254+0000", 2024-08-30T23:38:31.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.166315+0000", 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0946000000000002e-05 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481856+0000", 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31554066800000002 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481863+0000", 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0960000000000003e-06 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481922+0000", 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8789000000000002e-05 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.481941+0000", 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9307999999999999e-05 2024-08-30T23:38:31.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.483268+0000", 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013262339999999999 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.483302+0000", 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4947999999999998e-05 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.296 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62105 4.4 e337/331 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head v 337'3875, mlcod=336'3872)", 2024-08-30T23:38:31.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:31.297 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:38:31.297 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 214.053050727, 2024-08-30T23:38:31.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34727296499999999, 2024-08-30T23:38:31.297 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.297 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.297 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179840+0000", 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179841+0000", 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.68e-07 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.179895+0000", 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4670999999999999e-05 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525681+0000", 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34578567199999999 2024-08-30T23:38:31.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525688+0000", 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4069999999999998e-06 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525744+0000", 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6588000000000001e-05 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525761+0000", 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6642000000000001e-05 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.527080+0000", 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013189460000000001 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.527113+0000", 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3470999999999999e-05 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.301 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62174 4.4 e337/331 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head v 337'3877, mlcod=336'3873)", 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.70728577200001, 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57728454699999998, 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525605+0000", 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525606+0000", 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1429999999999999e-06 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.525661+0000", 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4715e-05 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101380+0000", 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.57571867700000001 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101387+0000", 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.746e-06 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101441+0000", 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3916999999999999e-05 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.101460+0000", 2024-08-30T23:38:31.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8185999999999998e-05 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.102857+0000", 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013971820000000001 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.102890+0000", 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2981000000000002e-05 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.306 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:62186 4.4 e337/331 4:257aab52:::benchmark_data_smithi173_74914_object62185:head v 337'3878, mlcod=337'3874)", 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 213.67866080600001, 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55065025599999995, 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554230+0000", 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554231+0000", 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1099999999999995e-07 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.554289+0000", 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8440000000000003e-05 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103923+0000", 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54963335000000002 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103927+0000", 2024-08-30T23:38:31.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7199999999999997e-06 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103985+0000", 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7398999999999997e-05 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.103997+0000", 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2617e-05 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.104853+0000", 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085557500000000002 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:58.104880+0000", 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7444e-05 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.311 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98221 4.5 4:a4ee3b1f:::benchmark_data_smithi173_74914_object98220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.795700544, 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.225938741, 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98221 2024-08-30T23:38:31.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437190+0000", 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437192+0000", 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.598e-06 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437212+0000", 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0387999999999999e-05 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437225+0000", 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2393e-05 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437230+0000", 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5319999999999997e-06 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437273+0000", 2024-08-30T23:38:31.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2778999999999998e-05 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437360+0000", 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6718000000000006e-05 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.437446+0000", 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6215e-05 2024-08-30T23:38:31.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.439899+0000", 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024530929999999999 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447609+0000", 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.007709624 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.662941+0000", 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21533202500000001 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.663056+0000", 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115085 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.663129+0000", 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3291e-05 2024-08-30T23:38:31.318 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98225 4.5 4:ad425262:::benchmark_data_smithi173_74914_object98224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.788381233, 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32404802500000002, 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.319 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98225 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444510+0000", 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0200000000000001e-07 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444529+0000", 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8377999999999999e-05 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444538+0000", 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6719999999999999e-06 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444543+0000", 2024-08-30T23:38:31.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9119999999999997e-06 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444596+0000", 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2286e-05 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444722+0000", 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126229 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.444864+0000", 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014163000000000001 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447348+0000", 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024847910000000001 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.450736+0000", 2024-08-30T23:38:31.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0033874059999999999 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768431+0000", 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31769526799999998 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768502+0000", 2024-08-30T23:38:31.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0516e-05 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.768558+0000", 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6234999999999999e-05 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:98228 4.7 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:38:31.326 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 179.78525788499999, 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35701674100000003, 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 98228 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:38:31.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447633+0000", 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447634+0000", 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.035e-06 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447655+0000", 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0735000000000001e-05 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447665+0000", 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0353999999999999e-05 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447679+0000", 2024-08-30T23:38:31.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3804e-05 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447729+0000", 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9570000000000001e-05 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.447855+0000", 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000126453 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:38:31.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.448003+0000", 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014797200000000001 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.450323+0000", 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002320192 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.604622+0000", 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15429874599999999 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804484+0000", 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.199861699 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804600+0000", 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011622500000000001 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:31.804650+0000", 2024-08-30T23:38:31.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9956000000000001e-05 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:196465 4.4 e381/331 4:242a1b19:::benchmark_data_smithi173_74914_object98223:head v 381'12200, mlcod=381'12196)", 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.016412+0000", 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.216478491999993, 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45701166199999999, 2024-08-30T23:38:31.333 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016412+0000", 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016412+0000", 2024-08-30T23:38:31.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016413+0000", 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2800000000000005e-07 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016420+0000", 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.392e-06 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.469072+0000", 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45265234100000001 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.469088+0000", 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6232999999999999e-05 2024-08-30T23:38:31.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.469207+0000", 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118843 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.469230+0000", 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3153e-05 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.473378+0000", 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041476270000000001 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.473424+0000", 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6144999999999999e-05 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:196478 4.a 4:550417b9:::benchmark_data_smithi173_74914_object98236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.028886+0000", 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 73.204005018000004, 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.407278205, 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:38:31.339 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 196478 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.028886+0000", 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.028886+0000", 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.028887+0000", 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4249999999999999e-06 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.028893+0000", 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0129999999999999e-06 2024-08-30T23:38:31.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.028910+0000", 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6473999999999998e-05 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.028918+0000", 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6969999999999993e-06 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:38:31.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.029016+0000", 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8335e-05 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.029137+0000", 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120988 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7", 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.029245+0000", 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108206 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.033473+0000", 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004227771 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.435988+0000", 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40251555 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.436059+0000", 2024-08-30T23:38:31.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0889999999999994e-05 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.436164+0000", 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104856 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:38:31.345 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:38:31.346 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6882/157632 objects degraded (4.366%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:31.346 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:31 smithi173 ceph-mon[20554]: osdmap e418: 12 total, 11 up, 11 in 2024-08-30T23:38:31.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6882/157632 objects degraded (4.366%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:31.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:31 smithi106 ceph-mon[16603]: osdmap e418: 12 total, 11 up, 11 in 2024-08-30T23:38:31.369 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:31.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:31.494 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6882/157632 objects degraded (4.366%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:31 smithi136 ceph-mon[20207]: osdmap e418: 12 total, 11 up, 11 in 2024-08-30T23:38:31.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:31.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:31.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:31.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:31.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:31.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:31.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:31.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:32.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:32.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 58934 58918 73.6404 65.5625 0.00448429 0.0135391 2024-08-30T23:38:32.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:32.110 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:38:32.110 INFO:teuthology.orchestra.run.smithi106.stderr:marked in osd.10. 2024-08-30T23:38:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.128 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:32.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:32.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:38:32.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:32.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:32.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/381241356' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:32.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:32 smithi173 ceph-mon[20554]: pgmap v998: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 6882/166428 objects degraded (4.135%); 3540/166428 objects misplaced (2.127%) 2024-08-30T23:38:32.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:32 smithi173 ceph-mon[20554]: from='client.16818 172.21.15.106:0/3226832469' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:38:32.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:32.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:31 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:38:31.928+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e418 definitely_dead 0 2024-08-30T23:38:32.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/381241356' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:32.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:32 smithi106 ceph-mon[16603]: pgmap v998: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 6882/166428 objects degraded (4.135%); 3540/166428 objects misplaced (2.127%) 2024-08-30T23:38:32.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:32 smithi106 ceph-mon[16603]: from='client.16818 172.21.15.106:0/3226832469' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:38:32.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:32.409 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:32.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.500 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:32.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/381241356' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:32 smithi136 ceph-mon[20207]: pgmap v998: 17 pgs: 1 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 3.4 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 6882/166428 objects degraded (4.135%); 3540/166428 objects misplaced (2.127%) 2024-08-30T23:38:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:32 smithi136 ceph-mon[20207]: from='client.16818 172.21.15.106:0/3226832469' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:38:32.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:38:32.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:32.623 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:32.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:32.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:32.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:32.853 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-08-30T23:38:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:32.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:32.905 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:38:32.906 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:38:32.906 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:38:32.906 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:38:32.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:32.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:32.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:33.001 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 59017 59001 72.2982 5.1875 0.162117 0.013628 2024-08-30T23:38:33.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:33.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:33.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:33.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:33.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:33 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/381241356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:33 smithi106 ceph-mon[16603]: from='client.16818 172.21.15.106:0/3226832469' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:38:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:33 smithi106 ceph-mon[16603]: osdmap e419: 12 total, 11 up, 12 in 2024-08-30T23:38:33.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:33.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:33.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.495 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:33.539 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:33 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/381241356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:33 smithi136 ceph-mon[20207]: from='client.16818 172.21.15.106:0/3226832469' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:38:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:33 smithi136 ceph-mon[20207]: osdmap e419: 12 total, 11 up, 12 in 2024-08-30T23:38:33.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:33 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:33.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/381241356' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:33.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:33 smithi173 ceph-mon[20554]: from='client.16818 172.21.15.106:0/3226832469' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:38:33.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:33 smithi173 ceph-mon[20554]: osdmap e419: 12 total, 11 up, 12 in 2024-08-30T23:38:33.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:33.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:38:33.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:33.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:33.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:33.837 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:33.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:33.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:33.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:33.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:33.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:34.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 59017 59001 70.9078 0 - 0.013628 2024-08-30T23:38:34.065 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:34.065 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:34.065 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:34.086 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:34.086 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:34.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:34.107 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:34.107 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:34.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:34.136 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:34.136 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:34.171 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:34.171 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:34.205 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:34.205 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:34.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:34.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:34 smithi173 ceph-mon[20554]: osdmap e420: 12 total, 11 up, 12 in 2024-08-30T23:38:34.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:34 smithi173 ceph-mon[20554]: pgmap v1001: 17 pgs: 2 remapped+peering, 1 active+clean+remapped, 2 active+undersized+degraded, 12 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 7280/172065 objects degraded (4.231%); 3540/172065 objects misplaced (2.057%) 2024-08-30T23:38:34.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/960904010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:34.231 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:34.231 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:34.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:34.254 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:34.254 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:34.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:34.285 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:34.310 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:34.311 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:34.339 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:34.340 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:34 smithi106 ceph-mon[16603]: osdmap e420: 12 total, 11 up, 12 in 2024-08-30T23:38:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:34 smithi106 ceph-mon[16603]: pgmap v1001: 17 pgs: 2 remapped+peering, 1 active+clean+remapped, 2 active+undersized+degraded, 12 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 7280/172065 objects degraded (4.231%); 3540/172065 objects misplaced (2.057%) 2024-08-30T23:38:34.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/960904010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:34.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:34.373 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:34.373 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:34.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:34.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:34.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:34 smithi136 ceph-mon[20207]: osdmap e420: 12 total, 11 up, 12 in 2024-08-30T23:38:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:34 smithi136 ceph-mon[20207]: pgmap v1001: 17 pgs: 2 remapped+peering, 1 active+clean+remapped, 2 active+undersized+degraded, 12 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 7280/172065 objects degraded (4.231%); 3540/172065 objects misplaced (2.057%) 2024-08-30T23:38:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/960904010' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:34.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:34.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:34.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:34.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:34.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:34.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:34.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:34.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:34.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:35.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 59824 59808 70.5215 25.2188 0.00828862 0.0141719 2024-08-30T23:38:35.020 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:35.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:35.147 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:35.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:35.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:35 smithi173 ceph-mon[20554]: osdmap e421: 12 total, 11 up, 12 in 2024-08-30T23:38:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:35 smithi106 ceph-mon[16603]: osdmap e421: 12 total, 11 up, 12 in 2024-08-30T23:38:35.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:35.396 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:35.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:38:35.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:35.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:38:35.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:35.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:35 smithi136 ceph-mon[20207]: osdmap e421: 12 total, 11 up, 12 in 2024-08-30T23:38:35.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:35.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:35.720 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:38:35.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:35.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:35.798 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:35.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:35.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:35.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:36.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 60596 60580 70.109 48.25 0.0129493 0.014256 2024-08-30T23:38:36.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:36.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:36.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:36.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:36.194 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:36 smithi106 ceph-mon[16603]: pgmap v1003: 17 pgs: 3 remapped+peering, 1 active+clean+remapped, 2 active+undersized+degraded, 11 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 56 MiB/s wr, 894 op/s; 7533/176790 objects degraded (4.261%); 3540/176790 objects misplaced (2.002%) 2024-08-30T23:38:36.194 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7280/172065 objects degraded (4.231%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:36.194 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:36 smithi106 ceph-mon[16603]: osdmap e422: 12 total, 11 up, 12 in 2024-08-30T23:38:36.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:36.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:36.298 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:36.298 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:36.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:36.415 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:36.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:36 smithi136 ceph-mon[20207]: pgmap v1003: 17 pgs: 3 remapped+peering, 1 active+clean+remapped, 2 active+undersized+degraded, 11 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 56 MiB/s wr, 894 op/s; 7533/176790 objects degraded (4.261%); 3540/176790 objects misplaced (2.002%) 2024-08-30T23:38:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7280/172065 objects degraded (4.231%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:36 smithi136 ceph-mon[20207]: osdmap e422: 12 total, 11 up, 12 in 2024-08-30T23:38:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.541 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:36.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:36 smithi173 ceph-mon[20554]: pgmap v1003: 17 pgs: 3 remapped+peering, 1 active+clean+remapped, 2 active+undersized+degraded, 11 active+clean; 3.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 56 MiB/s wr, 894 op/s; 7533/176790 objects degraded (4.261%); 3540/176790 objects misplaced (2.002%) 2024-08-30T23:38:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7280/172065 objects degraded (4.231%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:38:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:36 smithi173 ceph-mon[20554]: osdmap e422: 12 total, 11 up, 12 in 2024-08-30T23:38:36.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:36.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:36.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:36.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:36.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:36.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:36.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:36.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:38:37.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 61693 61677 70.0808 68.5625 0.00327835 0.014263 2024-08-30T23:38:37.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:37.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:37.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:38:37.129 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:38:37.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:37.196 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:37.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:37.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3448078054' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:38:37.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3448078054' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:38:37.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:37.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3448078054' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:38:37.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:37.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:37.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:37.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:37.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:37.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:37.854 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:38:37.854 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:38:37.854 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-30T23:38:37.854 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:38:37.854 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:38:37.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-30T23:38:37.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:37.950 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:37.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:38.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 62200 62184 69.3952 31.6875 0.00353505 0.0142863 2024-08-30T23:38:38.021 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:38.030 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:38.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:38.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:38.126 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:38.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:38.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3448078054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:38:38.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: osdmap e423: 12 total, 11 up, 12 in 2024-08-30T23:38:38.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:38:38.159 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:38:38.159 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: pgmap v1006: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 remapped+peering, 2 active+undersized+degraded, 10 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 731 op/s; 7707/180426 objects degraded (4.272%); 7370/180426 objects misplaced (4.085%); 1.9 MiB/s, 30 objects/s recovering 2024-08-30T23:38:38.159 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:38:38.159 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:38 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:38:38.159 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:38:38.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:38.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:38.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:38.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.331 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:38.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:38.433 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3448078054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:38:38.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: osdmap e423: 12 total, 11 up, 12 in 2024-08-30T23:38:38.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:38:38.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:38:38.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: pgmap v1006: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 remapped+peering, 2 active+undersized+degraded, 10 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 731 op/s; 7707/180426 objects degraded (4.272%); 7370/180426 objects misplaced (4.085%); 1.9 MiB/s, 30 objects/s recovering 2024-08-30T23:38:38.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:38:38.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:38 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:38:38.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:38.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3448078054' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: osdmap e423: 12 total, 11 up, 12 in 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: pgmap v1006: 17 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+remapped, 3 remapped+peering, 2 active+undersized+degraded, 10 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 731 op/s; 7707/180426 objects degraded (4.272%); 7370/180426 objects misplaced (4.085%); 1.9 MiB/s, 30 objects/s recovering 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:38:38.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:38 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:38:38.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:38.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:38.688 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:38:38.688 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:38:38.688 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:38:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.711 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:38.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:38.759 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:38.759 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:38.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:38.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:38.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:38.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:38.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:39.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 62263 62247 68.2468 3.9375 0.0126096 0.0143033 2024-08-30T23:38:39.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:39.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:39.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:39.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:39.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:39.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:39.336 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:39 smithi106 ceph-mon[16603]: 5.d scrub starts 2024-08-30T23:38:39.336 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:39 smithi106 ceph-mon[16603]: 5.1 scrub starts 2024-08-30T23:38:39.336 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:39 smithi106 ceph-mon[16603]: osdmap e424: 12 total, 11 up, 12 in 2024-08-30T23:38:39.391 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:39 smithi136 ceph-mon[20207]: 5.d scrub starts 2024-08-30T23:38:39.391 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:39 smithi136 ceph-mon[20207]: 5.1 scrub starts 2024-08-30T23:38:39.391 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:39 smithi136 ceph-mon[20207]: osdmap e424: 12 total, 11 up, 12 in 2024-08-30T23:38:39.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:39.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:39.551 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:39 smithi173 ceph-mon[20554]: 5.d scrub starts 2024-08-30T23:38:39.551 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:39 smithi173 ceph-mon[20554]: 5.1 scrub starts 2024-08-30T23:38:39.551 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:39 smithi173 ceph-mon[20554]: osdmap e424: 12 total, 11 up, 12 in 2024-08-30T23:38:39.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:39.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:39.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:39.702 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:39.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:38:39.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:39.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:39.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:39.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:38:39.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:39.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:39.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:39.964 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:40.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 62286 62270 67.0949 1.4375 0.00283153 0.0143151 2024-08-30T23:38:40.012 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:40.012 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:40.012 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:40.034 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:40.034 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:40.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:40.057 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:40.057 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:40.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:40.077 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:40.077 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:40.100 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:40.122 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:40.144 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:40.144 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:40.165 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:40.166 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:40.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:40.187 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:40.188 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:40.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:40.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:40 smithi106 ceph-mon[16603]: 5.1 scrub ok 2024-08-30T23:38:40.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4103583728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:40.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:40 smithi106 ceph-mon[16603]: pgmap v1008: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 599 op/s; 11649/184710 objects degraded (6.307%); 14255/184710 objects misplaced (7.718%); 7.0 MiB/s, 112 objects/s recovering 2024-08-30T23:38:40.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:40.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:40.411 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:40.411 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:40.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:40.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:40.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.449 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:40.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:40.449 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:38:40.449 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:38:40.449 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:38:40.450 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:38:40.450 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:38:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:40 smithi136 ceph-mon[20207]: 5.1 scrub ok 2024-08-30T23:38:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4103583728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:40 smithi136 ceph-mon[20207]: pgmap v1008: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 599 op/s; 11649/184710 objects degraded (6.307%); 14255/184710 objects misplaced (7.718%); 7.0 MiB/s, 112 objects/s recovering 2024-08-30T23:38:40.550 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:40.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:40.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:40 smithi173 ceph-mon[20554]: 5.1 scrub ok 2024-08-30T23:38:40.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4103583728' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:40 smithi173 ceph-mon[20554]: pgmap v1008: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 599 op/s; 11649/184710 objects degraded (6.307%); 14255/184710 objects misplaced (7.718%); 7.0 MiB/s, 112 objects/s recovering 2024-08-30T23:38:40.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:40.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:40.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:40.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:40.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:40.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:40.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:40.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:41.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 62310 62294 65.9831 1.5 0.00285951 0.0143191 2024-08-30T23:38:41.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:41.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:41.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:41.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:41.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:41.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:41.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:41.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:38:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:38:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:41 smithi136 ceph-mon[20207]: osdmap e425: 12 total, 11 up, 12 in 2024-08-30T23:38:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 11649/184710 objects degraded (6.307%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:41 smithi136 ceph-mon[20207]: osdmap e426: 12 total, 11 up, 12 in 2024-08-30T23:38:41.540 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:41.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:38:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:38:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:41 smithi173 ceph-mon[20554]: osdmap e425: 12 total, 11 up, 12 in 2024-08-30T23:38:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 11649/184710 objects degraded (6.307%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:41 smithi173 ceph-mon[20554]: osdmap e426: 12 total, 11 up, 12 in 2024-08-30T23:38:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:38:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:38:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:41 smithi106 ceph-mon[16603]: osdmap e425: 12 total, 11 up, 12 in 2024-08-30T23:38:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 11649/184710 objects degraded (6.307%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:41 smithi106 ceph-mon[16603]: osdmap e426: 12 total, 11 up, 12 in 2024-08-30T23:38:41.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:41.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:41.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:41.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:41.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:41.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:41.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:41.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:42.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:38:42.003471+0000 min lat: 0.00108473 max lat: 9.72704 avg lat: 0.0143191 2024-08-30T23:38:42.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:38:42.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 62310 62294 64.8834 0 - 0.0143191 2024-08-30T23:38:42.024 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:42.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:42.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:42.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:42.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:42.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:42.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:42.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:42.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:42.456 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:42 smithi106 ceph-mon[16603]: 5.d scrub ok 2024-08-30T23:38:42.456 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:42 smithi106 ceph-mon[16603]: pgmap v1011: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 510 op/s; 11649/186501 objects degraded (6.246%); 14321/186501 objects misplaced (7.679%); 6.7 MiB/s, 107 objects/s recovering 2024-08-30T23:38:42.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:42 smithi136 ceph-mon[20207]: 5.d scrub ok 2024-08-30T23:38:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:42 smithi136 ceph-mon[20207]: pgmap v1011: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 510 op/s; 11649/186501 objects degraded (6.246%); 14321/186501 objects misplaced (7.679%); 6.7 MiB/s, 107 objects/s recovering 2024-08-30T23:38:42.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:42 smithi173 ceph-mon[20554]: 5.d scrub ok 2024-08-30T23:38:42.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:42 smithi173 ceph-mon[20554]: pgmap v1011: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 510 op/s; 11649/186501 objects degraded (6.246%); 14321/186501 objects misplaced (7.679%); 6.7 MiB/s, 107 objects/s recovering 2024-08-30T23:38:42.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:42.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:42.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:42.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:42.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:42.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:42.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:43.002 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 62310 62294 63.8197 0 - 0.0143191 2024-08-30T23:38:43.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:43.027 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:43.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:43.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:43.156 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:38:43.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:43.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:43.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3396625409' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:43.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:43.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:43.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3396625409' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:43.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3396625409' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:43.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:43.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:43.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:43.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:43.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:38:43.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:43.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:43.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:44.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 62310 62294 62.7903 0 - 0.0143191 2024-08-30T23:38:44.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:44.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:44.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:44.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:44.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:44.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:44.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:44.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:44.419 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:38:44.419 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:38:44.419 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T23:38:44.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:38:44.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:44 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:44.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3396625409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:44.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:44 smithi106 ceph-mon[16603]: osdmap e427: 12 total, 11 up, 12 in 2024-08-30T23:38:44.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:44 smithi106 ceph-mon[16603]: pgmap v1013: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 401 op/s; 11678/186888 objects degraded (6.249%); 13897/186888 objects misplaced (7.436%); 10 MiB/s, 166 objects/s recovering 2024-08-30T23:38:44.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:44.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:44 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3396625409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:44 smithi136 ceph-mon[20207]: osdmap e427: 12 total, 11 up, 12 in 2024-08-30T23:38:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:44 smithi136 ceph-mon[20207]: pgmap v1013: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 401 op/s; 11678/186888 objects degraded (6.249%); 13897/186888 objects misplaced (7.436%); 10 MiB/s, 166 objects/s recovering 2024-08-30T23:38:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:44 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:38:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3396625409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:44 smithi173 ceph-mon[20554]: osdmap e427: 12 total, 11 up, 12 in 2024-08-30T23:38:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:44 smithi173 ceph-mon[20554]: pgmap v1013: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 401 op/s; 11678/186888 objects degraded (6.249%); 13897/186888 objects misplaced (7.436%); 10 MiB/s, 166 objects/s recovering 2024-08-30T23:38:44.637 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:44.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:44.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:44.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:44.784 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:44.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:44.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:44.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:45.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 62310 62294 61.7936 0 - 0.0143191 2024-08-30T23:38:45.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:45.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:38:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:45.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:45.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:45.144 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:45.145 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:38:45.145 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:38:45.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:38:45.145 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:38:45.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.24460.0:193978 2.5 2:a2ee5e67:::benchmark_data_smithi173_47195_object96970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 542.387847588, 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.283382829, 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.24460", 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193978 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.146 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756146+0000", 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756148+0000", 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5370000000000001e-06 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756154+0000", 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756166+0000", 2024-08-30T23:38:45.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2041e-05 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756172+0000", 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2349999999999996e-06 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756333+0000", 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000160767 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756484+0000", 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015133799999999999 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.756627+0000", 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014263700000000001 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764342+0000", 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0077147439999999999 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.769055+0000", 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047136399999999998 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039012+0000", 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26995667699999998 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039055+0000", 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2567000000000001e-05 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039529+0000", 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00047478400000000001 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.152 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.24460.0:193978 2.5 e200/182 ondisk, result = 0)", 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 542.37989232200005, 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27543190299999998, 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764102+0000", 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764103+0000", 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.035e-06 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.764108+0000", 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9080000000000003e-06 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038928+0000", 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27481996400000003 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038937+0000", 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7860000000000008e-06 2024-08-30T23:38:45.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.038998+0000", 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0915999999999997e-05 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039007+0000", 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4319999999999999e-06 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.156 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.039534+0000", 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00052686200000000001 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(2.5 e200,182 2:a1c9afc1:::benchmark_data_smithi173_47195_object1378:head,146'6135)", 2024-08-30T23:38:45.157 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 542.36118411999996, 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42837196799999999, 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782810+0000", 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782811+0000", 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2799999999999996e-07 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.782816+0000", 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6969999999999998e-06 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210395+0000", 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42757847500000001 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210404+0000", 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0000000000000002e-06 2024-08-30T23:38:45.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.210493+0000", 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9438000000000007e-05 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.211182+0000", 2024-08-30T23:38:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068873000000000003 2024-08-30T23:38:45.162 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:2685 3.6 3:669aa85d:::benchmark_data_smithi173_60780_object2684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 512.45046039299996, 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22443937, 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:38:45.163 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 2685 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693534+0000", 2024-08-30T23:38:45.164 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693535+0000", 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.009e-06 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693573+0000", 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7843999999999999e-05 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.165 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693589+0000", 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6359e-05 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693603+0000", 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.412e-05 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693657+0000", 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3757e-05 2024-08-30T23:38:45.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693735+0000", 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8444999999999999e-05 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.693820+0000", 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4290999999999996e-05 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:38:45.167 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.694189+0000", 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036947500000000002 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.874073+0000", 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.179884508 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917876+0000", 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.043802633000000001 2024-08-30T23:38:45.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917931+0000", 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5297999999999998e-05 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:30:12.917973+0000", 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1631000000000002e-05 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.169 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176646 3.e 3:78ef2980:::benchmark_data_smithi173_60780_object73192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 340.40916041700001, 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22395779099999999, 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176646 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734834+0000", 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734835+0000", 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7399999999999991e-07 2024-08-30T23:38:45.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734839+0000", 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5800000000000002e-06 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734853+0000", 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.375e-05 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.172 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.734858+0000", 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738711+0000", 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003852847 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738850+0000", 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000138818 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.738954+0000", 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0001037 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748083+0000", 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0091293829999999996 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748375+0000", 2024-08-30T23:38:45.174 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000292427 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748430+0000", 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4191999999999999e-05 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.756943+0000", 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085137770000000002 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.175 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958661+0000", 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.201717853 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958707+0000", 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5840999999999997e-05 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958791+0000", 2024-08-30T23:38:45.176 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4349000000000002e-05 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:176647 3.e 3:751af89f:::benchmark_data_smithi173_60780_object73193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 340.40791165899998, 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.223004061, 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.177 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 176647 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.178 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736082+0000", 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736083+0000", 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8299999999999995e-07 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736088+0000", 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2370000000000003e-06 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736098+0000", 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0835e-05 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.736104+0000", 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742826+0000", 2024-08-30T23:38:45.180 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006721505 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.742967+0000", 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140977 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,4,9,11", 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.743123+0000", 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015588399999999999 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:38:45.181 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.748383+0000", 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052605059999999999 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.758513+0000", 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010130116 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.767529+0000", 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090161789999999992 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.182 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.771374+0000", 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038445660000000002 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958849+0000", 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.18747534399999999 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.958930+0000", 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0463000000000005e-05 2024-08-30T23:38:45.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:04.959086+0000", 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000156701 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:31057 4.b 4:d5378b27:::benchmark_data_smithi173_74914_object31056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 264.45663612200002, 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.641950197, 2024-08-30T23:38:45.184 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 31057 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687360+0000", 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0669999999999999e-06 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687397+0000", 2024-08-30T23:38:45.186 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6677999999999997e-05 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687413+0000", 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532999999999999e-05 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687421+0000", 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9410000000000007e-06 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.187 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687486+0000", 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4894000000000006e-05 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687630+0000", 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014421900000000001 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687780+0000", 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000149748 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689089+0000", 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013092609999999999 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689219+0000", 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013019 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329200+0000", 2024-08-30T23:38:45.189 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63998067999999997 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329262+0000", 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1508999999999997e-05 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329308+0000", 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6477000000000002e-05 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.0 319/318 [PushOp(4:08a755c5:::benchmark_data_smithi173_74914_object390:head, version: 311'18, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:08a755c5:::benchmark_data_smithi173_74914_object390:head@311'18, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 262.220164195, 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29680203999999999, 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:38:45.191 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923830+0000", 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923831+0000", 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:22.923901+0000", 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0507999999999996e-05 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219654+0000", 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.295752615 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219681+0000", 2024-08-30T23:38:45.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7390999999999999e-05 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219790+0000", 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108904 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.219807+0000", 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6871000000000002e-05 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:23.220632+0000", 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082508799999999997 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:61123 4.b 4:d896c312:::benchmark_data_smithi173_74914_object61122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 230.93059679199999, 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93993815199999997, 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 61123 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213400+0000", 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5000000000000002e-06 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213421+0000", 2024-08-30T23:38:45.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1537000000000001e-05 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213434+0000", 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2478e-05 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213440+0000", 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8699999999999997e-06 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213485+0000", 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5439999999999999e-05 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213576+0000", 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0767999999999996e-05 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213667+0000", 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1135000000000001e-05 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215888+0000", 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022207139999999999 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215938+0000", 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0022000000000003e-05 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153264+0000", 2024-08-30T23:38:45.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93732649300000004 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153304+0000", 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0139999999999999e-05 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153335+0000", 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1055e-05 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:61123 4.b e335/306 ondisk, result = 0)", 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 230.92795599300001, 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93730281299999996, 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216039+0000", 2024-08-30T23:38:45.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4099999999999999e-07 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216042+0000", 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7529999999999999e-06 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153195+0000", 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93715309700000005 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153202+0000", 2024-08-30T23:38:45.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7340000000000002e-06 2024-08-30T23:38:45.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153255+0000", 2024-08-30T23:38:45.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:45.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2420999999999998e-05 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153260+0000", 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7370000000000003e-06 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153341+0000", 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0530000000000003e-05 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.5 335/318 [PushOp(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head, version: 316'1784, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head@316'1784, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:38:45.208 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 229.96974338999999, 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.83137486000000005, 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174252+0000", 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174402+0000", 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015002700000000001 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003927+0000", 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82952564699999998 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003946+0000", 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8678999999999999e-05 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004043+0000", 2024-08-30T23:38:45.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6718999999999993e-05 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004054+0000", 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0925e-05 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.005626+0000", 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001571813 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98225 4.5 e353/337 4:ad425262:::benchmark_data_smithi173_74914_object98224:head v 353'6223, mlcod=353'6220)", 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.69902454999999, 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32306562100000003, 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:38:45.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444971+0000", 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0699999999999999e-06 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.214 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.445067+0000", 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5938000000000002e-05 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765335+0000", 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32026851299999998 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765347+0000", 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.151e-05 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765448+0000", 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100967 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765468+0000", 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0219000000000001e-05 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.767987+0000", 2024-08-30T23:38:45.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025194890000000002 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.768035+0000", 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7914999999999998e-05 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98228 4.7 e353/329 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head v 353'6056, mlcod=353'6052)", 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.695887865, 2024-08-30T23:38:45.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35600754000000001, 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448107+0000", 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3300000000000001e-07 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448202+0000", 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4503e-05 2024-08-30T23:38:45.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801211+0000", 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35300926399999999 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801228+0000", 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6694999999999999e-05 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801373+0000", 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014589400000000001 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801409+0000", 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6025999999999997e-05 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804078+0000", 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026682939999999999 2024-08-30T23:38:45.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804114+0000", 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6031000000000001e-05 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:143754 4.0 e365/329 4:0c834e11:::benchmark_data_smithi173_74914_object45512:head v 365'9153, mlcod=365'9150)", 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:19.176383+0000", 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 145.96761096399999, 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.225303222, 2024-08-30T23:38:45.222 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.223 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.223 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176383+0000", 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176383+0000", 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176385+0000", 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4309999999999999e-06 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.176394+0000", 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8230000000000006e-06 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398124+0000", 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22172945599999999 2024-08-30T23:38:45.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398137+0000", 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3519999999999999e-05 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398255+0000", 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011801100000000001 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.398297+0000", 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1956000000000003e-05 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.401655+0000", 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003358069 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.401686+0000", 2024-08-30T23:38:45.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0956e-05 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:143768 4.3 e365/306 4:c93e37ef:::benchmark_data_smithi173_74914_object45526:head v 365'8833, mlcod=365'8830)", 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 145.712027882, 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.374659609, 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.228 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431967+0000", 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5000000000000001e-07 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431975+0000", 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4139999999999997e-06 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.793620+0000", 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36164507800000001 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.793628+0000", 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000005e-06 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.794714+0000", 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010868989999999999 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.794732+0000", 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7396000000000001e-05 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.806585+0000", 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011852713000000001 2024-08-30T23:38:45.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.806626+0000", 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1248999999999998e-05 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:31249 5.8 e406/394 5:11e97cde:::benchmark_data_smithi173_87639_object31248:head v 406'1968, mlcod=406'1965)", 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:01.238485+0000", 2024-08-30T23:38:45.233 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.905508591, 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20899014199999999, 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.238485+0000", 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.238485+0000", 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.238486+0000", 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1100000000000001e-07 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.238555+0000", 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8844999999999996e-05 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.445668+0000", 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20711373499999999 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.445683+0000", 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4178999999999999e-05 2024-08-30T23:38:45.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.445785+0000", 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101909 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.445805+0000", 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0758e-05 2024-08-30T23:38:45.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.447441+0000", 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016361080000000001 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.447476+0000", 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4196999999999998e-05 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.238 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:47905 5.5 e415/389 5:a41778e5:::benchmark_data_smithi173_87639_object47904:head v 415'3097, mlcod=415'3095)", 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:23.049722+0000", 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 22.094272082, 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21226472900000001, 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.049722+0000", 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.049722+0000", 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.049723+0000", 2024-08-30T23:38:45.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5700000000000002e-07 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.049780+0000", 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7739999999999999e-05 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.258938+0000", 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20915741299999999 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.258949+0000", 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1032e-05 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.259012+0000", 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3178999999999999e-05 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.259034+0000", 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.215e-05 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.261950+0000", 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029157570000000002 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.261987+0000", 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6801000000000002e-05 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.243 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:47928 5.8 e415/394 5:118ac48f:::benchmark_data_smithi173_87639_object47927:head v 415'3008, mlcod=415'3005)", 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:23.062779+0000", 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 22.081215414999999, 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20133398999999999, 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.062779+0000", 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.062779+0000", 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.062780+0000", 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1500000000000003e-07 2024-08-30T23:38:45.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.062907+0000", 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000127785 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.262681+0000", 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19977336400000001 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.262688+0000", 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.041e-06 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.262745+0000", 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7479000000000003e-05 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.262763+0000", 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8039000000000001e-05 2024-08-30T23:38:45.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.264081+0000", 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013177480000000001 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:23.264113+0000", 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1619000000000003e-05 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:58925 5.8 e418/394 5:1dcd983f:::benchmark_data_smithi173_87639_object58924:head v 418'3666, mlcod=418'3663)", 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:31.869947+0000", 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.274047018999999, 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20185862600000001, 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:31.869947+0000", 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:31.869947+0000", 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:31.869947+0000", 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4099999999999999e-07 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:31.870055+0000", 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107118 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.069946+0000", 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19989181 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.069959+0000", 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2666999999999999e-05 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.070088+0000", 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012865000000000001 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.070153+0000", 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5382000000000005e-05 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.071776+0000", 2024-08-30T23:38:45.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016233739999999999 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.071806+0000", 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9184e-05 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:58980 5.8 e418/394)", 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:32.105688+0000", 2024-08-30T23:38:45.253 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.038306234, 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.458888293, 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105688+0000", 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:38:45.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105688+0000", 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105689+0000", 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4399999999999999e-07 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105791+0000", 2024-08-30T23:38:45.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102163 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564508+0000", 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45871740599999999 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564515+0000", 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3150000000000003e-06 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564567+0000", 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1997999999999999e-05 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564576+0000", 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6670000000000002e-06 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.257 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.258 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:38:45.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:45.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:45.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:45.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:45.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:45.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:45.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:45.703 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:45.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:45.826 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:45 smithi106 ceph-mon[16603]: pgmap v1014: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 123 op/s; 11679/186936 objects degraded (6.248%); 13451/186936 objects misplaced (7.196%); 9.0 MiB/s, 144 objects/s recovering 2024-08-30T23:38:45.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:45.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:45.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 428, 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:38:45.691146+0000", 2024-08-30T23:38:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:36:27.655854+0000", 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:38:32.101509+0000", 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5799936, 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 26, 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:38:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 5, 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "jewel", 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:38:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:38:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 1, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 1, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 1, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 1, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 1, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:38:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "343", 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:38:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:38:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:38:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:38:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 10, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 10, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 0.30000001192092896, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.77478784322738647, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.93000000715255737, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.76999998092651367 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 5, 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_3", 2024-08-30T23:38:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:37:32.432920+0000", 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:38:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "392", 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:38:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:38:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:38:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.0699999332427979, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 2.5, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.380000114440918, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.75, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.77478784322738647, 2024-08-30T23:38:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.93000000715255737, 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.76999998092651367, 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "error_message": "pool 5 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.923797607421875, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 306, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:38:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:38:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:38:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:38:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:38:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.5043182373046875, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 13, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 332, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1861847252 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/1861847252", 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/1861847252", 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/1861847252", 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/1861847252", 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 420, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:38:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.8072052001953125, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 389, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:38:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:38:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.527435302734375, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 419, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:38:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 39, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 420, 2024-08-30T23:38:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2282248833 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2282248833", 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2282248833", 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2282248833", 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2282248833", 2024-08-30T23:38:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7101593017578125, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 420, 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 389, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:38:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 206, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:38:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 394, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:38:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 420, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.589813232421875, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:38:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 394, 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:38:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:38:46.053 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.054 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:49.855958+0000", 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:35:29.938964+0000", 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.055 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:38:39.979238+0000", 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:28.761800+0000", 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.057 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:45.024203+0000", 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:38:37.399476+0000", 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.058 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:43.562361+0000", 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.059 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:54.812182+0000", 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.060 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:46.062 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:45 smithi136 ceph-mon[20207]: pgmap v1014: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 123 op/s; 11679/186936 objects degraded (6.248%); 13451/186936 objects misplaced (7.196%); 9.0 MiB/s, 144 objects/s recovering 2024-08-30T23:38:46.062 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 62310 62294 60.828 0 - 0.0143191 2024-08-30T23:38:46.062 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:38:37.766071+0000", 2024-08-30T23:38:46.062 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:22.931472+0000", 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.063 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:45.032895+0000", 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:38:46.064 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:37:46.049864+0000", 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [], 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [ 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "5.0", 2024-08-30T23:38:46.065 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: 10, 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: 4, 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: 0 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "5.2", 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: 0 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "5.4", 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: 5, 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: 2 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "5.8", 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: 9 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "5.a", 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: 8 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:38:46.068 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:38:46.069 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:38:46.070 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:38:46.071 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:38:46.071 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:38:46.071 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:38:46.071 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:38:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:45 smithi173 ceph-mon[20554]: pgmap v1014: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s wr, 123 op/s; 11679/186936 objects degraded (6.248%); 13451/186936 objects misplaced (7.196%); 9.0 MiB/s, 144 objects/s recovering 2024-08-30T23:38:46.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:46.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:46.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:46.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:46.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:46.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:46.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:46.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:46.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:38:46.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:46.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:46.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:46.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:46.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 11679/186936 objects degraded (6.248%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:46 smithi106 ceph-mon[16603]: osdmap e428: 12 total, 11 up, 12 in 2024-08-30T23:38:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/976691887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:38:46.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:46.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:46.987 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:46.999 INFO:tasks.thrashosds.thrasher:j is {'epoch': 428, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:38:45.691146+0000', 'last_up_change': '2024-08-30T23:36:27.655854+0000', 'last_in_change': '2024-08-30T23:38:32.101509+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '343', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 10, 'score_stable': 10, 'optimal_score': 0.30000001192092896, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.7747878432273865, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-08-30T23:37:32.432920+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '392', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.069999933242798, 'score_stable': 2.5, 'optimal_score': 1.100000023841858, 'raw_score_acting': 3.380000114440918, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.7747878432273865, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.7699999809265137, 'error_message': 'pool 5 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 0.923797607421875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 0, 'in': 1, 'weight': 0.5043182373046875, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 332, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 1861847252}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 1861847252}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 1861847252}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 1861847252}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 1861847252}]}, 'public_addr': '172.21.15.106:6811/1861847252', 'cluster_addr': '172.21.15.106:6813/1861847252', 'heartbeat_back_addr': '172.21.15.106:6817/1861847252', 'heartbeat_front_addr': '172.21.15.106:6815/1861847252', 'state': ['exists']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 0.8072052001953125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 389, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 0.527435302734375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 419, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2282248833}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2282248833}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2282248833}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2282248833}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2282248833}]}, 'public_addr': '172.21.15.136:6809/2282248833', 'cluster_addr': '172.21.15.136:6811/2282248833', 'heartbeat_back_addr': '172.21.15.136:6815/2282248833', 'heartbeat_front_addr': '172.21.15.136:6813/2282248833', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 0.7101593017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 389, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 394, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 0.589813232421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 394, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:49.855958+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:35:29.938964+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:38:39.979238+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:28.761800+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:45.024203+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:38:37.399476+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:43.562361+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:54.812182+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:38:37.766071+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:22.931472+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:45.032895+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:37:46.049864+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '5.0', 'osds': [10, 4, 0]}, {'pgid': '5.2', 'osds': [6, 0]}, {'pgid': '5.4', 'osds': [5, 2]}, {'pgid': '5.8', 'osds': [2, 0, 9]}, {'pgid': '5.a', 'osds': [2, 8]}], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:38:46.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:38:47.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 62310 62294 59.8922 0 - 0.0143191 2024-08-30T23:38:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 11679/186936 objects degraded (6.248%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:46 smithi136 ceph-mon[20207]: osdmap e428: 12 total, 11 up, 12 in 2024-08-30T23:38:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/976691887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:38:47.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 11679/186936 objects degraded (6.248%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:46 smithi173 ceph-mon[20554]: osdmap e428: 12 total, 11 up, 12 in 2024-08-30T23:38:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/976691887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:38:47.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:47.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:47.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:47.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:47.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:47.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:47.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:47.512 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:47.531 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:47.531 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:47.531 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:47.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:47.548 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:47.558 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:47.558 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:47.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:47.583 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:47.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:47.604 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:47.604 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:47.626 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:47.626 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:47.651 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:47.651 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:47.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:47.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:47.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3357495417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:47.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:47 smithi106 ceph-mon[16603]: pgmap v1016: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 42 op/s; 11679/186936 objects degraded (6.248%); 12984/186936 objects misplaced (6.946%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:38:47.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:47.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:47.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:47.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:47.941 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:48.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 62310 62294 58.9848 0 - 0.0143191 2024-08-30T23:38:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3357495417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:47 smithi136 ceph-mon[20207]: pgmap v1016: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 42 op/s; 11679/186936 objects degraded (6.248%); 12984/186936 objects misplaced (6.946%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:38:48.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:48.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3357495417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:47 smithi173 ceph-mon[20554]: pgmap v1016: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 42 op/s; 11679/186936 objects degraded (6.248%); 12984/186936 objects misplaced (6.946%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:38:48.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:48.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:48.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:38:48.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:48.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:48.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:48.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:48.475 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:48.475 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:48.497 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:48.497 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:48.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:48.519 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:48.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:48.541 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:48.541 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:48.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:48.562 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:48.562 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:48.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:48.677 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:38:48.678 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:38:48.678 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:48 smithi106 ceph-mon[16603]: osdmap e429: 12 total, 11 up, 12 in 2024-08-30T23:38:48.679 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:48.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:48.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:48.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:48.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:48.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:49.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 62310 62294 58.1044 0 - 0.0143191 2024-08-30T23:38:49.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:38:49.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:38:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:48 smithi136 ceph-mon[20207]: osdmap e429: 12 total, 11 up, 12 in 2024-08-30T23:38:49.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:38:49.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:38:49.046 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:48 smithi173 ceph-mon[20554]: osdmap e429: 12 total, 11 up, 12 in 2024-08-30T23:38:49.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:49.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:49.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:49.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:49.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:49.386 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:49.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:49.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:49.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:49.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:49.623 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:38:49.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:49.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:49.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:49.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:49.852 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:38:49.853 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:38:49.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:49.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:49.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:50.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 62310 62294 57.25 0 - 0.0143191 2024-08-30T23:38:50.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:50.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/7170632' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:49 smithi136 ceph-mon[20207]: pgmap v1018: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s wr, 3 op/s; 11679/186936 objects degraded (6.248%); 12277/186936 objects misplaced (6.567%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:38:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/7170632' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:49 smithi173 ceph-mon[20554]: pgmap v1018: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s wr, 3 op/s; 11679/186936 objects degraded (6.248%); 12277/186936 objects misplaced (6.567%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:38:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/7170632' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:38:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:49 smithi106 ceph-mon[16603]: pgmap v1018: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s wr, 3 op/s; 11679/186936 objects degraded (6.248%); 12277/186936 objects misplaced (6.567%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:38:50.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:50.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:50.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:50.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:50.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:50.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:50.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:50.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.533 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:50.619 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:50.619 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/7170632' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:50.619 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: osdmap e430: 12 total, 11 up, 12 in 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:38:50.620 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:50 smithi106 ceph-mon[16603]: from='client.16866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:38:50.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:50.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:50.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:50.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:50.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:50.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:50.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:51.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 62310 62294 56.4203 0 - 0.0143191 2024-08-30T23:38:51.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:51.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/7170632' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: osdmap e430: 12 total, 11 up, 12 in 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:38:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:50 smithi136 ceph-mon[20207]: from='client.16866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:38:51.042 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:51.068 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1019,"stamp":"2024-08-30T23:38:49.626716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4084138528,"num_objects":62312,"num_object_clones":0,"num_object_copies":186936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11679,"num_objects_misplaced":12277,"num_objects_unfound":0,"num_objects_dirty":62312,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":62433,"num_write_kb":3989990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2655,"num_bytes_recovered":175191648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62402,"ondisk_log_size":62402,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16324652,"kb_used_data":12812848,"kb_used_omap":18,"kb_used_meta":3511533,"kb_avail":1108617172,"statfs":{"total":1151940427776,"available":1135223984128,"internally_reserved":0,"allocated":13120356352,"data_stored":13115247162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":3595810105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":-1620,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1620,"num_bytes_recovered":106168320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.118702"},"pg_stats":[{"pgid":"5.f","version":"425'3856","reported_seq":3947,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-08-30T23:38:47.629451+0000","last_change":"2024-08-30T23:37:34.040566+0000","last_active":"2024-08-30T23:38:47.629451+0000","last_peered":"2024-08-30T23:38:47.629451+0000","last_clean":"2024-08-30T23:38:47.629451+0000","last_became_active":"2024-08-30T23:37:34.040273+0000","last_became_peered":"2024-08-30T23:37:34.040273+0000","last_unstale":"2024-08-30T23:38:47.629451+0000","last_undegraded":"2024-08-30T23:38:47.629451+0000","last_fullsized":"2024-08-30T23:38:47.629451+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:00.148306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252706816,"num_objects":3856,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3856,"num_write_kb":246784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,2],"acting":[5,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"425'3750","reported_seq":3837,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-30T23:38:44.280790+0000","last_change":"2024-08-30T23:37:34.072395+0000","last_active":"2024-08-30T23:38:44.280790+0000","last_peered":"2024-08-30T23:38:44.280790+0000","last_clean":"2024-08-30T23:38:44.280790+0000","last_became_active":"2024-08-30T23:37:34.071585+0000","last_became_peered":"2024-08-30T23:37:34.071585+0000","last_unstale":"2024-08-30T23:38:44.280790+0000","last_undegraded":"2024-08-30T23:38:44.280790+0000","last_fullsized":"2024-08-30T23:38:44.280790+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3750,"log_dups_size":0,"ondisk_log_size":3750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:52.584305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245760000,"num_objects":3750,"num_object_clones":0,"num_object_copies":11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3750,"num_write_kb":240000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"425'3876","reported_seq":4011,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-08-30T23:38:45.883685+0000","last_change":"2024-08-30T23:38:40.517933+0000","last_active":"2024-08-30T23:38:45.883685+0000","last_peered":"2024-08-30T23:38:45.883685+0000","last_clean":"2024-08-30T23:38:45.883685+0000","last_became_active":"2024-08-30T23:37:34.040753+0000","last_became_peered":"2024-08-30T23:37:34.040753+0000","last_unstale":"2024-08-30T23:38:45.883685+0000","last_undegraded":"2024-08-30T23:38:45.883685+0000","last_fullsized":"2024-08-30T23:38:45.883685+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'3876","last_scrub_stamp":"2024-08-30T23:38:40.517849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:40.517849+0000","objects_scrubbed":3874,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:53.360081+0000","scrub_duration":2833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254017536,"num_objects":3876,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":248064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"424'4020","reported_seq":4107,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-30T23:38:44.287190+0000","last_change":"2024-08-30T23:37:34.072314+0000","last_active":"2024-08-30T23:38:44.287190+0000","last_peered":"2024-08-30T23:38:44.287190+0000","last_clean":"2024-08-30T23:38:44.287190+0000","last_became_active":"2024-08-30T23:37:34.071745+0000","last_became_peered":"2024-08-30T23:37:34.071745+0000","last_unstale":"2024-08-30T23:38:44.287190+0000","last_undegraded":"2024-08-30T23:38:44.287190+0000","last_fullsized":"2024-08-30T23:38:44.287190+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4020,"log_dups_size":0,"ondisk_log_size":4020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:51.834379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263454720,"num_objects":4020,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4020,"num_write_kb":257280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"425'3974","reported_seq":4061,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-30T23:38:44.280783+0000","last_change":"2024-08-30T23:37:34.072046+0000","last_active":"2024-08-30T23:38:44.280783+0000","last_peered":"2024-08-30T23:38:44.280783+0000","last_clean":"2024-08-30T23:38:44.280783+0000","last_became_active":"2024-08-30T23:37:34.071651+0000","last_became_peered":"2024-08-30T23:37:34.071651+0000","last_unstale":"2024-08-30T23:38:44.280783+0000","last_undegraded":"2024-08-30T23:38:44.280783+0000","last_fullsized":"2024-08-30T23:38:44.280783+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:02.689327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260440064,"num_objects":3974,"num_object_clones":0,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":254336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"425'3917","reported_seq":3999,"reported_epoch":429,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:38:47.619227+0000","last_change":"2024-08-30T23:38:34.267437+0000","last_active":"2024-08-30T23:38:47.619227+0000","last_peered":"2024-08-30T23:38:47.619227+0000","last_clean":"2024-08-30T23:38:32.087384+0000","last_became_active":"2024-08-30T23:38:34.258131+0000","last_became_peered":"2024-08-30T23:38:34.258131+0000","last_unstale":"2024-08-30T23:38:47.619227+0000","last_undegraded":"2024-08-30T23:38:34.136754+0000","last_fullsized":"2024-08-30T23:38:34.117998+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3917,"log_dups_size":0,"ondisk_log_size":3917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256704512,"num_objects":3917,"num_object_clones":0,"num_object_copies":11751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3917,"num_write_kb":250688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[2,8],"avail_no_missing":["2","8","11"],"object_location_counts":[{"shards":"2,8","objects":203},{"shards":"2,8,11","objects":3714}],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"424'3908","reported_seq":3997,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-08-30T23:38:45.883579+0000","last_change":"2024-08-30T23:37:34.038908+0000","last_active":"2024-08-30T23:38:45.883579+0000","last_peered":"2024-08-30T23:38:45.883579+0000","last_clean":"2024-08-30T23:38:45.883579+0000","last_became_active":"2024-08-30T23:37:34.038547+0000","last_became_peered":"2024-08-30T23:37:34.038547+0000","last_unstale":"2024-08-30T23:38:45.883579+0000","last_undegraded":"2024-08-30T23:38:45.883579+0000","last_fullsized":"2024-08-30T23:38:45.883579+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3908,"log_dups_size":0,"ondisk_log_size":3908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:50.587122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256114688,"num_objects":3908,"num_object_clones":0,"num_object_copies":11724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3908,"num_write_kb":250112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"424'3883","reported_seq":5466,"reported_epoch":429,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:38:48.198334+0000","last_change":"2024-08-30T23:38:34.309882+0000","last_active":"2024-08-30T23:38:48.198334+0000","last_peered":"2024-08-30T23:38:48.198334+0000","last_clean":"2024-08-30T23:38:32.105026+0000","last_became_active":"2024-08-30T23:38:34.257797+0000","last_became_peered":"2024-08-30T23:38:34.257797+0000","last_unstale":"2024-08-30T23:38:48.198334+0000","last_undegraded":"2024-08-30T23:38:48.198334+0000","last_fullsized":"2024-08-30T23:38:48.198334+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254476288,"num_objects":3883,"num_object_clones":0,"num_object_copies":11649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2921,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3883,"num_write_kb":248512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":748,"num_bytes_recovered":49020928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[2,0,9],"avail_no_missing":["2","0","9"],"object_location_counts":[{"shards":"0,2,9","objects":962}],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"424'3961","reported_seq":4049,"reported_epoch":429,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:38:47.629456+0000","last_change":"2024-08-30T23:37:42.009508+0000","last_active":"2024-08-30T23:38:47.629456+0000","last_peered":"2024-08-30T23:38:47.629456+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.039578+0000","last_became_peered":"2024-08-30T23:37:34.039578+0000","last_unstale":"2024-08-30T23:38:47.629456+0000","last_undegraded":"2024-08-30T23:37:42.008557+0000","last_fullsized":"2024-08-30T23:37:34.037051+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:53.741229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259588096,"num_objects":3961,"num_object_clones":0,"num_object_copies":11883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":253504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":["5","9"],"object_location_counts":[{"shards":"5,9","objects":3961}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"425'3818","reported_seq":3893,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-08-30T23:38:47.629431+0000","last_change":"2024-08-30T23:37:34.040398+0000","last_active":"2024-08-30T23:38:47.629431+0000","last_peered":"2024-08-30T23:38:47.629431+0000","last_clean":"2024-08-30T23:38:47.629431+0000","last_became_active":"2024-08-30T23:37:34.039974+0000","last_became_peered":"2024-08-30T23:37:34.039974+0000","last_unstale":"2024-08-30T23:38:47.629431+0000","last_undegraded":"2024-08-30T23:38:47.629431+0000","last_fullsized":"2024-08-30T23:38:47.629431+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3818,"log_dups_size":0,"ondisk_log_size":3818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:54.643521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250216448,"num_objects":3818,"num_object_clones":0,"num_object_copies":11454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3818,"num_write_kb":244352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"425'4004","reported_seq":4089,"reported_epoch":428,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:38:45.727343+0000","last_change":"2024-08-30T23:37:42.007680+0000","last_active":"2024-08-30T23:38:45.727343+0000","last_peered":"2024-08-30T23:38:45.727343+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.071741+0000","last_became_peered":"2024-08-30T23:37:34.071741+0000","last_unstale":"2024-08-30T23:38:45.727343+0000","last_undegraded":"2024-08-30T23:37:42.007336+0000","last_fullsized":"2024-08-30T23:37:34.037577+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:58.233137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262406144,"num_objects":4004,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4004,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4004,"num_write_kb":256256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":4004}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"424'3879","reported_seq":3967,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-08-30T23:38:45.727326+0000","last_change":"2024-08-30T23:37:34.071270+0000","last_active":"2024-08-30T23:38:45.727326+0000","last_peered":"2024-08-30T23:38:45.727326+0000","last_clean":"2024-08-30T23:38:45.727326+0000","last_became_active":"2024-08-30T23:37:34.070943+0000","last_became_peered":"2024-08-30T23:37:34.070943+0000","last_unstale":"2024-08-30T23:38:45.727326+0000","last_undegraded":"2024-08-30T23:38:45.727326+0000","last_fullsized":"2024-08-30T23:38:45.727326+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3879,"log_dups_size":0,"ondisk_log_size":3879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254214144,"num_objects":3879,"num_object_clones":0,"num_object_copies":11637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3879,"num_write_kb":248256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"424'3843","reported_seq":5506,"reported_epoch":428,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:38:47.033825+0000","last_change":"2024-08-30T23:38:34.286521+0000","last_active":"2024-08-30T23:38:47.033825+0000","last_peered":"2024-08-30T23:38:47.033825+0000","last_clean":"2024-08-30T23:38:31.890254+0000","last_became_active":"2024-08-30T23:38:34.258074+0000","last_became_peered":"2024-08-30T23:38:34.258074+0000","last_unstale":"2024-08-30T23:38:47.033825+0000","last_undegraded":"2024-08-30T23:38:47.033825+0000","last_fullsized":"2024-08-30T23:38:34.119189+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3843,"log_dups_size":0,"ondisk_log_size":3843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251854848,"num_objects":3843,"num_object_clones":0,"num_object_copies":11529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2849,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3843,"num_write_kb":245952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":51249152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":2849},{"shards":"0,6","objects":994}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"424'3807","reported_seq":3944,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-08-30T23:38:47.751021+0000","last_change":"2024-08-30T23:38:39.040065+0000","last_active":"2024-08-30T23:38:47.751021+0000","last_peered":"2024-08-30T23:38:47.751021+0000","last_clean":"2024-08-30T23:38:47.751021+0000","last_became_active":"2024-08-30T23:37:34.043753+0000","last_became_peered":"2024-08-30T23:37:34.043753+0000","last_unstale":"2024-08-30T23:38:47.751021+0000","last_undegraded":"2024-08-30T23:38:47.751021+0000","last_fullsized":"2024-08-30T23:38:47.751021+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'3805","last_scrub_stamp":"2024-08-30T23:38:39.039968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:39.039968+0000","objects_scrubbed":3803,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:40:03.713913+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249495552,"num_objects":3807,"num_object_clones":0,"num_object_copies":11421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":243648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"424'3938","reported_seq":4020,"reported_epoch":428,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:38:45.719298+0000","last_change":"2024-08-30T23:38:34.132086+0000","last_active":"2024-08-30T23:38:45.719298+0000","last_peered":"2024-08-30T23:38:45.719298+0000","last_clean":"2024-08-30T23:38:45.719298+0000","last_became_active":"2024-08-30T23:38:34.126744+0000","last_became_peered":"2024-08-30T23:38:34.126744+0000","last_unstale":"2024-08-30T23:38:45.719298+0000","last_undegraded":"2024-08-30T23:38:45.719298+0000","last_fullsized":"2024-08-30T23:38:45.719298+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3938,"log_dups_size":0,"ondisk_log_size":3938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:03.002777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258080768,"num_objects":3938,"num_object_clones":0,"num_object_copies":11814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3938,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3938,"num_write_kb":252032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4,0],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":3938}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"425'3876","reported_seq":6198,"reported_epoch":429,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:38:48.767605+0000","last_change":"2024-08-30T23:38:34.270052+0000","last_active":"2024-08-30T23:38:48.767605+0000","last_peered":"2024-08-30T23:38:48.767605+0000","last_clean":"2024-08-30T23:38:32.091146+0000","last_became_active":"2024-08-30T23:38:34.242472+0000","last_became_peered":"2024-08-30T23:38:34.242472+0000","last_unstale":"2024-08-30T23:38:48.767605+0000","last_undegraded":"2024-08-30T23:38:48.767605+0000","last_fullsized":"2024-08-30T23:38:34.119231+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":3876,"log_dups_size":0,"ondisk_log_size":3876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254017536,"num_objects":3876,"num_object_clones":0,"num_object_copies":11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2569,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":248064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1119,"num_bytes_recovered":73334784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[5,2],"avail_no_missing":["5","2","8"],"object_location_counts":[{"shards":"2,5","objects":1307},{"shards":"2,5,8","objects":2569}],"blocked_by":[],"up_primary":10,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":843,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-08-30T23:38:45.883556+0000","last_change":"2024-08-30T23:38:00.011325+0000","last_active":"2024-08-30T23:38:45.883556+0000","last_peered":"2024-08-30T23:38:45.883556+0000","last_clean":"2024-08-30T23:38:45.883556+0000","last_became_active":"2024-08-30T23:34:35.630884+0000","last_became_peered":"2024-08-30T23:34:35.630884+0000","last_unstale":"2024-08-30T23:38:45.883556+0000","last_undegraded":"2024-08-30T23:38:45.883556+0000","last_fullsized":"2024-08-30T23:38:45.883556+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:38:00.011247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:38:00.011247+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:03.137916+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4083548160,"num_objects":62310,"num_object_clones":0,"num_object_copies":186930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11679,"num_objects_misplaced":12277,"num_objects_unfound":0,"num_objects_dirty":62310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62310,"num_write_kb":3987840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":173604864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11888689152,"data_stored":11888689152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62310,"ondisk_log_size":62310,"up":45,"acting":43,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187627,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330640,"kb_used_data":984312,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92414512,"statfs":{"total":95995035648,"available":94632460288,"internally_reserved":0,"allocated":1007935488,"data_stored":1007495573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":354613700},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449266,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623504,"kb_used_data":439612,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93121648,"statfs":{"total":95995035648,"available":95356567552,"internally_reserved":0,"allocated":450162688,"data_stored":449721950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":188283328},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710901,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1611076,"kb_used_data":1241136,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92134076,"statfs":{"total":95995035648,"available":94345293824,"internally_reserved":0,"allocated":1270923264,"data_stored":1270486921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":378796480},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907588,"kb_used_data":727856,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92837564,"statfs":{"total":95995035648,"available":95065665536,"internally_reserved":0,"allocated":745324544,"data_stored":744888201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234174,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1342872,"kb_used_data":986820,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92402280,"statfs":{"total":95995035648,"available":94619934720,"internally_reserved":0,"allocated":1010503680,"data_stored":1010059390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":364575178},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528516,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529496,"kb_used_data":1259524,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92215656,"statfs":{"total":95995035648,"available":94428831744,"internally_reserved":0,"allocated":1289752576,"data_stored":1289308286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724744,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1970080,"kb_used_data":1744520,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":91775072,"statfs":{"total":95995035648,"available":93977673728,"internally_reserved":0,"allocated":1786388480,"data_stored":1785943397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820668,"kb_used_data":510436,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92924484,"statfs":{"total":95995035648,"available":95154671616,"internally_reserved":0,"allocated":522686464,"data_stored":522254516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149907,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578644,"kb_used_data":244796,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93166508,"statfs":{"total":95995035648,"available":95402504192,"internally_reserved":0,"allocated":250671104,"data_stored":250230366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411544,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1665564,"kb_used_data":1471944,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92079588,"statfs":{"total":95995035648,"available":94289498112,"internally_reserved":0,"allocated":1507270656,"data_stored":1506825573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526392,"kb_used_data":1198120,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92218760,"statfs":{"total":95995035648,"available":94432010240,"internally_reserved":0,"allocated":1226874880,"data_stored":1226611199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738594,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2418128,"kb_used_data":2003772,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91327024,"statfs":{"total":95995035648,"available":93518872576,"internally_reserved":0,"allocated":2051862528,"data_stored":2051421790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2050686976,"data_stored":2050686976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1506082816,"data_stored":1506082816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":249495552,"data_stored":249495552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":521535488,"data_stored":521535488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1785200640,"data_stored":1785200640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1288044544,"data_stored":1288044544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1008795648,"data_stored":1008795648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":744161280,"data_stored":744161280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1269760000,"data_stored":1269760000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":458686464,"data_stored":458686464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1006239744,"data_stored":1006239744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:38:51.069 INFO:tasks.thrashosds.thrasher:Setting 5.3 to [8, 7, 10] 2024-08-30T23:38:51.069 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.3', '8', '7', '10'] 2024-08-30T23:38:51.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 5.3 8 7 10 2024-08-30T23:38:51.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/7170632' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: osdmap e430: 12 total, 11 up, 12 in 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:38:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:50 smithi173 ceph-mon[20554]: from='client.16866 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:38:51.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:51.172 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:51.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:51.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:51.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:38:51.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:51.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:51.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:51.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:51.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:51.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:51.726 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:51.726 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:51.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:51.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:51.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:51.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:51.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:52.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 62310 62294 55.6143 0 - 0.0143191 2024-08-30T23:38:52.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:52.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:51 smithi136 ceph-mon[20207]: pgmap v1020: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11679/186936 objects degraded (6.248%); 12277/186936 objects misplaced (6.567%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:38:52.064 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:51 smithi106 ceph-mon[16603]: pgmap v1020: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11679/186936 objects degraded (6.248%); 12277/186936 objects misplaced (6.567%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:38:52.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:51 smithi173 ceph-mon[20554]: pgmap v1020: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 11679/186936 objects degraded (6.248%); 12277/186936 objects misplaced (6.567%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:38:52.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:52.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:52.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:52.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:52.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:52.403 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:38:52.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:52.484 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:52.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:52.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:38:52.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:52.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:52.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:52.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:52.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:52.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:52.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:52.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:52.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:53.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 63239 63223 55.6487 4.83854 0.00807838 0.017964 2024-08-30T23:38:53.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:53.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:53.099 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:53.108 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:53.108 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:53.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:53.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:53.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:53.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:53.357 INFO:teuthology.orchestra.run.smithi106.stderr:set 5.3 pg_upmap mapping to [8,7,10] 2024-08-30T23:38:53.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:53.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3903154615' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.3", "id": [8, 7, 10]} : dispatch 2024-08-30T23:38:53.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:53.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:53.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:53.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3903154615' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.3", "id": [8, 7, 10]} : dispatch 2024-08-30T23:38:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3903154615' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "5.3", "id": [8, 7, 10]} : dispatch 2024-08-30T23:38:53.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:53.890 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:53.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:53.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:54.003 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 63651 63635 55.2334 25.75 0.0029044 0.0179979 2024-08-30T23:38:54.013 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:54.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:54.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:54.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:54.263 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:54.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.389 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:54.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:54.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:38:54.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:38:54.557 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3903154615' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [8, 7, 10]}]': finished 2024-08-30T23:38:54.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:54 smithi106 ceph-mon[16603]: osdmap e431: 12 total, 11 up, 12 in 2024-08-30T23:38:54.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:38:54.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:38:54.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:54 smithi106 ceph-mon[16603]: pgmap v1022: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s wr, 43 op/s; 11679/187713 objects degraded (6.222%); 11164/187713 objects misplaced (5.947%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:38:54.571 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:54.571 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:54.571 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:54.592 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:54.593 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:54.612 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:54.613 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:54.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:54.633 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:54.633 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:54.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:54.655 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:54.655 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:54.675 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:54.676 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:54.696 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:54.697 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:54.718 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:54.718 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:54.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:54.739 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:54.739 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:54.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:54.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:54.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:54.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:54.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3903154615' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [8, 7, 10]}]': finished 2024-08-30T23:38:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:54 smithi136 ceph-mon[20207]: osdmap e431: 12 total, 11 up, 12 in 2024-08-30T23:38:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:38:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:38:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:54 smithi136 ceph-mon[20207]: pgmap v1022: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s wr, 43 op/s; 11679/187713 objects degraded (6.222%); 11164/187713 objects misplaced (5.947%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:38:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3903154615' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.3", "id": [8, 7, 10]}]': finished 2024-08-30T23:38:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:54 smithi173 ceph-mon[20554]: osdmap e431: 12 total, 11 up, 12 in 2024-08-30T23:38:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:38:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:38:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:54 smithi173 ceph-mon[20554]: pgmap v1022: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s wr, 43 op/s; 11679/187713 objects degraded (6.222%); 11164/187713 objects misplaced (5.947%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:38:54.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:54.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:54.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:54.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:55.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 63667 63651 54.4905 1 0.00588241 0.0180047 2024-08-30T23:38:55.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:55.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:55.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:55.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:55.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:55.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:55.370 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:38:55.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:55.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:38:55.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:38:55.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.631 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:55.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:55.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:55.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:55.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:55 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 11679/187713 objects degraded (6.222%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:55.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:55 smithi106 ceph-mon[16603]: osdmap e432: 12 total, 11 up, 12 in 2024-08-30T23:38:55.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2318119499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:55.765 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/240957634' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:55 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 11679/187713 objects degraded (6.222%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:55 smithi136 ceph-mon[20207]: osdmap e432: 12 total, 11 up, 12 in 2024-08-30T23:38:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2318119499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/240957634' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:38:55.807 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:38:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:55 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 11679/187713 objects degraded (6.222%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:38:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:55 smithi173 ceph-mon[20554]: osdmap e432: 12 total, 11 up, 12 in 2024-08-30T23:38:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2318119499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/240957634' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:38:55.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:55.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:55.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:55.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:56.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 64149 64133 54.1612 30.125 0.0042768 0.0184567 2024-08-30T23:38:56.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:56.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:56.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:56.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:56.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:56.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:56.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:56.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:56.418 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:56 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:56.418 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/240957634' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:56.418 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:56 smithi106 ceph-mon[16603]: osdmap e433: 12 total, 11 up, 12 in 2024-08-30T23:38:56.419 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:56 smithi106 ceph-mon[16603]: pgmap v1025: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s wr, 139 op/s; 11756/189411 objects degraded (6.207%); 10841/189411 objects misplaced (5.724%); 15 MiB/s, 243 objects/s recovering 2024-08-30T23:38:56.419 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:56 smithi106 ceph-mon[16603]: osdmap e434: 12 total, 11 up, 12 in 2024-08-30T23:38:56.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:56.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:56.542 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:38:56.543 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:38:56.543 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:38:56.543 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:38:56.543 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:38:56.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:56.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:56.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:56 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/240957634' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:56 smithi136 ceph-mon[20207]: osdmap e433: 12 total, 11 up, 12 in 2024-08-30T23:38:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:56 smithi136 ceph-mon[20207]: pgmap v1025: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s wr, 139 op/s; 11756/189411 objects degraded (6.207%); 10841/189411 objects misplaced (5.724%); 15 MiB/s, 243 objects/s recovering 2024-08-30T23:38:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:56 smithi136 ceph-mon[20207]: osdmap e434: 12 total, 11 up, 12 in 2024-08-30T23:38:56.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:56 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:38:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/240957634' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:38:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:56 smithi173 ceph-mon[20554]: osdmap e433: 12 total, 11 up, 12 in 2024-08-30T23:38:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:56 smithi173 ceph-mon[20554]: pgmap v1025: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 10 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s wr, 139 op/s; 11756/189411 objects degraded (6.207%); 10841/189411 objects misplaced (5.724%); 15 MiB/s, 243 objects/s recovering 2024-08-30T23:38:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:56 smithi173 ceph-mon[20554]: osdmap e434: 12 total, 11 up, 12 in 2024-08-30T23:38:56.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:56.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:56.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:57.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 65298 65282 54.3964 71.8125 0.00327658 0.0183783 2024-08-30T23:38:57.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:57.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:57.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:57.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:57.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:57.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:57.334 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:57.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:57.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:38:57.461 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:38:57.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:57.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:57.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:57.722 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:57.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:38:57.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:38:57.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:38:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:57 smithi106 ceph-mon[16603]: pgmap v1027: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 354 op/s; 11961/191913 objects degraded (6.233%); 10454/191913 objects misplaced (5.447%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:38:57.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:57.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:57.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:58.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 66323 66307 54.5235 64.0625 0.00448944 0.0183353 2024-08-30T23:38:58.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:38:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:57 smithi136 ceph-mon[20207]: pgmap v1027: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 354 op/s; 11961/191913 objects degraded (6.233%); 10454/191913 objects misplaced (5.447%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:38:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:57 smithi173 ceph-mon[20554]: pgmap v1027: 17 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 10 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 354 op/s; 11961/191913 objects degraded (6.233%); 10454/191913 objects misplaced (5.447%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:38:58.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:58.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:58.170 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:58.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:38:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:38:58.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:38:58.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:58.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:58.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:58.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:58.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:58.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:58.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:38:58.726 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:38:58.728 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:38:58.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:38:58.745 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:38:58.756 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:38:58.756 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:38:58.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:38:58.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:38:58.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:38:58.813 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:38:58.814 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:38:58.841 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:38:58.841 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:38:58.841 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/808377167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:58.869 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:38:58.869 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:38:58.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:58.900 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:38:58.901 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:38:58.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:58.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:58.927 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:38:58.928 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:38:58.957 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:38:58.957 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:38:58.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:38:58.991 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:38:58.991 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:38:59.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 67555 67539 54.8153 77 0.00411711 0.0182372 2024-08-30T23:38:59.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:59.024 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:38:59.024 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:38:59.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:59.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/808377167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:59.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/808377167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:38:59.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:59.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:59.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:38:59.284 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:38:59.284 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:38:59.284 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-30T23:38:59.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:38:59.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:38:59.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:38:59.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:38:59.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:38:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:59.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:38:59.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:59.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:38:59.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:38:59.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:38:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.817 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:38:59.831 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:38:59.831 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:38:59.918 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:38:59.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:38:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:38:59.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:00.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 68641 68625 54.9827 67.875 0.00320511 0.01818 2024-08-30T23:39:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:38:59 smithi136 ceph-mon[20207]: pgmap v1028: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 538 op/s; 20105/197400 objects degraded (10.185%); 13666/197400 objects misplaced (6.923%); 17 MiB/s, 264 objects/s recovering 2024-08-30T23:39:00.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:00.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:38:59 smithi173 ceph-mon[20554]: pgmap v1028: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 538 op/s; 20105/197400 objects degraded (10.185%); 13666/197400 objects misplaced (6.923%); 17 MiB/s, 264 objects/s recovering 2024-08-30T23:39:00.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:38:59 smithi106 ceph-mon[16603]: pgmap v1028: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 538 op/s; 20105/197400 objects degraded (10.185%); 13666/197400 objects misplaced (6.923%); 17 MiB/s, 264 objects/s recovering 2024-08-30T23:39:00.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:00.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:00.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:00.328 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:39:00.373 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:39:00.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:00.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:00.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:00.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:00.641 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:00.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:00.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:00.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:00.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:00.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:00.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:00.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:01.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 69551 69535 55.0066 56.875 0.0156711 0.0181723 2024-08-30T23:39:01.012 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:00 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 20105/197400 objects degraded (10.185%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:00 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 20105/197400 objects degraded (10.185%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:01.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:00 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 20105/197400 objects degraded (10.185%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:01.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:01.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:01.226 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:01.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:01.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:01.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:01.482 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:01.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:01.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:01.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:39:01.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:01.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:01.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:01 smithi106 ceph-mon[16603]: pgmap v1029: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 623 op/s; 20105/202056 objects degraded (9.950%); 13984/202056 objects misplaced (6.921%); 11 MiB/s, 176 objects/s recovering 2024-08-30T23:39:01.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:01.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:01.957 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:01.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:01.995 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:02.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:02.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:39:02.005376+0000 min lat: 0.00108473 max lat: 15.0138 avg lat: 0.0181474 2024-08-30T23:39:02.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:39:02.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 70465 70449 55.033 57.125 0.00754394 0.0181474 2024-08-30T23:39:02.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:01 smithi136 ceph-mon[20207]: pgmap v1029: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 623 op/s; 20105/202056 objects degraded (9.950%); 13984/202056 objects misplaced (6.921%); 11 MiB/s, 176 objects/s recovering 2024-08-30T23:39:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:01 smithi173 ceph-mon[20554]: pgmap v1029: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 623 op/s; 20105/202056 objects degraded (9.950%); 13984/202056 objects misplaced (6.921%); 11 MiB/s, 176 objects/s recovering 2024-08-30T23:39:02.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:02.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:39:02.178 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:39:02.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:02.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:02.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:02.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:39:02.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:02.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:02.512 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:02.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:02.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:02.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:02.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:02 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:02 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:02 smithi106 ceph-mon[16603]: osdmap e435: 12 total, 11 up, 12 in 2024-08-30T23:39:02.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:02 smithi106 ceph-mon[16603]: from='client.26372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:39:02.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:02.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:02.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:03.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:03.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 71362 71346 55.0456 56.0625 0.00484974 0.0181609 2024-08-30T23:39:03.017 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1030,"stamp":"2024-08-30T23:39:01.757931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4414439968,"num_objects":67352,"num_object_clones":0,"num_object_copies":202056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20105,"num_objects_misplaced":13984,"num_objects_unfound":0,"num_objects_dirty":67352,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":67473,"num_write_kb":4312550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5356,"num_bytes_recovered":352204384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67442,"ondisk_log_size":67442,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":62,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17592060,"kb_used_data":13955372,"kb_used_omap":18,"kb_used_meta":3636461,"kb_avail":1107349764,"statfs":{"total":1151940427776,"available":1133926158336,"internally_reserved":0,"allocated":14290300928,"data_stored":14284892750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":3723736377},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,4,5,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":276234240,"num_objects":4215,"num_object_clones":0,"num_object_copies":12645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8349,"num_objects_misplaced":3143,"num_objects_unfound":0,"num_objects_dirty":4215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4215,"num_write_kb":269760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1265,"num_bytes_recovered":82903040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.384436"},"pg_stats":[{"pgid":"5.f","version":"434'4166","reported_seq":4267,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:58.779434+0000","last_change":"2024-08-30T23:37:34.040566+0000","last_active":"2024-08-30T23:38:58.779434+0000","last_peered":"2024-08-30T23:38:58.779434+0000","last_clean":"2024-08-30T23:38:58.779434+0000","last_became_active":"2024-08-30T23:37:34.040273+0000","last_became_peered":"2024-08-30T23:37:34.040273+0000","last_unstale":"2024-08-30T23:38:58.779434+0000","last_undegraded":"2024-08-30T23:38:58.779434+0000","last_fullsized":"2024-08-30T23:38:58.779434+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:00.148306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273022976,"num_objects":4166,"num_object_clones":0,"num_object_copies":12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4166,"num_write_kb":266624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,2],"acting":[5,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"434'4127","reported_seq":4227,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:59.984451+0000","last_change":"2024-08-30T23:37:34.072395+0000","last_active":"2024-08-30T23:38:59.984451+0000","last_peered":"2024-08-30T23:38:59.984451+0000","last_clean":"2024-08-30T23:38:59.984451+0000","last_became_active":"2024-08-30T23:37:34.071585+0000","last_became_peered":"2024-08-30T23:37:34.071585+0000","last_unstale":"2024-08-30T23:38:59.984451+0000","last_undegraded":"2024-08-30T23:38:59.984451+0000","last_fullsized":"2024-08-30T23:38:59.984451+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270467072,"num_objects":4127,"num_object_clones":0,"num_object_copies":12381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":264128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.d","version":"434'4305","reported_seq":4451,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:39:01.440131+0000","last_change":"2024-08-30T23:38:40.517933+0000","last_active":"2024-08-30T23:39:01.440131+0000","last_peered":"2024-08-30T23:39:01.440131+0000","last_clean":"2024-08-30T23:39:01.440131+0000","last_became_active":"2024-08-30T23:37:34.040753+0000","last_became_peered":"2024-08-30T23:37:34.040753+0000","last_unstale":"2024-08-30T23:39:01.440131+0000","last_undegraded":"2024-08-30T23:39:01.440131+0000","last_fullsized":"2024-08-30T23:39:01.440131+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'3876","last_scrub_stamp":"2024-08-30T23:38:40.517849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:40.517849+0000","objects_scrubbed":3874,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:53.360081+0000","scrub_duration":2833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282132480,"num_objects":4305,"num_object_clones":0,"num_object_copies":12915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4305,"num_write_kb":275520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.c","version":"434'4412","reported_seq":4513,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:59.718067+0000","last_change":"2024-08-30T23:37:34.072314+0000","last_active":"2024-08-30T23:38:59.718067+0000","last_peered":"2024-08-30T23:38:59.718067+0000","last_clean":"2024-08-30T23:38:59.718067+0000","last_became_active":"2024-08-30T23:37:34.071745+0000","last_became_peered":"2024-08-30T23:37:34.071745+0000","last_unstale":"2024-08-30T23:38:59.718067+0000","last_undegraded":"2024-08-30T23:38:59.718067+0000","last_fullsized":"2024-08-30T23:38:59.718067+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:03.875220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289144832,"num_objects":4412,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":282368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"434'4389","reported_seq":4490,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:59.990942+0000","last_change":"2024-08-30T23:37:34.072046+0000","last_active":"2024-08-30T23:38:59.990942+0000","last_peered":"2024-08-30T23:38:59.990942+0000","last_clean":"2024-08-30T23:38:59.990942+0000","last_became_active":"2024-08-30T23:37:34.071651+0000","last_became_peered":"2024-08-30T23:37:34.071651+0000","last_unstale":"2024-08-30T23:38:59.990942+0000","last_undegraded":"2024-08-30T23:38:59.990942+0000","last_fullsized":"2024-08-30T23:38:59.990942+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4389,"log_dups_size":0,"ondisk_log_size":4389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:02.689327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287637504,"num_objects":4389,"num_object_clones":0,"num_object_copies":13167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":280896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"434'4193","reported_seq":4284,"reported_epoch":434,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:38:58.291570+0000","last_change":"2024-08-30T23:38:34.267437+0000","last_active":"2024-08-30T23:38:58.291570+0000","last_peered":"2024-08-30T23:38:58.291570+0000","last_clean":"2024-08-30T23:38:32.087384+0000","last_became_active":"2024-08-30T23:38:34.258131+0000","last_became_peered":"2024-08-30T23:38:34.258131+0000","last_unstale":"2024-08-30T23:38:58.291570+0000","last_undegraded":"2024-08-30T23:38:34.136754+0000","last_fullsized":"2024-08-30T23:38:34.117998+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4193,"num_write_kb":268352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[2,8],"avail_no_missing":["2","8","11"],"object_location_counts":[{"shards":"2,8","objects":479},{"shards":"2,8,11","objects":3714}],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"434'4113","reported_seq":4214,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:57.484396+0000","last_change":"2024-08-30T23:37:34.038908+0000","last_active":"2024-08-30T23:38:57.484396+0000","last_peered":"2024-08-30T23:38:57.484396+0000","last_clean":"2024-08-30T23:38:57.484396+0000","last_became_active":"2024-08-30T23:37:34.038547+0000","last_became_peered":"2024-08-30T23:37:34.038547+0000","last_unstale":"2024-08-30T23:38:57.484396+0000","last_undegraded":"2024-08-30T23:38:57.484396+0000","last_fullsized":"2024-08-30T23:38:57.484396+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4113,"log_dups_size":0,"ondisk_log_size":4113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269549568,"num_objects":4113,"num_object_clones":0,"num_object_copies":12339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4113,"num_write_kb":263232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"434'4158","reported_seq":7553,"reported_epoch":434,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:38:58.301399+0000","last_change":"2024-08-30T23:38:34.309882+0000","last_active":"2024-08-30T23:38:58.301399+0000","last_peered":"2024-08-30T23:38:58.301399+0000","last_clean":"2024-08-30T23:38:32.105026+0000","last_became_active":"2024-08-30T23:38:34.257797+0000","last_became_peered":"2024-08-30T23:38:34.257797+0000","last_unstale":"2024-08-30T23:38:58.301399+0000","last_undegraded":"2024-08-30T23:38:58.301399+0000","last_fullsized":"2024-08-30T23:38:58.301399+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272498688,"num_objects":4158,"num_object_clones":0,"num_object_copies":12474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2020,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":266112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":108068864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[2,0,9],"avail_no_missing":["2","0","9"],"object_location_counts":[{"shards":"0,2,9","objects":2138}],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"434'4288","reported_seq":4386,"reported_epoch":434,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:38:58.731545+0000","last_change":"2024-08-30T23:37:42.009508+0000","last_active":"2024-08-30T23:38:58.731545+0000","last_peered":"2024-08-30T23:38:58.731545+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.039578+0000","last_became_peered":"2024-08-30T23:37:34.039578+0000","last_unstale":"2024-08-30T23:38:58.731545+0000","last_undegraded":"2024-08-30T23:37:42.008557+0000","last_fullsized":"2024-08-30T23:37:34.037051+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":4288,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4288,"num_write_kb":274432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":["5","9"],"object_location_counts":[{"shards":"5,9","objects":4288}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"434'4143","reported_seq":4244,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:58.779246+0000","last_change":"2024-08-30T23:37:34.040398+0000","last_active":"2024-08-30T23:38:58.779246+0000","last_peered":"2024-08-30T23:38:58.779246+0000","last_clean":"2024-08-30T23:38:58.779246+0000","last_became_active":"2024-08-30T23:37:34.039974+0000","last_became_peered":"2024-08-30T23:37:34.039974+0000","last_unstale":"2024-08-30T23:38:58.779246+0000","last_undegraded":"2024-08-30T23:38:58.779246+0000","last_fullsized":"2024-08-30T23:38:58.779246+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4143,"log_dups_size":0,"ondisk_log_size":4143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271515648,"num_objects":4143,"num_object_clones":0,"num_object_copies":12429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":265152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"434'4209","reported_seq":4306,"reported_epoch":434,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:38:57.010001+0000","last_change":"2024-08-30T23:37:42.007680+0000","last_active":"2024-08-30T23:38:57.010001+0000","last_peered":"2024-08-30T23:38:57.010001+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.071741+0000","last_became_peered":"2024-08-30T23:37:34.071741+0000","last_unstale":"2024-08-30T23:38:57.010001+0000","last_undegraded":"2024-08-30T23:37:42.007336+0000","last_fullsized":"2024-08-30T23:37:34.037577+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:38:58.233137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275841024,"num_objects":4209,"num_object_clones":0,"num_object_copies":12627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4209,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":4209}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"434'4156","reported_seq":4262,"reported_epoch":434,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:38:58.171562+0000","last_change":"2024-08-30T23:38:55.586630+0000","last_active":"2024-08-30T23:38:58.171562+0000","last_peered":"2024-08-30T23:38:58.171562+0000","last_clean":"2024-08-30T23:38:53.153561+0000","last_became_active":"2024-08-30T23:38:55.581448+0000","last_became_peered":"2024-08-30T23:38:55.581448+0000","last_unstale":"2024-08-30T23:38:58.171562+0000","last_undegraded":"2024-08-30T23:38:55.486733+0000","last_fullsized":"2024-08-30T23:38:55.373545+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272367616,"num_objects":4156,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7894,"num_objects_misplaced":3947,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":265984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[0,7],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":3947},{"shards":"0,7","objects":209}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"434'4019","reported_seq":7890,"reported_epoch":434,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:38:57.047215+0000","last_change":"2024-08-30T23:38:34.286521+0000","last_active":"2024-08-30T23:38:57.047215+0000","last_peered":"2024-08-30T23:38:57.047215+0000","last_clean":"2024-08-30T23:38:31.890254+0000","last_became_active":"2024-08-30T23:38:34.258074+0000","last_became_peered":"2024-08-30T23:38:34.258074+0000","last_unstale":"2024-08-30T23:38:57.047215+0000","last_undegraded":"2024-08-30T23:38:57.047215+0000","last_fullsized":"2024-08-30T23:38:34.119189+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":12057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1751,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4019,"num_write_kb":257216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1880,"num_bytes_recovered":123207680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":1751},{"shards":"0,6","objects":2268}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"434'4093","reported_seq":4240,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:58.413234+0000","last_change":"2024-08-30T23:38:39.040065+0000","last_active":"2024-08-30T23:38:58.413234+0000","last_peered":"2024-08-30T23:38:58.413234+0000","last_clean":"2024-08-30T23:38:58.413234+0000","last_became_active":"2024-08-30T23:37:34.043753+0000","last_became_peered":"2024-08-30T23:37:34.043753+0000","last_unstale":"2024-08-30T23:38:58.413234+0000","last_undegraded":"2024-08-30T23:38:58.413234+0000","last_fullsized":"2024-08-30T23:38:58.413234+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'3805","last_scrub_stamp":"2024-08-30T23:38:39.039968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:39.039968+0000","objects_scrubbed":3803,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:40:03.713913+0000","scrub_duration":1274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268238848,"num_objects":4093,"num_object_clones":0,"num_object_copies":12279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":261952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'4398","reported_seq":4492,"reported_epoch":434,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:39:01.342185+0000","last_change":"2024-08-30T23:38:34.132086+0000","last_active":"2024-08-30T23:39:01.342185+0000","last_peered":"2024-08-30T23:39:01.342185+0000","last_clean":"2024-08-30T23:39:01.342185+0000","last_became_active":"2024-08-30T23:38:34.126744+0000","last_became_peered":"2024-08-30T23:38:34.126744+0000","last_unstale":"2024-08-30T23:39:01.342185+0000","last_undegraded":"2024-08-30T23:39:01.342185+0000","last_fullsized":"2024-08-30T23:39:01.342185+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:03.002777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288227328,"num_objects":4398,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4398,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":281472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4,0],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":4398}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"434'4181","reported_seq":7913,"reported_epoch":434,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:38:58.778481+0000","last_change":"2024-08-30T23:38:34.270052+0000","last_active":"2024-08-30T23:38:58.778481+0000","last_peered":"2024-08-30T23:38:58.778481+0000","last_clean":"2024-08-30T23:38:32.091146+0000","last_became_active":"2024-08-30T23:38:34.242472+0000","last_became_peered":"2024-08-30T23:38:34.242472+0000","last_unstale":"2024-08-30T23:38:58.778481+0000","last_undegraded":"2024-08-30T23:38:58.778481+0000","last_fullsized":"2024-08-30T23:38:34.119231+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274006016,"num_objects":4181,"num_object_clones":0,"num_object_copies":12543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1868,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":267584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1820,"num_bytes_recovered":119275520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[5,2],"avail_no_missing":["5","2","8"],"object_location_counts":[{"shards":"2,5","objects":2313},{"shards":"2,5,8","objects":1868}],"blocked_by":[],"up_primary":10,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":855,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-08-30T23:38:55.699172+0000","last_change":"2024-08-30T23:38:00.011325+0000","last_active":"2024-08-30T23:38:55.699172+0000","last_peered":"2024-08-30T23:38:55.699172+0000","last_clean":"2024-08-30T23:38:55.699172+0000","last_became_active":"2024-08-30T23:34:35.630884+0000","last_became_peered":"2024-08-30T23:34:35.630884+0000","last_unstale":"2024-08-30T23:38:55.699172+0000","last_undegraded":"2024-08-30T23:38:55.699172+0000","last_fullsized":"2024-08-30T23:38:55.699172+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:38:00.011247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:38:00.011247+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:39:03.137916+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4413849600,"num_objects":67350,"num_object_clones":0,"num_object_copies":202050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20105,"num_objects_misplaced":13984,"num_objects_unfound":0,"num_objects_dirty":67350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67350,"num_write_kb":4310400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5350,"num_bytes_recovered":350617600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13168214016,"data_stored":13168214016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67350,"ondisk_log_size":67350,"up":45,"acting":42,"num_store_stats":11},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187630,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1417436,"kb_used_data":1065548,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92327716,"statfs":{"total":95995035648,"available":94543581184,"internally_reserved":0,"allocated":1091121152,"data_stored":1090650869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":360315332},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449269,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031252,"kb_used_data":855684,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92713900,"statfs":{"total":95995035648,"available":94939033600,"internally_reserved":0,"allocated":876220416,"data_stored":875753705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710904,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1733908,"kb_used_data":1355652,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92011244,"statfs":{"total":95995035648,"available":94219513856,"internally_reserved":0,"allocated":1388187648,"data_stored":1387720937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972542,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1003476,"kb_used_data":801540,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92741676,"statfs":{"total":95995035648,"available":94967476224,"internally_reserved":0,"allocated":820776960,"data_stored":820310249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234176,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1424988,"kb_used_data":1046732,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92320164,"statfs":{"total":95995035648,"available":94535847936,"internally_reserved":0,"allocated":1071853568,"data_stored":1071383305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528518,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567276,"kb_used_data":1297292,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92177876,"statfs":{"total":95995035648,"available":94390145024,"internally_reserved":0,"allocated":1328427008,"data_stored":1327956745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":276429254},"hb_peers":[0,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724746,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2149332,"kb_used_data":1861316,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91595820,"statfs":{"total":95995035648,"available":93794119680,"internally_reserved":0,"allocated":1905987584,"data_stored":1905520873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953681,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860756,"kb_used_data":550532,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92884396,"statfs":{"total":95995035648,"available":95113621504,"internally_reserved":0,"allocated":563744768,"data_stored":563278057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149909,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592660,"kb_used_data":258820,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93152492,"statfs":{"total":95995035648,"available":95388151808,"internally_reserved":0,"allocated":265031680,"data_stored":264564969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411546,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1749716,"kb_used_data":1551940,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":91995436,"statfs":{"total":95995035648,"available":94203326464,"internally_reserved":0,"allocated":1589186560,"data_stored":1588719849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526392,"kb_used_data":1198120,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92218760,"statfs":{"total":95995035648,"available":94432010240,"internally_reserved":0,"allocated":1226874880,"data_stored":1226611199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738596,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534868,"kb_used_data":2112196,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":91210284,"statfs":{"total":95995035648,"available":93399330816,"internally_reserved":0,"allocated":2162888704,"data_stored":2162421993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2188902400,"data_stored":2188902400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1596391424,"data_stored":1596391424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":268238848,"data_stored":268238848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":571342848,"data_stored":571342848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1911488512,"data_stored":1911488512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1342242816,"data_stored":1342242816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1072824320,"data_stored":1072824320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":826015744,"data_stored":826015744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1401225216,"data_stored":1401225216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":894173184,"data_stored":894173184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1095368704,"data_stored":1095368704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:39:03.018 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-30T23:39:03.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-08-30T23:39:03.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:03.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:02 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:03.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:02 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:02 smithi136 ceph-mon[20207]: osdmap e435: 12 total, 11 up, 12 in 2024-08-30T23:39:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:02 smithi136 ceph-mon[20207]: from='client.26372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:39:03.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:02 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:02 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:02 smithi173 ceph-mon[20554]: osdmap e435: 12 total, 11 up, 12 in 2024-08-30T23:39:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:02 smithi173 ceph-mon[20554]: from='client.26372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:39:03.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:03.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:03.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:03.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.296 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:03.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:03.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:03.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:03.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:03.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:03.704 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:03.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:03.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:03 smithi106 ceph-mon[16603]: pgmap v1031: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 796 op/s; 20440/208539 objects degraded (9.802%); 13248/208539 objects misplaced (6.353%); 16 MiB/s, 250 objects/s recovering 2024-08-30T23:39:03.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:03.949 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:03.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:04.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 72370 72354 55.1426 63 0.00344382 0.0181282 2024-08-30T23:39:04.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:03 smithi136 ceph-mon[20207]: pgmap v1031: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 796 op/s; 20440/208539 objects degraded (9.802%); 13248/208539 objects misplaced (6.353%); 16 MiB/s, 250 objects/s recovering 2024-08-30T23:39:04.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:03 smithi173 ceph-mon[20554]: pgmap v1031: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 796 op/s; 20440/208539 objects degraded (9.802%); 13248/208539 objects misplaced (6.353%); 16 MiB/s, 250 objects/s recovering 2024-08-30T23:39:04.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:04.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:04.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:04.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:04.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:04.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:04.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:04.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:04.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:04.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:04.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:04.724 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:04.769 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:39:04.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:04.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:04.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:04.980 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:04.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:05.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 73303 73287 55.1807 58.3125 0.00374779 0.0181144 2024-08-30T23:39:05.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1660415737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:39:05.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1660415737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:39:05.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1660415737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:39:05.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:05.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:05.235 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:05.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:05.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:05.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:05.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:05.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:05.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:05.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:05.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:05.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:05.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:05.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:05.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:06.004 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 74101 74085 55.1175 49.875 0.0267931 0.0181331 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1660415737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: osdmap e436: 12 total, 11 up, 12 in 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: pgmap v1033: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s; 20756/215097 objects degraded (9.650%); 12847/215097 objects misplaced (5.973%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:39:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 20440/208539 objects degraded (9.802%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:06.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1660415737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:39:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: osdmap e436: 12 total, 11 up, 12 in 2024-08-30T23:39:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: pgmap v1033: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s; 20756/215097 objects degraded (9.650%); 12847/215097 objects misplaced (5.973%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:39:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 20440/208539 objects degraded (9.802%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:06.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1660415737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: osdmap e436: 12 total, 11 up, 12 in 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: pgmap v1033: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 965 op/s; 20756/215097 objects degraded (9.650%); 12847/215097 objects misplaced (5.973%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:39:06.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 20440/208539 objects degraded (9.802%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:06.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:06.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:06.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:06.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:06.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:06.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:06.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:06.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:06.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:06.648 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:06.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:06.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:06.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:06.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:06.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:06.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:07.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 75065 75049 55.1778 60.25 0.00911134 0.0181171 2024-08-30T23:39:07.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:07.038 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:07.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:07.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:07.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:07.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:07.398 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:07.422 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:07.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:07.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:07.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:07.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:07.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:07.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:07.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:07.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:07.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:08.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 76041 76025 55.2455 61 0.00351825 0.0180946 2024-08-30T23:39:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:07 smithi136 ceph-mon[20207]: osdmap e437: 12 total, 11 up, 12 in 2024-08-30T23:39:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:07 smithi136 ceph-mon[20207]: pgmap v1035: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 963 op/s; 21064/219396 objects degraded (9.601%); 12738/219396 objects misplaced (5.806%); 16 MiB/s, 252 objects/s recovering 2024-08-30T23:39:08.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:08.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:07 smithi173 ceph-mon[20554]: osdmap e437: 12 total, 11 up, 12 in 2024-08-30T23:39:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:07 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:07 smithi173 ceph-mon[20554]: pgmap v1035: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 963 op/s; 21064/219396 objects degraded (9.601%); 12738/219396 objects misplaced (5.806%); 16 MiB/s, 252 objects/s recovering 2024-08-30T23:39:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:07 smithi106 ceph-mon[16603]: osdmap e437: 12 total, 11 up, 12 in 2024-08-30T23:39:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:07 smithi106 ceph-mon[16603]: pgmap v1035: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 9 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 963 op/s; 21064/219396 objects degraded (9.601%); 12738/219396 objects misplaced (5.806%); 16 MiB/s, 252 objects/s recovering 2024-08-30T23:39:08.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:08.196 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:08.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:08.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:08.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:08.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:08.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:08.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:08.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:08.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:08.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:08.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:08.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:08.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:08.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:09.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 76969 76953 55.2771 58 0.0159663 0.0180855 2024-08-30T23:39:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:08 smithi173 ceph-mon[20554]: osdmap e438: 12 total, 11 up, 12 in 2024-08-30T23:39:09.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:08 smithi106 ceph-mon[16603]: osdmap e438: 12 total, 11 up, 12 in 2024-08-30T23:39:09.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:09.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:09.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:08 smithi136 ceph-mon[20207]: osdmap e438: 12 total, 11 up, 12 in 2024-08-30T23:39:09.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:09.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.378 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:09.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:39:09.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:09.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:09.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:09.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:09.668 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:09.670 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:09.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:09.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:09.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:09.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:09.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:09.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:10.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:10.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 15 78065 78050 55.428 68.5625 0.0215835 0.0180359 2024-08-30T23:39:10.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:10.069 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:09 smithi106 ceph-mon[16603]: 5.c scrub starts 2024-08-30T23:39:10.069 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:09 smithi106 ceph-mon[16603]: pgmap v1037: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 994 op/s; 21370/226437 objects degraded (9.438%); 11822/226437 objects misplaced (5.221%); 18 MiB/s, 282 objects/s recovering 2024-08-30T23:39:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:09 smithi173 ceph-mon[20554]: 5.c scrub starts 2024-08-30T23:39:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:09 smithi173 ceph-mon[20554]: pgmap v1037: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 994 op/s; 21370/226437 objects degraded (9.438%); 11822/226437 objects misplaced (5.221%); 18 MiB/s, 282 objects/s recovering 2024-08-30T23:39:10.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:10.160 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:10.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:09 smithi136 ceph-mon[20207]: 5.c scrub starts 2024-08-30T23:39:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:09 smithi136 ceph-mon[20207]: pgmap v1037: 17 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 994 op/s; 21370/226437 objects degraded (9.438%); 11822/226437 objects misplaced (5.221%); 18 MiB/s, 282 objects/s recovering 2024-08-30T23:39:10.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:10.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:10.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:10.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:10.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:10.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:10.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:10.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:10.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:10 smithi106 ceph-mon[16603]: 5.c scrub ok 2024-08-30T23:39:10.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 21370/226437 objects degraded (9.438%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:10.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:10.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:10.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:11.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 79212 79196 55.6099 71.625 0.00720779 0.0179772 2024-08-30T23:39:11.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:10 smithi173 ceph-mon[20554]: 5.c scrub ok 2024-08-30T23:39:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 21370/226437 objects degraded (9.438%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:11.079 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:11.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:11.210 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:10 smithi136 ceph-mon[20207]: 5.c scrub ok 2024-08-30T23:39:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 21370/226437 objects degraded (9.438%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:11.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:11.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:11.362 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:11.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:11.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:11.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:11.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:11.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:11.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:11.823 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pg_num to 11 2024-08-30T23:39:11.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:11.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:11 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:11.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:11 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:11.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2594087608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:11.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/756382558' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-30T23:39:11.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:11 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-30T23:39:11.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:11 smithi106 ceph-mon[16603]: pgmap v1038: 17 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 8 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 924 op/s; 21370/231048 objects degraded (9.249%); 12146/231048 objects misplaced (5.257%); 12 MiB/s, 191 objects/s recovering 2024-08-30T23:39:11.877 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:11.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:11.884 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:39:11.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:39:11.885 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:39:11.925 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:39:11.925 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:39:11.961 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:39:11.962 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:39:11.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:11.999 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:39:11.999 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:39:12.005 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 6 80237 80231 55.7107 64.6875 0.0131255 0.0179445 2024-08-30T23:39:12.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:12.020 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:39:12.020 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:39:12.042 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:39:12.042 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:39:12.062 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:39:12.063 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:39:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:11 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:11 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2594087608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/756382558' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-30T23:39:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:11 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-30T23:39:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:11 smithi173 ceph-mon[20554]: pgmap v1038: 17 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 8 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 924 op/s; 21370/231048 objects degraded (9.249%); 12146/231048 objects misplaced (5.257%); 12 MiB/s, 191 objects/s recovering 2024-08-30T23:39:12.083 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:39:12.084 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:39:12.104 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:39:12.104 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:39:12.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:39:12.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:12.124 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:39:12.125 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:39:12.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:12.145 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:39:12.145 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:39:12.195 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:39:12.195 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:39:12.195 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:39:12.195 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:39:12.195 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:39:12.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:12.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:11 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:11 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2594087608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/756382558' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-30T23:39:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:11 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-30T23:39:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:11 smithi136 ceph-mon[20207]: pgmap v1038: 17 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 8 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 924 op/s; 21370/231048 objects degraded (9.249%); 12146/231048 objects misplaced (5.257%); 12 MiB/s, 191 objects/s recovering 2024-08-30T23:39:12.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:12.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:12.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:12.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:12.503 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:12.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:12.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:12.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:12.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:12.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:12.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:39:12.916 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:12.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:13.008 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.6377 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 80237 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 55.3281 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 41.2819 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 150.688 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 885 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 660.531 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2411 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-08-30T23:39:13.009 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0179659 2024-08-30T23:39:13.010 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.247437 2024-08-30T23:39:13.010 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 15.0138 2024-08-30T23:39:13.010 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00108473 2024-08-30T23:39:13.012 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:39:13.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:13.047 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:12 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-30T23:39:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:12 smithi173 ceph-mon[20554]: osdmap e439: 12 total, 11 up, 12 in 2024-08-30T23:39:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:12 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-30T23:39:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:12 smithi106 ceph-mon[16603]: osdmap e439: 12 total, 11 up, 12 in 2024-08-30T23:39:13.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:13.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:12 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-30T23:39:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:12 smithi136 ceph-mon[20207]: osdmap e439: 12 total, 11 up, 12 in 2024-08-30T23:39:13.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:13.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:13.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:13.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:13.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:13.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:13.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.694 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:13.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:13.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:13.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:13.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:14.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:13 smithi173 ceph-mon[20554]: pgmap v1040: 17 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 8 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 993 op/s; 21674/236310 objects degraded (9.172%); 11310/236310 objects misplaced (4.786%); 20 MiB/s, 316 objects/s recovering 2024-08-30T23:39:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-30T23:39:14.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:13 smithi106 ceph-mon[16603]: pgmap v1040: 17 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 8 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 993 op/s; 21674/236310 objects degraded (9.172%); 11310/236310 objects misplaced (4.786%); 20 MiB/s, 316 objects/s recovering 2024-08-30T23:39:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-30T23:39:14.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:14.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:13 smithi136 ceph-mon[20207]: pgmap v1040: 17 pgs: 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 8 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 993 op/s; 21674/236310 objects degraded (9.172%); 11310/236310 objects misplaced (4.786%); 20 MiB/s, 316 objects/s recovering 2024-08-30T23:39:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-30T23:39:14.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:14.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:14.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:14.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:14.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:14.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:14.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:14.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:14.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:14.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:14.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:14.917 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:39:14.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:39:14.953 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:14.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:14.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:14.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:15.080 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-30T23:39:15.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:14 smithi106 ceph-mon[16603]: osdmap e440: 12 total, 11 up, 12 in 2024-08-30T23:39:15.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:15.116 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:15.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:15.235 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:15.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-30T23:39:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:14 smithi136 ceph-mon[20207]: osdmap e440: 12 total, 11 up, 12 in 2024-08-30T23:39:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-30T23:39:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:14 smithi173 ceph-mon[20554]: osdmap e440: 12 total, 11 up, 12 in 2024-08-30T23:39:15.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:15.376 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:15.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:15.501 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.24460.0:194011 2.1 e200/97 2:83dfe0ed:::benchmark_data_smithi173_47195_object97003:head v 200'12204, mlcod=200'12201)", 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:39:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 572.70140385499997, 2024-08-30T23:39:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82744930299999997, 2024-08-30T23:39:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:39:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797677+0000", 2024-08-30T23:39:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797678+0000", 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1800000000000005e-07 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:42.797683+0000", 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6529999999999999e-06 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620982+0000", 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82329911700000002 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.620996+0000", 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3995999999999999e-05 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621133+0000", 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013682700000000001 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.621198+0000", 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4368000000000004e-05 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625091+0000", 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003893194 2024-08-30T23:39:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:29:43.625126+0000", 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5429999999999998e-05 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:103424 3.2 3:4caab458:::benchmark_data_smithi173_60780_object103423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 454.23945724999999, 2024-08-30T23:39:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.221222537, 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103424 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:39:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259624+0000", 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0800000000000004e-07 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259682+0000", 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7364999999999997e-05 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259692+0000", 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0686000000000001e-05 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259698+0000", 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.361e-06 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261702+0000", 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020041289999999999 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261778+0000", 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6148000000000003e-05 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261872+0000", 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4598000000000005e-05 2024-08-30T23:39:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.263845+0000", 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001972419 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.264707+0000", 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086252099999999999 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480753+0000", 2024-08-30T23:39:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.216045457 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480795+0000", 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2605000000000003e-05 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480846+0000", 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0540000000000001e-05 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198168 3.2 3:46c77281:::benchmark_data_smithi173_60780_object94714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 342.53686457600003, 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22735555199999999, 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198168 2024-08-30T23:39:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962217+0000", 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2099999999999995e-07 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962221+0000", 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999999e-06 2024-08-30T23:39:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962234+0000", 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2412000000000001e-05 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962241+0000", 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6789999999999999e-06 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962296+0000", 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5347999999999999e-05 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962411+0000", 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011498299999999999 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962524+0000", 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011293700000000001 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.967040+0000", 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045164259999999996 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.968062+0000", 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010217270000000001 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189490+0000", 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22142835899999999 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189535+0000", 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4338999999999997e-05 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189572+0000", 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6990999999999999e-05 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198178 3.2 3:47f13797:::benchmark_data_smithi173_60780_object94724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 342.52607514800002, 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21670980100000001, 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198178 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973007+0000", 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973013+0000", 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1140000000000002e-06 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973023+0000", 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0516000000000001e-05 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973030+0000", 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8390000000000001e-06 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973072+0000", 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1919000000000003e-05 2024-08-30T23:39:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973158+0000", 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6148000000000003e-05 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973234+0000", 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6165e-05 2024-08-30T23:39:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.976842+0000", 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036077290000000001 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.978153+0000", 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013109930000000001 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189617+0000", 2024-08-30T23:39:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21146347300000001 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189674+0000", 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7082e-05 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189715+0000", 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1656999999999998e-05 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206885 3.5 e300/206 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head v 300'12859, mlcod=300'12855)", 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 331.45783740899998, 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23467711799999999, 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:39:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041244+0000", 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041249+0000", 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2360000000000004e-06 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271945+0000", 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.230695859 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271961+0000", 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5815999999999999e-05 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272088+0000", 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126721 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272127+0000", 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8872000000000003e-05 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275888+0000", 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037616350000000002 2024-08-30T23:39:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275920+0000", 2024-08-30T23:39:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2255000000000002e-05 2024-08-30T23:39:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206897 3.b 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 331.44199639999999, 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58358057900000004, 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206897 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057085+0000", 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3399999999999999e-07 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057088+0000", 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8609999999999998e-06 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057094+0000", 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5429999999999999e-06 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057098+0000", 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9720000000000003e-06 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057149+0000", 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0683000000000002e-05 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057264+0000", 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114772 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057387+0000", 2024-08-30T23:39:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012322799999999999 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.061158+0000", 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037710170000000002 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062218+0000", 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001059948 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640580+0000", 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57836202199999998 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640625+0000", 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5116e-05 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640665+0000", 2024-08-30T23:39:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9882999999999997e-05 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206901 3.7 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 331.43631514200001, 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38317995100000002, 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206901 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062767+0000", 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062771+0000", 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6779999999999999e-06 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062785+0000", 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3839000000000001e-05 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062793+0000", 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063873+0000", 2024-08-30T23:39:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001080632 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063965+0000", 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1717000000000003e-05 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.064031+0000", 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5729999999999996e-05 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.067745+0000", 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037140430000000002 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.071385+0000", 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036407620000000001 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445862+0000", 2024-08-30T23:39:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37447608799999998 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445912+0000", 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0059999999999998e-05 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445946+0000", 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3927000000000002e-05 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.550 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206900 3.5 e300/206 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head v 300'12860, mlcod=300'12856)", 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 331.43587387500003, 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36104319299999998, 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:39:15.551 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063208+0000", 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.552 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063213+0000", 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0080000000000002e-06 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420406+0000", 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35719326299999998 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420426+0000", 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0253000000000001e-05 2024-08-30T23:39:15.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420597+0000", 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170786 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420630+0000", 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3578999999999998e-05 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424218+0000", 2024-08-30T23:39:15.554 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035875490000000002 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424250+0000", 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2030999999999999e-05 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62091 4.4 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:39:15.555 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 258.33341291099998, 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318614905, 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62091 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.556 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165670+0000", 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.557 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165713+0000", 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3368000000000002e-05 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165732+0000", 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9031000000000001e-05 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165740+0000", 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8050000000000007e-06 2024-08-30T23:39:15.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165802+0000", 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2279999999999993e-05 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165946+0000", 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014352899999999999 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.166046+0000", 2024-08-30T23:39:15.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100046 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169317+0000", 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003271267 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169909+0000", 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059201200000000005 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484178+0000", 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31426850299999998 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484234+0000", 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6678000000000002e-05 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484283+0000", 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8457e-05 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.561 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62105 4.4 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 258.31982857899999, 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34846845199999998, 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.562 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62105 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179254+0000", 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179297+0000", 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3108000000000001e-05 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179311+0000", 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.345e-05 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179316+0000", 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179371+0000", 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5316999999999999e-05 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179508+0000", 2024-08-30T23:39:15.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643600000000001 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179665+0000", 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015680499999999999 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182071+0000", 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024068090000000002 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182226+0000", 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015503899999999999 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527606+0000", 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34537996300000001 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527669+0000", 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2966999999999995e-05 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527721+0000", 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1224000000000003e-05 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62174 4.4 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 257.97461679899999, 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57863658799999995, 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.568 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62174 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524466+0000", 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0600000000000002e-06 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524501+0000", 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4635999999999998e-05 2024-08-30T23:39:15.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524521+0000", 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0193000000000001e-05 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524531+0000", 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7780000000000002e-06 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524604+0000", 2024-08-30T23:39:15.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3646000000000006e-05 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524763+0000", 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015852999999999999 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524940+0000", 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017771799999999999 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.572 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527733+0000", 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027928789999999998 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.528529+0000", 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000795487 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103030+0000", 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57450106899999998 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103068+0000", 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8398000000000001e-05 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103100+0000", 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2193999999999997e-05 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62186 4.4 4:257aab52:::benchmark_data_smithi173_74914_object62185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:39:15.574 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 257.94550467800002, 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55186797300000001, 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62186 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.575 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553578+0000", 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553612+0000", 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3773999999999999e-05 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553627+0000", 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5410999999999999e-05 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553637+0000", 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0101e-05 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553709+0000", 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2365000000000003e-05 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553885+0000", 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017522699999999999 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.554065+0000", 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000180101 2024-08-30T23:39:15.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556530+0000", 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024650240000000001 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556570+0000", 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9975999999999998e-05 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105343+0000", 2024-08-30T23:39:15.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54877291299999997 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105399+0000", 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6648999999999999e-05 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105444+0000", 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4660000000000003e-05 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.580 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:93692 4.4 4:24d1e68e:::benchmark_data_smithi173_74914_object93691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.47530102799999, 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28134646200000002, 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.581 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 93692 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023781+0000", 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023840+0000", 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9305999999999998e-05 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023854+0000", 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3773e-05 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023872+0000", 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7708000000000001e-05 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023930+0000", 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8007999999999998e-05 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024050+0000", 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120209 2024-08-30T23:39:15.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024158+0000", 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010744299999999999 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026599+0000", 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024417430000000001 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026641+0000", 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305020+0000", 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27837906600000001 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305079+0000", 2024-08-30T23:39:15.586 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9080999999999999e-05 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305126+0000", 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6657999999999999e-05 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:93692 4.4 e350/331 ondisk, result = 0)", 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:39:15.587 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.472024691, 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27807699200000002, 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:39:15.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027057+0000", 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027060+0000", 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.01e-06 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304915+0000", 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27785516100000002 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304921+0000", 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1800000000000001e-06 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305007+0000", 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5783000000000002e-05 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305015+0000", 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7659999999999998e-06 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305133+0000", 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118472 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:107608 4.f e354/306 4:fc26249f:::benchmark_data_smithi173_74914_object9366:head v 354'6690, mlcod=354'6687)", 2024-08-30T23:39:15.591 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 214.00126761999999, 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22329728099999999, 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497814+0000", 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3399999999999998e-07 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497821+0000", 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8900000000000001e-06 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717759+0000", 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21993858999999999 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717767+0000", 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6000000000000001e-06 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717856+0000", 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8516000000000004e-05 2024-08-30T23:39:15.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717893+0000", 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7342999999999998e-05 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721075+0000", 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031817579999999998 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721110+0000", 2024-08-30T23:39:15.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5750000000000002e-05 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:168699 4.f 4:f9ac8f5d:::benchmark_data_smithi173_74914_object70457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 148.09148945499999, 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23153700799999999, 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.596 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168699 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407593+0000", 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.308e-06 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407598+0000", 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5899999999999998e-06 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407609+0000", 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0975000000000001e-05 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407616+0000", 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0910000000000004e-06 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407664+0000", 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7513999999999999e-05 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407761+0000", 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6817999999999994e-05 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407838+0000", 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6971000000000006e-05 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.411723+0000", 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038854430000000001 2024-08-30T23:39:15.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639022+0000", 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.227298575 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639056+0000", 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4623999999999999e-05 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.601 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639128+0000", 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2099000000000002e-05 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:193335 4.f 4:f0f0a8f4:::benchmark_data_smithi173_74914_object95093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 120.87050835399999, 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17749384200000001, 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.602 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193335 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628573+0000", 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.119e-06 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628580+0000", 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1569999999999997e-06 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628592+0000", 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2866e-05 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628599+0000", 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4030000000000004e-06 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628655+0000", 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6322e-05 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628791+0000", 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013567700000000001 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628934+0000", 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014291299999999999 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.633290+0000", 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043565920000000003 2024-08-30T23:39:15.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.805971+0000", 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17268071500000001 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:15.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.806014+0000", 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3257999999999998e-05 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.806066+0000", 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1820000000000002e-05 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:196465 4.4 4:242a1b19:::benchmark_data_smithi173_74914_object98223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 117.482970126, 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45786220799999999, 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.609 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196465 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016114+0000", 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999999e-06 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016121+0000", 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9240000000000001e-06 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016125+0000", 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.704e-06 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016162+0000", 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6528e-05 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016238+0000", 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6246000000000001e-05 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016310+0000", 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2475e-05 2024-08-30T23:39:15.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020389+0000", 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040785120000000003 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020517+0000", 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012816600000000001 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.614 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473901+0000", 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45338352300000001 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473940+0000", 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9224000000000003e-05 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473973+0000", 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3034000000000003e-05 2024-08-30T23:39:15.615 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:15231 5.1 5:8ac1186e:::benchmark_data_smithi173_87639_object15230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:50.725069+0000", 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 84.774011711, 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214480904, 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.616 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 15231 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725069+0000", 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725069+0000", 2024-08-30T23:39:15.617 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725070+0000", 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8299999999999995e-07 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725160+0000", 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9543000000000006e-05 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.618 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725170+0000", 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0767e-05 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725178+0000", 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2749999999999998e-06 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725228+0000", 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0067e-05 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725318+0000", 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9888000000000004e-05 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725435+0000", 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011709699999999999 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.726054+0000", 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00061947499999999997 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.726412+0000", 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035780399999999998 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.939440+0000", 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21302797600000001 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.939497+0000", 2024-08-30T23:39:15.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7380999999999998e-05 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.939550+0000", 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2648000000000001e-05 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:17923 5.1 5:89278b47:::benchmark_data_smithi173_87639_object17922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:52.420774+0000", 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 83.078306546999997, 2024-08-30T23:39:15.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17294684799999999, 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 17923 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420774+0000", 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420774+0000", 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420775+0000", 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7899999999999997e-07 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420841+0000", 2024-08-30T23:39:15.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6388000000000002e-05 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420852+0000", 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0192e-05 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420857+0000", 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0830000000000003e-06 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.421298+0000", 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00044180100000000002 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.421420+0000", 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012193699999999999 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.421500+0000", 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9931000000000004e-05 2024-08-30T23:39:15.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.422068+0000", 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056767099999999995 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.422613+0000", 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054493900000000004 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:15.627 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.593659+0000", 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17104583400000001 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.593691+0000", 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2496000000000002e-05 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.593721+0000", 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9697e-05 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.628 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.629 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.629 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:39:15.629 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:39:15.629 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:39:15.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:15.736 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:15.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:15.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:15.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:15.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:15.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:15.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:16.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:15 smithi106 ceph-mon[16603]: osdmap e441: 12 total, 11 up, 12 in 2024-08-30T23:39:16.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:15 smithi106 ceph-mon[16603]: pgmap v1043: 27 pgs: 10 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 891 op/s; 21851/238506 objects degraded (9.162%); 10788/238506 objects misplaced (4.523%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:39:16.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-30T23:39:16.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 21674/236310 objects degraded (9.172%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:16.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-30T23:39:16.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:15 smithi106 ceph-mon[16603]: osdmap e442: 12 total, 11 up, 12 in 2024-08-30T23:39:16.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:16.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:16.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:16.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:15 smithi136 ceph-mon[20207]: osdmap e441: 12 total, 11 up, 12 in 2024-08-30T23:39:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:15 smithi136 ceph-mon[20207]: pgmap v1043: 27 pgs: 10 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 891 op/s; 21851/238506 objects degraded (9.162%); 10788/238506 objects misplaced (4.523%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:39:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-30T23:39:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 21674/236310 objects degraded (9.172%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-30T23:39:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:15 smithi136 ceph-mon[20207]: osdmap e442: 12 total, 11 up, 12 in 2024-08-30T23:39:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:15 smithi173 ceph-mon[20554]: osdmap e441: 12 total, 11 up, 12 in 2024-08-30T23:39:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:15 smithi173 ceph-mon[20554]: pgmap v1043: 27 pgs: 10 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 9 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 891 op/s; 21851/238506 objects degraded (9.162%); 10788/238506 objects misplaced (4.523%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:39:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-30T23:39:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 21674/236310 objects degraded (9.172%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-30T23:39:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:15 smithi173 ceph-mon[20554]: osdmap e442: 12 total, 11 up, 12 in 2024-08-30T23:39:16.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:16.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:16.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:16.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:16.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:16.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:16.702 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:39:16.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:16.760 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:16.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:16.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:16.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:16.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:39:16.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:17.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:17.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:39:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:39:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:39:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/687785528' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:17.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:39:17.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/687785528' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:17.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:16 smithi106 ceph-mon[16603]: osdmap e443: 12 total, 11 up, 12 in 2024-08-30T23:39:17.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:17.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:17.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:17.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:17.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/687785528' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/687785528' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:16 smithi136 ceph-mon[20207]: osdmap e443: 12 total, 11 up, 12 in 2024-08-30T23:39:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:39:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:39:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/687785528' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:39:17.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/687785528' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:17.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:16 smithi173 ceph-mon[20554]: osdmap e443: 12 total, 11 up, 12 in 2024-08-30T23:39:17.348 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:17.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:17.394 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:17.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:17.529 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:17.597 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:17.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:17.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:17.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:17.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:17.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:17.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:17.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:17.982 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: pgmap v1046: 27 pgs: 4 peering, 3 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 753 op/s; 21628/235383 objects degraded (9.188%); 10097/235383 objects misplaced (4.290%); 16 MiB/s, 257 objects/s recovering 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:39:17.983 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:39:17.984 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:39:17.984 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:17 smithi136 ceph-mon[20207]: osdmap e444: 12 total, 11 up, 12 in 2024-08-30T23:39:18.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:18.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:18.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:18.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: pgmap v1046: 27 pgs: 4 peering, 3 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 753 op/s; 21628/235383 objects degraded (9.188%); 10097/235383 objects misplaced (4.290%); 16 MiB/s, 257 objects/s recovering 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:39:18.186 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:39:18.187 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:17 smithi106 ceph-mon[16603]: osdmap e444: 12 total, 11 up, 12 in 2024-08-30T23:39:18.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:18.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:18.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:39:18.304 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:39:18.304 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:39:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:39:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:39:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:39:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:39:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: pgmap v1046: 27 pgs: 4 peering, 3 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+clean+remapped, 1 active+recovering+remapped, 2 active+undersized+degraded, 12 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 753 op/s; 21628/235383 objects degraded (9.188%); 10097/235383 objects misplaced (4.290%); 16 MiB/s, 257 objects/s recovering 2024-08-30T23:39:18.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:39:18.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:39:18.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:39:18.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:39:18.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:17 smithi173 ceph-mon[20554]: osdmap e444: 12 total, 11 up, 12 in 2024-08-30T23:39:18.328 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:39:18.328 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:39:18.355 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:39:18.355 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:39:18.363 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:39:18.363 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:39:18.363 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:39:18.363 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:39:18.377 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:39:18.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:39:18.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:18.398 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:39:18.399 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:39:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:18.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:18.422 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:39:18.422 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:39:18.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.540 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:18.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:18.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:18.650 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:39:18.650 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:39:18.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:18.673 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:39:18.674 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:39:18.694 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:39:18.694 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:39:18.720 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:39:18.720 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:39:18.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:39:18.750 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:39:18.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:18.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:18.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:39:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:18.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:18.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:39:18.971 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:39:18.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:39:18.972 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:39:18.972 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-30T23:39:18.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5 2024-08-30T23:39:19.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:19.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:39:19.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:39:19.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3038673471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:19.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:18 smithi106 ceph-mon[16603]: osdmap e445: 12 total, 11 up, 12 in 2024-08-30T23:39:19.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:19.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:19.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:19 smithi136 systemd[1]: Stopping Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:39:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:39:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:39:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:39:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:39:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-08-30T23:39:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:39:19.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:39:19.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3038673471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:19.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:18 smithi136 ceph-mon[20207]: osdmap e445: 12 total, 11 up, 12 in 2024-08-30T23:39:19.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:39:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3038673471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:18 smithi173 ceph-mon[20554]: osdmap e445: 12 total, 11 up, 12 in 2024-08-30T23:39:19.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:19.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:19.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:39:19.530 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:39:19.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:19.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:19.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:19.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:19.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:19.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:39:19.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:19.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:19.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:20.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:20.085 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:20.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:20.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:20.214 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:39:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:19 smithi136 ceph-mon[20207]: pgmap v1049: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 2 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 13 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s wr, 787 op/s; 17364/229326 objects degraded (7.572%); 12405/229326 objects misplaced (5.409%); 25 MiB/s, 410 objects/s recovering 2024-08-30T23:39:20.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:39:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:19 smithi173 ceph-mon[20554]: pgmap v1049: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 2 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 13 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s wr, 787 op/s; 17364/229326 objects degraded (7.572%); 12405/229326 objects misplaced (5.409%); 25 MiB/s, 410 objects/s recovering 2024-08-30T23:39:20.334 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:39:20.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:19 smithi106 ceph-mon[16603]: pgmap v1049: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 4 peering, 2 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 2 active+undersized+degraded, 13 active+clean; 4.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s wr, 787 op/s; 17364/229326 objects degraded (7.572%); 12405/229326 objects misplaced (5.409%); 25 MiB/s, 410 objects/s recovering 2024-08-30T23:39:20.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:20.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:20.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:39:20.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:20.583 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:20.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:20.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:20.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:20.832 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:20.867 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:20.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:20.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:20.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:20 smithi136 ceph-mon[20207]: osdmap e446: 12 total, 11 up, 12 in 2024-08-30T23:39:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:20 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 17364/229326 objects degraded (7.572%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:20 smithi136 ceph-mon[20207]: osd.5 marked itself down and dead 2024-08-30T23:39:21.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[30446]: 2024-08-30T23:39:20.784+0000 7f6ea1afe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-30T23:39:21.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[30446]: 2024-08-30T23:39:20.784+0000 7f6ea1afe640 -1 osd.5 446 *** Got signal Terminated *** 2024-08-30T23:39:21.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[30446]: 2024-08-30T23:39:20.784+0000 7f6ea1afe640 -1 osd.5 446 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-30T23:39:21.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:21.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:20 smithi106 ceph-mon[16603]: osdmap e446: 12 total, 11 up, 12 in 2024-08-30T23:39:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:20 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 17364/229326 objects degraded (7.572%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:20 smithi106 ceph-mon[16603]: osd.5 marked itself down and dead 2024-08-30T23:39:21.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:21.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:21.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:21.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:20 smithi173 ceph-mon[20554]: osdmap e446: 12 total, 11 up, 12 in 2024-08-30T23:39:21.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:20 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 17364/229326 objects degraded (7.572%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:21.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:20 smithi173 ceph-mon[20554]: osd.5 marked itself down and dead 2024-08-30T23:39:21.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:21.378 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:21.378 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:21.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:21.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:21.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:21.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:39:21.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:21.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:21.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.722 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:21.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:21.789 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:21 smithi136 podman[93947]: 2024-08-30 23:39:21.586280097 +0000 UTC m=+0.910730924 container died 0b63d44cb6ea04d1ef511f4a2e3cdf72a08c204b2575423b7cf39a514c295334 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-08-30T23:39:21.797 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:21.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:21.845 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:21.872 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:39:21.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:21.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:21.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:22.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:22.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:22.160 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:22 smithi136 podman[93947]: 2024-08-30 23:39:21.99996136 +0000 UTC m=+1.324412189 container cleanup 0b63d44cb6ea04d1ef511f4a2e3cdf72a08c204b2575423b7cf39a514c295334 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-30T23:39:22.160 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:22 smithi136 bash[93947]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: Health check update: 2 osds down (OSD_DOWN) 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: osdmap e447: 12 total, 10 up, 12 in 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1850048870' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: pgmap v1052: 27 pgs: 2 peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 stale+active+undersized+degraded, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 17364/225477 objects degraded (7.701%); 11819/225477 objects misplaced (5.242%); 24 MiB/s, 386 objects/s recovering 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1850048870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:39:22.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:21 smithi136 ceph-mon[20207]: osdmap e448: 12 total, 10 up, 12 in 2024-08-30T23:39:22.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:22.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:22.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:22.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: Health check update: 2 osds down (OSD_DOWN) 2024-08-30T23:39:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: osdmap e447: 12 total, 10 up, 12 in 2024-08-30T23:39:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1850048870' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:39:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: pgmap v1052: 27 pgs: 2 peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 stale+active+undersized+degraded, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 17364/225477 objects degraded (7.701%); 11819/225477 objects misplaced (5.242%); 24 MiB/s, 386 objects/s recovering 2024-08-30T23:39:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1850048870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:39:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:21 smithi173 ceph-mon[20554]: osdmap e448: 12 total, 10 up, 12 in 2024-08-30T23:39:22.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:22.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: Health check update: 2 osds down (OSD_DOWN) 2024-08-30T23:39:22.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: osdmap e447: 12 total, 10 up, 12 in 2024-08-30T23:39:22.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1850048870' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:39:22.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: pgmap v1052: 27 pgs: 2 peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 stale+active+undersized+degraded, 2 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 14 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s; 17364/225477 objects degraded (7.701%); 11819/225477 objects misplaced (5.242%); 24 MiB/s, 386 objects/s recovering 2024-08-30T23:39:22.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:22.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1850048870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:39:22.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:21 smithi106 ceph-mon[16603]: osdmap e448: 12 total, 10 up, 12 in 2024-08-30T23:39:22.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:22.464 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:22.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:39:22.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:22.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:22.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:22.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:22.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:22.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:22.877 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:22.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:22.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:22.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:23.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:23.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:23.137 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:22 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-08-30T23:39:23.137 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:22 smithi136 ceph-mon[20207]: osdmap e449: 12 total, 10 up, 12 in 2024-08-30T23:39:23.137 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:22 smithi136 podman[94103]: 2024-08-30 23:39:22.82424059 +0000 UTC m=+1.307351015 container remove 0b63d44cb6ea04d1ef511f4a2e3cdf72a08c204b2575423b7cf39a514c295334 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-30T23:39:23.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:23.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:23.239 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:22 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-08-30T23:39:23.239 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:22 smithi106 ceph-mon[16603]: osdmap e449: 12 total, 10 up, 12 in 2024-08-30T23:39:23.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:22 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-08-30T23:39:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:22 smithi173 ceph-mon[20554]: osdmap e449: 12 total, 10 up, 12 in 2024-08-30T23:39:23.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:23.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:23.388 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:23.388 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:23.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.470 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:23.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:23.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:23.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:23.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:23.719 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:23.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:23.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:23.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:23.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:23.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:24.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:39:24.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:24.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:24.185 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:39:24.191 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:39:24.191 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4241677780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:24.192 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:23 smithi106 ceph-mon[16603]: pgmap v1055: 27 pgs: 2 peering, 1 unknown, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 stale+active+undersized+degraded, 2 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 17932/223152 objects degraded (8.036%); 14587/223152 objects misplaced (6.537%); 17 MiB/s, 273 objects/s recovering 2024-08-30T23:39:24.192 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:39:24.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:24.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4241677780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:24.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:23 smithi136 ceph-mon[20207]: pgmap v1055: 27 pgs: 2 peering, 1 unknown, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 stale+active+undersized+degraded, 2 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 17932/223152 objects degraded (8.036%); 14587/223152 objects misplaced (6.537%); 17 MiB/s, 273 objects/s recovering 2024-08-30T23:39:24.209 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:39:24.209 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:39:24.215 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:24.231 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:39:24.231 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:39:24.252 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:39:24.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:39:24.253 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:39:24.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:39:24.274 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:39:24.298 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:39:24.298 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:39:24.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4241677780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:23 smithi173 ceph-mon[20554]: pgmap v1055: 27 pgs: 2 peering, 1 unknown, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 1 stale+active+undersized+degraded, 2 active+clean+remapped, 2 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 500 op/s; 17932/223152 objects degraded (8.036%); 14587/223152 objects misplaced (6.537%); 17 MiB/s, 273 objects/s recovering 2024-08-30T23:39:24.325 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:39:24.325 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:39:24.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:24.353 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:39:24.353 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:39:24.374 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:39:24.374 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:39:24.397 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:39:24.397 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:39:24.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:24.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:24.527 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:24 smithi136 podman[94389]: 2024-08-30 23:39:24.107489107 +0000 UTC m=+0.016270956 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:39:24.527 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:24 smithi136 podman[94389]: 2024-08-30 23:39:24.224316481 +0000 UTC m=+0.133098312 container create 3138373a83704ae37c170bfa40d6c5590ef7ca957e0a941c891f7c661cfc0d63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:39:24.527 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:24 smithi136 podman[94389]: 2024-08-30 23:39:24.435654144 +0000 UTC m=+0.344435975 container init 3138373a83704ae37c170bfa40d6c5590ef7ca957e0a941c891f7c661cfc0d63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-08-30T23:39:24.527 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:24 smithi136 podman[94389]: 2024-08-30 23:39:24.438967634 +0000 UTC m=+0.347749466 container start 3138373a83704ae37c170bfa40d6c5590ef7ca957e0a941c891f7c661cfc0d63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-08-30T23:39:24.527 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:24 smithi136 podman[94389]: 2024-08-30 23:39:24.527216383 +0000 UTC m=+0.435998234 container attach 3138373a83704ae37c170bfa40d6c5590ef7ca957e0a941c891f7c661cfc0d63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-30T23:39:24.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:24.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:24.604 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:24.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:24.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:24.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:24.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:24.940 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:24.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:24.962 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:25.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:39:25.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:25.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:25.159 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:39:24 smithi136 podman[94389]: 2024-08-30 23:39:24.891583157 +0000 UTC m=+0.800364989 container died 3138373a83704ae37c170bfa40d6c5590ef7ca957e0a941c891f7c661cfc0d63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-08-30T23:39:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:39:25.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:39:25.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:25.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:25.345 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:25.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:25.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:25.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:25.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service' 2024-08-30T23:39:25.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:25.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:25.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-30T23:39:25.695 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-30T23:39:25.695 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-08-30T23:39:25.695 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi136.front.sepia.ceph.com 2024-08-30T23:39:25.695 INFO:tasks.thrashosds.thrasher:remote for exp osd 1 is ubuntu@smithi106.front.sepia.ceph.com 2024-08-30T23:39:25.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-08-30T23:39:25.720 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:25 smithi106 ceph-mon[16603]: pgmap v1056: 27 pgs: 2 peering, 1 unknown, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 2 active+clean+remapped, 1 undersized+degraded+peered, 3 active+undersized+degraded, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 27392/222636 objects degraded (12.303%); 14587/222636 objects misplaced (6.552%); 12 MiB/s, 193 objects/s recovering 2024-08-30T23:39:25.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:25.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:25.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:25.905 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:39:25.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:25.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:25.962 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:25 smithi136 ceph-mon[20207]: pgmap v1056: 27 pgs: 2 peering, 1 unknown, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 2 active+clean+remapped, 1 undersized+degraded+peered, 3 active+undersized+degraded, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 27392/222636 objects degraded (12.303%); 14587/222636 objects misplaced (6.552%); 12 MiB/s, 193 objects/s recovering 2024-08-30T23:39:26.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:26.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:25 smithi173 ceph-mon[20554]: pgmap v1056: 27 pgs: 2 peering, 1 unknown, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 2 active+clean+remapped, 1 undersized+degraded+peered, 3 active+undersized+degraded, 13 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 27392/222636 objects degraded (12.303%); 14587/222636 objects misplaced (6.552%); 12 MiB/s, 193 objects/s recovering 2024-08-30T23:39:26.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:26.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:26.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:26.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:26.362 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:39:26.362 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:39:26.362 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:39:26.362 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:39:26.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:26.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:26.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:26.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:26.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:26.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:26.710 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:39:26.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:26.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:26.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:26.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:26.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:26.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:26.908 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 27392/222636 objects degraded (12.303%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:39:26.908 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:26 smithi136 ceph-mon[20207]: osdmap e450: 12 total, 10 up, 12 in 2024-08-30T23:39:26.908 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4282061153' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:26.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:39:27.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:27.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 27392/222636 objects degraded (12.303%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:39:27.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:26 smithi173 ceph-mon[20554]: osdmap e450: 12 total, 10 up, 12 in 2024-08-30T23:39:27.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4282061153' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 27392/222636 objects degraded (12.303%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:39:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:26 smithi106 ceph-mon[16603]: osdmap e450: 12 total, 10 up, 12 in 2024-08-30T23:39:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4282061153' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:27.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:27.148 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:27.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:39:27.224 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:27.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:27.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:27.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:27.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:27.469 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:39:27.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:27.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:27.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:27.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:27.675 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:27.713 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:39:27.713 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:39:27.713 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:39:27.714 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:39:27.714 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:39:27.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:27.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:27.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:27.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:27.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:27.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:28.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4282061153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:27 smithi136 ceph-mon[20207]: osdmap e451: 12 total, 10 up, 12 in 2024-08-30T23:39:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:27 smithi136 ceph-mon[20207]: pgmap v1059: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 41101/222132 objects degraded (18.503%); 14560/222132 objects misplaced (6.555%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:39:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4282061153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:27 smithi173 ceph-mon[20554]: osdmap e451: 12 total, 10 up, 12 in 2024-08-30T23:39:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:27 smithi173 ceph-mon[20554]: pgmap v1059: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 41101/222132 objects degraded (18.503%); 14560/222132 objects misplaced (6.555%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:39:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4282061153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:27 smithi106 ceph-mon[16603]: osdmap e451: 12 total, 10 up, 12 in 2024-08-30T23:39:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:27 smithi106 ceph-mon[16603]: pgmap v1059: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 194 op/s; 41101/222132 objects degraded (18.503%); 14560/222132 objects misplaced (6.555%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:39:28.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:28.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:28.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:28.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:28.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:39:28.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:28.383 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:28.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:28.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:28.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:28.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:28.576 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:28.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:28.631 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:28.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:28.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:28.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:28.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:28.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:28.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:29.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.24460.0:193992 2.7 2:e8d619b4:::benchmark_data_smithi173_47195_object96984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-08-30T23:39:29.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 586.27862421999998, 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60785984599999998, 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.24460", 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/4144298757", 2024-08-30T23:39:29.056 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 193992 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775251+0000", 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775252+0000", 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.528e-06 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775258+0000", 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.485e-06 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775273+0000", 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5355000000000001e-05 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775281+0000", 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5140000000000004e-06 2024-08-30T23:39:29.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775342+0000", 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1334000000000006e-05 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775448+0000", 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010587 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.775514+0000", 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5747000000000006e-05 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.779105+0000", 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035911110000000001 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:42.780025+0000", 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092040100000000005 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383045+0000", 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60302017900000005 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383083+0000", 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7670999999999999e-05 2024-08-30T23:39:29.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:29:43.383111+0000", 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7651e-05 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:22811 3.c e215/206 3:362f756c:::benchmark_data_smithi173_60780_object22810:head v 215'1421, mlcod=215'1418)", 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 543.04848708999998, 2024-08-30T23:39:29.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32690804400000001, 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005388+0000", 2024-08-30T23:39:29.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005389+0000", 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.005420+0000", 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1340000000000001e-05 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331407+0000", 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32598661899999998 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331414+0000", 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7409999999999992e-06 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331466+0000", 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1319000000000001e-05 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.331490+0000", 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4544000000000001e-05 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332257+0000", 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076687200000000004 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:30:26.332296+0000", 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8828000000000002e-05 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.068 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:206871 3.a 3:5d60dc93:::benchmark_data_smithi173_60780_object103417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 345.03332657499999, 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45068615499999998, 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 206871 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020548+0000", 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020550+0000", 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.559e-06 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020557+0000", 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5810000000000002e-06 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020573+0000", 2024-08-30T23:39:29.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6767999999999999e-05 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020581+0000", 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7109999999999995e-06 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020645+0000", 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3559000000000006e-05 2024-08-30T23:39:29.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020750+0000", 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010555100000000001 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.020853+0000", 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103207 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.025592+0000", 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047385489999999999 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.053826+0000", 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028234459999999999 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471165+0000", 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41733862300000002 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471204+0000", 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8769000000000001e-05 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.471235+0000", 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0818000000000002e-05 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24732 4.a 4:5449864d:::benchmark_data_smithi173_74914_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 312.54498109000002, 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24409322, 2024-08-30T23:39:29.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24732 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508894+0000", 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9800000000000003e-07 2024-08-30T23:39:29.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.508987+0000", 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2602999999999994e-05 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509002+0000", 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509009+0000", 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509048+0000", 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9002999999999999e-05 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509127+0000", 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9573000000000005e-05 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.509220+0000", 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2218000000000004e-05 2024-08-30T23:39:29.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.510205+0000", 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00098522999999999992 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.550346+0000", 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040140995999999998 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752876+0000", 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20253054600000001 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752927+0000", 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0686999999999998e-05 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752987+0000", 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9998999999999999e-05 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:24732 4.a 4:5449864d:::benchmark_data_smithi173_74914_object24731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e315)", 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 312.53915958800002, 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23822328400000001, 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for ondisk", 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 24732 2024-08-30T23:39:29.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514715+0000", 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514716+0000", 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4300000000000003e-07 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514742+0000", 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6389000000000001e-05 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514749+0000", 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0879999999999997e-06 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514755+0000", 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999998e-06 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514806+0000", 2024-08-30T23:39:29.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1598999999999998e-05 2024-08-30T23:39:29.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for ondisk", 2024-08-30T23:39:29.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.514835+0000", 2024-08-30T23:39:29.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8839999999999998e-05 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:16.752939+0000", 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23810340499999999 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.091 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49440 4.9 4:9de739e2:::benchmark_data_smithi173_74914_object49439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 288.42381795900002, 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22086719499999999, 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49440 2024-08-30T23:39:29.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630057+0000", 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5999999999999999e-07 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630090+0000", 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2196000000000001e-05 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630096+0000", 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3160000000000002e-06 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630102+0000", 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9050000000000002e-06 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630553+0000", 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045081400000000001 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630557+0000", 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6979999999999997e-06 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845169+0000", 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21461201499999999 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845239+0000", 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0062999999999996e-05 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.845330+0000", 2024-08-30T23:39:29.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0912e-05 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848252+0000", 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002922043 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848274+0000", 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1472999999999999e-05 2024-08-30T23:39:29.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850752+0000", 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024776540000000001 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850813+0000", 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1593999999999993e-05 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.850924+0000", 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011105200000000001 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49445 4.9 4:915080b3:::benchmark_data_smithi173_74914_object49444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:39:29.100 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 288.42376648700002, 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22553746099999999, 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49445 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630108+0000", 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630109+0000", 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630137+0000", 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8206999999999999e-05 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630143+0000", 2024-08-30T23:39:29.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0329999999999997e-06 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630148+0000", 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4769999999999997e-06 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630574+0000", 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042622100000000001 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.630579+0000", 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9989999999999999e-06 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846364+0000", 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21578499100000001 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846489+0000", 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012525599999999999 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.846624+0000", 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013508199999999999 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848317+0000", 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016928519999999999 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.848441+0000", 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012401 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855479+0000", 2024-08-30T23:39:29.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070380499999999997 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855541+0000", 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1371000000000006e-05 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.855646+0000", 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010543300000000001 2024-08-30T23:39:29.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62189 4.9 4:952cc0c3:::benchmark_data_smithi173_74914_object62188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.49074741800001, 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.475062977, 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62189 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563128+0000", 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3799999999999996e-07 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563190+0000", 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1838e-05 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563201+0000", 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0794e-05 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563206+0000", 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.429e-06 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563251+0000", 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4363999999999997e-05 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563339+0000", 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8675000000000001e-05 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.563428+0000", 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8559000000000004e-05 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.564850+0000", 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014217069999999999 2024-08-30T23:39:29.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.565634+0000", 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00078473200000000001 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038076+0000", 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.47244159499999999 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038142+0000", 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6005000000000002e-05 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038191+0000", 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8541000000000002e-05 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:62196 4.9 4:91ca1c4e:::benchmark_data_smithi173_74914_object62195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.28070995899998, 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310692415, 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62196 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773165+0000", 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773167+0000", 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5409999999999999e-06 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773206+0000", 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9888000000000002e-05 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773222+0000", 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5690000000000001e-05 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773228+0000", 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9819999999999996e-06 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773279+0000", 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0417000000000002e-05 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773374+0000", 2024-08-30T23:39:29.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5396000000000006e-05 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773464+0000", 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0315000000000005e-05 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775336+0000", 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018712749999999999 2024-08-30T23:39:29.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.779671+0000", 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043352089999999996 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083790+0000", 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.304119056 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083827+0000", 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6863999999999998e-05 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083857+0000", 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0781999999999998e-05 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:62189 4.9 e337/327 ondisk, result = 0)", 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.27839373900002, 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26272571900000002, 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775481+0000", 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775482+0000", 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.64e-07 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.775487+0000", 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.037983+0000", 2024-08-30T23:39:29.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26249586200000002 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.037988+0000", 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.561e-06 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038060+0000", 2024-08-30T23:39:29.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2241000000000002e-05 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038070+0000", 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.038207+0000", 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013733899999999999 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.129 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:44467 5.5 5:aedb68b9:::benchmark_data_smithi173_87639_object44466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:11.683084+0000", 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.370790618000001, 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23497832199999999, 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44467 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683084+0000", 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683084+0000", 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683089+0000", 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8840000000000002e-06 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683121+0000", 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1977000000000002e-05 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683144+0000", 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3078000000000002e-05 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683154+0000", 2024-08-30T23:39:29.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.005e-05 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683220+0000", 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5650999999999999e-05 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683333+0000", 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011322799999999999 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.683440+0000", 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010686799999999999 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.684869+0000", 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014291390000000001 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917948+0000", 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23307857600000001 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917999+0000", 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0929e-05 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.918063+0000", 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3942000000000005e-05 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:44467 5.5 e410/389 ondisk, result = 0)", 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:11.685295+0000", 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.368580058000006, 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23277415100000001, 2024-08-30T23:39:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.685295+0000", 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.685295+0000", 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.685296+0000", 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.68e-07 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.685302+0000", 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5409999999999997e-06 2024-08-30T23:39:29.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917886+0000", 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23258390300000001 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917889+0000", 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2349999999999999e-06 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917937+0000", 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7783e-05 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917944+0000", 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7000000000000002e-06 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.918069+0000", 2024-08-30T23:39:29.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012532099999999999 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:44479 5.2 5:4271f637:::benchmark_data_smithi173_87639_object44478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:11.915069+0000", 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.138806348000003, 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25316217499999999, 2024-08-30T23:39:29.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44479 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915069+0000", 2024-08-30T23:39:29.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915069+0000", 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915071+0000", 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.419e-06 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915109+0000", 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8260000000000003e-05 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915129+0000", 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9783999999999999e-05 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915137+0000", 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9359999999999999e-06 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915202+0000", 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5240000000000006e-05 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915369+0000", 2024-08-30T23:39:29.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000167035 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.915528+0000", 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159166 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917077+0000", 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001548359 2024-08-30T23:39:29.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917893+0000", 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081605200000000005 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168145+0000", 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25025234400000002 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168191+0000", 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5785999999999998e-05 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168231+0000", 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9793999999999999e-05 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:44479 5.2 e410/394 ondisk, result = 0)", 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:11.917872+0000", 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.136003052000007, 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25036549899999999, 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917872+0000", 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917872+0000", 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917872+0000", 2024-08-30T23:39:29.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9999999999999999e-07 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:11.917874+0000", 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1859999999999999e-06 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168034+0000", 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25015959799999998 2024-08-30T23:39:29.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168047+0000", 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2541000000000001e-05 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168129+0000", 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2651999999999994e-05 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168141+0000", 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1494e-05 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.168237+0000", 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6728e-05 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59003 5.2 5.20ed2842 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.906681040999999, 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96588689100000003, 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:29.155 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59003 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147195+0000", 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.429e-06 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147224+0000", 2024-08-30T23:39:29.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9014999999999999e-05 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147237+0000", 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.311e-05 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147244+0000", 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1160000000000004e-06 2024-08-30T23:39:29.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147299+0000", 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5275000000000002e-05 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147305+0000", 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.302e-06 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113050+0000", 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96574470800000001 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113081+0000", 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0936000000000003e-05 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59011 5.2 5.db576df2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.595735883000003, 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65497856600000004, 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:39:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59011 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:39:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458149+0000", 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8779999999999993e-06 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458186+0000", 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6718000000000003e-05 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458199+0000", 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3811999999999999e-05 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458205+0000", 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6879999999999996e-06 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458260+0000", 2024-08-30T23:39:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.516e-05 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458269+0000", 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3680000000000002e-06 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113109+0000", 2024-08-30T23:39:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65484074999999997 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113118+0000", 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1920000000000005e-06 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59016 5.2 5.92c60782 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.483906187000002, 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54317872599999995, 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59016 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569970+0000", 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2449999999999999e-06 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570027+0000", 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7083999999999998e-05 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570039+0000", 2024-08-30T23:39:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1664999999999999e-05 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570044+0000", 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6189999999999999e-06 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570095+0000", 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1085999999999998e-05 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570103+0000", 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5830000000000001e-06 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113141+0000", 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54303761500000003 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113148+0000", 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8290000000000002e-06 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59017 5.2 5.eccc78f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.327822230000002, 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38712284699999999, 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59017 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726060+0000", 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2359999999999998e-06 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726121+0000", 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0909000000000002e-05 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726151+0000", 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0292999999999999e-05 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726165+0000", 2024-08-30T23:39:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.428e-05 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726324+0000", 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015868300000000001 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:39:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726349+0000", 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4508e-05 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113170+0000", 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38682103699999998 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113176+0000", 2024-08-30T23:39:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.2 426/420 [PushReplyOp(5:4213227c:::benchmark_data_smithi173_87639_object5797:head)])", 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.530900543999998, 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64882206499999995, 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522976+0000", 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136e-06 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522980+0000", 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3109999999999999e-06 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169121+0000", 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64614092499999998 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169129+0000", 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0099999999999995e-06 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169226+0000", 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7223000000000001e-05 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.171796+0000", 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025704600000000001 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.2 432/420 [PushReplyOp(5:4f272be2:::benchmark_data_smithi173_87639_object27138:head)])", 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 34.181599466000002, 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43174907800000001, 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:39:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872277+0000", 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.674e-06 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872285+0000", 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4529999999999996e-06 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302111+0000", 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42982598799999999 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302119+0000", 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1990000000000006e-06 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302204+0000", 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5348999999999999e-05 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:39:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.304025+0000", 2024-08-30T23:39:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018204149999999999 2024-08-30T23:39:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:39:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:39:29.188 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:39:29.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:29.189 INFO:teuthology.orchestra.run.smithi106.stdout:4.d 2024-08-30T23:39:29.189 INFO:teuthology.orchestra.run.smithi106.stdout:4.b 2024-08-30T23:39:29.190 INFO:teuthology.orchestra.run.smithi106.stdout:4.f 2024-08-30T23:39:29.190 INFO:teuthology.orchestra.run.smithi106.stdout:4.a 2024-08-30T23:39:29.190 INFO:teuthology.orchestra.run.smithi106.stdout:4.6 2024-08-30T23:39:29.190 INFO:teuthology.orchestra.run.smithi106.stdout:4.3 2024-08-30T23:39:29.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:29.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/116673160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:29.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:29.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:29.350 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:39:29.351 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:39:29.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:39:29.372 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:39:29.393 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:39:29.393 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:39:29.394 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/116673160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:29.415 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:39:29.416 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:39:29.416 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:39:29.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:29.438 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:39:29.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:39:29.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:29.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:39:29.460 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:39:29.488 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:39:29.489 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:39:29.515 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:39:29.516 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:39:29.538 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:39:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:39:29.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:29.561 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:39:29.561 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:39:29.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/116673160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:29.584 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:29.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:29.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:29.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:29.830 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:29.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:29.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:29.955 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:30.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:30.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:30.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:39:30.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:39:30.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:30.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:30.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:30.349 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:30.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:30 smithi106 ceph-mon[16603]: pgmap v1060: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 923 B/s rd, 29 KiB/s wr, 97 op/s; 41016/222120 objects degraded (18.466%); 13943/222120 objects misplaced (6.277%); 15 MiB/s, 240 objects/s recovering 2024-08-30T23:39:30.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:30.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 4.a --file /var/log/ceph/exp.4.a.1 2024-08-30T23:39:30.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:30.470 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:30 smithi136 ceph-mon[20207]: pgmap v1060: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 923 B/s rd, 29 KiB/s wr, 97 op/s; 41016/222120 objects degraded (18.466%); 13943/222120 objects misplaced (6.277%); 15 MiB/s, 240 objects/s recovering 2024-08-30T23:39:30.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:30.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:30 smithi173 ceph-mon[20554]: pgmap v1060: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 923 B/s rd, 29 KiB/s wr, 97 op/s; 41016/222120 objects degraded (18.466%); 13943/222120 objects misplaced (6.277%); 15 MiB/s, 240 objects/s recovering 2024-08-30T23:39:30.590 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.612 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:30.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:30.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:30.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:30.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:30.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:30.868 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:30.868 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:30.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:30.981 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:30.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:31.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:31.106 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:39:31.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:39:31.172 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:39:31.201 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:31.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:31.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:31.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:31.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:31.438 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:31 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:31.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 41016/222120 objects degraded (18.466%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:39:31.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:31 smithi106 ceph-mon[16603]: osdmap e452: 12 total, 10 up, 12 in 2024-08-30T23:39:31.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:31.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:31.488 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:31 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:31.488 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 41016/222120 objects degraded (18.466%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:39:31.488 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:31 smithi136 ceph-mon[20207]: osdmap e452: 12 total, 10 up, 12 in 2024-08-30T23:39:31.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:39:31.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:31 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:31.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 41016/222120 objects degraded (18.466%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:39:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:31 smithi173 ceph-mon[20554]: osdmap e452: 12 total, 10 up, 12 in 2024-08-30T23:39:31.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:31.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:31.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:31.732 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:31.802 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:31.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:31.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:31.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:31.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:31.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:32.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:32.105 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:32.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:32.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:32.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:32.283 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:32.283 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:32.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:32.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:32.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:32.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:32 smithi136 ceph-mon[20207]: pgmap v1062: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s; 41016/222120 objects degraded (18.466%); 13943/222120 objects misplaced (6.277%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:39:32.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:32 smithi173 ceph-mon[20554]: pgmap v1062: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s; 41016/222120 objects degraded (18.466%); 13943/222120 objects misplaced (6.277%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:39:32.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:32.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:32.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:32 smithi106 ceph-mon[16603]: pgmap v1062: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s; 41016/222120 objects degraded (18.466%); 13943/222120 objects misplaced (6.277%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:39:32.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:32.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:32.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:32.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:32.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:32.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:39:32.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:32.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:33.076 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.099 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:33.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:33.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:39:33.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:33.215 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:39:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.231 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:33.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:33.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:33.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:33.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:33.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:33.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:33.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:33.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:33.733 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:33.733 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:33.739 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:39:33.831 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:33.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:33.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:33.971 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:33 smithi106 ceph-mon[16603]: pgmap v1063: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s; 40828/222120 objects degraded (18.381%); 13090/222120 objects misplaced (5.893%); 19 MiB/s, 310 objects/s recovering 2024-08-30T23:39:33.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:33.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:33 smithi136 ceph-mon[20207]: pgmap v1063: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s; 40828/222120 objects degraded (18.381%); 13090/222120 objects misplaced (5.893%); 19 MiB/s, 310 objects/s recovering 2024-08-30T23:39:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:33 smithi173 ceph-mon[20554]: pgmap v1063: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 21 op/s; 40828/222120 objects degraded (18.381%); 13090/222120 objects misplaced (5.893%); 19 MiB/s, 310 objects/s recovering 2024-08-30T23:39:34.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:34.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:34.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:34.223 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:34.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:34.348 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:34.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:39:34.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:34.471 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:34.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:34.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:34.640 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:34.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:34.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:34.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:34.841 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:34.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:34.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:34.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:35.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:35.089 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:35.163 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:35.163 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:35.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:35.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:35.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:35.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:35.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:35.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:35.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:35.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:35.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:35.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:35.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:35.836 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:35.935 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:35 smithi106 ceph-mon[16603]: pgmap v1064: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 7 op/s; 40828/222120 objects degraded (18.381%); 13090/222120 objects misplaced (5.893%); 14 MiB/s, 219 objects/s recovering 2024-08-30T23:39:35.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:35.951 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:35 smithi173 ceph-mon[20554]: pgmap v1064: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 7 op/s; 40828/222120 objects degraded (18.381%); 13090/222120 objects misplaced (5.893%); 14 MiB/s, 219 objects/s recovering 2024-08-30T23:39:35.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:35.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:35.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:39:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:35 smithi136 ceph-mon[20207]: pgmap v1064: 27 pgs: 1 active+recovering+degraded+remapped, 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 7 op/s; 40828/222120 objects degraded (18.381%); 13090/222120 objects misplaced (5.893%); 14 MiB/s, 219 objects/s recovering 2024-08-30T23:39:36.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:36.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:36.185 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:36.206 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:36.229 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:36.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.330 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:36.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:36.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:36.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:36.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:36.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:36.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:36.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:36.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:36.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:36.873 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 40828/222120 objects degraded (18.381%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:36.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:36.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:36.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:37.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 40828/222120 objects degraded (18.381%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:37.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:37.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.075 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:37.086 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:37.086 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 40828/222120 objects degraded (18.381%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:37.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:37.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:37.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:37.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:37.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:37.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:37.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:37.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:37.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:37.643 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3211136763' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:39:37.643 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:37 smithi173 ceph-mon[20554]: pgmap v1065: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 36249/222120 objects degraded (16.320%); 17669/222120 objects misplaced (7.955%); 12 MiB/s, 234 objects/s recovering 2024-08-30T23:39:37.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:37.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:39:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:37.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:37.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:37.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:37.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:37.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:37.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:37.944 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3211136763' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:39:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:37 smithi136 ceph-mon[20207]: pgmap v1065: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 36249/222120 objects degraded (16.320%); 17669/222120 objects misplaced (7.955%); 12 MiB/s, 234 objects/s recovering 2024-08-30T23:39:38.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:38.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3211136763' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:39:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:37 smithi106 ceph-mon[16603]: pgmap v1065: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 36249/222120 objects degraded (16.320%); 17669/222120 objects misplaced (7.955%); 12 MiB/s, 234 objects/s recovering 2024-08-30T23:39:38.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:38.252 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:39:38.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:38.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:38.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:38.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:38.464 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:38.464 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:38.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:38.557 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:38.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:38.682 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:38.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:38.806 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:38.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:38.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:38.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:38.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:38 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:38.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3211136763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:39:38.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:38 smithi136 ceph-mon[20207]: osdmap e453: 12 total, 10 up, 12 in 2024-08-30T23:39:39.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:39.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:38 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:39.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3211136763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:39:39.047 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:38 smithi173 ceph-mon[20554]: osdmap e453: 12 total, 10 up, 12 in 2024-08-30T23:39:39.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:38 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:39:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3211136763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:39:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:38 smithi106 ceph-mon[16603]: osdmap e453: 12 total, 10 up, 12 in 2024-08-30T23:39:39.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:39.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:39.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:39.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:39.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:39.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:39.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:39.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:39.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:39:39.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:39.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:39.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:39.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:39.855 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:39.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:39.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:39.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:39.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:39 smithi173 ceph-mon[20554]: 5.e scrub starts 2024-08-30T23:39:39.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:39 smithi173 ceph-mon[20554]: pgmap v1067: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36042/222120 objects degraded (16.226%); 16898/222120 objects misplaced (7.608%); 14 MiB/s, 269 objects/s recovering 2024-08-30T23:39:40.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:39 smithi136 ceph-mon[20207]: 5.e scrub starts 2024-08-30T23:39:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:39 smithi136 ceph-mon[20207]: pgmap v1067: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36042/222120 objects degraded (16.226%); 16898/222120 objects misplaced (7.608%); 14 MiB/s, 269 objects/s recovering 2024-08-30T23:39:40.050 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:39 smithi106 ceph-mon[16603]: 5.e scrub starts 2024-08-30T23:39:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:39 smithi106 ceph-mon[16603]: pgmap v1067: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36042/222120 objects degraded (16.226%); 16898/222120 objects misplaced (7.608%); 14 MiB/s, 269 objects/s recovering 2024-08-30T23:39:40.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:40.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:40.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:40.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:40.398 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:40.398 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:40.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:40.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:40.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:40.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:40.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:40.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:40.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:40.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:40.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:40.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:40.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:41.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:41.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:40 smithi136 ceph-mon[20207]: 5.e scrub ok 2024-08-30T23:39:41.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.043 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:40 smithi173 ceph-mon[20554]: 5.e scrub ok 2024-08-30T23:39:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:40 smithi106 ceph-mon[16603]: 5.e scrub ok 2024-08-30T23:39:41.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:39:41.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:41.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:41.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:41.310 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:41.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:41.428 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:41.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:41.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:41.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:41.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:41.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:41.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:41.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:41 smithi173 ceph-mon[20554]: osdmap e454: 12 total, 10 up, 12 in 2024-08-30T23:39:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 36042/222120 objects degraded (16.226%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:41 smithi173 ceph-mon[20554]: pgmap v1069: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36042/222120 objects degraded (16.226%); 16898/222120 objects misplaced (7.608%); 4.8 MiB/s, 122 objects/s recovering 2024-08-30T23:39:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:41 smithi106 ceph-mon[16603]: osdmap e454: 12 total, 10 up, 12 in 2024-08-30T23:39:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 36042/222120 objects degraded (16.226%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:41 smithi106 ceph-mon[16603]: pgmap v1069: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36042/222120 objects degraded (16.226%); 16898/222120 objects misplaced (7.608%); 4.8 MiB/s, 122 objects/s recovering 2024-08-30T23:39:41.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:41.933 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:41.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:41.941 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:41.941 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:42.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:42.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:41 smithi136 ceph-mon[20207]: osdmap e454: 12 total, 10 up, 12 in 2024-08-30T23:39:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 36042/222120 objects degraded (16.226%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:41 smithi136 ceph-mon[20207]: pgmap v1069: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36042/222120 objects degraded (16.226%); 16898/222120 objects misplaced (7.608%); 4.8 MiB/s, 122 objects/s recovering 2024-08-30T23:39:42.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:42.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:42.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:42.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:42.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:42.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:42.426 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:42.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:42.549 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:42.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:42.674 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:42.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:42.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:42.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:42.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:42.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:42.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:39:43.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:43.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:43.173 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:43.241 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:43.274 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:43.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:43.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:43.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:43.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:43.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:43.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:43.693 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:43.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:43.758 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:43.758 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:43.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:43.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:43.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:43.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:43.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:43 smithi136 ceph-mon[20207]: pgmap v1070: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 12 MiB/s, 234 objects/s recovering 2024-08-30T23:39:44.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:44.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:43 smithi173 ceph-mon[20554]: pgmap v1070: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 12 MiB/s, 234 objects/s recovering 2024-08-30T23:39:44.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:43 smithi106 ceph-mon[16603]: pgmap v1070: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 12 MiB/s, 234 objects/s recovering 2024-08-30T23:39:44.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:44.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:44.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:44.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:44.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:44.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:44.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:44.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:44.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.698 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:44.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:44.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:44.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:44.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:44.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:44.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3047237920,config=2353167246,health=2038898362,kv=2447206621}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2978295890,logm=1312154168}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=633776533}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218094949}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132101477}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1777845836}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3386658990}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684336725}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777394118}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089242139}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612666901}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=31712457}) 2024-08-30T23:39:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882604424}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603964739}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179441750}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076717494}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702105279}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397296879}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887269384}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500304889}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1201922507}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981530326}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414290934}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2949668266}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3574382334}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709707267}) 2024-08-30T23:39:45.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=327708740}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=729367256}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3288855105}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3235327617}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471264107}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321471807}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653267943}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408656975}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87401936}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106089076}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699995984}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311783227}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=356784792}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2833739379}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374327615}) 2024-08-30T23:39:45.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894447369}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252784114}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=62} crc {logm=2138478742,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=937537078,mon_config_key=3306395670}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1379215915}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=860558}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=29,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=53} crc {mon_config_key=653502843,monmap=1193225519,osd_metadata=3634387181,osd_pg_creating=3202411791,osdmap=2764434270}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3809202636}) 2024-08-30T23:39:45.044 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=57} crc {osdmap=4260582738}) 2024-08-30T23:39:45.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:45.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3047237920,config=2353167246,health=2038898362,kv=2447206621}) 2024-08-30T23:39:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2978295890,logm=1312154168}) 2024-08-30T23:39:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=633776533}) 2024-08-30T23:39:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218094949}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132101477}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1777845836}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3386658990}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684336725}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777394118}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089242139}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612666901}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=31712457}) 2024-08-30T23:39:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882604424}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603964739}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179441750}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076717494}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702105279}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397296879}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887269384}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500304889}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1201922507}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981530326}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414290934}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2949668266}) 2024-08-30T23:39:45.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3574382334}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709707267}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=327708740}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=729367256}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3288855105}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3235327617}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471264107}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321471807}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653267943}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408656975}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87401936}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106089076}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699995984}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311783227}) 2024-08-30T23:39:45.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=356784792}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2833739379}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374327615}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894447369}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252784114}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=62} crc {logm=2138478742,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=937537078,mon_config_key=3306395670}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1379215915}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=860558}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=29,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=53} crc {mon_config_key=653502843,monmap=1193225519,osd_metadata=3634387181,osd_pg_creating=3202411791,osdmap=2764434270}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3809202636}) 2024-08-30T23:39:45.078 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=57} crc {osdmap=4260582738}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=3047237920,config=2353167246,health=2038898362,kv=2447206621}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=2978295890,logm=1312154168}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=633776533}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1218094949}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132101477}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1777845836}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3386658990}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684336725}) 2024-08-30T23:39:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777394118}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089242139}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3612666901}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=31712457}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=882604424}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=603964739}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3179441750}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2076717494}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2702105279}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397296879}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887269384}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500304889}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1201922507}) 2024-08-30T23:39:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981530326}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414290934}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2949668266}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3574382334}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1709707267}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=327708740}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=729367256}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3288855105}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3235327617}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1471264107}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321471807}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653267943}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408656975}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=87401936}) 2024-08-30T23:39:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106089076}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699995984}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2311783227}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=356784792}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2833739379}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374327615}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894447369}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4252784114}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=2,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=12,mon_config_key=62} crc {logm=2138478742,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=937537078,mon_config_key=3306395670}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1379215915}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=860558}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=29,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=53} crc {mon_config_key=653502843,monmap=1193225519,osd_metadata=3634387181,osd_pg_creating=3202411791,osdmap=2764434270}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3809202636}) 2024-08-30T23:39:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=57} crc {osdmap=4260582738}) 2024-08-30T23:39:45.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:45.179 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:45.192 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:45.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:45.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:45.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.439 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:45.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:45.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:45.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:45.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:45.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:45.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:45.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:45.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:45.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:45.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:46.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:45 smithi136 ceph-mon[20207]: pgmap v1071: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 11 MiB/s, 183 objects/s recovering 2024-08-30T23:39:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:45 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 35788/222120 objects degraded (16.112%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:46.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:45 smithi173 ceph-mon[20554]: pgmap v1071: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 11 MiB/s, 183 objects/s recovering 2024-08-30T23:39:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:45 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 35788/222120 objects degraded (16.112%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:45 smithi106 ceph-mon[16603]: pgmap v1071: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 11 MiB/s, 183 objects/s recovering 2024-08-30T23:39:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:45 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 35788/222120 objects degraded (16.112%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:46.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:46.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:46.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:46.307 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:46.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:46.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:46.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:39:46.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:46.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:46.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:46.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:46.730 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:46.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:46.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:46.903 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:46.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:46.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:47.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:47.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:47.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:47.171 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:47.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:47.277 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:47.277 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:47.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:47.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:47.418 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:47.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:47.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:47.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:47.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:47.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:47.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:47 smithi106 ceph-mon[16603]: pgmap v1072: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:39:47.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:47.903 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:47 smithi173 ceph-mon[20554]: pgmap v1072: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:39:47.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:47.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:47.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:39:48.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:48.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:47 smithi136 ceph-mon[20207]: pgmap v1072: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35788/222120 objects degraded (16.112%); 15744/222120 objects misplaced (7.088%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:39:48.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:48.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:48.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:48.254 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:48.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:48.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:48.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:39:48.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:48.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:48.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.533 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:48.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:48.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:48.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:48.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:48.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:48.799 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:48.799 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:48.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:48.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:48.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:49.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:49.027 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:49.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:48 smithi173 ceph-mon[20554]: 5.2 scrub starts 2024-08-30T23:39:49.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:48 smithi173 ceph-mon[20554]: 5.2 scrub ok 2024-08-30T23:39:49.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:48 smithi106 ceph-mon[16603]: 5.2 scrub starts 2024-08-30T23:39:49.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:48 smithi106 ceph-mon[16603]: 5.2 scrub ok 2024-08-30T23:39:49.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:49.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:49.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:49.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:48 smithi136 ceph-mon[20207]: 5.2 scrub starts 2024-08-30T23:39:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:48 smithi136 ceph-mon[20207]: 5.2 scrub ok 2024-08-30T23:39:49.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:49.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:49.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:49.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:49.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:49.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:39:49.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:49.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:49.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.775 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:49.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:49.828 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:49.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:49.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:49.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:49.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:50.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:50.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.148 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:50.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:50.203 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:49 smithi173 ceph-mon[20554]: pgmap v1073: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35550/222120 objects degraded (16.005%); 14556/222120 objects misplaced (6.553%); 12 MiB/s, 184 objects/s recovering 2024-08-30T23:39:50.203 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:49 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:39:50.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:49 smithi136 ceph-mon[20207]: pgmap v1073: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35550/222120 objects degraded (16.005%); 14556/222120 objects misplaced (6.553%); 12 MiB/s, 184 objects/s recovering 2024-08-30T23:39:50.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:49 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:39:50.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:50.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:50.319 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:50.319 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:49 smithi106 ceph-mon[16603]: pgmap v1073: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35550/222120 objects degraded (16.005%); 14556/222120 objects misplaced (6.553%); 12 MiB/s, 184 objects/s recovering 2024-08-30T23:39:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:49 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:39:50.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:50.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:50.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:50.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:50.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:50.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:50.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:50.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:50.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:50.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:50.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:50.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:51.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:39:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.016 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:51.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:51.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:51.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:51.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:51.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:51.265 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Exporting 4.a info 4.a( v 368'9375 (0'0,368'9375] local-lis/les=332/333 n=2757 ec=306/306 lis/c=332/332 les/c/f=333/333/0 sis=332) 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50012d5d:::benchmark_data_smithi173_74914_object92236:head# 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5001d08a:::benchmark_data_smithi173_74914_object73608:head# 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50020e18:::benchmark_data_smithi173_74914_object80244:head# 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50055b0f:::benchmark_data_smithi173_74914_object72183:head# 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:500a02f1:::benchmark_data_smithi173_74914_object80382:head# 2024-08-30T23:39:51.274 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:500bc44c:::benchmark_data_smithi173_74914_object69343:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:500beaf6:::benchmark_data_smithi173_74914_object70538:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:500e6df1:::benchmark_data_smithi173_74914_object55471:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:500ffd62:::benchmark_data_smithi173_74914_object77580:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5010507f:::benchmark_data_smithi173_74914_object82204:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5011f8c5:::benchmark_data_smithi173_74914_object81161:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:501507ba:::benchmark_data_smithi173_74914_object85009:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5015f817:::benchmark_data_smithi173_74914_object53673:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5016a9a9:::benchmark_data_smithi173_74914_object72494:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5017fcc0:::benchmark_data_smithi173_74914_object53080:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50189899:::benchmark_data_smithi173_74914_object65983:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50194c1e:::benchmark_data_smithi173_74914_object62701:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:501aa6eb:::benchmark_data_smithi173_74914_object53491:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:501b0bd3:::benchmark_data_smithi173_74914_object76480:head# 2024-08-30T23:39:51.275 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:501c3568:::benchmark_data_smithi173_74914_object65823:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:501d5cdf:::benchmark_data_smithi173_74914_object94175:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50216877:::benchmark_data_smithi173_74914_object57545:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5021aeea:::benchmark_data_smithi173_74914_object92858:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5025edf5:::benchmark_data_smithi173_74914_object69381:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5026f9ab:::benchmark_data_smithi173_74914_object68072:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50278f46:::benchmark_data_smithi173_74914_object83187:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:502849d1:::benchmark_data_smithi173_74914_object53075:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:502b0972:::benchmark_data_smithi173_74914_object56548:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:502b8062:::benchmark_data_smithi173_74914_object89018:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:502c7392:::benchmark_data_smithi173_74914_object73754:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:502d233d:::benchmark_data_smithi173_74914_object57476:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:502f4131:::benchmark_data_smithi173_74914_object67957:head# 2024-08-30T23:39:51.276 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50304878:::benchmark_data_smithi173_74914_object67462:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:503260f3:::benchmark_data_smithi173_74914_object63568:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5033b7ac:::benchmark_data_smithi173_74914_object73509:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50351923:::benchmark_data_smithi173_74914_object97683:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:503d109e:::benchmark_data_smithi173_74914_object57753:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:503d2e69:::benchmark_data_smithi173_74914_object68096:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:503d519d:::benchmark_data_smithi173_74914_object60644:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:503dbc54:::benchmark_data_smithi173_74914_object84508:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:503f993d:::benchmark_data_smithi173_74914_object60789:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:504050ec:::benchmark_data_smithi173_74914_object56585:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50491efd:::benchmark_data_smithi173_74914_object94701:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:504cd2f7:::benchmark_data_smithi173_74914_object87084:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:504d4424:::benchmark_data_smithi173_74914_object76211:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:504de322:::benchmark_data_smithi173_74914_object76014:head# 2024-08-30T23:39:51.277 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:504f2eb9:::benchmark_data_smithi173_74914_object80885:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:504f7689:::benchmark_data_smithi173_74914_object64303:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5050ba8f:::benchmark_data_smithi173_74914_object63207:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5050c07c:::benchmark_data_smithi173_74914_object82186:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50544a40:::benchmark_data_smithi173_74914_object90718:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5055679c:::benchmark_data_smithi173_74914_object53384:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:505a3787:::benchmark_data_smithi173_74914_object65615:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:505e472f:::benchmark_data_smithi173_74914_object83847:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:505ea163:::benchmark_data_smithi173_74914_object95065:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:505fa108:::benchmark_data_smithi173_74914_object80103:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50610b7b:::benchmark_data_smithi173_74914_object64871:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5061920c:::benchmark_data_smithi173_74914_object91092:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506272ee:::benchmark_data_smithi173_74914_object95243:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50642c57:::benchmark_data_smithi173_74914_object58709:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506762e0:::benchmark_data_smithi173_74914_object90947:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5067f5ce:::benchmark_data_smithi173_74914_object88694:head# 2024-08-30T23:39:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506a7309:::benchmark_data_smithi173_74914_object94515:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506b4781:::benchmark_data_smithi173_74914_object68638:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506c3cb7:::benchmark_data_smithi173_74914_object96146:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506d683f:::benchmark_data_smithi173_74914_object75340:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:506e20c1:::benchmark_data_smithi173_74914_object70481:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5070a173:::benchmark_data_smithi173_74914_object97109:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50725731:::benchmark_data_smithi173_74914_object92411:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:507290b3:::benchmark_data_smithi173_74914_object79204:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5077780e:::benchmark_data_smithi173_74914_object65225:head# 2024-08-30T23:39:51.279 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50779c0d:::benchmark_data_smithi173_74914_object54438:head# 2024-08-30T23:39:51.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:50 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:51.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:50 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:51.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:50 smithi106 ceph-mon[16603]: osdmap e455: 12 total, 10 up, 12 in 2024-08-30T23:39:51.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:50 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 35550/222120 objects degraded (16.005%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:51.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:50 smithi106 ceph-mon[16603]: osdmap e456: 12 total, 10 up, 12 in 2024-08-30T23:39:51.283 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5078054d:::benchmark_data_smithi173_74914_object86721:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:507bd69f:::benchmark_data_smithi173_74914_object98188:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:507dceb6:::benchmark_data_smithi173_74914_object65375:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5081c6b0:::benchmark_data_smithi173_74914_object77939:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:508260a3:::benchmark_data_smithi173_74914_object88837:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:508295f2:::benchmark_data_smithi173_74914_object69334:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5085f008:::benchmark_data_smithi173_74914_object57760:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50864bf2:::benchmark_data_smithi173_74914_object59355:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5087812d:::benchmark_data_smithi173_74914_object87345:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:508b6d6d:::benchmark_data_smithi173_74914_object60215:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:508bb2b8:::benchmark_data_smithi173_74914_object86110:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:508df272:::benchmark_data_smithi173_74914_object78905:head# 2024-08-30T23:39:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50945e1c:::benchmark_data_smithi173_74914_object62283:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5094fd24:::benchmark_data_smithi173_74914_object75658:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5095a5ad:::benchmark_data_smithi173_74914_object81779:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50964891:::benchmark_data_smithi173_74914_object63484:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50965dc5:::benchmark_data_smithi173_74914_object82658:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:509827fc:::benchmark_data_smithi173_74914_object68669:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:509c42ad:::benchmark_data_smithi173_74914_object63795:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:509c9025:::benchmark_data_smithi173_74914_object53981:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:509cd51c:::benchmark_data_smithi173_74914_object72264:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:509df41f:::benchmark_data_smithi173_74914_object57720:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:509f2217:::benchmark_data_smithi173_74914_object93589:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50a21eaf:::benchmark_data_smithi173_74914_object84797:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50a2cbb5:::benchmark_data_smithi173_74914_object68108:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50a398c4:::benchmark_data_smithi173_74914_object58646:head# 2024-08-30T23:39:51.285 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50a41be3:::benchmark_data_smithi173_74914_object88986:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50a4443f:::benchmark_data_smithi173_74914_object67106:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50a8d12a:::benchmark_data_smithi173_74914_object90670:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50aa190a:::benchmark_data_smithi173_74914_object68818:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50aa26cf:::benchmark_data_smithi173_74914_object87172:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ac25fc:::benchmark_data_smithi173_74914_object95176:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ac61e6:::benchmark_data_smithi173_74914_object74971:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ac771f:::benchmark_data_smithi173_74914_object97034:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ac7f9e:::benchmark_data_smithi173_74914_object58280:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b02680:::benchmark_data_smithi173_74914_object87428:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b044b9:::benchmark_data_smithi173_74914_object74763:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b281a3:::benchmark_data_smithi173_74914_object56376:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b54077:::benchmark_data_smithi173_74914_object75250:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b70fd2:::benchmark_data_smithi173_74914_object71564:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b7c50e:::benchmark_data_smithi173_74914_object74964:head# 2024-08-30T23:39:51.286 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b82744:::benchmark_data_smithi173_74914_object89770:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50b97896:::benchmark_data_smithi173_74914_object93483:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50bc2a77:::benchmark_data_smithi173_74914_object59201:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c1c5c5:::benchmark_data_smithi173_74914_object73019:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c1d8ab:::benchmark_data_smithi173_74914_object84879:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c31f00:::benchmark_data_smithi173_74914_object69235:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c44146:::benchmark_data_smithi173_74914_object59064:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c45e00:::benchmark_data_smithi173_74914_object94775:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c47044:::benchmark_data_smithi173_74914_object92103:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c4dc2e:::benchmark_data_smithi173_74914_object61866:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c7b61d:::benchmark_data_smithi173_74914_object63345:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c7c2e8:::benchmark_data_smithi173_74914_object53327:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c9523f:::benchmark_data_smithi173_74914_object83929:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50c9bd98:::benchmark_data_smithi173_74914_object91982:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50cab5a4:::benchmark_data_smithi173_74914_object60882:head# 2024-08-30T23:39:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50cc5be2:::benchmark_data_smithi173_74914_object90421:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50cee19b:::benchmark_data_smithi173_74914_object96283:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50cf0677:::benchmark_data_smithi173_74914_object53602:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d0af38:::benchmark_data_smithi173_74914_object79906:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d0b7f0:::benchmark_data_smithi173_74914_object56744:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d0e8ce:::benchmark_data_smithi173_74914_object53911:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d3370a:::benchmark_data_smithi173_74914_object69371:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d387a7:::benchmark_data_smithi173_74914_object72660:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d38b97:::benchmark_data_smithi173_74914_object78007:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d45e97:::benchmark_data_smithi173_74914_object63850:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d71f0d:::benchmark_data_smithi173_74914_object62507:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d86989:::benchmark_data_smithi173_74914_object65353:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d8c973:::benchmark_data_smithi173_74914_object97666:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50d8f581:::benchmark_data_smithi173_74914_object94626:head# 2024-08-30T23:39:51.288 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50dc3dba:::benchmark_data_smithi173_74914_object62630:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50dc5297:::benchmark_data_smithi173_74914_object68764:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50dcaaf6:::benchmark_data_smithi173_74914_object94662:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50dd08e4:::benchmark_data_smithi173_74914_object80388:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50dea12e:::benchmark_data_smithi173_74914_object91829:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50e01aea:::benchmark_data_smithi173_74914_object96031:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50e6770b:::benchmark_data_smithi173_74914_object71874:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50e7cb14:::benchmark_data_smithi173_74914_object93646:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50e915d6:::benchmark_data_smithi173_74914_object69154:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ea7b62:::benchmark_data_smithi173_74914_object97014:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ec2fb4:::benchmark_data_smithi173_74914_object70180:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ecde64:::benchmark_data_smithi173_74914_object66793:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ed41a1:::benchmark_data_smithi173_74914_object65467:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50eec46d:::benchmark_data_smithi173_74914_object92932:head# 2024-08-30T23:39:51.289 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50efda6f:::benchmark_data_smithi173_74914_object79047:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50f26e00:::benchmark_data_smithi173_74914_object59532:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50f521e2:::benchmark_data_smithi173_74914_object93819:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50f5a545:::benchmark_data_smithi173_74914_object88412:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50f5c430:::benchmark_data_smithi173_74914_object76889:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50f8fee8:::benchmark_data_smithi173_74914_object85818:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50fa8765:::benchmark_data_smithi173_74914_object87419:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50fbd12b:::benchmark_data_smithi173_74914_object87250:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50feedce:::benchmark_data_smithi173_74914_object73672:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:50ff0cac:::benchmark_data_smithi173_74914_object88469:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5100bf0e:::benchmark_data_smithi173_74914_object83756:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51011c45:::benchmark_data_smithi173_74914_object61579:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5101dd17:::benchmark_data_smithi173_74914_object79209:head# 2024-08-30T23:39:51.290 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510241c8:::benchmark_data_smithi173_74914_object53070:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51029551:::benchmark_data_smithi173_74914_object56285:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5102cf25:::benchmark_data_smithi173_74914_object95418:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5102d049:::benchmark_data_smithi173_74914_object80827:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51043265:::benchmark_data_smithi173_74914_object66073:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51052bf4:::benchmark_data_smithi173_74914_object65908:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51077d81:::benchmark_data_smithi173_74914_object55476:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510b796f:::benchmark_data_smithi173_74914_object75302:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510c61f9:::benchmark_data_smithi173_74914_object63623:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510cc862:::benchmark_data_smithi173_74914_object59053:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510d6f81:::benchmark_data_smithi173_74914_object74387:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510e42f9:::benchmark_data_smithi173_74914_object97805:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510e7972:::benchmark_data_smithi173_74914_object61937:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:510ee9c8:::benchmark_data_smithi173_74914_object90237:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511018cb:::benchmark_data_smithi173_74914_object60107:head# 2024-08-30T23:39:51.291 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5110baf1:::benchmark_data_smithi173_74914_object69376:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5112c115:::benchmark_data_smithi173_74914_object88320:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51139579:::benchmark_data_smithi173_74914_object95288:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51140266:::benchmark_data_smithi173_74914_object87924:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5116139a:::benchmark_data_smithi173_74914_object66869:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511937e5:::benchmark_data_smithi173_74914_object73080:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5119b5c3:::benchmark_data_smithi173_74914_object89809:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511a3095:::benchmark_data_smithi173_74914_object60328:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511a7601:::benchmark_data_smithi173_74914_object60443:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511a82c0:::benchmark_data_smithi173_74914_object91430:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511bdab0:::benchmark_data_smithi173_74914_object76040:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:511fdbbc:::benchmark_data_smithi173_74914_object88064:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5122dc09:::benchmark_data_smithi173_74914_object76997:head# 2024-08-30T23:39:51.292 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5124934d:::benchmark_data_smithi173_74914_object97549:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5124d10d:::benchmark_data_smithi173_74914_object73708:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:512580e1:::benchmark_data_smithi173_74914_object63397:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5125be2c:::benchmark_data_smithi173_74914_object66456:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51268a2d:::benchmark_data_smithi173_74914_object54048:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51283175:::benchmark_data_smithi173_74914_object81221:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:512cdf10:::benchmark_data_smithi173_74914_object84465:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:512fc1d0:::benchmark_data_smithi173_74914_object82262:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51315445:::benchmark_data_smithi173_74914_object61257:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51339e5f:::benchmark_data_smithi173_74914_object54852:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5133ce3e:::benchmark_data_smithi173_74914_object54971:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5135c3cd:::benchmark_data_smithi173_74914_object79080:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5135e4bb:::benchmark_data_smithi173_74914_object77715:head# 2024-08-30T23:39:51.293 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513679f5:::benchmark_data_smithi173_74914_object72542:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5138705e:::benchmark_data_smithi173_74914_object90642:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5138aa2b:::benchmark_data_smithi173_74914_object81102:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513a3eb6:::benchmark_data_smithi173_74914_object93403:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513a6b8a:::benchmark_data_smithi173_74914_object71308:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513b1d0e:::benchmark_data_smithi173_74914_object88086:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513b9015:::benchmark_data_smithi173_74914_object61652:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513d35d1:::benchmark_data_smithi173_74914_object70964:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:513dadb9:::benchmark_data_smithi173_74914_object81767:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51447ed4:::benchmark_data_smithi173_74914_object68941:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51499929:::benchmark_data_smithi173_74914_object92262:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:514a6bcc:::benchmark_data_smithi173_74914_object87989:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:514aff54:::benchmark_data_smithi173_74914_object67730:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:514bd5d7:::benchmark_data_smithi173_74914_object85248:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:515102bb:::benchmark_data_smithi173_74914_object57940:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5152b9b1:::benchmark_data_smithi173_74914_object67370:head# 2024-08-30T23:39:51.294 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5157ad9f:::benchmark_data_smithi173_74914_object97376:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51589a4a:::benchmark_data_smithi173_74914_object97168:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51597cba:::benchmark_data_smithi173_74914_object74376:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5159bf94:::benchmark_data_smithi173_74914_object63963:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:515b32ae:::benchmark_data_smithi173_74914_object60431:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:515c2bc8:::benchmark_data_smithi173_74914_object89537:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:515e5313:::benchmark_data_smithi173_74914_object81013:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:515f452a:::benchmark_data_smithi173_74914_object95238:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:516021f2:::benchmark_data_smithi173_74914_object98020:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5160f6b6:::benchmark_data_smithi173_74914_object90179:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:516113ad:::benchmark_data_smithi173_74914_object76466:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5165a7be:::benchmark_data_smithi173_74914_object86569:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5165f50a:::benchmark_data_smithi173_74914_object96075:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51664c0c:::benchmark_data_smithi173_74914_object66235:head# 2024-08-30T23:39:51.295 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51670c80:::benchmark_data_smithi173_74914_object79987:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5168e0a4:::benchmark_data_smithi173_74914_object82238:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:516a6442:::benchmark_data_smithi173_74914_object90399:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:516cb8b2:::benchmark_data_smithi173_74914_object95925:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:516ed3f4:::benchmark_data_smithi173_74914_object98173:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:516ff2b8:::benchmark_data_smithi173_74914_object97562:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51700cad:::benchmark_data_smithi173_74914_object86369:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51726edc:::benchmark_data_smithi173_74914_object63168:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:517480f9:::benchmark_data_smithi173_74914_object94368:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5174a520:::benchmark_data_smithi173_74914_object66143:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5174b8e7:::benchmark_data_smithi173_74914_object90080:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5175d5b8:::benchmark_data_smithi173_74914_object88681:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51776f63:::benchmark_data_smithi173_74914_object59191:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5178dd10:::benchmark_data_smithi173_74914_object94956:head# 2024-08-30T23:39:51.296 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5179c1a1:::benchmark_data_smithi173_74914_object79810:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:517a25a6:::benchmark_data_smithi173_74914_object68877:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:517b60d1:::benchmark_data_smithi173_74914_object62546:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:517cc245:::benchmark_data_smithi173_74914_object90247:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:517cefc0:::benchmark_data_smithi173_74914_object70637:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:517d9342:::benchmark_data_smithi173_74914_object56410:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51827196:::benchmark_data_smithi173_74914_object69954:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518624fe:::benchmark_data_smithi173_74914_object94441:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5186a63e:::benchmark_data_smithi173_74914_object96009:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518a2397:::benchmark_data_smithi173_74914_object62205:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518bf464:::benchmark_data_smithi173_74914_object55631:head# 2024-08-30T23:39:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518c0e9c:::benchmark_data_smithi173_74914_object86483:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518c3cdf:::benchmark_data_smithi173_74914_object69183:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518e045d:::benchmark_data_smithi173_74914_object61033:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518e5919:::benchmark_data_smithi173_74914_object91606:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518f34a2:::benchmark_data_smithi173_74914_object96625:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:518f7f63:::benchmark_data_smithi173_74914_object56334:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51907f29:::benchmark_data_smithi173_74914_object54981:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51909360:::benchmark_data_smithi173_74914_object93066:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5193c73a:::benchmark_data_smithi173_74914_object67709:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5193ce26:::benchmark_data_smithi173_74914_object79509:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51943f65:::benchmark_data_smithi173_74914_object95372:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:519444ad:::benchmark_data_smithi173_74914_object67551:head# 2024-08-30T23:39:51.298 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5196ae39:::benchmark_data_smithi173_74914_object96607:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51976844:::benchmark_data_smithi173_74914_object66243:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5198818d:::benchmark_data_smithi173_74914_object88201:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5198f126:::benchmark_data_smithi173_74914_object73078:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:519b4469:::benchmark_data_smithi173_74914_object95479:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:519cb7ff:::benchmark_data_smithi173_74914_object81883:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:519d43a7:::benchmark_data_smithi173_74914_object65858:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:519d5751:::benchmark_data_smithi173_74914_object96371:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:519ebf4f:::benchmark_data_smithi173_74914_object54177:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a29afb:::benchmark_data_smithi173_74914_object53776:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a37d27:::benchmark_data_smithi173_74914_object68692:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a4169a:::benchmark_data_smithi173_74914_object64590:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a4e5d5:::benchmark_data_smithi173_74914_object76521:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a544c7:::benchmark_data_smithi173_74914_object72618:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a54f48:::benchmark_data_smithi173_74914_object79120:head# 2024-08-30T23:39:51.299 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51a8c77c:::benchmark_data_smithi173_74914_object75546:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ac554d:::benchmark_data_smithi173_74914_object59113:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51b30158:::benchmark_data_smithi173_74914_object91810:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51b7e3e9:::benchmark_data_smithi173_74914_object87318:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51bab31d:::benchmark_data_smithi173_74914_object75056:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51bbf72f:::benchmark_data_smithi173_74914_object74645:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51bd4957:::benchmark_data_smithi173_74914_object71945:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c0200f:::benchmark_data_smithi173_74914_object65776:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c06b48:::benchmark_data_smithi173_74914_object55541:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c2f444:::benchmark_data_smithi173_74914_object82420:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c383a2:::benchmark_data_smithi173_74914_object53212:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c428b3:::benchmark_data_smithi173_74914_object54248:head# 2024-08-30T23:39:51.300 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c6c786:::benchmark_data_smithi173_74914_object59870:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51c8bf04:::benchmark_data_smithi173_74914_object71948:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ca1f37:::benchmark_data_smithi173_74914_object93548:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51cba72f:::benchmark_data_smithi173_74914_object71521:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51cda074:::benchmark_data_smithi173_74914_object58436:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ce62b8:::benchmark_data_smithi173_74914_object65541:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51d00907:::benchmark_data_smithi173_74914_object86799:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51d0a6f4:::benchmark_data_smithi173_74914_object66680:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51d249b5:::benchmark_data_smithi173_74914_object94014:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51d3222f:::benchmark_data_smithi173_74914_object57070:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51d6fc11:::benchmark_data_smithi173_74914_object97848:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51d74444:::benchmark_data_smithi173_74914_object76430:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51db00a6:::benchmark_data_smithi173_74914_object78168:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51db40c9:::benchmark_data_smithi173_74914_object81853:head# 2024-08-30T23:39:51.301 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51db86bf:::benchmark_data_smithi173_74914_object71101:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e00a97:::benchmark_data_smithi173_74914_object72350:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e200dc:::benchmark_data_smithi173_74914_object71829:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e51347:::benchmark_data_smithi173_74914_object88346:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e5cb7f:::benchmark_data_smithi173_74914_object53909:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e6dfdc:::benchmark_data_smithi173_74914_object91931:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e79e42:::benchmark_data_smithi173_74914_object57054:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e82aa0:::benchmark_data_smithi173_74914_object97821:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51e9bf38:::benchmark_data_smithi173_74914_object94990:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ea00ca:::benchmark_data_smithi173_74914_object89395:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ea0de3:::benchmark_data_smithi173_74914_object70698:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ea8c56:::benchmark_data_smithi173_74914_object72945:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51eb2681:::benchmark_data_smithi173_74914_object92666:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ec82ba:::benchmark_data_smithi173_74914_object70717:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ecda2e:::benchmark_data_smithi173_74914_object56756:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51ed2a24:::benchmark_data_smithi173_74914_object81520:head# 2024-08-30T23:39:51.302 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f0adc4:::benchmark_data_smithi173_74914_object53829:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f227d4:::benchmark_data_smithi173_74914_object71322:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f5ab85:::benchmark_data_smithi173_74914_object77249:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f78b58:::benchmark_data_smithi173_74914_object83409:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f7cb0b:::benchmark_data_smithi173_74914_object93832:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f7fc41:::benchmark_data_smithi173_74914_object60891:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51f8c15b:::benchmark_data_smithi173_74914_object96608:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51fa594e:::benchmark_data_smithi173_74914_object75568:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51fa920f:::benchmark_data_smithi173_74914_object63337:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51fb0b05:::benchmark_data_smithi173_74914_object61496:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51fc9aaf:::benchmark_data_smithi173_74914_object94597:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51fce2c5:::benchmark_data_smithi173_74914_object89238:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:51fd4755:::benchmark_data_smithi173_74914_object55221:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52021b2f:::benchmark_data_smithi173_74914_object79902:head# 2024-08-30T23:39:51.303 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5203b962:::benchmark_data_smithi173_74914_object72168:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:520420bc:::benchmark_data_smithi173_74914_object90635:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5204474b:::benchmark_data_smithi173_74914_object73425:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5204c4b2:::benchmark_data_smithi173_74914_object85583:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52083000:::benchmark_data_smithi173_74914_object86948:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:520a18aa:::benchmark_data_smithi173_74914_object76315:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:520bd1d7:::benchmark_data_smithi173_74914_object55986:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:520ccf24:::benchmark_data_smithi173_74914_object68932:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5212691f:::benchmark_data_smithi173_74914_object78126:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5212901f:::benchmark_data_smithi173_74914_object54347:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:521358c8:::benchmark_data_smithi173_74914_object81732:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5214e42d:::benchmark_data_smithi173_74914_object79493:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5217d0cb:::benchmark_data_smithi173_74914_object61297:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5217ef40:::benchmark_data_smithi173_74914_object87186:head# 2024-08-30T23:39:51.304 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5218fcf6:::benchmark_data_smithi173_74914_object72974:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52199e05:::benchmark_data_smithi173_74914_object83149:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5219f485:::benchmark_data_smithi173_74914_object86452:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:521c46cb:::benchmark_data_smithi173_74914_object62725:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:521c5f84:::benchmark_data_smithi173_74914_object77214:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5221b3c1:::benchmark_data_smithi173_74914_object60205:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5221cfba:::benchmark_data_smithi173_74914_object64325:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52224f3e:::benchmark_data_smithi173_74914_object71927:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:522570ab:::benchmark_data_smithi173_74914_object84938:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5225a34e:::benchmark_data_smithi173_74914_object83982:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52264229:::benchmark_data_smithi173_74914_object87267:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5226989a:::benchmark_data_smithi173_74914_object87098:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5226f701:::benchmark_data_smithi173_74914_object64092:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5227b519:::benchmark_data_smithi173_74914_object57284:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:522e4531:::benchmark_data_smithi173_74914_object68045:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:523086ab:::benchmark_data_smithi173_74914_object69674:head# 2024-08-30T23:39:51.305 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52311518:::benchmark_data_smithi173_74914_object93946:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5232249f:::benchmark_data_smithi173_74914_object92945:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5232fa83:::benchmark_data_smithi173_74914_object85948:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52352544:::benchmark_data_smithi173_74914_object58086:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:523695ca:::benchmark_data_smithi173_74914_object84040:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52370f15:::benchmark_data_smithi173_74914_object54032:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5237e00b:::benchmark_data_smithi173_74914_object91842:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5239556d:::benchmark_data_smithi173_74914_object60279:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:523bcd06:::benchmark_data_smithi173_74914_object76982:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:523dafb4:::benchmark_data_smithi173_74914_object88935:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:523efcf3:::benchmark_data_smithi173_74914_object53662:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:523ff00e:::benchmark_data_smithi173_74914_object61519:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52420363:::benchmark_data_smithi173_74914_object77929:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52455e0c:::benchmark_data_smithi173_74914_object71576:head# 2024-08-30T23:39:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5248c698:::benchmark_data_smithi173_74914_object64636:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5249507f:::benchmark_data_smithi173_74914_object59242:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:524b21e3:::benchmark_data_smithi173_74914_object80986:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:524c3811:::benchmark_data_smithi173_74914_object57646:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:524ce85b:::benchmark_data_smithi173_74914_object86944:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52570cf9:::benchmark_data_smithi173_74914_object68762:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5259b3c8:::benchmark_data_smithi173_74914_object66730:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525b3288:::benchmark_data_smithi173_74914_object53995:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525b6cad:::benchmark_data_smithi173_74914_object55688:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525bc18d:::benchmark_data_smithi173_74914_object67545:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525c34fc:::benchmark_data_smithi173_74914_object66417:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525c91d9:::benchmark_data_smithi173_74914_object76648:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525fb08c:::benchmark_data_smithi173_74914_object88088:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:525fd75f:::benchmark_data_smithi173_74914_object57988:head# 2024-08-30T23:39:51.307 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5261eee6:::benchmark_data_smithi173_74914_object59009:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52670186:::benchmark_data_smithi173_74914_object60101:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5267a7ad:::benchmark_data_smithi173_74914_object93250:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5268235e:::benchmark_data_smithi173_74914_object58678:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:526d9640:::benchmark_data_smithi173_74914_object54300:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:526ffc3b:::benchmark_data_smithi173_74914_object63374:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527088ca:::benchmark_data_smithi173_74914_object77979:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52710dc9:::benchmark_data_smithi173_74914_object85023:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5273d5bd:::benchmark_data_smithi173_74914_object88272:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52740581:::benchmark_data_smithi173_74914_object84183:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5274ff57:::benchmark_data_smithi173_74914_object75429:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52752fd8:::benchmark_data_smithi173_74914_object81200:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5276afce:::benchmark_data_smithi173_74914_object56877:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5277aa6e:::benchmark_data_smithi173_74914_object84956:head# 2024-08-30T23:39:51.308 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5279007a:::benchmark_data_smithi173_74914_object94902:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5279d048:::benchmark_data_smithi173_74914_object82839:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527afcee:::benchmark_data_smithi173_74914_object92630:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527b77b8:::benchmark_data_smithi173_74914_object85504:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527bb291:::benchmark_data_smithi173_74914_object91776:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527bfd88:::benchmark_data_smithi173_74914_object78620:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527e2137:::benchmark_data_smithi173_74914_object70665:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527edd07:::benchmark_data_smithi173_74914_object64494:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:527f68d5:::benchmark_data_smithi173_74914_object58091:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52818bc2:::benchmark_data_smithi173_74914_object62316:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5281c0b8:::benchmark_data_smithi173_74914_object68278:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:528229bd:::benchmark_data_smithi173_74914_object89259:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:528264f9:::benchmark_data_smithi173_74914_object86905:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52837efc:::benchmark_data_smithi173_74914_object75561:head# 2024-08-30T23:39:51.309 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5283f11c:::benchmark_data_smithi173_74914_object97448:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52844ade:::benchmark_data_smithi173_74914_object53323:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5285c984:::benchmark_data_smithi173_74914_object84735:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52873355:::benchmark_data_smithi173_74914_object72572:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52882cbc:::benchmark_data_smithi173_74914_object73315:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5288453a:::benchmark_data_smithi173_74914_object85989:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:528968f8:::benchmark_data_smithi173_74914_object66752:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:528de0ba:::benchmark_data_smithi173_74914_object65396:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:528e1a75:::benchmark_data_smithi173_74914_object56723:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:528f3e29:::benchmark_data_smithi173_74914_object77759:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52915e06:::benchmark_data_smithi173_74914_object85058:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52918c97:::benchmark_data_smithi173_74914_object89084:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52936d21:::benchmark_data_smithi173_74914_object66282:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529498a6:::benchmark_data_smithi173_74914_object70880:head# 2024-08-30T23:39:51.310 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5295622e:::benchmark_data_smithi173_74914_object79648:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52963d5b:::benchmark_data_smithi173_74914_object57285:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5298e62d:::benchmark_data_smithi173_74914_object79286:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5299d58b:::benchmark_data_smithi173_74914_object89501:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529a2678:::benchmark_data_smithi173_74914_object58042:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529c4ced:::benchmark_data_smithi173_74914_object57334:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529caf4a:::benchmark_data_smithi173_74914_object53860:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529dd2d0:::benchmark_data_smithi173_74914_object91632:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529e7aab:::benchmark_data_smithi173_74914_object80340:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:529ff913:::benchmark_data_smithi173_74914_object85873:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a09875:::benchmark_data_smithi173_74914_object58163:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a3f8b0:::benchmark_data_smithi173_74914_object81602:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a5bb1a:::benchmark_data_smithi173_74914_object95843:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a79877:::benchmark_data_smithi173_74914_object91535:head# 2024-08-30T23:39:51.311 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a8cfe6:::benchmark_data_smithi173_74914_object91827:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a908a0:::benchmark_data_smithi173_74914_object82018:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52a9fb00:::benchmark_data_smithi173_74914_object86177:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52acc5e7:::benchmark_data_smithi173_74914_object88557:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52ae0434:::benchmark_data_smithi173_74914_object63171:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52af0e27:::benchmark_data_smithi173_74914_object86359:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52b60410:::benchmark_data_smithi173_74914_object91136:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52b95f2f:::benchmark_data_smithi173_74914_object92403:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52b9e6a4:::benchmark_data_smithi173_74914_object97615:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52bb97c4:::benchmark_data_smithi173_74914_object97868:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52bd6891:::benchmark_data_smithi173_74914_object57986:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52c04029:::benchmark_data_smithi173_74914_object59341:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52c0c818:::benchmark_data_smithi173_74914_object88317:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52c3b3ed:::benchmark_data_smithi173_74914_object83541:head# 2024-08-30T23:39:51.312 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52c60d8c:::benchmark_data_smithi173_74914_object57503:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52c62a56:::benchmark_data_smithi173_74914_object54682:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52c7b68e:::benchmark_data_smithi173_74914_object82448:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52cb1a63:::benchmark_data_smithi173_74914_object66126:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52cbdf95:::benchmark_data_smithi173_74914_object88342:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d0a885:::benchmark_data_smithi173_74914_object66599:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d42df1:::benchmark_data_smithi173_74914_object79335:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d49f49:::benchmark_data_smithi173_74914_object79247:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d5993b:::benchmark_data_smithi173_74914_object88737:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d5b9d8:::benchmark_data_smithi173_74914_object92158:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d7228e:::benchmark_data_smithi173_74914_object53523:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52d78665:::benchmark_data_smithi173_74914_object63729:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52db6c67:::benchmark_data_smithi173_74914_object75673:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52dbe37e:::benchmark_data_smithi173_74914_object75094:head# 2024-08-30T23:39:51.313 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52dc2367:::benchmark_data_smithi173_74914_object68836:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52dc8738:::benchmark_data_smithi173_74914_object87766:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52df2cd8:::benchmark_data_smithi173_74914_object71735:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52dfb7df:::benchmark_data_smithi173_74914_object57183:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52dfbc68:::benchmark_data_smithi173_74914_object69817:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52e3b41e:::benchmark_data_smithi173_74914_object80004:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52e41bae:::benchmark_data_smithi173_74914_object60897:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52e61803:::benchmark_data_smithi173_74914_object73275:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52e932de:::benchmark_data_smithi173_74914_object79045:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52e953fe:::benchmark_data_smithi173_74914_object87017:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52eb5334:::benchmark_data_smithi173_74914_object66919:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52ebd8e0:::benchmark_data_smithi173_74914_object56304:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52ee5d34:::benchmark_data_smithi173_74914_object55085:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52f3e882:::benchmark_data_smithi173_74914_object59431:head# 2024-08-30T23:39:51.314 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52f67d9d:::benchmark_data_smithi173_74914_object70875:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52f6a395:::benchmark_data_smithi173_74914_object83614:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52f7966b:::benchmark_data_smithi173_74914_object66296:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52f836a1:::benchmark_data_smithi173_74914_object86886:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52f962f9:::benchmark_data_smithi173_74914_object53057:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52fa372d:::benchmark_data_smithi173_74914_object54580:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52fe85ba:::benchmark_data_smithi173_74914_object94221:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:52fee71c:::benchmark_data_smithi173_74914_object68458:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5300a58e:::benchmark_data_smithi173_74914_object84303:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5302453e:::benchmark_data_smithi173_74914_object74498:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5302f1e6:::benchmark_data_smithi173_74914_object75153:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53043a2b:::benchmark_data_smithi173_74914_object85538:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53071c99:::benchmark_data_smithi173_74914_object73738:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530ae62f:::benchmark_data_smithi173_74914_object58244:head# 2024-08-30T23:39:51.315 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530afee8:::benchmark_data_smithi173_74914_object53763:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530bb102:::benchmark_data_smithi173_74914_object74978:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530bef9d:::benchmark_data_smithi173_74914_object81415:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530e3dc1:::benchmark_data_smithi173_74914_object77196:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530e4c8d:::benchmark_data_smithi173_74914_object62512:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:530ec32f:::benchmark_data_smithi173_74914_object69008:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5310c27f:::benchmark_data_smithi173_74914_object68071:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53113fba:::benchmark_data_smithi173_74914_object55326:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:531146af:::benchmark_data_smithi173_74914_object62943:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53115099:::benchmark_data_smithi173_74914_object78812:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53123c04:::benchmark_data_smithi173_74914_object54105:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:531255fd:::benchmark_data_smithi173_74914_object78194:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53153cfd:::benchmark_data_smithi173_74914_object59034:head# 2024-08-30T23:39:51.316 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:531646a7:::benchmark_data_smithi173_74914_object85918:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:531696e7:::benchmark_data_smithi173_74914_object79349:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:531d7d4f:::benchmark_data_smithi173_74914_object89650:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532093bd:::benchmark_data_smithi173_74914_object95335:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5320ea9c:::benchmark_data_smithi173_74914_object80158:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532192e7:::benchmark_data_smithi173_74914_object65002:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5322b5b8:::benchmark_data_smithi173_74914_object82320:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53234a62:::benchmark_data_smithi173_74914_object87347:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53265b51:::benchmark_data_smithi173_74914_object83251:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53273211:::benchmark_data_smithi173_74914_object56238:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5327671a:::benchmark_data_smithi173_74914_object63381:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532807e2:::benchmark_data_smithi173_74914_object92161:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53282c7a:::benchmark_data_smithi173_74914_object74946:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532e5b10:::benchmark_data_smithi173_74914_object88891:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532ede51:::benchmark_data_smithi173_74914_object65465:head# 2024-08-30T23:39:51.317 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532ee0b2:::benchmark_data_smithi173_74914_object61798:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:532f71f4:::benchmark_data_smithi173_74914_object62705:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5331c3e5:::benchmark_data_smithi173_74914_object88045:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5333f79a:::benchmark_data_smithi173_74914_object70664:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5335c257:::benchmark_data_smithi173_74914_object86080:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:533676f4:::benchmark_data_smithi173_74914_object59513:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5338641f:::benchmark_data_smithi173_74914_object66034:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53387568:::benchmark_data_smithi173_74914_object89636:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5339ffdb:::benchmark_data_smithi173_74914_object83751:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:533b4cdb:::benchmark_data_smithi173_74914_object92468:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:533e3d4c:::benchmark_data_smithi173_74914_object84931:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53432378:::benchmark_data_smithi173_74914_object73979:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53446b2c:::benchmark_data_smithi173_74914_object68786:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53459a89:::benchmark_data_smithi173_74914_object66578:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:534642c3:::benchmark_data_smithi173_74914_object77369:head# 2024-08-30T23:39:51.318 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:534879bf:::benchmark_data_smithi173_74914_object90531:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:534b7412:::benchmark_data_smithi173_74914_object81010:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:534c1ef8:::benchmark_data_smithi173_74914_object75090:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:534d2342:::benchmark_data_smithi173_74914_object94986:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:534fee98:::benchmark_data_smithi173_74914_object64616:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5350ec7a:::benchmark_data_smithi173_74914_object80503:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5351c284:::benchmark_data_smithi173_74914_object74734:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53526ae4:::benchmark_data_smithi173_74914_object61607:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5353b4b5:::benchmark_data_smithi173_74914_object61467:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5354fe9a:::benchmark_data_smithi173_74914_object56020:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53571da7:::benchmark_data_smithi173_74914_object52800:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5357f02e:::benchmark_data_smithi173_74914_object92869:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535847e0:::benchmark_data_smithi173_74914_object54813:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53591ddc:::benchmark_data_smithi173_74914_object63330:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53592439:::benchmark_data_smithi173_74914_object56711:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5359b159:::benchmark_data_smithi173_74914_object68812:head# 2024-08-30T23:39:51.319 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535ac720:::benchmark_data_smithi173_74914_object70490:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535b04a5:::benchmark_data_smithi173_74914_object58872:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535be0bd:::benchmark_data_smithi173_74914_object58207:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535bf0e4:::benchmark_data_smithi173_74914_object93186:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535c689a:::benchmark_data_smithi173_74914_object93396:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535e24bd:::benchmark_data_smithi173_74914_object88836:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535e78b7:::benchmark_data_smithi173_74914_object56002:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:535f1ebd:::benchmark_data_smithi173_74914_object80331:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53619e24:::benchmark_data_smithi173_74914_object76452:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:536259b4:::benchmark_data_smithi173_74914_object58829:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5365032f:::benchmark_data_smithi173_74914_object60536:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53658538:::benchmark_data_smithi173_74914_object80667:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:536626be:::benchmark_data_smithi173_74914_object86677:head# 2024-08-30T23:39:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53671db2:::benchmark_data_smithi173_74914_object74012:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5367a9fa:::benchmark_data_smithi173_74914_object71878:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5367d0ae:::benchmark_data_smithi173_74914_object88651:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5369fb7e:::benchmark_data_smithi173_74914_object78134:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:536d0397:::benchmark_data_smithi173_74914_object67097:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:536dc556:::benchmark_data_smithi173_74914_object59893:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:536e94f8:::benchmark_data_smithi173_74914_object93386:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5370c30b:::benchmark_data_smithi173_74914_object70365:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5372d437:::benchmark_data_smithi173_74914_object70232:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5372e374:::benchmark_data_smithi173_74914_object88962:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53743d9a:::benchmark_data_smithi173_74914_object78467:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5375500d:::benchmark_data_smithi173_74914_object85012:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:537661a8:::benchmark_data_smithi173_74914_object55225:head# 2024-08-30T23:39:51.321 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:537e436e:::benchmark_data_smithi173_74914_object71266:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:537fd325:::benchmark_data_smithi173_74914_object68126:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5380080f:::benchmark_data_smithi173_74914_object91261:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5380d892:::benchmark_data_smithi173_74914_object90493:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5382f165:::benchmark_data_smithi173_74914_object84073:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53880416:::benchmark_data_smithi173_74914_object55595:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:538846f2:::benchmark_data_smithi173_74914_object56420:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53899361:::benchmark_data_smithi173_74914_object84547:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:538a494f:::benchmark_data_smithi173_74914_object54169:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:538b24b5:::benchmark_data_smithi173_74914_object80490:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:538e8eac:::benchmark_data_smithi173_74914_object55810:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53901b76:::benchmark_data_smithi173_74914_object93942:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:539105f7:::benchmark_data_smithi173_74914_object78998:head# 2024-08-30T23:39:51.322 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53914b4c:::benchmark_data_smithi173_74914_object76974:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5391c3bd:::benchmark_data_smithi173_74914_object64052:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5392326b:::benchmark_data_smithi173_74914_object55446:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5395cc31:::benchmark_data_smithi173_74914_object98113:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5398f530:::benchmark_data_smithi173_74914_object85366:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:539c95c5:::benchmark_data_smithi173_74914_object55529:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:539cb659:::benchmark_data_smithi173_74914_object90392:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:539df1a3:::benchmark_data_smithi173_74914_object72088:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53a08c0d:::benchmark_data_smithi173_74914_object55511:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53a3ff4e:::benchmark_data_smithi173_74914_object88160:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53a9badb:::benchmark_data_smithi173_74914_object55817:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ac4d24:::benchmark_data_smithi173_74914_object79264:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53adc18d:::benchmark_data_smithi173_74914_object61965:head# 2024-08-30T23:39:51.323 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ae24e0:::benchmark_data_smithi173_74914_object89521:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53b2f4b2:::benchmark_data_smithi173_74914_object89372:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53b3a935:::benchmark_data_smithi173_74914_object68984:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53b6c561:::benchmark_data_smithi173_74914_object84527:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53b87edb:::benchmark_data_smithi173_74914_object63284:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53bfa15f:::benchmark_data_smithi173_74914_object72699:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c25352:::benchmark_data_smithi173_74914_object81763:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c38152:::benchmark_data_smithi173_74914_object94489:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c4a702:::benchmark_data_smithi173_74914_object81220:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c66e07:::benchmark_data_smithi173_74914_object67741:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c7cd89:::benchmark_data_smithi173_74914_object53185:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c85877:::benchmark_data_smithi173_74914_object87156:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53c9a7b9:::benchmark_data_smithi173_74914_object67089:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ca486d:::benchmark_data_smithi173_74914_object85797:head# 2024-08-30T23:39:51.324 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ca8bf5:::benchmark_data_smithi173_74914_object61466:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53cc863f:::benchmark_data_smithi173_74914_object59503:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53cf76db:::benchmark_data_smithi173_74914_object85536:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53cffc10:::benchmark_data_smithi173_74914_object92772:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53d141ba:::benchmark_data_smithi173_74914_object69456:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53d38dac:::benchmark_data_smithi173_74914_object92817:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53d44e35:::benchmark_data_smithi173_74914_object81782:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53d4a2cb:::benchmark_data_smithi173_74914_object56625:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53d7b6bf:::benchmark_data_smithi173_74914_object97941:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53d980b6:::benchmark_data_smithi173_74914_object90748:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53daa268:::benchmark_data_smithi173_74914_object79615:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53de3b5b:::benchmark_data_smithi173_74914_object97231:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53de89a9:::benchmark_data_smithi173_74914_object97978:head# 2024-08-30T23:39:51.325 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53df6343:::benchmark_data_smithi173_74914_object81029:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53e0c92c:::benchmark_data_smithi173_74914_object67580:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53e1b37c:::benchmark_data_smithi173_74914_object88846:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53e2bddf:::benchmark_data_smithi173_74914_object85635:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53e3ab1a:::benchmark_data_smithi173_74914_object76280:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53e5fa06:::benchmark_data_smithi173_74914_object74815:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ea6f18:::benchmark_data_smithi173_74914_object93979:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ec248c:::benchmark_data_smithi173_74914_object54011:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ed68e2:::benchmark_data_smithi173_74914_object56853:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ee36ca:::benchmark_data_smithi173_74914_object80966:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53eee3df:::benchmark_data_smithi173_74914_object77665:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f05796:::benchmark_data_smithi173_74914_object55672:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f08892:::benchmark_data_smithi173_74914_object56748:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f0cb46:::benchmark_data_smithi173_74914_object76361:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f0d2bf:::benchmark_data_smithi173_74914_object82641:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f1df36:::benchmark_data_smithi173_74914_object97590:head# 2024-08-30T23:39:51.326 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f202eb:::benchmark_data_smithi173_74914_object65857:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f24c41:::benchmark_data_smithi173_74914_object59006:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f24ff4:::benchmark_data_smithi173_74914_object70517:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f3b283:::benchmark_data_smithi173_74914_object92957:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f40286:::benchmark_data_smithi173_74914_object91257:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f5af1f:::benchmark_data_smithi173_74914_object72743:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53f95bc6:::benchmark_data_smithi173_74914_object98000:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fa366c:::benchmark_data_smithi173_74914_object95270:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fac52d:::benchmark_data_smithi173_74914_object97388:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fcda7b:::benchmark_data_smithi173_74914_object89220:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fde5e2:::benchmark_data_smithi173_74914_object93440:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fe24b8:::benchmark_data_smithi173_74914_object62863:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fe9795:::benchmark_data_smithi173_74914_object53445:head# 2024-08-30T23:39:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53fed48c:::benchmark_data_smithi173_74914_object75027:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ff250b:::benchmark_data_smithi173_74914_object79172:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:53ff323a:::benchmark_data_smithi173_74914_object64557:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54058b29:::benchmark_data_smithi173_74914_object81054:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54097b65:::benchmark_data_smithi173_74914_object65275:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:540998fd:::benchmark_data_smithi173_74914_object86824:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:540acd0c:::benchmark_data_smithi173_74914_object97292:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:540b01dc:::benchmark_data_smithi173_74914_object92899:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:540c66ef:::benchmark_data_smithi173_74914_object96488:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:540db8c2:::benchmark_data_smithi173_74914_object67014:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54108a1c:::benchmark_data_smithi173_74914_object88774:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54110482:::benchmark_data_smithi173_74914_object82224:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5413e84c:::benchmark_data_smithi173_74914_object68480:head# 2024-08-30T23:39:51.328 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54151004:::benchmark_data_smithi173_74914_object63771:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:541554b0:::benchmark_data_smithi173_74914_object63724:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5417b869:::benchmark_data_smithi173_74914_object72929:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:541b3f67:::benchmark_data_smithi173_74914_object74561:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5420e56f:::benchmark_data_smithi173_74914_object67259:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5420ed9a:::benchmark_data_smithi173_74914_object94339:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542282a7:::benchmark_data_smithi173_74914_object57352:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54266be2:::benchmark_data_smithi173_74914_object82653:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5426b7a4:::benchmark_data_smithi173_74914_object77619:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542a54c7:::benchmark_data_smithi173_74914_object83371:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542a7ce2:::benchmark_data_smithi173_74914_object81216:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542cfa8e:::benchmark_data_smithi173_74914_object69323:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542cfc12:::benchmark_data_smithi173_74914_object60902:head# 2024-08-30T23:39:51.329 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542d091f:::benchmark_data_smithi173_74914_object76657:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:542ec078:::benchmark_data_smithi173_74914_object82052:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54306698:::benchmark_data_smithi173_74914_object80649:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:543211a0:::benchmark_data_smithi173_74914_object83190:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5433999e:::benchmark_data_smithi173_74914_object92235:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54379561:::benchmark_data_smithi173_74914_object85951:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5437adeb:::benchmark_data_smithi173_74914_object59723:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:543899b3:::benchmark_data_smithi173_74914_object87023:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:543a2961:::benchmark_data_smithi173_74914_object52987:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:543a6c17:::benchmark_data_smithi173_74914_object63167:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:543c2a27:::benchmark_data_smithi173_74914_object84366:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:543f46d6:::benchmark_data_smithi173_74914_object76391:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5440fd3b:::benchmark_data_smithi173_74914_object75303:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:544122c0:::benchmark_data_smithi173_74914_object82152:head# 2024-08-30T23:39:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54414ca8:::benchmark_data_smithi173_74914_object75551:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5446622b:::benchmark_data_smithi173_74914_object55462:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54486253:::benchmark_data_smithi173_74914_object79001:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54486412:::benchmark_data_smithi173_74914_object54315:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:544ccd1e:::benchmark_data_smithi173_74914_object76811:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:544d5b55:::benchmark_data_smithi173_74914_object96257:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:545689d4:::benchmark_data_smithi173_74914_object84468:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:545799f2:::benchmark_data_smithi173_74914_object73929:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54594904:::benchmark_data_smithi173_74914_object67263:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:545951fb:::benchmark_data_smithi173_74914_object54217:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54599b51:::benchmark_data_smithi173_74914_object73250:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5459d4cf:::benchmark_data_smithi173_74914_object66434:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54604fc0:::benchmark_data_smithi173_74914_object58333:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54607243:::benchmark_data_smithi173_74914_object60609:head# 2024-08-30T23:39:51.331 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5461a93f:::benchmark_data_smithi173_74914_object82863:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5462f6e3:::benchmark_data_smithi173_74914_object58913:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5463bdfa:::benchmark_data_smithi173_74914_object68683:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:546607ff:::benchmark_data_smithi173_74914_object82284:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54674b15:::benchmark_data_smithi173_74914_object72698:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5467d0bd:::benchmark_data_smithi173_74914_object70621:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5467ea61:::benchmark_data_smithi173_74914_object80671:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54689e81:::benchmark_data_smithi173_74914_object73720:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:546a43c3:::benchmark_data_smithi173_74914_object73100:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:546a9855:::benchmark_data_smithi173_74914_object64070:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:546e1571:::benchmark_data_smithi173_74914_object56617:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5473139a:::benchmark_data_smithi173_74914_object86209:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:547346a6:::benchmark_data_smithi173_74914_object61800:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5473e620:::benchmark_data_smithi173_74914_object80872:head# 2024-08-30T23:39:51.332 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54744157:::benchmark_data_smithi173_74914_object64238:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5476680c:::benchmark_data_smithi173_74914_object96793:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:547886cc:::benchmark_data_smithi173_74914_object97424:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54793b85:::benchmark_data_smithi173_74914_object94171:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:547d1f59:::benchmark_data_smithi173_74914_object80532:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:547ec065:::benchmark_data_smithi173_74914_object93769:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548669ba:::benchmark_data_smithi173_74914_object55126:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54871372:::benchmark_data_smithi173_74914_object72087:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548741f8:::benchmark_data_smithi173_74914_object82438:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5487c743:::benchmark_data_smithi173_74914_object87736:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5487e994:::benchmark_data_smithi173_74914_object87833:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5488e5f1:::benchmark_data_smithi173_74914_object97272:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5489dfff:::benchmark_data_smithi173_74914_object69503:head# 2024-08-30T23:39:51.333 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5489f676:::benchmark_data_smithi173_74914_object71340:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548a80b8:::benchmark_data_smithi173_74914_object83740:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548b62a0:::benchmark_data_smithi173_74914_object70569:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548c4c33:::benchmark_data_smithi173_74914_object66347:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548d3675:::benchmark_data_smithi173_74914_object79140:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:548e869c:::benchmark_data_smithi173_74914_object71868:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54913427:::benchmark_data_smithi173_74914_object79170:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5492df68:::benchmark_data_smithi173_74914_object74346:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549449ad:::benchmark_data_smithi173_74914_object55668:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5495400d:::benchmark_data_smithi173_74914_object90358:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5495ba83:::benchmark_data_smithi173_74914_object77684:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5498e9ed:::benchmark_data_smithi173_74914_object64849:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5499bece:::benchmark_data_smithi173_74914_object97178:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549b1824:::benchmark_data_smithi173_74914_object93397:head# 2024-08-30T23:39:51.334 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549bc598:::benchmark_data_smithi173_74914_object72815:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549c63b7:::benchmark_data_smithi173_74914_object66246:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549c7ea8:::benchmark_data_smithi173_74914_object91609:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549cd0af:::benchmark_data_smithi173_74914_object54583:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549d14c0:::benchmark_data_smithi173_74914_object77520:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:549ed284:::benchmark_data_smithi173_74914_object82820:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54a153f7:::benchmark_data_smithi173_74914_object66151:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54a5fa40:::benchmark_data_smithi173_74914_object90002:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54a63a92:::benchmark_data_smithi173_74914_object58385:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54a79056:::benchmark_data_smithi173_74914_object66510:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54a825ed:::benchmark_data_smithi173_74914_object92439:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54a83025:::benchmark_data_smithi173_74914_object53708:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54acb5b1:::benchmark_data_smithi173_74914_object87980:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54b11a5c:::benchmark_data_smithi173_74914_object94969:head# 2024-08-30T23:39:51.335 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54b11e51:::benchmark_data_smithi173_74914_object90575:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54b27a4b:::benchmark_data_smithi173_74914_object77769:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54b41c36:::benchmark_data_smithi173_74914_object70359:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54b5c2a9:::benchmark_data_smithi173_74914_object61131:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54b64873:::benchmark_data_smithi173_74914_object66678:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ba10b1:::benchmark_data_smithi173_74914_object80223:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ba8b7f:::benchmark_data_smithi173_74914_object80343:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54bad9c6:::benchmark_data_smithi173_74914_object92407:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54bb6efe:::benchmark_data_smithi173_74914_object93482:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54bb905e:::benchmark_data_smithi173_74914_object65666:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54bbc2c2:::benchmark_data_smithi173_74914_object52845:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54bc20b4:::benchmark_data_smithi173_74914_object85207:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54be0d55:::benchmark_data_smithi173_74914_object95992:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54be67d6:::benchmark_data_smithi173_74914_object94538:head# 2024-08-30T23:39:51.336 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54bf86d2:::benchmark_data_smithi173_74914_object60539:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54c3769f:::benchmark_data_smithi173_74914_object92735:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54c427db:::benchmark_data_smithi173_74914_object74347:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54c59955:::benchmark_data_smithi173_74914_object68743:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54c8ec23:::benchmark_data_smithi173_74914_object73835:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54cb25e2:::benchmark_data_smithi173_74914_object68712:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54cca437:::benchmark_data_smithi173_74914_object56322:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54cfcde8:::benchmark_data_smithi173_74914_object79258:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54cfd1a5:::benchmark_data_smithi173_74914_object77446:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d0ba25:::benchmark_data_smithi173_74914_object76634:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d3457f:::benchmark_data_smithi173_74914_object89324:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d42d38:::benchmark_data_smithi173_74914_object76961:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d621bc:::benchmark_data_smithi173_74914_object63883:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d81d0a:::benchmark_data_smithi173_74914_object65401:head# 2024-08-30T23:39:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d8be8e:::benchmark_data_smithi173_74914_object69344:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54d9af5e:::benchmark_data_smithi173_74914_object70230:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54dec6ed:::benchmark_data_smithi173_74914_object79440:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54e2a0fe:::benchmark_data_smithi173_74914_object97106:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54e3d8ab:::benchmark_data_smithi173_74914_object59078:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54e45766:::benchmark_data_smithi173_74914_object92446:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54e58c27:::benchmark_data_smithi173_74914_object79453:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ea2582:::benchmark_data_smithi173_74914_object65599:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ea997d:::benchmark_data_smithi173_74914_object58077:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ebe553:::benchmark_data_smithi173_74914_object78405:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54eca389:::benchmark_data_smithi173_74914_object95833:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ee36cd:::benchmark_data_smithi173_74914_object77332:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54eefe88:::benchmark_data_smithi173_74914_object84712:head# 2024-08-30T23:39:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54ef3aff:::benchmark_data_smithi173_74914_object93631:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f26c7d:::benchmark_data_smithi173_74914_object53749:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f5b79f:::benchmark_data_smithi173_74914_object81751:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f68224:::benchmark_data_smithi173_74914_object88708:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f6f7c2:::benchmark_data_smithi173_74914_object56108:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f786a4:::benchmark_data_smithi173_74914_object59841:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f8c2c6:::benchmark_data_smithi173_74914_object53101:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54f9d6ac:::benchmark_data_smithi173_74914_object77669:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54fa0d9b:::benchmark_data_smithi173_74914_object89687:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54fd0b86:::benchmark_data_smithi173_74914_object71957:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:54fe0a3d:::benchmark_data_smithi173_74914_object77185:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550061f7:::benchmark_data_smithi173_74914_object61806:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55017d95:::benchmark_data_smithi173_74914_object70447:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550233a8:::benchmark_data_smithi173_74914_object95471:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550417b9:::benchmark_data_smithi173_74914_object98236:head# 2024-08-30T23:39:51.339 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5506fc2f:::benchmark_data_smithi173_74914_object98213:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5508703f:::benchmark_data_smithi173_74914_object74951:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5508814d:::benchmark_data_smithi173_74914_object58082:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550a42fb:::benchmark_data_smithi173_74914_object60017:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550a8c34:::benchmark_data_smithi173_74914_object77424:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550be634:::benchmark_data_smithi173_74914_object82285:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:550deccb:::benchmark_data_smithi173_74914_object70197:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55110c56:::benchmark_data_smithi173_74914_object96725:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5518b4df:::benchmark_data_smithi173_74914_object92245:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5519a91f:::benchmark_data_smithi173_74914_object91883:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551a19b5:::benchmark_data_smithi173_74914_object73522:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551cc646:::benchmark_data_smithi173_74914_object94281:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551ccfe7:::benchmark_data_smithi173_74914_object74656:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551d3e3c:::benchmark_data_smithi173_74914_object61878:head# 2024-08-30T23:39:51.340 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551e249f:::benchmark_data_smithi173_74914_object71331:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551f1010:::benchmark_data_smithi173_74914_object74809:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551f4f81:::benchmark_data_smithi173_74914_object85799:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:551fd15f:::benchmark_data_smithi173_74914_object88709:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5523674b:::benchmark_data_smithi173_74914_object61289:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5526723e:::benchmark_data_smithi173_74914_object86219:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5526fc76:::benchmark_data_smithi173_74914_object63060:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5527ba95:::benchmark_data_smithi173_74914_object76275:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552bb7a1:::benchmark_data_smithi173_74914_object90423:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552c3107:::benchmark_data_smithi173_74914_object62290:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552cc0d0:::benchmark_data_smithi173_74914_object82955:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552e2ef9:::benchmark_data_smithi173_74914_object90188:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552ee1b8:::benchmark_data_smithi173_74914_object64508:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552f4ffb:::benchmark_data_smithi173_74914_object96742:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:552ff152:::benchmark_data_smithi173_74914_object93794:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5530122f:::benchmark_data_smithi173_74914_object58425:head# 2024-08-30T23:39:51.341 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55313dfc:::benchmark_data_smithi173_74914_object68933:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5531e0ea:::benchmark_data_smithi173_74914_object64410:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5532648b:::benchmark_data_smithi173_74914_object76314:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5536f335:::benchmark_data_smithi173_74914_object53688:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5537c966:::benchmark_data_smithi173_74914_object59875:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5537ef06:::benchmark_data_smithi173_74914_object96783:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553807b7:::benchmark_data_smithi173_74914_object83387:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5538aeae:::benchmark_data_smithi173_74914_object76489:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553ad4be:::benchmark_data_smithi173_74914_object78282:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553b94c5:::benchmark_data_smithi173_74914_object94371:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553c85ee:::benchmark_data_smithi173_74914_object63444:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553cb41f:::benchmark_data_smithi173_74914_object78146:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553cca47:::benchmark_data_smithi173_74914_object83571:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553d6b2e:::benchmark_data_smithi173_74914_object87322:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553e32b4:::benchmark_data_smithi173_74914_object58035:head# 2024-08-30T23:39:51.342 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:553fb1d3:::benchmark_data_smithi173_74914_object94883:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55423588:::benchmark_data_smithi173_74914_object74409:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55440313:::benchmark_data_smithi173_74914_object78161:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55441cbc:::benchmark_data_smithi173_74914_object72865:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5544cb5e:::benchmark_data_smithi173_74914_object83065:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5545b7e9:::benchmark_data_smithi173_74914_object88588:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5548acc3:::benchmark_data_smithi173_74914_object71660:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:554953ae:::benchmark_data_smithi173_74914_object81899:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55505a38:::benchmark_data_smithi173_74914_object80033:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555092d1:::benchmark_data_smithi173_74914_object78339:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5550ae49:::benchmark_data_smithi173_74914_object80847:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55515f61:::benchmark_data_smithi173_74914_object59051:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55520a67:::benchmark_data_smithi173_74914_object76258:head# 2024-08-30T23:39:51.343 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55536843:::benchmark_data_smithi173_74914_object82977:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5553ec6f:::benchmark_data_smithi173_74914_object77891:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55542a31:::benchmark_data_smithi173_74914_object60860:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55566473:::benchmark_data_smithi173_74914_object58662:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5556e414:::benchmark_data_smithi173_74914_object86271:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55570c0b:::benchmark_data_smithi173_74914_object91452:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555ac412:::benchmark_data_smithi173_74914_object55650:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555adad3:::benchmark_data_smithi173_74914_object60161:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555b159a:::benchmark_data_smithi173_74914_object72419:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555bc016:::benchmark_data_smithi173_74914_object93054:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555c01df:::benchmark_data_smithi173_74914_object79084:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555cbc75:::benchmark_data_smithi173_74914_object56357:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:555da819:::benchmark_data_smithi173_74914_object89436:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5560889b:::benchmark_data_smithi173_74914_object64576:head# 2024-08-30T23:39:51.344 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5561555b:::benchmark_data_smithi173_74914_object77936:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5561b3d9:::benchmark_data_smithi173_74914_object74342:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5563ba29:::benchmark_data_smithi173_74914_object56456:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55647d56:::benchmark_data_smithi173_74914_object53973:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5564f316:::benchmark_data_smithi173_74914_object53758:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5565630a:::benchmark_data_smithi173_74914_object85266:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55680af1:::benchmark_data_smithi173_74914_object98157:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5568981f:::benchmark_data_smithi173_74914_object84698:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:556aba3a:::benchmark_data_smithi173_74914_object61169:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:556b591d:::benchmark_data_smithi173_74914_object54800:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:556c1ff9:::benchmark_data_smithi173_74914_object64139:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:556d61b9:::benchmark_data_smithi173_74914_object94907:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55700d92:::benchmark_data_smithi173_74914_object57283:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5571b007:::benchmark_data_smithi173_74914_object78926:head# 2024-08-30T23:39:51.345 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55733b30:::benchmark_data_smithi173_74914_object56450:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5573cdc7:::benchmark_data_smithi173_74914_object60486:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5578e0d3:::benchmark_data_smithi173_74914_object65621:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:557952c8:::benchmark_data_smithi173_74914_object62069:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:557e4f82:::benchmark_data_smithi173_74914_object73856:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:557f3a98:::benchmark_data_smithi173_74914_object96182:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:557fa886:::benchmark_data_smithi173_74914_object54855:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55887047:::benchmark_data_smithi173_74914_object86374:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5588d881:::benchmark_data_smithi173_74914_object89913:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:558caada:::benchmark_data_smithi173_74914_object92786:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:558d75dd:::benchmark_data_smithi173_74914_object61410:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55909fd4:::benchmark_data_smithi173_74914_object89371:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5591e1ac:::benchmark_data_smithi173_74914_object86293:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55922559:::benchmark_data_smithi173_74914_object95978:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5593fc3f:::benchmark_data_smithi173_74914_object56400:head# 2024-08-30T23:39:51.346 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5594f1bc:::benchmark_data_smithi173_74914_object92594:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:559c9cee:::benchmark_data_smithi173_74914_object64123:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:559d5371:::benchmark_data_smithi173_74914_object73082:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:559de5fa:::benchmark_data_smithi173_74914_object69366:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:559e16d4:::benchmark_data_smithi173_74914_object89882:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:559f2dd8:::benchmark_data_smithi173_74914_object83061:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55a0b552:::benchmark_data_smithi173_74914_object95699:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55a25e82:::benchmark_data_smithi173_74914_object63906:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55a2ce4b:::benchmark_data_smithi173_74914_object87893:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55a3cb74:::benchmark_data_smithi173_74914_object70948:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55a4e20a:::benchmark_data_smithi173_74914_object75617:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55adabf4:::benchmark_data_smithi173_74914_object78944:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ae9a5f:::benchmark_data_smithi173_74914_object59378:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55b0811d:::benchmark_data_smithi173_74914_object93796:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55b26e35:::benchmark_data_smithi173_74914_object88235:head# 2024-08-30T23:39:51.347 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55b2831e:::benchmark_data_smithi173_74914_object64383:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55b43e86:::benchmark_data_smithi173_74914_object78654:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55b55479:::benchmark_data_smithi173_74914_object74893:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55b6a8ba:::benchmark_data_smithi173_74914_object70807:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55bdcb7f:::benchmark_data_smithi173_74914_object56780:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55bdfe66:::benchmark_data_smithi173_74914_object55440:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55be2c30:::benchmark_data_smithi173_74914_object65116:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55bed879:::benchmark_data_smithi173_74914_object60361:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55bf76c9:::benchmark_data_smithi173_74914_object95382:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c1ce51:::benchmark_data_smithi173_74914_object73935:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c25f6d:::benchmark_data_smithi173_74914_object57200:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c4b5db:::benchmark_data_smithi173_74914_object56816:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c68051:::benchmark_data_smithi173_74914_object97808:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c698a4:::benchmark_data_smithi173_74914_object63610:head# 2024-08-30T23:39:51.348 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c6bc42:::benchmark_data_smithi173_74914_object95780:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55c8c3e4:::benchmark_data_smithi173_74914_object60920:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ca2cbb:::benchmark_data_smithi173_74914_object81380:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ca62fa:::benchmark_data_smithi173_74914_object77391:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55caf47c:::benchmark_data_smithi173_74914_object62031:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55cb3708:::benchmark_data_smithi173_74914_object96057:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ccaae3:::benchmark_data_smithi173_74914_object59304:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ce22b5:::benchmark_data_smithi173_74914_object78808:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d01024:::benchmark_data_smithi173_74914_object56920:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d104b5:::benchmark_data_smithi173_74914_object78188:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d2d219:::benchmark_data_smithi173_74914_object86321:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d3d81d:::benchmark_data_smithi173_74914_object63325:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d44bb6:::benchmark_data_smithi173_74914_object72859:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d59e41:::benchmark_data_smithi173_74914_object59522:head# 2024-08-30T23:39:51.349 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d743ff:::benchmark_data_smithi173_74914_object61888:head# 2024-08-30T23:39:51.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55d91d59:::benchmark_data_smithi173_74914_object71591:head# 2024-08-30T23:39:51.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55da35ce:::benchmark_data_smithi173_74914_object78096:head# 2024-08-30T23:39:51.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55da9934:::benchmark_data_smithi173_74914_object60483:head# 2024-08-30T23:39:51.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55db2b6f:::benchmark_data_smithi173_74914_object72298:head# 2024-08-30T23:39:51.350 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55db4b52:::benchmark_data_smithi173_74914_object64021:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55dbceca:::benchmark_data_smithi173_74914_object57792:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55df163b:::benchmark_data_smithi173_74914_object79330:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e0020e:::benchmark_data_smithi173_74914_object68847:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e25844:::benchmark_data_smithi173_74914_object93152:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e3ad34:::benchmark_data_smithi173_74914_object66939:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e401d9:::benchmark_data_smithi173_74914_object82970:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e4f0ad:::benchmark_data_smithi173_74914_object81904:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e7c8ca:::benchmark_data_smithi173_74914_object77282:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e7cfbd:::benchmark_data_smithi173_74914_object72553:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e84ba2:::benchmark_data_smithi173_74914_object94088:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55e887b3:::benchmark_data_smithi173_74914_object63999:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55eb33d8:::benchmark_data_smithi173_74914_object81978:head# 2024-08-30T23:39:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55eb5474:::benchmark_data_smithi173_74914_object55530:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ec6947:::benchmark_data_smithi173_74914_object97426:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ed7261:::benchmark_data_smithi173_74914_object67467:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ee0809:::benchmark_data_smithi173_74914_object92444:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55ee13f6:::benchmark_data_smithi173_74914_object57312:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55eea9f7:::benchmark_data_smithi173_74914_object74780:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55eef716:::benchmark_data_smithi173_74914_object82945:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f1e795:::benchmark_data_smithi173_74914_object72500:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f34859:::benchmark_data_smithi173_74914_object68680:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f57ad4:::benchmark_data_smithi173_74914_object79082:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f7b367:::benchmark_data_smithi173_74914_object85031:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f82c48:::benchmark_data_smithi173_74914_object59719:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f8cd67:::benchmark_data_smithi173_74914_object86502:head# 2024-08-30T23:39:51.352 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55f9dbaf:::benchmark_data_smithi173_74914_object66498:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55fc54cc:::benchmark_data_smithi173_74914_object62880:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:55fe1897:::benchmark_data_smithi173_74914_object77231:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56008c81:::benchmark_data_smithi173_74914_object87171:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5600b4a8:::benchmark_data_smithi173_74914_object79490:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56012c00:::benchmark_data_smithi173_74914_object75510:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56013386:::benchmark_data_smithi173_74914_object69711:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56023efe:::benchmark_data_smithi173_74914_object77094:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:560307c8:::benchmark_data_smithi173_74914_object91243:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5605eabb:::benchmark_data_smithi173_74914_object85869:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:560e576f:::benchmark_data_smithi173_74914_object85847:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:560f332f:::benchmark_data_smithi173_74914_object58134:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:560fa0f1:::benchmark_data_smithi173_74914_object70658:head# 2024-08-30T23:39:51.353 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56109b7a:::benchmark_data_smithi173_74914_object56932:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5610df8a:::benchmark_data_smithi173_74914_object92975:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5614e227:::benchmark_data_smithi173_74914_object94550:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56153b07:::benchmark_data_smithi173_74914_object60791:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5616a114:::benchmark_data_smithi173_74914_object85386:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561776cb:::benchmark_data_smithi173_74914_object96747:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5618521b:::benchmark_data_smithi173_74914_object76939:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5618be80:::benchmark_data_smithi173_74914_object80292:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561a8b15:::benchmark_data_smithi173_74914_object77499:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561bc3a7:::benchmark_data_smithi173_74914_object90290:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561be8a9:::benchmark_data_smithi173_74914_object73737:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561c44d6:::benchmark_data_smithi173_74914_object70480:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561d5bf5:::benchmark_data_smithi173_74914_object77351:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:561e985a:::benchmark_data_smithi173_74914_object80672:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56201a8d:::benchmark_data_smithi173_74914_object95366:head# 2024-08-30T23:39:51.354 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5621e4fc:::benchmark_data_smithi173_74914_object59407:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56225867:::benchmark_data_smithi173_74914_object56596:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:562297f3:::benchmark_data_smithi173_74914_object76849:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:562348b6:::benchmark_data_smithi173_74914_object90851:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5623ec8e:::benchmark_data_smithi173_74914_object72252:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56244496:::benchmark_data_smithi173_74914_object64489:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5625eeb8:::benchmark_data_smithi173_74914_object81890:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:562a55dd:::benchmark_data_smithi173_74914_object97289:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:562b22f4:::benchmark_data_smithi173_74914_object60900:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:562bdd53:::benchmark_data_smithi173_74914_object71839:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:562e3b58:::benchmark_data_smithi173_74914_object63338:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5630af54:::benchmark_data_smithi173_74914_object96654:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5630ca64:::benchmark_data_smithi173_74914_object91411:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:563424fd:::benchmark_data_smithi173_74914_object69945:head# 2024-08-30T23:39:51.355 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:563a1f78:::benchmark_data_smithi173_74914_object58253:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:563b3823:::benchmark_data_smithi173_74914_object76398:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:563c12e9:::benchmark_data_smithi173_74914_object73227:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:563d4336:::benchmark_data_smithi173_74914_object60547:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5640cd9b:::benchmark_data_smithi173_74914_object93617:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56411269:::benchmark_data_smithi173_74914_object79539:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56421733:::benchmark_data_smithi173_74914_object69950:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56436466:::benchmark_data_smithi173_74914_object82431:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5643e5ca:::benchmark_data_smithi173_74914_object68660:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:564432bc:::benchmark_data_smithi173_74914_object74998:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56450c50:::benchmark_data_smithi173_74914_object67894:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:564809df:::benchmark_data_smithi173_74914_object68749:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5648b01f:::benchmark_data_smithi173_74914_object88395:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5648fd61:::benchmark_data_smithi173_74914_object54684:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5649686d:::benchmark_data_smithi173_74914_object83996:head# 2024-08-30T23:39:51.356 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:564ae084:::benchmark_data_smithi173_74914_object52807:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:564f5e97:::benchmark_data_smithi173_74914_object61217:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56502aa8:::benchmark_data_smithi173_74914_object68942:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565039fc:::benchmark_data_smithi173_74914_object86419:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56503d0b:::benchmark_data_smithi173_74914_object97477:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565165e4:::benchmark_data_smithi173_74914_object65643:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56520cd9:::benchmark_data_smithi173_74914_object92467:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5653a3ac:::benchmark_data_smithi173_74914_object91942:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5654c4f5:::benchmark_data_smithi173_74914_object57697:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5654f681:::benchmark_data_smithi173_74914_object95018:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56595b18:::benchmark_data_smithi173_74914_object60840:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565a406d:::benchmark_data_smithi173_74914_object96877:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565aefaa:::benchmark_data_smithi173_74914_object68656:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565d0ec2:::benchmark_data_smithi173_74914_object54467:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565e90e4:::benchmark_data_smithi173_74914_object66460:head# 2024-08-30T23:39:51.357 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565eb750:::benchmark_data_smithi173_74914_object81117:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:565fe10c:::benchmark_data_smithi173_74914_object93195:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:566207af:::benchmark_data_smithi173_74914_object90853:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56633461:::benchmark_data_smithi173_74914_object94149:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56644430:::benchmark_data_smithi173_74914_object57348:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56648f88:::benchmark_data_smithi173_74914_object76920:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:566600f2:::benchmark_data_smithi173_74914_object74831:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56685a0e:::benchmark_data_smithi173_74914_object86755:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5668e556:::benchmark_data_smithi173_74914_object79015:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56691d3c:::benchmark_data_smithi173_74914_object64062:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56694c98:::benchmark_data_smithi173_74914_object53477:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:566c7ae6:::benchmark_data_smithi173_74914_object78686:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:566f9433:::benchmark_data_smithi173_74914_object93452:head# 2024-08-30T23:39:51.358 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:566fa3b2:::benchmark_data_smithi173_74914_object70562:head# 2024-08-30T23:39:51.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:566fa9f0:::benchmark_data_smithi173_74914_object56724:head# 2024-08-30T23:39:51.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:56711e25:::benchmark_data_smithi173_74914_object69441:head# 2024-08-30T23:39:51.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:567183cd:::benchmark_data_smithi173_74914_object89933:head# 2024-08-30T23:39:51.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5675627d:::benchmark_data_smithi173_74914_object54556:head# 2024-08-30T23:39:51.359 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5677407d:::benchmark_data_smithi173_74914_object86989:head# 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:50 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:50 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:50 smithi173 ceph-mon[20554]: osdmap e455: 12 total, 10 up, 12 in 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:50 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 35550/222120 objects degraded (16.005%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:50 smithi173 ceph-mon[20554]: osdmap e456: 12 total, 10 up, 12 in 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:50 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:51.359 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:50 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:51.360 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:50 smithi136 ceph-mon[20207]: osdmap e455: 12 total, 10 up, 12 in 2024-08-30T23:39:51.360 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:50 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 35550/222120 objects degraded (16.005%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:51.360 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:50 smithi136 ceph-mon[20207]: osdmap e456: 12 total, 10 up, 12 in 2024-08-30T23:39:51.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:39:51.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:39:51.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:51.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:51.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:51.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:51.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:51.746 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:51.746 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:51.756 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:51.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:51.882 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:51.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:51.984 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:52.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:52.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:52.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:52.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:52.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:52.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:52.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:52.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:39:52.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:52.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:52.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:52.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.629 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:52.697 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:52.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:52.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:52 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:52.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:52 smithi136 ceph-mon[20207]: pgmap v1076: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35550/222120 objects degraded (16.005%); 14556/222120 objects misplaced (6.553%); 7.4 MiB/s, 118 objects/s recovering 2024-08-30T23:39:52.817 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:52.817 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:52 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:52.817 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:52 smithi173 ceph-mon[20554]: pgmap v1076: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35550/222120 objects degraded (16.005%); 14556/222120 objects misplaced (6.553%); 7.4 MiB/s, 118 objects/s recovering 2024-08-30T23:39:52.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:52 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:52 smithi106 ceph-mon[16603]: pgmap v1076: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35550/222120 objects degraded (16.005%); 14556/222120 objects misplaced (6.553%); 7.4 MiB/s, 118 objects/s recovering 2024-08-30T23:39:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:52.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:52.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:52.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:53.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:53.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:53.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:53.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.128 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:53.137 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:53.137 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:53.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:53.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:53.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:53.375 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:53.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:53.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:53.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:53.623 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:53.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:53.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:53.749 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:53 smithi173 ceph-mon[20554]: osdmap e457: 12 total, 10 up, 12 in 2024-08-30T23:39:53.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:53.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:39:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:53 smithi136 ceph-mon[20207]: osdmap e457: 12 total, 10 up, 12 in 2024-08-30T23:39:53.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:53 smithi106 ceph-mon[16603]: osdmap e457: 12 total, 10 up, 12 in 2024-08-30T23:39:53.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:53.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:53.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:54.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:54.019 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:39:54.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:54.127 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:54.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:54.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:54.281 INFO:teuthology.orchestra.run.smithi106.stderr:Read #4:5678c031: 2024-08-30T23:39:54.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.1 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 4.a 2024-08-30T23:39:54.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:54.378 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:54.480 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:54.500 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:54.569 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:39:54.569 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:39:54.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:54 smithi106 ceph-mon[16603]: osdmap e458: 12 total, 10 up, 12 in 2024-08-30T23:39:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:54 smithi106 ceph-mon[16603]: pgmap v1079: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35361/222120 objects degraded (15.920%); 14180/222120 objects misplaced (6.384%); 2.9 MiB/s, 46 objects/s recovering 2024-08-30T23:39:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:39:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:54 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:39:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:54.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:54.724 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:39:54.724 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:39:54.725 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:39:54.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:54.746 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:39:54.746 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:39:54.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:54.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:39:54.766 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:39:54.787 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:54.788 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:39:54.788 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:39:54.788 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:39:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:54 smithi136 ceph-mon[20207]: osdmap e458: 12 total, 10 up, 12 in 2024-08-30T23:39:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:54 smithi136 ceph-mon[20207]: pgmap v1079: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35361/222120 objects degraded (15.920%); 14180/222120 objects misplaced (6.384%); 2.9 MiB/s, 46 objects/s recovering 2024-08-30T23:39:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:39:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:54 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:39:54.811 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:39:54.811 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:39:54.812 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.1/config 2024-08-30T23:39:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:54 smithi173 ceph-mon[20554]: osdmap e458: 12 total, 10 up, 12 in 2024-08-30T23:39:54.889 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:54 smithi173 ceph-mon[20554]: pgmap v1079: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35361/222120 objects degraded (15.920%); 14180/222120 objects misplaced (6.384%); 2.9 MiB/s, 46 objects/s recovering 2024-08-30T23:39:54.889 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:39:54.889 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:54 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:39:54.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:54.890 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:39:54.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:39:54.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:39:54.891 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:39:54.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:39:54.891 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:39:54.898 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:39:54.899 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:39:54.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:54.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:54.920 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:39:54.920 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:39:55.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:55.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:55.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:55.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:55.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:55.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:39:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.284 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:55.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:55.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:55.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:55.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:39:55.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:39:55.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:55.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/438616202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:55.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:55.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:55.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:55.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/438616202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:39:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/438616202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:39:55.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:55.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:55.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:56.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:39:56.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:39:56.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:56.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:56.148 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:56.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:56.276 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:56.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:56.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:56.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:56.479 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:39:56.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:56.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:56 smithi106 ceph-mon[16603]: pgmap v1080: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35361/222120 objects degraded (15.920%); 14180/222120 objects misplaced (6.384%); 2.1 MiB/s, 33 objects/s recovering 2024-08-30T23:39:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:39:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:39:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 35361/222120 objects degraded (15.920%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/249055612' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:56.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:56.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:56.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:56.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:56.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:56 smithi136 ceph-mon[20207]: pgmap v1080: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35361/222120 objects degraded (15.920%); 14180/222120 objects misplaced (6.384%); 2.1 MiB/s, 33 objects/s recovering 2024-08-30T23:39:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:39:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:39:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 35361/222120 objects degraded (15.920%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/249055612' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:56 smithi173 ceph-mon[20554]: pgmap v1080: 27 pgs: 2 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 14 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 35361/222120 objects degraded (15.920%); 14180/222120 objects misplaced (6.384%); 2.1 MiB/s, 33 objects/s recovering 2024-08-30T23:39:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:39:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:39:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 35361/222120 objects degraded (15.920%), 8 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:39:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/249055612' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:39:56.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:56.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:56.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:57.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:57.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:57.080 INFO:teuthology.orchestra.run.smithi106.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:39:57.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:57.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:57.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:57.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:57.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:57.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:57.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:57.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:57 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:39:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/249055612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:57 smithi106 ceph-mon[16603]: osdmap e459: 12 total, 10 up, 12 in 2024-08-30T23:39:57.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:39:57.651 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:39:57.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:39:57.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:39:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:57 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:39:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/249055612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:57 smithi136 ceph-mon[20207]: osdmap e459: 12 total, 10 up, 12 in 2024-08-30T23:39:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:57 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:39:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/249055612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:39:57.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:57 smithi173 ceph-mon[20554]: osdmap e459: 12 total, 10 up, 12 in 2024-08-30T23:39:57.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:57.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:57.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:58.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:58.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:39:58.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: marking collection for removal 2024-08-30T23:39:58.113 INFO:teuthology.orchestra.run.smithi106.stdout:setting '_remove' omap key 2024-08-30T23:39:58.113 INFO:teuthology.orchestra.run.smithi106.stdout:finish_remove_pgs 4.a_head removing 4.a 2024-08-30T23:39:58.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:58.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:58.188 INFO:teuthology.orchestra.run.smithi106.stdout:Remove successful 2024-08-30T23:39:58.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:58.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:58.368 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:58.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:58.403 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:58.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:58.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:58.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:39:58.649 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:39:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:39:58.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:58.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:58.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:58 smithi136 ceph-mon[20207]: pgmap v1082: 27 pgs: 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35089/222120 objects degraded (15.797%); 13775/222120 objects misplaced (6.202%); 6.2 MiB/s, 144 objects/s recovering 2024-08-30T23:39:58.802 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:58 smithi106 ceph-mon[16603]: pgmap v1082: 27 pgs: 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35089/222120 objects degraded (15.797%); 13775/222120 objects misplaced (6.202%); 6.2 MiB/s, 144 objects/s recovering 2024-08-30T23:39:58.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:58 smithi173 ceph-mon[20554]: pgmap v1082: 27 pgs: 2 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 35089/222120 objects degraded (15.797%); 13775/222120 objects misplaced (6.202%); 6.2 MiB/s, 144 objects/s recovering 2024-08-30T23:39:58.829 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:39:58.830 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:39:58.830 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:39:58.830 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:39:58.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:39:58.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:58.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:39:59.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:59.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.029 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:59.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:59.207 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:39:59.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:59.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:59.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:39:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:59.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:39:59.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:39:59.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:39:59.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:39:59.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:39:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:39:59.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:59.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:39:59.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:39:59.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:39:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:39:59.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:00.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:00.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:39:59 smithi136 ceph-mon[20207]: pgmap v1083: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 34942/222120 objects degraded (15.731%); 13481/222120 objects misplaced (6.069%); 6.5 MiB/s, 143 objects/s recovering 2024-08-30T23:40:00.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:39:59 smithi173 ceph-mon[20554]: pgmap v1083: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 34942/222120 objects degraded (15.731%); 13481/222120 objects misplaced (6.069%); 6.5 MiB/s, 143 objects/s recovering 2024-08-30T23:40:00.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:39:59 smithi106 ceph-mon[16603]: pgmap v1083: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 34942/222120 objects degraded (15.731%); 13481/222120 objects misplaced (6.069%); 6.5 MiB/s, 143 objects/s recovering 2024-08-30T23:40:00.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:00.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:00.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:00.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:00.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:00.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:40:00.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:00.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:00.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:00.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:00.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: Health detail: HEALTH_WARN 2 osds down; Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: [WRN] OSD_DOWN: 2 osds down 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: osd.1 (root=default,host=smithi106) is down 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: osd.5 (root=default,host=smithi136) is down 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: [WRN] PG_DEGRADED: Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.3 is stuck undersized for 64s, current state active+recovering+undersized+degraded+remapped, last acting [0,7] 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.5 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,0] 2024-08-30T23:40:00.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.6 is active+undersized+degraded, acting [7,2] 2024-08-30T23:40:00.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.7 is undersized+degraded+peered, acting [9] 2024-08-30T23:40:00.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.b is active+undersized+degraded, acting [9,0] 2024-08-30T23:40:00.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.d is active+undersized+degraded, acting [11,7] 2024-08-30T23:40:00.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:00 smithi106 ceph-mon[16603]: pg 5.f is active+undersized+degraded, acting [11,2] 2024-08-30T23:40:00.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.5 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-08-30T23:40:00.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:00.778 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:00.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:00.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:00.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:00.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:00.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:00.923 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:00.924 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.5/config 2024-08-30T23:40:00.946 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:00.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:00.966 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:00.967 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:00.988 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:00.989 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:00.989 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:01.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:01.011 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:01.012 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:01.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:01.033 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:01.033 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: Health detail: HEALTH_WARN 2 osds down; Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: [WRN] OSD_DOWN: 2 osds down 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: osd.1 (root=default,host=smithi106) is down 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: osd.5 (root=default,host=smithi136) is down 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: [WRN] PG_DEGRADED: Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.3 is stuck undersized for 64s, current state active+recovering+undersized+degraded+remapped, last acting [0,7] 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.5 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,0] 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.6 is active+undersized+degraded, acting [7,2] 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.7 is undersized+degraded+peered, acting [9] 2024-08-30T23:40:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.b is active+undersized+degraded, acting [9,0] 2024-08-30T23:40:01.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.d is active+undersized+degraded, acting [11,7] 2024-08-30T23:40:01.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:00 smithi136 ceph-mon[20207]: pg 5.f is active+undersized+degraded, acting [11,2] 2024-08-30T23:40:01.054 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:01.054 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: Health detail: HEALTH_WARN 2 osds down; Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: [WRN] OSD_DOWN: 2 osds down 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: osd.1 (root=default,host=smithi106) is down 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: osd.5 (root=default,host=smithi136) is down 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: [WRN] PG_DEGRADED: Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.3 is stuck undersized for 64s, current state active+recovering+undersized+degraded+remapped, last acting [0,7] 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.5 is stuck undersized for 2m, current state active+undersized+degraded, last acting [6,0] 2024-08-30T23:40:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.6 is active+undersized+degraded, acting [7,2] 2024-08-30T23:40:01.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.7 is undersized+degraded+peered, acting [9] 2024-08-30T23:40:01.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.b is active+undersized+degraded, acting [9,0] 2024-08-30T23:40:01.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.d is active+undersized+degraded, acting [11,7] 2024-08-30T23:40:01.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:00 smithi173 ceph-mon[20554]: pg 5.f is active+undersized+degraded, acting [11,2] 2024-08-30T23:40:01.081 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:01.081 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:01.109 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:01.110 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:01.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:01.132 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:01.132 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:01.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:01.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:01.274 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:01.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:01.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:01.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:01.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:01.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:40:01.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:01.645 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:01.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:01.768 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:01.780 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:01 smithi106 ceph-mon[16603]: osdmap e460: 12 total, 10 up, 12 in 2024-08-30T23:40:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3656663914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:01 smithi106 ceph-mon[16603]: pgmap v1085: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 34942/222120 objects degraded (15.731%); 13481/222120 objects misplaced (6.069%); 4.3 MiB/s, 102 objects/s recovering 2024-08-30T23:40:01.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:01.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:01.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:01.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:02.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:02.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:01 smithi136 ceph-mon[20207]: osdmap e460: 12 total, 10 up, 12 in 2024-08-30T23:40:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3656663914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:01 smithi136 ceph-mon[20207]: pgmap v1085: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 34942/222120 objects degraded (15.731%); 13481/222120 objects misplaced (6.069%); 4.3 MiB/s, 102 objects/s recovering 2024-08-30T23:40:02.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:01 smithi173 ceph-mon[20554]: osdmap e460: 12 total, 10 up, 12 in 2024-08-30T23:40:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3656663914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:01 smithi173 ceph-mon[20554]: pgmap v1085: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 34942/222120 objects degraded (15.731%); 13481/222120 objects misplaced (6.069%); 4.3 MiB/s, 102 objects/s recovering 2024-08-30T23:40:02.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:02.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:02.192 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:40:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:02.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:02.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:02.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:02.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:02.387 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:02.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:02.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:02.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:02.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:02.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:02.751 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:02.759 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:02.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:02.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:02.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:02.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:03.005 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:40:03.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:03.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:03.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:03.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:03.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:03.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/138924566' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/138924566' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:03.329 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/138924566' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:03.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:03.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:03.480 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:03.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:03.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:03.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:03.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:03.749 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.f 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.b 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.7 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.6 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.4 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.d 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:5.3 2024-08-30T23:40:03.768 INFO:teuthology.orchestra.run.smithi136.stdout:1.8 2024-08-30T23:40:03.769 INFO:teuthology.orchestra.run.smithi136.stdout:1.7 2024-08-30T23:40:03.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:03.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:03.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:03.988 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:03.988 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:03.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:03.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:04 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/138924566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:04 smithi136 ceph-mon[20207]: osdmap e461: 12 total, 10 up, 12 in 2024-08-30T23:40:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:04 smithi136 ceph-mon[20207]: pgmap v1087: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 6.8 MiB/s, 147 objects/s recovering 2024-08-30T23:40:04.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:04.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.120 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:04.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:04.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:40:04.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:04.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:04 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:04.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/138924566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:04.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:04 smithi106 ceph-mon[16603]: osdmap e461: 12 total, 10 up, 12 in 2024-08-30T23:40:04.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:04 smithi106 ceph-mon[16603]: pgmap v1087: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 6.8 MiB/s, 147 objects/s recovering 2024-08-30T23:40:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:04 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/138924566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:04 smithi173 ceph-mon[20554]: osdmap e461: 12 total, 10 up, 12 in 2024-08-30T23:40:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:04 smithi173 ceph-mon[20554]: pgmap v1087: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 6.8 MiB/s, 147 objects/s recovering 2024-08-30T23:40:04.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:04.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:04.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:04.489 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:04.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:04.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:04.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:40:04.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:04.670 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:04.714 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:04.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:04.765 INFO:tasks.thrashosds.thrasher:Moving pg 4.a from osd.1 to osd.5 2024-08-30T23:40:04.765 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi106.front.sepia.ceph.com to ubuntu@smithi136.front.sepia.ceph.com 2024-08-30T23:40:04.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /var/log/ceph 2024-08-30T23:40:04.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-08-30T23:40:04.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:04.843 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-08-30T23:40:04.858 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.LVHlAAicQE 2024-08-30T23:40:04.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.4.a.1 /tmp/tmp.LVHlAAicQE 2024-08-30T23:40:04.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:04.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:05.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:05.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:05.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:05.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:05.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:05.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:05.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:05.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:05.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:05.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:05.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:05.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:05.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:40:05.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:05.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:05.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:05.912 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:05.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:06.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.035 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:06.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:06.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:05 smithi136 ceph-mon[20207]: pgmap v1088: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 2.8 MiB/s, 45 objects/s recovering 2024-08-30T23:40:06.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:05 smithi173 ceph-mon[20554]: pgmap v1088: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 2.8 MiB/s, 45 objects/s recovering 2024-08-30T23:40:06.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:05 smithi106 ceph-mon[16603]: pgmap v1088: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 2.8 MiB/s, 45 objects/s recovering 2024-08-30T23:40:06.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:06.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:06.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:06.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:06.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:06.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:06.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:06.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:06.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:06.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:06.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:06.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:06.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 34728/222120 objects degraded (15.635%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:06.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:06 smithi106 ceph-mon[16603]: osdmap e462: 12 total, 10 up, 12 in 2024-08-30T23:40:06.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1751136408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:06.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.LVHlAAicQE 2024-08-30T23:40:06.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:06.940 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:06.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:06.943 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:06.943 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:06.944 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:06.945 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:06.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:06.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:06.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:06.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:06.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:06.946 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.LVHlAAicQE is 177MB 2024-08-30T23:40:07.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:07.454 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 34728/222120 objects degraded (15.635%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:07.454 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:06 smithi173 ceph-mon[20554]: osdmap e462: 12 total, 10 up, 12 in 2024-08-30T23:40:07.454 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1751136408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:07.455 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 34728/222120 objects degraded (15.635%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:07.455 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:06 smithi136 ceph-mon[20207]: osdmap e462: 12 total, 10 up, 12 in 2024-08-30T23:40:07.455 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1751136408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:07.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:07.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:07.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:07.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:07.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:07.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:07.673 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:07.744 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:07.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:07.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:07.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:07.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:07.958 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:07.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:07.993 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:07 smithi106 ceph-mon[16603]: pgmap v1090: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:40:08.043 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:07 smithi136 ceph-mon[20207]: pgmap v1090: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:40:08.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:08.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:07 smithi173 ceph-mon[20554]: pgmap v1090: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34728/222120 objects degraded (15.635%); 13051/222120 objects misplaced (5.876%); 2.0 MiB/s, 31 objects/s recovering 2024-08-30T23:40:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:08.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:08.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:08.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:08.240 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:08.240 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:31057 4.b 4:d5378b27:::benchmark_data_smithi173_74914_object31056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 347.53905947599998, 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.641950197, 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:08.241 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 31057 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687358+0000", 2024-08-30T23:40:08.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687360+0000", 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0669999999999999e-06 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687397+0000", 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6677999999999997e-05 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687413+0000", 2024-08-30T23:40:08.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6532999999999999e-05 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687421+0000", 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9410000000000007e-06 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687486+0000", 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4894000000000006e-05 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687630+0000", 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014421900000000001 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.687780+0000", 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000149748 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689089+0000", 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013092609999999999 2024-08-30T23:40:08.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:20.689219+0000", 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013019 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329200+0000", 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63998067999999997 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329262+0000", 2024-08-30T23:40:08.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1508999999999997e-05 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:21.329308+0000", 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6477000000000002e-05 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:61123 4.b 4:d896c312:::benchmark_data_smithi173_74914_object61122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:40:08.247 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 314.01302014599997, 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93993815199999997, 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 61123 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213397+0000", 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213400+0000", 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5000000000000002e-06 2024-08-30T23:40:08.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213421+0000", 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1537000000000001e-05 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213434+0000", 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2478e-05 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213440+0000", 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8699999999999997e-06 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213485+0000", 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5439999999999999e-05 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213576+0000", 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0767999999999996e-05 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.213667+0000", 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1135000000000001e-05 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215888+0000", 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022207139999999999 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.215938+0000", 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0022000000000003e-05 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153264+0000", 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93732649300000004 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153304+0000", 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0139999999999999e-05 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153335+0000", 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1055e-05 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:61123 4.b e335/306 ondisk, result = 0)", 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 314.01037934700003, 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93730281299999996, 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:40:08.254 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216038+0000", 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216039+0000", 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4099999999999999e-07 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:54.216042+0000", 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7529999999999999e-06 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153195+0000", 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93715309700000005 2024-08-30T23:40:08.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153202+0000", 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7340000000000002e-06 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153255+0000", 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2420999999999998e-05 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153260+0000", 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7370000000000003e-06 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.153341+0000", 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0530000000000003e-05 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.258 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(4.5 335/318 [PushOp(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head, version: 316'1784, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aba41fea:::benchmark_data_smithi173_74914_object27791:head@316'1784, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 313.05216674399998, 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.83137486000000005, 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174251+0000", 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174252+0000", 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:55.174402+0000", 2024-08-30T23:40:08.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015002700000000001 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003927+0000", 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82952564699999998 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.003946+0000", 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8678999999999999e-05 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004043+0000", 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6718999999999993e-05 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.004054+0000", 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0925e-05 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:56.005626+0000", 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001571813 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.262 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98225 4.5 e353/337 4:ad425262:::benchmark_data_smithi173_74914_object98224:head v 353'6223, mlcod=353'6220)", 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.781447904, 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32306562100000003, 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444970+0000", 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.444971+0000", 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0699999999999999e-06 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.445067+0000", 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5938000000000002e-05 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765335+0000", 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32026851299999998 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765347+0000", 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.151e-05 2024-08-30T23:40:08.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765448+0000", 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100967 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.765468+0000", 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0219000000000001e-05 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.767987+0000", 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025194890000000002 2024-08-30T23:40:08.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.768035+0000", 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7914999999999998e-05 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:98228 4.7 e353/329 4:ef7954e5:::benchmark_data_smithi173_74914_object98227:head v 353'6056, mlcod=353'6052)", 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.77831121899999, 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35600754000000001, 2024-08-30T23:40:08.267 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448106+0000", 2024-08-30T23:40:08.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448107+0000", 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3300000000000001e-07 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.448202+0000", 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4503e-05 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801211+0000", 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35300926399999999 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801228+0000", 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6694999999999999e-05 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801373+0000", 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014589400000000001 2024-08-30T23:40:08.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.801409+0000", 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6025999999999997e-05 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804078+0000", 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026682939999999999 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:31.804114+0000", 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6031000000000001e-05 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:143768 4.3 e365/306 4:c93e37ef:::benchmark_data_smithi173_74914_object45526:head v 365'8833, mlcod=365'8830)", 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 228.79445123599999, 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.374659609, 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431966+0000", 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431967+0000", 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5000000000000001e-07 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.431975+0000", 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4139999999999997e-06 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.793620+0000", 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36164507800000001 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.793628+0000", 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9100000000000005e-06 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.794714+0000", 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010868989999999999 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.794732+0000", 2024-08-30T23:40:08.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7396000000000001e-05 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.806585+0000", 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011852713000000001 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:19.806626+0000", 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1248999999999998e-05 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:58980 5.8 e418/394)", 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:32.105688+0000", 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 96.120729588000003, 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.458888293, 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105688+0000", 2024-08-30T23:40:08.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105688+0000", 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105689+0000", 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4399999999999999e-07 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.105791+0000", 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000102163 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564508+0000", 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45871740599999999 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564515+0000", 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3150000000000003e-06 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564567+0000", 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1997999999999999e-05 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.564576+0000", 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6670000000000002e-06 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:80226 5.5 e439/389 5:ada75ddb:::benchmark_data_smithi173_87639_object80225:head v 439'5176, mlcod=439'5172)", 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:11.987304+0000", 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 56.239113711999998, 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.443118495, 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.987304+0000", 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.987304+0000", 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.987304+0000", 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9400000000000005e-07 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.987338+0000", 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4047000000000001e-05 2024-08-30T23:40:08.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.427892+0000", 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44055327799999999 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.427907+0000", 2024-08-30T23:40:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:08.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:08.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5057e-05 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.428051+0000", 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014391999999999999 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.428104+0000", 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3038999999999998e-05 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.430391+0000", 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022870820000000002 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.430422+0000", 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1377999999999997e-05 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.306 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:80231 5.2 e439/420 5:4ddaa76e:::benchmark_data_smithi173_87639_object80230:head v 439'4926, mlcod=415'2985)", 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:11.991271+0000", 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 56.235146270000001, 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64341151600000002, 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991271+0000", 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991271+0000", 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991272+0000", 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9699999999999996e-07 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991419+0000", 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014760399999999999 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632382+0000", 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64096311900000003 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632392+0000", 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1400000000000006e-06 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632443+0000", 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1415000000000002e-05 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632464+0000", 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0752999999999999e-05 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.634636+0000", 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021725220000000001 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.634683+0000", 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6366000000000003e-05 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:80234 5.b e439/389 5:d0f4e873:::benchmark_data_smithi173_87639_object80233:head v 439'5114, mlcod=439'5110)", 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:11.995050+0000", 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 56.231367347999999, 2024-08-30T23:40:08.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35324898500000002, 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.995050+0000", 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.995050+0000", 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.995051+0000", 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1999999999999998e-07 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.995159+0000", 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000108009 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.345774+0000", 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35061526700000001 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.345784+0000", 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0307e-05 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.345895+0000", 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000110103 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.345915+0000", 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0662e-05 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.348261+0000", 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023462079999999998 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.348299+0000", 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7608999999999999e-05 2024-08-30T23:40:08.315 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 449/432 [PushReplyOp(5:c200ac77:::benchmark_data_smithi173_87639_object8428:head)])", 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:24.221649+0000", 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 44.004768544999997, 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.49411498799999998, 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.221649+0000", 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.221649+0000", 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.221649+0000", 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.75e-07 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.221653+0000", 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.89e-06 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.713790+0000", 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.49213716400000002 2024-08-30T23:40:08.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.713814+0000", 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3462999999999999e-05 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.713879+0000", 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4667999999999998e-05 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:24.715764+0000", 2024-08-30T23:40:08.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018853279999999999 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:c8aed71c:::benchmark_data_smithi173_87639_object26550:head)])", 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 13.915238279, 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94060026299999999, 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.332 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.333 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311180+0000", 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5199999999999996e-07 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311185+0000", 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8019999999999999e-06 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.249924+0000", 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93873898200000006 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.334 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.249936+0000", 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2034e-05 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.250051+0000", 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115751 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.251779+0000", 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017280419999999999 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.335 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:cdfa4373:::benchmark_data_smithi173_87639_object26587:head)])", 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.937237625, 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64991660600000001, 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.336 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2300000000000001e-07 2024-08-30T23:40:08.337 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289185+0000", 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2810000000000002e-06 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937334+0000", 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64814879999999997 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937343+0000", 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6199999999999994e-06 2024-08-30T23:40:08.338 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937437+0000", 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3407999999999997e-05 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.939096+0000", 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001659874 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.339 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:c3bec23f:::benchmark_data_smithi173_87639_object26821:head)])", 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:56.121622+0000", 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.104795232000001, 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37426185499999998, 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.121622+0000", 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.340 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.121622+0000", 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.121622+0000", 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6e-07 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.341 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.121624+0000", 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8530000000000001e-06 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.493921+0000", 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37229643200000001 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.493930+0000", 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1779999999999992e-06 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.342 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.494023+0000", 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3000999999999999e-05 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:56.495884+0000", 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001861131 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:c1c728e1:::benchmark_data_smithi173_87639_object28064:head)])", 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:40:08.343 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 10.935041014999999, 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58924576699999998, 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.344 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291377+0000", 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8200000000000001e-07 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291379+0000", 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.17e-06 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.878910+0000", 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58753071199999995 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.345 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.878927+0000", 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7733000000000001e-05 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.879062+0000", 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013429699999999999 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.880622+0000", 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015604729999999999 2024-08-30T23:40:08.346 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:c3b8f5dc:::benchmark_data_smithi173_87639_object28564:head)])", 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 9.1047433770000001, 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82014575300000003, 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.347 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8200000000000001e-07 2024-08-30T23:40:08.348 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121677+0000", 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.807e-06 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939832+0000", 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81815479400000002 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.349 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939842+0000", 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9399999999999997e-06 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939959+0000", 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117068 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.941820+0000", 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001860762 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.350 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:ca3229d2:::benchmark_data_smithi173_87639_object28892:head)])", 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.0467633979999995, 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62786811200000003, 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:40:08.351 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.35e-07 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.352 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179657+0000", 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4949999999999998e-06 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.805946+0000", 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62628936400000002 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.805956+0000", 2024-08-30T23:40:08.353 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7599999999999997e-06 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.806029+0000", 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3069000000000001e-05 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.807522+0000", 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001493189 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.354 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 462/432 [PushReplyOp(5:c69c4cd1:::benchmark_data_smithi173_87639_object33999:head)])", 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.932739749, 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71283231999999996, 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.355 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1300000000000001e-07 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293681+0000", 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6290000000000002e-06 2024-08-30T23:40:08.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004558+0000", 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71087762700000001 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004576+0000", 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7363999999999998e-05 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004759+0000", 2024-08-30T23:40:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018305599999999999 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.006510+0000", 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017513310000000001 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 462/432 [PushReplyOp(5:c7f18ecf:::benchmark_data_smithi173_87639_object34330:head)])", 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:07.229909+0000", 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.99650854799999999, 2024-08-30T23:40:08.358 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.30998268699999998, 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.229909+0000", 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.229909+0000", 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:08.359 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.229909+0000", 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.229912+0000", 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9909999999999998e-06 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:08.360 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.538261+0000", 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.308348597 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.538278+0000", 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7439999999999999e-05 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.538462+0000", 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000183862 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:08.361 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.539892+0000", 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00142931 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:08.362 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:40:08.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:08.466 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:08.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:08.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:08.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:08.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:08.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:08.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:08.757 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:08.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:08.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:08.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:09.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:09.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:09.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.215 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:09.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:09.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:09.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:09.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:09.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:09.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:09.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:09.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:40:09.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:09.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:09.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:09.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:09.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:09.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:09 smithi173 ceph-mon[20554]: osdmap e463: 12 total, 10 up, 12 in 2024-08-30T23:40:09.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:09.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:09.948 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:09 smithi173 ceph-mon[20554]: pgmap v1092: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34547/222120 objects degraded (15.553%); 12691/222120 objects misplaced (5.714%); 3.8 MiB/s, 60 objects/s recovering 2024-08-30T23:40:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:09.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:09.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:10.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:10.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:10.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:09 smithi136 ceph-mon[20207]: osdmap e463: 12 total, 10 up, 12 in 2024-08-30T23:40:10.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:10.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:10.056 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:09 smithi136 ceph-mon[20207]: pgmap v1092: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34547/222120 objects degraded (15.553%); 12691/222120 objects misplaced (5.714%); 3.8 MiB/s, 60 objects/s recovering 2024-08-30T23:40:10.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:10.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:10.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:10.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:09 smithi106 ceph-mon[16603]: osdmap e463: 12 total, 10 up, 12 in 2024-08-30T23:40:10.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:10.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:10.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:09 smithi106 ceph-mon[16603]: pgmap v1092: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34547/222120 objects degraded (15.553%); 12691/222120 objects misplaced (5.714%); 3.8 MiB/s, 60 objects/s recovering 2024-08-30T23:40:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:10.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:10.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:10.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:10.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:10.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:10.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:10.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:10.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:10.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:10.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:10.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:10.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:10.772 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:10.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:10.813 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.LVHlAAicQE 2024-08-30T23:40:10.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:10.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:10.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:11.066 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:10 smithi136 ceph-mon[20207]: osdmap e464: 12 total, 10 up, 12 in 2024-08-30T23:40:11.071 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:11.102 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:10 smithi173 ceph-mon[20554]: osdmap e464: 12 total, 10 up, 12 in 2024-08-30T23:40:11.114 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:10 smithi106 ceph-mon[16603]: osdmap e464: 12 total, 10 up, 12 in 2024-08-30T23:40:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:11.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:11.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:11.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:11.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:11.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:11.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:11.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:11.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:11.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:11.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:11.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:11.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:11.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:11.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:11.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:11.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:12.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:12.057 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 34547/222120 objects degraded (15.553%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:12.057 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:11 smithi136 ceph-mon[20207]: pgmap v1094: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34547/222120 objects degraded (15.553%); 12691/222120 objects misplaced (5.714%); 1.9 MiB/s, 29 objects/s recovering 2024-08-30T23:40:12.082 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 34547/222120 objects degraded (15.553%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:12.082 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:11 smithi173 ceph-mon[20554]: pgmap v1094: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34547/222120 objects degraded (15.553%); 12691/222120 objects misplaced (5.714%); 1.9 MiB/s, 29 objects/s recovering 2024-08-30T23:40:12.103 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 34547/222120 objects degraded (15.553%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:12.103 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:11 smithi106 ceph-mon[16603]: pgmap v1094: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34547/222120 objects degraded (15.553%); 12691/222120 objects misplaced (5.714%); 1.9 MiB/s, 29 objects/s recovering 2024-08-30T23:40:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:12.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:12.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:12.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:12.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:12.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:12.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:12.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:12.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:12.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:12.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:12.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:12.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:12.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:12.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:12.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:12.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:12.850 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:12.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:12.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.004 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:13.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:13.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:13.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:13.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:13.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:13.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:13.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:13.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:13.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:13.639 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:40:13.639 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/96122643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:13.639 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:13 smithi106 ceph-mon[16603]: from='client.16986 172.21.15.106:0/96122643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:13.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:13.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:13.809 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/96122643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:13.809 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:13 smithi136 ceph-mon[20207]: from='client.16986 172.21.15.106:0/96122643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:13.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/96122643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:13.836 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:13 smithi173 ceph-mon[20554]: from='client.16986 172.21.15.106:0/96122643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:13.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:13.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv /var/log/ceph/exp.4.a.1 /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.4.a.1 2024-08-30T23:40:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:13.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:13.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:13.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.5 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.a.1 2024-08-30T23:40:14.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:14.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:14.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:14.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.184 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:14.197 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.5/config 2024-08-30T23:40:14.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:14.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:14.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:14.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:40:14.431 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:14.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:14.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:14.591 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: from='client.16986 172.21.15.106:0/96122643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: osdmap e465: 12 total, 10 up, 12 in 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: pgmap v1096: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 4.8 MiB/s, 76 objects/s recovering 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:14.592 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:14 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:14.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.681 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:14.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:14.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:14.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: from='client.16986 172.21.15.106:0/96122643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: osdmap e465: 12 total, 10 up, 12 in 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: pgmap v1096: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 4.8 MiB/s, 76 objects/s recovering 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:14 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: from='client.16986 172.21.15.106:0/96122643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: osdmap e465: 12 total, 10 up, 12 in 2024-08-30T23:40:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: pgmap v1096: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 4.8 MiB/s, 76 objects/s recovering 2024-08-30T23:40:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:14.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:14.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:14.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:14 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:14.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:14.927 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:14.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:15.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:15.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:15.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:15.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:15.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:15.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:15.369 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:40:15.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:15.392 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:15.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:15.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:15 smithi106 ceph-mon[16603]: osdmap e466: 12 total, 10 up, 12 in 2024-08-30T23:40:15.402 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2243741919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:15.402 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:15.402 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:15.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:15.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:15.427 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:15.448 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:15.448 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:15.449 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:15.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:15.469 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:15.489 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:15.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:15.510 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:15.511 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:15.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:15.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:15.532 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:15.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:15.552 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:15.553 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:15.574 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:15.574 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:15.612 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:15 smithi136 ceph-mon[20207]: osdmap e466: 12 total, 10 up, 12 in 2024-08-30T23:40:15.613 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2243741919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:15.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:15.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:15.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:15.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:15 smithi173 ceph-mon[20554]: osdmap e466: 12 total, 10 up, 12 in 2024-08-30T23:40:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2243741919' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:15.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:15.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:15.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:16.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:16.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:16.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:16.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:16.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:16.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:16.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:16.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:16.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:16.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:16.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:16.637 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:16.637 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:16.637 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:16 smithi106 ceph-mon[16603]: pgmap v1098: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 2.9 MiB/s, 46 objects/s recovering 2024-08-30T23:40:16.637 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 34270/222120 objects degraded (15.429%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:16.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:16.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:16 smithi136 ceph-mon[20207]: pgmap v1098: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 2.9 MiB/s, 46 objects/s recovering 2024-08-30T23:40:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 34270/222120 objects degraded (15.429%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:16.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:16.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:16 smithi173 ceph-mon[20554]: pgmap v1098: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 2.9 MiB/s, 46 objects/s recovering 2024-08-30T23:40:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 34270/222120 objects degraded (15.429%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:16.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:40:16.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:16.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:16.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:17.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:17.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:17.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:17.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:17.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:17.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:17.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:17.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:17.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:17.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:17.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:17.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:17.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:17.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:40:17.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:17.792 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:17.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:17.864 INFO:teuthology.orchestra.run.smithi136.stderr:Pool 4 no longer exists 2024-08-30T23:40:17.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:17.912 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:17 smithi106 ceph-mon[16603]: pgmap v1099: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 2.2 MiB/s, 35 objects/s recovering 2024-08-30T23:40:17.922 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:17.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:18.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:17 smithi136 ceph-mon[20207]: pgmap v1099: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 2.2 MiB/s, 35 objects/s recovering 2024-08-30T23:40:18.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:17 smithi173 ceph-mon[20554]: pgmap v1099: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34270/222120 objects degraded (15.429%); 12137/222120 objects misplaced (5.464%); 2.2 MiB/s, 35 objects/s recovering 2024-08-30T23:40:18.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:18.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:18.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:18.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:18.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:18.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:18.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:18.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:40:18.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:18.542 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:18.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:18.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:18.761 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:18 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:18.761 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:18 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:18.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:18.778 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:18 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:18.778 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:18 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:18.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:18.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:18.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:18.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:19.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:19.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:19.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:18 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:19.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:18 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:19.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:19.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 10 2024-08-30T23:40:19.183 INFO:tasks.thrashosds.thrasher:Pool went away before processing an import...ignored 2024-08-30T23:40:19.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.4.a.1 2024-08-30T23:40:19.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.4.a.1 2024-08-30T23:40:19.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:19.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:19.359 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:19.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:19.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:19.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:19.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:19.626 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:40:19.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:19.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:19.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:19.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:19 smithi106 ceph-mon[16603]: osdmap e467: 12 total, 10 up, 12 in 2024-08-30T23:40:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:19 smithi106 ceph-mon[16603]: pgmap v1101: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33962/222120 objects degraded (15.290%); 11521/222120 objects misplaced (5.187%); 3.1 MiB/s, 49 objects/s recovering 2024-08-30T23:40:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/856655249' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:40:19.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:19 smithi136 ceph-mon[20207]: osdmap e467: 12 total, 10 up, 12 in 2024-08-30T23:40:19.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:19 smithi136 ceph-mon[20207]: pgmap v1101: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33962/222120 objects degraded (15.290%); 11521/222120 objects misplaced (5.187%); 3.1 MiB/s, 49 objects/s recovering 2024-08-30T23:40:19.886 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/856655249' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:40:19.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:19.932 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:19.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:20.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:20.056 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:19 smithi173 ceph-mon[20554]: osdmap e467: 12 total, 10 up, 12 in 2024-08-30T23:40:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:19 smithi173 ceph-mon[20554]: pgmap v1101: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33962/222120 objects degraded (15.290%); 11521/222120 objects misplaced (5.187%); 3.1 MiB/s, 49 objects/s recovering 2024-08-30T23:40:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/856655249' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:40:20.103 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:20.103 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:20.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:20.180 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:20.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:20.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:20.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:20.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:20.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:20.551 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:20.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:20.673 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:20.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:20.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:20 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/856655249' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:40:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:20 smithi106 ceph-mon[16603]: osdmap e468: 12 total, 10 up, 12 in 2024-08-30T23:40:20.857 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:20.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:20.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:20.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:21.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:20 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/856655249' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:40:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:20 smithi136 ceph-mon[20207]: osdmap e468: 12 total, 10 up, 12 in 2024-08-30T23:40:21.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:20 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/856655249' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:40:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:20 smithi173 ceph-mon[20554]: osdmap e468: 12 total, 10 up, 12 in 2024-08-30T23:40:21.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:21.166 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:21.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:21.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:40:21.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:21.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.413 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:21.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:21.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:21.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:21.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:21.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:21.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:21.773 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:21.774 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:21.774 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:21.778 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:21.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:21.800 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:21.801 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:21.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 33962/222120 objects degraded (15.290%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:21.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:21 smithi106 ceph-mon[16603]: pgmap v1103: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33962/222120 objects degraded (15.290%); 11521/222120 objects misplaced (5.187%); 2.7 MiB/s, 42 objects/s recovering 2024-08-30T23:40:21.822 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:21.823 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:21.846 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:21.847 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:21.847 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:21.868 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:21.868 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:21.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:21.888 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:21.889 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:21.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:21.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:21.910 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:21.910 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:21.933 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:21.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:21.956 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:21.956 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:21.977 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:21.977 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:22.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:22.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 33962/222120 objects degraded (15.290%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:21 smithi136 ceph-mon[20207]: pgmap v1103: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33962/222120 objects degraded (15.290%); 11521/222120 objects misplaced (5.187%); 2.7 MiB/s, 42 objects/s recovering 2024-08-30T23:40:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 33962/222120 objects degraded (15.290%), 7 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:21 smithi173 ceph-mon[20554]: pgmap v1103: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33962/222120 objects degraded (15.290%); 11521/222120 objects misplaced (5.187%); 2.7 MiB/s, 42 objects/s recovering 2024-08-30T23:40:22.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:22.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:22.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:22.275 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:22.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:22.399 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:1186 4.4 e311/306 4:2bc10e34:::benchmark_data_smithi173_74914_object1185:head v 311'76, mlcod=311'72)", 2024-08-30T23:40:22.457 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 380.59360510900001, 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21510685600000001, 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.458 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862593+0000", 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4200000000000005e-07 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862614+0000", 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1129e-05 2024-08-30T23:40:22.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077250+0000", 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214636047 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077258+0000", 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9789999999999993e-06 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077310+0000", 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2367999999999997e-05 2024-08-30T23:40:22.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077334+0000", 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4168e-05 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077669+0000", 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033434399999999998 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.461 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077699+0000", 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9978999999999999e-05 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:168798 4.6 e375/369 4:68592339:::benchmark_data_smithi173_74914_object70556:head v 375'10605, mlcod=375'10602)", 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 214.929088224, 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208871322, 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.462 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527110+0000", 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5499999999999997e-07 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527116+0000", 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0209999999999996e-06 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732384+0000", 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205268487 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732390+0000", 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0419999999999999e-06 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732440+0000", 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9663000000000002e-05 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732470+0000", 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0412e-05 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735945+0000", 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034745869999999999 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735980+0000", 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5255e-05 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:15236 5.4 e403/394 5:275cc9e1:::benchmark_data_smithi173_87639_object15235:head v 403'976, mlcod=403'973)", 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:50.728823+0000", 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.727373629, 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.198409419, 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728823+0000", 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728823+0000", 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728830+0000", 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0389999999999999e-06 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728929+0000", 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9114e-05 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926322+0000", 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19739287599999999 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926328+0000", 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6849999999999998e-06 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926390+0000", 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1619999999999996e-05 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926403+0000", 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3327e-05 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927185+0000", 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078186300000000005 2024-08-30T23:40:22.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927233+0000", 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7895000000000001e-05 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:15243 5.2 e403/394 5:4bef17d2:::benchmark_data_smithi173_87639_object15242:head v 403'951, mlcod=403'948)", 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.724416367, 2024-08-30T23:40:22.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19543065400000001, 2024-08-30T23:40:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.472 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7800000000000001e-07 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731845+0000", 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3625000000000002e-05 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926238+0000", 2024-08-30T23:40:22.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19439292499999999 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926246+0000", 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5380000000000002e-06 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926310+0000", 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3526000000000001e-05 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926339+0000", 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9323e-05 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927173+0000", 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00083399799999999997 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927211+0000", 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8040999999999997e-05 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.476 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:31241 5.4 e406/394 5:27c8315f:::benchmark_data_smithi173_87639_object31240:head v 406'1956, mlcod=406'1953)", 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 141.22599193799999, 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.165546794, 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6499999999999999e-07 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230313+0000", 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107126 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394252+0000", 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16393967000000001 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394272+0000", 2024-08-30T23:40:22.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0098999999999998e-05 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394426+0000", 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000153843 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394467+0000", 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1226000000000001e-05 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.395715+0000", 2024-08-30T23:40:22.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012479399999999999 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.395752+0000", 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6325000000000002e-05 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:59001 5.6 e419/389 5:6208a760:::benchmark_data_smithi173_87639_object59000:head v 419'3623, mlcod=418'3620)", 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 110.31312977499999, 2024-08-30T23:40:22.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.314356937, 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143070+0000", 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5739999999999998e-06 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143128+0000", 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7890000000000003e-05 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455493+0000", 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31236522100000003 2024-08-30T23:40:22.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455506+0000", 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3655999999999999e-05 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455662+0000", 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015594900000000001 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455681+0000", 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8301e-05 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.457390+0000", 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017094860000000001 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.457424+0000", 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3859999999999998e-05 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.485 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 424/420 [PushReplyOp(5:11696329:::benchmark_data_smithi173_87639_object5188:head)])", 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 102.634872409, 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45081538799999998, 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821326+0000", 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3340000000000001e-06 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821332+0000", 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0360000000000003e-06 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269704+0000", 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44837247699999999 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269713+0000", 2024-08-30T23:40:22.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269806+0000", 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3058000000000002e-05 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.272140+0000", 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023334639999999999 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.489 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 425/420 [PushReplyOp(5:10493f51:::benchmark_data_smithi173_87639_object5215:head)])", 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 102.094005647, 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318688992, 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:40:22.490 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362192+0000", 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.141e-06 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362198+0000", 2024-08-30T23:40:22.491 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6130000000000003e-06 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678705+0000", 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31650737299999998 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678716+0000", 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0288e-05 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678809+0000", 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3018999999999998e-05 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.680880+0000", 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020715579999999998 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 427/420 [PushReplyOp(5:123eeb63:::benchmark_data_smithi173_87639_object7071:head)])", 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:43.522780+0000", 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 98.933416940000001, 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19251843099999999, 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522780+0000", 2024-08-30T23:40:22.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522780+0000", 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522781+0000", 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522791+0000", 2024-08-30T23:40:22.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0169999999999999e-05 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.712969+0000", 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.190178027 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.712987+0000", 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8073000000000001e-05 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.713142+0000", 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000154993 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.715298+0000", 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021558879999999999 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 427/420 [PushReplyOp(5:1ccb0413:::benchmark_data_smithi173_87639_object8971:head)])", 2024-08-30T23:40:22.497 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.952505978999994, 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60307924400000001, 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:40:22.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503692+0000", 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.02e-07 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503698+0000", 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9819999999999996e-06 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104246+0000", 2024-08-30T23:40:22.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60054844500000004 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104278+0000", 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.18e-05 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104366+0000", 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7817000000000003e-05 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.106770+0000", 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002404298 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:81366 5.8 5:189e6c06:::benchmark_data_smithi173_87639_object1126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:14.116539+0000", 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.339657653000003, 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.104431312, 2024-08-30T23:40:22.501 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 81366 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.502 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116539+0000", 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116539+0000", 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116542+0000", 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5229999999999998e-06 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116554+0000", 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.27e-05 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116591+0000", 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6328999999999997e-05 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116597+0000", 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6290000000000004e-06 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116658+0000", 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0804999999999998e-05 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116789+0000", 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000131259 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:40:22.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116931+0000", 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014145 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.121433+0000", 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045022489999999998 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.210573+0000", 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.089139754000000002 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218603+0000", 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0080298829999999998 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.220576+0000", 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019733799999999998 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.220645+0000", 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8522000000000006e-05 2024-08-30T23:40:22.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.220971+0000", 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032582899999999997 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:81366 5.8 e440/420 ondisk, result = 0)", 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:14.119138+0000", 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.337058786, 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099470546000000007, 2024-08-30T23:40:22.508 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119138+0000", 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119138+0000", 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119139+0000", 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8899999999999999e-07 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119143+0000", 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7349999999999998e-06 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218234+0000", 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099091913000000004 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218238+0000", 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2490000000000001e-06 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218592+0000", 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000354173 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218598+0000", 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0480000000000004e-06 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218609+0000", 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0739e-05 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 440/420 [PushReplyOp(5:154bd465:::benchmark_data_smithi173_87639_object47039:head)])", 2024-08-30T23:40:22.512 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:14.122831+0000", 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 68.33336611, 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.095671489999999998, 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.122831+0000", 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.122831+0000", 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.122832+0000", 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.92e-07 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.122837+0000", 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1819999999999997e-06 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.217642+0000", 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.094805558999999998 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.217650+0000", 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.525e-06 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.217690+0000", 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0535999999999999e-05 2024-08-30T23:40:22.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218502+0000", 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00081189599999999999 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84105 5.8 5:17a91613:::benchmark_data_smithi173_87639_object3865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:40:22.516 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 65.283870695999994, 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22008060500000001, 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84105 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:40:22.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172327+0000", 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.124e-06 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172332+0000", 2024-08-30T23:40:22.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9910000000000002e-06 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172344+0000", 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1755e-05 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172351+0000", 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5699999999999998e-06 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172564+0000", 2024-08-30T23:40:22.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000213317 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172648+0000", 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3610999999999996e-05 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172745+0000", 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7783999999999998e-05 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:22.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.174806+0000", 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020607030000000001 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175526+0000", 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071991399999999999 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.176022+0000", 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049577499999999995 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391996+0000", 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21597425100000001 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392038+0000", 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2058999999999998e-05 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392407+0000", 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036875100000000001 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:84105 5.8 e443/420 ondisk, result = 0)", 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 65.281035463999999, 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21725276199999999, 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175162+0000", 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175166+0000", 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2189999999999998e-06 2024-08-30T23:40:22.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391931+0000", 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21676427100000001 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391937+0000", 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0270000000000001e-06 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391986+0000", 2024-08-30T23:40:22.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9175999999999998e-05 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391992+0000", 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0929999999999999e-06 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392414+0000", 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042228600000000002 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.526 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84505 5.a 5:51170427:::benchmark_data_smithi173_87639_object4265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.766478+0000", 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.689719077000007, 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.111502222, 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84505 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766478+0000", 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766478+0000", 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766479+0000", 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1599999999999999e-06 2024-08-30T23:40:22.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766485+0000", 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2450000000000003e-06 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766496+0000", 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.128e-05 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766503+0000", 2024-08-30T23:40:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3589999999999996e-06 2024-08-30T23:40:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.769391+0000", 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028883820000000001 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.769487+0000", 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6069000000000005e-05 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 8,10", 2024-08-30T23:40:22.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.769593+0000", 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105752 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.774123+0000", 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045304359999999997 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.776252+0000", 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002128614 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.876452+0000", 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.100199811 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.876497+0000", 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5269999999999999e-05 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.877980+0000", 2024-08-30T23:40:22.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014828440000000001 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84662 5.8 5:1acada36:::benchmark_data_smithi173_87639_object4422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:18.051032+0000", 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.405164936000006, 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.161365491, 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:22.534 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84662 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051032+0000", 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051032+0000", 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051033+0000", 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.449e-06 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051040+0000", 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1600000000000003e-06 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051051+0000", 2024-08-30T23:40:22.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1821000000000001e-05 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051058+0000", 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.097e-06 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051130+0000", 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1766000000000004e-05 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:22.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051276+0000", 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145465 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051419+0000", 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014362199999999999 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.053932+0000", 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025125719999999998 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:22.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.053951+0000", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8984e-05 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.054643+0000", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000692295 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.212029+0000", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.157385993 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.212106+0000", 2024-08-30T23:40:22.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7330000000000007e-05 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.212397+0000", 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029093700000000001 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 444/420 [PushReplyOp(5:14a83234:::benchmark_data_smithi173_87639_object53167:head)])", 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:18.062274+0000", 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.393922426000003, 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14578411499999999, 2024-08-30T23:40:22.540 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062274+0000", 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062274+0000", 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062275+0000", 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7199999999999998e-07 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062280+0000", 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9729999999999997e-06 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.207439+0000", 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14515842300000001 2024-08-30T23:40:22.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.207444+0000", 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.207507+0000", 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2639999999999997e-05 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.208059+0000", 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00055189400000000004 2024-08-30T23:40:22.543 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:22.544 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:40:22.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.566 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:22.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:40:22.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:22.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:22.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:22.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:22.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:22.929 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:22 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:40:22.929 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3969012710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:22.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:22.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:23.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:22 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:40:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3969012710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:23.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:22 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:40:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3969012710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:23.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:23.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:23.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:23.307 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:23.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:23.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:23.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:23.533 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.555 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:23.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:23.656 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:23.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:23.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:23.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:40:23.784 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:23.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:23.905 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:23 smithi106 ceph-mon[16603]: pgmap v1104: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 5.9 MiB/s, 94 objects/s recovering 2024-08-30T23:40:23.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:23.926 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:23.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:24.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:23 smithi136 ceph-mon[20207]: pgmap v1104: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 5.9 MiB/s, 94 objects/s recovering 2024-08-30T23:40:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:23 smithi173 ceph-mon[20554]: pgmap v1104: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 5.9 MiB/s, 94 objects/s recovering 2024-08-30T23:40:24.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:24.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:24.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:24.298 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:24.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1, 5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-08-30T23:40:24.307 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:40:24.307 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:40:24.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:40:24.331 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:24.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:24.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:24.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:24.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:24.639 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:40:24.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:24.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:24.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1264819509' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:24.763 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:40:24.764 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:40:24.764 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:40:24.764 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:40:24.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:24.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:24.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:24.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:24.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:24.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:25.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:25.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1264819509' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:25.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:25.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1264819509' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:25.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:25.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:25.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:25.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:25.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:40:25.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:25.412 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:25.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:25.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:25.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:25.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:25.770 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:25 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:40:25.770 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1264819509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:25.770 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:25 smithi106 ceph-mon[16603]: osdmap e469: 12 total, 10 up, 12 in 2024-08-30T23:40:25.771 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:25 smithi106 ceph-mon[16603]: pgmap v1106: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 4.0 MiB/s, 64 objects/s recovering 2024-08-30T23:40:25.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:25.859 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:25.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:25.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:25.985 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:25 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:40:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1264819509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:25 smithi136 ceph-mon[20207]: osdmap e469: 12 total, 10 up, 12 in 2024-08-30T23:40:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:25 smithi136 ceph-mon[20207]: pgmap v1106: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 4.0 MiB/s, 64 objects/s recovering 2024-08-30T23:40:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:25 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:40:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1264819509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:25 smithi173 ceph-mon[20554]: osdmap e469: 12 total, 10 up, 12 in 2024-08-30T23:40:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:25 smithi173 ceph-mon[20554]: pgmap v1106: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 4.0 MiB/s, 64 objects/s recovering 2024-08-30T23:40:26.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:26.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:40:26.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:26.224 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:26.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:26.258 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:26.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:26.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:26.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:26.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:26.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:26.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:26.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:26.722 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:26.741 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:26.741 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:26.741 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:26.742 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:26.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 33516/222120 objects degraded (15.089%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:26.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:26 smithi106 ceph-mon[16603]: osdmap e470: 12 total, 10 up, 12 in 2024-08-30T23:40:26.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:26 smithi106 ceph-mon[16603]: 5.1 scrub starts 2024-08-30T23:40:26.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:26.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:26.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:26.973 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:26.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 33516/222120 objects degraded (15.089%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:26 smithi136 ceph-mon[20207]: osdmap e470: 12 total, 10 up, 12 in 2024-08-30T23:40:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:26 smithi136 ceph-mon[20207]: 5.1 scrub starts 2024-08-30T23:40:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 33516/222120 objects degraded (15.089%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:26 smithi173 ceph-mon[20554]: osdmap e470: 12 total, 10 up, 12 in 2024-08-30T23:40:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:26 smithi173 ceph-mon[20554]: 5.1 scrub starts 2024-08-30T23:40:27.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:27.095 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:27.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:27.218 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:27.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:27.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:27.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:27.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:40:27.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:27.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:27.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:27.685 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:27.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:40:27.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:27.708 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:27.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:27.889 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:27.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:27.890 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:27.897 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:27.897 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:27.897 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:27 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:40:27.898 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:27 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:40:27.898 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:27 smithi106 ceph-mon[16603]: pgmap v1108: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:40:27.928 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:27.929 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:27.929 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:27.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:27.957 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:27.963 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:27.963 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:27.964 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:27.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:27.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:27.982 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:27.997 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:27.997 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:28.021 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:28.021 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:27 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:40:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:27 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:40:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:27 smithi136 ceph-mon[20207]: pgmap v1108: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:40:28.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:28.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:28.067 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:28.068 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:28.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:27 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:40:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:27 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:40:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:27 smithi173 ceph-mon[20554]: pgmap v1108: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33516/222120 objects degraded (15.089%); 10627/222120 objects misplaced (4.784%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:40:28.091 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:28.093 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:28.094 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:28.116 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:28.116 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:28.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:28.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:28.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.339 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:28.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:28.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:28.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:28.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:28.585 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:28.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:28.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:28.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:28.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4042351966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:28.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:28 smithi106 ceph-mon[16603]: 5.1 scrub ok 2024-08-30T23:40:28.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:28.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:28.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4042351966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:28 smithi136 ceph-mon[20207]: 5.1 scrub ok 2024-08-30T23:40:29.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4042351966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:28 smithi173 ceph-mon[20554]: 5.1 scrub ok 2024-08-30T23:40:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:29.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:29.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:29.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:29.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:29.267 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:29.267 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:29.306 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:29.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:29.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:29.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:29.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:29.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:29.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:29.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:29.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:29.896 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:29 smithi106 ceph-mon[16603]: pgmap v1109: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33177/222120 objects degraded (14.937%); 9951/222120 objects misplaced (4.480%); 6.1 MiB/s, 98 objects/s recovering 2024-08-30T23:40:29.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:29.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:29.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:29.988 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:29 smithi136 ceph-mon[20207]: pgmap v1109: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33177/222120 objects degraded (14.937%); 9951/222120 objects misplaced (4.480%); 6.1 MiB/s, 98 objects/s recovering 2024-08-30T23:40:30.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:30.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:29 smithi173 ceph-mon[20554]: pgmap v1109: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33177/222120 objects degraded (14.937%); 9951/222120 objects misplaced (4.480%); 6.1 MiB/s, 98 objects/s recovering 2024-08-30T23:40:30.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:30.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:30.200 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:30.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:30.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:30.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:30.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:30.451 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:30.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:30.574 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:30.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:30.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:30.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:30.753 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:30.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:40:30.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:30.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:30.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:30.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:30.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:31.030 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:30 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 33177/222120 objects degraded (14.937%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:30 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 33177/222120 objects degraded (14.937%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:31.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:31.067 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:30 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 33177/222120 objects degraded (14.937%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:31.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:40:31.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:31.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:31.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:31.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:31.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:40:31.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:31.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:31.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:31.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:31.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:31.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:31.734 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:31.751 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:31.760 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:31.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:31.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:31.865 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:31 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:40:31.865 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:31 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:40:31.865 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:31 smithi106 ceph-mon[16603]: pgmap v1110: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33177/222120 objects degraded (14.937%); 9951/222120 objects misplaced (4.480%); 2.6 MiB/s, 42 objects/s recovering 2024-08-30T23:40:31.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:31.931 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:31.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:32.032 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:31 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:40:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:31 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:40:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:31 smithi136 ceph-mon[20207]: pgmap v1110: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33177/222120 objects degraded (14.937%); 9951/222120 objects misplaced (4.480%); 2.6 MiB/s, 42 objects/s recovering 2024-08-30T23:40:32.052 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:31 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:40:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:31 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:40:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:31 smithi173 ceph-mon[20554]: pgmap v1110: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 33177/222120 objects degraded (14.937%); 9951/222120 objects misplaced (4.480%); 2.6 MiB/s, 42 objects/s recovering 2024-08-30T23:40:32.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:32.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:32.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:32.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:32.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:32.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:32.432 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:32.432 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:32.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:32.456 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:32.457 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:32.478 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:32.478 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:32.501 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:32.502 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:32.502 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:32.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:32.524 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:32.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:32.546 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:32.546 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:32.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:32.567 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:32.568 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:32.589 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:32.589 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:32.609 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:32.609 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:32.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:32.631 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:32.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:32.654 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:32.655 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:32.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:32.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:32.788 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:40:32.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:32.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:32.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:32.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:33.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:33.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/281920463' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:33.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1482095307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:33.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/281920463' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1482095307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/281920463' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1482095307' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:33.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:33.165 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:33.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:33.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:33.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:33.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:33.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:33.534 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:33.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:33.657 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:33.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:33.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/281920463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: osdmap e471: 12 total, 10 up, 12 in 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:33.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:33.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:33.859 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:33.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:33.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:33.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:33.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:33.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:33.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:33.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:33.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:33.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.861 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:33.862 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:33.862 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:33.862 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:33.862 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:33 smithi106 ceph-mon[16603]: pgmap v1112: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32817/222120 objects degraded (14.774%); 9231/222120 objects misplaced (4.156%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:40:33.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:33.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:33.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:34.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:34.027 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/281920463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: osdmap e471: 12 total, 10 up, 12 in 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:33 smithi136 ceph-mon[20207]: pgmap v1112: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32817/222120 objects degraded (14.774%); 9231/222120 objects misplaced (4.156%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:40:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/281920463' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: osdmap e471: 12 total, 10 up, 12 in 2024-08-30T23:40:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:40:34.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:33 smithi173 ceph-mon[20554]: pgmap v1112: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32817/222120 objects degraded (14.774%); 9231/222120 objects misplaced (4.156%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:40:34.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:34.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:34.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:34.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:34.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:34.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:34.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:34.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:34.574 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:34.574 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:40:34.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:34.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:34.751 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:40:34.752 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:40:34.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:40:34.752 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:40:34.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:103424 3.2 3:4caab458:::benchmark_data_smithi173_60780_object103423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 533.49061275999998, 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.221222537, 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.753 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 103424 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259623+0000", 2024-08-30T23:40:34.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259624+0000", 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0800000000000004e-07 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259682+0000", 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7364999999999997e-05 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259692+0000", 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0686000000000001e-05 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.259698+0000", 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.361e-06 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261702+0000", 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020041289999999999 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261778+0000", 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6148000000000003e-05 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.261872+0000", 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4598000000000005e-05 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.263845+0000", 2024-08-30T23:40:34.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001972419 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.264707+0000", 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086252099999999999 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480753+0000", 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.216045457 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480795+0000", 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2605000000000003e-05 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:31:41.480846+0000", 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0540000000000001e-05 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198168 3.2 3:46c77281:::benchmark_data_smithi173_60780_object94714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:40:34.759 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 421.78802008600002, 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22735555199999999, 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198168 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962216+0000", 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962217+0000", 2024-08-30T23:40:34.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2099999999999995e-07 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962221+0000", 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999999e-06 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962234+0000", 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2412000000000001e-05 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962241+0000", 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6789999999999999e-06 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962296+0000", 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5347999999999999e-05 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962411+0000", 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011498299999999999 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.962524+0000", 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011293700000000001 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.967040+0000", 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045164259999999996 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.968062+0000", 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010217270000000001 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189490+0000", 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22142835899999999 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189535+0000", 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4338999999999997e-05 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189572+0000", 2024-08-30T23:40:34.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6990999999999999e-05 2024-08-30T23:40:34.766 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.766 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.766 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.767 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:198178 3.2 3:47f13797:::benchmark_data_smithi173_60780_object94724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e298)", 2024-08-30T23:40:34.767 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:40:34.767 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 421.77723065800001, 2024-08-30T23:40:34.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21670980100000001, 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 198178 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:40:34.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973006+0000", 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973007+0000", 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973013+0000", 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1140000000000002e-06 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973023+0000", 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0516000000000001e-05 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973030+0000", 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8390000000000001e-06 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973072+0000", 2024-08-30T23:40:34.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1919000000000003e-05 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973158+0000", 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6148000000000003e-05 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.973234+0000", 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6165e-05 2024-08-30T23:40:34.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.976842+0000", 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036077290000000001 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:32.978153+0000", 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013109930000000001 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189617+0000", 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21146347300000001 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189674+0000", 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7082e-05 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:33.189715+0000", 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1656999999999998e-05 2024-08-30T23:40:34.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206885 3.5 e300/206 3:a4db87f4:::benchmark_data_smithi173_60780_object103431:head v 300'12859, mlcod=300'12855)", 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.70899291900002, 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23467711799999999, 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041243+0000", 2024-08-30T23:40:34.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041244+0000", 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.041249+0000", 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2360000000000004e-06 2024-08-30T23:40:34.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271945+0000", 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.230695859 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.271961+0000", 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5815999999999999e-05 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272088+0000", 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000126721 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.272127+0000", 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8872000000000003e-05 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275888+0000", 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037616350000000002 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.275920+0000", 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2255000000000002e-05 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206897 3.b 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:40:34.780 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.69315190999998, 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58358057900000004, 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206897 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057084+0000", 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057085+0000", 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3399999999999999e-07 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057088+0000", 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8609999999999998e-06 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057094+0000", 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5429999999999999e-06 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057098+0000", 2024-08-30T23:40:34.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9720000000000003e-06 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057149+0000", 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0683000000000002e-05 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057264+0000", 2024-08-30T23:40:34.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114772 2024-08-30T23:40:34.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:34.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:40:34.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.057387+0000", 2024-08-30T23:40:34.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012322799999999999 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.061158+0000", 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037710170000000002 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062218+0000", 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001059948 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640580+0000", 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57836202199999998 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640625+0000", 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5116e-05 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.640665+0000", 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9882999999999997e-05 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.789 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.15351.0:206901 3.7 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.687470652, 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38317995100000002, 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.15351", 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 206901 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062766+0000", 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062767+0000", 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2619999999999999e-06 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062771+0000", 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6779999999999999e-06 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062785+0000", 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3839000000000001e-05 2024-08-30T23:40:34.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.062793+0000", 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2130000000000002e-06 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063873+0000", 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001080632 2024-08-30T23:40:34.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063965+0000", 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1717000000000003e-05 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.064031+0000", 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5729999999999996e-05 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.067745+0000", 2024-08-30T23:40:34.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037140430000000002 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.071385+0000", 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036407620000000001 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445862+0000", 2024-08-30T23:40:34.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.37447608799999998 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445912+0000", 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0059999999999998e-05 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.445946+0000", 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3927000000000002e-05 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.15351.0:206900 3.5 e300/206 3:a700f8ce:::benchmark_data_smithi173_60780_object103446:head v 300'12860, mlcod=300'12856)", 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 410.68702938500002, 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36104319299999998, 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063207+0000", 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063208+0000", 2024-08-30T23:40:34.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2399999999999997e-07 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.063213+0000", 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0080000000000002e-06 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420406+0000", 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35719326299999998 2024-08-30T23:40:34.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420426+0000", 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0253000000000001e-05 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420597+0000", 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000170786 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.420630+0000", 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3578999999999998e-05 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424218+0000", 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035875490000000002 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:33:44.424250+0000", 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2030999999999999e-05 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.801 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62091 4.4 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 337.58456842099997, 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318614905, 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.802 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62091 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165670+0000", 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165713+0000", 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3368000000000002e-05 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165732+0000", 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9031000000000001e-05 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165740+0000", 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8050000000000007e-06 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165802+0000", 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2279999999999993e-05 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165946+0000", 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014352899999999999 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.166046+0000", 2024-08-30T23:40:34.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100046 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169317+0000", 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003271267 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169909+0000", 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059201200000000005 2024-08-30T23:40:34.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484178+0000", 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31426850299999998 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484234+0000", 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6678000000000002e-05 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484283+0000", 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8457e-05 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62105 4.4 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 337.57098408899998, 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34846845199999998, 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.809 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62105 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179254+0000", 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179297+0000", 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3108000000000001e-05 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179311+0000", 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.345e-05 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179316+0000", 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179371+0000", 2024-08-30T23:40:34.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5316999999999999e-05 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179508+0000", 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643600000000001 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179665+0000", 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015680499999999999 2024-08-30T23:40:34.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182071+0000", 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024068090000000002 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182226+0000", 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015503899999999999 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527606+0000", 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34537996300000001 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527669+0000", 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2966999999999995e-05 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527721+0000", 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1224000000000003e-05 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.815 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62174 4.4 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 337.22577230899998, 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57863658799999995, 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.816 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62174 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524466+0000", 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0600000000000002e-06 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524501+0000", 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4635999999999998e-05 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524521+0000", 2024-08-30T23:40:34.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0193000000000001e-05 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524531+0000", 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7780000000000002e-06 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524604+0000", 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3646000000000006e-05 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524763+0000", 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015852999999999999 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524940+0000", 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017771799999999999 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527733+0000", 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027928789999999998 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.528529+0000", 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000795487 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103030+0000", 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57450106899999998 2024-08-30T23:40:34.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103068+0000", 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8398000000000001e-05 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103100+0000", 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2193999999999997e-05 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.822 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62186 4.4 4:257aab52:::benchmark_data_smithi173_74914_object62185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 337.19666018800001, 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55186797300000001, 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62186 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553578+0000", 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553612+0000", 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3773999999999999e-05 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553627+0000", 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5410999999999999e-05 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553637+0000", 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0101e-05 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553709+0000", 2024-08-30T23:40:34.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2365000000000003e-05 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553885+0000", 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017522699999999999 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.554065+0000", 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000180101 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556530+0000", 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024650240000000001 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556570+0000", 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9975999999999998e-05 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105343+0000", 2024-08-30T23:40:34.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54877291299999997 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105399+0000", 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6648999999999999e-05 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105444+0000", 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4660000000000003e-05 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.829 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:93692 4.4 4:24d1e68e:::benchmark_data_smithi173_74914_object93691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 307.72645653799998, 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28134646200000002, 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.830 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 93692 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023780+0000", 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023781+0000", 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023840+0000", 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9305999999999998e-05 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023854+0000", 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3773e-05 2024-08-30T23:40:34.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023872+0000", 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7708000000000001e-05 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.023930+0000", 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8007999999999998e-05 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024050+0000", 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120209 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.024158+0000", 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010744299999999999 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026599+0000", 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024417430000000001 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.026641+0000", 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2008999999999997e-05 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305020+0000", 2024-08-30T23:40:34.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27837906600000001 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305079+0000", 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9080999999999999e-05 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305126+0000", 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6657999999999999e-05 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.836 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.25654.0:93692 4.4 e350/331 ondisk, result = 0)", 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 307.72318020099999, 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27807699200000002, 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.837 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027056+0000", 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027057+0000", 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.027060+0000", 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.01e-06 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304915+0000", 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27785516100000002 2024-08-30T23:40:34.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.304921+0000", 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1800000000000001e-06 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305007+0000", 2024-08-30T23:40:34.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5783000000000002e-05 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305015+0000", 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7659999999999998e-06 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:27.305133+0000", 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118472 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.841 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:107608 4.f e354/306 4:fc26249f:::benchmark_data_smithi173_74914_object9366:head v 354'6690, mlcod=354'6687)", 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 293.25242313000001, 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22329728099999999, 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:40:34.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497813+0000", 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497814+0000", 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3399999999999998e-07 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.497821+0000", 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8900000000000001e-06 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717759+0000", 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21993858999999999 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717767+0000", 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6000000000000001e-06 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717856+0000", 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8516000000000004e-05 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.717893+0000", 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7342999999999998e-05 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721075+0000", 2024-08-30T23:40:34.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0031817579999999998 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:35:41.721110+0000", 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5750000000000002e-05 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:168699 4.f 4:f9ac8f5d:::benchmark_data_smithi173_74914_object70457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 227.34264496500001, 2024-08-30T23:40:34.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23153700799999999, 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 168699 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407591+0000", 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407593+0000", 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.308e-06 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407598+0000", 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5899999999999998e-06 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407609+0000", 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0975000000000001e-05 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407616+0000", 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0910000000000004e-06 2024-08-30T23:40:34.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407664+0000", 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7513999999999999e-05 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407761+0000", 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6817999999999994e-05 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-08-30T23:40:34.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.407838+0000", 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6971000000000006e-05 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.411723+0000", 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038854430000000001 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639022+0000", 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.227298575 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639056+0000", 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4623999999999999e-05 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.639128+0000", 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2099000000000002e-05 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.852 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:193335 4.f 4:f0f0a8f4:::benchmark_data_smithi173_74914_object95093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 200.121663864, 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17749384200000001, 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193335 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628572+0000", 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628573+0000", 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.119e-06 2024-08-30T23:40:34.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628580+0000", 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1569999999999997e-06 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628592+0000", 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2866e-05 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628599+0000", 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4030000000000004e-06 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628655+0000", 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6322e-05 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628791+0000", 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013567700000000001 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 9", 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.628934+0000", 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014291299999999999 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.633290+0000", 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043565920000000003 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.805971+0000", 2024-08-30T23:40:34.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17268071500000001 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.806014+0000", 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3257999999999998e-05 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:14.806066+0000", 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1820000000000002e-05 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:196465 4.4 4:242a1b19:::benchmark_data_smithi173_74914_object98223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.73412563599999, 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45786220799999999, 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196465 2024-08-30T23:40:34.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016114+0000", 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999999e-06 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016121+0000", 2024-08-30T23:40:34.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9240000000000001e-06 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016125+0000", 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.704e-06 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016162+0000", 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6528e-05 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016238+0000", 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6246000000000001e-05 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016310+0000", 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2475e-05 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020389+0000", 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040785120000000003 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020517+0000", 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012816600000000001 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473901+0000", 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45338352300000001 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473940+0000", 2024-08-30T23:40:34.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9224000000000003e-05 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473973+0000", 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3034000000000003e-05 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:15231 5.1 5:8ac1186e:::benchmark_data_smithi173_87639_object15230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:50.725069+0000", 2024-08-30T23:40:34.865 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.025167221, 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214480904, 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 15231 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725069+0000", 2024-08-30T23:40:34.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725069+0000", 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725070+0000", 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8299999999999995e-07 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725160+0000", 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9543000000000006e-05 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725170+0000", 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0767e-05 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725178+0000", 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2749999999999998e-06 2024-08-30T23:40:34.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725228+0000", 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0067e-05 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725318+0000", 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9888000000000004e-05 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:40:34.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.725435+0000", 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011709699999999999 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.726054+0000", 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00061947499999999997 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.726412+0000", 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035780399999999998 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.939440+0000", 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21302797600000001 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.939497+0000", 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7380999999999998e-05 2024-08-30T23:40:34.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.939550+0000", 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2648000000000001e-05 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:17923 5.1 5:89278b47:::benchmark_data_smithi173_87639_object17922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e403)", 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:52.420774+0000", 2024-08-30T23:40:34.872 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 162.329462057, 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17294684799999999, 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 17923 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.873 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420774+0000", 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420774+0000", 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420775+0000", 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7899999999999997e-07 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420841+0000", 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6388000000000002e-05 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420852+0000", 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0192e-05 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.420857+0000", 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0830000000000003e-06 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.421298+0000", 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00044180100000000002 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.421420+0000", 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012193699999999999 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.421500+0000", 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9931000000000004e-05 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.422068+0000", 2024-08-30T23:40:34.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056767099999999995 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.422613+0000", 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00054493900000000004 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.593659+0000", 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17104583400000001 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.593691+0000", 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2496000000000002e-05 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:52.593721+0000", 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9697e-05 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.879 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "rep_scrubmap(5.1 e389 from shard 9)", 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:25.986087+0000", 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 8.7641492529999994, 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.935597817, 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:40:34.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986087+0000", 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986087+0000", 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986088+0000", 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5239999999999999e-06 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986137+0000", 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8992000000000001e-05 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921482+0000", 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93534420799999995 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921488+0000", 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7340000000000002e-06 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921558+0000", 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9606999999999999e-05 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921568+0000", 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7340000000000003e-06 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:40:34.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921685+0000", 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117018 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:40:34.884 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:40:34.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:34.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:35.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:35.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:35.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:35.161 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:35.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:35.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:35.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:35.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:35.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:35.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:35.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:40:35.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:35.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:35.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:35.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:35.787 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:35.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:35.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:35.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:36.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:36.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:35 smithi173 ceph-mon[20554]: osdmap e472: 12 total, 10 up, 12 in 2024-08-30T23:40:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:35 smithi173 ceph-mon[20554]: pgmap v1114: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32817/222120 objects degraded (14.774%); 9231/222120 objects misplaced (4.156%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:40:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 32817/222120 objects degraded (14.774%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:35 smithi106 ceph-mon[16603]: osdmap e472: 12 total, 10 up, 12 in 2024-08-30T23:40:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:35 smithi106 ceph-mon[16603]: pgmap v1114: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32817/222120 objects degraded (14.774%); 9231/222120 objects misplaced (4.156%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:40:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 32817/222120 objects degraded (14.774%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:36.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:36.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:36.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:40:36.203 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:36.225 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:35 smithi136 ceph-mon[20207]: osdmap e472: 12 total, 10 up, 12 in 2024-08-30T23:40:36.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:35 smithi136 ceph-mon[20207]: pgmap v1114: 27 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32817/222120 objects degraded (14.774%); 9231/222120 objects misplaced (4.156%); 5.5 MiB/s, 87 objects/s recovering 2024-08-30T23:40:36.226 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 32817/222120 objects degraded (14.774%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:36.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:36.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:36.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:36.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:36.446 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:36.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.538 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:36.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:36.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:36.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:36.704 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:36.737 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:36.737 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:36.737 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:36.738 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:36.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:36.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:36.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:36.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:36.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:36 smithi136 ceph-mon[20207]: osdmap e473: 12 total, 10 up, 12 in 2024-08-30T23:40:36.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:36.821 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:36.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:36.907 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:36.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:36.942 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:40:36.942 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:40:36.942 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:40:36.942 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:40:37.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:37.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:36 smithi173 ceph-mon[20554]: osdmap e473: 12 total, 10 up, 12 in 2024-08-30T23:40:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:36 smithi106 ceph-mon[16603]: osdmap e473: 12 total, 10 up, 12 in 2024-08-30T23:40:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:37.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:37.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:37.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:37.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:37.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:37.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:37.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:37.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:37.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:40:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:37.581 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:37.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:40:37.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:37.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:37.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:40:37.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:37.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:37.817 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:37.849 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:37.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:37.906 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:37.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:37.970 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:37 smithi106 ceph-mon[16603]: osdmap e474: 12 total, 10 up, 12 in 2024-08-30T23:40:37.970 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:37 smithi106 ceph-mon[16603]: pgmap v1117: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:40:38.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:38.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:37 smithi173 ceph-mon[20554]: osdmap e474: 12 total, 10 up, 12 in 2024-08-30T23:40:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:37 smithi173 ceph-mon[20554]: pgmap v1117: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:40:38.079 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:37 smithi136 ceph-mon[20207]: osdmap e474: 12 total, 10 up, 12 in 2024-08-30T23:40:38.079 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:37 smithi136 ceph-mon[20207]: pgmap v1117: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 5.0 MiB/s, 79 objects/s recovering 2024-08-30T23:40:38.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:38.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:38.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:38.285 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:40:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:40:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:40:38.285 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:40:38.285 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:40:38.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:38.307 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:38.307 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:38.307 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:38.307 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:38.307 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:38.308 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:38.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:38.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:38.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:38.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:38.560 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:38.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:38.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:38.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:38.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:38.819 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:40:38.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:38.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:38.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:38.921 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1491650647' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:38.942 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1491650647' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:38.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:40:38.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:38.970 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:38.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-30T23:40:39.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:39.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1491650647' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:39.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:39.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:39.190 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:39.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:39.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:39.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:39.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:39.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:39.533 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:39.583 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:206897 3.b e300/206 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head v 300'12978, mlcod=300'12975)", 2024-08-30T23:40:39.626 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.057579+0000", 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 415.567228301, 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.58266937699999999, 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057579+0000", 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057579+0000", 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057580+0000", 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.99e-07 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057584+0000", 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4100000000000001e-06 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636116+0000", 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.57853164000000001 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636133+0000", 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.668e-05 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636261+0000", 2024-08-30T23:40:39.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012832100000000001 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636303+0000", 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1845999999999999e-05 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.640219+0000", 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039165789999999999 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.640249+0000", 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9201999999999999e-05 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:1794 4.e 4:768be7b6:::benchmark_data_smithi173_74914_object1793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:02.513298+0000", 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 397.11150910800001, 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21590541399999999, 2024-08-30T23:40:39.631 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 1794 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513298+0000", 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513298+0000", 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513299+0000", 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5899999999999995e-07 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513395+0000", 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5822999999999994e-05 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513407+0000", 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1646e-05 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513414+0000", 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4880000000000001e-06 2024-08-30T23:40:39.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513450+0000", 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5716000000000002e-05 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513519+0000", 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9332999999999995e-05 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,10", 2024-08-30T23:40:39.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513598+0000", 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8918999999999994e-05 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.513931+0000", 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033251299999999999 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.514357+0000", 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042641199999999998 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.729107+0000", 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.214749937 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.729164+0000", 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6567999999999998e-05 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:02.729204+0000", 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0299999999999997e-05 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:31053 4.c 4:3c66407d:::benchmark_data_smithi173_74914_object31052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:20.682766+0000", 2024-08-30T23:40:39.638 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 378.94204141, 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60411518900000005, 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 31053 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682766+0000", 2024-08-30T23:40:39.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682766+0000", 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682767+0000", 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682833+0000", 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6268999999999999e-05 2024-08-30T23:40:39.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682849+0000", 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5753e-05 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682857+0000", 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.106e-06 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.683370+0000", 2024-08-30T23:40:39.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051270999999999999 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.683454+0000", 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3888999999999996e-05 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.683552+0000", 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8455999999999994e-05 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684220+0000", 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066772100000000003 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.756181+0000", 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071960969 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286726+0000", 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53054503799999997 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286800+0000", 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4201000000000003e-05 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286881+0000", 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1201999999999999e-05 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:31053 4.c e317/306 ondisk, result = 0)", 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:20.684501+0000", 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 378.94030654900001, 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60239176800000005, 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684501+0000", 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684501+0000", 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684502+0000", 2024-08-30T23:40:39.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5700000000000002e-07 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684505+0000", 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.422e-06 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286533+0000", 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60202851700000004 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286549+0000", 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5939000000000001e-05 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286703+0000", 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015350699999999999 2024-08-30T23:40:39.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286718+0000", 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4832e-05 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286893+0000", 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017499400000000001 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49449 4.c 4:30eeb856:::benchmark_data_smithi173_74914_object49448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.563639+0000", 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 359.06116889800001, 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.225408365, 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49449 2024-08-30T23:40:39.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563639+0000", 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563639+0000", 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563642+0000", 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7570000000000002e-06 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563676+0000", 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3778000000000001e-05 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563696+0000", 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9587999999999999e-05 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563705+0000", 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8419999999999994e-06 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563758+0000", 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3327e-05 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563878+0000", 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120111 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563988+0000", 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109616 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.565407+0000", 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014193630000000001 2024-08-30T23:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.719803+0000", 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15439636400000001 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.788921+0000", 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.069117888000000002 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.788998+0000", 2024-08-30T23:40:39.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6662999999999994e-05 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.789047+0000", 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9067999999999998e-05 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:39.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:62196 4.9 e337/327 4:91ca1c4e:::benchmark_data_smithi173_74914_object62195:head v 337'3860, mlcod=337'3856)", 2024-08-30T23:40:39.659 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.773569+0000", 2024-08-30T23:40:39.659 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 341.85123866800001, 2024-08-30T23:40:39.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310100351, 2024-08-30T23:40:39.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773569+0000", 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773569+0000", 2024-08-30T23:40:39.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773570+0000", 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773594+0000", 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4323000000000001e-05 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.081936+0000", 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30834138 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.081942+0000", 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.081994+0000", 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1647999999999997e-05 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.082008+0000", 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4134000000000001e-05 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083640+0000", 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016318050000000001 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083669+0000", 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9275999999999999e-05 2024-08-30T23:40:39.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:71384 4.e 4:706b20d0:::benchmark_data_smithi173_74914_object71383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:06.471925+0000", 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 333.152882256, 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21602839500000001, 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 71384 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.471925+0000", 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.471925+0000", 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.471927+0000", 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9570000000000001e-06 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.471980+0000", 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2394e-05 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.472001+0000", 2024-08-30T23:40:39.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1260999999999999e-05 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.472010+0000", 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2450000000000004e-06 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.472068+0000", 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7540000000000001e-05 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.472184+0000", 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011599000000000001 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.472301+0000", 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117075 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.474348+0000", 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020473980000000002 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.687064+0000", 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21271557899999999 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.687837+0000", 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077367199999999999 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.687894+0000", 2024-08-30T23:40:39.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6539000000000002e-05 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:06.687954+0000", 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9744999999999998e-05 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:122182 4.e 4:7aa6f75c:::benchmark_data_smithi173_74914_object23940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-08-30T23:40:39.671 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:56.818707+0000", 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 282.806100587, 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21992162000000001, 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122182 2024-08-30T23:40:39.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818707+0000", 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818707+0000", 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818708+0000", 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818712+0000", 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5549999999999996e-06 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818725+0000", 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2499e-05 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818731+0000", 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6559999999999997e-06 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818770+0000", 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9826000000000001e-05 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818852+0000", 2024-08-30T23:40:39.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1679000000000002e-05 2024-08-30T23:40:39.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:40:39.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818919+0000", 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7219999999999997e-05 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.821868+0000", 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029483970000000002 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.822176+0000", 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000307903 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:57.038559+0000", 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21638298 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:57.038599+0000", 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0735000000000002e-05 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:57.038629+0000", 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9306e-05 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:188382 4.c 4:38db3f02:::benchmark_data_smithi173_74914_object90140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:09.185479+0000", 2024-08-30T23:40:39.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.439328889, 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22511474200000001, 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 188382 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185479+0000", 2024-08-30T23:40:39.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185479+0000", 2024-08-30T23:40:39.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185480+0000", 2024-08-30T23:40:39.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3659999999999999e-06 2024-08-30T23:40:39.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185485+0000", 2024-08-30T23:40:39.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3190000000000002e-06 2024-08-30T23:40:39.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185499+0000", 2024-08-30T23:40:39.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4055e-05 2024-08-30T23:40:39.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185507+0000", 2024-08-30T23:40:39.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2590000000000003e-06 2024-08-30T23:40:39.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185553+0000", 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5943999999999999e-05 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185701+0000", 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148658 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185837+0000", 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000135313 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.190396+0000", 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045590400000000003 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.190998+0000", 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060265399999999999 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.410522+0000", 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21952385399999999 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.410554+0000", 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1832000000000003e-05 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.410593+0000", 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9447999999999999e-05 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:188386 4.e 4:747907c3:::benchmark_data_smithi173_74914_object90144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.435087985, 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21908520400000001, 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 188386 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4499999999999995e-07 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189725+0000", 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7079999999999996e-06 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189741+0000", 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5769000000000001e-05 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189750+0000", 2024-08-30T23:40:39.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8769999999999999e-06 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189812+0000", 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249000000000007e-05 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189929+0000", 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011684 2024-08-30T23:40:39.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.190053+0000", 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124556 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.193881+0000", 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038277680000000001 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.197367+0000", 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034854560000000001 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.408731+0000", 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21136460600000001 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.408772+0000", 2024-08-30T23:40:39.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0602999999999997e-05 2024-08-30T23:40:39.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.408805+0000", 2024-08-30T23:40:39.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2827000000000003e-05 2024-08-30T23:40:39.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:28981 5.c e406/389 ondisk, result = 0)", 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 160.18268693900001, 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21804868699999999, 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.15e-07 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442123+0000", 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.39e-06 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.659052+0000", 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21692828 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.659059+0000", 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.827e-06 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.660059+0000", 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010008630000000001 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.660068+0000", 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2430000000000005e-06 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.660169+0000", 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010166900000000001 2024-08-30T23:40:39.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:44553 5.c 5:323e8416:::benchmark_data_smithi173_87639_object44552:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 147.17624115999999, 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83468573899999998, 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44553 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:40:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:40:39.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448567+0000", 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.093e-06 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448624+0000", 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6285000000000001e-05 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448633+0000", 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9830000000000002e-06 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448639+0000", 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.725e-06 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448672+0000", 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2815999999999999e-05 2024-08-30T23:40:39.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448753+0000", 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1149999999999994e-05 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:40:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448841+0000", 2024-08-30T23:40:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7267000000000003e-05 2024-08-30T23:40:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.449924+0000", 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001083533 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450381+0000", 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045717400000000002 2024-08-30T23:40:39.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283161+0000", 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83277994799999999 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283216+0000", 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5067e-05 2024-08-30T23:40:39.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283252+0000", 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5698000000000003e-05 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:44553 5.c e410/389 ondisk, result = 0)", 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:40:39.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 147.17480642999999, 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83325748899999996, 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450002+0000", 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450005+0000", 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.58e-06 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283028+0000", 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.833022924 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283038+0000", 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.713e-06 2024-08-30T23:40:39.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283141+0000", 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102903 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283154+0000", 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.289e-05 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283259+0000", 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010485899999999999 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:45508 5.0 5:052f1fbc:::benchmark_data_smithi173_87639_object45507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:21.430861+0000", 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 138.19394622199999, 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.217851873, 2024-08-30T23:40:39.719 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 45508 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430861+0000", 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430861+0000", 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430863+0000", 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.81e-06 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430905+0000", 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1909000000000001e-05 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430922+0000", 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6750000000000001e-05 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430929+0000", 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5259999999999996e-06 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430984+0000", 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4923000000000003e-05 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.431138+0000", 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000154244 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.431233+0000", 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4503e-05 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.432903+0000", 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016699810000000001 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.445800+0000", 2024-08-30T23:40:39.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012896960000000001 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.648509+0000", 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.202709108 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.648576+0000", 2024-08-30T23:40:39.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7254000000000004e-05 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.648713+0000", 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013690499999999999 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58901 5.c 5:343594e4:::benchmark_data_smithi173_87639_object58900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:31.825342+0000", 2024-08-30T23:40:39.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.799465262, 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.249786806, 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58901 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825342+0000", 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825342+0000", 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825343+0000", 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825411+0000", 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7438e-05 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825422+0000", 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1089e-05 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825430+0000", 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8949999999999998e-06 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825507+0000", 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7114999999999996e-05 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825663+0000", 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015647999999999999 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825793+0000", 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129617 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.827641+0000", 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018479060000000001 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.833079+0000", 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054384139999999999 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.075071+0000", 2024-08-30T23:40:39.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24199152600000001 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.075103+0000", 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2162000000000001e-05 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.075129+0000", 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5998000000000001e-05 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.733 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58921 5.c 5:32b6f731:::benchmark_data_smithi173_87639_object58920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:31.862617+0000", 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.762190559, 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21638538500000001, 2024-08-30T23:40:39.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58921 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862617+0000", 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862617+0000", 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862618+0000", 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.347e-06 2024-08-30T23:40:39.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862675+0000", 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6928000000000001e-05 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862689+0000", 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3545e-05 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862697+0000", 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862744+0000", 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7339999999999997e-05 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862838+0000", 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3870999999999996e-05 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.862924+0000", 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5885999999999997e-05 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.864589+0000", 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001665314 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.874784+0000", 2024-08-30T23:40:39.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010195197 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.078932+0000", 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20414813900000001 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.078968+0000", 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5451999999999999e-05 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.079002+0000", 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4551000000000002e-05 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58987 5.0 5.b63d9900 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.51001366600001, 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99942602000000003, 2024-08-30T23:40:39.741 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58987 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114795+0000", 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114853+0000", 2024-08-30T23:40:39.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7525000000000002e-05 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114863+0000", 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9899999999999992e-06 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114871+0000", 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5709999999999998e-06 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114907+0000", 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5661000000000003e-05 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114912+0000", 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2809999999999999e-06 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114202+0000", 2024-08-30T23:40:39.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99928982799999999 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114220+0000", 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7734e-05 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58996 5.0 5.b98b8b60 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:40:39.746 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.497605917, 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.987150694, 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58996 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127203+0000", 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.347e-06 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127240+0000", 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7403000000000001e-05 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127253+0000", 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2870999999999999e-05 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127260+0000", 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8800000000000002e-06 2024-08-30T23:40:39.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127312+0000", 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1431000000000003e-05 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:40:39.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127318+0000", 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6229999999999999e-06 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114337+0000", 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.987019172 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114352+0000", 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4966999999999999e-05 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.752 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59005 5.0 5.61e98650 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.467117694, 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95670884499999997, 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59005 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157691+0000", 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181e-06 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157717+0000", 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5612000000000001e-05 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157729+0000", 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-08-30T23:40:39.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157734+0000", 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1800000000000004e-06 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157780+0000", 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5779999999999999e-05 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:40:39.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157787+0000", 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.815e-06 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114390+0000", 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95660253100000003 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114399+0000", 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1249999999999999e-06 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59006 5.0 5.aa303940 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.406510298, 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89614382599999998, 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:40:39.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59006 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218298+0000", 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218324+0000", 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5857e-05 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218335+0000", 2024-08-30T23:40:39.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0716999999999999e-05 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218340+0000", 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9490000000000004e-06 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218423+0000", 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3065000000000005e-05 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218430+0000", 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1400000000000002e-06 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114428+0000", 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89599851500000005 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114441+0000", 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2727e-05 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.762 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:39.763 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:39.763 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:40:39.784 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:39.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:39.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:39.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:39.967 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:39 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:39.967 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1491650647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:39.967 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:39 smithi106 ceph-mon[16603]: osdmap e475: 12 total, 10 up, 12 in 2024-08-30T23:40:39.967 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:39 smithi106 ceph-mon[16603]: pgmap v1119: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 242 KiB/s, 3 objects/s recovering 2024-08-30T23:40:39.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:39 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:39.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1491650647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:39.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:39 smithi136 ceph-mon[20207]: osdmap e475: 12 total, 10 up, 12 in 2024-08-30T23:40:39.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:39 smithi136 ceph-mon[20207]: pgmap v1119: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 242 KiB/s, 3 objects/s recovering 2024-08-30T23:40:40.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:40.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:39 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1491650647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:39 smithi173 ceph-mon[20554]: osdmap e475: 12 total, 10 up, 12 in 2024-08-30T23:40:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:39 smithi173 ceph-mon[20554]: pgmap v1119: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 242 KiB/s, 3 objects/s recovering 2024-08-30T23:40:40.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:40.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:40.184 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:40.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:40.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:40.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:40.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:40.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:40:40.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:40.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:40.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:40.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:40.710 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:40.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:40.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:40.845 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:40.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-30T23:40:40.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:40.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:40.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:41.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:41.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:40 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 32800/222120 objects degraded (14.767%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:40 smithi173 ceph-mon[20554]: osdmap e476: 12 total, 10 up, 12 in 2024-08-30T23:40:41.085 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:40 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 32800/222120 objects degraded (14.767%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:41.085 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:40 smithi136 ceph-mon[20207]: osdmap e476: 12 total, 10 up, 12 in 2024-08-30T23:40:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:40 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 32800/222120 objects degraded (14.767%), 6 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:40:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:40 smithi106 ceph-mon[16603]: osdmap e476: 12 total, 10 up, 12 in 2024-08-30T23:40:41.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:41.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:41.185 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:40:41.185 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:40:41.185 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:40:41.185 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:40:41.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:41.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:41.308 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:41.308 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:41.309 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:41.325 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:41.330 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:41.331 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:41.352 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:41.352 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:41.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:41.393 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:41.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:41.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:41.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:40:41.576 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:41.577 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:41.577 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:41.601 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:41.604 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:41.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:41.631 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:41.631 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:41.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:41.652 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:41.652 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:41.673 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:41.673 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:41.694 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:41.694 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:41.715 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:41.716 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:41.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:41.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:41.816 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:41.816 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:41.817 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:41.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:40:41.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:41.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:41.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:42.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:42.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:42.084 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2694592482' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:41 smithi106 ceph-mon[16603]: pgmap v1121: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 201 KiB/s, 3 objects/s recovering 2024-08-30T23:40:42.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:42.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:42.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2694592482' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:42.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:41 smithi136 ceph-mon[20207]: pgmap v1121: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 201 KiB/s, 3 objects/s recovering 2024-08-30T23:40:42.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:42.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2694592482' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:41 smithi173 ceph-mon[20554]: pgmap v1121: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%); 201 KiB/s, 3 objects/s recovering 2024-08-30T23:40:42.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:42.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:42.464 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:42.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:42.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:42.542 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:42.542 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:42.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:42.651 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:42.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:42.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.655 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:42.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:42.777 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:42.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:42.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:42.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:43.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:43.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:43.064 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:43.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-30T23:40:43.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:43.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:43.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:40:43.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:43.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:43.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:43.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:43.449 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:43.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:43.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:43.589 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:43.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:43.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:43.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:43.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.775 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:43.826 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:43 smithi136 ceph-mon[20207]: pgmap v1122: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:43 smithi106 ceph-mon[16603]: pgmap v1122: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:43.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:43.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:43.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:43.925 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:43.925 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:44.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:44.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:44.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:43 smithi173 ceph-mon[20554]: pgmap v1122: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:44.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:44.146 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:44.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:44.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:44.322 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:44.323 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:44.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:44.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:44.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:40:44.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.533 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:44.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:44.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:40:44.625 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:40:44.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:44.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:44.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:44.780 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:44.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3400820377' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:40:44.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:44.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:44.914 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:45.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3400820377' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:40:45.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:45.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:45.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-30T23:40:45.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:45.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3400820377' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:40:45.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:45.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:45.219 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:45.219 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:45.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:45.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.293 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:45.355 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:45.366 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:45.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:45.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:45.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:45.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:45.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:45.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:45 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:45.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:45 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:45.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:45 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:45.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3400820377' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:40:45.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:45 smithi106 ceph-mon[16603]: osdmap e477: 12 total, 10 up, 12 in 2024-08-30T23:40:45.731 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:45 smithi106 ceph-mon[16603]: pgmap v1124: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:45.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:45.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:45.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:40:45.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:45.912 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:45.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:45.918 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:45 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:45.918 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:45 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:45.918 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:45 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:45.918 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3400820377' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:40:45.918 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:45 smithi136 ceph-mon[20207]: osdmap e477: 12 total, 10 up, 12 in 2024-08-30T23:40:45.918 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:45 smithi136 ceph-mon[20207]: pgmap v1124: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:45.942 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:45.942 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:45.943 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:45.964 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:45.964 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:45.984 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:45.984 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:46.006 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:46.006 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:46.006 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:46.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:46.027 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:46.027 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:46.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:46.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:46.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:46.068 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:46.068 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:46.069 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:45 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:46.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:45 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:45 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3400820377' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:40:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:45 smithi173 ceph-mon[20554]: osdmap e477: 12 total, 10 up, 12 in 2024-08-30T23:40:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:45 smithi173 ceph-mon[20554]: pgmap v1124: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:46.090 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:46.090 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:46.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:46.160 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:46.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:46.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:46.313 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:46.313 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:46.333 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:46.334 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:46.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:46.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:46.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:46.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:46.600 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:40:46.600 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:40:46.600 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:40:46.600 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:40:46.601 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:40:46.630 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:46.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:46.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:46.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:46.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:46.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:46.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:47.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:47.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.025 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:47.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:46 smithi106 ceph-mon[16603]: 5.9 scrub starts 2024-08-30T23:40:47.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:46 smithi106 ceph-mon[16603]: 5.a scrub starts 2024-08-30T23:40:47.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:46 smithi106 ceph-mon[16603]: osdmap e478: 12 total, 10 up, 12 in 2024-08-30T23:40:47.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2865409561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:47.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:46 smithi106 ceph-mon[16603]: 5.a scrub ok 2024-08-30T23:40:47.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:46 smithi106 ceph-mon[16603]: 5.9 scrub ok 2024-08-30T23:40:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:47.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:47.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:47.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:47.196 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:46 smithi136 ceph-mon[20207]: 5.9 scrub starts 2024-08-30T23:40:47.196 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:46 smithi136 ceph-mon[20207]: 5.a scrub starts 2024-08-30T23:40:47.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:46 smithi136 ceph-mon[20207]: osdmap e478: 12 total, 10 up, 12 in 2024-08-30T23:40:47.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2865409561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:47.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:46 smithi136 ceph-mon[20207]: 5.a scrub ok 2024-08-30T23:40:47.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:46 smithi136 ceph-mon[20207]: 5.9 scrub ok 2024-08-30T23:40:47.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:40:47.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:47.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:47.307 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:47.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:47.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:47.308 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:47.308 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:47.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:46 smithi173 ceph-mon[20554]: 5.9 scrub starts 2024-08-30T23:40:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:46 smithi173 ceph-mon[20554]: 5.a scrub starts 2024-08-30T23:40:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:46 smithi173 ceph-mon[20554]: osdmap e478: 12 total, 10 up, 12 in 2024-08-30T23:40:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2865409561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:46 smithi173 ceph-mon[20554]: 5.a scrub ok 2024-08-30T23:40:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:46 smithi173 ceph-mon[20554]: 5.9 scrub ok 2024-08-30T23:40:47.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:47.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:47.468 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:40:47.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:47.536 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:47.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:47.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:47.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:47.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:47.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:47.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:47.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:47.959 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:40:47.959 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:40:47.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:40:47.959 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:40:47.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.15351.0:74450 3.d 3:b854981d:::benchmark_data_smithi173_60780_object74449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e243)", 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 575.99176137500001, 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29260630999999998, 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:47.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.15351", 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74450 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965931+0000", 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965932+0000", 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.71e-07 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.965987+0000", 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5448000000000002e-05 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966002+0000", 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4419999999999999e-05 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.966007+0000", 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0119999999999996e-06 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:47.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967853+0000", 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001846547 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.967977+0000", 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123404 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.968112+0000", 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013568200000000001 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970457+0000", 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023443460000000002 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.013531+0000", 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.043074440999999998 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258300+0000", 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.244768449 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258497+0000", 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000197209 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258537+0000", 2024-08-30T23:40:47.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0481000000000001e-05 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.15351.0:74450 3.d e243/206 ondisk, result = 0)", 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 575.98744745299996, 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28829785299999999, 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970245+0000", 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970246+0000", 2024-08-30T23:40:47.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:11.970250+0000", 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0330000000000002e-06 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258240+0000", 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28799039900000001 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258244+0000", 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1309999999999999e-06 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258287+0000", 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2347e-05 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258294+0000", 2024-08-30T23:40:47.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2250000000000003e-06 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:31:12.258543+0000", 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000249001 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61137 4.0 4:05184379:::benchmark_data_smithi173_74914_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 353.54462612899999, 2024-08-30T23:40:47.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0987384589999998, 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61137 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413067+0000", 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2640000000000001e-06 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413084+0000", 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6407999999999999e-05 2024-08-30T23:40:47.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413095+0000", 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277999999999999e-05 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413117+0000", 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1498e-05 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:47.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413168+0000", 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1266e-05 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413265+0000", 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6620000000000007e-05 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413359+0000", 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4003000000000001e-05 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415078+0000", 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719478 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415468+0000", 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038999199999999998 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511679+0000", 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0962105000000002 2024-08-30T23:40:47.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511758+0000", 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9426999999999997e-05 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511805+0000", 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6724999999999997e-05 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61138 4.0 4:0601b2bd:::benchmark_data_smithi173_74914_object61137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 352.80397137, 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582538660000001, 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61138 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153722+0000", 2024-08-30T23:40:47.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-06 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153826+0000", 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103683 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153839+0000", 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3169e-05 2024-08-30T23:40:47.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153863+0000", 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4326000000000001e-05 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153921+0000", 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7283e-05 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154051+0000", 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012986399999999999 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154154+0000", 2024-08-30T23:40:47.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103188 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156034+0000", 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018801009999999999 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156113+0000", 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8807000000000006e-05 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511881+0000", 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3557679949999999 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511932+0000", 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1134999999999997e-05 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:47.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511975+0000", 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2973999999999999e-05 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:62193 4.8 e337/329 4:198a939b:::benchmark_data_smithi173_74914_object62192:head v 337'3972, mlcod=337'3968)", 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 350.35256697400001, 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.349666123, 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:47.985 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605126+0000", 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.13e-06 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605148+0000", 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1759e-05 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952851+0000", 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34770309199999999 2024-08-30T23:40:47.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952856+0000", 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1220000000000003e-06 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952891+0000", 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4292999999999998e-05 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:47.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952908+0000", 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7277000000000001e-05 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954758+0000", 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018501889999999999 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954791+0000", 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3260999999999999e-05 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.990 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:47 smithi106 ceph-mon[16603]: pgmap v1126: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:107569 4.0 4:06be00e7:::benchmark_data_smithi173_74914_object9327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-08-30T23:40:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:41.457255+0000", 2024-08-30T23:40:47.991 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 306.50043762000001, 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23871579300000001, 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 107569 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457255+0000", 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457255+0000", 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457256+0000", 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.173e-06 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457260+0000", 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5199999999999999e-06 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457274+0000", 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3309000000000001e-05 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:47.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457281+0000", 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.798e-06 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457331+0000", 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0538000000000003e-05 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457461+0000", 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013044099999999999 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.457579+0000", 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011773300000000001 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.460634+0000", 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030552209999999999 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.510843+0000", 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050208424000000001 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695905+0000", 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18506230500000001 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695942+0000", 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7061999999999999e-05 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695971+0000", 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8269e-05 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:107569 4.0 e354/329 ondisk, result = 0)", 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 306.49610154599998, 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.234384061, 2024-08-30T23:40:47.998 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:40:47.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461591+0000", 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000003e-07 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.461596+0000", 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695857+0000", 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23426132899999999 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695861+0000", 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4880000000000001e-06 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695895+0000", 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3485999999999998e-05 2024-08-30T23:40:48.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695901+0000", 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1410000000000002e-06 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:41.695975+0000", 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3751000000000006e-05 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:143768 4.3 4:c93e37ef:::benchmark_data_smithi173_74914_object45526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 268.52607301299997, 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37550649899999999, 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 143768 2024-08-30T23:40:48.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431621+0000", 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431627+0000", 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431639+0000", 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2404e-05 2024-08-30T23:40:48.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431646+0000", 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431724+0000", 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7655000000000001e-05 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431814+0000", 2024-08-30T23:40:48.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0143999999999996e-05 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431885+0000", 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1421999999999995e-05 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.435174+0000", 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003288394 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.435587+0000", 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041313500000000002 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807016+0000", 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37142968700000001 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807076+0000", 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9561000000000001e-05 2024-08-30T23:40:48.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807126+0000", 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9864999999999998e-05 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:196473 4.0 4:0415f716:::benchmark_data_smithi173_74914_object98231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 209.93343172900001, 2024-08-30T23:40:48.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49994438299999999, 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 196473 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024266+0000", 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1779999999999996e-06 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024277+0000", 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1567e-05 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024284+0000", 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4389999999999997e-06 2024-08-30T23:40:48.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024336+0000", 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2145000000000003e-05 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024443+0000", 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107633 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024556+0000", 2024-08-30T23:40:48.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011263 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029319+0000", 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047626949999999999 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029471+0000", 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015204800000000001 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524119+0000", 2024-08-30T23:40:48.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49464797199999999 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524156+0000", 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7716000000000003e-05 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524205+0000", 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8591000000000003e-05 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:196478 4.a e381/369 4:550417b9:::benchmark_data_smithi173_74914_object98236:head v 381'12132, mlcod=381'12130)", 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 209.928336946, 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.406346965, 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029356+0000", 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5400000000000001e-07 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029361+0000", 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6739999999999996e-06 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431719+0000", 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.402358571 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431734+0000", 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4963000000000001e-05 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431842+0000", 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107765 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431888+0000", 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6269e-05 2024-08-30T23:40:48.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.435671+0000", 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037823639999999999 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.435702+0000", 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1705000000000002e-05 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.26173.0:48252 5.6 e415/389 5:6df5d833:::benchmark_data_smithi173_87639_object48251:head v 415'2973, mlcod=415'2969)", 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:23.505066+0000", 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.45262681, 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23519159100000001, 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.505066+0000", 2024-08-30T23:40:48.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.505066+0000", 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.505066+0000", 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6600000000000005e-07 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.505089+0000", 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2684000000000002e-05 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.738637+0000", 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.233548318 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.738643+0000", 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5509999999999999e-06 2024-08-30T23:40:48.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.738693+0000", 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9633999999999999e-05 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.738713+0000", 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0290000000000001e-05 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.740219+0000", 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015057880000000001 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:23.740257+0000", 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8460000000000001e-05 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.025 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.26173.0:84738 5.6 e444/389 5:65c90a2e:::benchmark_data_smithi173_87639_object4498:head v 444'5252, mlcod=444'5248)", 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:18.311910+0000", 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.645782182999994, 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31850212, 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.311910+0000", 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.311910+0000", 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.311911+0000", 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17e-06 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.311921+0000", 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5519999999999993e-06 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.628067+0000", 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31614592200000002 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.628075+0000", 2024-08-30T23:40:48.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2330000000000006e-06 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.628131+0000", 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6124e-05 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.628153+0000", 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1620999999999998e-05 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:40:48.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.630363+0000", 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022102929999999999 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:18.630412+0000", 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9205000000000001e-05 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 451/432 [PushOp(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head, version: 400'576, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head@400'576, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:40:48.030 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.358163340999994, 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44893841800000001, 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:40:48.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599530+0000", 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4339999999999999e-06 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599595+0000", 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4184999999999996e-05 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046675+0000", 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.447080796 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046706+0000", 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0505999999999999e-05 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046916+0000", 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021055600000000001 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046944+0000", 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7492999999999999e-05 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.048467+0000", 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015234479999999999 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 451/432 [PushOp(5:c3d2b08f:::benchmark_data_smithi173_87639_object9384:head, version: 400'582, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c3d2b08f:::benchmark_data_smithi173_87639_object9384:head@400'582, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:48.034 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:28.145491+0000", 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 79.812201802000004, 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.39316452299999999, 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145491+0000", 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145491+0000", 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145492+0000", 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145609+0000", 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117631 2024-08-30T23:40:48.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536849+0000", 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.39123995900000003 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536864+0000", 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5037e-05 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536953+0000", 2024-08-30T23:40:48.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8413999999999997e-05 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536963+0000", 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0265000000000001e-05 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.538655+0000", 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016920769999999999 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 463/432 [PushOp(5:c0945bbd:::benchmark_data_smithi173_87639_object35921:head, version: 407'2270, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c0945bbd:::benchmark_data_smithi173_87639_object35921:head@407'2270, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:09.511082+0000", 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.446610083000003, 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32755928600000001, 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511082+0000", 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511082+0000", 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511083+0000", 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5099999999999999e-07 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511113+0000", 2024-08-30T23:40:48.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0315999999999999e-05 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836525+0000", 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32541161000000002 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836544+0000", 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8987e-05 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836678+0000", 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013441899999999999 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836692+0000", 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.381e-05 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.838642+0000", 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019494930000000001 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 464/432 [PushOp(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head, version: 408'2477, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head@408'2477, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 34.88040934, 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52881942000000004, 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:40:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077310+0000", 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6896000000000001e-05 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077371+0000", 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1471999999999997e-05 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604348+0000", 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52697629300000004 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604378+0000", 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0654999999999997e-05 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604592+0000", 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021338600000000001 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604609+0000", 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7244999999999998e-05 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.606102+0000", 2024-08-30T23:40:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014934729999999999 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "replica_scrub(pg: 5.1,from:0'0,to:445'5203,epoch:470/389,start:5:8a8f2f50:::benchmark_data_smithi173_87639_object56981:0,end:5:8ae6a8d1:::benchmark_data_smithi173_87639_object64309:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 20.834872016999999, 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74865697399999998, 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122821+0000", 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7599999999999997e-07 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122828+0000", 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-06 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871217+0000", 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74838935200000001 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871235+0000", 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7708999999999999e-05 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871407+0000", 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172153 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871447+0000", 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9582999999999997e-05 2024-08-30T23:40:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871477+0000", 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0501000000000002e-05 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.9 e389 from shard 11)", 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:46.051110+0000", 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.9065820570000001, 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00030611400000000002, 2024-08-30T23:40:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051110+0000", 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051110+0000", 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051111+0000", 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4399999999999999e-07 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051138+0000", 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7240999999999999e-05 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051167+0000", 2024-08-30T23:40:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8677e-05 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051175+0000", 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2449999999999998e-06 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051235+0000", 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9993e-05 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051250+0000", 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4548e-05 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051416+0000", 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016656599999999999 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.9 e389 from shard 6)", 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:46.051582+0000", 2024-08-30T23:40:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.906110634, 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025920199999999999, 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051582+0000", 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051582+0000", 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051582+0000", 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0999999999999998e-07 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051598+0000", 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5421999999999999e-05 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051623+0000", 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4969000000000001e-05 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051627+0000", 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0400000000000003e-06 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051664+0000", 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6949000000000001e-05 2024-08-30T23:40:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051674+0000", 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0315e-05 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.051841+0000", 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000166897 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.9 e389 from shard 11)", 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:46.115794+0000", 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.8418984090000001, 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027283100000000002, 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115794+0000", 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115794+0000", 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115794+0000", 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.98e-07 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115809+0000", 2024-08-30T23:40:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.468e-05 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115842+0000", 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3083999999999997e-05 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115846+0000", 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.065e-06 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:40:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115977+0000", 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130922 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.115982+0000", 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:46.116067+0000", 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4253999999999997e-05 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:40:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:40:48.064 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:40:48.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:48.065 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:48.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-30T23:40:48.127 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:48.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:48.250 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:48.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:40:48.287 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:47 smithi136 ceph-mon[20207]: pgmap v1126: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:47 smithi173 ceph-mon[20554]: pgmap v1126: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:48.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:48.357 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:48.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:48.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:48.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:48.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:48.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:48.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:48.800 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:48.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:48.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:40:48.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:48.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:49.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:49.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:49.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:40:49.096 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:48 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:40:49.096 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:48 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:40:49.101 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:49.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:49.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:49.223 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:48 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:40:49.223 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:48 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:40:49.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:48 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:40:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:48 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:40:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:49.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:49.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:49.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:49.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:49.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:49.556 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:49.614 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:40:49.614 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:40:49.614 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:40:49.615 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:49.632 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:49.633 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:49.633 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:49.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:49.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:49.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:49.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:49.854 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:40:49.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:49.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:49.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:50.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:50.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:49 smithi106 ceph-mon[16603]: 5.3 scrub starts 2024-08-30T23:40:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:49 smithi106 ceph-mon[16603]: 5.c scrub starts 2024-08-30T23:40:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:49 smithi106 ceph-mon[16603]: 5.e scrub starts 2024-08-30T23:40:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:49 smithi106 ceph-mon[16603]: pgmap v1127: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3730290949' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:50.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:50.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:50.193 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:49 smithi136 ceph-mon[20207]: 5.3 scrub starts 2024-08-30T23:40:50.193 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:49 smithi136 ceph-mon[20207]: 5.c scrub starts 2024-08-30T23:40:50.193 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:49 smithi136 ceph-mon[20207]: 5.e scrub starts 2024-08-30T23:40:50.193 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:49 smithi136 ceph-mon[20207]: pgmap v1127: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:50.193 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3730290949' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:50.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:49 smithi173 ceph-mon[20554]: 5.3 scrub starts 2024-08-30T23:40:50.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:49 smithi173 ceph-mon[20554]: 5.c scrub starts 2024-08-30T23:40:50.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:49 smithi173 ceph-mon[20554]: 5.e scrub starts 2024-08-30T23:40:50.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:49 smithi173 ceph-mon[20554]: pgmap v1127: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:50.253 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3730290949' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:40:50.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:50.284 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:50.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:50.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:50.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:50.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:50.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:50.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:40:50.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:50.659 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:50.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:50.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:50.824 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:50.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:50.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:50.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:51.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:51.033 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:51.108 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3730290949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: osdmap e479: 12 total, 10 up, 12 in 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: 5.e scrub ok 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: 5.3 scrub ok 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: 5.c scrub ok 2024-08-30T23:40:51.109 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:50 smithi173 ceph-mon[20554]: osdmap e480: 12 total, 10 up, 12 in 2024-08-30T23:40:51.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:51.266 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:51.286 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:40:51.287 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:40:51.287 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:40:51.287 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:40:51.287 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3730290949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: osdmap e479: 12 total, 10 up, 12 in 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: 5.e scrub ok 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: 5.3 scrub ok 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: 5.c scrub ok 2024-08-30T23:40:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:50 smithi136 ceph-mon[20207]: osdmap e480: 12 total, 10 up, 12 in 2024-08-30T23:40:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:40:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:40:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:40:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3730290949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:40:51.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: osdmap e479: 12 total, 10 up, 12 in 2024-08-30T23:40:51.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: 5.e scrub ok 2024-08-30T23:40:51.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: 5.3 scrub ok 2024-08-30T23:40:51.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: 5.c scrub ok 2024-08-30T23:40:51.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:50 smithi106 ceph-mon[16603]: osdmap e480: 12 total, 10 up, 12 in 2024-08-30T23:40:51.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:51.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:51.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:51.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:51.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:51.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:51.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:51.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:51.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:51.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:51.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:51.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:40:52.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:52.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:52.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:52.151 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:52.153 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:52.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:52.258 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:51 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:40:52.259 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:51 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:40:52.259 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:51 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:40:52.259 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:51 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:40:52.259 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:51 smithi173 ceph-mon[20554]: pgmap v1130: 27 pgs: 1 active+clean+scrubbing, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:52.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:51 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:40:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:51 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:40:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:51 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:40:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:51 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:40:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:51 smithi136 ceph-mon[20207]: pgmap v1130: 27 pgs: 1 active+clean+scrubbing, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:51 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:40:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:51 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:40:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:51 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:40:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:51 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:40:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:51 smithi106 ceph-mon[16603]: pgmap v1130: 27 pgs: 1 active+clean+scrubbing, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:52.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:52.403 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:52.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:52.529 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:103438 3.9 3:9093f9df:::benchmark_data_smithi173_60780_object103437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 551.31029314800003, 2024-08-30T23:40:52.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38848340100000001, 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 103438 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:40:52.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274866+0000", 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274868+0000", 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-06 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274894+0000", 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5692000000000001e-05 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274909+0000", 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5378000000000001e-05 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274918+0000", 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6489999999999997e-06 2024-08-30T23:40:52.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.274990+0000", 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2335000000000005e-05 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275112+0000", 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012186 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:40:52.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.275235+0000", 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123585 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278451+0000", 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003215954 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.279271+0000", 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081947200000000002 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663147+0000", 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38387581399999998 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663253+0000", 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106125 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663350+0000", 2024-08-30T23:40:52.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6832999999999999e-05 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.15351.0:103438 3.9 e255/206 ondisk, result = 0)", 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 551.30704387399999, 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38524248500000002, 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.593 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278115+0000", 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278116+0000", 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9199999999999999e-07 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.278121+0000", 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7609999999999998e-06 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662960+0000", 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38483874699999998 2024-08-30T23:40:52.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.662985+0000", 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5055999999999999e-05 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663120+0000", 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013537999999999999 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663136+0000", 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6220000000000001e-05 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:31:41.663358+0000", 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022142900000000001 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.15351.0:163721 3.9 3:9f4ef878:::benchmark_data_smithi173_60780_object60267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:40:52.597 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 483.93192010400003, 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.179110358, 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.15351", 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/283472260", 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 163721 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653239+0000", 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653241+0000", 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.013e-06 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653250+0000", 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653272+0000", 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2860000000000001e-05 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.653281+0000", 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8289999999999997e-06 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654712+0000", 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014306620000000001 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654854+0000", 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000141755 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.654980+0000", 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012682000000000001 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.658949+0000", 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039685739999999999 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.714803+0000", 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.055854251000000001 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832230+0000", 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.117426525 2024-08-30T23:40:52.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832301+0000", 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0921999999999996e-05 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:32:48.832350+0000", 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8835000000000003e-05 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:26909 4.0 e315/306 4:05316899:::benchmark_data_smithi173_74914_object26908:head v 315'1674, mlcod=315'1671)", 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 394.38867374900002, 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.194211306, 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196486+0000", 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196487+0000", 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.196605+0000", 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011812299999999999 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389338+0000", 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19273368799999999 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389343+0000", 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6589999999999998e-06 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389405+0000", 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1645999999999999e-05 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.389423+0000", 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8641e-05 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390666+0000", 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001242993 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:18.390697+0000", 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0565000000000003e-05 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:49431 4.2 e327/306 4:49678b2f:::benchmark_data_smithi173_74914_object49430:head v 327'3104, mlcod=326'3101)", 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 372.04309123299998, 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.203368459, 2024-08-30T23:40:52.609 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542068+0000", 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0900000000000002e-07 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.542091+0000", 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2452000000000001e-05 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743809+0000", 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20171845999999999 2024-08-30T23:40:52.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743817+0000", 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0960000000000001e-06 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743892+0000", 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4893000000000003e-05 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.743911+0000", 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8479999999999999e-05 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745394+0000", 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001483617 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:40.745437+0000", 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2052000000000003e-05 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.613 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:84204 4.0 e346/329 4:023d35bd:::benchmark_data_smithi173_74914_object84203:head v 346'5351, mlcod=346'5348)", 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 335.184227416, 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20227883999999999, 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400932+0000", 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400933+0000", 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1990000000000001e-06 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.400986+0000", 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3118000000000002e-05 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601032+0000", 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.200045414 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601038+0000", 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4330000000000001e-06 2024-08-30T23:40:52.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601086+0000", 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7720999999999999e-05 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.601104+0000", 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8267999999999999e-05 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603163+0000", 2024-08-30T23:40:52.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020593569999999999 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:35:17.603211+0000", 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7330000000000003e-05 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.25654.0:196473 4.0 e381/329 4:0415f716:::benchmark_data_smithi173_74914_object98231:head v 381'12558, mlcod=381'12555)", 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.024645+0000", 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 214.56051434599999, 2024-08-30T23:40:52.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.49918435300000003, 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024645+0000", 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024645+0000", 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024646+0000", 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4160000000000001e-06 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.024656+0000", 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1430000000000004e-06 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519446+0000", 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.494790018 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519459+0000", 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3637e-05 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519611+0000", 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000151867 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.519654+0000", 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2935000000000001e-05 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.523788+0000", 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041342030000000004 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.523829+0000", 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1134000000000003e-05 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.622 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:12179 5.4 e401/394 5:20ccc99b:::benchmark_data_smithi173_87639_object12178:head v 401'786, mlcod=401'783)", 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:49.201090+0000", 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.38406921000001, 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.166900032, 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201090+0000", 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201090+0000", 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201091+0000", 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0699999999999996e-07 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.201149+0000", 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8604000000000003e-05 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367308+0000", 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16615836 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367313+0000", 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4949999999999999e-06 2024-08-30T23:40:52.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367362+0000", 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9107000000000003e-05 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367377+0000", 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5007999999999999e-05 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367957+0000", 2024-08-30T23:40:52.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00058009700000000001 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:49.367990+0000", 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2654000000000003e-05 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:34695 5.a e407/394 5:5ba104af:::benchmark_data_smithi173_87639_object34694:head v 407'2145, mlcod=407'2142)", 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:03.756488+0000", 2024-08-30T23:40:52.627 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.828671063, 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15478784600000001, 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756488+0000", 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756488+0000", 2024-08-30T23:40:52.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756489+0000", 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.756513+0000", 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3717999999999999e-05 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910265+0000", 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15375180799999999 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910269+0000", 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6920000000000001e-06 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910313+0000", 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3556000000000001e-05 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.910325+0000", 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2661e-05 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.911242+0000", 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091632699999999998 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:03.911276+0000", 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4303e-05 2024-08-30T23:40:52.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:44435 5.4 e410/394 5:21107bab:::benchmark_data_smithi173_87639_object44434:head v 410'2764, mlcod=410'2761)", 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:11.418128+0000", 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 161.16703087900001, 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16557548499999999, 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418128+0000", 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418128+0000", 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418129+0000", 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0279999999999999e-06 2024-08-30T23:40:52.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.418188+0000", 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8851999999999998e-05 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582325+0000", 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16413667500000001 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582331+0000", 2024-08-30T23:40:52.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8730000000000004e-06 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582383+0000", 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2444999999999997e-05 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582404+0000", 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0818999999999998e-05 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.583669+0000", 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012650890000000001 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.583704+0000", 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4703999999999998e-05 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:44461 5.4 e410/394 5:2bfa2ed6:::benchmark_data_smithi173_87639_object44460:head v 410'2765, mlcod=410'2762)", 2024-08-30T23:40:52.636 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:11.582282+0000", 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 161.00287689500001, 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41074240899999998, 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582282+0000", 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582282+0000", 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582283+0000", 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4099999999999998e-07 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.582308+0000", 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4638e-05 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991587+0000", 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40927872999999998 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991605+0000", 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8839999999999999e-05 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991745+0000", 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014014 2024-08-30T23:40:52.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.991760+0000", 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4893e-05 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.992996+0000", 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012353889999999999 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:11.993025+0000", 2024-08-30T23:40:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9037999999999999e-05 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 459/432 [PushOp(5:c960c4e7:::benchmark_data_smithi173_87639_object28434:head, version: 406'1805, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c960c4e7:::benchmark_data_smithi173_87639_object28434:head@406'1805, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:58.153228+0000", 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.431931034000002, 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61530680400000004, 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.153228+0000", 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.153228+0000", 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.153229+0000", 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0500000000000003e-07 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.153304+0000", 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4801000000000004e-05 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.766993+0000", 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61368893999999996 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.767011+0000", 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8314000000000001e-05 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.767114+0000", 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010354200000000001 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.767125+0000", 2024-08-30T23:40:52.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0618999999999999e-05 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.768535+0000", 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014099830000000001 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 459/432 [PushOp(5:cb0f6473:::benchmark_data_smithi173_87639_object28435:head, version: 406'1806, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cb0f6473:::benchmark_data_smithi173_87639_object28435:head@406'1806, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:58.780208+0000", 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 53.804951441, 2024-08-30T23:40:52.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27630119400000003, 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.780208+0000", 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.780208+0000", 2024-08-30T23:40:52.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.780209+0000", 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5030000000000001e-06 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:58.780365+0000", 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000155403 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:59.054932+0000", 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27456710600000001 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:59.054945+0000", 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3464e-05 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:59.055047+0000", 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101957 2024-08-30T23:40:52.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:59.055058+0000", 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0947e-05 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:59.056509+0000", 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014508139999999999 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 464/432 [PushOp(5:c62fd5fc:::benchmark_data_smithi173_87639_object37303:head, version: 408'2349, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c62fd5fc:::benchmark_data_smithi173_87639_object37303:head@408'2349, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:10.892888+0000", 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.692270886999999, 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.371672263, 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:10.892888+0000", 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:10.892888+0000", 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:10.892889+0000", 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4000000000000001e-07 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:10.893191+0000", 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000301954 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:11.262660+0000", 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.369469351 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:11.262679+0000", 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8797999999999999e-05 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:11.262793+0000", 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113579 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:11.262804+0000", 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0916e-05 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:11.264561+0000", 2024-08-30T23:40:52.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017570249999999999 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 468/432 [PushOp(5:cd9fdb30:::benchmark_data_smithi173_87639_object50077:head, version: 415'3158, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cd9fdb30:::benchmark_data_smithi173_87639_object50077:head@415'3158, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:22.218825+0000", 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 30.366334074000001, 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58546559600000003, 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.218825+0000", 2024-08-30T23:40:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.218825+0000", 2024-08-30T23:40:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.218826+0000", 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8899999999999999e-07 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.218875+0000", 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9094000000000001e-05 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.802497+0000", 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58362190700000005 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.802518+0000", 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1420000000000002e-05 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.802630+0000", 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011179 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.802648+0000", 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7938e-05 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:22.804291+0000", 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016428580000000001 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 468/432 [PushOp(5:cf5427eb:::benchmark_data_smithi173_87639_object52812:head, version: 416'3330, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cf5427eb:::benchmark_data_smithi173_87639_object52812:head@416'3330, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:24.454217+0000", 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.130941930999999, 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.80342548899999999, 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454217+0000", 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454217+0000", 2024-08-30T23:40:52.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454218+0000", 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0500000000000003e-07 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454326+0000", 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108422 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.255685+0000", 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.80135822199999995 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.255715+0000", 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0593000000000003e-05 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.256047+0000", 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000332121 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.256081+0000", 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3705999999999999e-05 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.257643+0000", 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00156182 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 470/432 [PushOp(5:c32e7fa5:::benchmark_data_smithi173_87639_object55010:head, version: 417'3465, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c32e7fa5:::benchmark_data_smithi173_87639_object55010:head@417'3465, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:26.477253+0000", 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 26.107906286999999, 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93278049699999999, 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477253+0000", 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477253+0000", 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477254+0000", 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1960000000000001e-06 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477375+0000", 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012103399999999999 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408122+0000", 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93074724099999995 2024-08-30T23:40:52.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408136+0000", 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3864e-05 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408249+0000", 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113028 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408261+0000", 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1348e-05 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.410034+0000", 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017727859999999999 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(5.3 e472 from shard 7)", 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:48.805101+0000", 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.7800580579999998, 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.280582257, 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:48.805101+0000", 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:48.805101+0000", 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:48.805102+0000", 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:48.805132+0000", 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9801000000000001e-05 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.085378+0000", 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.280246035 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.085391+0000", 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3196e-05 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.085474+0000", 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2885000000000003e-05 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.085489+0000", 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5393e-05 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.085683+0000", 2024-08-30T23:40:52.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000194298 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(5.3 e472 from shard 10)", 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:49.153699+0000", 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.4314606149999998, 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.317477026, 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.678 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.153699+0000", 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.153699+0000", 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.153699+0000", 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2500000000000005e-07 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.153714+0000", 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4894e-05 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.470913+0000", 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31719884599999998 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.470918+0000", 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0069999999999997e-06 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.470969+0000", 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0963e-05 2024-08-30T23:40:52.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.470981+0000", 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1878999999999999e-05 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.471176+0000", 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019481199999999999 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.682 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(5.3 e472 from shard 10)", 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:49.479248+0000", 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.1059110520000002, 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22191761900000001, 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:40:52.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.479248+0000", 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.479248+0000", 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.479249+0000", 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9400000000000005e-07 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:40:52.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.479265+0000", 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6373000000000001e-05 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.700998+0000", 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22173234999999999 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:40:52.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.701004+0000", 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1600000000000003e-06 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.701047+0000", 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3173999999999997e-05 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.701054+0000", 2024-08-30T23:40:52.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5849999999999996e-06 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:49.701166+0000", 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112283 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:40:52.688 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:40:52.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:52.797 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:52.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-30T23:40:52.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:52.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:52.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:52.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:52.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:52.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:53.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:53.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:53.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:53.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:53.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:53.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:53.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:53.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:53.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:53.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:53 smithi173 ceph-mon[20554]: 5.8 scrub starts 2024-08-30T23:40:53.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:53 smithi173 ceph-mon[20554]: 5.8 scrub ok 2024-08-30T23:40:53.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:53.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:53.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:53.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:53 smithi106 ceph-mon[16603]: 5.8 scrub starts 2024-08-30T23:40:53.732 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:53 smithi106 ceph-mon[16603]: 5.8 scrub ok 2024-08-30T23:40:53.749 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:53.749 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:53.749 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:53.767 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:53.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:40:53.773 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:53.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:53 smithi136 ceph-mon[20207]: 5.8 scrub starts 2024-08-30T23:40:53.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:53 smithi136 ceph-mon[20207]: 5.8 scrub ok 2024-08-30T23:40:53.801 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:53.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:53.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:53.824 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:53.825 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:53.825 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:53.831 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:53.847 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:53.848 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:53.869 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:53.869 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:53.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:53.891 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:53.912 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:53.912 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:53.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:53.934 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:53.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:53.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:53.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:53.956 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:53.956 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:54.041 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:54.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:54.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:54.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:54.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:54.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:54.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:54.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:54.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:54.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:54.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:54.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:54.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:54.704 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:54 smithi106 ceph-mon[16603]: pgmap v1131: 27 pgs: 1 active+clean+scrubbing, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:54.704 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:54 smithi106 ceph-mon[16603]: 5.4 scrub starts 2024-08-30T23:40:54.704 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1862892710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:54.704 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:54 smithi106 ceph-mon[16603]: 5.4 scrub ok 2024-08-30T23:40:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:40:54.705 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:40:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:54.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:54.731 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:54 smithi173 ceph-mon[20554]: pgmap v1131: 27 pgs: 1 active+clean+scrubbing, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:54.731 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:54 smithi173 ceph-mon[20554]: 5.4 scrub starts 2024-08-30T23:40:54.731 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1862892710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:54.731 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:54 smithi173 ceph-mon[20554]: 5.4 scrub ok 2024-08-30T23:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:54.731 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:54 smithi136 ceph-mon[20207]: pgmap v1131: 27 pgs: 1 active+clean+scrubbing, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:54 smithi136 ceph-mon[20207]: 5.4 scrub starts 2024-08-30T23:40:54.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1862892710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:54 smithi136 ceph-mon[20207]: 5.4 scrub ok 2024-08-30T23:40:54.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:54.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:54.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:54.964 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:54.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:54.966 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:55.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:55.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:55.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:55.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:55.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:55.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:55.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:55.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:55.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:40:55.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:40:55.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:55 smithi106 ceph-mon[16603]: 5.0 scrub starts 2024-08-30T23:40:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:55 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:40:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:55 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:40:55.617 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:55.659 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:55.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-08-30T23:40:55.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:55.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:55 smithi136 ceph-mon[20207]: 5.0 scrub starts 2024-08-30T23:40:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:55 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:40:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:55 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:40:55.801 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:55.810 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:55 smithi173 ceph-mon[20554]: 5.0 scrub starts 2024-08-30T23:40:55.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:55 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:40:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:55 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:40:55.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:55.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:55.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:55.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:55.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:56.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:56.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:56.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:56.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:56.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:56.257 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:56.272 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:56.273 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:56.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:56.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:56.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:56.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: osdmap e481: 12 total, 10 up, 12 in 2024-08-30T23:40:56.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: pgmap v1133: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:56.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:40:56.485 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:40:56.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: 5.0 scrub ok 2024-08-30T23:40:56.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: osdmap e482: 12 total, 10 up, 12 in 2024-08-30T23:40:56.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/917486536' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:56.501 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:56.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:56.624 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:40:56.703 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:40:56.704 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:40:56.704 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:40:56.704 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:40:56.704 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:40:56.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:40:56.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:56.740 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:40:56.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: osdmap e481: 12 total, 10 up, 12 in 2024-08-30T23:40:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: pgmap v1133: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:40:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:40:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: 5.0 scrub ok 2024-08-30T23:40:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: osdmap e482: 12 total, 10 up, 12 in 2024-08-30T23:40:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/917486536' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: osdmap e481: 12 total, 10 up, 12 in 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: pgmap v1133: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: 5.0 scrub ok 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: osdmap e482: 12 total, 10 up, 12 in 2024-08-30T23:40:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/917486536' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:40:56.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:56.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:56.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:56.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:40:56.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:57.009 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:57.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:57.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:57.180 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:57.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:57.258 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:57.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:40:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:57.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:40:57.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:40:57.493 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:40:57.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:57.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:57 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:40:57.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:57 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:40:57.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:57 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:57.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/917486536' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:57.563 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:57 smithi173 ceph-mon[20554]: osdmap e483: 12 total, 10 up, 12 in 2024-08-30T23:40:57.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:57.628 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:57.671 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:57.671 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:57.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:57.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:57 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:40:57.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:57 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:40:57.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:57 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:57.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/917486536' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:57.743 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:57 smithi106 ceph-mon[16603]: osdmap e483: 12 total, 10 up, 12 in 2024-08-30T23:40:57.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:57 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:40:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:57 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:40:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:57 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:40:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/917486536' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:40:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:57 smithi136 ceph-mon[20207]: osdmap e483: 12 total, 10 up, 12 in 2024-08-30T23:40:57.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:57.879 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:57.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:57.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:58.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:58.087 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:40:58.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:40:58.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:40:58.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:40:58.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:40:58.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:40:58.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.379 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:58.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:58.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:40:58.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:40:58.515 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:58 smithi173 ceph-mon[20554]: pgmap v1136: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:58 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:40:58.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:58 smithi106 ceph-mon[16603]: pgmap v1136: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:58 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:40:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:58.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:58.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:58.751 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:58 smithi136 ceph-mon[20207]: pgmap v1136: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:40:58.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:58 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:40:58.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:58.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:58.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:58.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:40:59.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:40:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:40:59.020 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:40:59.020 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:40:59.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:40:59.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:40:59.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:40:59.251 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:40:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:40:59.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:59.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:59.390 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:40:59.390 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:40:59.390 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:40:59.412 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:40:59.412 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:40:59.433 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:40:59.433 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:40:59.455 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:40:59.455 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:40:59.456 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:40:59.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:40:59.478 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:40:59.478 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:40:59.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:40:59.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:40:59.500 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:40:59.520 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:40:59.521 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:40:59.542 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:40:59.542 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:40:59.563 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:40:59.563 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:40:59.583 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:40:59.584 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:40:59.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:59.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:59.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:59.658 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:40:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1361914706' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:59.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:40:59.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:40:59.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:40:59.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:40:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1361914706' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:40:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1361914706' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:40:59.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:40:59.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:40:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:40:59.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:40:59.980 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:40:59.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:00.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:00.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:41:00.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:00.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:00.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:41:00.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:41:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:00.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:00.291 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:00.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:00.379 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:00.479 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:41:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:41:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:41:00.480 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:41:00.480 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:41:00.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:00.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:00.516 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:00 smithi106 ceph-mon[16603]: pgmap v1137: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:00.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:00.631 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:00.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:00.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:00 smithi136 ceph-mon[20207]: pgmap v1137: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:00.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:00 smithi173 ceph-mon[20554]: pgmap v1137: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:00.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:00.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:00.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:01.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:01.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:01.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:01.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:41:01.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:01.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:01.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:01.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:01.418 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:41:01.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:01.514 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:01.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:01.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:01.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:01.758 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:41:01.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:01.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:01 smithi173 ceph-mon[20554]: osdmap e484: 12 total, 10 up, 12 in 2024-08-30T23:41:01.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3997109444' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:01.811 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:01 smithi173 ceph-mon[20554]: pgmap v1139: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:01.869 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:01.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:01.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(3.1 e301,280 3:8490e9ac:::benchmark_data_smithi173_60780_object7986:head,258'6976)", 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:41:01.911 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 434.73097619800001, 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70969468099999999, 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178978+0000", 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4199999999999995e-07 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.178982+0000", 2024-08-30T23:41:01.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9779999999999999e-06 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887644+0000", 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.70866116300000004 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887661+0000", 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6954999999999999e-05 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.887778+0000", 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117812 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:47.888673+0000", 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00089413100000000005 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.915 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31037 4.5 4.306472d5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.22754109300001, 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0020195190000001, 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.916 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31037 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682413+0000", 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682415+0000", 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6589999999999999e-06 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682488+0000", 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3084999999999996e-05 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682498+0000", 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0644e-05 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682506+0000", 2024-08-30T23:41:01.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0229999999999992e-06 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682702+0000", 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019531299999999999 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682713+0000", 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1778e-05 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684400+0000", 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016865800000001 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684432+0000", 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2437000000000001e-05 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31040 4.5 4.eb15935 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.22738350600002, 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.001902058, 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31040 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682571+0000", 2024-08-30T23:41:01.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682627+0000", 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5911000000000001e-05 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682637+0000", 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0363999999999999e-05 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682642+0000", 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8509999999999998e-06 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682765+0000", 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123311 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.682799+0000", 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3145000000000002e-05 2024-08-30T23:41:01.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684465+0000", 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0016660740000001 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684473+0000", 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9149999999999996e-06 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31060 4.5 4.c6dc7475 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.21797828199999, 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99255960700000001, 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.927 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:41:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:41:01.928 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31060 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691976+0000", 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691977+0000", 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.066e-06 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.691998+0000", 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.158e-05 2024-08-30T23:41:01.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692011+0000", 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2848e-05 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692019+0000", 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8990000000000001e-06 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692069+0000", 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0080999999999997e-05 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.692076+0000", 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1450000000000002e-06 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684524+0000", 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99244810000000006 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.684535+0000", 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0888e-05 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58973 5.4 5.af96ba24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.114618+0000", 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.795335577, 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.997737031, 2024-08-30T23:41:01.933 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58973 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114618+0000", 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114618+0000", 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114620+0000", 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114646+0000", 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5905999999999999e-05 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114658+0000", 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2362e-05 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114667+0000", 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7819999999999999e-06 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114717+0000", 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9570000000000001e-05 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114726+0000", 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1640000000000007e-06 2024-08-30T23:41:01.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112346+0000", 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99762036799999998 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112355+0000", 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2499999999999995e-06 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58980 5.8 5.98770828 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.114757+0000", 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.79519677499999, 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99760046099999999, 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58980 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.939 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114757+0000", 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114757+0000", 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114758+0000", 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6000000000000003e-07 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114778+0000", 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9809e-05 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114787+0000", 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4450000000000002e-06 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114793+0000", 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6289999999999998e-06 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114842+0000", 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9650000000000001e-05 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114850+0000", 2024-08-30T23:41:01.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3350000000000001e-06 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112343+0000", 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99749290499999999 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112358+0000", 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5028000000000001e-05 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58983 5.4 5.4902cd24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.114807+0000", 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.795146862, 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99758265700000004, 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58983 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114807+0000", 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114807+0000", 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114808+0000", 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2899999999999999e-07 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114829+0000", 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1347000000000001e-05 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114836+0000", 2024-08-30T23:41:01.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2509999999999997e-06 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114841+0000", 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.933e-06 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114882+0000", 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1069000000000003e-05 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114894+0000", 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1980999999999999e-05 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112382+0000", 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99748783600000002 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112390+0000", 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8110000000000003e-06 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58991 5.8 5.4658f208 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.116833+0000", 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.79312125000001, 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99558751199999995, 2024-08-30T23:41:01.949 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58991 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116833+0000", 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116833+0000", 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116834+0000", 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6339999999999999e-06 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116859+0000", 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4153999999999999e-05 2024-08-30T23:41:01.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116871+0000", 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.234e-05 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116879+0000", 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.106e-06 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116923+0000", 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4403999999999997e-05 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116929+0000", 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7690000000000002e-06 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112412+0000", 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99548259500000003 2024-08-30T23:41:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112420+0000", 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5099999999999998e-06 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59002 5.8 5.40785bb8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.145317+0000", 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.76463736100001, 2024-08-30T23:41:01.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96713300800000002, 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59002 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145317+0000", 2024-08-30T23:41:01.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145317+0000", 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145319+0000", 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2589999999999999e-06 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145344+0000", 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4689e-05 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145362+0000", 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8624999999999999e-05 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145370+0000", 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.576e-06 2024-08-30T23:41:01.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145432+0000", 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2663000000000006e-05 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145442+0000", 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5049999999999996e-06 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112442+0000", 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96700018099999996 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112450+0000", 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5100000000000001e-06 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.959 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59008 5.4 5.ad204744 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.336873+0000", 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.5730811, 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77569046100000005, 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59008 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.960 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336873+0000", 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336873+0000", 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336875+0000", 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1289999999999999e-06 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336899+0000", 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4173e-05 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336918+0000", 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8940000000000002e-05 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336927+0000", 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5720000000000003e-06 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336989+0000", 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2285000000000004e-05 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336996+0000", 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2439999999999996e-06 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112555+0000", 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77555825599999995 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112563+0000", 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8619999999999992e-06 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.964 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59009 5.a 5.536bab9a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.341155+0000", 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.568799321, 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77138790199999996, 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.965 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59009 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341155+0000", 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341155+0000", 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341157+0000", 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.695e-06 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341208+0000", 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.041e-05 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341230+0000", 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.247e-05 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341240+0000", 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0069e-05 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341303+0000", 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2879000000000006e-05 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341312+0000", 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4100000000000008e-06 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112520+0000", 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.771208067 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112543+0000", 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2901999999999999e-05 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59010 5.8 5.34167998 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.447203+0000", 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.462751274, 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66527532499999997, 2024-08-30T23:41:01.970 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59010 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447203+0000", 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447203+0000", 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447205+0000", 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0020000000000001e-06 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447252+0000", 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7345999999999997e-05 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447269+0000", 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6755000000000001e-05 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447276+0000", 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6449999999999999e-06 2024-08-30T23:41:01.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447336+0000", 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.02e-05 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.447346+0000", 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0394000000000001e-05 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112471+0000", 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.66512450899999997 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112478+0000", 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4739999999999999e-06 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.2 425/420 [PushOp(5:4ffcd3d6:::benchmark_data_smithi173_87639_object4590:head, version: 398'287, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:4ffcd3d6:::benchmark_data_smithi173_87639_object4590:head@398'287, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:01.975 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 141.51788448400001, 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94084964800000004, 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1600000000000001e-07 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392115+0000", 2024-08-30T23:41:01.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5108999999999998e-05 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331203+0000", 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93908753700000003 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331236+0000", 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2728999999999998e-05 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331454+0000", 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021814600000000001 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331479+0000", 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5789e-05 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.332919+0000", 2024-08-30T23:41:01.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014397220000000001 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.4 425/420 [PushOp(5:2400e38c:::benchmark_data_smithi173_87639_object6909:head, version: 399'435, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:2400e38c:::benchmark_data_smithi173_87639_object6909:head@399'435, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:40.586744+0000", 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 141.323210101, 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84143205700000001, 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.980 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586744+0000", 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586744+0000", 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586745+0000", 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3799999999999999e-06 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586787+0000", 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1313999999999998e-05 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426748+0000", 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83996137699999995 2024-08-30T23:41:01.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426778+0000", 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0380000000000001e-05 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426960+0000", 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018135499999999999 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:01.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426980+0000", 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0245999999999999e-05 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.428176+0000", 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001196005 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.8 426/420 [PushOp(5:1176d18c:::benchmark_data_smithi173_87639_object6158:head, version: 399'397, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1176d18c:::benchmark_data_smithi173_87639_object6158:head@399'397, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:01.984 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:41.768366+0000", 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 140.14158755700001, 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91116549400000002, 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768366+0000", 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768366+0000", 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768367+0000", 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8100000000000003e-07 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768398+0000", 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0908000000000003e-05 2024-08-30T23:41:01.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678320+0000", 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90992154700000005 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678348+0000", 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8279000000000001e-05 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678600+0000", 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025234299999999998 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678614+0000", 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4294000000000001e-05 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.679532+0000", 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091754200000000003 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.2 427/420 [PushOp(5:4ae76459:::benchmark_data_smithi173_87639_object8076:head, version: 400'527, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:4ae76459:::benchmark_data_smithi173_87639_object8076:head@400'527, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:44.354983+0000", 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 137.554970879, 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87486126600000003, 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.354983+0000", 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.354983+0000", 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.354985+0000", 2024-08-30T23:41:01.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.463e-06 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.355050+0000", 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5874e-05 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228350+0000", 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87329974600000004 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228379+0000", 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8702000000000001e-05 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228600+0000", 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022151199999999999 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228616+0000", 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5291999999999999e-05 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.229844+0000", 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012286770000000001 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 449/432 [PushOp(5:c55d6b41:::benchmark_data_smithi173_87639_object8462:head, version: 400'521, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c55d6b41:::benchmark_data_smithi173_87639_object8462:head@400'521, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.928138316000002, 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73936645499999998, 2024-08-30T23:41:01.993 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9900000000000001e-07 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981923+0000", 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010642 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719510+0000", 2024-08-30T23:41:01.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73758770799999995 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719527+0000", 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6744000000000001e-05 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719625+0000", 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7499999999999998e-05 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:01.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719634+0000", 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2820000000000002e-06 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.721182+0000", 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015483020000000001 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.997 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 450/432 [PushOp(5:cdb9415d:::benchmark_data_smithi173_87639_object8623:head, version: 400'532, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cdb9415d:::benchmark_data_smithi173_87639_object8623:head@400'532, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:25.931361+0000", 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 95.978592914999993, 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72661587400000005, 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931361+0000", 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:01.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931361+0000", 2024-08-30T23:41:01.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:01.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:01.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:01.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:01.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:02.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931366+0000", 2024-08-30T23:41:02.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4739999999999999e-06 2024-08-30T23:41:02.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:02.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931429+0000", 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3728999999999999e-05 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656006+0000", 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72457618999999995 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656019+0000", 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3563e-05 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656127+0000", 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108312 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656138+0000", 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0519999999999999e-05 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.657977+0000", 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001839086 2024-08-30T23:41:02.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.4 450/448 [PushReplyOp(5:2985048c:::benchmark_data_smithi173_87639_object67330:head)])", 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 95.950520393000005, 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77832121700000001, 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:02.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:41:02.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9399999999999995e-07 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959438+0000", 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.715e-06 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737214+0000", 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77777609000000003 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737219+0000", 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737291+0000", 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1145999999999999e-05 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737755+0000", 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00046422199999999997 2024-08-30T23:41:02.006 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.a 451/420 [PushOp(5:506865e8:::benchmark_data_smithi173_87639_object18842:head, version: 403'1181, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:506865e8:::benchmark_data_smithi173_87639_object18842:head@403'1181, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:28.772486+0000", 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 93.137467701000006, 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62979779599999997, 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:41:02.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:28.772486+0000", 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:28.772486+0000", 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:28.772487+0000", 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.75e-07 2024-08-30T23:41:02.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:28.772513+0000", 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6236000000000001e-05 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:29.399977+0000", 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62746431999999996 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:41:02.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:29.399991+0000", 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3879e-05 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:29.400108+0000", 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117067 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:29.400120+0000", 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1973999999999999e-05 2024-08-30T23:41:02.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:29.402284+0000", 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021637449999999999 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:41:02.011 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:41:02.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:02.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:01 smithi136 ceph-mon[20207]: osdmap e484: 12 total, 10 up, 12 in 2024-08-30T23:41:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3997109444' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:01 smithi136 ceph-mon[20207]: pgmap v1139: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:01 smithi106 ceph-mon[16603]: osdmap e484: 12 total, 10 up, 12 in 2024-08-30T23:41:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3997109444' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:01 smithi106 ceph-mon[16603]: pgmap v1139: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:02.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:02.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:02.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:41:02.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:02.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:02.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:02.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:02.495 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1, 5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-08-30T23:41:02.495 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:41:02.495 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-08-30T23:41:02.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:41:02.508 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:02.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:02.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:02.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:02.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:02.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:41:02.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:02.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3997109444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:02.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:02 smithi173 ceph-mon[20554]: osdmap e485: 12 total, 10 up, 12 in 2024-08-30T23:41:02.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:02.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:02.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:02.976 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:41:02.979 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:41:02.979 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:41:02.979 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:41:02.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3997109444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:02.980 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:02 smithi106 ceph-mon[16603]: osdmap e485: 12 total, 10 up, 12 in 2024-08-30T23:41:03.000 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:41:03.000 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:41:03.013 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:03.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:03.022 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:41:03.022 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:41:03.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3997109444' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:03.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:02 smithi136 ceph-mon[20207]: osdmap e485: 12 total, 10 up, 12 in 2024-08-30T23:41:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:03.048 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:41:03.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:41:03.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:41:03.070 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:41:03.070 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:41:03.091 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:41:03.092 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:41:03.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:41:03.113 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:41:03.134 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:41:03.134 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:41:03.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:03.155 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:41:03.155 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:41:03.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:03.176 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:41:03.176 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:41:03.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:03.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:03.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:03.442 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:03.470 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:03.470 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:03.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:03.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:03.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:03.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:03.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:41:03.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:41:03.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:03.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:03.942 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:03.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:04.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/953828216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:04.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:03 smithi106 ceph-mon[16603]: pgmap v1141: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:04.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/953828216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:04.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:03 smithi136 ceph-mon[20207]: pgmap v1141: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:04.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:04.065 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/953828216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:03 smithi173 ceph-mon[20554]: pgmap v1141: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:04.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:41:04.157 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:41:04.158 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:41:04.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:04.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:04.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:04.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:04.371 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:41:04.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:04.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:04.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:41:04.541 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:04.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:04.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:04.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:04.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:04.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:04.855 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:04.855 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:04.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:04.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:04.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:05.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:05.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:04 smithi106 ceph-mon[16603]: from='client.17166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:41:05.039 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1141,"stamp":"2024-08-30T23:41:03.553939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4852535742,"num_objects":74040,"num_object_clones":0,"num_object_copies":222120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32800,"num_objects_misplaced":9195,"num_objects_unfound":0,"num_objects_dirty":74040,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":86448,"num_write_kb":5135364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20299,"num_bytes_recovered":1288551380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87449,"ondisk_log_size":87449,"up":69,"acting":72,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19234404,"kb_used_data":15406048,"kb_used_omap":18,"kb_used_meta":3828077,"kb_avail":1105707420,"statfs":{"total":1151940427776,"available":1132244398080,"internally_reserved":0,"allocated":15775793152,"data_stored":15768169119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":3919951161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.811602"},"pg_stats":[{"pgid":"5.f","version":"449'5410","reported_seq":5614,"reported_epoch":484,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:00.824904+0000","last_change":"2024-08-30T23:39:21.877768+0000","last_active":"2024-08-30T23:41:00.824904+0000","last_peered":"2024-08-30T23:41:00.824904+0000","last_clean":"2024-08-30T23:39:20.857133+0000","last_became_active":"2024-08-30T23:39:21.877768+0000","last_became_peered":"2024-08-30T23:39:21.877768+0000","last_unstale":"2024-08-30T23:41:00.824904+0000","last_undegraded":"2024-08-30T23:39:21.874443+0000","last_fullsized":"2024-08-30T23:39:21.874315+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:03.897134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300679168,"num_objects":4588,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":319936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":["11","2"],"object_location_counts":[{"shards":"2,11","objects":4588}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":107,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-08-30T23:40:57.614671+0000","last_change":"2024-08-30T23:40:26.468648+0000","last_active":"2024-08-30T23:40:57.614671+0000","last_peered":"2024-08-30T23:40:57.614671+0000","last_clean":"2024-08-30T23:40:57.614671+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:40:57.614671+0000","last_undegraded":"2024-08-30T23:40:57.614671+0000","last_fullsized":"2024-08-30T23:40:57.614671+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:26.468590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:26.468590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:56.142859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.e","version":"449'5233","reported_seq":5536,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-08-30T23:40:57.614880+0000","last_change":"2024-08-30T23:40:49.893551+0000","last_active":"2024-08-30T23:40:57.614880+0000","last_peered":"2024-08-30T23:40:57.614880+0000","last_clean":"2024-08-30T23:40:57.614880+0000","last_became_active":"2024-08-30T23:37:34.071585+0000","last_became_peered":"2024-08-30T23:37:34.071585+0000","last_unstale":"2024-08-30T23:40:57.614880+0000","last_undegraded":"2024-08-30T23:40:57.614880+0000","last_fullsized":"2024-08-30T23:40:57.614880+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5233","last_scrub_stamp":"2024-08-30T23:40:49.893457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.893457+0000","objects_scrubbed":4463,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:02.450977+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292487168,"num_objects":4463,"num_object_clones":0,"num_object_copies":13389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5233,"num_write_kb":310272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":989,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.798105+0000","last_change":"2024-08-30T23:40:49.616038+0000","last_active":"2024-08-30T23:41:01.798105+0000","last_peered":"2024-08-30T23:41:01.798105+0000","last_clean":"2024-08-30T23:41:01.798105+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:41:01.798105+0000","last_undegraded":"2024-08-30T23:41:01.798105+0000","last_fullsized":"2024-08-30T23:41:01.798105+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:49.615961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.615961+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:08.534735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.d","version":"449'5311","reported_seq":5563,"reported_epoch":484,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:00.824900+0000","last_change":"2024-08-30T23:39:21.875568+0000","last_active":"2024-08-30T23:41:00.824900+0000","last_peered":"2024-08-30T23:41:00.824900+0000","last_clean":"2024-08-30T23:39:20.866225+0000","last_became_active":"2024-08-30T23:39:21.875568+0000","last_became_peered":"2024-08-30T23:39:21.875568+0000","last_unstale":"2024-08-30T23:41:00.824900+0000","last_undegraded":"2024-08-30T23:39:21.874572+0000","last_fullsized":"2024-08-30T23:39:21.874461+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'3876","last_scrub_stamp":"2024-08-30T23:38:40.517849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:40.517849+0000","objects_scrubbed":3874,"log_size":5311,"log_dups_size":0,"ondisk_log_size":5311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299630616,"num_objects":4573,"num_object_clones":0,"num_object_copies":13719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5311,"num_write_kb":316225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","7"],"object_location_counts":[{"shards":"7,11","objects":4573}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":84,"reported_epoch":485,"state":"active+undersized","last_fresh":"2024-08-30T23:41:01.797292+0000","last_change":"2024-08-30T23:39:21.884892+0000","last_active":"2024-08-30T23:41:01.797292+0000","last_peered":"2024-08-30T23:41:01.797292+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:39:21.884892+0000","last_became_peered":"2024-08-30T23:39:21.884892+0000","last_unstale":"2024-08-30T23:41:01.797292+0000","last_undegraded":"2024-08-30T23:41:01.797292+0000","last_fullsized":"2024-08-30T23:39:21.873655+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:39:17.564717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:39:17.564717+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:41:04.962720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"449'5607","reported_seq":5921,"reported_epoch":483,"state":"active+clean","last_fresh":"2024-08-30T23:40:56.743924+0000","last_change":"2024-08-30T23:40:50.258887+0000","last_active":"2024-08-30T23:40:56.743924+0000","last_peered":"2024-08-30T23:40:56.743924+0000","last_clean":"2024-08-30T23:40:56.743924+0000","last_became_active":"2024-08-30T23:37:34.071745+0000","last_became_peered":"2024-08-30T23:37:34.071745+0000","last_unstale":"2024-08-30T23:40:56.743924+0000","last_undegraded":"2024-08-30T23:40:56.743924+0000","last_fullsized":"2024-08-30T23:40:56.743924+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5607","last_scrub_stamp":"2024-08-30T23:40:50.258779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.258779+0000","objects_scrubbed":4793,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:14.459973+0000","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5607,"num_write_kb":332800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"449'5473","reported_seq":5676,"reported_epoch":483,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:40:57.614713+0000","last_change":"2024-08-30T23:39:21.876317+0000","last_active":"2024-08-30T23:40:57.614713+0000","last_peered":"2024-08-30T23:40:57.614713+0000","last_clean":"2024-08-30T23:39:20.385995+0000","last_became_active":"2024-08-30T23:39:21.876317+0000","last_became_peered":"2024-08-30T23:39:21.876317+0000","last_unstale":"2024-08-30T23:40:57.614713+0000","last_undegraded":"2024-08-30T23:39:21.874534+0000","last_fullsized":"2024-08-30T23:39:21.874354+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:01.358484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":14265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5473,"num_write_kb":327296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[{"shards":"0,9","objects":4755}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.a","version":"449'5457","reported_seq":6493,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.032265+0000","last_change":"2024-08-30T23:40:46.105090+0000","last_active":"2024-08-30T23:41:01.032265+0000","last_peered":"2024-08-30T23:41:01.032265+0000","last_clean":"2024-08-30T23:41:01.032265+0000","last_became_active":"2024-08-30T23:39:53.469025+0000","last_became_peered":"2024-08-30T23:39:53.469025+0000","last_unstale":"2024-08-30T23:41:01.032265+0000","last_undegraded":"2024-08-30T23:41:01.032265+0000","last_fullsized":"2024-08-30T23:41:01.032265+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5457","last_scrub_stamp":"2024-08-30T23:40:46.105001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:46.105001+0000","objects_scrubbed":4671,"log_size":5457,"log_dups_size":0,"ondisk_log_size":5457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:14.401724+0000","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306118656,"num_objects":4671,"num_object_clones":0,"num_object_copies":14013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5457,"num_write_kb":324096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3714,"num_bytes_recovered":225574912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"449'5356","reported_seq":5614,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.798244+0000","last_change":"2024-08-30T23:40:46.194299+0000","last_active":"2024-08-30T23:41:01.798244+0000","last_peered":"2024-08-30T23:41:01.798244+0000","last_clean":"2024-08-30T23:41:01.798244+0000","last_became_active":"2024-08-30T23:37:34.038547+0000","last_became_peered":"2024-08-30T23:37:34.038547+0000","last_unstale":"2024-08-30T23:41:01.798244+0000","last_undegraded":"2024-08-30T23:41:01.798244+0000","last_fullsized":"2024-08-30T23:41:01.798244+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5356","last_scrub_stamp":"2024-08-30T23:40:46.194227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:46.194227+0000","objects_scrubbed":4614,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:01.798730+0000","scrub_duration":664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302383104,"num_objects":4614,"num_object_clones":0,"num_object_copies":13842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":319040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"449'5406","reported_seq":13003,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.032347+0000","last_change":"2024-08-30T23:40:53.177084+0000","last_active":"2024-08-30T23:41:01.032347+0000","last_peered":"2024-08-30T23:41:01.032347+0000","last_clean":"2024-08-30T23:41:01.032347+0000","last_became_active":"2024-08-30T23:39:21.876871+0000","last_became_peered":"2024-08-30T23:39:21.876871+0000","last_unstale":"2024-08-30T23:41:01.032347+0000","last_undegraded":"2024-08-30T23:41:01.032347+0000","last_fullsized":"2024-08-30T23:41:01.032347+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5406","last_scrub_stamp":"2024-08-30T23:40:53.176979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:53.176979+0000","objects_scrubbed":4596,"log_size":5406,"log_dups_size":0,"ondisk_log_size":5406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:17.547511+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301203456,"num_objects":4596,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":320064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3669,"num_bytes_recovered":240451584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":109,"reported_epoch":484,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:41:00.736463+0000","last_change":"2024-08-30T23:40:50.732043+0000","last_active":"2024-08-30T23:41:00.736463+0000","last_peered":"2024-08-30T23:41:00.736463+0000","last_clean":"2024-08-30T23:41:00.736463+0000","last_became_active":"2024-08-30T23:39:17.748045+0000","last_became_peered":"2024-08-30T23:39:17.748045+0000","last_unstale":"2024-08-30T23:41:00.736463+0000","last_undegraded":"2024-08-30T23:41:00.736463+0000","last_fullsized":"2024-08-30T23:41:00.736463+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:50.731969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.731969+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:13.207668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10,6],"avail_no_missing":["8","6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.7","version":"446'5508","reported_seq":5706,"reported_epoch":483,"state":"undersized+degraded+peered","last_fresh":"2024-08-30T23:40:57.614735+0000","last_change":"2024-08-30T23:39:21.874931+0000","last_active":"2024-08-30T23:39:20.873413+0000","last_peered":"2024-08-30T23:40:57.614735+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.039578+0000","last_became_peered":"2024-08-30T23:39:21.874931+0000","last_unstale":"2024-08-30T23:40:57.614735+0000","last_undegraded":"2024-08-30T23:39:21.874526+0000","last_fullsized":"2024-08-30T23:39:21.874442+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5508,"log_dups_size":0,"ondisk_log_size":5508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312868864,"num_objects":4774,"num_object_clones":0,"num_object_copies":14322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5508,"num_write_kb":329024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":4774}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":989,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.797906+0000","last_change":"2024-08-30T23:40:50.601518+0000","last_active":"2024-08-30T23:41:01.797906+0000","last_peered":"2024-08-30T23:41:01.797906+0000","last_clean":"2024-08-30T23:41:01.797906+0000","last_became_active":"2024-08-30T23:39:16.708792+0000","last_became_peered":"2024-08-30T23:39:16.708792+0000","last_unstale":"2024-08-30T23:41:01.797906+0000","last_undegraded":"2024-08-30T23:41:01.797906+0000","last_fullsized":"2024-08-30T23:41:01.797906+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:50.601452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.601452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:53.505995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"449'5354","reported_seq":5560,"reported_epoch":485,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:01.798073+0000","last_change":"2024-08-30T23:39:21.875697+0000","last_active":"2024-08-30T23:41:01.798073+0000","last_peered":"2024-08-30T23:41:01.798073+0000","last_clean":"2024-08-30T23:39:20.863070+0000","last_became_active":"2024-08-30T23:39:21.875697+0000","last_became_peered":"2024-08-30T23:39:21.875697+0000","last_unstale":"2024-08-30T23:41:01.798073+0000","last_undegraded":"2024-08-30T23:39:21.873746+0000","last_fullsized":"2024-08-30T23:39:21.873562+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5354,"log_dups_size":0,"ondisk_log_size":5354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":4544,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4544,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5354,"num_write_kb":316736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":4544}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":105,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.032260+0000","last_change":"2024-08-30T23:41:01.032260+0000","last_active":"2024-08-30T23:41:01.032260+0000","last_peered":"2024-08-30T23:41:01.032260+0000","last_clean":"2024-08-30T23:41:01.032260+0000","last_became_active":"2024-08-30T23:39:16.714172+0000","last_became_peered":"2024-08-30T23:39:16.714172+0000","last_unstale":"2024-08-30T23:41:01.032260+0000","last_undegraded":"2024-08-30T23:41:01.032260+0000","last_fullsized":"2024-08-30T23:41:01.032260+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:39:18.024617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:39:18.024617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:02.615306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.5","version":"449'5560","reported_seq":5760,"reported_epoch":485,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:01.797915+0000","last_change":"2024-08-30T23:37:42.007680+0000","last_active":"2024-08-30T23:41:01.797915+0000","last_peered":"2024-08-30T23:41:01.797915+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.071741+0000","last_became_peered":"2024-08-30T23:37:34.071741+0000","last_unstale":"2024-08-30T23:41:01.797915+0000","last_undegraded":"2024-08-30T23:37:42.007336+0000","last_fullsized":"2024-08-30T23:37:34.037577+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314048512,"num_objects":4792,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5560,"num_write_kb":331264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":4792}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":136,"reported_epoch":485,"state":"active+undersized","last_fresh":"2024-08-30T23:41:01.797925+0000","last_change":"2024-08-30T23:39:21.876060+0000","last_active":"2024-08-30T23:41:01.797925+0000","last_peered":"2024-08-30T23:41:01.797925+0000","last_clean":"2024-08-30T23:39:19.875655+0000","last_became_active":"2024-08-30T23:39:21.876060+0000","last_became_peered":"2024-08-30T23:39:21.876060+0000","last_unstale":"2024-08-30T23:41:01.797925+0000","last_undegraded":"2024-08-30T23:41:01.797925+0000","last_fullsized":"2024-08-30T23:39:21.872797+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:39:17.257778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:39:17.257778+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:02.881616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"449'5359","reported_seq":7579,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:00.736607+0000","last_change":"2024-08-30T23:40:49.976715+0000","last_active":"2024-08-30T23:41:00.736607+0000","last_peered":"2024-08-30T23:41:00.736607+0000","last_clean":"2024-08-30T23:41:00.736607+0000","last_became_active":"2024-08-30T23:40:35.810044+0000","last_became_peered":"2024-08-30T23:40:35.810044+0000","last_unstale":"2024-08-30T23:41:00.736607+0000","last_undegraded":"2024-08-30T23:41:00.736607+0000","last_fullsized":"2024-08-30T23:41:00.736607+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5359","last_scrub_stamp":"2024-08-30T23:40:49.976620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.976620+0000","objects_scrubbed":4583,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:05.351439+0000","scrub_duration":1280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300351488,"num_objects":4583,"num_object_clones":0,"num_object_copies":13749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5359,"num_write_kb":318144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3948,"num_bytes_recovered":258736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":110,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.032372+0000","last_change":"2024-08-30T23:40:55.609577+0000","last_active":"2024-08-30T23:41:01.032372+0000","last_peered":"2024-08-30T23:41:01.032372+0000","last_clean":"2024-08-30T23:41:01.032372+0000","last_became_active":"2024-08-30T23:39:16.713802+0000","last_became_peered":"2024-08-30T23:39:16.713802+0000","last_unstale":"2024-08-30T23:41:01.032372+0000","last_undegraded":"2024-08-30T23:41:01.032372+0000","last_fullsized":"2024-08-30T23:41:01.032372+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:55.609499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:55.609499+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:17.675158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"449'5320","reported_seq":12857,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.798308+0000","last_change":"2024-08-30T23:39:48.635566+0000","last_active":"2024-08-30T23:41:01.798308+0000","last_peered":"2024-08-30T23:41:01.798308+0000","last_clean":"2024-08-30T23:41:01.798308+0000","last_became_active":"2024-08-30T23:39:18.708292+0000","last_became_peered":"2024-08-30T23:39:18.708292+0000","last_unstale":"2024-08-30T23:41:01.798308+0000","last_undegraded":"2024-08-30T23:41:01.798308+0000","last_fullsized":"2024-08-30T23:41:01.798308+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5320","last_scrub_stamp":"2024-08-30T23:39:48.635484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:39:48.635484+0000","objects_scrubbed":4532,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:12.493340+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297009152,"num_objects":4532,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5320,"num_write_kb":315264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3631,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":110,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.032294+0000","last_change":"2024-08-30T23:40:56.616141+0000","last_active":"2024-08-30T23:41:01.032294+0000","last_peered":"2024-08-30T23:41:01.032294+0000","last_clean":"2024-08-30T23:41:01.032294+0000","last_became_active":"2024-08-30T23:39:16.724034+0000","last_became_peered":"2024-08-30T23:39:16.724034+0000","last_unstale":"2024-08-30T23:41:01.032294+0000","last_undegraded":"2024-08-30T23:41:01.032294+0000","last_fullsized":"2024-08-30T23:41:01.032294+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:56.616093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:56.616093+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:20.249226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.1","version":"449'5278","reported_seq":5580,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.797340+0000","last_change":"2024-08-30T23:40:28.061401+0000","last_active":"2024-08-30T23:41:01.797340+0000","last_peered":"2024-08-30T23:41:01.797340+0000","last_clean":"2024-08-30T23:41:01.797340+0000","last_became_active":"2024-08-30T23:37:34.043753+0000","last_became_peered":"2024-08-30T23:37:34.043753+0000","last_unstale":"2024-08-30T23:41:01.797340+0000","last_undegraded":"2024-08-30T23:41:01.797340+0000","last_fullsized":"2024-08-30T23:41:01.797340+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5278","last_scrub_stamp":"2024-08-30T23:40:28.061270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:28.061270+0000","objects_scrubbed":4566,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:48.939703+0000","scrub_duration":2178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299237376,"num_objects":4566,"num_object_clones":0,"num_object_copies":13698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5278,"num_write_kb":315008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":248,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-08-30T23:41:00.824901+0000","last_change":"2024-08-30T23:40:31.094070+0000","last_active":"2024-08-30T23:41:00.824901+0000","last_peered":"2024-08-30T23:41:00.824901+0000","last_clean":"2024-08-30T23:41:00.824901+0000","last_became_active":"2024-08-30T23:39:16.709980+0000","last_became_peered":"2024-08-30T23:39:16.709980+0000","last_unstale":"2024-08-30T23:41:00.824901+0000","last_undegraded":"2024-08-30T23:41:00.824901+0000","last_fullsized":"2024-08-30T23:41:00.824901+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:31.094018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:31.094018+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:45.291464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"449'5442","reported_seq":5691,"reported_epoch":484,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:41:01.032253+0000","last_change":"2024-08-30T23:40:55.652525+0000","last_active":"2024-08-30T23:41:01.032253+0000","last_peered":"2024-08-30T23:41:01.032253+0000","last_clean":"2024-08-30T23:41:01.032253+0000","last_became_active":"2024-08-30T23:38:34.126744+0000","last_became_peered":"2024-08-30T23:38:34.126744+0000","last_unstale":"2024-08-30T23:41:01.032253+0000","last_undegraded":"2024-08-30T23:41:01.032253+0000","last_fullsized":"2024-08-30T23:41:01.032253+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5442","last_scrub_stamp":"2024-08-30T23:40:55.652430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:55.652430+0000","objects_scrubbed":4616,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:07.158490+0000","scrub_duration":1018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302514176,"num_objects":4616,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4616,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":321856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4,0],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":4616}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.4","version":"449'5363","reported_seq":16295,"reported_epoch":484,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:41:01.032311+0000","last_change":"2024-08-30T23:40:54.104884+0000","last_active":"2024-08-30T23:41:01.032311+0000","last_peered":"2024-08-30T23:41:01.032311+0000","last_clean":"2024-08-30T23:41:01.032311+0000","last_became_active":"2024-08-30T23:39:22.924034+0000","last_became_peered":"2024-08-30T23:39:22.924034+0000","last_unstale":"2024-08-30T23:41:01.032311+0000","last_undegraded":"2024-08-30T23:41:01.032311+0000","last_fullsized":"2024-08-30T23:41:01.032311+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5363","last_scrub_stamp":"2024-08-30T23:40:54.104767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:54.104767+0000","objects_scrubbed":4579,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:18.481328+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300089344,"num_objects":4579,"num_object_clones":0,"num_object_copies":13737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4579,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5363,"num_write_kb":318144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5337,"num_bytes_recovered":325779456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2,8],"avail_no_missing":["10","2","8"],"object_location_counts":[{"shards":"2,8,10","objects":4579}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":984,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-08-30T23:41:01.798049+0000","last_change":"2024-08-30T23:40:47.613938+0000","last_active":"2024-08-30T23:41:01.798049+0000","last_peered":"2024-08-30T23:41:01.798049+0000","last_clean":"2024-08-30T23:41:01.798049+0000","last_became_active":"2024-08-30T23:39:14.854372+0000","last_became_peered":"2024-08-30T23:39:14.854372+0000","last_unstale":"2024-08-30T23:41:01.798049+0000","last_undegraded":"2024-08-30T23:41:01.798049+0000","last_fullsized":"2024-08-30T23:41:01.798049+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:47.613889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:47.613889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:09.886849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4852154392,"num_objects":74039,"num_object_clones":0,"num_object_copies":222117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32800,"num_objects_misplaced":9195,"num_objects_unfound":0,"num_objects_dirty":74039,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":86437,"num_write_kb":5135169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20299,"num_bytes_recovered":1288503296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14524624896,"data_stored":14524612680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86437,"ondisk_log_size":86437,"up":39,"acting":41,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":381350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":30,"acting":31,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1248052,"kb_used_data":883680,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92497100,"statfs":{"total":95995035648,"available":94717030400,"internally_reserved":0,"allocated":904888320,"data_stored":904201338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":373094852},"hb_peers":[0,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2026984,"kb_used_data":1766736,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":91718168,"statfs":{"total":95995035648,"available":93919404032,"internally_reserved":0,"allocated":1809137664,"data_stored":1808459178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":266467776},"hb_peers":[0,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710928,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1871656,"kb_used_data":1483664,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91873496,"statfs":{"total":95995035648,"available":94078459904,"internally_reserved":0,"allocated":1519271936,"data_stored":1518593450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[0,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972566,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1362544,"kb_used_data":1173084,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92382608,"statfs":{"total":95995035648,"available":94599790592,"internally_reserved":0,"allocated":1201238016,"data_stored":1200555114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234201,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1854388,"kb_used_data":1466400,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91890764,"statfs":{"total":95995035648,"available":94096142336,"internally_reserved":0,"allocated":1501593600,"data_stored":1500906626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":397277642},"hb_peers":[0,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528543,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1498352,"kb_used_data":1200604,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92246800,"statfs":{"total":95995035648,"available":94460723200,"internally_reserved":0,"allocated":1229418496,"data_stored":1228735594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304871878},"hb_peers":[0,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724750,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2374104,"kb_used_data":2072200,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91371048,"statfs":{"total":95995035648,"available":93563953152,"internally_reserved":0,"allocated":2121932800,"data_stored":2121416724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953705,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910060,"kb_used_data":603984,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92835092,"statfs":{"total":95995035648,"available":95063134208,"internally_reserved":0,"allocated":618479616,"data_stored":617801130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149934,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636424,"kb_used_data":294256,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93108728,"statfs":{"total":95995035648,"available":95343337472,"internally_reserved":0,"allocated":301318144,"data_stored":300627274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":350353863},"hb_peers":[0,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1408200,"kb_used_data":1178480,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92336952,"statfs":{"total":95995035648,"available":94553038848,"internally_reserved":0,"allocated":1206763520,"data_stored":1206072650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526392,"kb_used_data":1198120,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92218760,"statfs":{"total":95995035648,"available":94432010240,"internally_reserved":0,"allocated":1226874880,"data_stored":1226611199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738621,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2517248,"kb_used_data":2084840,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91227904,"statfs":{"total":95995035648,"available":93417373696,"internally_reserved":0,"allocated":2134876160,"data_stored":2134188842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2133000192,"data_stored":2133000192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1204682752,"data_stored":1204682752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":299237376,"data_stored":299237376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":616628224,"data_stored":616628224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2117668864,"data_stored":2117664792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1227554816,"data_stored":1227554816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1499402240,"data_stored":1499398168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1199046656,"data_stored":1199046656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1517420544,"data_stored":1517420544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1807286272,"data_stored":1807286272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":902696960,"data_stored":902692888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:41:05.040 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-30T23:41:05.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_3 pg_num 26 2024-08-30T23:41:05.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:04 smithi136 ceph-mon[20207]: from='client.17166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:41:05.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:05.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:04 smithi173 ceph-mon[20554]: from='client.17166 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:41:05.126 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:05.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:05.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:05.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:05.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:05.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:05.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:41:05.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:05.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:05.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:05.654 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:41:05.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:05.684 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:05.694 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:05.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:05.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:41:05.811 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:05.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:05.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:05.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:05.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:06.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:05 smithi173 ceph-mon[20554]: pgmap v1142: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:06.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:05 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:06.031 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:05 smithi173 ceph-mon[20554]: osdmap e486: 12 total, 10 up, 12 in 2024-08-30T23:41:06.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:06.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:05 smithi136 ceph-mon[20207]: pgmap v1142: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:06.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:05 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:05 smithi136 ceph-mon[20207]: osdmap e486: 12 total, 10 up, 12 in 2024-08-30T23:41:06.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:06.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:05 smithi106 ceph-mon[16603]: pgmap v1142: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:06.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:05 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:06.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:05 smithi106 ceph-mon[16603]: osdmap e486: 12 total, 10 up, 12 in 2024-08-30T23:41:06.139 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:06.139 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:06.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:06.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:06.279 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:06.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:06.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:06.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:06.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.429 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:06.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:06.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:06.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:06.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:06.710 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:41:06.710 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:41:06.710 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:41:06.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:41:06.731 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:41:06.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:41:06.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:41:06.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:41:06.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:41:06.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:41:06.781 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 5 pg_num to 26 2024-08-30T23:41:06.781 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:41:06.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:06.796 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:41:06.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:41:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:06.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:06.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:41:06.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:41:06.827 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:41:06.848 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:41:06.848 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:41:06.869 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:41:06.869 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:41:06.890 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:41:06.891 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:41:06.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:06.912 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:41:06.912 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:41:06.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:06.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:07.025 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/558141715' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:41:07.025 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2625342394' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-30T23:41:07.025 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2386413963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:07.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:07.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/558141715' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:41:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2625342394' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-30T23:41:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2386413963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:07.055 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/558141715' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:41:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2625342394' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"} : dispatch 2024-08-30T23:41:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2386413963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:07.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:07.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:07.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:07.306 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:07.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:07.423 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:07.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:07.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:07.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:07.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:07.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:07.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:07.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:07.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.806 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:07.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:07.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:07.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:08.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:08.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/558141715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:41:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2625342394' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-30T23:41:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:07 smithi173 ceph-mon[20554]: osdmap e487: 12 total, 10 up, 12 in 2024-08-30T23:41:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:07 smithi173 ceph-mon[20554]: pgmap v1145: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:07 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:41:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/558141715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:41:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2625342394' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-30T23:41:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:07 smithi106 ceph-mon[16603]: osdmap e487: 12 total, 10 up, 12 in 2024-08-30T23:41:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:07 smithi106 ceph-mon[16603]: pgmap v1145: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:07 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:41:08.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:08.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:08.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:08.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/558141715' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:41:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2625342394' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-08-30T23:41:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:07 smithi136 ceph-mon[20207]: osdmap e487: 12 total, 10 up, 12 in 2024-08-30T23:41:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:07 smithi136 ceph-mon[20207]: pgmap v1145: 27 pgs: 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:07 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:41:08.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:08.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:08.435 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:08.537 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:08.558 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:08.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:08.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:08.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:08.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:08 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-30T23:41:08.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:08 smithi106 ceph-mon[16603]: osdmap e488: 12 total, 10 up, 12 in 2024-08-30T23:41:08.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:08.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:08.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:08.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:09.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:09.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:08 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-30T23:41:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:08 smithi173 ceph-mon[20554]: osdmap e488: 12 total, 10 up, 12 in 2024-08-30T23:41:09.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:09.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:09.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:08 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-30T23:41:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:08 smithi136 ceph-mon[20207]: osdmap e488: 12 total, 10 up, 12 in 2024-08-30T23:41:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.305 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:09.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:09.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:41:09.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:41:09.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:09.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:09.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:09.614 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:41:09.614 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:41:09.614 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:41:09.615 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:41:09.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:09.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:09.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:09.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:09.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:09.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:09.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:10.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:10.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:09 smithi173 ceph-mon[20554]: osdmap e489: 12 total, 10 up, 12 in 2024-08-30T23:41:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:09 smithi173 ceph-mon[20554]: 5.1 scrub starts 2024-08-30T23:41:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:09 smithi173 ceph-mon[20554]: pgmap v1148: 37 pgs: 1 peering, 10 unknown, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:09 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:41:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:09 smithi106 ceph-mon[16603]: osdmap e489: 12 total, 10 up, 12 in 2024-08-30T23:41:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:09 smithi106 ceph-mon[16603]: 5.1 scrub starts 2024-08-30T23:41:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:09 smithi106 ceph-mon[16603]: pgmap v1148: 37 pgs: 1 peering, 10 unknown, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:09 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:41:10.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:10.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:10.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:09 smithi136 ceph-mon[20207]: osdmap e489: 12 total, 10 up, 12 in 2024-08-30T23:41:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:09 smithi136 ceph-mon[20207]: 5.1 scrub starts 2024-08-30T23:41:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:09 smithi136 ceph-mon[20207]: pgmap v1148: 37 pgs: 1 peering, 10 unknown, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 15 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32800/222120 objects degraded (14.767%); 9195/222120 objects misplaced (4.140%) 2024-08-30T23:41:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:09 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:41:10.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:10.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:41:10.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:10.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:10.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:41:10.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:10.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:10.635 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:10.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:10.672 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:10.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:10.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:10.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:10.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:10.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:11.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:11.034 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: 5.8 scrub starts 2024-08-30T23:41:11.035 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: 5.3 scrub starts 2024-08-30T23:41:11.035 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-30T23:41:11.035 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: osdmap e490: 12 total, 10 up, 12 in 2024-08-30T23:41:11.035 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: 5.1 scrub ok 2024-08-30T23:41:11.035 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: 5.8 scrub ok 2024-08-30T23:41:11.035 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:11.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:10 smithi106 ceph-mon[16603]: osdmap e491: 12 total, 10 up, 12 in 2024-08-30T23:41:11.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: 5.8 scrub starts 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: 5.3 scrub starts 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: osdmap e490: 12 total, 10 up, 12 in 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: 5.1 scrub ok 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: 5.8 scrub ok 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:10 smithi173 ceph-mon[20554]: osdmap e491: 12 total, 10 up, 12 in 2024-08-30T23:41:11.144 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:41:11.144 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:41:11.144 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:41:11.144 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:41:11.145 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:41:11.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:11.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:11.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: 5.8 scrub starts 2024-08-30T23:41:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: 5.3 scrub starts 2024-08-30T23:41:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-30T23:41:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: osdmap e490: 12 total, 10 up, 12 in 2024-08-30T23:41:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: 5.1 scrub ok 2024-08-30T23:41:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: 5.8 scrub ok 2024-08-30T23:41:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:10 smithi136 ceph-mon[20207]: osdmap e491: 12 total, 10 up, 12 in 2024-08-30T23:41:11.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:11.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:11.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:11.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:41:11.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:11.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:11.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:11.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:11.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:11.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:11.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:41:11.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:11.920 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:11.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:11.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:11.945 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:12.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:12.053 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:11 smithi106 ceph-mon[16603]: 5.4 scrub starts 2024-08-30T23:41:12.053 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:11 smithi106 ceph-mon[16603]: 5.9 scrub starts 2024-08-30T23:41:12.053 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:11 smithi106 ceph-mon[16603]: 5.3 scrub ok 2024-08-30T23:41:12.053 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:11 smithi106 ceph-mon[16603]: 5.4 scrub ok 2024-08-30T23:41:12.054 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:11 smithi106 ceph-mon[16603]: pgmap v1151: 37 pgs: 1 peering, 2 remapped+peering, 7 unknown, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28026/208434 objects degraded (13.446%); 4619/208434 objects misplaced (2.216%) 2024-08-30T23:41:12.054 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:11 smithi106 ceph-mon[16603]: osdmap e492: 12 total, 10 up, 12 in 2024-08-30T23:41:12.056 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:12.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:12.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:11 smithi173 ceph-mon[20554]: 5.4 scrub starts 2024-08-30T23:41:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:11 smithi173 ceph-mon[20554]: 5.9 scrub starts 2024-08-30T23:41:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:11 smithi173 ceph-mon[20554]: 5.3 scrub ok 2024-08-30T23:41:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:11 smithi173 ceph-mon[20554]: 5.4 scrub ok 2024-08-30T23:41:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:11 smithi173 ceph-mon[20554]: pgmap v1151: 37 pgs: 1 peering, 2 remapped+peering, 7 unknown, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28026/208434 objects degraded (13.446%); 4619/208434 objects misplaced (2.216%) 2024-08-30T23:41:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:11 smithi173 ceph-mon[20554]: osdmap e492: 12 total, 10 up, 12 in 2024-08-30T23:41:12.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:12.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:12.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:11 smithi136 ceph-mon[20207]: 5.4 scrub starts 2024-08-30T23:41:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:11 smithi136 ceph-mon[20207]: 5.9 scrub starts 2024-08-30T23:41:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:11 smithi136 ceph-mon[20207]: 5.3 scrub ok 2024-08-30T23:41:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:11 smithi136 ceph-mon[20207]: 5.4 scrub ok 2024-08-30T23:41:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:11 smithi136 ceph-mon[20207]: pgmap v1151: 37 pgs: 1 peering, 2 remapped+peering, 7 unknown, 2 active+undersized, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28026/208434 objects degraded (13.446%); 4619/208434 objects misplaced (2.216%) 2024-08-30T23:41:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:11 smithi136 ceph-mon[20207]: osdmap e492: 12 total, 10 up, 12 in 2024-08-30T23:41:12.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:12.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:12.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:12.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:12.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:12.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:12.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:12.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:12.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:12 smithi106 ceph-mon[16603]: 5.9 scrub ok 2024-08-30T23:41:12.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:12 smithi106 ceph-mon[16603]: 5.0 scrub starts 2024-08-30T23:41:12.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:12 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 28026/208434 objects degraded (13.446%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:12.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:12 smithi106 ceph-mon[16603]: 5.2 scrub starts 2024-08-30T23:41:12.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:12 smithi106 ceph-mon[16603]: 5.0 scrub ok 2024-08-30T23:41:12.835 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1401549732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:12.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:12.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:41:12.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:41:12.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:41:12.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:41:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:41:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:41:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:41:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:41:12.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:41:12.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:12.931 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:12.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:41:12.945 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:41:12.968 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:12.972 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:41:12.972 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:41:13.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:13.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:12 smithi173 ceph-mon[20554]: 5.9 scrub ok 2024-08-30T23:41:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:12 smithi173 ceph-mon[20554]: 5.0 scrub starts 2024-08-30T23:41:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:12 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 28026/208434 objects degraded (13.446%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:12 smithi173 ceph-mon[20554]: 5.2 scrub starts 2024-08-30T23:41:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:12 smithi173 ceph-mon[20554]: 5.0 scrub ok 2024-08-30T23:41:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1401549732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:13.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:13.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:13.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:12 smithi136 ceph-mon[20207]: 5.9 scrub ok 2024-08-30T23:41:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:12 smithi136 ceph-mon[20207]: 5.0 scrub starts 2024-08-30T23:41:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:12 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 28026/208434 objects degraded (13.446%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:12 smithi136 ceph-mon[20207]: 5.2 scrub starts 2024-08-30T23:41:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:12 smithi136 ceph-mon[20207]: 5.0 scrub ok 2024-08-30T23:41:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1401549732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:13.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:13.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:13.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:13.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:13.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:13.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:13.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:13.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:13.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:13.817 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:41:13.861 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1, 5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-08-30T23:41:13.861 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:41:13.862 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-30T23:41:13.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:41:13.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:13.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:13.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:13.993 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:13 smithi106 ceph-mon[16603]: 5.2 scrub ok 2024-08-30T23:41:13.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:13 smithi106 ceph-mon[16603]: 5.14 scrub starts 2024-08-30T23:41:13.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:13.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:13.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3165171814' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:13.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:13 smithi106 ceph-mon[16603]: pgmap v1153: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44453/215502 objects degraded (20.628%); 18219/215502 objects misplaced (8.454%) 2024-08-30T23:41:14.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:14.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:13 smithi173 ceph-mon[20554]: 5.2 scrub ok 2024-08-30T23:41:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:13 smithi173 ceph-mon[20554]: 5.14 scrub starts 2024-08-30T23:41:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3165171814' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:13 smithi173 ceph-mon[20554]: pgmap v1153: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44453/215502 objects degraded (20.628%); 18219/215502 objects misplaced (8.454%) 2024-08-30T23:41:14.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:14.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:14.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:13 smithi136 ceph-mon[20207]: 5.2 scrub ok 2024-08-30T23:41:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:13 smithi136 ceph-mon[20207]: 5.14 scrub starts 2024-08-30T23:41:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3165171814' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:13 smithi136 ceph-mon[20207]: pgmap v1153: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 1 unknown, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 44453/215502 objects degraded (20.628%); 18219/215502 objects misplaced (8.454%) 2024-08-30T23:41:14.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:14.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:14.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:14.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:14.533 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:14.555 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:14.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:14.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:14.727 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:14.727 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:14.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:14.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:14 smithi106 ceph-mon[16603]: 5.14 scrub ok 2024-08-30T23:41:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:14 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:41:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3165171814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:14 smithi106 ceph-mon[16603]: osdmap e493: 12 total, 10 up, 12 in 2024-08-30T23:41:14.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:14.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:14.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:15.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:15.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:14 smithi173 ceph-mon[20554]: 5.14 scrub ok 2024-08-30T23:41:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:14 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:41:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3165171814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:14 smithi173 ceph-mon[20554]: osdmap e493: 12 total, 10 up, 12 in 2024-08-30T23:41:15.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:15.174 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:15.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:14 smithi136 ceph-mon[20207]: 5.14 scrub ok 2024-08-30T23:41:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:14 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:41:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3165171814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:14 smithi136 ceph-mon[20207]: osdmap e493: 12 total, 10 up, 12 in 2024-08-30T23:41:15.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:15.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:15.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.426 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:15.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:15.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:15.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:41:15.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:15.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:15.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:15.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:15.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:15.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:15.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:16.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:16.050 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:15 smithi173 ceph-mon[20554]: pgmap v1155: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48910/222471 objects degraded (21.985%); 20456/222471 objects misplaced (9.195%) 2024-08-30T23:41:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:15 smithi173 ceph-mon[20554]: osdmap e494: 12 total, 10 up, 12 in 2024-08-30T23:41:16.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:15 smithi106 ceph-mon[16603]: pgmap v1155: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48910/222471 objects degraded (21.985%); 20456/222471 objects misplaced (9.195%) 2024-08-30T23:41:16.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:15 smithi106 ceph-mon[16603]: osdmap e494: 12 total, 10 up, 12 in 2024-08-30T23:41:16.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:16.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:16.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:16.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:15 smithi136 ceph-mon[20207]: pgmap v1155: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 16 active+clean; 4.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 48910/222471 objects degraded (21.985%); 20456/222471 objects misplaced (9.195%) 2024-08-30T23:41:16.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:15 smithi136 ceph-mon[20207]: osdmap e494: 12 total, 10 up, 12 in 2024-08-30T23:41:16.301 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:16.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:16.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:16.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:16.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:16.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:41:16.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:16.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:16.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:16.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:16.848 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:41:16.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:41:16.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:16.929 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:16.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:17.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:17.042 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:17.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:17.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:17.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:17.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:17.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:17.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:17.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:17.428 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:17.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:17.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:17.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:17.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:17.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:17.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:17.782 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:17.833 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:17.896 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:17 smithi106 ceph-mon[16603]: pgmap v1157: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53820/222198 objects degraded (24.222%); 24199/222198 objects misplaced (10.891%) 2024-08-30T23:41:17.896 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:17 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 53820/222198 objects degraded (24.222%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:17.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:17.961 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:17.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:17 smithi136 ceph-mon[20207]: pgmap v1157: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53820/222198 objects degraded (24.222%); 24199/222198 objects misplaced (10.891%) 2024-08-30T23:41:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:17 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 53820/222198 objects degraded (24.222%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:18.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:17 smithi173 ceph-mon[20554]: pgmap v1157: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53820/222198 objects degraded (24.222%); 24199/222198 objects misplaced (10.891%) 2024-08-30T23:41:18.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:17 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 53820/222198 objects degraded (24.222%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:18.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:18.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:18.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:18.220 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:41:18.221 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:41:18.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:18.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:18.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:18.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:18.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:18.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:18.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:18.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:18.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:18.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:18 smithi106 ceph-mon[16603]: from='client.17202 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:41:18.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:41:18.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:18.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:18.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:19.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:18 smithi136 ceph-mon[20207]: from='client.17202 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:41:19.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:19.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:18 smithi173 ceph-mon[20554]: from='client.17202 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:41:19.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:19.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:19.201 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:19.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:19.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:19.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:19.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:19.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:19.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:19.582 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1157,"stamp":"2024-08-30T23:41:17.557134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4854206910,"num_objects":74066,"num_object_clones":0,"num_object_copies":222198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53820,"num_objects_misplaced":24199,"num_objects_unfound":0,"num_objects_dirty":74066,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":65236,"num_write_kb":3862532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10928,"num_bytes_recovered":698334164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141638,"ondisk_log_size":141638,"up":92,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19449712,"kb_used_data":15468684,"kb_used_omap":18,"kb_used_meta":3980781,"kb_avail":1105492112,"statfs":{"total":1151940427776,"available":1132023922688,"internally_reserved":0,"allocated":15839932416,"data_stored":15831939164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":4076320057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":300711936,"num_objects":4588,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25794,"num_objects_misplaced":19580,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7692,"num_write_kb":-470112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1060,"num_bytes_recovered":-57507840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001341"},"pg_stats":[{"pgid":"5.19","version":"492'5362","reported_seq":6919,"reported_epoch":494,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:17.509282+0000","last_change":"2024-08-30T23:41:11.918499+0000","last_active":"2024-08-30T23:41:17.509282+0000","last_peered":"2024-08-30T23:41:17.509282+0000","last_clean":"2024-08-30T23:41:08.818941+0000","last_became_active":"2024-08-30T23:41:11.912523+0000","last_became_peered":"2024-08-30T23:41:11.912523+0000","last_unstale":"2024-08-30T23:41:17.509282+0000","last_undegraded":"2024-08-30T23:41:11.750556+0000","last_fullsized":"2024-08-30T23:41:11.742168+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5356","last_scrub_stamp":"2024-08-30T23:40:46.194227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:46.194227+0000","objects_scrubbed":4614,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150798336,"num_objects":2301,"num_object_clones":0,"num_object_copies":6903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":1579,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":159520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":634,"num_bytes_recovered":41549824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8],"acting":[7,6],"avail_no_missing":["7","6","11"],"object_location_counts":[{"shards":"6,7","objects":722},{"shards":"6,7,11","objects":1579}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"492'5412","reported_seq":12912,"reported_epoch":492,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:12.869612+0000","last_change":"2024-08-30T23:41:11.922780+0000","last_active":"2024-08-30T23:41:12.869612+0000","last_peered":"2024-08-30T23:41:12.869612+0000","last_clean":"2024-08-30T23:39:23.658243+0000","last_became_active":"2024-08-30T23:41:11.914359+0000","last_became_peered":"2024-08-30T23:41:11.914359+0000","last_unstale":"2024-08-30T23:41:12.869612+0000","last_undegraded":"2024-08-30T23:41:11.832906+0000","last_fullsized":"2024-08-30T23:41:11.742652+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5406","last_scrub_stamp":"2024-08-30T23:40:53.176979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:53.176979+0000","objects_scrubbed":0,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150208512,"num_objects":2292,"num_object_clones":0,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4773,"num_objects_misplaced":2481,"num_objects_unfound":0,"num_objects_dirty":2292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2709,"num_write_kb":160032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":120619008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[0,6],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,6","objects":21},{"shards":"0,9,10","objects":2271}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"492'5526","reported_seq":6185,"reported_epoch":493,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:14.256072+0000","last_change":"2024-08-30T23:41:11.933443+0000","last_active":"2024-08-30T23:41:14.256072+0000","last_peered":"2024-08-30T23:41:14.256072+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:41:11.909880+0000","last_became_peered":"2024-08-30T23:41:11.909880+0000","last_unstale":"2024-08-30T23:41:14.256072+0000","last_undegraded":"2024-08-30T23:41:11.860771+0000","last_fullsized":"2024-08-30T23:41:11.743511+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155254784,"num_objects":2369,"num_object_clones":0,"num_object_copies":7107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4472,"num_objects_misplaced":2237,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2772,"num_write_kb":164512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":7077888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[9,3],"avail_no_missing":["9"],"object_location_counts":[{"shards":"3,9","objects":132},{"shards":"3,9,11","objects":1},{"shards":"9","objects":2236}],"blocked_by":[],"up_primary":3,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.16","version":"492'5359","reported_seq":5858,"reported_epoch":492,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:13.359415+0000","last_change":"2024-08-30T23:41:11.868832+0000","last_active":"2024-08-30T23:41:13.359415+0000","last_peered":"2024-08-30T23:41:13.359415+0000","last_clean":"2024-08-30T23:39:20.863070+0000","last_became_active":"2024-08-30T23:41:11.849925+0000","last_became_peered":"2024-08-30T23:41:11.849925+0000","last_unstale":"2024-08-30T23:41:13.359415+0000","last_undegraded":"2024-08-30T23:41:11.859335+0000","last_fullsized":"2024-08-30T23:41:11.744009+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":6801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4534,"num_objects_misplaced":2100,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2682,"num_write_kb":158368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":11665408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":167}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"492'5563","reported_seq":5952,"reported_epoch":492,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:13.209344+0000","last_change":"2024-08-30T23:41:11.937294+0000","last_active":"2024-08-30T23:41:13.209344+0000","last_peered":"2024-08-30T23:41:13.209344+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:41:11.905359+0000","last_became_peered":"2024-08-30T23:41:11.905359+0000","last_unstale":"2024-08-30T23:41:13.209344+0000","last_undegraded":"2024-08-30T23:41:11.923607+0000","last_fullsized":"2024-08-30T23:41:11.742635+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156827648,"num_objects":2393,"num_object_clones":0,"num_object_copies":7179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4786,"num_objects_misplaced":2302,"num_objects_unfound":0,"num_objects_dirty":2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":8323072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":91}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"492'5371","reported_seq":16360,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.035447+0000","last_change":"2024-08-30T23:41:13.404655+0000","last_active":"2024-08-30T23:41:16.035447+0000","last_peered":"2024-08-30T23:41:16.035447+0000","last_clean":"2024-08-30T23:41:16.035447+0000","last_became_active":"2024-08-30T23:41:10.747773+0000","last_became_peered":"2024-08-30T23:41:10.747773+0000","last_unstale":"2024-08-30T23:41:16.035447+0000","last_undegraded":"2024-08-30T23:41:16.035447+0000","last_fullsized":"2024-08-30T23:41:16.035447+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5371","last_scrub_stamp":"2024-08-30T23:41:13.404571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:13.404571+0000","objects_scrubbed":2260,"log_size":5371,"log_dups_size":0,"ondisk_log_size":5371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:35.844314+0000","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.13","version":"492'5368","reported_seq":8424,"reported_epoch":494,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:41:15.742489+0000","last_change":"2024-08-30T23:41:11.874760+0000","last_active":"2024-08-30T23:41:15.742489+0000","last_peered":"2024-08-30T23:41:15.742489+0000","last_clean":"2024-08-30T23:41:06.785735+0000","last_became_active":"2024-08-30T23:41:11.850620+0000","last_became_peered":"2024-08-30T23:41:11.850620+0000","last_unstale":"2024-08-30T23:41:15.742489+0000","last_undegraded":"2024-08-30T23:41:15.742489+0000","last_fullsized":"2024-08-30T23:41:11.741437+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5359","last_scrub_stamp":"2024-08-30T23:40:49.976620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.976620+0000","objects_scrubbed":4583,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1280,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149585920,"num_objects":2283,"num_object_clones":0,"num_object_copies":6849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1805,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":159072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2381,"num_bytes_recovered":156041216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,10],"acting":[8,10],"avail_no_missing":["8","7","10"],"object_location_counts":[{"shards":"7,8,10","objects":1805},{"shards":"8,10","objects":478}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.12","version":"492'5323","reported_seq":12754,"reported_epoch":492,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:12.392613+0000","last_change":"2024-08-30T23:41:11.930167+0000","last_active":"2024-08-30T23:41:12.392613+0000","last_peered":"2024-08-30T23:41:12.392613+0000","last_clean":"2024-08-30T23:39:23.604420+0000","last_became_active":"2024-08-30T23:41:11.907266+0000","last_became_peered":"2024-08-30T23:41:11.907266+0000","last_unstale":"2024-08-30T23:41:12.392613+0000","last_undegraded":"2024-08-30T23:41:11.849155+0000","last_fullsized":"2024-08-30T23:41:11.742826+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5320","last_scrub_stamp":"2024-08-30T23:39:48.635484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:39:48.635484+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148307968,"num_objects":2263,"num_object_clones":0,"num_object_copies":6789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4712,"num_objects_misplaced":2449,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":157632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1819,"num_bytes_recovered":119177216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[0,7],"avail_no_missing":["0","6","10"],"object_location_counts":[{"shards":"0,6,10","objects":2244},{"shards":"0,7","objects":19}],"blocked_by":[],"up_primary":8,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":1003,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:15.741024+0000","last_change":"2024-08-30T23:40:47.613938+0000","last_active":"2024-08-30T23:41:15.741024+0000","last_peered":"2024-08-30T23:41:15.741024+0000","last_clean":"2024-08-30T23:41:15.741024+0000","last_became_active":"2024-08-30T23:39:14.854372+0000","last_became_peered":"2024-08-30T23:39:14.854372+0000","last_unstale":"2024-08-30T23:41:15.741024+0000","last_undegraded":"2024-08-30T23:41:15.741024+0000","last_fullsized":"2024-08-30T23:41:15.741024+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:47.613889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:47.613889+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:09.886849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"492'5368","reported_seq":16361,"reported_epoch":494,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:41:16.035447+0000","last_change":"2024-08-30T23:41:11.017070+0000","last_active":"2024-08-30T23:41:16.035447+0000","last_peered":"2024-08-30T23:41:16.035447+0000","last_clean":"2024-08-30T23:41:16.035447+0000","last_became_active":"2024-08-30T23:41:08.809599+0000","last_became_peered":"2024-08-30T23:41:08.809599+0000","last_unstale":"2024-08-30T23:41:16.035447+0000","last_undegraded":"2024-08-30T23:41:16.035447+0000","last_fullsized":"2024-08-30T23:41:16.035447+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5363","last_scrub_stamp":"2024-08-30T23:41:11.016975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:11.016975+0000","objects_scrubbed":2311,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:23.502992+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151126016,"num_objects":2306,"num_object_clones":0,"num_object_copies":6918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2306,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2,8],"avail_no_missing":["10","2","8"],"object_location_counts":[{"shards":"2,8,10","objects":2306}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"492'5453","reported_seq":5763,"reported_epoch":494,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:41:16.035071+0000","last_change":"2024-08-30T23:41:12.170496+0000","last_active":"2024-08-30T23:41:16.035071+0000","last_peered":"2024-08-30T23:41:16.035071+0000","last_clean":"2024-08-30T23:41:16.035071+0000","last_became_active":"2024-08-30T23:41:08.804773+0000","last_became_peered":"2024-08-30T23:41:08.804773+0000","last_unstale":"2024-08-30T23:41:16.035071+0000","last_undegraded":"2024-08-30T23:41:16.035071+0000","last_fullsized":"2024-08-30T23:41:16.035071+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'5449","last_scrub_stamp":"2024-08-30T23:41:12.170402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:12.170402+0000","objects_scrubbed":2333,"log_size":5453,"log_dups_size":0,"ondisk_log_size":5453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:21.406706+0000","scrub_duration":497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152371200,"num_objects":2325,"num_object_clones":0,"num_object_copies":6975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2325,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4],"acting":[10,4,0],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":2325}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":268,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.035805+0000","last_change":"2024-08-30T23:40:31.094070+0000","last_active":"2024-08-30T23:41:16.035805+0000","last_peered":"2024-08-30T23:41:16.035805+0000","last_clean":"2024-08-30T23:41:16.035805+0000","last_became_active":"2024-08-30T23:39:16.709980+0000","last_became_peered":"2024-08-30T23:39:16.709980+0000","last_unstale":"2024-08-30T23:41:16.035805+0000","last_undegraded":"2024-08-30T23:41:16.035805+0000","last_fullsized":"2024-08-30T23:41:16.035805+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:31.094018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:31.094018+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:45.291464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"492'5286","reported_seq":5642,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-08-30T23:41:12.685763+0000","last_change":"2024-08-30T23:41:10.050856+0000","last_active":"2024-08-30T23:41:12.685763+0000","last_peered":"2024-08-30T23:41:12.685763+0000","last_clean":"2024-08-30T23:41:12.685763+0000","last_became_active":"2024-08-30T23:41:08.808950+0000","last_became_peered":"2024-08-30T23:41:08.808950+0000","last_unstale":"2024-08-30T23:41:12.685763+0000","last_undegraded":"2024-08-30T23:41:12.685763+0000","last_fullsized":"2024-08-30T23:41:12.685763+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5278","last_scrub_stamp":"2024-08-30T23:41:10.050789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:10.050789+0000","objects_scrubbed":2310,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:38.328823+0000","scrub_duration":1117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150863872,"num_objects":2302,"num_object_clones":0,"num_object_copies":6906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":130,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.035340+0000","last_change":"2024-08-30T23:40:56.616141+0000","last_active":"2024-08-30T23:41:16.035340+0000","last_peered":"2024-08-30T23:41:16.035340+0000","last_clean":"2024-08-30T23:41:16.035340+0000","last_became_active":"2024-08-30T23:39:16.724034+0000","last_became_peered":"2024-08-30T23:39:16.724034+0000","last_unstale":"2024-08-30T23:41:16.035340+0000","last_undegraded":"2024-08-30T23:41:16.035340+0000","last_fullsized":"2024-08-30T23:41:16.035340+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:56.616093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:56.616093+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:20.249226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"492'5330","reported_seq":12926,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.034810+0000","last_change":"2024-08-30T23:41:12.175845+0000","last_active":"2024-08-30T23:41:16.034810+0000","last_peered":"2024-08-30T23:41:16.034810+0000","last_clean":"2024-08-30T23:41:16.034810+0000","last_became_active":"2024-08-30T23:41:08.807376+0000","last_became_peered":"2024-08-30T23:41:08.807376+0000","last_unstale":"2024-08-30T23:41:16.034810+0000","last_undegraded":"2024-08-30T23:41:16.034810+0000","last_fullsized":"2024-08-30T23:41:16.034810+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'5326","last_scrub_stamp":"2024-08-30T23:41:12.175620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:12.175620+0000","objects_scrubbed":2284,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:14.046911+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149094400,"num_objects":2275,"num_object_clones":0,"num_object_copies":6825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":130,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.035463+0000","last_change":"2024-08-30T23:40:55.609577+0000","last_active":"2024-08-30T23:41:16.035463+0000","last_peered":"2024-08-30T23:41:16.035463+0000","last_clean":"2024-08-30T23:41:16.035463+0000","last_became_active":"2024-08-30T23:39:16.713802+0000","last_became_peered":"2024-08-30T23:39:16.713802+0000","last_unstale":"2024-08-30T23:41:16.035463+0000","last_undegraded":"2024-08-30T23:41:16.035463+0000","last_fullsized":"2024-08-30T23:41:16.035463+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:55.609499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:55.609499+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:17.675158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"492'5365","reported_seq":7645,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:15.742704+0000","last_change":"2024-08-30T23:41:10.841288+0000","last_active":"2024-08-30T23:41:15.742704+0000","last_peered":"2024-08-30T23:41:15.742704+0000","last_clean":"2024-08-30T23:41:15.742704+0000","last_became_active":"2024-08-30T23:41:08.802906+0000","last_became_peered":"2024-08-30T23:41:08.802906+0000","last_unstale":"2024-08-30T23:41:15.742704+0000","last_undegraded":"2024-08-30T23:41:15.742704+0000","last_fullsized":"2024-08-30T23:41:15.742704+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5359","last_scrub_stamp":"2024-08-30T23:41:10.841189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:10.841189+0000","objects_scrubbed":2371,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:30.267024+0000","scrub_duration":1090,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154992640,"num_objects":2365,"num_object_clones":0,"num_object_copies":7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":154,"reported_epoch":494,"state":"active+undersized","last_fresh":"2024-08-30T23:41:16.034816+0000","last_change":"2024-08-30T23:39:21.876060+0000","last_active":"2024-08-30T23:41:16.034816+0000","last_peered":"2024-08-30T23:41:16.034816+0000","last_clean":"2024-08-30T23:39:19.875655+0000","last_became_active":"2024-08-30T23:39:21.876060+0000","last_became_peered":"2024-08-30T23:39:21.876060+0000","last_unstale":"2024-08-30T23:41:16.034816+0000","last_undegraded":"2024-08-30T23:41:16.034816+0000","last_fullsized":"2024-08-30T23:39:21.872797+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:39:17.257778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:39:17.257778+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"492'5565","reported_seq":5790,"reported_epoch":494,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:16.034898+0000","last_change":"2024-08-30T23:41:08.800566+0000","last_active":"2024-08-30T23:41:16.034898+0000","last_peered":"2024-08-30T23:41:16.034898+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:41:08.800566+0000","last_became_peered":"2024-08-30T23:41:08.800566+0000","last_unstale":"2024-08-30T23:41:16.034898+0000","last_undegraded":"2024-08-30T23:41:08.798588+0000","last_fullsized":"2024-08-30T23:41:08.798475+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156696576,"num_objects":2391,"num_object_clones":0,"num_object_copies":7173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2785,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":2391}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":125,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.034914+0000","last_change":"2024-08-30T23:41:01.032260+0000","last_active":"2024-08-30T23:41:16.034914+0000","last_peered":"2024-08-30T23:41:16.034914+0000","last_clean":"2024-08-30T23:41:16.034914+0000","last_became_active":"2024-08-30T23:39:16.714172+0000","last_became_peered":"2024-08-30T23:39:16.714172+0000","last_unstale":"2024-08-30T23:41:16.034914+0000","last_undegraded":"2024-08-30T23:41:16.034914+0000","last_fullsized":"2024-08-30T23:41:16.034914+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:39:18.024617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:39:18.024617+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"492'5367","reported_seq":5598,"reported_epoch":494,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:15.741216+0000","last_change":"2024-08-30T23:41:08.803040+0000","last_active":"2024-08-30T23:41:15.741216+0000","last_peered":"2024-08-30T23:41:15.741216+0000","last_clean":"2024-08-30T23:39:20.863070+0000","last_became_active":"2024-08-30T23:41:08.803040+0000","last_became_peered":"2024-08-30T23:41:08.803040+0000","last_unstale":"2024-08-30T23:41:15.741216+0000","last_undegraded":"2024-08-30T23:41:08.797881+0000","last_fullsized":"2024-08-30T23:41:08.797756+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5367,"log_dups_size":0,"ondisk_log_size":5367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:41:16.789617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148045824,"num_objects":2259,"num_object_clones":0,"num_object_copies":6777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":158368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":2259}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1007,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:15.741151+0000","last_change":"2024-08-30T23:40:50.601518+0000","last_active":"2024-08-30T23:41:15.741151+0000","last_peered":"2024-08-30T23:41:15.741151+0000","last_clean":"2024-08-30T23:41:15.741151+0000","last_became_active":"2024-08-30T23:39:16.708792+0000","last_became_peered":"2024-08-30T23:39:16.708792+0000","last_unstale":"2024-08-30T23:41:15.741151+0000","last_undegraded":"2024-08-30T23:41:15.741151+0000","last_fullsized":"2024-08-30T23:41:15.741151+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:50.601452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.601452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:53.505995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"446'5508","reported_seq":5729,"reported_epoch":493,"state":"undersized+degraded+peered","last_fresh":"2024-08-30T23:41:13.829356+0000","last_change":"2024-08-30T23:41:08.807007+0000","last_active":"2024-08-30T23:39:20.873413+0000","last_peered":"2024-08-30T23:41:13.829356+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:37:34.039578+0000","last_became_peered":"2024-08-30T23:41:08.807007+0000","last_unstale":"2024-08-30T23:41:13.829356+0000","last_undegraded":"2024-08-30T23:41:08.801822+0000","last_fullsized":"2024-08-30T23:41:08.801710+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5508,"log_dups_size":0,"ondisk_log_size":5508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156434432,"num_objects":2387,"num_object_clones":0,"num_object_copies":7161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2754,"num_write_kb":164512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"9","objects":2387}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":129,"reported_epoch":494,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:41:15.742489+0000","last_change":"2024-08-30T23:40:50.732043+0000","last_active":"2024-08-30T23:41:15.742489+0000","last_peered":"2024-08-30T23:41:15.742489+0000","last_clean":"2024-08-30T23:41:15.742489+0000","last_became_active":"2024-08-30T23:39:17.748045+0000","last_became_peered":"2024-08-30T23:39:17.748045+0000","last_unstale":"2024-08-30T23:41:15.742489+0000","last_undegraded":"2024-08-30T23:41:15.742489+0000","last_fullsized":"2024-08-30T23:41:15.742489+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:50.731969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.731969+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:13.207668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10,6],"avail_no_missing":["8","6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.8","version":"492'5415","reported_seq":13072,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.034954+0000","last_change":"2024-08-30T23:41:10.058306+0000","last_active":"2024-08-30T23:41:16.034954+0000","last_peered":"2024-08-30T23:41:16.034954+0000","last_clean":"2024-08-30T23:41:16.034954+0000","last_became_active":"2024-08-30T23:41:08.808304+0000","last_became_peered":"2024-08-30T23:41:08.808304+0000","last_unstale":"2024-08-30T23:41:16.034954+0000","last_undegraded":"2024-08-30T23:41:16.034954+0000","last_fullsized":"2024-08-30T23:41:16.034954+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5406","last_scrub_stamp":"2024-08-30T23:41:10.058231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:10.058231+0000","objects_scrubbed":2319,"log_size":5415,"log_dups_size":0,"ondisk_log_size":5415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:15.979103+0000","scrub_duration":328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151388160,"num_objects":2310,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,9],"acting":[10,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.9","version":"492'5368","reported_seq":5686,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:15.741202+0000","last_change":"2024-08-30T23:41:11.489609+0000","last_active":"2024-08-30T23:41:15.741202+0000","last_peered":"2024-08-30T23:41:15.741202+0000","last_clean":"2024-08-30T23:41:15.741202+0000","last_became_active":"2024-08-30T23:41:08.803797+0000","last_became_peered":"2024-08-30T23:41:08.803797+0000","last_unstale":"2024-08-30T23:41:15.741202+0000","last_undegraded":"2024-08-30T23:41:15.741202+0000","last_fullsized":"2024-08-30T23:41:15.741202+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5356","last_scrub_stamp":"2024-08-30T23:41:11.489550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:11.489550+0000","objects_scrubbed":2401,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:25.576527+0000","scrub_duration":747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565504,"num_objects":2389,"num_object_clones":0,"num_object_copies":7167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"492'5465","reported_seq":6524,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:16.035105+0000","last_change":"2024-08-30T23:40:46.105090+0000","last_active":"2024-08-30T23:41:16.035105+0000","last_peered":"2024-08-30T23:41:16.035105+0000","last_clean":"2024-08-30T23:41:16.035105+0000","last_became_active":"2024-08-30T23:39:53.469025+0000","last_became_peered":"2024-08-30T23:39:53.469025+0000","last_unstale":"2024-08-30T23:41:16.035105+0000","last_undegraded":"2024-08-30T23:41:16.035105+0000","last_fullsized":"2024-08-30T23:41:16.035105+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5457","last_scrub_stamp":"2024-08-30T23:40:46.105001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:46.105001+0000","objects_scrubbed":4671,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:02.986994+0000","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305594368,"num_objects":4663,"num_object_clones":0,"num_object_copies":13989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5465,"num_write_kb":324096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3714,"num_bytes_recovered":225574912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.b","version":"492'5488","reported_seq":5714,"reported_epoch":493,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:13.828894+0000","last_change":"2024-08-30T23:39:21.876317+0000","last_active":"2024-08-30T23:41:13.828894+0000","last_peered":"2024-08-30T23:41:13.828894+0000","last_clean":"2024-08-30T23:39:20.385995+0000","last_became_active":"2024-08-30T23:39:21.876317+0000","last_became_peered":"2024-08-30T23:39:21.876317+0000","last_unstale":"2024-08-30T23:41:13.828894+0000","last_undegraded":"2024-08-30T23:39:21.874534+0000","last_fullsized":"2024-08-30T23:39:21.874354+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310640640,"num_objects":4740,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4740,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":327296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[{"shards":"0,9","objects":4740}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"492'5623","reported_seq":5960,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-08-30T23:41:13.830004+0000","last_change":"2024-08-30T23:40:50.258887+0000","last_active":"2024-08-30T23:41:13.830004+0000","last_peered":"2024-08-30T23:41:13.830004+0000","last_clean":"2024-08-30T23:41:13.830004+0000","last_became_active":"2024-08-30T23:37:34.071745+0000","last_became_peered":"2024-08-30T23:37:34.071745+0000","last_unstale":"2024-08-30T23:41:13.830004+0000","last_undegraded":"2024-08-30T23:41:13.830004+0000","last_fullsized":"2024-08-30T23:41:13.830004+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5607","last_scrub_stamp":"2024-08-30T23:40:50.258779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.258779+0000","objects_scrubbed":4793,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:09.316418+0000","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313065472,"num_objects":4777,"num_object_clones":0,"num_object_copies":14331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":332800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":98,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-08-30T23:41:11.742391+0000","last_change":"2024-08-30T23:39:21.884892+0000","last_active":"2024-08-30T23:41:11.742391+0000","last_peered":"2024-08-30T23:41:11.742391+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:39:21.884892+0000","last_became_peered":"2024-08-30T23:39:21.884892+0000","last_unstale":"2024-08-30T23:41:11.742391+0000","last_undegraded":"2024-08-30T23:41:11.742391+0000","last_fullsized":"2024-08-30T23:39:21.873655+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:39:17.564717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:39:17.564717+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"492'5335","reported_seq":5610,"reported_epoch":494,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:16.035901+0000","last_change":"2024-08-30T23:39:21.875568+0000","last_active":"2024-08-30T23:41:16.035901+0000","last_peered":"2024-08-30T23:41:16.035901+0000","last_clean":"2024-08-30T23:39:20.866225+0000","last_became_active":"2024-08-30T23:39:21.875568+0000","last_became_peered":"2024-08-30T23:39:21.875568+0000","last_unstale":"2024-08-30T23:41:16.035901+0000","last_undegraded":"2024-08-30T23:39:21.874572+0000","last_fullsized":"2024-08-30T23:39:21.874461+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'3876","last_scrub_stamp":"2024-08-30T23:38:40.517849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:40.517849+0000","objects_scrubbed":3874,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057752,"num_objects":4549,"num_object_clones":0,"num_object_copies":13647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5335,"num_write_kb":316225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7],"acting":[11,7],"avail_no_missing":["11","7"],"object_location_counts":[{"shards":"7,11","objects":4549}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1007,"reported_epoch":494,"state":"active+clean","last_fresh":"2024-08-30T23:41:15.741041+0000","last_change":"2024-08-30T23:40:49.616038+0000","last_active":"2024-08-30T23:41:15.741041+0000","last_peered":"2024-08-30T23:41:15.741041+0000","last_clean":"2024-08-30T23:41:15.741041+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:41:15.741041+0000","last_undegraded":"2024-08-30T23:41:15.741041+0000","last_fullsized":"2024-08-30T23:41:15.741041+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:49.615961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.615961+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:08.534735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"492'5248","reported_seq":5574,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-08-30T23:41:13.828971+0000","last_change":"2024-08-30T23:40:49.893551+0000","last_active":"2024-08-30T23:41:13.828971+0000","last_peered":"2024-08-30T23:41:13.828971+0000","last_clean":"2024-08-30T23:41:13.828971+0000","last_became_active":"2024-08-30T23:37:34.071585+0000","last_became_peered":"2024-08-30T23:37:34.071585+0000","last_unstale":"2024-08-30T23:41:13.828971+0000","last_undegraded":"2024-08-30T23:41:13.828971+0000","last_fullsized":"2024-08-30T23:41:13.828971+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5233","last_scrub_stamp":"2024-08-30T23:40:49.893457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.893457+0000","objects_scrubbed":4463,"log_size":5248,"log_dups_size":0,"ondisk_log_size":5248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:54.990629+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291504128,"num_objects":4448,"num_object_clones":0,"num_object_copies":13344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5248,"num_write_kb":310272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":127,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-08-30T23:41:13.828850+0000","last_change":"2024-08-30T23:40:26.468648+0000","last_active":"2024-08-30T23:41:13.828850+0000","last_peered":"2024-08-30T23:41:13.828850+0000","last_clean":"2024-08-30T23:41:13.828850+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:41:13.828850+0000","last_undegraded":"2024-08-30T23:41:13.828850+0000","last_fullsized":"2024-08-30T23:41:13.828850+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:26.468590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:26.468590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:41:56.142859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"492'5421","reported_seq":5648,"reported_epoch":494,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:41:16.035914+0000","last_change":"2024-08-30T23:39:21.877768+0000","last_active":"2024-08-30T23:41:16.035914+0000","last_peered":"2024-08-30T23:41:16.035914+0000","last_clean":"2024-08-30T23:39:20.857133+0000","last_became_active":"2024-08-30T23:39:21.877768+0000","last_became_peered":"2024-08-30T23:39:21.877768+0000","last_unstale":"2024-08-30T23:41:16.035914+0000","last_undegraded":"2024-08-30T23:39:21.874443+0000","last_fullsized":"2024-08-30T23:39:21.874315+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299958272,"num_objects":4577,"num_object_clones":0,"num_object_copies":13731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5421,"num_write_kb":319936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":["11","2"],"object_location_counts":[{"shards":"2,11","objects":4577}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.10","version":"492'5451","reported_seq":5759,"reported_epoch":494,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:41:16.034894+0000","last_change":"2024-08-30T23:41:11.927060+0000","last_active":"2024-08-30T23:41:16.034894+0000","last_peered":"2024-08-30T23:41:16.034894+0000","last_clean":"2024-08-30T23:41:07.236730+0000","last_became_active":"2024-08-30T23:41:11.921995+0000","last_became_peered":"2024-08-30T23:41:11.921995+0000","last_unstale":"2024-08-30T23:41:16.034894+0000","last_undegraded":"2024-08-30T23:41:11.865506+0000","last_fullsized":"2024-08-30T23:41:11.746848+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5442","last_scrub_stamp":"2024-08-30T23:40:55.652430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:55.652430+0000","objects_scrubbed":4616,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1018,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150667264,"num_objects":2299,"num_object_clones":0,"num_object_copies":6897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4952,"num_objects_misplaced":2476,"num_objects_unfound":0,"num_objects_dirty":2299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":160928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,6],"acting":[10,6],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":2271},{"shards":"6,10","objects":28}],"blocked_by":[],"up_primary":7,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.11","version":"492'5286","reported_seq":5849,"reported_epoch":492,"state":"active+recovering+remapped","last_fresh":"2024-08-30T23:41:12.869528+0000","last_change":"2024-08-30T23:41:11.838695+0000","last_active":"2024-08-30T23:41:12.869528+0000","last_peered":"2024-08-30T23:41:12.869528+0000","last_clean":"2024-08-30T23:41:08.833246+0000","last_became_active":"2024-08-30T23:41:11.823071+0000","last_became_peered":"2024-08-30T23:41:11.823071+0000","last_unstale":"2024-08-30T23:41:12.869528+0000","last_undegraded":"2024-08-30T23:41:12.869528+0000","last_fullsized":"2024-08-30T23:41:12.869528+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5278","last_scrub_stamp":"2024-08-30T23:40:28.061270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:28.061270+0000","objects_scrubbed":4566,"log_size":5286,"log_dups_size":0,"ondisk_log_size":5286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149094400,"num_objects":2275,"num_object_clones":0,"num_object_copies":6825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2139,"num_objects_unfound":0,"num_objects_dirty":2275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2647,"num_write_kb":157504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":7667712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,7],"acting":[3,9,7],"avail_no_missing":["3","7","9"],"object_location_counts":[{"shards":"3,7,9","objects":136}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":4853825560,"num_objects":74065,"num_object_clones":0,"num_object_copies":222195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53820,"num_objects_misplaced":24199,"num_objects_unfound":0,"num_objects_dirty":74065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":65225,"num_write_kb":3862337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10928,"num_bytes_recovered":698286080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14608904192,"data_stored":14608891976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140626,"ondisk_log_size":140626,"up":62,"acting":63,"num_store_stats":11},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":381350,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2115712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":30,"acting":31,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187657,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1275356,"kb_used_data":888780,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":92469796,"statfs":{"total":95995035648,"available":94689071104,"internally_reserved":0,"allocated":910110720,"data_stored":909386416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":395835844},"hb_peers":[0,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449296,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2022300,"kb_used_data":1762056,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":91722852,"statfs":{"total":95995035648,"available":93924200448,"internally_reserved":0,"allocated":1804345344,"data_stored":1803625120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":266467776},"hb_peers":[0,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710931,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885152,"kb_used_data":1479112,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91860000,"statfs":{"total":95995035648,"available":94064640000,"internally_reserved":0,"allocated":1514610688,"data_stored":1513890464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":415758784},"hb_peers":[0,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972569,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1446684,"kb_used_data":1216968,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92298468,"statfs":{"total":95995035648,"available":94513631232,"internally_reserved":0,"allocated":1246175232,"data_stored":1245455008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1849260,"kb_used_data":1461272,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91895892,"statfs":{"total":95995035648,"available":94101393408,"internally_reserved":0,"allocated":1496342528,"data_stored":1495614081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":397277642},"hb_peers":[0,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528546,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1523688,"kb_used_data":1225940,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92221464,"statfs":{"total":95995035648,"available":94434779136,"internally_reserved":0,"allocated":1255362560,"data_stored":1254638185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":304871878},"hb_peers":[0,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724750,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2374104,"kb_used_data":2072200,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91371048,"statfs":{"total":95995035648,"available":93563953152,"internally_reserved":0,"allocated":2121932800,"data_stored":2121416724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953708,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908444,"kb_used_data":602376,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92836708,"statfs":{"total":95995035648,"available":95064788992,"internally_reserved":0,"allocated":616833024,"data_stored":616112800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149936,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":680856,"kb_used_data":302596,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":93064296,"statfs":{"total":95995035648,"available":95297839104,"internally_reserved":0,"allocated":309858304,"data_stored":309138629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":387316167},"hb_peers":[0,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1423128,"kb_used_data":1175364,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92322024,"statfs":{"total":95995035648,"available":94537752576,"internally_reserved":0,"allocated":1203572736,"data_stored":1202853061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526392,"kb_used_data":1198120,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92218760,"statfs":{"total":95995035648,"available":94432010240,"internally_reserved":0,"allocated":1226874880,"data_stored":1226611199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534348,"kb_used_data":2083900,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91210804,"statfs":{"total":95995035648,"available":93399863296,"internally_reserved":0,"allocated":2133913600,"data_stored":2133197477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2135621632,"data_stored":2135621632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1201405952,"data_stored":1201405952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312344576,"data_stored":312344576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":614858752,"data_stored":614858752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2117668864,"data_stored":2117664792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1253703680,"data_stored":1253703680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1494028288,"data_stored":1494024216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1254293504,"data_stored":1254293504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1512636416,"data_stored":1512636416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1802371072,"data_stored":1802371072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":909971456,"data_stored":909967384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:41:19.584 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-30T23:41:19.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-08-30T23:41:19.631 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:41:19.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:19.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:19.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:19.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3572020009' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:41:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:19 smithi106 ceph-mon[16603]: pgmap v1158: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53820/222198 objects degraded (24.222%); 22678/222198 objects misplaced (10.206%); 5.1 MiB/s, 58 objects/s recovering 2024-08-30T23:41:19.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:19.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:19.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3572020009' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:41:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:19 smithi136 ceph-mon[20207]: pgmap v1158: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53820/222198 objects degraded (24.222%); 22678/222198 objects misplaced (10.206%); 5.1 MiB/s, 58 objects/s recovering 2024-08-30T23:41:20.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:20.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3572020009' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:41:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:19 smithi173 ceph-mon[20554]: pgmap v1158: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53820/222198 objects degraded (24.222%); 22678/222198 objects misplaced (10.206%); 5.1 MiB/s, 58 objects/s recovering 2024-08-30T23:41:20.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:20.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:20.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:20.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:20.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:20.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:20.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:20.567 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:20.669 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:20.690 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:20.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.814 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:20.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:20.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:20.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:20.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:21.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:20 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3572020009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:41:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:20 smithi136 ceph-mon[20207]: osdmap e495: 12 total, 10 up, 12 in 2024-08-30T23:41:21.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:20 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:21.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3572020009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:41:21.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:20 smithi173 ceph-mon[20554]: osdmap e495: 12 total, 10 up, 12 in 2024-08-30T23:41:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:20 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3572020009' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:41:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:20 smithi106 ceph-mon[16603]: osdmap e495: 12 total, 10 up, 12 in 2024-08-30T23:41:21.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:21.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:21.291 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:21.291 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:21.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:21.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:21.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:21.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:21.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:21.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:21.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:21.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:21.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:21.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:21.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:21.963 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:21 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:21 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:21 smithi136 ceph-mon[20207]: osdmap e496: 12 total, 10 up, 12 in 2024-08-30T23:41:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:21 smithi136 ceph-mon[20207]: pgmap v1161: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53271/222198 objects degraded (23.975%); 21943/222198 objects misplaced (9.875%) 2024-08-30T23:41:22.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:21 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:21 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:21 smithi173 ceph-mon[20554]: osdmap e496: 12 total, 10 up, 12 in 2024-08-30T23:41:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:21 smithi173 ceph-mon[20554]: pgmap v1161: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53271/222198 objects degraded (23.975%); 21943/222198 objects misplaced (9.875%) 2024-08-30T23:41:22.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:22.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:21 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:22.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:21 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:22.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:21 smithi106 ceph-mon[16603]: osdmap e496: 12 total, 10 up, 12 in 2024-08-30T23:41:22.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:21 smithi106 ceph-mon[16603]: pgmap v1161: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53271/222198 objects degraded (23.975%); 21943/222198 objects misplaced (9.875%) 2024-08-30T23:41:22.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:22.212 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:22.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:22.339 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:22.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:22.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:22.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:22.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:22.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:22.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:22.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:22.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:22.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:22.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:22.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:22.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:22.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:23.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:22 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:23.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:22 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:23.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:22 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:23.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:22 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:23.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:22 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:22 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:23.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:23.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:23.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:23.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:23.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:23.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:23.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:23.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:23.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:23.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:23.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:23.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:23.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:23.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:23.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:24.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:23 smithi136 ceph-mon[20207]: osdmap e497: 12 total, 10 up, 12 in 2024-08-30T23:41:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:23 smithi136 ceph-mon[20207]: pgmap v1163: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53271/222198 objects degraded (23.975%); 19555/222198 objects misplaced (8.801%); 48 MiB/s, 773 objects/s recovering 2024-08-30T23:41:24.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:23 smithi173 ceph-mon[20554]: osdmap e497: 12 total, 10 up, 12 in 2024-08-30T23:41:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:23 smithi173 ceph-mon[20554]: pgmap v1163: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53271/222198 objects degraded (23.975%); 19555/222198 objects misplaced (8.801%); 48 MiB/s, 773 objects/s recovering 2024-08-30T23:41:24.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:23 smithi106 ceph-mon[16603]: osdmap e497: 12 total, 10 up, 12 in 2024-08-30T23:41:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:23 smithi106 ceph-mon[16603]: pgmap v1163: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 53271/222198 objects degraded (23.975%); 19555/222198 objects misplaced (8.801%); 48 MiB/s, 773 objects/s recovering 2024-08-30T23:41:24.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:24.199 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:24.301 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:24.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:24.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:24.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:24.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:24.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.573 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:24.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:24.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:24.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:24.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:24.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:24.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:24.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:25.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:25.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:25.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:25.192 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:25.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:25.315 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:25.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:25.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:25.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:25.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:25.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:25.686 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:25.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:25.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:25.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:25.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:25.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:26.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:25 smithi136 ceph-mon[20207]: pgmap v1164: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 19395/222198 objects misplaced (8.729%); 34 MiB/s, 547 objects/s recovering 2024-08-30T23:41:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:25 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 52952/222198 objects degraded (23.831%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:26.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:25 smithi173 ceph-mon[20554]: pgmap v1164: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 19395/222198 objects misplaced (8.729%); 34 MiB/s, 547 objects/s recovering 2024-08-30T23:41:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:25 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 52952/222198 objects degraded (23.831%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:26.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:25 smithi106 ceph-mon[16603]: pgmap v1164: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 19395/222198 objects misplaced (8.729%); 34 MiB/s, 547 objects/s recovering 2024-08-30T23:41:26.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:25 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 52952/222198 objects degraded (23.831%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:26.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:26.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:26.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:26.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.306 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:26.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:26.428 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:26.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:26.556 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:26.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:26.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:26.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:26.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:26.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:26.933 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:26.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:27.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:26 smithi136 ceph-mon[20207]: osdmap e498: 12 total, 10 up, 12 in 2024-08-30T23:41:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:27.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:27.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:26 smithi173 ceph-mon[20554]: osdmap e498: 12 total, 10 up, 12 in 2024-08-30T23:41:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:26 smithi106 ceph-mon[16603]: osdmap e498: 12 total, 10 up, 12 in 2024-08-30T23:41:27.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:27.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:27.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:27.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:27.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:27.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:27.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:27.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:27.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:27.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:27.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:27.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:27.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:27.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:27.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:28.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:27 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:27 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:27 smithi136 ceph-mon[20207]: osdmap e499: 12 total, 10 up, 12 in 2024-08-30T23:41:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:27 smithi136 ceph-mon[20207]: pgmap v1167: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 18401/222198 objects misplaced (8.281%); 37 MiB/s, 590 objects/s recovering 2024-08-30T23:41:28.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:27 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:27 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:27 smithi173 ceph-mon[20554]: osdmap e499: 12 total, 10 up, 12 in 2024-08-30T23:41:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:27 smithi173 ceph-mon[20554]: pgmap v1167: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 18401/222198 objects misplaced (8.281%); 37 MiB/s, 590 objects/s recovering 2024-08-30T23:41:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:27 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:27 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:27 smithi106 ceph-mon[16603]: osdmap e499: 12 total, 10 up, 12 in 2024-08-30T23:41:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:27 smithi106 ceph-mon[16603]: pgmap v1167: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 18401/222198 objects misplaced (8.281%); 37 MiB/s, 590 objects/s recovering 2024-08-30T23:41:28.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:28.187 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:28.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:28.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:28.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:28.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:28.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:28.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:28.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:28.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:28.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:28.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:28.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:28.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:28.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:29.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:28 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:28 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:28 smithi136 ceph-mon[20207]: osdmap e500: 12 total, 10 up, 12 in 2024-08-30T23:41:29.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:28 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:28 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:29.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:28 smithi173 ceph-mon[20554]: osdmap e500: 12 total, 10 up, 12 in 2024-08-30T23:41:29.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:28 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:29.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:28 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:29.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:28 smithi106 ceph-mon[16603]: osdmap e500: 12 total, 10 up, 12 in 2024-08-30T23:41:29.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:29.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:29.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:29.298 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:29.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:29.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:29.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:29.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:29.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:29.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:29.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:29.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:29.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:29.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:29.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:30.012 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:29 smithi106 ceph-mon[16603]: pgmap v1169: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 16564/222198 objects misplaced (7.455%); 31 MiB/s, 498 objects/s recovering 2024-08-30T23:41:30.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:29 smithi136 ceph-mon[20207]: pgmap v1169: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 16564/222198 objects misplaced (7.455%); 31 MiB/s, 498 objects/s recovering 2024-08-30T23:41:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:30.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:29 smithi173 ceph-mon[20554]: pgmap v1169: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52952/222198 objects degraded (23.831%); 16564/222198 objects misplaced (7.455%); 31 MiB/s, 498 objects/s recovering 2024-08-30T23:41:30.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.164 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:30.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:30.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:30.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:30.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:30.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:30.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:30.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:30.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:30.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:30.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:30.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:30.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:30.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:31.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:31.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:31.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:31.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:31.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:31.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:31.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:31.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:31.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.530 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:31.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:31.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:31.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:31.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:31.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:31.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:31.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:31.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:32.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:32.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:31 smithi136 ceph-mon[20207]: osdmap e501: 12 total, 10 up, 12 in 2024-08-30T23:41:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:31 smithi136 ceph-mon[20207]: pgmap v1171: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52293/222198 objects degraded (23.534%); 15435/222198 objects misplaced (6.947%); 43 MiB/s, 680 objects/s recovering 2024-08-30T23:41:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:31 smithi173 ceph-mon[20554]: osdmap e501: 12 total, 10 up, 12 in 2024-08-30T23:41:32.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:31 smithi173 ceph-mon[20554]: pgmap v1171: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52293/222198 objects degraded (23.534%); 15435/222198 objects misplaced (6.947%); 43 MiB/s, 680 objects/s recovering 2024-08-30T23:41:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:31 smithi106 ceph-mon[16603]: osdmap e501: 12 total, 10 up, 12 in 2024-08-30T23:41:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:31 smithi106 ceph-mon[16603]: pgmap v1171: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52293/222198 objects degraded (23.534%); 15435/222198 objects misplaced (6.947%); 43 MiB/s, 680 objects/s recovering 2024-08-30T23:41:32.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:32.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:32.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:32.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:32.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.400 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:32.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:32.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:32.523 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:32.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:32.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:32.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:32.770 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:32.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:41:32.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:32.873 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:32.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:32.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:32.897 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:32 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 52293/222198 objects degraded (23.534%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:32.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:33.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:32 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 52293/222198 objects degraded (23.534%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:32 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 52293/222198 objects degraded (23.534%), 13 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:33.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:33.144 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:33.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:33.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:33.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.269 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:33.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:33.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:33.432 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:41:33.432 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:41:33.433 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:41:33.462 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:41:33.462 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:41:33.473 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:33.480 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:33.491 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:41:33.491 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:41:33.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:33.513 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:33.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:41:33.518 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:41:33.519 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:41:33.541 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:41:33.541 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:41:33.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:41:33.564 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:41:33.592 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:41:33.592 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:41:33.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:33.620 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:41:33.620 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:41:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:33.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:33.642 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:41:33.642 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:41:33.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:41:33.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:41:33.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:33.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:33.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:33.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:33.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:33.946 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:33 smithi106 ceph-mon[16603]: osdmap e502: 12 total, 10 up, 12 in 2024-08-30T23:41:33.946 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2257407977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:33.946 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:33.947 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:33 smithi106 ceph-mon[16603]: pgmap v1173: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45278/222198 objects degraded (20.377%); 20190/222198 objects misplaced (9.086%); 49 MiB/s, 838 objects/s recovering 2024-08-30T23:41:33.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:34.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:33 smithi136 ceph-mon[20207]: osdmap e502: 12 total, 10 up, 12 in 2024-08-30T23:41:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2257407977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:33 smithi136 ceph-mon[20207]: pgmap v1173: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45278/222198 objects degraded (20.377%); 20190/222198 objects misplaced (9.086%); 49 MiB/s, 838 objects/s recovering 2024-08-30T23:41:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:33 smithi173 ceph-mon[20554]: osdmap e502: 12 total, 10 up, 12 in 2024-08-30T23:41:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2257407977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:33 smithi173 ceph-mon[20554]: pgmap v1173: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45278/222198 objects degraded (20.377%); 20190/222198 objects misplaced (9.086%); 49 MiB/s, 838 objects/s recovering 2024-08-30T23:41:34.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:34.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:34.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:34.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:34.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:34.288 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:41:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:41:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:41:34.289 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:41:34.289 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:41:34.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:34.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:34.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:34.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:34.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:34.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:34.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:41:34.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:34.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:34.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:34.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:34.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:34.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:35.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:35.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:34 smithi106 ceph-mon[16603]: osdmap e503: 12 total, 10 up, 12 in 2024-08-30T23:41:35.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:34 smithi106 ceph-mon[16603]: 5.13 scrub starts 2024-08-30T23:41:35.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:35.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:35.037 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:41:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:34 smithi136 ceph-mon[20207]: osdmap e503: 12 total, 10 up, 12 in 2024-08-30T23:41:35.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:34 smithi136 ceph-mon[20207]: 5.13 scrub starts 2024-08-30T23:41:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:41:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:34 smithi173 ceph-mon[20554]: osdmap e503: 12 total, 10 up, 12 in 2024-08-30T23:41:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:34 smithi173 ceph-mon[20554]: 5.13 scrub starts 2024-08-30T23:41:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:41:35.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:35.125 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:35.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:35.252 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:35.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:35.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:35.419 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:41:35.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:41:35.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:35.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:35.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:35.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.624 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:35.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:35.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:35.849 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:35.873 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:35.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:35.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:35.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:35 smithi173 ceph-mon[20554]: 5.11 scrub starts 2024-08-30T23:41:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:35 smithi173 ceph-mon[20554]: osdmap e504: 12 total, 10 up, 12 in 2024-08-30T23:41:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:35 smithi173 ceph-mon[20554]: 5.16 scrub starts 2024-08-30T23:41:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:35 smithi173 ceph-mon[20554]: pgmap v1176: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44747/222198 objects degraded (20.138%); 19925/222198 objects misplaced (8.967%); 37 MiB/s, 651 objects/s recovering 2024-08-30T23:41:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:35 smithi173 ceph-mon[20554]: osdmap e505: 12 total, 10 up, 12 in 2024-08-30T23:41:36.101 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:35 smithi106 ceph-mon[16603]: 5.11 scrub starts 2024-08-30T23:41:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:35 smithi106 ceph-mon[16603]: osdmap e504: 12 total, 10 up, 12 in 2024-08-30T23:41:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:35 smithi106 ceph-mon[16603]: 5.16 scrub starts 2024-08-30T23:41:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:35 smithi106 ceph-mon[16603]: pgmap v1176: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44747/222198 objects degraded (20.138%); 19925/222198 objects misplaced (8.967%); 37 MiB/s, 651 objects/s recovering 2024-08-30T23:41:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:35 smithi106 ceph-mon[16603]: osdmap e505: 12 total, 10 up, 12 in 2024-08-30T23:41:36.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:36.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:36.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:35 smithi136 ceph-mon[20207]: 5.11 scrub starts 2024-08-30T23:41:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:35 smithi136 ceph-mon[20207]: osdmap e504: 12 total, 10 up, 12 in 2024-08-30T23:41:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:35 smithi136 ceph-mon[20207]: 5.16 scrub starts 2024-08-30T23:41:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:35 smithi136 ceph-mon[20207]: pgmap v1176: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 17 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 44747/222198 objects degraded (20.138%); 19925/222198 objects misplaced (8.967%); 37 MiB/s, 651 objects/s recovering 2024-08-30T23:41:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:35 smithi136 ceph-mon[20207]: osdmap e505: 12 total, 10 up, 12 in 2024-08-30T23:41:36.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:36.371 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:36.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:41:36.430 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:36.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:36.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:36.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:36.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:36.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:36.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:36.762 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pg_num to 21 2024-08-30T23:41:36.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:36.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:36.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:36.989 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:36.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3610141168' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-30T23:41:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:36 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-30T23:41:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:36 smithi173 ceph-mon[20554]: 5.19 scrub starts 2024-08-30T23:41:37.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:36 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-30T23:41:37.077 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:36 smithi173 ceph-mon[20554]: osdmap e506: 12 total, 10 up, 12 in 2024-08-30T23:41:37.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3610141168' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-30T23:41:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:36 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-30T23:41:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:36 smithi106 ceph-mon[16603]: 5.19 scrub starts 2024-08-30T23:41:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:36 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-30T23:41:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:36 smithi106 ceph-mon[16603]: osdmap e506: 12 total, 10 up, 12 in 2024-08-30T23:41:37.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:37.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:37.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:37.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3610141168' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-30T23:41:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:36 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-30T23:41:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:36 smithi136 ceph-mon[20207]: 5.19 scrub starts 2024-08-30T23:41:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:36 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-30T23:41:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:36 smithi136 ceph-mon[20207]: osdmap e506: 12 total, 10 up, 12 in 2024-08-30T23:41:37.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:37.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:37.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:41:37.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.488 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:37.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:37.507 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:37.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:37.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:37.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:37.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:37.796 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:41:37.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:37.864 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:37.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:37.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:37.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:38.050 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:37 smithi106 ceph-mon[16603]: 5.15 scrub starts 2024-08-30T23:41:38.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3555978469' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:38.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:37 smithi106 ceph-mon[16603]: pgmap v1179: 37 pgs: 1 creating+remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32708/222198 objects degraded (14.720%); 26255/222198 objects misplaced (11.816%); 8.0 MiB/s, 292 objects/s recovering 2024-08-30T23:41:38.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-08-30T23:41:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:37 smithi173 ceph-mon[20554]: 5.15 scrub starts 2024-08-30T23:41:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3555978469' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:37 smithi173 ceph-mon[20554]: pgmap v1179: 37 pgs: 1 creating+remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32708/222198 objects degraded (14.720%); 26255/222198 objects misplaced (11.816%); 8.0 MiB/s, 292 objects/s recovering 2024-08-30T23:41:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-08-30T23:41:38.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:38.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:38.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:38.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:38.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:37 smithi136 ceph-mon[20207]: 5.15 scrub starts 2024-08-30T23:41:38.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3555978469' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:41:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:37 smithi136 ceph-mon[20207]: pgmap v1179: 37 pgs: 1 creating+remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 1 active+undersized+degraded+remapped, 4 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 32708/222198 objects degraded (14.720%); 26255/222198 objects misplaced (11.816%); 8.0 MiB/s, 292 objects/s recovering 2024-08-30T23:41:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-08-30T23:41:38.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:38.357 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:38.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:38.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:38.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.606 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:38.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:38.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:38.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:38.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:38.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.856 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:38.957 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:38.979 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:38.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:38 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 32708/222198 objects degraded (14.720%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:38 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3555978469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-30T23:41:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:38 smithi173 ceph-mon[20554]: osdmap e507: 12 total, 10 up, 12 in 2024-08-30T23:41:39.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:39.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:38 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 32708/222198 objects degraded (14.720%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:38 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3555978469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-30T23:41:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:38 smithi106 ceph-mon[16603]: osdmap e507: 12 total, 10 up, 12 in 2024-08-30T23:41:39.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:39.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:38 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 32708/222198 objects degraded (14.720%), 10 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:38 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3555978469' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:41:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-30T23:41:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:38 smithi136 ceph-mon[20207]: osdmap e507: 12 total, 10 up, 12 in 2024-08-30T23:41:39.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:39.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:39.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:39.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:39.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:39.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:39.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:39.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:39.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:39.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:39.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:39.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:40.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:39 smithi173 ceph-mon[20554]: osdmap e508: 12 total, 10 up, 12 in 2024-08-30T23:41:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:39 smithi173 ceph-mon[20554]: 5.13 scrub starts 2024-08-30T23:41:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:39 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:41:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:39 smithi173 ceph-mon[20554]: pgmap v1182: 47 pgs: 10 unknown, 1 creating+remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 5 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 30259/222198 objects degraded (13.618%); 27028/222198 objects misplaced (12.164%); 14 MiB/s, 384 objects/s recovering 2024-08-30T23:41:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:39 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-30T23:41:40.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:39 smithi106 ceph-mon[16603]: osdmap e508: 12 total, 10 up, 12 in 2024-08-30T23:41:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:39 smithi106 ceph-mon[16603]: 5.13 scrub starts 2024-08-30T23:41:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:39 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:41:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:39 smithi106 ceph-mon[16603]: pgmap v1182: 47 pgs: 10 unknown, 1 creating+remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 5 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 30259/222198 objects degraded (13.618%); 27028/222198 objects misplaced (12.164%); 14 MiB/s, 384 objects/s recovering 2024-08-30T23:41:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:39 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-30T23:41:40.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:40.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.219 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:40.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:39 smithi136 ceph-mon[20207]: osdmap e508: 12 total, 10 up, 12 in 2024-08-30T23:41:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:39 smithi136 ceph-mon[20207]: 5.13 scrub starts 2024-08-30T23:41:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:39 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:41:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:39 smithi136 ceph-mon[20207]: pgmap v1182: 47 pgs: 10 unknown, 1 creating+remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 5 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 19 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 30259/222198 objects degraded (13.618%); 27028/222198 objects misplaced (12.164%); 14 MiB/s, 384 objects/s recovering 2024-08-30T23:41:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:39 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-30T23:41:40.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.345 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:40.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:40.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:40.467 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:40.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:40.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:40.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:40.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:40.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:40.838 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:40.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:40.961 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:40.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:41.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: 5.11 scrub starts 2024-08-30T23:41:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-30T23:41:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: osdmap e509: 12 total, 10 up, 12 in 2024-08-30T23:41:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:41:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:41:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:41:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:40 smithi173 ceph-mon[20554]: osdmap e510: 12 total, 10 up, 12 in 2024-08-30T23:41:41.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:41.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: 5.11 scrub starts 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: osdmap e509: 12 total, 10 up, 12 in 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:41:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:40 smithi106 ceph-mon[16603]: osdmap e510: 12 total, 10 up, 12 in 2024-08-30T23:41:41.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:41.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: 5.11 scrub starts 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: osdmap e509: 12 total, 10 up, 12 in 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:41:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:40 smithi136 ceph-mon[20207]: osdmap e510: 12 total, 10 up, 12 in 2024-08-30T23:41:41.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:41.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:41.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:41.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:41.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:41.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:41.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:41.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:41.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:41.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:41.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:41.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:41.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:42.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:41:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:41:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:41:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:41:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:41:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: 5.16 scrub starts 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:41:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:41:42.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: pgmap v1185: 47 pgs: 1 active+recovering+undersized+remapped, 9 unknown, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 20 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 24 KiB/s wr, 2 op/s; 29520/222201 objects degraded (13.285%); 31105/222201 objects misplaced (13.999%); 17 MiB/s, 279 objects/s recovering 2024-08-30T23:41:42.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:41 smithi173 ceph-mon[20554]: osdmap e511: 12 total, 10 up, 12 in 2024-08-30T23:41:42.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:41:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: 5.16 scrub starts 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:41:42.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:41:42.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: pgmap v1185: 47 pgs: 1 active+recovering+undersized+remapped, 9 unknown, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 20 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 24 KiB/s wr, 2 op/s; 29520/222201 objects degraded (13.285%); 31105/222201 objects misplaced (13.999%); 17 MiB/s, 279 objects/s recovering 2024-08-30T23:41:42.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:41 smithi106 ceph-mon[16603]: osdmap e511: 12 total, 10 up, 12 in 2024-08-30T23:41:42.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:42.199 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:41:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: 5.16 scrub starts 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: pgmap v1185: 47 pgs: 1 active+recovering+undersized+remapped, 9 unknown, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 20 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 24 KiB/s wr, 2 op/s; 29520/222201 objects degraded (13.285%); 31105/222201 objects misplaced (13.999%); 17 MiB/s, 279 objects/s recovering 2024-08-30T23:41:42.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:41 smithi136 ceph-mon[20207]: osdmap e511: 12 total, 10 up, 12 in 2024-08-30T23:41:42.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:42.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:42.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:42.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:42.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:42.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:42.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:42.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:42.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:42.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:42.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:42.921 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:42.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:42.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:43.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:43.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:43.120 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1, 5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-08-30T23:41:43.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:41:43.120 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:41:43.120 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 0.245448 2024-08-30T23:41:43.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 9 0.2454481228772477 2024-08-30T23:41:43.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:43.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:43.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:42 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:41:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:42 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:41:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:42 smithi136 ceph-mon[20207]: 5.19 scrub starts 2024-08-30T23:41:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:42 smithi136 ceph-mon[20207]: 5.15 scrub starts 2024-08-30T23:41:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:42 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:41:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:42 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:41:43.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:43.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:42 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:41:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:42 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:41:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:42 smithi173 ceph-mon[20554]: 5.19 scrub starts 2024-08-30T23:41:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:42 smithi173 ceph-mon[20554]: 5.15 scrub starts 2024-08-30T23:41:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:42 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:41:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:42 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:41:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:42 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:41:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:42 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:41:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:42 smithi106 ceph-mon[16603]: 5.19 scrub starts 2024-08-30T23:41:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:42 smithi106 ceph-mon[16603]: 5.15 scrub starts 2024-08-30T23:41:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:42 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:41:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:42 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:41:43.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:43.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:43.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:43.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:43.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:43.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:43.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:43.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:43.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:43.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:43.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:44.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:44.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:44.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:44.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:44.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:44.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:43 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:41:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:43 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:41:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:43 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:41:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:43 smithi136 ceph-mon[20207]: pgmap v1187: 47 pgs: 1 active+clean+remapped+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 24 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 137 KiB/s wr, 13 op/s; 29520/222201 objects degraded (13.285%); 29910/222201 objects misplaced (13.461%); 14 MiB/s, 227 objects/s recovering 2024-08-30T23:41:44.302 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:43 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:41:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:43 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:41:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:43 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:41:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:43 smithi173 ceph-mon[20554]: pgmap v1187: 47 pgs: 1 active+clean+remapped+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 24 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 137 KiB/s wr, 13 op/s; 29520/222201 objects degraded (13.285%); 29910/222201 objects misplaced (13.461%); 14 MiB/s, 227 objects/s recovering 2024-08-30T23:41:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:43 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:41:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:43 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:41:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:43 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:41:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:43 smithi106 ceph-mon[16603]: pgmap v1187: 47 pgs: 1 active+clean+remapped+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 24 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 137 KiB/s wr, 13 op/s; 29520/222201 objects degraded (13.285%); 29910/222201 objects misplaced (13.461%); 14 MiB/s, 227 objects/s recovering 2024-08-30T23:41:44.404 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.426 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:44.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:44.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:44.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:44.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:44.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:44.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:44.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:44.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:44.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:44.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:45.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:45.047 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:45.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:45.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:45.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:45.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:44 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:41:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:44 smithi136 ceph-mon[20207]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:41:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:44 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 29520/222201 objects degraded (13.285%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:44 smithi136 ceph-mon[20207]: 5.11 scrub starts 2024-08-30T23:41:45.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:45.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:44 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:41:45.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:44 smithi173 ceph-mon[20554]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:41:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:44 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 29520/222201 objects degraded (13.285%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:44 smithi173 ceph-mon[20554]: 5.11 scrub starts 2024-08-30T23:41:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:44 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:41:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:44 smithi106 ceph-mon[16603]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:41:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:44 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 29520/222201 objects degraded (13.285%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:44 smithi106 ceph-mon[16603]: 5.11 scrub starts 2024-08-30T23:41:45.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:45.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:45.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:45.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:45.644 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:45.668 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:45.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:45.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:45.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:45.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:45.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:1186 4.4 e311/306 4:2bc10e34:::benchmark_data_smithi173_74914_object1185:head v 311'76, mlcod=311'72)", 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:41:45.948 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 464.08397655900001, 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21510685600000001, 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862592+0000", 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862593+0000", 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4200000000000005e-07 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:01.862614+0000", 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1129e-05 2024-08-30T23:41:45.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077250+0000", 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214636047 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077258+0000", 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9789999999999993e-06 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077310+0000", 2024-08-30T23:41:45.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2367999999999997e-05 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077334+0000", 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4168e-05 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077669+0000", 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00033434399999999998 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:02.077699+0000", 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9978999999999999e-05 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:168798 4.6 e375/369 4:68592339:::benchmark_data_smithi173_74914_object70556:head v 375'10605, mlcod=375'10602)", 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 298.419459674, 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208871322, 2024-08-30T23:41:45.953 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527110+0000", 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5499999999999997e-07 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527116+0000", 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0209999999999996e-06 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732384+0000", 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205268487 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732390+0000", 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0419999999999999e-06 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732440+0000", 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9663000000000002e-05 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732470+0000", 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0412e-05 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735945+0000", 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034745869999999999 2024-08-30T23:41:45.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735980+0000", 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5255e-05 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:15236 5.4 e403/394 5:275cc9e1:::benchmark_data_smithi173_87639_object15235:head v 403'976, mlcod=403'973)", 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:50.728823+0000", 2024-08-30T23:41:45.958 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 235.217745079, 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.198409419, 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728823+0000", 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728823+0000", 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728830+0000", 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0389999999999999e-06 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.728929+0000", 2024-08-30T23:41:45.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9114e-05 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926322+0000", 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19739287599999999 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926328+0000", 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6849999999999998e-06 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926390+0000", 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1619999999999996e-05 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926403+0000", 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3327e-05 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927185+0000", 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00078186300000000005 2024-08-30T23:41:45.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927233+0000", 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7895000000000001e-05 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:15243 5.2 e403/394 5:4bef17d2:::benchmark_data_smithi173_87639_object15242:head v 403'951, mlcod=403'948)", 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 235.214787817, 2024-08-30T23:41:45.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19543065400000001, 2024-08-30T23:41:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.964 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731781+0000", 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7800000000000001e-07 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.731845+0000", 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3625000000000002e-05 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926238+0000", 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19439292499999999 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926246+0000", 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5380000000000002e-06 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926310+0000", 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3526000000000001e-05 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.926339+0000", 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9323e-05 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:45.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927173+0000", 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00083399799999999997 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:50.927211+0000", 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8040999999999997e-05 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.969 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:31241 5.4 e406/394 5:27c8315f:::benchmark_data_smithi173_87639_object31240:head v 406'1956, mlcod=406'1953)", 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 224.71636338799999, 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.165546794, 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230205+0000", 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6499999999999999e-07 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.230313+0000", 2024-08-30T23:41:45.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107126 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394252+0000", 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16393967000000001 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394272+0000", 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0098999999999998e-05 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394426+0000", 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000153843 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.394467+0000", 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1226000000000001e-05 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.395715+0000", 2024-08-30T23:41:45.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012479399999999999 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:01.395752+0000", 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6325000000000002e-05 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:59001 5.6 e419/389 5:6208a760:::benchmark_data_smithi173_87639_object59000:head v 419'3623, mlcod=418'3620)", 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:41:45.974 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.80350122499999, 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.314356937, 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143070+0000", 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5739999999999998e-06 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143128+0000", 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7890000000000003e-05 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455493+0000", 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31236522100000003 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455506+0000", 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3655999999999999e-05 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455662+0000", 2024-08-30T23:41:45.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015594900000000001 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455681+0000", 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8301e-05 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.457390+0000", 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017094860000000001 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.457424+0000", 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3859999999999998e-05 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 424/420 [PushReplyOp(5:11696329:::benchmark_data_smithi173_87639_object5188:head)])", 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.12524385899999, 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45081538799999998, 2024-08-30T23:41:45.979 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:41:45.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821326+0000", 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3340000000000001e-06 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821332+0000", 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0360000000000003e-06 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269704+0000", 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44837247699999999 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269713+0000", 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269806+0000", 2024-08-30T23:41:45.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3058000000000002e-05 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.272140+0000", 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023334639999999999 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 425/420 [PushReplyOp(5:10493f51:::benchmark_data_smithi173_87639_object5215:head)])", 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:41:45.983 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 185.58437709699999, 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318688992, 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362192+0000", 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.141e-06 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362198+0000", 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6130000000000003e-06 2024-08-30T23:41:45.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678705+0000", 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31650737299999998 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678716+0000", 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0288e-05 2024-08-30T23:41:45.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678809+0000", 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3018999999999998e-05 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.680880+0000", 2024-08-30T23:41:45.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020715579999999998 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 427/420 [PushReplyOp(5:123eeb63:::benchmark_data_smithi173_87639_object7071:head)])", 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:43.522780+0000", 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 182.42378839, 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19251843099999999, 2024-08-30T23:41:45.988 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522780+0000", 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522780+0000", 2024-08-30T23:41:45.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522781+0000", 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.522791+0000", 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0169999999999999e-05 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.712969+0000", 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.190178027 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.712987+0000", 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8073000000000001e-05 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.713142+0000", 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000154993 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:43.715298+0000", 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021558879999999999 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 427/420 [PushReplyOp(5:1ccb0413:::benchmark_data_smithi173_87639_object8971:head)])", 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 181.44287742899999, 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60307924400000001, 2024-08-30T23:41:45.992 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503692+0000", 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.02e-07 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503698+0000", 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9819999999999996e-06 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104246+0000", 2024-08-30T23:41:45.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60054844500000004 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104278+0000", 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.18e-05 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104366+0000", 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7817000000000003e-05 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.106770+0000", 2024-08-30T23:41:45.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002404298 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:81366 5.8 5:189e6c06:::benchmark_data_smithi173_87639_object1126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:14.116539+0000", 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.83002910299999, 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.104431312, 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:45.996 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 81366 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116539+0000", 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:45.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116539+0000", 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116542+0000", 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5229999999999998e-06 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116554+0000", 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.27e-05 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116591+0000", 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6328999999999997e-05 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116597+0000", 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6290000000000004e-06 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116658+0000", 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0804999999999998e-05 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:45.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116789+0000", 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000131259 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.116931+0000", 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014145 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.121433+0000", 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045022489999999998 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.210573+0000", 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.089139754000000002 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218603+0000", 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0080298829999999998 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.220576+0000", 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019733799999999998 2024-08-30T23:41:46.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.220645+0000", 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8522000000000006e-05 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.220971+0000", 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032582899999999997 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:81366 5.8 e440/420 ondisk, result = 0)", 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:14.119138+0000", 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.827430236, 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099470546000000007, 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119138+0000", 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119138+0000", 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119139+0000", 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8899999999999999e-07 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.119143+0000", 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7349999999999998e-06 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218234+0000", 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099091913000000004 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218238+0000", 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2490000000000001e-06 2024-08-30T23:41:46.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218592+0000", 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000354173 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218598+0000", 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0480000000000004e-06 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:14.218609+0000", 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0739e-05 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84105 5.8 5:17a91613:::benchmark_data_smithi173_87639_object3865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 148.77424214600001, 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22008060500000001, 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.007 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84105 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172327+0000", 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.124e-06 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172332+0000", 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9910000000000002e-06 2024-08-30T23:41:46.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172344+0000", 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1755e-05 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172351+0000", 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5699999999999998e-06 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172564+0000", 2024-08-30T23:41:46.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000213317 2024-08-30T23:41:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.011 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172648+0000", 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3610999999999996e-05 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172745+0000", 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7783999999999998e-05 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.174806+0000", 2024-08-30T23:41:46.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020607030000000001 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175526+0000", 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071991399999999999 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.176022+0000", 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049577499999999995 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391996+0000", 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21597425100000001 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392038+0000", 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2058999999999998e-05 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392407+0000", 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036875100000000001 2024-08-30T23:41:46.014 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:84105 5.8 e443/420 ondisk, result = 0)", 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 148.77140691400001, 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21725276199999999, 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175162+0000", 2024-08-30T23:41:46.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175166+0000", 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2189999999999998e-06 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391931+0000", 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21676427100000001 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391937+0000", 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0270000000000001e-06 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391986+0000", 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9175999999999998e-05 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391992+0000", 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0929999999999999e-06 2024-08-30T23:41:46.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392414+0000", 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042228600000000002 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84505 5.a 5:51170427:::benchmark_data_smithi173_87639_object4265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.766478+0000", 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 148.180090527, 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.111502222, 2024-08-30T23:41:46.019 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84505 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766478+0000", 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766478+0000", 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766479+0000", 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1599999999999999e-06 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766485+0000", 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2450000000000003e-06 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766496+0000", 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.128e-05 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.766503+0000", 2024-08-30T23:41:46.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3589999999999996e-06 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.769391+0000", 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028883820000000001 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.769487+0000", 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6069000000000005e-05 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 8,10", 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.769593+0000", 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105752 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.774123+0000", 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045304359999999997 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.776252+0000", 2024-08-30T23:41:46.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002128614 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.876452+0000", 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.100199811 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.876497+0000", 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5269999999999999e-05 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.877980+0000", 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014828440000000001 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84662 5.8 5:1acada36:::benchmark_data_smithi173_87639_object4422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:18.051032+0000", 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 147.895536386, 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.161365491, 2024-08-30T23:41:46.026 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84662 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051032+0000", 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051032+0000", 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051033+0000", 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.449e-06 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051040+0000", 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1600000000000003e-06 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051051+0000", 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1821000000000001e-05 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051058+0000", 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.097e-06 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051130+0000", 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1766000000000004e-05 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051276+0000", 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000145465 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.051419+0000", 2024-08-30T23:41:46.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014362199999999999 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.053932+0000", 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025125719999999998 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.053951+0000", 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8984e-05 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.054643+0000", 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000692295 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.212029+0000", 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.157385993 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.212106+0000", 2024-08-30T23:41:46.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7330000000000007e-05 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.212397+0000", 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029093700000000001 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 444/420 [PushReplyOp(5:14a83234:::benchmark_data_smithi173_87639_object53167:head)])", 2024-08-30T23:41:46.033 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:18.062274+0000", 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 147.88429387599999, 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14578411499999999, 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062274+0000", 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062274+0000", 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062275+0000", 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7199999999999998e-07 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.062280+0000", 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9729999999999997e-06 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.207439+0000", 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14515842300000001 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.207444+0000", 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.207507+0000", 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2639999999999997e-05 2024-08-30T23:41:46.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:18.208059+0000", 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00055189400000000004 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:86535 5.f e492/447 5:ffbf1c07:::benchmark_data_smithi173_87639_object6295:head v 492'5415, mlcod=492'5412)", 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:12.074311+0000", 2024-08-30T23:41:46.037 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.872257316999999, 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.104082852, 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.074311+0000", 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.074311+0000", 2024-08-30T23:41:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.074312+0000", 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1699999999999998e-07 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.074318+0000", 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9150000000000001e-06 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.173882+0000", 2024-08-30T23:41:46.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.099563981999999995 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.173888+0000", 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8190000000000003e-06 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.173942+0000", 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3779999999999997e-05 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.173961+0000", 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8598999999999999e-05 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.178351+0000", 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043906509999999998 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.178394+0000", 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2488999999999999e-05 2024-08-30T23:41:46.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:86608 5.14 e492/490 5:2c182c99:::benchmark_data_smithi173_87639_object6368:head v 492'5368, mlcod=492'5366)", 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:12.254563+0000", 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.692005545000001, 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.121439582, 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.254563+0000", 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.254563+0000", 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.254564+0000", 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9600000000000008e-07 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.254570+0000", 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5220000000000004e-06 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.374674+0000", 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.120104026 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.374679+0000", 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0769999999999998e-06 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.374740+0000", 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0126e-05 2024-08-30T23:41:46.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.374765+0000", 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5026000000000001e-05 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.375970+0000", 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001205345 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.376002+0000", 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2464e-05 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:86645 5.4 e492/488 5:20864db1:::benchmark_data_smithi173_87639_object6405:head v 492'5366, mlcod=492'5364)", 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:12.422333+0000", 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 33.524235572999999, 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12711357700000001, 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:41:46.047 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.422333+0000", 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.422333+0000", 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:41:46.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.422334+0000", 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3900000000000003e-07 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.422341+0000", 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.968e-06 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.547923+0000", 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.12558201099999999 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.547928+0000", 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8930000000000004e-06 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.547972+0000", 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4328e-05 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.547990+0000", 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8275e-05 2024-08-30T23:41:46.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.549404+0000", 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014138709999999999 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:12.549446+0000", 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2292000000000001e-05 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:41:46.051 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:41:46.052 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:45 smithi106 ceph-mon[16603]: 5.13 scrub starts 2024-08-30T23:41:46.052 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:45 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:41:46.052 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:45 smithi106 ceph-mon[16603]: pgmap v1188: 47 pgs: 1 active+clean+remapped+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 24 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 KiB/s wr, 10 op/s; 29157/222201 objects degraded (13.122%); 29728/222201 objects misplaced (13.379%); 13 MiB/s, 211 objects/s recovering 2024-08-30T23:41:46.052 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:45 smithi106 ceph-mon[16603]: osdmap e512: 12 total, 10 up, 12 in 2024-08-30T23:41:46.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:46.074 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:46.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:46.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:46.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:45 smithi136 ceph-mon[20207]: 5.13 scrub starts 2024-08-30T23:41:46.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:45 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:41:46.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:45 smithi136 ceph-mon[20207]: pgmap v1188: 47 pgs: 1 active+clean+remapped+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 24 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 KiB/s wr, 10 op/s; 29157/222201 objects degraded (13.122%); 29728/222201 objects misplaced (13.379%); 13 MiB/s, 211 objects/s recovering 2024-08-30T23:41:46.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:45 smithi136 ceph-mon[20207]: osdmap e512: 12 total, 10 up, 12 in 2024-08-30T23:41:46.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:46.313 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:46.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:45 smithi173 ceph-mon[20554]: 5.13 scrub starts 2024-08-30T23:41:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:45 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:41:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:45 smithi173 ceph-mon[20554]: pgmap v1188: 47 pgs: 1 active+clean+remapped+scrubbing, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 24 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 109 KiB/s wr, 10 op/s; 29157/222201 objects degraded (13.122%); 29728/222201 objects misplaced (13.379%); 13 MiB/s, 211 objects/s recovering 2024-08-30T23:41:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:45 smithi173 ceph-mon[20554]: osdmap e512: 12 total, 10 up, 12 in 2024-08-30T23:41:46.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:46.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:46.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:46.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:46.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:46.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:46.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:46.815 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:46.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:46.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:46.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:46.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:41:47.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:47.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:47.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:47.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:47.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:46 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:41:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:46 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:41:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:46 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:41:47.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:47.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:47.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:46 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:41:47.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:46 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:41:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:46 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:41:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:46 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:41:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:46 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:41:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:46 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:41:47.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:41:47.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:47.424 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:47.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:47.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:47.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:47.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:47.685 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:41:47.685 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:41:47.685 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:41:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:47.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:47.706 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:41:47.706 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:41:47.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:41:47.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:41:47.748 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:41:47.748 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:41:47.748 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:41:47.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:41:47.769 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:41:47.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:47.792 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:41:47.792 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:41:47.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:47.812 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:41:47.812 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:41:47.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:47.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:47.936 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:41:47.937 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:41:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:47.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:47.959 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:41:47.959 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:41:47.982 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:41:47.982 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:41:48.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:48.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:48.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:48.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:41:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:41:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: 5.16 scrub starts 2024-08-30T23:41:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:41:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:41:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: pgmap v1190: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 23 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 KiB/s wr, 9 op/s; 29157/222201 objects degraded (13.122%); 29215/222201 objects misplaced (13.148%); 9.8 MiB/s, 156 objects/s recovering 2024-08-30T23:41:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3252083872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:47 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:41:48.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: 5.16 scrub starts 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: pgmap v1190: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 23 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 KiB/s wr, 9 op/s; 29157/222201 objects degraded (13.122%); 29215/222201 objects misplaced (13.148%); 9.8 MiB/s, 156 objects/s recovering 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3252083872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:47 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:41:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:41:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:41:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: 5.16 scrub starts 2024-08-30T23:41:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:41:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:41:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: pgmap v1190: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 23 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 96 KiB/s wr, 9 op/s; 29157/222201 objects degraded (13.122%); 29215/222201 objects misplaced (13.148%); 9.8 MiB/s, 156 objects/s recovering 2024-08-30T23:41:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3252083872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:41:48.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:47 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:41:48.362 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:48.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:48.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:48.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:48.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:48.585 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:48.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:48.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:48.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:48.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:48.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:48.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:48.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:48.968 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.9 primary-affinity to 0.245448 (8160852) 2024-08-30T23:41:49.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:48 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:41:49.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:48 smithi106 ceph-mon[16603]: 5.15 scrub starts 2024-08-30T23:41:49.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:48 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:41:49.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:48 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:41:49.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:48 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:41:49.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2499519154' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.2454481228772477} : dispatch 2024-08-30T23:41:49.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:49.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:49.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:49.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:49.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:48 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:41:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:48 smithi136 ceph-mon[20207]: 5.15 scrub starts 2024-08-30T23:41:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:48 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:41:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:48 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:41:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:48 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:41:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2499519154' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.2454481228772477} : dispatch 2024-08-30T23:41:49.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:49.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.295 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:49.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:48 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:41:49.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:48 smithi173 ceph-mon[20554]: 5.15 scrub starts 2024-08-30T23:41:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:48 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:41:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:48 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:41:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:48 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:41:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2499519154' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.2454481228772477} : dispatch 2024-08-30T23:41:49.325 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:49.325 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:49.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:49.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:49.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:49.541 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:49.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:49.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:49.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:49.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:49.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:49.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:49.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:49.978 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:41:50.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:50.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:50.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:41:50.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:50.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:50.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:50.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:41:50.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:41:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2499519154' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.2454481228772477}]': finished 2024-08-30T23:41:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: osdmap e513: 12 total, 10 up, 12 in 2024-08-30T23:41:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: 5.11 scrub starts 2024-08-30T23:41:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/999288748' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:41:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:49 smithi136 ceph-mon[20207]: pgmap v1192: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 2 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 8 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 21 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 71 KiB/s wr, 7 op/s; 29157/222201 objects degraded (13.122%); 28023/222201 objects misplaced (12.612%); 12 MiB/s, 198 objects/s recovering 2024-08-30T23:41:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:50.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:50.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:41:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:41:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2499519154' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.2454481228772477}]': finished 2024-08-30T23:41:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: osdmap e513: 12 total, 10 up, 12 in 2024-08-30T23:41:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: 5.11 scrub starts 2024-08-30T23:41:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/999288748' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:41:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:49 smithi173 ceph-mon[20554]: pgmap v1192: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 2 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 8 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 21 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 71 KiB/s wr, 7 op/s; 29157/222201 objects degraded (13.122%); 28023/222201 objects misplaced (12.612%); 12 MiB/s, 198 objects/s recovering 2024-08-30T23:41:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:41:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:41:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2499519154' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.2454481228772477}]': finished 2024-08-30T23:41:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: osdmap e513: 12 total, 10 up, 12 in 2024-08-30T23:41:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: 5.11 scrub starts 2024-08-30T23:41:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/999288748' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:41:50.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:49 smithi106 ceph-mon[16603]: pgmap v1192: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 2 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 8 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 21 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 71 KiB/s wr, 7 op/s; 29157/222201 objects degraded (13.122%); 28023/222201 objects misplaced (12.612%); 12 MiB/s, 198 objects/s recovering 2024-08-30T23:41:50.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:50.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:50.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:50.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:50.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:50.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:50.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:50.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:50.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:50.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:51.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:51.035 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:51.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:51.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:51.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:51.284 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:50 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/999288748' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:41:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:50 smithi136 ceph-mon[20207]: osdmap e514: 12 total, 10 up, 12 in 2024-08-30T23:41:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:50 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:41:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:50 smithi136 ceph-mon[20207]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:41:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:50 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 29157/222201 objects degraded (13.122%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:51.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:50 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:51.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/999288748' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:41:51.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:50 smithi106 ceph-mon[16603]: osdmap e514: 12 total, 10 up, 12 in 2024-08-30T23:41:51.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:50 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:41:51.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:50 smithi106 ceph-mon[16603]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:41:51.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:50 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 29157/222201 objects degraded (13.122%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:50 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:41:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/999288748' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:41:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:50 smithi173 ceph-mon[20554]: osdmap e514: 12 total, 10 up, 12 in 2024-08-30T23:41:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:50 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:41:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:50 smithi173 ceph-mon[20554]: Health check update: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:41:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:50 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 29157/222201 objects degraded (13.122%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:51.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:51.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:51.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:51.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:51.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:51.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:51.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:51.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:51.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:51.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:51.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:51.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:52.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:52.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:52.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:52.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:52.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:52.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: 5.13 scrub starts 2024-08-30T23:41:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: 5.19 scrub starts 2024-08-30T23:41:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: 5.13 scrub ok 2024-08-30T23:41:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: 5.19 scrub ok 2024-08-30T23:41:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:41:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:41:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:51 smithi136 ceph-mon[20207]: pgmap v1194: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 2 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 8 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 21 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28312/222201 objects degraded (12.742%); 27019/222201 objects misplaced (12.160%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:41:52.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: 5.13 scrub starts 2024-08-30T23:41:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: 5.19 scrub starts 2024-08-30T23:41:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: 5.13 scrub ok 2024-08-30T23:41:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: 5.19 scrub ok 2024-08-30T23:41:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:41:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:41:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:51 smithi173 ceph-mon[20554]: pgmap v1194: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 2 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 8 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 21 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28312/222201 objects degraded (12.742%); 27019/222201 objects misplaced (12.160%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:41:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: 5.13 scrub starts 2024-08-30T23:41:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: 5.19 scrub starts 2024-08-30T23:41:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: 5.13 scrub ok 2024-08-30T23:41:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: 5.19 scrub ok 2024-08-30T23:41:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:41:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:41:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:51 smithi106 ceph-mon[16603]: pgmap v1194: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 2 active+clean+remapped+scrubbing, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 8 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 21 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28312/222201 objects degraded (12.742%); 27019/222201 objects misplaced (12.160%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:41:52.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:52.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:52.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:52.521 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:52.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:52.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:52.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:52.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:52.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:52.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:52.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:53.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:53.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:53.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:53.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:53.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:53.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:41:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:41:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:41:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:41:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 5.16 scrub starts 2024-08-30T23:41:53.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:41:53.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:41:53.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:41:53.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:41:53.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:41:53.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:52 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:41:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 5.16 scrub starts 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:41:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:52 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:41:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:41:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:41:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:41:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:41:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 5.16 scrub starts 2024-08-30T23:41:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:41:53.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:41:53.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:41:53.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:41:53.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:41:53.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:52 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:41:53.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:53.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:53.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:53.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:53.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:53.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:53.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:53.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:53.767 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:53.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:53.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:53.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:53.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:54.015 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:54.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:54.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:54.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:54.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 5.16 scrub ok 2024-08-30T23:41:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:41:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:41:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:41:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:41:54.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:41:54.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:41:54.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:41:54.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:41:54.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:53 smithi136 ceph-mon[20207]: pgmap v1195: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28312/221940 objects degraded (12.757%); 25727/221940 objects misplaced (11.592%) 2024-08-30T23:41:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 5.16 scrub ok 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:41:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:53 smithi173 ceph-mon[20554]: pgmap v1195: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28312/221940 objects degraded (12.757%); 25727/221940 objects misplaced (11.592%) 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 5.16 scrub ok 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:41:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:41:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:41:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:53 smithi106 ceph-mon[16603]: pgmap v1195: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 28312/221940 objects degraded (12.757%); 25727/221940 objects misplaced (11.592%) 2024-08-30T23:41:54.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:41:54.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:41:54.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:54.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:54.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:54.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:54.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:54.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:54.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:41:54.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:41:54.885 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:54.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:54.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:55.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:55.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:55.053 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:55.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:41:55.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 5.15 scrub starts 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 5.15 scrub ok 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 5.11 scrub starts 2024-08-30T23:41:55.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:54 smithi106 ceph-mon[16603]: 5.11 scrub ok 2024-08-30T23:41:55.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:55.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:55.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:55.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 5.15 scrub starts 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 5.15 scrub ok 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 5.11 scrub starts 2024-08-30T23:41:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:54 smithi136 ceph-mon[20207]: 5.11 scrub ok 2024-08-30T23:41:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 5.15 scrub starts 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 5.15 scrub ok 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 5.11 scrub starts 2024-08-30T23:41:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:54 smithi173 ceph-mon[20554]: 5.11 scrub ok 2024-08-30T23:41:55.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:55.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.379 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:55.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:55.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:55.600 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:55.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:55.627 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:55.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:55.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:55.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:55.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:55.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:55.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:55.999 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:56.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:56.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:56.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:56.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:56.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:55 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:41:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:55 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:41:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:55 smithi136 ceph-mon[20207]: pgmap v1196: 47 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27847/221940 objects degraded (12.547%); 25493/221940 objects misplaced (11.486%) 2024-08-30T23:41:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:55 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 28312/221940 objects degraded (12.757%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:56.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:55 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:41:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:55 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:41:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:55 smithi173 ceph-mon[20554]: pgmap v1196: 47 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27847/221940 objects degraded (12.547%); 25493/221940 objects misplaced (11.486%) 2024-08-30T23:41:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:55 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 28312/221940 objects degraded (12.757%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:56.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:56.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:55 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:41:56.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:55 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:41:56.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:55 smithi106 ceph-mon[16603]: pgmap v1196: 47 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27847/221940 objects degraded (12.547%); 25493/221940 objects misplaced (11.486%) 2024-08-30T23:41:56.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:55 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 28312/221940 objects degraded (12.757%), 9 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:41:56.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:56.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:41:56.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:56.496 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:56.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:56.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:56.652 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:56.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:56.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:56.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:56.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:56.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:41:56.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:56.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:56.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:57.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:41:57.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:56 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:41:57.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:56 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:41:57.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:56 smithi106 ceph-mon[16603]: osdmap e515: 12 total, 10 up, 12 in 2024-08-30T23:41:57.051 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:56 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:41:57.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:57.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:57.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:57.209 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:41:57.209 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:41:57.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:57.254 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:56 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:41:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:56 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:41:57.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:56 smithi136 ceph-mon[20207]: osdmap e515: 12 total, 10 up, 12 in 2024-08-30T23:41:57.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:56 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:41:57.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:56 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:41:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:56 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:41:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:56 smithi173 ceph-mon[20554]: osdmap e515: 12 total, 10 up, 12 in 2024-08-30T23:41:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:56 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:41:57.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:57.378 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:57.480 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:57.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:57.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:57.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:57.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:57.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:57.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:57.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:57.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:57.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:57.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:58.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:58.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:58.220 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:41:58.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:41:58.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:58 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:41:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:58 smithi136 ceph-mon[20207]: osdmap e516: 12 total, 10 up, 12 in 2024-08-30T23:41:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:58 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:41:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:58 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:41:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:58 smithi136 ceph-mon[20207]: pgmap v1199: 47 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27847/221712 objects degraded (12.560%); 25003/221712 objects misplaced (11.277%) 2024-08-30T23:41:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:58 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:41:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:58 smithi173 ceph-mon[20554]: osdmap e516: 12 total, 10 up, 12 in 2024-08-30T23:41:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:58 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:41:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:58 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:41:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:58 smithi173 ceph-mon[20554]: pgmap v1199: 47 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27847/221712 objects degraded (12.560%); 25003/221712 objects misplaced (11.277%) 2024-08-30T23:41:58.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:58 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:41:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:58 smithi106 ceph-mon[16603]: osdmap e516: 12 total, 10 up, 12 in 2024-08-30T23:41:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:58 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:41:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:58 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:41:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:58 smithi106 ceph-mon[16603]: pgmap v1199: 47 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 25 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27847/221712 objects degraded (12.560%); 25003/221712 objects misplaced (11.277%) 2024-08-30T23:41:58.366 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:58.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:58.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:58.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:58.613 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:58.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:58.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:58.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:58.839 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:41:58.861 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:41:58.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:41:58.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:58.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:41:59.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:41:59.109 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:41:59.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:41:59.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:41:59.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:59 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:41:59.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:59 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-30T23:41:59.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:41:59 smithi136 ceph-mon[20207]: osdmap e517: 12 total, 10 up, 12 in 2024-08-30T23:41:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:59 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:41:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:59 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-30T23:41:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:41:59 smithi173 ceph-mon[20554]: osdmap e517: 12 total, 10 up, 12 in 2024-08-30T23:41:59.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:41:59.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:41:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:59 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:41:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:59 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-30T23:41:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:41:59 smithi106 ceph-mon[16603]: osdmap e517: 12 total, 10 up, 12 in 2024-08-30T23:41:59.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:59.463 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [1, 5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-08-30T23:41:59.463 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:41:59.463 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-08-30T23:41:59.463 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-08-30T23:41:59.463 DEBUG:teuthology.orchestra.run.smithi106:osd.1> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1.service 2024-08-30T23:41:59.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl start ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1 2024-08-30T23:41:59.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:59.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:59.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:59.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:41:59.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:41:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:41:59.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:41:59.851 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:41:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:41:59.857 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:41:59 smithi106 systemd[1]: Starting Ceph osd.1 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:41:59.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:41:59.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:41:59.968 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:41:59.977 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:41:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:41:59.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:41:59.987 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:00.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:00.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:00.176 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:00.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:00.226 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:00.228 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:00 smithi106 ceph-mon[16603]: 5.17 scrub starts 2024-08-30T23:42:00.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:00 smithi106 ceph-mon[16603]: 5.17 scrub ok 2024-08-30T23:42:00.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:00 smithi106 ceph-mon[16603]: pgmap v1201: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221712 objects degraded (12.559%); 23741/221712 objects misplaced (10.708%) 2024-08-30T23:42:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:00 smithi136 ceph-mon[20207]: 5.17 scrub starts 2024-08-30T23:42:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:00 smithi136 ceph-mon[20207]: 5.17 scrub ok 2024-08-30T23:42:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:00 smithi136 ceph-mon[20207]: pgmap v1201: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221712 objects degraded (12.559%); 23741/221712 objects misplaced (10.708%) 2024-08-30T23:42:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:00 smithi173 ceph-mon[20554]: 5.17 scrub starts 2024-08-30T23:42:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:00 smithi173 ceph-mon[20554]: 5.17 scrub ok 2024-08-30T23:42:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:00 smithi173 ceph-mon[20554]: pgmap v1201: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221712 objects degraded (12.559%); 23741/221712 objects misplaced (10.708%) 2024-08-30T23:42:00.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:00.351 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:00.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:00.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:00.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:00.550 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:00 smithi106 podman[164772]: 2024-08-30 23:42:00.271234955 +0000 UTC m=+0.266642067 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:42:00.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:00.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:00.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:00.707 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:00.707 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:00.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:00.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:00.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:00.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:00.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:00.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:01.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:01.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:01.048 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:42:01.071 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:01.071 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:01.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:01.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:01.102 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:01.103 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:01.128 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:01.128 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:01.128 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:01.149 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:01.149 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:01.170 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:01.171 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:01.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 27845/221712 objects degraded (12.559%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:01.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-mon[16603]: osdmap e518: 12 total, 10 up, 12 in 2024-08-30T23:42:01.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2426072365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:01.172 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 podman[164772]: 2024-08-30 23:42:01.118275405 +0000 UTC m=+1.113682497 container create 27e5cd59a62dd54b53b33a32fa60a188d5a17dd6c3aedb32ea8deddecf99ccf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-30T23:42:01.192 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:01.192 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:01.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:01.214 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:01.214 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:01.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:01.238 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:01.238 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:01.261 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:01.262 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 27845/221712 objects degraded (12.559%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:01 smithi136 ceph-mon[20207]: osdmap e518: 12 total, 10 up, 12 in 2024-08-30T23:42:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2426072365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 27845/221712 objects degraded (12.559%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:01 smithi173 ceph-mon[20554]: osdmap e518: 12 total, 10 up, 12 in 2024-08-30T23:42:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2426072365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:01.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:01.348 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.349 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:01.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:01.473 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:01.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:01.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:01.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:42:01.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:01.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:01.857 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 podman[164772]: 2024-08-30 23:42:01.547584053 +0000 UTC m=+1.542991147 container init 27e5cd59a62dd54b53b33a32fa60a188d5a17dd6c3aedb32ea8deddecf99ccf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 podman[164772]: 2024-08-30 23:42:01.551487265 +0000 UTC m=+1.546894360 container start 27e5cd59a62dd54b53b33a32fa60a188d5a17dd6c3aedb32ea8deddecf99ccf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 podman[164772]: 2024-08-30 23:42:01.647718723 +0000 UTC m=+1.643125835 container attach 27e5cd59a62dd54b53b33a32fa60a188d5a17dd6c3aedb32ea8deddecf99ccf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:42:01.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:01.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:01.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:01.963 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:01.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:02.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:02.018 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:42:02.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:02.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:02.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:02.258 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:02.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/24913268' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:02.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-mon[16603]: pgmap v1203: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221712 objects degraded (12.559%); 23129/221712 objects misplaced (10.432%) 2024-08-30T23:42:02.262 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-30T23:42:02.262 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-30T23:42:02.262 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-30T23:42:02.262 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:01 smithi106 bash[164772]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-30T23:42:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/24913268' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:02 smithi136 ceph-mon[20207]: pgmap v1203: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221712 objects degraded (12.559%); 23129/221712 objects misplaced (10.432%) 2024-08-30T23:42:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/24913268' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:02 smithi173 ceph-mon[20554]: pgmap v1203: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221712 objects degraded (12.559%); 23129/221712 objects misplaced (10.432%) 2024-08-30T23:42:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:02.325 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:02.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:02.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:02.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:02.451 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:42:02.451 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:42:02.451 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:42:02.451 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:42:02.451 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:42:02.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:02.506 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:02.607 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-30T23:42:02.607 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 bash[164772]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-30T23:42:02.607 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 bash[164772]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 bash[164772]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 bash[164772]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate[164953]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 bash[164772]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-30T23:42:02.608 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 podman[164772]: 2024-08-30 23:42:02.295047667 +0000 UTC m=+2.290454772 container died 27e5cd59a62dd54b53b33a32fa60a188d5a17dd6c3aedb32ea8deddecf99ccf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-30T23:42:02.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:02.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:02.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:02.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:02.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:02.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:02.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:02.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:03.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:03.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/24913268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: osdmap e519: 12 total, 10 up, 12 in 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:03.123 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:03.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:03.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:42:03.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:03.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:03.284 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:02 smithi106 podman[165100]: 2024-08-30 23:42:02.930851315 +0000 UTC m=+0.628231645 container remove 27e5cd59a62dd54b53b33a32fa60a188d5a17dd6c3aedb32ea8deddecf99ccf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/24913268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: osdmap e519: 12 total, 10 up, 12 in 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:03.285 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:42:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/24913268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: osdmap e519: 12 total, 10 up, 12 in 2024-08-30T23:42:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:03.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:03.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:03.426 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:03.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:03.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:03.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:03.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:03.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:03.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:03.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:03.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:03.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:03.927 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:42:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.15351.0:206897 3.b e300/206 3:d0177d76:::benchmark_data_smithi173_60780_object103443:head v 300'12978, mlcod=300'12975)", 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:33:44.057579+0000", 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 499.86860352500003, 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.58266937699999999, 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057579+0000", 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057579+0000", 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057580+0000", 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.99e-07 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.057584+0000", 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4100000000000001e-06 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636116+0000", 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.57853164000000001 2024-08-30T23:42:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636133+0000", 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.668e-05 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636261+0000", 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012832100000000001 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.636303+0000", 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1845999999999999e-05 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.640219+0000", 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039165789999999999 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:33:44.640249+0000", 2024-08-30T23:42:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9201999999999999e-05 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:31053 4.c 4:3c66407d:::benchmark_data_smithi173_74914_object31052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:20.682766+0000", 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 463.24341663400003, 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60411518900000005, 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 31053 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682766+0000", 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682766+0000", 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682767+0000", 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682833+0000", 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6268999999999999e-05 2024-08-30T23:42:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682849+0000", 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5753e-05 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.682857+0000", 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.106e-06 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.683370+0000", 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00051270999999999999 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.683454+0000", 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3888999999999996e-05 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.683552+0000", 2024-08-30T23:42:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8455999999999994e-05 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684220+0000", 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00066772100000000003 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.756181+0000", 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071960969 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286726+0000", 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53054503799999997 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286800+0000", 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4201000000000003e-05 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286881+0000", 2024-08-30T23:42:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1201999999999999e-05 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.25654.0:31053 4.c e317/306 ondisk, result = 0)", 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:20.684501+0000", 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 463.24168177299998, 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60239176800000005, 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684501+0000", 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684501+0000", 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684502+0000", 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5700000000000002e-07 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:20.684505+0000", 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.422e-06 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286533+0000", 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60202851700000004 2024-08-30T23:42:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286549+0000", 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5939000000000001e-05 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286703+0000", 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015350699999999999 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286718+0000", 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4832e-05 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:21.286893+0000", 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017499400000000001 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:49449 4.c 4:30eeb856:::benchmark_data_smithi173_74914_object49448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:40.563639+0000", 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 443.36254412199997, 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.225408365, 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49449 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563639+0000", 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563639+0000", 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563642+0000", 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7570000000000002e-06 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563676+0000", 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3778000000000001e-05 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563696+0000", 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9587999999999999e-05 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563705+0000", 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8419999999999994e-06 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563758+0000", 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3327e-05 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563878+0000", 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120111 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.563988+0000", 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109616 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.565407+0000", 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014193630000000001 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.719803+0000", 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15439636400000001 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.788921+0000", 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.069117888000000002 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.788998+0000", 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6662999999999994e-05 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:40.789047+0000", 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9067999999999998e-05 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:62196 4.9 e337/327 4:91ca1c4e:::benchmark_data_smithi173_74914_object62195:head v 337'3860, mlcod=337'3856)", 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.773569+0000", 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 426.15261389199998, 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.310100351, 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773569+0000", 2024-08-30T23:42:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773569+0000", 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773570+0000", 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.773594+0000", 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4323000000000001e-05 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.081936+0000", 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30834138 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.081942+0000", 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.081994+0000", 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1647999999999997e-05 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.082008+0000", 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4134000000000001e-05 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083640+0000", 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016318050000000001 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:58.083669+0000", 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9275999999999999e-05 2024-08-30T23:42:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:122182 4.e 4:7aa6f75c:::benchmark_data_smithi173_74914_object23940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:35:56.818707+0000", 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 367.10747581099997, 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21992162000000001, 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122182 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818707+0000", 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818707+0000", 2024-08-30T23:42:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818708+0000", 2024-08-30T23:42:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-30T23:42:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818712+0000", 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5549999999999996e-06 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818725+0000", 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2499e-05 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818731+0000", 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6559999999999997e-06 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818770+0000", 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9826000000000001e-05 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818852+0000", 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1679000000000002e-05 2024-08-30T23:42:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.818919+0000", 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7219999999999997e-05 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.821868+0000", 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029483970000000002 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:56.822176+0000", 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000307903 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:57.038559+0000", 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21638298 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:57.038599+0000", 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0735000000000002e-05 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:35:57.038629+0000", 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9306e-05 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:188382 4.c 4:38db3f02:::benchmark_data_smithi173_74914_object90140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:09.185479+0000", 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 294.74070411299999, 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22511474200000001, 2024-08-30T23:42:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 188382 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185479+0000", 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185479+0000", 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185480+0000", 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3659999999999999e-06 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185485+0000", 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3190000000000002e-06 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185499+0000", 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4055e-05 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185507+0000", 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2590000000000003e-06 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185553+0000", 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5943999999999999e-05 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185701+0000", 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148658 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,6", 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.185837+0000", 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000135313 2024-08-30T23:42:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.190396+0000", 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045590400000000003 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.190998+0000", 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00060265399999999999 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.410522+0000", 2024-08-30T23:42:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21952385399999999 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.410554+0000", 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1832000000000003e-05 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.410593+0000", 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9447999999999999e-05 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:188386 4.e 4:747907c3:::benchmark_data_smithi173_74914_object90144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 294.73646320900002, 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21908520400000001, 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 188386 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189720+0000", 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4499999999999995e-07 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189725+0000", 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7079999999999996e-06 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189741+0000", 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5769000000000001e-05 2024-08-30T23:42:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189750+0000", 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8769999999999999e-06 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189812+0000", 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249000000000007e-05 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.189929+0000", 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011684 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.190053+0000", 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124556 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.193881+0000", 2024-08-30T23:42:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038277680000000001 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.197367+0000", 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034854560000000001 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.408731+0000", 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21136460600000001 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.408772+0000", 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0602999999999997e-05 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:09.408805+0000", 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2827000000000003e-05 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:28981 5.c e406/389 ondisk, result = 0)", 2024-08-30T23:42:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 244.484062163, 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21804868699999999, 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442121+0000", 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.15e-07 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.442123+0000", 2024-08-30T23:42:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.39e-06 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.659052+0000", 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21692828 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.659059+0000", 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.827e-06 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.660059+0000", 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010008630000000001 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.660068+0000", 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2430000000000005e-06 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:59.660169+0000", 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010166900000000001 2024-08-30T23:42:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:44553 5.c 5:323e8416:::benchmark_data_smithi173_87639_object44552:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 231.47761638399999, 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83468573899999998, 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44553 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448567+0000", 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.093e-06 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448624+0000", 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6285000000000001e-05 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448633+0000", 2024-08-30T23:42:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9830000000000002e-06 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448639+0000", 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.725e-06 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448672+0000", 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2815999999999999e-05 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448753+0000", 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1149999999999994e-05 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448841+0000", 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7267000000000003e-05 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.449924+0000", 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001083533 2024-08-30T23:42:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450381+0000", 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045717400000000002 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283161+0000", 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83277994799999999 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283216+0000", 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5067e-05 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283252+0000", 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5698000000000003e-05 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:44553 5.c e410/389 ondisk, result = 0)", 2024-08-30T23:42:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 231.47618165399999, 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83325748899999996, 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450002+0000", 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450005+0000", 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.58e-06 2024-08-30T23:42:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283028+0000", 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.833022924 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283038+0000", 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.713e-06 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283141+0000", 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102903 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283154+0000", 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.289e-05 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283259+0000", 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010485899999999999 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:45508 5.0 5:052f1fbc:::benchmark_data_smithi173_87639_object45507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:21.430861+0000", 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 222.49532144599999, 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.217851873, 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 45508 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430861+0000", 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430861+0000", 2024-08-30T23:42:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430863+0000", 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.81e-06 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430905+0000", 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1909000000000001e-05 2024-08-30T23:42:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430922+0000", 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6750000000000001e-05 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430929+0000", 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5259999999999996e-06 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.430984+0000", 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4923000000000003e-05 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.431138+0000", 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000154244 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:42:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.431233+0000", 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4503e-05 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.432903+0000", 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016699810000000001 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.445800+0000", 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012896960000000001 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.648509+0000", 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.202709108 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.648576+0000", 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7254000000000004e-05 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:21.648713+0000", 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013690499999999999 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58901 5.c 5:343594e4:::benchmark_data_smithi173_87639_object58900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:31.825342+0000", 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 212.10084048600001, 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.249786806, 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58901 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825342+0000", 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825342+0000", 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825343+0000", 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825411+0000", 2024-08-30T23:42:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7438e-05 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825422+0000", 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1089e-05 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825430+0000", 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8949999999999998e-06 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825507+0000", 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7114999999999996e-05 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825663+0000", 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015647999999999999 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:42:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.825793+0000", 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129617 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.827641+0000", 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018479060000000001 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:31.833079+0000", 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054384139999999999 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.075071+0000", 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24199152600000001 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.075103+0000", 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2162000000000001e-05 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.075129+0000", 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5998000000000001e-05 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58987 5.0 5.b63d9900 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 211.81138888999999, 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99942602000000003, 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58987 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114795+0000", 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114853+0000", 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7525000000000002e-05 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114863+0000", 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9899999999999992e-06 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114871+0000", 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5709999999999998e-06 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114907+0000", 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5661000000000003e-05 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114912+0000", 2024-08-30T23:42:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2809999999999999e-06 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114202+0000", 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99928982799999999 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114220+0000", 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7734e-05 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58996 5.0 5.b98b8b60 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 211.79898114100001, 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.987150694, 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58996 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127203+0000", 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.347e-06 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127240+0000", 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7403000000000001e-05 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127253+0000", 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2870999999999999e-05 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127260+0000", 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8800000000000002e-06 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127312+0000", 2024-08-30T23:42:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1431000000000003e-05 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127318+0000", 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6229999999999999e-06 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114337+0000", 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.987019172 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114352+0000", 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4966999999999999e-05 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59005 5.0 5.61e98650 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 211.76849291799999, 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95670884499999997, 2024-08-30T23:42:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59005 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157691+0000", 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181e-06 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157717+0000", 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5612000000000001e-05 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157729+0000", 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157734+0000", 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1800000000000004e-06 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157780+0000", 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5779999999999999e-05 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157787+0000", 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.815e-06 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114390+0000", 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95660253100000003 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114399+0000", 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1249999999999999e-06 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.024 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59006 5.0 5.aa303940 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 211.707885522, 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89614382599999998, 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59006 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218298+0000", 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-08-30T23:42:04.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218324+0000", 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5857e-05 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218335+0000", 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0716999999999999e-05 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218340+0000", 2024-08-30T23:42:04.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9490000000000004e-06 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218423+0000", 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3065000000000005e-05 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218430+0000", 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1400000000000002e-06 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114428+0000", 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89599851500000005 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114441+0000", 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2727e-05 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.c e389 from shard 0)", 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.750328705000001, 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.840740928, 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175855+0000", 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175892+0000", 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7345000000000002e-05 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016334+0000", 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.84044191000000001 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016342+0000", 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0420000000000003e-06 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016459+0000", 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116371 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016469+0000", 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0713999999999999e-05 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016595+0000", 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012576500000000001 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:86627 5.c 5:399a37c9:::benchmark_data_smithi173_87639_object6387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e492)", 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:12.317645+0000", 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.608538058000001, 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35417571399999997, 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.034 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 86627 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317645+0000", 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317645+0000", 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317646+0000", 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.71e-07 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317649+0000", 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.63e-06 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317657+0000", 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0199999999999994e-06 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317662+0000", 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7029999999999997e-06 2024-08-30T23:42:04.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317698+0000", 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5827000000000001e-05 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317812+0000", 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114176 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:42:04.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.317924+0000", 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112537 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.319969+0000", 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020448229999999999 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.320724+0000", 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000755191 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671715+0000", 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35099032000000002 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671773+0000", 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8400999999999999e-05 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671820+0000", 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7215000000000001e-05 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:86627 5.c e492/389 ondisk, result = 0)", 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:12.320527+0000", 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.605655370999997, 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35129798899999998, 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:04.041 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.320527+0000", 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.320527+0000", 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.320528+0000", 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.99e-07 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.320534+0000", 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3949999999999998e-06 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671648+0000", 2024-08-30T23:42:04.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35111355 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671655+0000", 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549999999999997e-06 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671698+0000", 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3689999999999997e-05 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671708+0000", 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2329999999999995e-06 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:12.671825+0000", 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117667 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:04.045 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:04.046 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:42:04.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:04.046 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:03 smithi106 podman[165197]: 2024-08-30 23:42:03.775824882 +0000 UTC m=+0.017241414 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:42:04.047 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:03 smithi106 podman[165197]: 2024-08-30 23:42:03.894960116 +0000 UTC m=+0.136376650 container create 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-30T23:42:04.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:04.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:04.192 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:04.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:42:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:04 smithi136 ceph-mon[20207]: osdmap e520: 12 total, 10 up, 12 in 2024-08-30T23:42:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:04 smithi136 ceph-mon[20207]: pgmap v1206: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 21034/221586 objects misplaced (9.492%) 2024-08-30T23:42:04.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:04.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:04 smithi173 ceph-mon[20554]: osdmap e520: 12 total, 10 up, 12 in 2024-08-30T23:42:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:04 smithi173 ceph-mon[20554]: pgmap v1206: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 21034/221586 objects misplaced (9.492%) 2024-08-30T23:42:04.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:42:04.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:04 smithi106 ceph-mon[16603]: osdmap e520: 12 total, 10 up, 12 in 2024-08-30T23:42:04.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:04 smithi106 ceph-mon[16603]: pgmap v1206: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 21034/221586 objects misplaced (9.492%) 2024-08-30T23:42:04.349 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:04 smithi106 podman[165197]: 2024-08-30 23:42:04.148174408 +0000 UTC m=+0.389590937 container init 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-08-30T23:42:04.349 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:04 smithi106 podman[165197]: 2024-08-30 23:42:04.151521494 +0000 UTC m=+0.392938024 container start 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-08-30T23:42:04.349 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:04 smithi106 ceph-osd[165233]: -- 172.21.15.106:0/2213138687 <== mon.0 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5622a2cc5a40 con 0x5622a3a92400 2024-08-30T23:42:04.349 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:04 smithi106 bash[165197]: 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 2024-08-30T23:42:04.349 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:04 smithi106 systemd[1]: Started Ceph osd.1 for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:42:04.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:04.438 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:04.510 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:04.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:04.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:04.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:04.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.686 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:04.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:04.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:42:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:04.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:04.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:04.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:04.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:05.010 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:42:05.010 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:42:05.010 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:42:05.010 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:42:05.010 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:42:05.010 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:42:05.011 INFO:teuthology.orchestra.run.smithi106.stderr:config diff 2024-08-30T23:42:05.011 INFO:teuthology.orchestra.run.smithi106.stderr:config diff get 2024-08-30T23:42:05.011 INFO:teuthology.orchestra.run.smithi106.stderr:config get 2024-08-30T23:42:05.011 INFO:teuthology.orchestra.run.smithi106.stderr:config help [] 2024-08-30T23:42:05.011 INFO:teuthology.orchestra.run.smithi106.stderr:config set ... 2024-08-30T23:42:05.011 INFO:teuthology.orchestra.run.smithi106.stderr:admin_socket: invalid command 2024-08-30T23:42:05.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:05.051 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:05.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:05.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:05.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:05.221 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:05.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:05.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:05.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:05.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:05.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:05.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:05.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 520/491 [PushOp(5:0f9b2dde:::benchmark_data_smithi173_87639_object40895:head, version: 408'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0f9b2dde:::benchmark_data_smithi173_87639_object40895:head@408'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:05.651469+0000", 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00038266199999999998, 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041986299999999997, 2024-08-30T23:42:05.652 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651469+0000", 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651469+0000", 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:05.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651470+0000", 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3300000000000002e-07 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651544+0000", 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4833999999999995e-05 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:05.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651575+0000", 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0843999999999998e-05 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651586+0000", 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0616e-05 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651657+0000", 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0864000000000005e-05 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:05.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.651666+0000", 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1430000000000004e-06 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:42:05.656 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:42:05.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:05.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:42:05.699 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-08-30T23:42:05.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:42:05.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:05.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:05 smithi136 ceph-mon[20207]: pgmap v1207: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 21034/221586 objects misplaced (9.492%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:42:05.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:05.890 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:05 smithi106 ceph-mon[16603]: pgmap v1207: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 21034/221586 objects misplaced (9.492%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:42:05.890 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:05 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[165229]: 2024-08-30T23:42:05.809+0000 7f83b036f640 -1 Falling back to public interface 2024-08-30T23:42:05.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:05.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:05.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:06.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:06.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:06.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:05 smithi173 ceph-mon[20554]: pgmap v1207: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 21034/221586 objects misplaced (9.492%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:42:06.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:06.172 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:06.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:06.271 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:06.272 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:06.272 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:42:06.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:06.288 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:06.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:06.296 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:06.297 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:06.321 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:06.321 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:06.343 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:06.343 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:06.343 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:06.364 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:06.365 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:06.387 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:06.387 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:06.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:06.407 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:06.407 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:06.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:06.428 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:06.428 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:06.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:42:06.448 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:06.448 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:06.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:06.470 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:06.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:06.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:06.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:06.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:06.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:06.715 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:06.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:06.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 27845/221586 objects degraded (12.566%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/496592172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:06.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 27845/221586 objects degraded (12.566%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:06.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/496592172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:06.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:06.913 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:06.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:07.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:07.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 27845/221586 objects degraded (12.566%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/496592172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:07.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:07.160 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:07.252 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:07.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:07.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:07.321 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:07.321 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:07.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:07.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:07.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:07.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:07.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:07.660 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:07.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:07.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:07 smithi106 ceph-mon[16603]: pgmap v1208: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 20398/221586 objects misplaced (9.205%); 13 MiB/s, 211 objects/s recovering 2024-08-30T23:42:07.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:07.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:07.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:07.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:07 smithi136 ceph-mon[20207]: pgmap v1208: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 20398/221586 objects misplaced (9.205%); 13 MiB/s, 211 objects/s recovering 2024-08-30T23:42:07.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:42:08.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:08.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:08.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:07 smithi173 ceph-mon[20554]: pgmap v1208: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 20398/221586 objects misplaced (9.205%); 13 MiB/s, 211 objects/s recovering 2024-08-30T23:42:08.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:08.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:08.240 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:08.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:08.279 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:08.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:08.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:08.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:08.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:08.628 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:08.641 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:42:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.653 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:08.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:08.744 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:08.744 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:08.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:08.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:08.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:08.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:08.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:08.938 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:08 smithi106 ceph-mon[16603]: osdmap e521: 12 total, 10 up, 12 in 2024-08-30T23:42:08.938 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4230213850' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:09.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:09.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:08 smithi136 ceph-mon[20207]: osdmap e521: 12 total, 10 up, 12 in 2024-08-30T23:42:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4230213850' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:08 smithi173 ceph-mon[20554]: osdmap e521: 12 total, 10 up, 12 in 2024-08-30T23:42:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4230213850' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:09.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:09.146 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:09.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:09.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:42:09.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:09.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:09.392 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:09.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:42:09.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:09.510 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:09.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:09.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:09.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:09.730 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:09.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:09.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:09 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4230213850' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:42:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:09 smithi136 ceph-mon[20207]: osdmap e522: 12 total, 10 up, 12 in 2024-08-30T23:42:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:09 smithi136 ceph-mon[20207]: pgmap v1211: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 19926/221586 objects misplaced (8.992%); 14 MiB/s, 247 objects/s recovering 2024-08-30T23:42:09.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:09.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:09.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:09.918 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:09 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[165229]: 2024-08-30T23:42:09.798+0000 7f83b036f640 -1 osd.1 368 log_to_monitors true 2024-08-30T23:42:09.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:09 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:09.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4230213850' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:42:09.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:09 smithi106 ceph-mon[16603]: osdmap e522: 12 total, 10 up, 12 in 2024-08-30T23:42:09.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:09 smithi106 ceph-mon[16603]: pgmap v1211: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 19926/221586 objects misplaced (8.992%); 14 MiB/s, 247 objects/s recovering 2024-08-30T23:42:09.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:10.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:10.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:09 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4230213850' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:42:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:09 smithi173 ceph-mon[20554]: osdmap e522: 12 total, 10 up, 12 in 2024-08-30T23:42:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:09 smithi173 ceph-mon[20554]: pgmap v1211: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 10 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27845/221586 objects degraded (12.566%); 19926/221586 objects misplaced (8.992%); 14 MiB/s, 247 objects/s recovering 2024-08-30T23:42:10.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:10.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:10.151 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:10.151 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:10.151 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-30T23:42:10.172 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:10.172 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:10.192 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:10.192 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:10.199 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:10.199 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:10.212 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:10.213 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:10.213 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:10.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:10.234 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:10.234 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:10.254 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:10.254 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:10.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:10.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:10.274 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:10.295 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:10.295 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:10.316 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:10.316 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:10.336 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:10.336 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:10.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:10.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:10.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:10.500 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:10.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:10.623 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:10.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:42:10.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:10.747 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:10 smithi136 ceph-mon[20207]: 5.18 scrub starts 2024-08-30T23:42:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:10 smithi136 ceph-mon[20207]: osdmap e523: 12 total, 10 up, 12 in 2024-08-30T23:42:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:10 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-30T23:42:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/614243090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:10.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:10.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:42:10.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:10.922 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:10 smithi106 ceph-mon[16603]: 5.18 scrub starts 2024-08-30T23:42:10.922 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:10 smithi106 ceph-mon[16603]: osdmap e523: 12 total, 10 up, 12 in 2024-08-30T23:42:10.922 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:10 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-30T23:42:10.922 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/614243090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:10.923 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 30 23:42:10 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[165229]: 2024-08-30T23:42:10.703+0000 7f83a4111640 -1 osd.1 524 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:42:10.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:10.994 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:10.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:11.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:10 smithi173 ceph-mon[20554]: 5.18 scrub starts 2024-08-30T23:42:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:10 smithi173 ceph-mon[20554]: osdmap e523: 12 total, 10 up, 12 in 2024-08-30T23:42:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:10 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-30T23:42:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/614243090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:11.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:11.108 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:11.115 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:11.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:11.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:11.241 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:11.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:11.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:11.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:42:11.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:11.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:11.555 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:42:11.555 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:42:11.555 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:42:11.555 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:42:11.555 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:42:11.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:11.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:11.622 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:42:11.622 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:42:11.622 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:42:11.622 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:42:11.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:11.739 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:11.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:11 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-30T23:42:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:11 smithi106 ceph-mon[16603]: osdmap e524: 12 total, 10 up, 12 in 2024-08-30T23:42:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:11 smithi106 ceph-mon[16603]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:42:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 27845/221586 objects degraded (12.566%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:11 smithi106 ceph-mon[16603]: 5.12 scrub starts 2024-08-30T23:42:11.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:11 smithi106 ceph-mon[16603]: pgmap v1214: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 11 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25582/221586 objects degraded (11.545%); 21289/221586 objects misplaced (9.608%); 8.4 MiB/s, 185 objects/s recovering 2024-08-30T23:42:11.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:11 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-30T23:42:11.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:11 smithi136 ceph-mon[20207]: osdmap e524: 12 total, 10 up, 12 in 2024-08-30T23:42:11.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:11 smithi136 ceph-mon[20207]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:42:11.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 27845/221586 objects degraded (12.566%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:11.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:11 smithi136 ceph-mon[20207]: 5.12 scrub starts 2024-08-30T23:42:11.860 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:11 smithi136 ceph-mon[20207]: pgmap v1214: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 11 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25582/221586 objects degraded (11.545%); 21289/221586 objects misplaced (9.608%); 8.4 MiB/s, 185 objects/s recovering 2024-08-30T23:42:11.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:11.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:11.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:11.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:11 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-30T23:42:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:11 smithi173 ceph-mon[20554]: osdmap e524: 12 total, 10 up, 12 in 2024-08-30T23:42:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:11 smithi173 ceph-mon[20554]: from='osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]} : dispatch 2024-08-30T23:42:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 27845/221586 objects degraded (12.566%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:11 smithi173 ceph-mon[20554]: 5.12 scrub starts 2024-08-30T23:42:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:11 smithi173 ceph-mon[20554]: pgmap v1214: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 11 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25582/221586 objects degraded (11.545%); 21289/221586 objects misplaced (9.608%); 8.4 MiB/s, 185 objects/s recovering 2024-08-30T23:42:12.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:12.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:12.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:12.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:42:12.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:12.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:12.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:12.427 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:12.452 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:42:12.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:12.482 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:12.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:12.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:12.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:12.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:12.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:12.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:12.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:12.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:12 smithi136 ceph-mon[20207]: Health check update: 1 osds down (OSD_DOWN) 2024-08-30T23:42:12.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:42:12.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:12 smithi136 ceph-mon[20207]: osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139] boot 2024-08-30T23:42:12.833 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:12 smithi136 ceph-mon[20207]: osdmap e525: 12 total, 11 up, 12 in 2024-08-30T23:42:12.855 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:12.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:12.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:12.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:12.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:12 smithi106 ceph-mon[16603]: Health check update: 1 osds down (OSD_DOWN) 2024-08-30T23:42:12.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:42:12.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:12 smithi106 ceph-mon[16603]: osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139] boot 2024-08-30T23:42:12.903 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:12 smithi106 ceph-mon[16603]: osdmap e525: 12 total, 11 up, 12 in 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61137 4.0 4:05184379:::benchmark_data_smithi173_74914_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:42:12.939 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 438.52466722399998, 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0987384589999998, 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61137 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413067+0000", 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2640000000000001e-06 2024-08-30T23:42:12.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413084+0000", 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6407999999999999e-05 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413095+0000", 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277999999999999e-05 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413117+0000", 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1498e-05 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413168+0000", 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1266e-05 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413265+0000", 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6620000000000007e-05 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413359+0000", 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4003000000000001e-05 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415078+0000", 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719478 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415468+0000", 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038999199999999998 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511679+0000", 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0962105000000002 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511758+0000", 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9426999999999997e-05 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511805+0000", 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6724999999999997e-05 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61138 4.0 4:0601b2bd:::benchmark_data_smithi173_74914_object61137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 437.78401246499999, 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582538660000001, 2024-08-30T23:42:12.946 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61138 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153722+0000", 2024-08-30T23:42:12.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-06 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153826+0000", 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103683 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153839+0000", 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3169e-05 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153863+0000", 2024-08-30T23:42:12.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4326000000000001e-05 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153921+0000", 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7283e-05 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154051+0000", 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012986399999999999 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154154+0000", 2024-08-30T23:42:12.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103188 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156034+0000", 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018801009999999999 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156113+0000", 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8807000000000006e-05 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511881+0000", 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3557679949999999 2024-08-30T23:42:12.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511932+0000", 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1134999999999997e-05 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511975+0000", 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2973999999999999e-05 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:62193 4.8 e337/329 4:198a939b:::benchmark_data_smithi173_74914_object62192:head v 337'3972, mlcod=337'3968)", 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 435.332608069, 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.349666123, 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:42:12.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605125+0000", 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605126+0000", 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.13e-06 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.605148+0000", 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1759e-05 2024-08-30T23:42:12.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952851+0000", 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34770309199999999 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952856+0000", 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1220000000000003e-06 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952891+0000", 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4292999999999998e-05 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.952908+0000", 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7277000000000001e-05 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954758+0000", 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018501889999999999 2024-08-30T23:42:12.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:57.954791+0000", 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3260999999999999e-05 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:143768 4.3 4:c93e37ef:::benchmark_data_smithi173_74914_object45526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:42:12.958 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 353.50611410800002, 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37550649899999999, 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 143768 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431619+0000", 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431621+0000", 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-08-30T23:42:12.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431627+0000", 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431639+0000", 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2404e-05 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431646+0000", 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9199999999999998e-06 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431724+0000", 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7655000000000001e-05 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431814+0000", 2024-08-30T23:42:12.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0143999999999996e-05 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.431885+0000", 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1421999999999995e-05 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.435174+0000", 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003288394 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.435587+0000", 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041313500000000002 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807016+0000", 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37142968700000001 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807076+0000", 2024-08-30T23:42:12.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9561000000000001e-05 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:36:19.807126+0000", 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9864999999999998e-05 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:196473 4.0 4:0415f716:::benchmark_data_smithi173_74914_object98231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:42:12.965 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 294.913472824, 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49994438299999999, 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 196473 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:42:12.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024266+0000", 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1779999999999996e-06 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024277+0000", 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1567e-05 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024284+0000", 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4389999999999997e-06 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024336+0000", 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2145000000000003e-05 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024443+0000", 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107633 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024556+0000", 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011263 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029319+0000", 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047626949999999999 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029471+0000", 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015204800000000001 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524119+0000", 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49464797199999999 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524156+0000", 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7716000000000003e-05 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524205+0000", 2024-08-30T23:42:12.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8591000000000003e-05 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.25654.0:196478 4.a e381/369 4:550417b9:::benchmark_data_smithi173_74914_object98236:head v 381'12132, mlcod=381'12130)", 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 294.90837804099999, 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.406346965, 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029355+0000", 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029356+0000", 2024-08-30T23:42:12.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5400000000000001e-07 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029361+0000", 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6739999999999996e-06 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431719+0000", 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.402358571 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431734+0000", 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4963000000000001e-05 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431842+0000", 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107765 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.431888+0000", 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6269e-05 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:42:12.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.435671+0000", 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037823639999999999 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.435702+0000", 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1705000000000002e-05 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.976 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 451/432 [PushOp(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head, version: 400'576, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head@400'576, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 165.33820443600001, 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44893841800000001, 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599530+0000", 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4339999999999999e-06 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599595+0000", 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4184999999999996e-05 2024-08-30T23:42:12.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046675+0000", 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.447080796 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046706+0000", 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0505999999999999e-05 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046916+0000", 2024-08-30T23:42:12.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021055600000000001 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046944+0000", 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7492999999999999e-05 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.048467+0000", 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015234479999999999 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 451/432 [PushOp(5:c3d2b08f:::benchmark_data_smithi173_87639_object9384:head, version: 400'582, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c3d2b08f:::benchmark_data_smithi173_87639_object9384:head@400'582, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:28.145491+0000", 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.79224289699999, 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.39316452299999999, 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145491+0000", 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145491+0000", 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145492+0000", 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.145609+0000", 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117631 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536849+0000", 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.39123995900000003 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536864+0000", 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5037e-05 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536953+0000", 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8413999999999997e-05 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.536963+0000", 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0265000000000001e-05 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.538655+0000", 2024-08-30T23:42:12.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016920769999999999 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 463/432 [PushOp(5:c0945bbd:::benchmark_data_smithi173_87639_object35921:head, version: 407'2270, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c0945bbd:::benchmark_data_smithi173_87639_object35921:head@407'2270, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:09.511082+0000", 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 123.426651178, 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32755928600000001, 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511082+0000", 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511082+0000", 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511083+0000", 2024-08-30T23:42:12.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5099999999999999e-07 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.511113+0000", 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0315999999999999e-05 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836525+0000", 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32541161000000002 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836544+0000", 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8987e-05 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836678+0000", 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013441899999999999 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.836692+0000", 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.381e-05 2024-08-30T23:42:12.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:09.838642+0000", 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019494930000000001 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 464/432 [PushOp(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head, version: 408'2477, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head@408'2477, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 119.860450435, 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52881942000000004, 2024-08-30T23:42:12.989 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077310+0000", 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6896000000000001e-05 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077371+0000", 2024-08-30T23:42:12.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1471999999999997e-05 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604348+0000", 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52697629300000004 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604378+0000", 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0654999999999997e-05 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604592+0000", 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021338600000000001 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604609+0000", 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7244999999999998e-05 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.606102+0000", 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014934729999999999 2024-08-30T23:42:12.993 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "replica_scrub(pg: 5.1,from:0'0,to:445'5203,epoch:470/389,start:5:8a8f2f50:::benchmark_data_smithi173_87639_object56981:0,end:5:8ae6a8d1:::benchmark_data_smithi173_87639_object64309:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.814913112, 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74865697399999998, 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122821+0000", 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7599999999999997e-07 2024-08-30T23:42:12.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122828+0000", 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-06 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871217+0000", 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74838935200000001 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:12.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871235+0000", 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7708999999999999e-05 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871407+0000", 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172153 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871447+0000", 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9582999999999997e-05 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871477+0000", 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0501000000000002e-05 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.9 e488 from shard 6)", 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 62.060384904999999, 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49092997399999999, 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:12.998 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:12.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.34e-07 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877368+0000", 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8964999999999999e-05 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368058+0000", 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.490690554 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368065+0000", 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.72e-06 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368139+0000", 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3943000000000007e-05 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368148+0000", 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.67e-06 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368278+0000", 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013068800000000001 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e502,491 5:9f5e9ce2:::benchmark_data_smithi173_87639_object5778:head,446'5327)", 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:42:13.002 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 39.962898297999999, 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80846543500000001, 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:42:13.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9400000000000001e-07 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974839+0000", 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1080000000000001e-06 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783082+0000", 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80824320199999999 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783093+0000", 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1122e-05 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783147+0000", 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4397000000000001e-05 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783301+0000", 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015331200000000001 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 505/491 [PushOp(5:0a8a0c4f:::benchmark_data_smithi173_87639_object7264:head, version: 399'488, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0a8a0c4f:::benchmark_data_smithi173_87639_object7264:head@399'488, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.676728167, 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76577126500000003, 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.006 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261006+0000", 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9699999999999996e-07 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261041+0000", 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4728999999999999e-05 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025849+0000", 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76480819099999997 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025877+0000", 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8016000000000001e-05 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025956+0000", 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9369000000000005e-05 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025971+0000", 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4516e-05 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.026777+0000", 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080584700000000005 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.011 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 511/491 [PushOp(5:0e383089:::benchmark_data_smithi173_87639_object11978:head, version: 401'782, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0e383089:::benchmark_data_smithi173_87639_object11978:head@401'782, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.622713739000002, 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88692451299999997, 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315021+0000", 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3859999999999999e-06 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315100+0000", 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9388000000000006e-05 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200480+0000", 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88538003499999995 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200526+0000", 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5198999999999999e-05 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200744+0000", 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021846300000000001 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200767+0000", 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2996000000000001e-05 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.201944+0000", 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001177046 2024-08-30T23:42:13.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 511/491 [PushOp(5:0fb66bc9:::benchmark_data_smithi173_87639_object14176:head, version: 402'906, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0fb66bc9:::benchmark_data_smithi173_87639_object14176:head@402'906, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:44.718672+0000", 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.219061199999999, 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32876297700000001, 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:44.718672+0000", 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:44.718672+0000", 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:44.718673+0000", 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5000000000000002e-07 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:44.718731+0000", 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7995999999999998e-05 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.046464+0000", 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32773300300000002 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.046474+0000", 2024-08-30T23:42:13.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0234000000000001e-05 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.046537+0000", 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2681999999999994e-05 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.046544+0000", 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6559999999999992e-06 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.047435+0000", 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00089075600000000003 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 511/491 [PushOp(5:1e005da2:::benchmark_data_smithi173_87639_object35084:head, version: 407'2211, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1e005da2:::benchmark_data_smithi173_87639_object35084:head@407'2211, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.561107185000001, 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42476207799999999, 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376636+0000", 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0217000000000001e-05 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376737+0000", 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100571 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800459+0000", 2024-08-30T23:42:13.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.423722339 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800470+0000", 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0519e-05 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800543+0000", 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2799999999999994e-05 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800557+0000", 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4208e-05 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.801388+0000", 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083142399999999999 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 518/491 [PushOp(5:1ade8763:::benchmark_data_smithi173_87639_object73716:head, version: 436'4607, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1ade8763:::benchmark_data_smithi173_87639_object73716:head@436'4607, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.874708853, 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59454703399999997, 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8400000000000004e-07 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063054+0000", 2024-08-30T23:42:13.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9164999999999999e-05 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656490+0000", 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59343594399999999 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656518+0000", 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8041000000000001e-05 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656611+0000", 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3068999999999999e-05 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656622+0000", 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0903999999999999e-05 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.657572+0000", 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094932700000000003 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 519/491 [PushOp(5:092b6415:::benchmark_data_smithi173_87639_object36917:head, version: 408'2349, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:092b6415:::benchmark_data_smithi173_87639_object36917:head@408'2349, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.380616304, 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68386947499999995, 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557119+0000", 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4670000000000001e-06 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557185+0000", 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6804999999999995e-05 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239646+0000", 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68246094099999999 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239664+0000", 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8122999999999999e-05 2024-08-30T23:42:13.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239802+0000", 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013746299999999999 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239814+0000", 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2029e-05 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.240987+0000", 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001172647 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(5.18 e520,491 5:1cf428d8:::benchmark_data_smithi173_87639_object1164:head,440'5076)", 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.7564454840000003, 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54704231800000003, 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:42:13.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181289+0000", 2024-08-30T23:42:13.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.387e-06 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181305+0000", 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5756e-05 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728218+0000", 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54691292999999996 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728224+0000", 2024-08-30T23:42:13.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3759999999999996e-06 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728270+0000", 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6016000000000001e-05 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728277+0000", 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:42:13.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728330+0000", 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3631000000000002e-05 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:42:13.037 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:42:13.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:13.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:42:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:12 smithi173 ceph-mon[20554]: Health check update: 1 osds down (OSD_DOWN) 2024-08-30T23:42:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-30T23:42:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:12 smithi173 ceph-mon[20554]: osd.1 [v2:172.21.15.106:6810/2928503139,v1:172.21.15.106:6811/2928503139] boot 2024-08-30T23:42:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:12 smithi173 ceph-mon[20554]: osdmap e525: 12 total, 11 up, 12 in 2024-08-30T23:42:13.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:13.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:13.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:13.306 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:13.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:13.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:13.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.430 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:13.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:13.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:13.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:13.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:13.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:13.695 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:42:13.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:42:13.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:13.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:13.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:13.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:13.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:13.960 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:14.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:14.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: osdmap e526: 12 total, 11 up, 12 in 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3488222339' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: pgmap v1217: 47 pgs: 3 remapped+peering, 1 remapped, 1 active+recovering+undersized+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 20899/221586 objects degraded (9.432%); 16538/221586 objects misplaced (7.463%) 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3488222339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:13 smithi136 ceph-mon[20207]: osdmap e527: 12 total, 11 up, 12 in 2024-08-30T23:42:14.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: osdmap e526: 12 total, 11 up, 12 in 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3488222339' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: pgmap v1217: 47 pgs: 3 remapped+peering, 1 remapped, 1 active+recovering+undersized+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 20899/221586 objects degraded (9.432%); 16538/221586 objects misplaced (7.463%) 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3488222339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:13 smithi173 ceph-mon[20554]: osdmap e527: 12 total, 11 up, 12 in 2024-08-30T23:42:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: osdmap e526: 12 total, 11 up, 12 in 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3488222339' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: pgmap v1217: 47 pgs: 3 remapped+peering, 1 remapped, 1 active+recovering+undersized+remapped, 2 active+undersized, 9 active+clean+remapped, 1 undersized+degraded+peered, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 20899/221586 objects degraded (9.432%); 16538/221586 objects misplaced (7.463%) 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3488222339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:13 smithi106 ceph-mon[16603]: osdmap e527: 12 total, 11 up, 12 in 2024-08-30T23:42:14.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:14.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:14.274 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:14.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:14.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:14.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:14.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:87141 5.a e527/457 5:5736895d:::benchmark_data_smithi173_87639_object6901:head v 527'5497, mlcod=527'5495)", 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.527714+0000", 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0066801610000000004, 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067066469999999996, 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.536 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.527714+0000", 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.527714+0000", 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.527715+0000", 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5099999999999999e-07 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.527717+0000", 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.514e-06 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.527794+0000", 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6888e-05 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.527798+0000", 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2180000000000001e-06 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528665+0000", 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00086686900000000004 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528682+0000", 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7152000000000001e-05 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:87142 5.14 e527/490 5:2e290d11:::benchmark_data_smithi173_87639_object6902:head v 527'5386, mlcod=527'5384)", 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.528567+0000", 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0058280160000000001, 2024-08-30T23:42:14.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00591253, 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528567+0000", 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528567+0000", 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528567+0000", 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6600000000000002e-07 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528573+0000", 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3539999999999996e-06 2024-08-30T23:42:14.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528608+0000", 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4567999999999998e-05 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528614+0000", 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2709999999999998e-06 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528662+0000", 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8149999999999998e-05 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.528680+0000", 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7277000000000001e-05 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.544 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:87144 5.15 5:af23f0d8:::benchmark_data_smithi173_87639_object6904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.529143+0000", 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.005251548, 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053896810000000003, 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87144 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.529143+0000", 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.529143+0000", 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.529144+0000", 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4079999999999999e-06 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.529151+0000", 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3729999999999998e-06 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.529162+0000", 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1294999999999999e-05 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.529169+0000", 2024-08-30T23:42:14.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8419999999999999e-06 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.531998+0000", 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028291269999999999 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532134+0000", 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013560199999999999 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532271+0000", 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013764300000000001 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:87143 5.14 e527/490 5:2dad4f31:::benchmark_data_smithi173_87639_object6903:head v 527'5387, mlcod=527'5384)", 2024-08-30T23:42:14.549 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.530194+0000", 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.004200807, 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044027110000000001, 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530194+0000", 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530194+0000", 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530195+0000", 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7700000000000003e-07 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530201+0000", 2024-08-30T23:42:14.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8340000000000001e-06 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530233+0000", 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1674999999999997e-05 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530239+0000", 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1700000000000002e-06 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532184+0000", 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019448239999999999 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532206+0000", 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1838e-05 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:87145 5.12 5:4b0602d8:::benchmark_data_smithi173_87639_object6905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.530225+0000", 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0041692719999999999, 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044244829999999999, 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:14.555 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87145 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530225+0000", 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530225+0000", 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530226+0000", 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8299999999999995e-07 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530239+0000", 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2320000000000001e-05 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530251+0000", 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2041e-05 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530258+0000", 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9970000000000001e-06 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530443+0000", 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018576500000000001 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530561+0000", 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118071 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530674+0000", 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011272700000000001 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.533841+0000", 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003166468 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:87147 5.12 5:4faf8b70:::benchmark_data_smithi173_87639_object6907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.530899+0000", 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0034955289999999998, 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0038153369999999998, 2024-08-30T23:42:14.560 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87147 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530899+0000", 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530899+0000", 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530900+0000", 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.78e-07 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530905+0000", 2024-08-30T23:42:14.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7809999999999996e-06 2024-08-30T23:42:14.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530917+0000", 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2038000000000001e-05 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.530924+0000", 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3250000000000002e-06 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532173+0000", 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012484180000000001 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532281+0000", 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010875 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.532687+0000", 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040526800000000002 2024-08-30T23:42:14.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:87129 5.15 e527/526 ondisk, result = 0)", 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.534164+0000", 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.000230676, 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00060968999999999997, 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:42:14.567 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534164+0000", 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534164+0000", 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534165+0000", 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4799999999999998e-07 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534170+0000", 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0889999999999999e-06 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534225+0000", 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5049000000000001e-05 2024-08-30T23:42:14.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534231+0000", 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5490000000000003e-06 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534708+0000", 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00047669999999999999 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534717+0000", 2024-08-30T23:42:14.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0219999999999993e-06 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:87144 5.15 e527/526)", 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.534507+0000", 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4294967295.9998879, 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031112300000000001, 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534507+0000", 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534507+0000", 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534507+0000", 2024-08-30T23:42:14.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9999999999999996e-07 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534513+0000", 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.13e-06 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534573+0000", 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0302e-05 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534579+0000", 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5690000000000001e-06 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:87147 5.12 e527/523 ondisk, result = 0)", 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.534517+0000", 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4294967295.9998775, 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00033453699999999999, 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.574 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534517+0000", 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534517+0000", 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534517+0000", 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0200000000000001e-07 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534519+0000", 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5239999999999999e-06 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534595+0000", 2024-08-30T23:42:14.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6457000000000003e-05 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534599+0000", 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.177e-06 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534823+0000", 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000223365 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534829+0000", 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.212e-06 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:87149 5.4 e527/488 5:2400e38c:::benchmark_data_smithi173_87639_object6909:head v 527'5382, mlcod=527'5380)", 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.534525+0000", 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4294967295.9998698, 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000371994, 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.578 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534525+0000", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534525+0000", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534525+0000", 2024-08-30T23:42:14.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.91e-07 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534530+0000", 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7929999999999997e-06 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534616+0000", 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6456e-05 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534621+0000", 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.702e-06 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534664+0000", 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2568999999999998e-05 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534687+0000", 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3546000000000001e-05 2024-08-30T23:42:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:87129 5.15 5:aff25046:::benchmark_data_smithi173_87639_object6889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e527)", 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:14.520071+0000", 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.014323377, 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014870603, 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:42:14.582 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87129 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.520071+0000", 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.520071+0000", 2024-08-30T23:42:14.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.520072+0000", 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.136e-06 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.520077+0000", 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8949999999999997e-06 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.520089+0000", 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2184999999999999e-05 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.520097+0000", 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8390000000000007e-06 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:42:14.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.524052+0000", 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039550430000000001 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.524185+0000", 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013222099999999999 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.524351+0000", 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000165973 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.531955+0000", 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0076040279999999997 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.531973+0000", 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8559999999999998e-05 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534724+0000", 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0027509180000000002 2024-08-30T23:42:14.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:14.534785+0000", 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1072000000000001e-05 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 11 2024-08-30T23:42:14.588 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:42:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:14.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:14.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:14.725 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:14.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:14.855 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:14.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:14.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:14 smithi106 ceph-mon[16603]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:42:14.957 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:15.000 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:14 smithi136 ceph-mon[20207]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:42:15.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:14 smithi173 ceph-mon[20554]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:42:15.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:15.136 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:15.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:15.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:15.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:15.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:15.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:42:15.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:15.518 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:15.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:15.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:15.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:15.734 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:15 smithi173 ceph-mon[20554]: pgmap v1219: 47 pgs: 3 remapped+peering, 1 remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 53 op/s; 20725/220791 objects degraded (9.387%); 16538/220791 objects misplaced (7.490%) 2024-08-30T23:42:15.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:15.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:15.842 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:15.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:15.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:15.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:16.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:16.029 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:15 smithi106 ceph-mon[16603]: pgmap v1219: 47 pgs: 3 remapped+peering, 1 remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 53 op/s; 20725/220791 objects degraded (9.387%); 16538/220791 objects misplaced (7.490%) 2024-08-30T23:42:16.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:16.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:15 smithi136 ceph-mon[20207]: pgmap v1219: 47 pgs: 3 remapped+peering, 1 remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 9 active+clean+remapped, 4 active+undersized+degraded, 26 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 53 op/s; 20725/220791 objects degraded (9.387%); 16538/220791 objects misplaced (7.490%) 2024-08-30T23:42:16.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:16.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:16.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:16.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:16.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:16.412 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:42:16.412 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:42:16.412 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:42:16.412 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:42:16.412 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:42:16.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:16.522 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:16.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:16.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:16.674 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:16.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:16.799 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:16.886 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:16.886 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:16.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:16.910 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:42:16.910 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-30T23:42:16.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:16.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:16.934 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:16.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:16.958 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:16.958 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:16.982 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:16.982 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:16.982 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:17.005 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:17.006 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 20725/220791 objects degraded (9.387%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: osdmap e528: 12 total, 11 up, 12 in 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/399228636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:42:17.006 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:16 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:17.032 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:17.032 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:17.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 20725/220791 objects degraded (9.387%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: osdmap e528: 12 total, 11 up, 12 in 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/399228636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:42:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:16 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:17.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:17.057 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:17.057 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:17.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 20725/220791 objects degraded (9.387%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: osdmap e528: 12 total, 11 up, 12 in 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/399228636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:42:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:16 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:17.084 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:17.085 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:17.112 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:17.112 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:17.139 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:17.140 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:17.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:17.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:17.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:42:17.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:17.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:17.373 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:42:17.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:42:17.373 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-30T23:42:17.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-08-30T23:42:17.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:17.443 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:17.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:42:17.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:17.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:17.589 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:17.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:17.698 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:17.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:17.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:17.927 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:17.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:17.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:17.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:18.010 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:17 smithi173 ceph-mon[20554]: 5.12 scrub starts 2024-08-30T23:42:18.010 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:17 smithi173 ceph-mon[20554]: pgmap v1221: 47 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 4 active+undersized+degraded, 28 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 369 op/s; 25172/215040 objects degraded (11.706%); 12834/215040 objects misplaced (5.968%) 2024-08-30T23:42:18.015 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:17 smithi136 ceph-mon[20207]: 5.12 scrub starts 2024-08-30T23:42:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:17 smithi136 ceph-mon[20207]: pgmap v1221: 47 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 4 active+undersized+degraded, 28 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 369 op/s; 25172/215040 objects degraded (11.706%); 12834/215040 objects misplaced (5.968%) 2024-08-30T23:42:18.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:18.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:17 smithi106 ceph-mon[16603]: 5.12 scrub starts 2024-08-30T23:42:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:17 smithi106 ceph-mon[16603]: pgmap v1221: 47 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 4 active+undersized+degraded, 28 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 369 op/s; 25172/215040 objects degraded (11.706%); 12834/215040 objects misplaced (5.968%) 2024-08-30T23:42:18.141 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:18.141 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:18.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:18.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.209 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:18.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:18.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:18.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:18.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:18.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:18.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:18.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:18.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:18.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:18.867 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:18.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:18.926 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:42:18.926 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:42:18.926 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:42:18.927 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:42:18.927 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:42:18.927 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:42:18.927 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:18.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:42:18.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:18.999 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:42:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:19.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:42:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:42:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:19.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:19.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:19.263 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:19.312 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:19.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:19.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:19.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:19.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:19.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:19.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:19.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:moved 9 / 141 (6.38298%) 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:avg 12.8182 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 4.74646 -> 4.3275 (expected baseline 3.41363) 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.4 with 3 -> 3 pgs (0.234043 -> 0.234043 * mean) 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.7 with 19 -> 19 pgs (1.48227 -> 1.48227 * mean) 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:oload 110 2024-08-30T23:42:19.751 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0170 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0187 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1 weight 0.5043 -> 1.0000 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 0.9238 -> 0.5508 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5 weight 1.0000 -> 0.6698 2024-08-30T23:42:19.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3 weight 0.8072 -> 1.0000 2024-08-30T23:42:19.755 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:42:19.772 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:19.862 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:19.862 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:19.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:19.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:19.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:19.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:42:19.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:20.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3810330204' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:42:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:19 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:42:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:19 smithi136 ceph-mon[20207]: from='client.26776 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'36098','1':'65536','3':'65536','5':'43896'}"} : dispatch 2024-08-30T23:42:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:19 smithi136 ceph-mon[20207]: pgmap v1222: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 4 active+undersized+degraded, 28 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 401 op/s; 29865/213336 objects degraded (13.999%); 14856/213336 objects misplaced (6.964%); 8.0 MiB/s, 128 objects/s recovering 2024-08-30T23:42:20.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3810330204' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:42:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:19 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:42:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:19 smithi173 ceph-mon[20554]: from='client.26776 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'36098','1':'65536','3':'65536','5':'43896'}"} : dispatch 2024-08-30T23:42:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:19 smithi173 ceph-mon[20554]: pgmap v1222: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 4 active+undersized+degraded, 28 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 401 op/s; 29865/213336 objects degraded (13.999%); 14856/213336 objects misplaced (6.964%); 8.0 MiB/s, 128 objects/s recovering 2024-08-30T23:42:20.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3810330204' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:42:20.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:19 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:42:20.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:19 smithi106 ceph-mon[16603]: from='client.26776 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:20.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'36098','1':'65536','3':'65536','5':'43896'}"} : dispatch 2024-08-30T23:42:20.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:19 smithi106 ceph-mon[16603]: pgmap v1222: 47 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 6 active+clean+remapped, 4 active+undersized+degraded, 28 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 401 op/s; 29865/213336 objects degraded (13.999%); 14856/213336 objects misplaced (6.964%); 8.0 MiB/s, 128 objects/s recovering 2024-08-30T23:42:20.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:20.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:20.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:20.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:20.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:20.391 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:20.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:20.514 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:20.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:20.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:20.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:42:20.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:20.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:20.889 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:20.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:20.938 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:20.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:21.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:21.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:20 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-30T23:42:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:20 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:20 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:42:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'36098','1':'65536','3':'65536','5':'43896'}"}]': finished 2024-08-30T23:42:21.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:20 smithi136 ceph-mon[20207]: osdmap e529: 12 total, 11 up, 12 in 2024-08-30T23:42:21.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:20 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-30T23:42:21.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:20 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:21.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:20 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:42:21.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'36098','1':'65536','3':'65536','5':'43896'}"}]': finished 2024-08-30T23:42:21.061 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:20 smithi173 ceph-mon[20554]: osdmap e529: 12 total, 11 up, 12 in 2024-08-30T23:42:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:20 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-30T23:42:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:20 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:20 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:42:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'36098','1':'65536','3':'65536','5':'43896'}"}]': finished 2024-08-30T23:42:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:20 smithi106 ceph-mon[16603]: osdmap e529: 12 total, 11 up, 12 in 2024-08-30T23:42:21.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:21.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:21.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:21.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:21.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:21.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:21.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:21.473 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:21.473 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:21.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:21.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:21.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:21.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:21.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:21.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:21.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:21.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:21.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:21.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:22.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:21 smithi136 ceph-mon[20207]: osdmap e530: 12 total, 11 up, 12 in 2024-08-30T23:42:22.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 29865/213336 objects degraded (13.999%), 9 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:21 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:42:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:21 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:42:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:21 smithi136 ceph-mon[20207]: pgmap v1225: 47 pgs: 1 peering, 4 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 5 active+clean+remapped, 3 active+undersized+degraded, 26 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 20456/206583 objects degraded (9.902%); 11233/206583 objects misplaced (5.438%); 16 MiB/s, 263 objects/s recovering 2024-08-30T23:42:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:21 smithi136 ceph-mon[20207]: osdmap e531: 12 total, 11 up, 12 in 2024-08-30T23:42:22.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:21 smithi173 ceph-mon[20554]: osdmap e530: 12 total, 11 up, 12 in 2024-08-30T23:42:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 29865/213336 objects degraded (13.999%), 9 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:21 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:42:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:21 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:42:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:21 smithi173 ceph-mon[20554]: pgmap v1225: 47 pgs: 1 peering, 4 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 5 active+clean+remapped, 3 active+undersized+degraded, 26 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 20456/206583 objects degraded (9.902%); 11233/206583 objects misplaced (5.438%); 16 MiB/s, 263 objects/s recovering 2024-08-30T23:42:22.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:21 smithi173 ceph-mon[20554]: osdmap e531: 12 total, 11 up, 12 in 2024-08-30T23:42:22.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:21 smithi106 ceph-mon[16603]: osdmap e530: 12 total, 11 up, 12 in 2024-08-30T23:42:22.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 29865/213336 objects degraded (13.999%), 9 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:22.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:21 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:42:22.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:21 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:42:22.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:21 smithi106 ceph-mon[16603]: pgmap v1225: 47 pgs: 1 peering, 4 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized, 5 active+clean+remapped, 3 active+undersized+degraded, 26 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 789 op/s; 20456/206583 objects degraded (9.902%); 11233/206583 objects misplaced (5.438%); 16 MiB/s, 263 objects/s recovering 2024-08-30T23:42:22.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:21 smithi106 ceph-mon[16603]: osdmap e531: 12 total, 11 up, 12 in 2024-08-30T23:42:22.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:22.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:22.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:22.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:22.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:22.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:22.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.515 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:22.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:22.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:22.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:22.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:22.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.768 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:22.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:22.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:22.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:22.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:23.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:23.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:42:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:22 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:42:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:22 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:42:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:22 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:42:23.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:23.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:23.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:23.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:23.268 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:22 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:42:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:22 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:42:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:22 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:42:23.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:22 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:42:23.387 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:22 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:42:23.387 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:22 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:42:23.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:23.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:23.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:23.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:23.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:23.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:23.762 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:23.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:23.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:23.907 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:23.907 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:23.907 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:23 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:42:23.908 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:23 smithi173 ceph-mon[20554]: pgmap v1227: 47 pgs: 1 peering, 4 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 3 active+undersized+degraded, 27 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 22327/204384 objects degraded (10.924%); 12623/204384 objects misplaced (6.176%); 21 MiB/s, 335 objects/s recovering 2024-08-30T23:42:23.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:23.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:24.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:24.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:24.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:24.145 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:23 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:42:24.145 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:23 smithi106 ceph-mon[16603]: pgmap v1227: 47 pgs: 1 peering, 4 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 3 active+undersized+degraded, 27 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 22327/204384 objects degraded (10.924%); 12623/204384 objects misplaced (6.176%); 21 MiB/s, 335 objects/s recovering 2024-08-30T23:42:24.175 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:24.278 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:23 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:42:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:23 smithi136 ceph-mon[20207]: pgmap v1227: 47 pgs: 1 peering, 4 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 3 active+undersized+degraded, 27 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 590 op/s; 22327/204384 objects degraded (10.924%); 12623/204384 objects misplaced (6.176%); 21 MiB/s, 335 objects/s recovering 2024-08-30T23:42:24.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:24.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:24.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:24.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:24.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:42:24.570 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.571 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:24.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:24.694 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:24.729 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-08-30T23:42:24.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-08-30T23:42:24.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:24.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:24.881 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:24.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:24.966 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:24.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:25.068 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:25.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:25.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:25.227 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:25.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:25.344 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:25.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:25.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:25.481 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:25.482 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:25.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:25.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:25.619 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:25.646 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:25 smithi106 ceph-mon[16603]: pgmap v1228: 47 pgs: 4 remapped+peering, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 2 active+undersized+degraded, 27 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 27991/203034 objects degraded (13.786%); 12623/203034 objects misplaced (6.217%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:42:25.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:25.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:25.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:25.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.873 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:25.924 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:25.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:25.977 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:25.977 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:25.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:25.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:26.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:25 smithi136 ceph-mon[20207]: pgmap v1228: 47 pgs: 4 remapped+peering, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 2 active+undersized+degraded, 27 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 27991/203034 objects degraded (13.786%); 12623/203034 objects misplaced (6.217%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:42:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:25 smithi173 ceph-mon[20554]: pgmap v1228: 47 pgs: 4 remapped+peering, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 2 active+undersized+degraded, 27 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 27991/203034 objects degraded (13.786%); 12623/203034 objects misplaced (6.217%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:42:26.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:26.124 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:26.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:42:26.219 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:42:26.219 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:42:26.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:26.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:26.270 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:26.271 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:26.308 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:26.308 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:26.342 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:26.342 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:26.345 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:26.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:26.384 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:26.384 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:26.415 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:26.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:26.417 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:26.450 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:26.451 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:26.490 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:26.490 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:26.505 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:26.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:26.527 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:26.527 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:26.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:26.568 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:26.568 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:26.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:26.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:26.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:26.815 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:26.855 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 27991/203034 objects degraded (13.786%), 10 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:26.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:26 smithi106 ceph-mon[16603]: osdmap e532: 12 total, 11 up, 12 in 2024-08-30T23:42:26.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/975277456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout:moved 12 / 141 (8.51064%) 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout:avg 12.8182 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 4.3275 -> 2.29624 (expected baseline 3.41363) 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.4 with 3 -> 12 pgs (0.234043 -> 0.93617 * mean) 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.7 with 19 -> 16 pgs (1.48227 -> 1.24823 * mean) 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:42:26.856 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 1 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0164 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.1640 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 weight 0.5274 -> 0.9328 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 weight 0.5898 -> 0.7566 2024-08-30T23:42:26.857 INFO:teuthology.orchestra.run.smithi106.stderr:no change 2024-08-30T23:42:26.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:26.943 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 27991/203034 objects degraded (13.786%), 10 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:26.943 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:26 smithi173 ceph-mon[20554]: osdmap e532: 12 total, 11 up, 12 in 2024-08-30T23:42:26.943 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/975277456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:26.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:26.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:27.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 27991/203034 objects degraded (13.786%), 10 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:26 smithi136 ceph-mon[20207]: osdmap e532: 12 total, 11 up, 12 in 2024-08-30T23:42:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/975277456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:27.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:27.071 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:42:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:42:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:42:27.071 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:42:27.071 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:42:27.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:27.107 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 30 23:42:26 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-30T23:42:26.853+0000 7f2243404640 -1 mgr.server reply reply (2) No such file or directory no change 2024-08-30T23:42:27.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:27.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:27.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:27.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:27.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:42:27.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.478 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:27.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:27.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:27.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:27.613 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:27.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:27.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:27.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:27 smithi106 ceph-mon[16603]: from='client.26785 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:27 smithi106 ceph-mon[16603]: pgmap v1230: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 38439/199329 objects degraded (19.284%); 16758/199329 objects misplaced (8.407%); 28 MiB/s, 450 objects/s recovering 2024-08-30T23:42:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.858 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:27.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:27.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:27.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:27.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:27 smithi136 ceph-mon[20207]: from='client.26785 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:27 smithi136 ceph-mon[20207]: pgmap v1230: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 38439/199329 objects degraded (19.284%); 16758/199329 objects misplaced (8.407%); 28 MiB/s, 450 objects/s recovering 2024-08-30T23:42:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:27 smithi173 ceph-mon[20554]: from='client.26785 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:27 smithi173 ceph-mon[20554]: pgmap v1230: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 38439/199329 objects degraded (19.284%); 16758/199329 objects misplaced (8.407%); 28 MiB/s, 450 objects/s recovering 2024-08-30T23:42:28.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.106 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:28.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:28.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:28.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:28.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:28.375 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:28.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:28.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:28.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:28.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:28.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.768 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:28.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:28.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:28.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:28.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:28.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:29.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:29.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:29.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:29.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:29.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:29.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:29.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:29.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:29.536 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:29.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:42:29.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:29.682 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:29.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:29.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:29.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:29.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:29.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:30.014 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:29 smithi106 ceph-mon[16603]: pgmap v1231: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 38358/198408 objects degraded (19.333%); 15958/198408 objects misplaced (8.043%); 27 MiB/s, 428 objects/s recovering 2024-08-30T23:42:30.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:29 smithi136 ceph-mon[20207]: pgmap v1231: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 38358/198408 objects degraded (19.333%); 15958/198408 objects misplaced (8.043%); 27 MiB/s, 428 objects/s recovering 2024-08-30T23:42:30.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:30.071 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-30T23:42:30.071 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-08-30T23:42:30.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:29 smithi173 ceph-mon[20554]: pgmap v1231: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 4.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 38358/198408 objects degraded (19.333%); 15958/198408 objects misplaced (8.043%); 27 MiB/s, 428 objects/s recovering 2024-08-30T23:42:30.124 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:30.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:30.189 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:30.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:30.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:30.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:30.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:30.493 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:30.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:30.627 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:30.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:30.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:30.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:30.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:30.896 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:30.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:30.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:31.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:30 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 38358/198408 objects degraded (19.333%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:31.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:30 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 38358/198408 objects degraded (19.333%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:31.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:31.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:31.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:42:31.268 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:31.269 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:31.287 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:31.288 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:31.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:30 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 38358/198408 objects degraded (19.333%), 14 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:31.290 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:31.293 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:31.332 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:31.333 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:31.369 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:31.370 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:31.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:31.410 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:31.410 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:31.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:42:31.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:31.451 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:31.451 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:31.485 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:31.485 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:31.517 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:31.518 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:31.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:31.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:31.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:31.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:31.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:31.801 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:31.820 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:42:31.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:31.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:31.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:32.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:32.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4190224953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2855249269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:31 smithi173 ceph-mon[20554]: pgmap v1232: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 37806/192762 objects degraded (19.613%); 15050/192762 objects misplaced (7.808%); 27 MiB/s, 453 objects/s recovering 2024-08-30T23:42:32.086 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4190224953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2855249269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:31 smithi106 ceph-mon[16603]: pgmap v1232: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 37806/192762 objects degraded (19.613%); 15050/192762 objects misplaced (7.808%); 27 MiB/s, 453 objects/s recovering 2024-08-30T23:42:32.160 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:32.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:32.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4190224953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2855249269' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:32.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:31 smithi136 ceph-mon[20207]: pgmap v1232: 47 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 37806/192762 objects degraded (19.613%); 15050/192762 objects misplaced (7.808%); 27 MiB/s, 453 objects/s recovering 2024-08-30T23:42:32.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:32.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:32.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:32.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:32.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:32.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:32.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:32.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:32.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:32.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:32.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:32.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:33.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:32 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:42:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2855249269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:32 smithi173 ceph-mon[20554]: osdmap e533: 12 total, 11 up, 12 in 2024-08-30T23:42:33.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:33.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:32 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:42:33.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2855249269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:33.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:32 smithi106 ceph-mon[16603]: osdmap e533: 12 total, 11 up, 12 in 2024-08-30T23:42:33.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:33.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:33.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:32 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:42:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2855249269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:32 smithi136 ceph-mon[20207]: osdmap e533: 12 total, 11 up, 12 in 2024-08-30T23:42:33.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:33.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:33.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:33.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:33.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:33.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:33.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:33.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:33.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:33.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:33.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:33.850 INFO:teuthology.orchestra.run.smithi106.stdout:moved 12 / 141 (8.51064%) 2024-08-30T23:42:33.850 INFO:teuthology.orchestra.run.smithi106.stdout:avg 12.8182 2024-08-30T23:42:33.850 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 4.3275 -> 2.29624 (expected baseline 3.41363) 2024-08-30T23:42:33.850 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.4 with 3 -> 12 pgs (0.234043 -> 0.93617 * mean) 2024-08-30T23:42:33.850 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.7 with 19 -> 16 pgs (1.48227 -> 1.24823 * mean) 2024-08-30T23:42:33.851 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:42:33.851 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:42:33.851 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 1 2024-08-30T23:42:33.851 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:42:33.851 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0162 2024-08-30T23:42:33.851 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.1622 2024-08-30T23:42:33.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 weight 0.5274 -> 0.9317 2024-08-30T23:42:33.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 weight 0.5898 -> 0.7879 2024-08-30T23:42:33.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:33.960 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:33.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:34.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:34.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:33 smithi106 ceph-mon[16603]: osdmap e534: 12 total, 11 up, 12 in 2024-08-30T23:42:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:33 smithi106 ceph-mon[16603]: from='client.17334 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'4':'61060','11':'51638'}"} : dispatch 2024-08-30T23:42:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:33 smithi106 ceph-mon[16603]: pgmap v1235: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 37589/190638 objects degraded (19.717%); 13672/190638 objects misplaced (7.172%); 37 MiB/s, 619 objects/s recovering 2024-08-30T23:42:34.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:34.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:34.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:33 smithi136 ceph-mon[20207]: osdmap e534: 12 total, 11 up, 12 in 2024-08-30T23:42:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:33 smithi136 ceph-mon[20207]: from='client.17334 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'4':'61060','11':'51638'}"} : dispatch 2024-08-30T23:42:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:33 smithi136 ceph-mon[20207]: pgmap v1235: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 37589/190638 objects degraded (19.717%); 13672/190638 objects misplaced (7.172%); 37 MiB/s, 619 objects/s recovering 2024-08-30T23:42:34.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:33 smithi173 ceph-mon[20554]: osdmap e534: 12 total, 11 up, 12 in 2024-08-30T23:42:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:33 smithi173 ceph-mon[20554]: from='client.17334 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:34.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'4':'61060','11':'51638'}"} : dispatch 2024-08-30T23:42:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:33 smithi173 ceph-mon[20554]: pgmap v1235: 47 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 2 active+undersized+degraded, 28 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 37589/190638 objects degraded (19.717%); 13672/190638 objects misplaced (7.172%); 37 MiB/s, 619 objects/s recovering 2024-08-30T23:42:34.340 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:34.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:34.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:34.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.589 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:34.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:34.624 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-30T23:42:34.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-08-30T23:42:34.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:34.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:42:34.734 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:34.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:34.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:34.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:34.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:34.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:34.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:35.028 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:35.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.117 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:35.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:35.157 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:34 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'61060','11':'51638'}"}]': finished 2024-08-30T23:42:35.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:34 smithi106 ceph-mon[16603]: osdmap e535: 12 total, 11 up, 12 in 2024-08-30T23:42:35.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:34 smithi106 ceph-mon[16603]: 5.10 scrub starts 2024-08-30T23:42:35.158 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:34 smithi106 ceph-mon[16603]: osdmap e536: 12 total, 11 up, 12 in 2024-08-30T23:42:35.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:35.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:34 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'61060','11':'51638'}"}]': finished 2024-08-30T23:42:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:34 smithi136 ceph-mon[20207]: osdmap e535: 12 total, 11 up, 12 in 2024-08-30T23:42:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:34 smithi136 ceph-mon[20207]: 5.10 scrub starts 2024-08-30T23:42:35.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:34 smithi136 ceph-mon[20207]: osdmap e536: 12 total, 11 up, 12 in 2024-08-30T23:42:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:34 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'61060','11':'51638'}"}]': finished 2024-08-30T23:42:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:34 smithi173 ceph-mon[20554]: osdmap e535: 12 total, 11 up, 12 in 2024-08-30T23:42:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:34 smithi173 ceph-mon[20554]: 5.10 scrub starts 2024-08-30T23:42:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:34 smithi173 ceph-mon[20554]: osdmap e536: 12 total, 11 up, 12 in 2024-08-30T23:42:35.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:35.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:35.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:35.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:35.550 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:42:35.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:35.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:35.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:35.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:35.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:35.897 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:35.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:35.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:35.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:36.026 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:36.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:36.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:36.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:36.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: 5.2 scrub starts 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: 5.2 scrub ok 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: 5.10 scrub ok 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: pgmap v1238: 47 pgs: 2 unknown, 2 peering, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 25 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 0 op/s; 32722/172047 objects degraded (19.019%); 13672/172047 objects misplaced (7.947%); 24 MiB/s, 384 objects/s recovering 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 37589/190638 objects degraded (19.717%), 15 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:36.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:35 smithi136 ceph-mon[20207]: osdmap e537: 12 total, 11 up, 12 in 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:35 smithi173 ceph-mon[20554]: 5.2 scrub starts 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: 5.2 scrub ok 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: 5.10 scrub ok 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: pgmap v1238: 47 pgs: 2 unknown, 2 peering, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 25 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 0 op/s; 32722/172047 objects degraded (19.019%); 13672/172047 objects misplaced (7.947%); 24 MiB/s, 384 objects/s recovering 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 37589/190638 objects degraded (19.717%), 15 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: osdmap e537: 12 total, 11 up, 12 in 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: 5.2 scrub starts 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: 5.2 scrub ok 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: 5.10 scrub ok 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: pgmap v1238: 47 pgs: 2 unknown, 2 peering, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 25 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 0 op/s; 32722/172047 objects degraded (19.019%); 13672/172047 objects misplaced (7.947%); 24 MiB/s, 384 objects/s recovering 2024-08-30T23:42:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 37589/190638 objects degraded (19.717%), 15 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:42:36.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:35 smithi106 ceph-mon[16603]: osdmap e537: 12 total, 11 up, 12 in 2024-08-30T23:42:36.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:36.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:36.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:36.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:36.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:36.697 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:36.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:36.823 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:36.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:36.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:36.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:37.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:37.081 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:37.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:37.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:37.146 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:36 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:42:37.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:37.209 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:37.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:36 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:42:37.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:37.340 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:36 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:42:37.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:37.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:37.561 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:37.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:37.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:37.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:37.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:37.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:37.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:37.901 INFO:teuthology.orchestra.run.smithi106.stdout:moved 8 / 141 (5.67376%) 2024-08-30T23:42:37.901 INFO:teuthology.orchestra.run.smithi106.stdout:avg 12.8182 2024-08-30T23:42:37.901 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.29624 -> 2.75441 (expected baseline 3.41363) 2024-08-30T23:42:37.901 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.2 with 9 -> 10 pgs (0.702128 -> 0.780142 * mean) 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.7 with 16 -> 12 pgs (1.24823 -> 0.93617 * mean) 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:oload 110 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0161 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.0177 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0 weight 0.5508 -> 0.3393 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5 weight 0.6698 -> 0.4267 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4 weight 0.9317 -> 1.0000 2024-08-30T23:42:37.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7 weight 1.0000 -> 0.7503 2024-08-30T23:42:37.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:37.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:37.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:38.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:38.110 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:38.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: osdmap e538: 12 total, 11 up, 12 in 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: from='client.17340 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'22237','4':'65536','5':'27966','7':'49173'}"} : dispatch 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: pgmap v1241: 47 pgs: 2 unknown, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 25 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 0 op/s; 34728/169605 objects degraded (20.476%); 13827/169605 objects misplaced (8.152%) 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:38.223 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:37 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.236 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:38.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: osdmap e538: 12 total, 11 up, 12 in 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: from='client.17340 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'22237','4':'65536','5':'27966','7':'49173'}"} : dispatch 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: pgmap v1241: 47 pgs: 2 unknown, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 25 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 0 op/s; 34728/169605 objects degraded (20.476%); 13827/169605 objects misplaced (8.152%) 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:37 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: osdmap e538: 12 total, 11 up, 12 in 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: from='client.17340 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'22237','4':'65536','5':'27966','7':'49173'}"} : dispatch 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: pgmap v1241: 47 pgs: 2 unknown, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 25 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 0 op/s; 34728/169605 objects degraded (20.476%); 13827/169605 objects misplaced (8.152%) 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:37 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:38.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:38.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:38.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:38.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:38.486 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:38.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:38.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:38.659 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:42:38.660 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:42:38.681 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-08-30T23:42:38.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-08-30T23:42:38.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:38.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:38.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:38.880 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:38.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:38.916 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:38 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'22237','4':'65536','5':'27966','7':'49173'}"}]': finished 2024-08-30T23:42:38.916 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:38 smithi106 ceph-mon[16603]: osdmap e539: 12 total, 11 up, 12 in 2024-08-30T23:42:38.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.014 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:39.059 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:39.099 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:39.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:39.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:39.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:39.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:39.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:38 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'22237','4':'65536','5':'27966','7':'49173'}"}]': finished 2024-08-30T23:42:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:38 smithi136 ceph-mon[20207]: osdmap e539: 12 total, 11 up, 12 in 2024-08-30T23:42:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:38 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'22237','4':'65536','5':'27966','7':'49173'}"}]': finished 2024-08-30T23:42:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:38 smithi173 ceph-mon[20554]: osdmap e539: 12 total, 11 up, 12 in 2024-08-30T23:42:39.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:39.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:39.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:39.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.525 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:39.626 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:39.650 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:39.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:39.776 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:39.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:39.903 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:39.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:39.905 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:42:40.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:40.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:40.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:40.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:40.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:40.286 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: osdmap e540: 12 total, 11 up, 12 in 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/799159578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: pgmap v1244: 47 pgs: 1 remapped, 2 remapped+peering, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 24 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 1.12k op/s; 36608/186183 objects degraded (19.662%); 16494/186183 objects misplaced (8.859%); 11 MiB/s, 163 objects/s recovering 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:42:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:39 smithi136 ceph-mon[20207]: osdmap e541: 12 total, 11 up, 12 in 2024-08-30T23:42:40.320 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:40.320 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: osdmap e540: 12 total, 11 up, 12 in 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/799159578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: pgmap v1244: 47 pgs: 1 remapped, 2 remapped+peering, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 24 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 1.12k op/s; 36608/186183 objects degraded (19.662%); 16494/186183 objects misplaced (8.859%); 11 MiB/s, 163 objects/s recovering 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:42:40.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:39 smithi106 ceph-mon[16603]: osdmap e541: 12 total, 11 up, 12 in 2024-08-30T23:42:40.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: osdmap e540: 12 total, 11 up, 12 in 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/799159578' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: pgmap v1244: 47 pgs: 1 remapped, 2 remapped+peering, 2 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 24 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 1.12k op/s; 36608/186183 objects degraded (19.662%); 16494/186183 objects misplaced (8.859%); 11 MiB/s, 163 objects/s recovering 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:42:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:39 smithi173 ceph-mon[20554]: osdmap e541: 12 total, 11 up, 12 in 2024-08-30T23:42:40.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:40.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:40.442 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:40.442 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:40.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:42:40.469 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:42:40.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:40.500 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:40.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:40.526 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:40.527 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:40.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:40.549 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:40.550 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:40.550 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:40.575 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:40.575 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:40.597 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:40.597 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:40.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:40.662 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:40.709 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:40.709 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:40.738 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:40.738 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:40.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:40.767 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:40.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:40.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:40.796 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:40.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:40.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:40.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:40.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:41.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:41.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:41.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:41.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.168 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:41.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/334695785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:40 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 36608/186183 objects degraded (19.662%), 15 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:40 smithi136 ceph-mon[20207]: osdmap e542: 12 total, 11 up, 12 in 2024-08-30T23:42:41.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/334695785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:41.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:40 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 36608/186183 objects degraded (19.662%), 15 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:41.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:40 smithi173 ceph-mon[20554]: osdmap e542: 12 total, 11 up, 12 in 2024-08-30T23:42:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/334695785' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:40 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 36608/186183 objects degraded (19.662%), 15 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:40 smithi106 ceph-mon[16603]: osdmap e542: 12 total, 11 up, 12 in 2024-08-30T23:42:41.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:41.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:41.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:41.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:41.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:41.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:41.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:41.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:41.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:41.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:41.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:42.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:42.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:41 smithi106 ceph-mon[16603]: pgmap v1247: 47 pgs: 1 peering, 1 remapped, 1 remapped+peering, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.84k op/s; 36052/184398 objects degraded (19.551%); 13324/184398 objects misplaced (7.226%); 42 MiB/s, 677 objects/s recovering 2024-08-30T23:42:42.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:42.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:42.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:42.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:41 smithi136 ceph-mon[20207]: pgmap v1247: 47 pgs: 1 peering, 1 remapped, 1 remapped+peering, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.84k op/s; 36052/184398 objects degraded (19.551%); 13324/184398 objects misplaced (7.226%); 42 MiB/s, 677 objects/s recovering 2024-08-30T23:42:42.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:41 smithi173 ceph-mon[20554]: pgmap v1247: 47 pgs: 1 peering, 1 remapped, 1 remapped+peering, 5 active+recovery_wait+undersized+degraded+remapped, 9 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 26 active+clean; 3.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.84k op/s; 36052/184398 objects degraded (19.551%); 13324/184398 objects misplaced (7.226%); 42 MiB/s, 677 objects/s recovering 2024-08-30T23:42:42.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:42.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:42.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:42.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:42.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:42.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:42.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:42.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:42.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:42.928 INFO:teuthology.orchestra.run.smithi106.stdout:moved 2 / 141 (1.41844%) 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:avg 12.8182 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:stddev 2.75441 -> 2.32486 (expected baseline 3.41363) 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:min osd.0 with 8 -> 8 pgs (0.624113 -> 0.624113 * mean) 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:max osd.8 with 18 -> 16 pgs (1.40426 -> 1.24823 * mean) 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:oload 1000 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:max_change 3 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:max_change_osds 4 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:average_utilization 0.0160 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:overload_utilization 0.1600 2024-08-30T23:42:42.929 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11 weight 0.7879 -> 1.0000 2024-08-30T23:42:42.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:42.944 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:43.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:43.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:42 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:42:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'11':'65536'}"} : dispatch 2024-08-30T23:42:43.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:43.207 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:43.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:43.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:42 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:42:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:42 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'11':'65536'}"} : dispatch 2024-08-30T23:42:43.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:42 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:42:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'11':'65536'}"} : dispatch 2024-08-30T23:42:43.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:43.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:43.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:43.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:43.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:43.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:43.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:43.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:43.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:43.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:43.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:44.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.028 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:44.129 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:44.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:44.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:44.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:43 smithi136 ceph-mon[20207]: from='client.26735 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'11':'65536'}"}]': finished 2024-08-30T23:42:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:43 smithi136 ceph-mon[20207]: osdmap e543: 12 total, 11 up, 12 in 2024-08-30T23:42:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:43 smithi136 ceph-mon[20207]: pgmap v1249: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 activating+degraded, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 41079/183390 objects degraded (22.400%); 13966/183390 objects misplaced (7.615%); 46 MiB/s, 730 objects/s recovering 2024-08-30T23:42:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:43 smithi173 ceph-mon[20554]: from='client.26735 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'11':'65536'}"}]': finished 2024-08-30T23:42:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:43 smithi173 ceph-mon[20554]: osdmap e543: 12 total, 11 up, 12 in 2024-08-30T23:42:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:43 smithi173 ceph-mon[20554]: pgmap v1249: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 activating+degraded, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 41079/183390 objects degraded (22.400%); 13966/183390 objects misplaced (7.615%); 46 MiB/s, 730 objects/s recovering 2024-08-30T23:42:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: from='client.26735 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-30T23:42:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'11':'65536'}"}]': finished 2024-08-30T23:42:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: osdmap e543: 12 total, 11 up, 12 in 2024-08-30T23:42:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:44.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:43 smithi106 ceph-mon[16603]: pgmap v1249: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 activating+degraded, 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 198 op/s; 41079/183390 objects degraded (22.400%); 13966/183390 objects misplaced (7.615%); 46 MiB/s, 730 objects/s recovering 2024-08-30T23:42:44.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:44.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:44.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:44.540 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:44.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:44.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:44.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:44.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:44.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:44.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:44.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:45.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:45.079 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:45.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:45.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:45.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:44 smithi136 ceph-mon[20207]: osdmap e544: 12 total, 11 up, 12 in 2024-08-30T23:42:45.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:45.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:44 smithi173 ceph-mon[20554]: osdmap e544: 12 total, 11 up, 12 in 2024-08-30T23:42:45.328 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:44 smithi106 ceph-mon[16603]: osdmap e544: 12 total, 11 up, 12 in 2024-08-30T23:42:45.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:45.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:45.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:45.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:45.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:45.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:45.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:45.845 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:45.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:45.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:45.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:46.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:45 smithi106 ceph-mon[16603]: pgmap v1251: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 activating+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 43382/182355 objects degraded (23.790%); 13966/182355 objects misplaced (7.659%); 40 MiB/s, 638 objects/s recovering 2024-08-30T23:42:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:45 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 41079/183390 objects degraded (22.400%), 17 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:46.115 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:46.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:46.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:46.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:45 smithi136 ceph-mon[20207]: pgmap v1251: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 activating+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 43382/182355 objects degraded (23.790%); 13966/182355 objects misplaced (7.659%); 40 MiB/s, 638 objects/s recovering 2024-08-30T23:42:46.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:45 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 41079/183390 objects degraded (22.400%), 17 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:45 smithi173 ceph-mon[20554]: pgmap v1251: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 activating+degraded, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 43382/182355 objects degraded (23.790%); 13966/182355 objects misplaced (7.659%); 40 MiB/s, 638 objects/s recovering 2024-08-30T23:42:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:45 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 41079/183390 objects degraded (22.400%), 17 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:46.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:46.380 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:46.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:46.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:46.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:46.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:46.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:46.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:46.856 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:46.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:46.918 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:46.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:47.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:47.052 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:47.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:47.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:47.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:47.315 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:47.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:47.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:47.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:47.580 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:47.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:47.709 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:47.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:47.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:47 smithi106 ceph-mon[16603]: pgmap v1252: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 367 op/s; 42475/177930 objects degraded (23.872%); 11659/177930 objects misplaced (6.553%); 56 MiB/s, 905 objects/s recovering 2024-08-30T23:42:47.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:47.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:47.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:47 smithi136 ceph-mon[20207]: pgmap v1252: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 367 op/s; 42475/177930 objects degraded (23.872%); 11659/177930 objects misplaced (6.553%); 56 MiB/s, 905 objects/s recovering 2024-08-30T23:42:48.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:47 smithi173 ceph-mon[20554]: pgmap v1252: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 367 op/s; 42475/177930 objects degraded (23.872%); 11659/177930 objects misplaced (6.553%); 56 MiB/s, 905 objects/s recovering 2024-08-30T23:42:48.094 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:48.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:48.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:48.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:48.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:42:48.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:48.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:48.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:48.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:48.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:48.742 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:48.788 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:48.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:48.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:42:48.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:48.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:48.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:42:48.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:49.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:49.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:49.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:49.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:49.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:49.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:49.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:49.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:49.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:49.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:49.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:49.736 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:49.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:49.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:49 smithi106 ceph-mon[16603]: osdmap e545: 12 total, 11 up, 12 in 2024-08-30T23:42:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:49 smithi106 ceph-mon[16603]: pgmap v1254: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 42288/176670 objects degraded (23.936%); 11121/176670 objects misplaced (6.295%); 40 MiB/s, 676 objects/s recovering 2024-08-30T23:42:49.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:49.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:49.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:50.023 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:49 smithi136 ceph-mon[20207]: osdmap e545: 12 total, 11 up, 12 in 2024-08-30T23:42:50.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:49 smithi136 ceph-mon[20207]: pgmap v1254: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 42288/176670 objects degraded (23.936%); 11121/176670 objects misplaced (6.295%); 40 MiB/s, 676 objects/s recovering 2024-08-30T23:42:50.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:49 smithi173 ceph-mon[20554]: osdmap e545: 12 total, 11 up, 12 in 2024-08-30T23:42:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:49 smithi173 ceph-mon[20554]: pgmap v1254: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 10 active+recovering+undersized+degraded+remapped, 2 active+clean+remapped, 1 active+undersized+degraded, 27 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 351 op/s; 42288/176670 objects degraded (23.936%); 11121/176670 objects misplaced (6.295%); 40 MiB/s, 676 objects/s recovering 2024-08-30T23:42:50.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:50.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:50.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:50.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:50.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:50.423 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:50.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:50.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:50.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:50.658 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:42:50.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:50.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:50.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:50.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:50.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:50.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:50 smithi136 ceph-mon[20207]: osdmap e546: 12 total, 11 up, 12 in 2024-08-30T23:42:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2251697899' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3669433261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:51.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:51.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:50 smithi173 ceph-mon[20554]: osdmap e546: 12 total, 11 up, 12 in 2024-08-30T23:42:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2251697899' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3669433261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:51.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:51.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:50 smithi106 ceph-mon[16603]: osdmap e546: 12 total, 11 up, 12 in 2024-08-30T23:42:51.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2251697899' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:42:51.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3669433261' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:51.108 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:51.108 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:51.138 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:42:51.138 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:42:51.166 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:51.167 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:51.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:51.193 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:51.194 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:51.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:51.226 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:51.227 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:51.227 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:51.256 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:51.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:51.287 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:51.287 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:51.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:51.320 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:51.320 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:51.338 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:51.351 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:51.351 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:51.380 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:51.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:51.410 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:51.410 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:51.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:51.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:51.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:51.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:51.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:51.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:51.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:51.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:51.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2251697899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: osdmap e547: 12 total, 11 up, 12 in 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 42288/176670 objects degraded (23.936%), 18 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: osdmap e548: 12 total, 11 up, 12 in 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: 5.12 scrub starts 2024-08-30T23:42:51.919 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:51 smithi106 ceph-mon[16603]: pgmap v1258: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 28 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 40003/173274 objects degraded (23.087%); 12273/173274 objects misplaced (7.083%); 22 MiB/s, 580 objects/s recovering 2024-08-30T23:42:51.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:51.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:51.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2251697899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: osdmap e547: 12 total, 11 up, 12 in 2024-08-30T23:42:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 42288/176670 objects degraded (23.936%), 18 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: osdmap e548: 12 total, 11 up, 12 in 2024-08-30T23:42:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: 5.12 scrub starts 2024-08-30T23:42:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:51 smithi136 ceph-mon[20207]: pgmap v1258: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 28 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 40003/173274 objects degraded (23.087%); 12273/173274 objects misplaced (7.083%); 22 MiB/s, 580 objects/s recovering 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2251697899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: osdmap e547: 12 total, 11 up, 12 in 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 42288/176670 objects degraded (23.936%), 18 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: osdmap e548: 12 total, 11 up, 12 in 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: 5.12 scrub starts 2024-08-30T23:42:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:51 smithi173 ceph-mon[20554]: pgmap v1258: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+undersized+degraded, 28 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 40003/173274 objects degraded (23.087%); 12273/173274 objects misplaced (7.083%); 22 MiB/s, 580 objects/s recovering 2024-08-30T23:42:52.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:52.122 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:52.224 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:52.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:42:52.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:42:52.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:52.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:52.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:52.508 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:52.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:42:52.568 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:42:52.569 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-08-30T23:42:52.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:42:52.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:52.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:52.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:52.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:52.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:52.886 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:52.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:52.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:52.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:53.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:53.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:53.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:53.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:53.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:53.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:53.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:53.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:53.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:53.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:42:53.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:53.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:53.696 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:42:53.697 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:42:53.780 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:53.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:53.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:53.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:53.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:54.038 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:54.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:53 smithi136 ceph-mon[20207]: pgmap v1259: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 36763/169896 objects degraded (21.639%); 12803/169896 objects misplaced (7.536%); 32 MiB/s, 701 objects/s recovering 2024-08-30T23:42:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.062 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:54.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:53 smithi173 ceph-mon[20554]: pgmap v1259: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 36763/169896 objects degraded (21.639%); 12803/169896 objects misplaced (7.536%); 32 MiB/s, 701 objects/s recovering 2024-08-30T23:42:54.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:53 smithi106 ceph-mon[16603]: pgmap v1259: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 36763/169896 objects degraded (21.639%); 12803/169896 objects misplaced (7.536%); 32 MiB/s, 701 objects/s recovering 2024-08-30T23:42:54.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:54.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:54.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:42:54.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:54.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:54.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:54.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:54.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:54.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:54.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:54.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:54.807 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:42:54.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:54.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:54.942 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:54 smithi173 ceph-mon[20554]: from='client.17373 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:42:54.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:54.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:54.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:54 smithi136 ceph-mon[20207]: from='client.17373 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:42:55.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:55.104 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:55.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:54 smithi106 ceph-mon[16603]: from='client.17373 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:42:55.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:55.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:55.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:55.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:55.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:55.500 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:55.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:55.631 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:55.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:55.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:55.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:55.885 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:55.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:55.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:56.017 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:55 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:42:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:55 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:42:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:55 smithi136 ceph-mon[20207]: osdmap e549: 12 total, 11 up, 12 in 2024-08-30T23:42:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:55 smithi136 ceph-mon[20207]: pgmap v1261: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 36412/168117 objects degraded (21.659%); 12803/168117 objects misplaced (7.616%); 30 MiB/s, 642 objects/s recovering 2024-08-30T23:42:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:55 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:42:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:55 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:42:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:55 smithi173 ceph-mon[20554]: osdmap e549: 12 total, 11 up, 12 in 2024-08-30T23:42:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:55 smithi173 ceph-mon[20554]: pgmap v1261: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 36412/168117 objects degraded (21.659%); 12803/168117 objects misplaced (7.616%); 30 MiB/s, 642 objects/s recovering 2024-08-30T23:42:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:55 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:42:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:55 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:42:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:55 smithi106 ceph-mon[16603]: osdmap e549: 12 total, 11 up, 12 in 2024-08-30T23:42:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:55 smithi106 ceph-mon[16603]: pgmap v1261: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 36412/168117 objects degraded (21.659%); 12803/168117 objects misplaced (7.616%); 30 MiB/s, 642 objects/s recovering 2024-08-30T23:42:56.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:42:56.144 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:42:56.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:56.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:56.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:56.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:56.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:56.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:56.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:56.668 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:56.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:56.793 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:56.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:56.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:56.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:42:56.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 36412/168117 objects degraded (21.659%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:56.963 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:56 smithi173 ceph-mon[20554]: osdmap e550: 12 total, 11 up, 12 in 2024-08-30T23:42:57.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 36412/168117 objects degraded (21.659%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:56 smithi136 ceph-mon[20207]: osdmap e550: 12 total, 11 up, 12 in 2024-08-30T23:42:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:57.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 36412/168117 objects degraded (21.659%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:42:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:56 smithi106 ceph-mon[16603]: osdmap e550: 12 total, 11 up, 12 in 2024-08-30T23:42:57.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.185 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:57.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:57.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:42:57.316 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:42:57.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:57.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:57.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:57.570 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:57.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:57.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:57.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:57.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:57.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:57.947 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:57 smithi106 ceph-mon[16603]: 5.12 scrub starts 2024-08-30T23:42:57.948 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:57 smithi106 ceph-mon[16603]: pgmap v1263: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 35493/162207 objects degraded (21.881%); 11382/162207 objects misplaced (7.017%); 39 MiB/s, 762 objects/s recovering 2024-08-30T23:42:57.949 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1259,"stamp":"2024-08-30T23:42:53.582515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3711828536,"num_objects":56632,"num_object_clones":0,"num_object_copies":169896,"num_objects_missing_on_primary":6356,"num_objects_missing":6356,"num_objects_degraded":36763,"num_objects_misplaced":12803,"num_objects_unfound":0,"num_objects_dirty":56632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":63092,"num_write_kb":2736801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30087,"num_bytes_recovered":1852768256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159761,"ondisk_log_size":159761,"up":135,"acting":129,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17388352,"kb_used_data":13336996,"kb_used_omap":104,"kb_used_meta":4051031,"kb_avail":1107553472,"statfs":{"total":1151940427776,"available":1134134755328,"internally_reserved":0,"allocated":13657083904,"data_stored":13646282082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107501,"internal_metadata":4148255763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,7,4,4,3,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-147980288,"num_objects":-2258,"num_object_clones":0,"num_object_copies":-6774,"num_objects_missing_on_primary":-307,"num_objects_missing":-307,"num_objects_degraded":-5525,"num_objects_misplaced":1682,"num_objects_unfound":0,"num_objects_dirty":-2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3458,"num_bytes_recovered":164691968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.929897"},"pg_stats":[{"pgid":"5.19","version":"548'5910","reported_seq":13754,"reported_epoch":548,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:52.536789+0000","last_change":"2024-08-30T23:42:37.916773+0000","last_active":"2024-08-30T23:42:52.536789+0000","last_peered":"2024-08-30T23:42:52.536789+0000","last_clean":"2024-08-30T23:42:19.710372+0000","last_became_active":"2024-08-30T23:42:21.900988+0000","last_became_peered":"2024-08-30T23:42:21.900988+0000","last_unstale":"2024-08-30T23:42:52.536789+0000","last_undegraded":"2024-08-30T23:42:21.768321+0000","last_fullsized":"2024-08-30T23:42:21.755939+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":505,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5362","last_scrub_stamp":"2024-08-30T23:41:51.170609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.170609+0000","objects_scrubbed":2207,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108724224,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1659,"num_objects_misplaced":701,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1300,"num_bytes_recovered":85065728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,6],"avail_no_missing":["7","6","8"],"object_location_counts":[{"shards":"6,7","objects":958},{"shards":"6,7,8","objects":527}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"548'5933","reported_seq":18696,"reported_epoch":548,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-08-30T23:42:52.041073+0000","last_change":"2024-08-30T23:42:42.023768+0000","last_active":"2024-08-30T23:42:52.041073+0000","last_peered":"2024-08-30T23:42:52.041073+0000","last_clean":"2024-08-30T23:42:10.750637+0000","last_became_active":"2024-08-30T23:42:42.014778+0000","last_became_peered":"2024-08-30T23:42:42.014778+0000","last_unstale":"2024-08-30T23:42:52.041073+0000","last_undegraded":"2024-08-30T23:42:39.976294+0000","last_fullsized":"2024-08-30T23:42:52.041073+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5406","last_scrub_stamp":"2024-08-30T23:40:53.176979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:53.176979+0000","objects_scrubbed":0,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116064256,"num_objects":1771,"num_object_clones":0,"num_object_copies":5313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2450,"num_objects_misplaced":2348,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":160032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4364,"num_bytes_recovered":278560768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,1],"acting":[6,11,1],"avail_no_missing":["6","0","7","10"],"object_location_counts":[{"shards":"0,1,6,7","objects":1750},{"shards":"1,6,11","objects":21}],"blocked_by":[],"up_primary":4,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":105,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.769976+0000","last_change":"2024-08-30T23:42:13.921090+0000","last_active":"2024-08-30T23:42:50.769976+0000","last_peered":"2024-08-30T23:42:50.769976+0000","last_clean":"2024-08-30T23:42:50.769976+0000","last_became_active":"2024-08-30T23:42:13.920647+0000","last_became_peered":"2024-08-30T23:42:13.920647+0000","last_unstale":"2024-08-30T23:42:50.769976+0000","last_undegraded":"2024-08-30T23:42:50.769976+0000","last_fullsized":"2024-08-30T23:42:50.769976+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:54.135149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:54.135149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:23.372887+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.17","version":"548'6080","reported_seq":6492,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:53.362599+0000","last_change":"2024-08-30T23:41:59.176807+0000","last_active":"2024-08-30T23:42:53.362599+0000","last_peered":"2024-08-30T23:42:53.362599+0000","last_clean":"2024-08-30T23:42:53.362599+0000","last_became_active":"2024-08-30T23:41:58.009965+0000","last_became_peered":"2024-08-30T23:41:58.009965+0000","last_unstale":"2024-08-30T23:42:53.362599+0000","last_undegraded":"2024-08-30T23:42:53.362599+0000","last_fullsized":"2024-08-30T23:42:53.362599+0000","mapping_epoch":516,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5526","last_scrub_stamp":"2024-08-30T23:41:59.176735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:59.176735+0000","objects_scrubbed":2327,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:27.419406+0000","scrub_duration":233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116195328,"num_objects":1773,"num_object_clones":0,"num_object_copies":5319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":554,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,11],"acting":[3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":67,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.772120+0000","last_change":"2024-08-30T23:42:43.952146+0000","last_active":"2024-08-30T23:42:50.772120+0000","last_peered":"2024-08-30T23:42:50.772120+0000","last_clean":"2024-08-30T23:42:50.772120+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:42:50.772120+0000","last_undegraded":"2024-08-30T23:42:50.772120+0000","last_fullsized":"2024-08-30T23:42:50.772120+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:21.921671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:21.921671+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:44.804661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"548'5898","reported_seq":10572,"reported_epoch":548,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:53.355201+0000","last_change":"2024-08-30T23:42:40.095373+0000","last_active":"2024-08-30T23:42:53.355201+0000","last_peered":"2024-08-30T23:42:53.355201+0000","last_clean":"2024-08-30T23:42:11.015645+0000","last_became_active":"2024-08-30T23:42:40.086037+0000","last_became_peered":"2024-08-30T23:42:40.086037+0000","last_unstale":"2024-08-30T23:42:53.355201+0000","last_undegraded":"2024-08-30T23:42:39.956680+0000","last_fullsized":"2024-08-30T23:42:39.907205+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5359","last_scrub_stamp":"2024-08-30T23:41:52.457261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:52.457261+0000","objects_scrubbed":2273,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113639424,"num_objects":1734,"num_object_clones":0,"num_object_copies":5202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":539,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":136118272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,10],"acting":[3,1],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"1,2,3,7","objects":1538},{"shards":"1,3","objects":-156},{"shards":"2,3,7","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":109,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.769681+0000","last_change":"2024-08-30T23:42:20.761884+0000","last_active":"2024-08-30T23:42:50.769681+0000","last_peered":"2024-08-30T23:42:50.769681+0000","last_clean":"2024-08-30T23:42:50.769681+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:42:50.769681+0000","last_undegraded":"2024-08-30T23:42:50.769681+0000","last_fullsized":"2024-08-30T23:42:50.769681+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:52.827190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:52.827190+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:59.644426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.15","version":"547'6135","reported_seq":7916,"reported_epoch":548,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:50.769919+0000","last_change":"2024-08-30T23:42:38.952061+0000","last_active":"2024-08-30T23:42:50.769919+0000","last_peered":"2024-08-30T23:42:50.769919+0000","last_clean":"2024-08-30T23:42:10.661546+0000","last_became_active":"2024-08-30T23:42:38.942580+0000","last_became_peered":"2024-08-30T23:42:38.942580+0000","last_unstale":"2024-08-30T23:42:50.769919+0000","last_undegraded":"2024-08-30T23:42:38.906709+0000","last_fullsized":"2024-08-30T23:42:38.903131+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5563","last_scrub_stamp":"2024-08-30T23:41:54.102519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:54.102519+0000","objects_scrubbed":2426,"log_size":6135,"log_dups_size":0,"ondisk_log_size":6135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121503744,"num_objects":1854,"num_object_clones":0,"num_object_copies":5562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":572,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":669,"num_bytes_recovered":43778048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,0],"avail_no_missing":["8","0","6"],"object_location_counts":[{"shards":"0,6,8","objects":1548},{"shards":"0,8","objects":306}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":1142,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.767048+0000","last_change":"2024-08-30T23:42:40.017890+0000","last_active":"2024-08-30T23:42:50.767048+0000","last_peered":"2024-08-30T23:42:50.767048+0000","last_clean":"2024-08-30T23:42:50.767048+0000","last_became_active":"2024-08-30T23:42:40.017555+0000","last_became_peered":"2024-08-30T23:42:40.017555+0000","last_unstale":"2024-08-30T23:42:50.767048+0000","last_undegraded":"2024-08-30T23:42:50.767048+0000","last_fullsized":"2024-08-30T23:42:50.767048+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:58.049995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:58.049995+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:21.601048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"548'5928","reported_seq":21955,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:53.432745+0000","last_change":"2024-08-30T23:42:50.678052+0000","last_active":"2024-08-30T23:42:53.432745+0000","last_peered":"2024-08-30T23:42:53.432745+0000","last_clean":"2024-08-30T23:42:53.432745+0000","last_became_active":"2024-08-30T23:42:50.667392+0000","last_became_peered":"2024-08-30T23:42:50.667392+0000","last_unstale":"2024-08-30T23:42:53.432745+0000","last_undegraded":"2024-08-30T23:42:53.432745+0000","last_fullsized":"2024-08-30T23:42:53.432745+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5371","last_scrub_stamp":"2024-08-30T23:41:13.404571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:13.404571+0000","objects_scrubbed":2260,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":749,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111607808,"num_objects":1703,"num_object_clones":0,"num_object_copies":5109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2459,"num_bytes_recovered":135069696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"547'5859","reported_seq":17948,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.770235+0000","last_change":"2024-08-30T23:42:49.679708+0000","last_active":"2024-08-30T23:42:50.770235+0000","last_peered":"2024-08-30T23:42:50.770235+0000","last_clean":"2024-08-30T23:42:50.770235+0000","last_became_active":"2024-08-30T23:42:49.670661+0000","last_became_peered":"2024-08-30T23:42:49.670661+0000","last_unstale":"2024-08-30T23:42:50.770235+0000","last_undegraded":"2024-08-30T23:42:50.770235+0000","last_fullsized":"2024-08-30T23:42:50.770235+0000","mapping_epoch":545,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5368","last_scrub_stamp":"2024-08-30T23:41:51.083921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.083921+0000","objects_scrubbed":2203,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:01.078549+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2420,"num_bytes_recovered":132972544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,3],"acting":[8,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.12","version":"548'5865","reported_seq":18278,"reported_epoch":548,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:42:50.791723+0000","last_change":"2024-08-30T23:42:50.673453+0000","last_active":"2024-08-30T23:42:50.791723+0000","last_peered":"2024-08-30T23:42:50.791723+0000","last_clean":"2024-08-30T23:42:50.791723+0000","last_became_active":"2024-08-30T23:42:50.664451+0000","last_became_peered":"2024-08-30T23:42:50.664451+0000","last_unstale":"2024-08-30T23:42:50.791723+0000","last_undegraded":"2024-08-30T23:42:50.791723+0000","last_fullsized":"2024-08-30T23:42:50.791723+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":5,"last_scrub":"449'5320","last_scrub_stamp":"2024-08-30T23:39:48.635484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:39:48.635484+0000","objects_scrubbed":0,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1721,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":157632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6720,"num_bytes_recovered":400130048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3],"acting":[8,3,7],"avail_no_missing":["8","3","7"],"object_location_counts":[{"shards":"3,7,8","objects":1721}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":170,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.788822+0000","last_change":"2024-08-30T23:42:39.063209+0000","last_active":"2024-08-30T23:42:50.788822+0000","last_peered":"2024-08-30T23:42:50.788822+0000","last_clean":"2024-08-30T23:42:50.788822+0000","last_became_active":"2024-08-30T23:42:38.914560+0000","last_became_peered":"2024-08-30T23:42:38.914560+0000","last_unstale":"2024-08-30T23:42:50.788822+0000","last_undegraded":"2024-08-30T23:42:50.788822+0000","last_fullsized":"2024-08-30T23:42:50.788822+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:39.063148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:39.063148+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:42.778726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.4","version":"548'5931","reported_seq":17033,"reported_epoch":548,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:42:51.415620+0000","last_change":"2024-08-30T23:41:11.017070+0000","last_active":"2024-08-30T23:42:51.415620+0000","last_peered":"2024-08-30T23:42:51.415620+0000","last_clean":"2024-08-30T23:42:51.415620+0000","last_became_active":"2024-08-30T23:41:08.809599+0000","last_became_peered":"2024-08-30T23:41:08.809599+0000","last_unstale":"2024-08-30T23:42:51.415620+0000","last_undegraded":"2024-08-30T23:42:51.415620+0000","last_fullsized":"2024-08-30T23:42:51.415620+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5363","last_scrub_stamp":"2024-08-30T23:41:11.016975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:11.016975+0000","objects_scrubbed":2311,"log_size":5931,"log_dups_size":0,"ondisk_log_size":5931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114229248,"num_objects":1743,"num_object_clones":0,"num_object_copies":5229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1743,"num_objects_unfound":0,"num_objects_dirty":1743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":568,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2],"acting":[10,2,8],"avail_no_missing":["10","2","8"],"object_location_counts":[{"shards":"2,8,10","objects":1743}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.0","version":"548'5991","reported_seq":9067,"reported_epoch":548,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:51.501295+0000","last_change":"2024-08-30T23:42:33.864088+0000","last_active":"2024-08-30T23:42:51.501295+0000","last_peered":"2024-08-30T23:42:51.501295+0000","last_clean":"2024-08-30T23:42:10.748576+0000","last_became_active":"2024-08-30T23:42:14.130267+0000","last_became_peered":"2024-08-30T23:42:14.130267+0000","last_unstale":"2024-08-30T23:42:51.501295+0000","last_undegraded":"2024-08-30T23:42:13.712728+0000","last_fullsized":"2024-08-30T23:42:13.698241+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'5449","last_scrub_stamp":"2024-08-30T23:41:12.170402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:12.170402+0000","objects_scrubbed":2333,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117112832,"num_objects":1787,"num_object_clones":0,"num_object_copies":5361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":334,"num_objects_misplaced":664,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":546,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1327,"num_bytes_recovered":86573056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,4],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":664},{"shards":"4,10","objects":789}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":405,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.769850+0000","last_change":"2024-08-30T23:41:56.277682+0000","last_active":"2024-08-30T23:42:50.769850+0000","last_peered":"2024-08-30T23:42:50.769850+0000","last_clean":"2024-08-30T23:42:50.769850+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:42:50.769850+0000","last_undegraded":"2024-08-30T23:42:50.769850+0000","last_fullsized":"2024-08-30T23:42:50.769850+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:56.277620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:56.277620+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:14.390680+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"546'5790","reported_seq":6505,"reported_epoch":546,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-30T23:42:49.814801+0000","last_change":"2024-08-30T23:42:41.677439+0000","last_active":"2024-08-30T23:42:49.814801+0000","last_peered":"2024-08-30T23:42:49.814801+0000","last_clean":"2024-08-30T23:42:33.810376+0000","last_became_active":"2024-08-30T23:42:41.661333+0000","last_became_peered":"2024-08-30T23:42:41.661333+0000","last_unstale":"2024-08-30T23:42:49.814801+0000","last_undegraded":"2024-08-30T23:42:41.038505+0000","last_fullsized":"2024-08-30T23:42:49.814801+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5278","last_scrub_stamp":"2024-08-30T23:41:10.050789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:10.050789+0000","objects_scrubbed":2310,"log_size":5790,"log_dups_size":0,"ondisk_log_size":5790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117833728,"num_objects":1798,"num_object_clones":0,"num_object_copies":5394,"num_objects_missing_on_primary":2374,"num_objects_missing":2374,"num_objects_degraded":2395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":8060928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7,9","objects":1798}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":279,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.774389+0000","last_change":"2024-08-30T23:42:34.874715+0000","last_active":"2024-08-30T23:42:50.774389+0000","last_peered":"2024-08-30T23:42:50.774389+0000","last_clean":"2024-08-30T23:42:50.774389+0000","last_became_active":"2024-08-30T23:42:34.874330+0000","last_became_peered":"2024-08-30T23:42:34.874330+0000","last_unstale":"2024-08-30T23:42:50.774389+0000","last_undegraded":"2024-08-30T23:42:50.774389+0000","last_fullsized":"2024-08-30T23:42:50.774389+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:53.774753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:53.774753+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:19.261375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"548'5854","reported_seq":13585,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:52.066661+0000","last_change":"2024-08-30T23:42:34.395099+0000","last_active":"2024-08-30T23:42:52.066661+0000","last_peered":"2024-08-30T23:42:52.066661+0000","last_clean":"2024-08-30T23:42:52.066661+0000","last_became_active":"2024-08-30T23:41:08.807376+0000","last_became_peered":"2024-08-30T23:41:08.807376+0000","last_unstale":"2024-08-30T23:42:52.066661+0000","last_undegraded":"2024-08-30T23:42:52.066661+0000","last_fullsized":"2024-08-30T23:42:52.066661+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'5677","last_scrub_stamp":"2024-08-30T23:42:34.395019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:42:34.395019+0000","objects_scrubbed":1928,"log_size":5854,"log_dups_size":0,"ondisk_log_size":5854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:58.877473+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114753536,"num_objects":1751,"num_object_clones":0,"num_object_copies":5253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":262,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.774676+0000","last_change":"2024-08-30T23:41:52.863983+0000","last_active":"2024-08-30T23:42:50.774676+0000","last_peered":"2024-08-30T23:42:50.774676+0000","last_clean":"2024-08-30T23:42:50.774676+0000","last_became_active":"2024-08-30T23:41:38.809561+0000","last_became_peered":"2024-08-30T23:41:38.809561+0000","last_unstale":"2024-08-30T23:42:50.774676+0000","last_undegraded":"2024-08-30T23:42:50.774676+0000","last_fullsized":"2024-08-30T23:42:50.774676+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:52.863920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:52.863920+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:20.522310+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"548'5916","reported_seq":8305,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.863366+0000","last_change":"2024-08-30T23:41:10.841288+0000","last_active":"2024-08-30T23:42:50.863366+0000","last_peered":"2024-08-30T23:42:50.863366+0000","last_clean":"2024-08-30T23:42:50.863366+0000","last_became_active":"2024-08-30T23:41:08.802906+0000","last_became_peered":"2024-08-30T23:41:08.802906+0000","last_unstale":"2024-08-30T23:42:50.863366+0000","last_undegraded":"2024-08-30T23:42:50.863366+0000","last_fullsized":"2024-08-30T23:42:50.863366+0000","mapping_epoch":472,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5359","last_scrub_stamp":"2024-08-30T23:41:10.841189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:10.841189+0000","objects_scrubbed":2371,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1090,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118882304,"num_objects":1814,"num_object_clones":0,"num_object_copies":5442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":557,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":284,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.773329+0000","last_change":"2024-08-30T23:42:21.207791+0000","last_active":"2024-08-30T23:42:50.773329+0000","last_peered":"2024-08-30T23:42:50.773329+0000","last_clean":"2024-08-30T23:42:50.773329+0000","last_became_active":"2024-08-30T23:42:20.761486+0000","last_became_peered":"2024-08-30T23:42:20.761486+0000","last_unstale":"2024-08-30T23:42:50.773329+0000","last_undegraded":"2024-08-30T23:42:50.773329+0000","last_fullsized":"2024-08-30T23:42:50.773329+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:21.207735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:21.207735+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:35.850271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"548'6106","reported_seq":7444,"reported_epoch":548,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:52.081408+0000","last_change":"2024-08-30T23:42:36.020215+0000","last_active":"2024-08-30T23:42:52.081408+0000","last_peered":"2024-08-30T23:42:52.081408+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:42:36.009783+0000","last_became_peered":"2024-08-30T23:42:36.009783+0000","last_unstale":"2024-08-30T23:42:52.081408+0000","last_undegraded":"2024-08-30T23:42:35.936791+0000","last_fullsized":"2024-08-30T23:42:35.855836+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":6106,"log_dups_size":0,"ondisk_log_size":6106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121241600,"num_objects":1850,"num_object_clones":0,"num_object_copies":5550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":22937600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[6,1],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1817},{"shards":"1,6","objects":27}],"blocked_by":[],"up_primary":1,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":264,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.773996+0000","last_change":"2024-08-30T23:42:38.913443+0000","last_active":"2024-08-30T23:42:50.773996+0000","last_peered":"2024-08-30T23:42:50.773996+0000","last_clean":"2024-08-30T23:42:50.773996+0000","last_became_active":"2024-08-30T23:42:38.913092+0000","last_became_peered":"2024-08-30T23:42:38.913092+0000","last_unstale":"2024-08-30T23:42:50.773996+0000","last_undegraded":"2024-08-30T23:42:50.773996+0000","last_fullsized":"2024-08-30T23:42:50.773996+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.695539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.695539+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:52.069054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"548'5902","reported_seq":6242,"reported_epoch":548,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:42:52.384929+0000","last_change":"2024-08-30T23:41:08.803040+0000","last_active":"2024-08-30T23:42:52.384929+0000","last_peered":"2024-08-30T23:42:52.384929+0000","last_clean":"2024-08-30T23:39:20.863070+0000","last_became_active":"2024-08-30T23:41:08.803040+0000","last_became_peered":"2024-08-30T23:41:08.803040+0000","last_unstale":"2024-08-30T23:42:52.384929+0000","last_undegraded":"2024-08-30T23:41:08.797881+0000","last_fullsized":"2024-08-30T23:41:08.797756+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":158368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1724}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1137,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-08-30T23:42:49.658906+0000","last_change":"2024-08-30T23:42:38.912360+0000","last_active":"2024-08-30T23:42:49.658906+0000","last_peered":"2024-08-30T23:42:49.658906+0000","last_clean":"2024-08-30T23:42:49.658906+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:42:49.658906+0000","last_undegraded":"2024-08-30T23:42:49.658906+0000","last_fullsized":"2024-08-30T23:42:49.658906+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:55.978214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:55.978214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:19.994020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"547'6093","reported_seq":8843,"reported_epoch":548,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:50.814159+0000","last_change":"2024-08-30T23:42:23.561367+0000","last_active":"2024-08-30T23:42:50.814159+0000","last_peered":"2024-08-30T23:42:50.814159+0000","last_clean":"2024-08-30T23:37:33.020570+0000","last_became_active":"2024-08-30T23:42:23.440929+0000","last_became_peered":"2024-08-30T23:42:23.440929+0000","last_unstale":"2024-08-30T23:42:50.814159+0000","last_undegraded":"2024-08-30T23:42:23.424586+0000","last_fullsized":"2024-08-30T23:42:21.758857+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":1675,"num_objects_missing":1675,"num_objects_degraded":3163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3339,"num_write_kb":164512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":939,"num_bytes_recovered":61341696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,10],"acting":[1,9],"avail_no_missing":["9"],"object_location_counts":[{"shards":"1,9","objects":132},{"shards":"9","objects":1493}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":32,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-08-30T23:42:49.658791+0000","last_change":"2024-08-30T23:42:34.865291+0000","last_active":"2024-08-30T23:42:49.658791+0000","last_peered":"2024-08-30T23:42:49.658791+0000","last_clean":"2024-08-30T23:42:49.658791+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:42:49.658791+0000","last_undegraded":"2024-08-30T23:42:49.658791+0000","last_fullsized":"2024-08-30T23:42:49.658791+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.830758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.830758+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:01.087297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":109,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.776031+0000","last_change":"2024-08-30T23:41:52.944641+0000","last_active":"2024-08-30T23:42:50.776031+0000","last_peered":"2024-08-30T23:42:50.776031+0000","last_clean":"2024-08-30T23:42:50.776031+0000","last_became_active":"2024-08-30T23:41:40.754524+0000","last_became_peered":"2024-08-30T23:41:40.754524+0000","last_unstale":"2024-08-30T23:42:50.776031+0000","last_undegraded":"2024-08-30T23:42:50.776031+0000","last_fullsized":"2024-08-30T23:42:50.776031+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:52.944589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:52.944589+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:13.676860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"548'5961","reported_seq":14144,"reported_epoch":548,"state":"active+recovering","last_fresh":"2024-08-30T23:42:52.034261+0000","last_change":"2024-08-30T23:42:48.209395+0000","last_active":"2024-08-30T23:42:52.034261+0000","last_peered":"2024-08-30T23:42:52.034261+0000","last_clean":"2024-08-30T23:42:33.453117+0000","last_became_active":"2024-08-30T23:42:42.017899+0000","last_became_peered":"2024-08-30T23:42:42.017899+0000","last_unstale":"2024-08-30T23:42:52.034261+0000","last_undegraded":"2024-08-30T23:42:52.034261+0000","last_fullsized":"2024-08-30T23:42:52.034261+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5406","last_scrub_stamp":"2024-08-30T23:41:10.058231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:10.058231+0000","objects_scrubbed":2319,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115605504,"num_objects":1764,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":2307,"num_objects_missing":2307,"num_objects_degraded":0,"num_objects_misplaced":1718,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":13434880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":["0","10"],"object_location_counts":[{"shards":"0,9,10","objects":1718},{"shards":"0,9,10,11","objects":1},{"shards":"9,10,11","objects":45}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":102,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-08-30T23:42:49.658909+0000","last_change":"2024-08-30T23:42:38.915147+0000","last_active":"2024-08-30T23:42:49.658909+0000","last_peered":"2024-08-30T23:42:49.658909+0000","last_clean":"2024-08-30T23:42:49.658909+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:42:49.658909+0000","last_undegraded":"2024-08-30T23:42:49.658909+0000","last_fullsized":"2024-08-30T23:42:49.658909+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:57.017557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:57.017557+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:42:59.992257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"548'5930","reported_seq":6357,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:52.469714+0000","last_change":"2024-08-30T23:41:11.489609+0000","last_active":"2024-08-30T23:42:52.469714+0000","last_peered":"2024-08-30T23:42:52.469714+0000","last_clean":"2024-08-30T23:42:52.469714+0000","last_became_active":"2024-08-30T23:41:08.803797+0000","last_became_peered":"2024-08-30T23:41:08.803797+0000","last_unstale":"2024-08-30T23:42:52.469714+0000","last_undegraded":"2024-08-30T23:42:52.469714+0000","last_fullsized":"2024-08-30T23:42:52.469714+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5356","last_scrub_stamp":"2024-08-30T23:41:11.489550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:11.489550+0000","objects_scrubbed":2401,"log_size":5930,"log_dups_size":0,"ondisk_log_size":5930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119734272,"num_objects":1827,"num_object_clones":0,"num_object_copies":5481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":574,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":40,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-08-30T23:42:49.658881+0000","last_change":"2024-08-30T23:42:34.888040+0000","last_active":"2024-08-30T23:42:49.658881+0000","last_peered":"2024-08-30T23:42:49.658881+0000","last_clean":"2024-08-30T23:42:49.658881+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:42:49.658881+0000","last_undegraded":"2024-08-30T23:42:49.658881+0000","last_fullsized":"2024-08-30T23:42:49.658881+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:34.887995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:34.887995+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:35.698420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"548'6550","reported_seq":7718,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:51.469228+0000","last_change":"2024-08-30T23:40:46.105090+0000","last_active":"2024-08-30T23:42:51.469228+0000","last_peered":"2024-08-30T23:42:51.469228+0000","last_clean":"2024-08-30T23:42:51.469228+0000","last_became_active":"2024-08-30T23:39:53.469025+0000","last_became_peered":"2024-08-30T23:39:53.469025+0000","last_unstale":"2024-08-30T23:42:51.469228+0000","last_undegraded":"2024-08-30T23:42:51.469228+0000","last_fullsized":"2024-08-30T23:42:51.469228+0000","mapping_epoch":457,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":458,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5457","last_scrub_stamp":"2024-08-30T23:40:46.105001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:46.105001+0000","objects_scrubbed":4671,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234487808,"num_objects":3578,"num_object_clones":0,"num_object_copies":10734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":324096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3714,"num_bytes_recovered":225574912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,8],"acting":[10,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":291,"reported_epoch":548,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:42:50.773132+0000","last_change":"2024-08-30T23:42:35.864929+0000","last_active":"2024-08-30T23:42:50.773132+0000","last_peered":"2024-08-30T23:42:50.773132+0000","last_clean":"2024-08-30T23:42:50.773132+0000","last_became_active":"2024-08-30T23:42:35.864395+0000","last_became_peered":"2024-08-30T23:42:35.864395+0000","last_unstale":"2024-08-30T23:42:50.773132+0000","last_undegraded":"2024-08-30T23:42:50.773132+0000","last_fullsized":"2024-08-30T23:42:50.773132+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.288446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.288446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:16.763830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,3],"avail_no_missing":["6","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.b","version":"546'6565","reported_seq":8334,"reported_epoch":546,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:50.076502+0000","last_change":"2024-08-30T23:42:31.889749+0000","last_active":"2024-08-30T23:42:50.076502+0000","last_peered":"2024-08-30T23:42:50.076502+0000","last_clean":"2024-08-30T23:39:20.385995+0000","last_became_active":"2024-08-30T23:42:21.943496+0000","last_became_peered":"2024-08-30T23:42:21.943496+0000","last_unstale":"2024-08-30T23:42:50.076502+0000","last_undegraded":"2024-08-30T23:42:21.915592+0000","last_fullsized":"2024-08-30T23:42:21.757590+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240058368,"num_objects":3663,"num_object_clones":0,"num_object_copies":10989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3643,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6565,"num_write_kb":327296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":46858240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[9,0],"avail_no_missing":["9","0"],"object_location_counts":[{"shards":"0,9","objects":20}],"blocked_by":[],"up_primary":7,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.c","version":"546'6624","reported_seq":7080,"reported_epoch":546,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:49.741688+0000","last_change":"2024-08-30T23:42:22.025443+0000","last_active":"2024-08-30T23:42:49.741688+0000","last_peered":"2024-08-30T23:42:49.741688+0000","last_clean":"2024-08-30T23:42:19.742860+0000","last_became_active":"2024-08-30T23:42:22.015025+0000","last_became_peered":"2024-08-30T23:42:22.015025+0000","last_unstale":"2024-08-30T23:42:49.741688+0000","last_undegraded":"2024-08-30T23:42:21.782114+0000","last_fullsized":"2024-08-30T23:42:21.755587+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5607","last_scrub_stamp":"2024-08-30T23:40:50.258779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:50.258779+0000","objects_scrubbed":4793,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":3776,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":332800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,0],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4","objects":-632},{"shards":"0,4,6","objects":3776}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":233,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.775897+0000","last_change":"2024-08-30T23:42:20.900182+0000","last_active":"2024-08-30T23:42:50.775897+0000","last_peered":"2024-08-30T23:42:50.775897+0000","last_clean":"2024-08-30T23:42:50.775897+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:42:50.775897+0000","last_undegraded":"2024-08-30T23:42:50.775897+0000","last_fullsized":"2024-08-30T23:42:50.775897+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:20.900120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:20.900120+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:46.627535+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"548'6389","reported_seq":8384,"reported_epoch":548,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:52.055525+0000","last_change":"2024-08-30T23:42:33.865344+0000","last_active":"2024-08-30T23:42:52.055525+0000","last_peered":"2024-08-30T23:42:52.055525+0000","last_clean":"2024-08-30T23:39:20.866225+0000","last_became_active":"2024-08-30T23:42:21.982780+0000","last_became_peered":"2024-08-30T23:42:21.982780+0000","last_unstale":"2024-08-30T23:42:52.055525+0000","last_undegraded":"2024-08-30T23:42:21.792563+0000","last_fullsized":"2024-08-30T23:42:21.757892+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'3876","last_scrub_stamp":"2024-08-30T23:38:40.517849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:38:40.517849+0000","objects_scrubbed":3874,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228982808,"num_objects":3495,"num_object_clones":0,"num_object_copies":10485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6389,"num_write_kb":316225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":804,"num_bytes_recovered":52559872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,0],"acting":[11,7],"avail_no_missing":["11","7"],"object_location_counts":[{"shards":"7,11","objects":107}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1119,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.768602+0000","last_change":"2024-08-30T23:40:49.616038+0000","last_active":"2024-08-30T23:42:50.768602+0000","last_peered":"2024-08-30T23:42:50.768602+0000","last_clean":"2024-08-30T23:42:50.768602+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:42:50.768602+0000","last_undegraded":"2024-08-30T23:42:50.768602+0000","last_fullsized":"2024-08-30T23:42:50.768602+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:49.615961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.615961+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"546'6237","reported_seq":6670,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-08-30T23:42:49.983704+0000","last_change":"2024-08-30T23:40:49.893551+0000","last_active":"2024-08-30T23:42:49.983704+0000","last_peered":"2024-08-30T23:42:49.983704+0000","last_clean":"2024-08-30T23:42:49.983704+0000","last_became_active":"2024-08-30T23:37:34.071585+0000","last_became_peered":"2024-08-30T23:37:34.071585+0000","last_unstale":"2024-08-30T23:42:49.983704+0000","last_undegraded":"2024-08-30T23:42:49.983704+0000","last_fullsized":"2024-08-30T23:42:49.983704+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'5233","last_scrub_stamp":"2024-08-30T23:40:49.893457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:40:49.893457+0000","objects_scrubbed":4463,"log_size":6237,"log_dups_size":0,"ondisk_log_size":6237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226689024,"num_objects":3459,"num_object_clones":0,"num_object_copies":10377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6237,"num_write_kb":310272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":237,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-08-30T23:42:49.671715+0000","last_change":"2024-08-30T23:40:26.468648+0000","last_active":"2024-08-30T23:42:49.671715+0000","last_peered":"2024-08-30T23:42:49.671715+0000","last_clean":"2024-08-30T23:42:49.671715+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:42:49.671715+0000","last_undegraded":"2024-08-30T23:42:49.671715+0000","last_fullsized":"2024-08-30T23:42:49.671715+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:40:26.468590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:40:26.468590+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":20,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.769509+0000","last_change":"2024-08-30T23:42:43.957815+0000","last_active":"2024-08-30T23:42:50.769509+0000","last_peered":"2024-08-30T23:42:50.769509+0000","last_clean":"2024-08-30T23:42:50.769509+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:42:50.769509+0000","last_undegraded":"2024-08-30T23:42:50.769509+0000","last_fullsized":"2024-08-30T23:42:50.769509+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:52.789800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:52.789800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:21.758151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.f","version":"548'6574","reported_seq":8499,"reported_epoch":548,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:53.380843+0000","last_change":"2024-08-30T23:42:37.928811+0000","last_active":"2024-08-30T23:42:53.380843+0000","last_peered":"2024-08-30T23:42:53.380843+0000","last_clean":"2024-08-30T23:39:20.857133+0000","last_became_active":"2024-08-30T23:42:36.049504+0000","last_became_peered":"2024-08-30T23:42:36.049504+0000","last_unstale":"2024-08-30T23:42:53.380843+0000","last_undegraded":"2024-08-30T23:42:35.889038+0000","last_fullsized":"2024-08-30T23:42:35.856307+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":389,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:37:33.020570+0000","objects_scrubbed":0,"log_size":6574,"log_dups_size":0,"ondisk_log_size":6574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3424,"num_objects_misplaced":3093,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6574,"num_write_kb":319936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":794,"num_bytes_recovered":52035584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11],"acting":[2,11],"avail_no_missing":["2","11"],"object_location_counts":[{"shards":"2,11","objects":331}],"blocked_by":[],"up_primary":4,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":976,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:50.767144+0000","last_change":"2024-08-30T23:41:53.017039+0000","last_active":"2024-08-30T23:42:50.767144+0000","last_peered":"2024-08-30T23:42:50.767144+0000","last_clean":"2024-08-30T23:42:50.767144+0000","last_became_active":"2024-08-30T23:41:40.752006+0000","last_became_peered":"2024-08-30T23:41:40.752006+0000","last_unstale":"2024-08-30T23:42:50.767144+0000","last_undegraded":"2024-08-30T23:42:50.767144+0000","last_fullsized":"2024-08-30T23:42:50.767144+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:53.016971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:53.016971+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:43:21.498152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"548'6006","reported_seq":16365,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-08-30T23:42:52.397796+0000","last_change":"2024-08-30T23:42:34.586807+0000","last_active":"2024-08-30T23:42:52.397796+0000","last_peered":"2024-08-30T23:42:52.397796+0000","last_clean":"2024-08-30T23:42:52.397796+0000","last_became_active":"2024-08-30T23:42:33.860387+0000","last_became_peered":"2024-08-30T23:42:33.860387+0000","last_unstale":"2024-08-30T23:42:52.397796+0000","last_undegraded":"2024-08-30T23:42:52.397796+0000","last_fullsized":"2024-08-30T23:42:52.397796+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":5,"last_scrub":"535'5800","last_scrub_stamp":"2024-08-30T23:42:34.586736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:42:34.586736+0000","objects_scrubbed":1922,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:44:03.692674+0000","scrub_duration":320,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112459776,"num_objects":1716,"num_object_clones":0,"num_object_copies":5148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":206,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,6],"acting":[7,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"548'5802","reported_seq":8666,"reported_epoch":548,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:42:52.091892+0000","last_change":"2024-08-30T23:42:36.000238+0000","last_active":"2024-08-30T23:42:52.091892+0000","last_peered":"2024-08-30T23:42:52.091892+0000","last_clean":"2024-08-30T23:42:33.834226+0000","last_became_active":"2024-08-30T23:42:35.983589+0000","last_became_peered":"2024-08-30T23:42:35.983589+0000","last_unstale":"2024-08-30T23:42:52.091892+0000","last_undegraded":"2024-08-30T23:42:35.870378+0000","last_fullsized":"2024-08-30T23:42:35.855639+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":5,"last_scrub":"492'5286","last_scrub_stamp":"2024-08-30T23:41:54.528134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:37:33.020570+0000","last_clean_scrub_stamp":"2024-08-30T23:41:54.528134+0000","objects_scrubbed":2248,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113508352,"num_objects":1732,"num_object_clones":0,"num_object_copies":5196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":815,"num_objects_unfound":0,"num_objects_dirty":1732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":516,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1094,"num_bytes_recovered":71696384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,9],"avail_no_missing":["6","3","7","9"],"object_location_counts":[{"shards":"6,7,9","objects":815},{"shards":"6,9","objects":916}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":26,"stat_sum":{"num_bytes":3711238168,"num_objects":56630,"num_object_clones":0,"num_object_copies":169890,"num_objects_missing_on_primary":6356,"num_objects_missing":6356,"num_objects_degraded":36763,"num_objects_misplaced":12803,"num_objects_unfound":0,"num_objects_dirty":56630,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":63092,"num_write_kb":2736801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30087,"num_bytes_recovered":1852768256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13444329472,"data_stored":13444317256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157829,"ondisk_log_size":157829,"up":73,"acting":66,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":62,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187676,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063152,"kb_used_data":684896,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92682000,"statfs":{"total":95995035648,"available":94906368000,"internally_reserved":0,"allocated":701333504,"data_stored":700392390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":387316164},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,3,2,1,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449315,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1675700,"kb_used_data":1405724,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92069452,"statfs":{"total":95995035648,"available":94279118848,"internally_reserved":0,"allocated":1439461376,"data_stored":1438524342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":276429248},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710950,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1335116,"kb_used_data":929080,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92410036,"statfs":{"total":95995035648,"available":94627876864,"internally_reserved":0,"allocated":951377920,"data_stored":950452992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":415758784},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972588,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1845276,"kb_used_data":1512844,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":91899876,"statfs":{"total":95995035648,"available":94105473024,"internally_reserved":0,"allocated":1549152256,"data_stored":1548219782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":340392389},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234223,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1938936,"kb_used_data":1484320,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91806216,"statfs":{"total":95995035648,"available":94009565184,"internally_reserved":0,"allocated":1519943680,"data_stored":1519002574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":465500618},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528566,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1542636,"kb_used_data":1226844,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92202516,"statfs":{"total":95995035648,"available":94415376384,"internally_reserved":0,"allocated":1256288256,"data_stored":1255351222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724750,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2374104,"kb_used_data":2072200,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91371048,"statfs":{"total":95995035648,"available":93563953152,"internally_reserved":0,"allocated":2121932800,"data_stored":2121416724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953727,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884872,"kb_used_data":542712,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92860280,"statfs":{"total":95995035648,"available":95088926720,"internally_reserved":0,"allocated":555737088,"data_stored":554812160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149956,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107508,"kb_used_data":693156,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92637644,"statfs":{"total":95995035648,"available":94860947456,"internally_reserved":0,"allocated":709791744,"data_stored":708851158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411594,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1169140,"kb_used_data":903332,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92576012,"statfs":{"total":95995035648,"available":94797836288,"internally_reserved":0,"allocated":925011968,"data_stored":924071382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830409,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492700,"kb_used_data":382852,"kb_used_omap":87,"kb_used_meta":109736,"kb_avail":93252452,"statfs":{"total":95995035648,"available":95490510848,"internally_reserved":0,"allocated":392040448,"data_stored":391111526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":112369823},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738643,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959212,"kb_used_data":1499036,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91785940,"statfs":{"total":95995035648,"available":93988802560,"internally_reserved":0,"allocated":1535012864,"data_stored":1534075830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1525153792,"data_stored":1525153792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":395706368,"data_stored":395706368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":914489344,"data_stored":914489344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":701759488,"data_stored":701759488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":554565632,"data_stored":554565632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2117668864,"data_stored":2117664792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1252327424,"data_stored":1252327424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1506414592,"data_stored":1506410520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1407385600,"data_stored":1407385600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":943390720,"data_stored":943390720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1427177472,"data_stored":1427177472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":698290176,"data_stored":698286104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:42:57.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_3 pgp_num 26 2024-08-30T23:42:57.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:57.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:57.976 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:57 smithi136 ceph-mon[20207]: 5.12 scrub starts 2024-08-30T23:42:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:57 smithi136 ceph-mon[20207]: pgmap v1263: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 35493/162207 objects degraded (21.881%); 11382/162207 objects misplaced (7.017%); 39 MiB/s, 762 objects/s recovering 2024-08-30T23:42:58.042 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:58.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:57 smithi173 ceph-mon[20554]: 5.12 scrub starts 2024-08-30T23:42:58.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:57 smithi173 ceph-mon[20554]: pgmap v1263: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 35493/162207 objects degraded (21.881%); 11382/162207 objects misplaced (7.017%); 39 MiB/s, 762 objects/s recovering 2024-08-30T23:42:58.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:58.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:58.234 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:58.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:58.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:58.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:42:58.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:42:58.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:58.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:58.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:58.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:58.832 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:42:58.833 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:42:58.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:42:58.864 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:42:58.864 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:42:58.889 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:42:58.889 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:42:58.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:58.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:42:58.915 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:42:58.916 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:42:58.933 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3623560153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:58.942 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:42:58.942 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:42:58.942 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:42:58.971 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:42:58.971 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:42:58.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:42:58.997 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:42:58.998 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:42:59.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:59.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:42:59.033 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:42:59.033 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:42:59.044 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:42:59.056 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:42:59.056 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:42:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3623560153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:59.117 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:42:59.118 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:42:59.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:42:59.150 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:42:59.151 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:42:59.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:42:59.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:42:59.286 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:42:59.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3623560153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:42:59.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:42:59.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:42:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:42:59.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:59.546 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:59.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:42:59.675 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:42:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:42:59.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:42:59.802 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:42:59.864 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:42:59.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:42:59.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:42:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:42:59.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:00.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:00.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:00.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1753171682' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:00.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:59 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:00.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:42:59 smithi106 ceph-mon[16603]: pgmap v1264: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 521 op/s; 35295/160752 objects degraded (21.956%); 11151/160752 objects misplaced (6.937%); 30 MiB/s, 512 objects/s recovering 2024-08-30T23:43:00.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:00.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:00.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1753171682' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:59 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:42:59 smithi136 ceph-mon[20207]: pgmap v1264: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 521 op/s; 35295/160752 objects degraded (21.956%); 11151/160752 objects misplaced (6.937%); 30 MiB/s, 512 objects/s recovering 2024-08-30T23:43:00.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1753171682' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:59 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:42:59 smithi173 ceph-mon[20554]: pgmap v1264: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 521 op/s; 35295/160752 objects degraded (21.956%); 11151/160752 objects misplaced (6.937%); 30 MiB/s, 512 objects/s recovering 2024-08-30T23:43:00.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:00.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:00.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:00.574 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:00.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:00.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:00.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:00.830 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:00.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:00.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:00.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:01.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:43:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: osdmap e551: 12 total, 11 up, 12 in 2024-08-30T23:43:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 35295/160752 objects degraded (21.956%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: osdmap e552: 12 total, 11 up, 12 in 2024-08-30T23:43:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1336257638' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-08-30T23:43:01.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:00 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-08-30T23:43:01.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:43:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: osdmap e551: 12 total, 11 up, 12 in 2024-08-30T23:43:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 35295/160752 objects degraded (21.956%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: osdmap e552: 12 total, 11 up, 12 in 2024-08-30T23:43:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1336257638' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-08-30T23:43:01.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:00 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-08-30T23:43:01.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:01.214 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:01.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:43:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: osdmap e551: 12 total, 11 up, 12 in 2024-08-30T23:43:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 35295/160752 objects degraded (21.956%), 14 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: osdmap e552: 12 total, 11 up, 12 in 2024-08-30T23:43:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1336257638' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-08-30T23:43:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:00 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"} : dispatch 2024-08-30T23:43:01.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:01.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:01.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:01.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:01.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:01.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:01.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:01.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:01.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:01.772 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 5 pgp_num to 26 2024-08-30T23:43:01.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:01.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:01.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:01.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:01.982 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:02.083 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:01 smithi106 ceph-mon[16603]: pgmap v1267: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 34659/156261 objects degraded (22.180%); 10660/156261 objects misplaced (6.822%); 27 MiB/s, 494 objects/s recovering 2024-08-30T23:43:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:01 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-08-30T23:43:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:01 smithi106 ceph-mon[16603]: osdmap e553: 12 total, 11 up, 12 in 2024-08-30T23:43:02.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:02.177 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:43:02.177 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:43:02.177 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:43:02.177 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:43:02.178 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:01 smithi173 ceph-mon[20554]: pgmap v1267: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 34659/156261 objects degraded (22.180%); 10660/156261 objects misplaced (6.822%); 27 MiB/s, 494 objects/s recovering 2024-08-30T23:43:02.178 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:01 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-08-30T23:43:02.178 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:01 smithi173 ceph-mon[20554]: osdmap e553: 12 total, 11 up, 12 in 2024-08-30T23:43:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:43:02.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:43:02.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:02.247 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:01 smithi136 ceph-mon[20207]: pgmap v1267: 47 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 3 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 34659/156261 objects degraded (22.180%); 10660/156261 objects misplaced (6.822%); 27 MiB/s, 494 objects/s recovering 2024-08-30T23:43:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:01 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "26"}]': finished 2024-08-30T23:43:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:01 smithi136 ceph-mon[20207]: osdmap e553: 12 total, 11 up, 12 in 2024-08-30T23:43:02.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:02.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:02.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:02.603 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:02.628 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:02.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:02.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:02.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:02.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:02.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:03.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:03.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:02 smithi106 ceph-mon[16603]: 5.12 scrub starts 2024-08-30T23:43:03.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:03.165 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:03.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:03.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:02 smithi136 ceph-mon[20207]: 5.12 scrub starts 2024-08-30T23:43:03.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.292 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:02 smithi173 ceph-mon[20554]: 5.12 scrub starts 2024-08-30T23:43:03.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:03.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:03.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:03.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:43:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:03.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:03.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:03.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:03.771 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:03.829 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:03.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:03.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:03.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:04.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:04.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:04.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:04.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:04.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:43:04.216 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:03 smithi106 ceph-mon[16603]: 5.12 scrub ok 2024-08-30T23:43:04.216 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:03 smithi106 ceph-mon[16603]: osdmap e554: 12 total, 11 up, 12 in 2024-08-30T23:43:04.216 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:03 smithi106 ceph-mon[16603]: pgmap v1270: 47 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 29867/152604 objects degraded (19.572%); 12568/152604 objects misplaced (8.236%); 33 MiB/s, 766 objects/s recovering 2024-08-30T23:43:04.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:04.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:03 smithi173 ceph-mon[20554]: 5.12 scrub ok 2024-08-30T23:43:04.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:03 smithi173 ceph-mon[20554]: osdmap e554: 12 total, 11 up, 12 in 2024-08-30T23:43:04.398 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:03 smithi173 ceph-mon[20554]: pgmap v1270: 47 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 29867/152604 objects degraded (19.572%); 12568/152604 objects misplaced (8.236%); 33 MiB/s, 766 objects/s recovering 2024-08-30T23:43:04.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:03 smithi136 ceph-mon[20207]: 5.12 scrub ok 2024-08-30T23:43:04.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:03 smithi136 ceph-mon[20207]: osdmap e554: 12 total, 11 up, 12 in 2024-08-30T23:43:04.398 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:03 smithi136 ceph-mon[20207]: pgmap v1270: 47 pgs: 1 active+recovery_wait+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 29867/152604 objects degraded (19.572%); 12568/152604 objects misplaced (8.236%); 33 MiB/s, 766 objects/s recovering 2024-08-30T23:43:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.427 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:04.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:04.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:04.562 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:04.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:04.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:04.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:04.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:04.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:04.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.832 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:04.896 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:43:04.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:04.956 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:04 smithi106 ceph-mon[16603]: osdmap e555: 12 total, 11 up, 12 in 2024-08-30T23:43:04.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1651569983' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:04.957 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:04 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:04.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:04.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:05.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:05.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:05.104 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:05.104 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:05.130 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:05.130 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:05.156 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:05.156 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:05.183 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:05.183 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:05.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:05.212 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:05.213 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:05.214 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:43:05.223 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:04 smithi136 ceph-mon[20207]: osdmap e555: 12 total, 11 up, 12 in 2024-08-30T23:43:05.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1651569983' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:05.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:04 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:05.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:04 smithi173 ceph-mon[20554]: osdmap e555: 12 total, 11 up, 12 in 2024-08-30T23:43:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1651569983' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:04 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:05.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:05.350 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:05.447 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:05.448 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:05.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:05.477 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:05.477 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:05.479 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:05.508 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:05.509 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:05.576 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:05.577 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:05.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:05.608 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:05.609 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:05.609 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:05.638 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:05.639 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:05.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:05.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:05.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:05.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:05.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:05.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:05.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:06.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:06.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:06.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.126 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:06.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:06.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:05 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:43:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:05 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:05 smithi136 ceph-mon[20207]: osdmap e556: 12 total, 11 up, 12 in 2024-08-30T23:43:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1625376045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:05 smithi136 ceph-mon[20207]: pgmap v1273: 47 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 25416/151200 objects degraded (16.810%); 15678/151200 objects misplaced (10.369%); 31 MiB/s, 637 objects/s recovering 2024-08-30T23:43:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 29867/152604 objects degraded (19.572%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:05 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:43:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:05 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:05 smithi173 ceph-mon[20554]: osdmap e556: 12 total, 11 up, 12 in 2024-08-30T23:43:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1625376045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:05 smithi173 ceph-mon[20554]: pgmap v1273: 47 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 25416/151200 objects degraded (16.810%); 15678/151200 objects misplaced (10.369%); 31 MiB/s, 637 objects/s recovering 2024-08-30T23:43:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 29867/152604 objects degraded (19.572%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:05 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:43:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:05 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:05 smithi106 ceph-mon[16603]: osdmap e556: 12 total, 11 up, 12 in 2024-08-30T23:43:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1625376045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:05 smithi106 ceph-mon[16603]: pgmap v1273: 47 pgs: 1 active+recovery_wait+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 29 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 25416/151200 objects degraded (16.810%); 15678/151200 objects misplaced (10.369%); 31 MiB/s, 637 objects/s recovering 2024-08-30T23:43:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 29867/152604 objects degraded (19.572%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:06.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:06.388 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:06.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:06.519 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:06.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:06.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:06.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:06.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:06.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:06.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:06.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:07.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:07.048 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:07.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:07.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:07.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:07.305 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:07.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.433 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:07.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:07.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:07.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:07.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:07.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:07.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:07.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:07.893 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:43:07.894 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:43:07.894 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-08-30T23:43:07.894 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-08-30T23:43:07.894 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service 2024-08-30T23:43:07.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl start ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5 2024-08-30T23:43:07.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:07.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:07.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:08.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:08.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:08.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:08.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:07 smithi136 ceph-mon[20207]: osdmap e557: 12 total, 11 up, 12 in 2024-08-30T23:43:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:07 smithi136 ceph-mon[20207]: pgmap v1275: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 30 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21810/146343 objects degraded (14.903%); 16433/146343 objects misplaced (11.229%) 2024-08-30T23:43:08.291 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:07 smithi136 systemd[1]: Starting Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:43:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:08 smithi173 ceph-mon[20554]: osdmap e557: 12 total, 11 up, 12 in 2024-08-30T23:43:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:08 smithi173 ceph-mon[20554]: pgmap v1275: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 30 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21810/146343 objects degraded (14.903%); 16433/146343 objects misplaced (11.229%) 2024-08-30T23:43:08.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:08.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:07 smithi106 ceph-mon[16603]: osdmap e557: 12 total, 11 up, 12 in 2024-08-30T23:43:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:07 smithi106 ceph-mon[16603]: pgmap v1275: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 30 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21810/146343 objects degraded (14.903%); 16433/146343 objects misplaced (11.229%) 2024-08-30T23:43:08.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:08.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:08.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:08.607 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:08.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:08.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:08.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:08 smithi136 podman[106930]: 2024-08-30 23:43:08.22625521 +0000 UTC m=+0.016170675 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:43:08.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:08.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:08.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:08.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:08.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:09.000 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:09.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:09.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:09.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:09.262 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:09.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:09.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:09.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:09.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:09.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:09.639 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:09.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:09.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:09.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:09.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:09.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:09.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:09.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:10.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:10.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:09 smithi136 podman[106930]: 2024-08-30 23:43:09.766133558 +0000 UTC m=+1.556049021 container create e04af61b57fbfa503f18a09bb247fc7fb41a12fcc0ed7acaec13da81ce0511a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-08-30T23:43:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:09 smithi136 ceph-mon[20207]: osdmap e558: 12 total, 11 up, 12 in 2024-08-30T23:43:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:09 smithi136 ceph-mon[20207]: pgmap v1277: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 30 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21742/145128 objects degraded (14.981%); 15840/145128 objects misplaced (10.915%) 2024-08-30T23:43:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:43:10.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:43:10.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:10.155 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:10.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:10.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:09 smithi173 ceph-mon[20554]: osdmap e558: 12 total, 11 up, 12 in 2024-08-30T23:43:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:09 smithi173 ceph-mon[20554]: pgmap v1277: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 30 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21742/145128 objects degraded (14.981%); 15840/145128 objects misplaced (10.915%) 2024-08-30T23:43:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:09 smithi106 ceph-mon[16603]: osdmap e558: 12 total, 11 up, 12 in 2024-08-30T23:43:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:09 smithi106 ceph-mon[16603]: pgmap v1277: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+recovering, 1 active+undersized+degraded, 30 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21742/145128 objects degraded (14.981%); 15840/145128 objects misplaced (10.915%) 2024-08-30T23:43:10.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:10.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:10.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:10.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:10.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:10.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:10.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:10.812 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:10.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:10.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:10.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:11.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:11.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:11.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:11.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:10 smithi136 ceph-mon[20207]: osdmap e559: 12 total, 11 up, 12 in 2024-08-30T23:43:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 21742/145128 objects degraded (14.981%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:11 smithi136 ceph-mon[20207]: osdmap e560: 12 total, 11 up, 12 in 2024-08-30T23:43:11.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:10 smithi173 ceph-mon[20554]: osdmap e559: 12 total, 11 up, 12 in 2024-08-30T23:43:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 21742/145128 objects degraded (14.981%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:10 smithi173 ceph-mon[20554]: osdmap e560: 12 total, 11 up, 12 in 2024-08-30T23:43:11.333 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:10 smithi106 ceph-mon[16603]: osdmap e559: 12 total, 11 up, 12 in 2024-08-30T23:43:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 21742/145128 objects degraded (14.981%), 12 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:10 smithi106 ceph-mon[16603]: osdmap e560: 12 total, 11 up, 12 in 2024-08-30T23:43:11.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:11.461 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:11.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:43:11.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:43:11.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:11.590 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:11.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:11.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:11.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:11.845 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:11.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:11.892 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:43:11.954 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:11.986 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:11.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:12.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:43:12.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:12.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:12.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:12.236 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:11 smithi106 ceph-mon[16603]: pgmap v1280: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 2 active+recovering, 1 active+undersized+degraded, 30 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 19068/141228 objects degraded (13.502%); 16941/141228 objects misplaced (11.995%); 25 MiB/s, 499 objects/s recovering 2024-08-30T23:43:12.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:11 smithi136 ceph-mon[20207]: pgmap v1280: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 2 active+recovering, 1 active+undersized+degraded, 30 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 19068/141228 objects degraded (13.502%); 16941/141228 objects misplaced (11.995%); 25 MiB/s, 499 objects/s recovering 2024-08-30T23:43:12.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:11 smithi173 ceph-mon[20554]: pgmap v1280: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 8 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 2 active+recovering, 1 active+undersized+degraded, 30 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 19068/141228 objects degraded (13.502%); 16941/141228 objects misplaced (11.995%); 25 MiB/s, 499 objects/s recovering 2024-08-30T23:43:12.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:12.376 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:12.440 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:12.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.512 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:12.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:12.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:12.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:12.658 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 podman[106930]: 2024-08-30 23:43:12.393715723 +0000 UTC m=+4.183631180 container init e04af61b57fbfa503f18a09bb247fc7fb41a12fcc0ed7acaec13da81ce0511a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-08-30T23:43:12.659 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 podman[106930]: 2024-08-30 23:43:12.398592298 +0000 UTC m=+4.188507763 container start e04af61b57fbfa503f18a09bb247fc7fb41a12fcc0ed7acaec13da81ce0511a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-30T23:43:12.659 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:12.659 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:12.659 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-30T23:43:12.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:12.764 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:12.866 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:12.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:12.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:12.978 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:12.979 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:12.990 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:43:12.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:13.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:13.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:13.152 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:13.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:13.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:13.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:43:13.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-08-30T23:43:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/805713484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:13.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:12 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/805713484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:12 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/805713484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:13.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:12 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:13.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:13.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:13.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:13.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:13.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:13.682 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:13.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:13.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:43:13.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:43:13.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-30T23:43:13.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[107080]: --> ceph-volume lvm activate successful for osd ID: 5 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-30T23:43:13.792 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:43:13.793 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 bash[106930]: --> ceph-volume lvm activate successful for osd ID: 5 2024-08-30T23:43:13.793 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 podman[106930]: 2024-08-30 23:43:13.52121205 +0000 UTC m=+5.311127507 container attach e04af61b57fbfa503f18a09bb247fc7fb41a12fcc0ed7acaec13da81ce0511a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-30T23:43:13.793 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:13 smithi136 podman[106930]: 2024-08-30 23:43:13.545589507 +0000 UTC m=+5.335504963 container died e04af61b57fbfa503f18a09bb247fc7fb41a12fcc0ed7acaec13da81ce0511a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-08-30T23:43:13.812 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:43:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:43:13.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:13.892 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:13.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:13.921 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:13.921 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:13.938 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:13.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:13.946 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:13.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:13.971 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:13.971 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:13.996 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:13.996 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:13.996 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:43:14.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:14.018 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:14.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:14.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:13 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:13 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:43:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:13 smithi173 ceph-mon[20554]: osdmap e561: 12 total, 11 up, 12 in 2024-08-30T23:43:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/638022117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:14.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:13 smithi173 ceph-mon[20554]: pgmap v1282: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 18217/137361 objects degraded (13.262%); 15707/137361 objects misplaced (11.435%); 28 MiB/s, 508 objects/s recovering 2024-08-30T23:43:14.105 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:14.105 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:14.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:13 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:14.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:13 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:43:14.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:13 smithi106 ceph-mon[16603]: osdmap e561: 12 total, 11 up, 12 in 2024-08-30T23:43:14.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/638022117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:14.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:13 smithi106 ceph-mon[16603]: pgmap v1282: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 18217/137361 objects degraded (13.262%); 15707/137361 objects misplaced (11.435%); 28 MiB/s, 508 objects/s recovering 2024-08-30T23:43:14.134 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:14.134 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:14.164 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:14.165 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:14.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:14.199 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:14.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:14.200 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:14.222 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:14.223 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:43:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-mon[20207]: osdmap e561: 12 total, 11 up, 12 in 2024-08-30T23:43:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/638022117' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:13 smithi136 ceph-mon[20207]: pgmap v1282: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 18217/137361 objects degraded (13.262%); 15707/137361 objects misplaced (11.435%); 28 MiB/s, 508 objects/s recovering 2024-08-30T23:43:14.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:14.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:14.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:14.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:14.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:14.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:14.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:14.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:14.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:14.862 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:14.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:43:14.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:43:14.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:14.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:43:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:14.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:14.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:15.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:15.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:14 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:15.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:14 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:15.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:14 smithi106 ceph-mon[16603]: osdmap e562: 12 total, 11 up, 12 in 2024-08-30T23:43:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:15.129 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:15.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:15.259 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:14 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:14 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:14 smithi136 ceph-mon[20207]: osdmap e562: 12 total, 11 up, 12 in 2024-08-30T23:43:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:14 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:14 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:14 smithi173 ceph-mon[20554]: osdmap e562: 12 total, 11 up, 12 in 2024-08-30T23:43:15.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:15.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:15.394 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:43:15.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:15.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:15.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:15.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:15.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:15.775 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:15.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:43:15.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:15.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:15.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:15.973 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:15 smithi136 podman[107272]: 2024-08-30 23:43:15.720220276 +0000 UTC m=+2.162766690 container remove e04af61b57fbfa503f18a09bb247fc7fb41a12fcc0ed7acaec13da81ce0511a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-30T23:43:16.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.046 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:16.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:16 smithi173 ceph-mon[20554]: pgmap v1284: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 18012/135294 objects degraded (13.313%); 15331/135294 objects misplaced (11.332%); 26 MiB/s, 467 objects/s recovering 2024-08-30T23:43:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 18217/137361 objects degraded (13.262%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:16.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:16.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:16.176 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:16.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:16 smithi136 ceph-mon[20207]: pgmap v1284: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 18012/135294 objects degraded (13.313%); 15331/135294 objects misplaced (11.332%); 26 MiB/s, 467 objects/s recovering 2024-08-30T23:43:16.249 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 18217/137361 objects degraded (13.262%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:16.249 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:15 smithi136 podman[107375]: 2024-08-30 23:43:15.878364189 +0000 UTC m=+0.024813975 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:43:16.249 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:16 smithi136 podman[107375]: 2024-08-30 23:43:16.004637586 +0000 UTC m=+0.151087354 container create ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-30T23:43:16.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:16.290 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:43:16.290 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:43:16.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.15351.0:206901 3.7 e300/206 3:e91c2dd8:::benchmark_data_smithi173_60780_object103447:head v 300'13076, mlcod=300'13071)", 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 571.90336277999995, 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38146454200000002, 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.291 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064145+0000", 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064146+0000", 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8000000000000005e-07 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.064151+0000", 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8270000000000003e-06 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441558+0000", 2024-08-30T23:43:16.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37740696499999998 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441573+0000", 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4623e-05 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441672+0000", 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9351999999999993e-05 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.441691+0000", 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8134999999999998e-05 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445570+0000", 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003879511 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:44.445609+0000", 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9348999999999999e-05 2024-08-30T23:43:16.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87cb8fd9:::benchmark_data_smithi173_60780_object16345:head,262'7462)", 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 564.02125663300001, 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72540086000000004, 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946251+0000", 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946252+0000", 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3399999999999999e-07 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:51.946257+0000", 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2349999999999999e-06 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671354+0000", 2024-08-30T23:43:16.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72509723199999998 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671379+0000", 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5279e-05 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671555+0000", 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000175203 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671572+0000", 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.774e-05 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:52.671652+0000", 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9536999999999994e-05 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDelete(3.1 e303,280 3:87ea66dc:::benchmark_data_smithi173_60780_object18546:head,263'7595)", 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:43:16.300 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 562.08490824299997, 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85628250299999997, 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882599+0000", 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882600+0000", 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3e-07 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:53.882604+0000", 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2289999999999997e-06 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738700+0000", 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85609620099999995 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738719+0000", 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8343e-05 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:33:54.738882+0000", 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016310000000000001 2024-08-30T23:43:16.303 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31056 4.8 4.b8866df8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 535.28145421399995, 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99756266500000002, 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:43:16.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31056 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686053+0000", 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686055+0000", 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6950000000000001e-06 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686078+0000", 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3113e-05 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686093+0000", 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4856e-05 2024-08-30T23:43:16.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686100+0000", 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3930000000000002e-06 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686163+0000", 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2131000000000005e-05 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:20.686172+0000", 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9849999999999995e-06 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683581+0000", 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99740900499999996 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683616+0000", 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5487000000000001e-05 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:31071 4.8 4.35742af8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 534.67179611999995, 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38794466599999999, 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.309 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 31071 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295711+0000", 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295712+0000", 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1299999999999998e-07 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295733+0000", 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0463e-05 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295744+0000", 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1534e-05 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295750+0000", 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0599999999999996e-06 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295804+0000", 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3983000000000001e-05 2024-08-30T23:43:16.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.295812+0000", 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4290000000000004e-06 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683649+0000", 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38783691999999997 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:21.683656+0000", 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3640000000000001e-06 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:62151 4.1 4:8b395779:::benchmark_data_smithi173_74914_object62150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 498.48920080599999, 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38737673, 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.314 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62151 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478307+0000", 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478309+0000", 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478334+0000", 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5167000000000001e-05 2024-08-30T23:43:16.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478351+0000", 2024-08-30T23:43:16.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7391e-05 2024-08-30T23:43:16.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478360+0000", 2024-08-30T23:43:16.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2579999999999995e-06 2024-08-30T23:43:16.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478413+0000", 2024-08-30T23:43:16.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3780999999999999e-05 2024-08-30T23:43:16.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478541+0000", 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000127358 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:43:16.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.478645+0000", 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001039 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480555+0000", 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019108790000000001 2024-08-30T23:43:16.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480787+0000", 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00023142799999999999 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865599+0000", 2024-08-30T23:43:16.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38481200700000001 2024-08-30T23:43:16.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865636+0000", 2024-08-30T23:43:16.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7416999999999998e-05 2024-08-30T23:43:16.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865684+0000", 2024-08-30T23:43:16.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7215000000000001e-05 2024-08-30T23:43:16.325 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.325 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.325 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:62151 4.1 e337/306 ondisk, result = 0)", 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 498.486787644, 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38498058299999999, 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480720+0000", 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480721+0000", 2024-08-30T23:43:16.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4000000000000001e-07 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.480728+0000", 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7719999999999994e-06 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865502+0000", 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38477343200000003 2024-08-30T23:43:16.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865508+0000", 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7300000000000002e-06 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865589+0000", 2024-08-30T23:43:16.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.119e-05 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865595+0000", 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4529999999999999e-06 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:34:57.865701+0000", 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105266 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:196463 4.1 4:8f28ee96:::benchmark_data_smithi173_74914_object98221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.015837+0000", 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 357.951670159, 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55394642699999996, 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.331 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 196463 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015837+0000", 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015837+0000", 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015839+0000", 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7290000000000001e-06 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015846+0000", 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7479999999999996e-06 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015863+0000", 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7431999999999998e-05 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015872+0000", 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0659999999999993e-06 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016253+0000", 2024-08-30T23:43:16.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000380987 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016363+0000", 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110047 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016488+0000", 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124957 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.020434+0000", 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039450889999999997 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.022249+0000", 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018158779999999999 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569606+0000", 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54735697400000005 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569708+0000", 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010207100000000001 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569784+0000", 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5449000000000002e-05 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:196463 4.1 e381/306 ondisk, result = 0)", 2024-08-30T23:43:16.337 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 357.945530889, 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.547816529, 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7999999999999996e-07 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021980+0000", 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.864e-06 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569398+0000", 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54741768300000004 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569420+0000", 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2169999999999999e-05 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569578+0000", 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001585 2024-08-30T23:43:16.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569596+0000", 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7725999999999999e-05 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569793+0000", 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019710600000000001 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 502/491 [PushOp(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head, version: 408'2535, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head@408'2535, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 102.768313609, 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59550546699999996, 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:43:16.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199198+0000", 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5470000000000001e-06 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199226+0000", 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8228000000000001e-05 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793558+0000", 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59433249099999996 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793576+0000", 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8108999999999999e-05 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793661+0000", 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5122000000000003e-05 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793671+0000", 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.794699+0000", 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001028581 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 509/491 [PushOp(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head, version: 417'3530, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head@417'3530, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 95.628720329999993, 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42136218600000003, 2024-08-30T23:43:16.346 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338788+0000", 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.33e-07 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338818+0000", 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9720999999999999e-05 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758562+0000", 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.419743904 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758588+0000", 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5839000000000001e-05 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758731+0000", 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014296099999999999 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758750+0000", 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9196000000000001e-05 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.760149+0000", 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001399632 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.350 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 510/491 [PushOp(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head, version: 418'3647, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head@418'3647, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 94.838756708000005, 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62881896699999995, 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.351 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128752+0000", 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2219999999999999e-06 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128784+0000", 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1445000000000001e-05 2024-08-30T23:43:16.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756187+0000", 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.627403029 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756208+0000", 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1382999999999998e-05 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756330+0000", 2024-08-30T23:43:16.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121612 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756346+0000", 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5962000000000001e-05 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.757570+0000", 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001224314 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:ef154109:::benchmark_data_smithi173_87639_object59696:head, version: 421'3773, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ef154109:::benchmark_data_smithi173_87639_object59696:head@421'3773, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 93.879231293000004, 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60072882599999999, 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088277+0000", 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4600000000000003e-07 2024-08-30T23:43:16.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088304+0000", 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6968999999999999e-05 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687871+0000", 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59956659400000001 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687891+0000", 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0307999999999999e-05 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688018+0000", 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012653499999999999 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688034+0000", 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6728999999999999e-05 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.689005+0000", 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097084500000000002 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 512/491 [PushOp(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head, version: 430'4003, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head@430'4003, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 89.703678619000002, 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48614643400000002, 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.359 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263830+0000", 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8500000000000001e-07 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263854+0000", 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3934000000000002e-05 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748876+0000", 2024-08-30T23:43:16.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.485022907 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748889+0000", 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2785999999999999e-05 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748973+0000", 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.331e-05 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748983+0000", 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.749975+0000", 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099270099999999991 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.d 542/530 [PushReplyOp(5:bce15829:::benchmark_data_smithi173_87639_object23219:head)])", 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:43:16.363 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.150696840000002, 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44671719199999999, 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:43:16.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816812+0000", 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1260000000000001e-06 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816821+0000", 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5510000000000005e-06 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260989+0000", 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44416764800000003 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260997+0000", 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5750000000000003e-06 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.261062+0000", 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5520999999999998e-05 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.263528+0000", 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024657709999999998 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99435 5.8 5:154d2bda:::benchmark_data_smithi173_87639_object19195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.609215734000003, 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39149939, 2024-08-30T23:43:16.367 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99435 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358293+0000", 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5540000000000001e-06 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358300+0000", 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4250000000000003e-06 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358312+0000", 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2333e-05 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358321+0000", 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4160000000000004e-06 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.383891+0000", 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025570521999999998 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.387506+0000", 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036149160000000001 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.705150+0000", 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31764353000000001 2024-08-30T23:43:16.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.707627+0000", 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024771609999999999 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.712526+0000", 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048989070000000001 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740121+0000", 2024-08-30T23:43:16.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027595773000000001 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740604+0000", 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000482196 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.741372+0000", 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076844900000000004 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.743276+0000", 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019037590000000001 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.749791+0000", 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065154490000000004 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99448 5.8 5:1453319e:::benchmark_data_smithi173_87639_object19208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:43:16.374 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.587441716000001, 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.415457309, 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99448 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380068+0000", 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.637e-06 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380074+0000", 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3500000000000002e-06 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380092+0000", 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7768999999999999e-05 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380101+0000", 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:43:16.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.389216+0000", 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0091151319999999997 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.392658+0000", 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034414720000000001 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.727246+0000", 2024-08-30T23:43:16.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33458855300000001 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.729727+0000", 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024804380000000002 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.733843+0000", 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041159079999999997 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.750905+0000", 2024-08-30T23:43:16.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017061951999999998 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.752106+0000", 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001200916 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.789800+0000", 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037693979000000002 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.790989+0000", 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001189563 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.795523+0000", 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045340220000000004 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.381 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102045 5.8 5:13843233:::benchmark_data_smithi173_87639_object21805:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.756247259999999, 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41801196699999998, 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102045 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.382 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211262+0000", 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7630000000000001e-06 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211268+0000", 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3489999999999997e-06 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211283+0000", 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4834000000000001e-05 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211293+0000", 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0037999999999999e-05 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.235799+0000", 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024505741000000001 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.239265+0000", 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034655129999999999 2024-08-30T23:43:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.592550+0000", 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35328515199999999 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.594844+0000", 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002294164 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.599415+0000", 2024-08-30T23:43:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045711010000000002 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.620491+0000", 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021075930999999999 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.621208+0000", 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071667200000000001 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.387 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.622191+0000", 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098297100000000002 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.623817+0000", 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016259740000000001 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629272+0000", 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054557640000000001 2024-08-30T23:43:16.388 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102049 5.8 5:1445c9f0:::benchmark_data_smithi173_87639_object21809:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.733231273000001, 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40420456799999999, 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.389 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102049 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.390 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234278+0000", 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.012e-06 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234286+0000", 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2799999999999998e-06 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234301+0000", 2024-08-30T23:43:16.391 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5590000000000002e-05 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234310+0000", 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.241629+0000", 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073184249999999999 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.392 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.244833+0000", 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032044450000000002 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.610115+0000", 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.365282208 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.612362+0000", 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022467279999999999 2024-08-30T23:43:16.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.616714+0000", 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043520410000000001 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629561+0000", 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012847305999999999 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.394 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.630448+0000", 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088642099999999998 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.631622+0000", 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011737430000000001 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.633508+0000", 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018869030000000001 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.395 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.638481+0000", 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049724469999999996 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102205 5.8 5:100735ae:::benchmark_data_smithi173_87639_object21965:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.292837165999998, 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41224830200000001, 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:43:16.396 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102205 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674673+0000", 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.063e-06 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674679+0000", 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3380000000000001e-06 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674692+0000", 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3383e-05 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674702+0000", 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4280000000000002e-06 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.691678+0000", 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016976577 2024-08-30T23:43:16.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.694973+0000", 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032944049999999998 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.051542+0000", 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35656930599999997 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.054103+0000", 2024-08-30T23:43:16.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025610979999999999 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.057950+0000", 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003847296 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.078442+0000", 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020491453 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.079182+0000", 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073979099999999995 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.080424+0000", 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00124194 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.081991+0000", 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015673 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:43:16.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.086919+0000", 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049279240000000002 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:43:16.403 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:43:16.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:16 smithi106 ceph-mon[16603]: pgmap v1284: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 18012/135294 objects degraded (13.313%); 15331/135294 objects misplaced (11.332%); 26 MiB/s, 467 objects/s recovering 2024-08-30T23:43:16.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 18217/137361 objects degraded (13.262%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:16.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:16.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:43:16.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:16.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:16.540 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:16 smithi136 podman[107375]: 2024-08-30 23:43:16.249211352 +0000 UTC m=+0.395661128 container init ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux ) 2024-08-30T23:43:16.540 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:16 smithi136 podman[107375]: 2024-08-30 23:43:16.254575063 +0000 UTC m=+0.401024835 container start ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-30T23:43:16.540 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:16 smithi136 ceph-osd[107394]: -- 172.21.15.136:0/77309247 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb78273a40 con 0x55cb78251c00 2024-08-30T23:43:16.540 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:16 smithi136 bash[107375]: ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 2024-08-30T23:43:16.540 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:16 smithi136 systemd[1]: Started Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:43:16.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:16.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:16.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:16.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:16.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:43:16.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:16.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:16.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:17.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:17.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:17.066 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:43:17.106 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:43:17.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:17.165 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1717013975' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:17.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:17.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:43:17.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1717013975' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:17.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:17.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:17.379 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1717013975' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:17.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:17.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:17.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:17.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.595 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:config diff 2024-08-30T23:43:17.660 INFO:teuthology.orchestra.run.smithi136.stderr:config diff get 2024-08-30T23:43:17.661 INFO:teuthology.orchestra.run.smithi136.stderr:config get 2024-08-30T23:43:17.661 INFO:teuthology.orchestra.run.smithi136.stderr:config help [] 2024-08-30T23:43:17.661 INFO:teuthology.orchestra.run.smithi136.stderr:config set ... 2024-08-30T23:43:17.661 INFO:teuthology.orchestra.run.smithi136.stderr:admin_socket: invalid command 2024-08-30T23:43:17.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:17.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:17.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:17.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:17.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:17.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.026 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:18.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:18.039 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[107390]: 2024-08-30T23:43:17.646+0000 7f89e8c2c640 -1 Falling back to public interface 2024-08-30T23:43:18.056 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:18.065 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:18.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:18.155 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:18.226 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1717013975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:18.226 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:18 smithi106 ceph-mon[16603]: osdmap e563: 12 total, 11 up, 12 in 2024-08-30T23:43:18.226 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:18 smithi106 ceph-mon[16603]: pgmap v1286: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 17025/129438 objects degraded (13.153%); 14497/129438 objects misplaced (11.200%); 31 MiB/s, 535 objects/s recovering 2024-08-30T23:43:18.226 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:43:18.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:18.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:18.376 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1717013975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:18.377 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:18 smithi136 ceph-mon[20207]: osdmap e563: 12 total, 11 up, 12 in 2024-08-30T23:43:18.377 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:18 smithi136 ceph-mon[20207]: pgmap v1286: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 17025/129438 objects degraded (13.153%); 14497/129438 objects misplaced (11.200%); 31 MiB/s, 535 objects/s recovering 2024-08-30T23:43:18.377 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:43:18.392 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:18.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:43:18.427 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-08-30T23:43:18.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.558 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:18.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1717013975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:18 smithi173 ceph-mon[20554]: osdmap e563: 12 total, 11 up, 12 in 2024-08-30T23:43:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:18 smithi173 ceph-mon[20554]: pgmap v1286: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 654 op/s; 17025/129438 objects degraded (13.153%); 14497/129438 objects misplaced (11.200%); 31 MiB/s, 535 objects/s recovering 2024-08-30T23:43:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:43:18.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:18.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:18.780 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:18.780 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:18.780 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:18.781 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:18.781 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:18.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:18.813 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:18.813 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:18.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:18.843 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:18.844 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:18.844 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:43:18.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:18.873 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:18.899 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:18.899 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:18.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:18.931 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:18.932 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:18.971 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:18.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:18.992 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:18.993 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:19.014 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:19.014 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:19.040 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:19.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:19.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:19.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:19.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:19.216 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:43:19.216 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:43:19.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:19.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:43:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:43:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:43:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/182127710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:19.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:19.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:19.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:43:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:43:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:43:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/182127710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:43:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:43:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:43:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/182127710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:19.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:19.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:19.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:19.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:19.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:19.894 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:19.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:19.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:43:19.939 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:43:19.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:20.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:43:20.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:20.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:20.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:20.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:20.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:20.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:20.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:20.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:20.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:20 smithi106 ceph-mon[16603]: pgmap v1287: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 16811/127839 objects degraded (13.150%); 13825/127839 objects misplaced (10.814%); 27 MiB/s, 441 objects/s recovering 2024-08-30T23:43:20.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:20.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:20.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:43:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:20 smithi136 ceph-mon[20207]: pgmap v1287: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 16811/127839 objects degraded (13.150%); 13825/127839 objects misplaced (10.814%); 27 MiB/s, 441 objects/s recovering 2024-08-30T23:43:20.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:20 smithi173 ceph-mon[20554]: pgmap v1287: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 16811/127839 objects degraded (13.150%); 13825/127839 objects misplaced (10.814%); 27 MiB/s, 441 objects/s recovering 2024-08-30T23:43:20.832 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.834 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:20.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:20.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:20.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:20.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:21.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:21.101 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:21.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:21.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:21.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:21.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:21.432 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:21.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.507 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:21.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:21.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:21 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:21.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 16811/127839 objects degraded (13.150%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:21.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:21 smithi106 ceph-mon[16603]: osdmap e564: 12 total, 11 up, 12 in 2024-08-30T23:43:21.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:21 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:21.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 16811/127839 objects degraded (13.150%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:21.608 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:21 smithi136 ceph-mon[20207]: osdmap e564: 12 total, 11 up, 12 in 2024-08-30T23:43:21.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:21.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:21.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:21 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 16811/127839 objects degraded (13.150%), 10 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:43:21.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:21 smithi173 ceph-mon[20554]: osdmap e564: 12 total, 11 up, 12 in 2024-08-30T23:43:21.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:21.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:21.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:21.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:21.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:21.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:43:22.004 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:43:22.004 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:43:22.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:22.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:22.039 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[107390]: 2024-08-30T23:43:21.607+0000 7f89e8c2c640 -1 osd.5 446 log_to_monitors true 2024-08-30T23:43:22.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:22.171 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:22.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:22.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:22.348 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:43:22.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:22.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:22.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:22.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:22 smithi106 ceph-mon[16603]: pgmap v1289: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 653 op/s; 16277/122460 objects degraded (13.292%); 13237/122460 objects misplaced (10.809%); 24 MiB/s, 391 objects/s recovering 2024-08-30T23:43:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:22 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:43:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4184832738' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:22.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:22.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:22 smithi136 ceph-mon[20207]: pgmap v1289: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 653 op/s; 16277/122460 objects degraded (13.292%); 13237/122460 objects misplaced (10.809%); 24 MiB/s, 391 objects/s recovering 2024-08-30T23:43:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:22 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:43:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4184832738' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:22.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:22.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:22 smithi173 ceph-mon[20554]: pgmap v1289: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 653 op/s; 16277/122460 objects degraded (13.292%); 13237/122460 objects misplaced (10.809%); 24 MiB/s, 391 objects/s recovering 2024-08-30T23:43:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:22 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:43:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4184832738' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:43:22.831 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:43:22.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:22.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:22.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:22.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:23.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:23.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:23.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:23.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:23.289 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:43:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[107390]: 2024-08-30T23:43:22.929+0000 7f89dc9ce640 -1 osd.5 565 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:43:23.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:23.350 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:23.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:43:23.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:23.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:23.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4184832738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: osdmap e565: 12 total, 11 up, 12 in 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:23 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' 2024-08-30T23:43:23.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:23.693 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:23.694 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:23.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:23.729 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:43:23.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4184832738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: osdmap e565: 12 total, 11 up, 12 in 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:23 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4184832738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: osdmap e565: 12 total, 11 up, 12 in 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:23 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692]' entity='osd.5' 2024-08-30T23:43:23.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:23.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:23.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:23.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:24.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:24.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:24.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:24.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:24.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:24.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:24.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:24.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:24.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:24.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-30T23:43:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692] boot 2024-08-30T23:43:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: osdmap e566: 12 total, 12 up, 12 in 2024-08-30T23:43:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:43:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:24 smithi106 ceph-mon[16603]: pgmap v1292: 47 pgs: 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 12936/117087 objects degraded (11.048%); 10450/117087 objects misplaced (8.925%); 31 MiB/s, 489 objects/s recovering 2024-08-30T23:43:24.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:24.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692] boot 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: osdmap e566: 12 total, 12 up, 12 in 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:24.649 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:24 smithi136 ceph-mon[20207]: pgmap v1292: 47 pgs: 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 12936/117087 objects degraded (11.048%); 10450/117087 objects misplaced (8.925%); 31 MiB/s, 489 objects/s recovering 2024-08-30T23:43:24.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:24.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:24.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:24.783 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:43:24.783 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:43:24.783 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:43:24.783 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: osd.5 [v2:172.21.15.136:6808/1428551692,v1:172.21.15.136:6809/1428551692] boot 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: osdmap e566: 12 total, 12 up, 12 in 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:24 smithi173 ceph-mon[20554]: pgmap v1292: 47 pgs: 1 remapped+peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 12936/117087 objects degraded (11.048%); 10450/117087 objects misplaced (8.925%); 31 MiB/s, 489 objects/s recovering 2024-08-30T23:43:24.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:24.898 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:24.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:25.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:25.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:25.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:25.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:25.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:25.279 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:25.380 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:25.380 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:25.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:25.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:25.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:25.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:25.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:25.406 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:25.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:25.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:25.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:25.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:25.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:43:25.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:25.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:25.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:25 smithi106 ceph-mon[16603]: osdmap e567: 12 total, 12 up, 12 in 2024-08-30T23:43:25.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1247704946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:25.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:25.765 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:25.765 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:25.765 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:25 smithi136 ceph-mon[20207]: osdmap e567: 12 total, 12 up, 12 in 2024-08-30T23:43:25.765 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1247704946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:25.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:25 smithi173 ceph-mon[20554]: osdmap e567: 12 total, 12 up, 12 in 2024-08-30T23:43:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1247704946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:25.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:25.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:25.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:26.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:26.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:26.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:26.185 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:26.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:26.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:26.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:26.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:26.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:26.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:26.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:26.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:26 smithi136 ceph-mon[20207]: osdmap e568: 12 total, 12 up, 12 in 2024-08-30T23:43:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:26 smithi136 ceph-mon[20207]: pgmap v1295: 47 pgs: 2 remapped+peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 9250/115221 objects degraded (8.028%); 11682/115221 objects misplaced (10.139%); 36 MiB/s, 579 objects/s recovering 2024-08-30T23:43:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:26 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 12936/117087 objects degraded (11.048%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:26.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:26 smithi173 ceph-mon[20554]: osdmap e568: 12 total, 12 up, 12 in 2024-08-30T23:43:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:26 smithi173 ceph-mon[20554]: pgmap v1295: 47 pgs: 2 remapped+peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 9250/115221 objects degraded (8.028%); 11682/115221 objects misplaced (10.139%); 36 MiB/s, 579 objects/s recovering 2024-08-30T23:43:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:26 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 12936/117087 objects degraded (11.048%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:26.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:26 smithi106 ceph-mon[16603]: osdmap e568: 12 total, 12 up, 12 in 2024-08-30T23:43:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:26 smithi106 ceph-mon[16603]: pgmap v1295: 47 pgs: 2 remapped+peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 4 active+clean+remapped, 1 active+undersized+degraded, 2 active+recovering, 31 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 603 op/s; 9250/115221 objects degraded (8.028%); 11682/115221 objects misplaced (10.139%); 36 MiB/s, 579 objects/s recovering 2024-08-30T23:43:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:26 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 12936/117087 objects degraded (11.048%), 9 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:26.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:26.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:26.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:27.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:27.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.088 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:27.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:27.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:27.326 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:27.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:27.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:27.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:27.590 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:27.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:27.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:27.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:27.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:27.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:27.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:28.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:28.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:27 smithi136 ceph-mon[20207]: pgmap v1296: 47 pgs: 1 active+recovery_wait+degraded+remapped, 2 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovering, 32 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 10847/111750 objects degraded (9.706%); 9781/111750 objects misplaced (8.753%); 30 MiB/s, 476 objects/s recovering 2024-08-30T23:43:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:27 smithi173 ceph-mon[20554]: pgmap v1296: 47 pgs: 1 active+recovery_wait+degraded+remapped, 2 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovering, 32 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 10847/111750 objects degraded (9.706%); 9781/111750 objects misplaced (8.753%); 30 MiB/s, 476 objects/s recovering 2024-08-30T23:43:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:27 smithi106 ceph-mon[16603]: pgmap v1296: 47 pgs: 1 active+recovery_wait+degraded+remapped, 2 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded, 2 active+recovering, 32 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 10847/111750 objects degraded (9.706%); 9781/111750 objects misplaced (8.753%); 30 MiB/s, 476 objects/s recovering 2024-08-30T23:43:28.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:28.166 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:28.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:28.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:28.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:28.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:28.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:28.560 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:28.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:28.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:28.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:28.821 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:43:28.821 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:43:28.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:43:28.821 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:43:28.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.821 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86177 5.7 5.7fdfa7e7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.613185459999997, 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97022776200000005, 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86177 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731726+0000", 2024-08-30T23:43:28.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.736e-06 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731731+0000", 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999998e-06 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731739+0000", 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6569999999999997e-06 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731746+0000", 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732139+0000", 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039219999999999999 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732147+0000", 2024-08-30T23:43:28.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8910000000000001e-06 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701940+0000", 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979272699999997 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701952+0000", 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1629e-05 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.826 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86218 5.7 5.665a2e07 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612890958999998, 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96998314699999999, 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.827 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86218 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.82e-07 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732023+0000", 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8060000000000001e-06 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732043+0000", 2024-08-30T23:43:28.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9873000000000001e-05 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732050+0000", 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4950000000000002e-06 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732187+0000", 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013689799999999999 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732194+0000", 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6039999999999998e-06 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701991+0000", 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979669899999998 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702002+0000", 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.119e-05 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.831 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86288 5.7 5.4dc78ea7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612850420000001, 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96997772299999996, 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86288 2024-08-30T23:43:28.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732060+0000", 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9799999999999999e-07 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732062+0000", 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2019999999999998e-06 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732068+0000", 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6710000000000002e-06 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.834 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732072+0000", 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9770000000000002e-06 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732233+0000", 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016027700000000001 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732240+0000", 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9990000000000002e-06 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702030+0000", 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979006599999995 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702037+0000", 2024-08-30T23:43:28.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86300 5.7 5.f6fa46a7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612832202000007, 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97000272399999998, 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.837 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86300 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:43:28.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732078+0000", 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4400000000000001e-07 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732079+0000", 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4839999999999999e-06 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.839 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732083+0000", 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2819999999999998e-06 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732094+0000", 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.075e-05 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732281+0000", 2024-08-30T23:43:28.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000186517 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732286+0000", 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.553e-06 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702067+0000", 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96978114199999998 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702080+0000", 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2751999999999999e-05 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86325 5.7 5.e2384ca7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612808482000005, 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97003113200000002, 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.842 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86325 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732102+0000", 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5499999999999998e-07 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732106+0000", 2024-08-30T23:43:28.844 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4460000000000003e-06 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732113+0000", 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7989999999999996e-06 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732119+0000", 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4080000000000003e-06 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732342+0000", 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00022250899999999999 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732347+0000", 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7969999999999997e-06 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702121+0000", 2024-08-30T23:43:28.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96977379200000002 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702132+0000", 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0926e-05 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86376 5.7 5.ad230c87 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612783820999994, 2024-08-30T23:43:28.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97006071599999999, 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86376 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:43:28.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4799999999999999e-07 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732129+0000", 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6529999999999998e-06 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732135+0000", 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2210000000000002e-06 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732139+0000", 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.211e-06 2024-08-30T23:43:28.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732389+0000", 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00024981999999999999 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732396+0000", 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1189999999999999e-06 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702177+0000", 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96978100300000003 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702186+0000", 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2410000000000001e-06 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86452 5.7 5.ed9357e7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.852 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612756093000002, 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97006287000000002, 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86452 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732154+0000", 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4200000000000002e-07 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732162+0000", 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6659999999999997e-06 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732169+0000", 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6839999999999999e-06 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732173+0000", 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1049999999999997e-06 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732425+0000", 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000251508 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732428+0000", 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.202e-06 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702206+0000", 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96977835099999998 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702216+0000", 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0012e-05 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.857 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86465 5.7 5.831aa467 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612730905999996, 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97009033300000003, 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86465 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6199999999999999e-07 2024-08-30T23:43:28.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732180+0000", 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1510000000000001e-06 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732183+0000", 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3289999999999998e-06 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732189+0000", 2024-08-30T23:43:28.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732442+0000", 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025276900000000002 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732445+0000", 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.638e-06 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702257+0000", 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96981182700000002 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702269+0000", 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2242000000000001e-05 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86565 5.7 5.628bf187 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.862 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612716574999993, 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97012332999999995, 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86565 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2499999999999999e-07 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732194+0000", 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.06e-06 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732198+0000", 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2650000000000001e-06 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732202+0000", 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4150000000000001e-06 2024-08-30T23:43:28.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732476+0000", 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027440400000000001 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732482+0000", 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1539999999999999e-06 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702307+0000", 2024-08-30T23:43:28.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982474900000004 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702316+0000", 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0580000000000004e-06 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86603 5.7 5.32d9a587 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612701845000004, 2024-08-30T23:43:28.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97014088799999998, 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86603 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4299999999999999e-07 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732210+0000", 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.655e-06 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732215+0000", 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6910000000000002e-06 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732220+0000", 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6489999999999999e-06 2024-08-30T23:43:28.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732509+0000", 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028865599999999998 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732511+0000", 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7049999999999999e-06 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702338+0000", 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.969826246 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702349+0000", 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0943e-05 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.872 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86625 5.7 5.9d66eb87 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612683533999999, 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97015716600000002, 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86625 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5199999999999998e-07 2024-08-30T23:43:28.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732228+0000", 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0389999999999999e-06 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732234+0000", 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3979999999999998e-06 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732239+0000", 2024-08-30T23:43:28.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3029999999999999e-06 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732541+0000", 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000301582 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732548+0000", 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8709999999999999e-06 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702376+0000", 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982850899999995 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702383+0000", 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1119999999999998e-06 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86638 5.7 5.ef24e247 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.877 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612664557000002, 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97016687199999996, 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86638 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8000000000000001e-07 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732247+0000", 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8500000000000001e-06 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732253+0000", 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5600000000000001e-06 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732257+0000", 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4939999999999997e-06 2024-08-30T23:43:28.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732573+0000", 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031563199999999999 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732576+0000", 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6120000000000001e-06 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702405+0000", 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982966100000001 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702412+0000", 2024-08-30T23:43:28.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6830000000000002e-06 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86682 5.7 5.ad2d3f47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:43:28.884 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612636074999998, 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97017973300000004, 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86682 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.885 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732274+0000", 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2000000000000001e-07 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732275+0000", 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.257e-06 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732279+0000", 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7469999999999999e-06 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732282+0000", 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1089999999999998e-06 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732606+0000", 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032443800000000003 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732613+0000", 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3810000000000004e-06 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702433+0000", 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982075199999995 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702453+0000", 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9729e-05 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.890 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86686 5.7 5.c68ec547 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612624170999993, 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97021215299999997, 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86686 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.891 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732286+0000", 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.28e-07 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732287+0000", 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.065e-06 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732290+0000", 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5499999999999999e-06 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732294+0000", 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2649999999999998e-06 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732637+0000", 2024-08-30T23:43:28.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034266199999999998 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732640+0000", 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5790000000000002e-06 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702487+0000", 2024-08-30T23:43:28.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96984751400000002 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702498+0000", 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.029e-05 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86696 5.7 5.36b4ca47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612610685999996, 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.970319865, 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.897 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86696 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:43:28.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.53e-07 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732301+0000", 2024-08-30T23:43:28.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.618e-06 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732308+0000", 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6399999999999997e-06 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732314+0000", 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2700000000000004e-06 2024-08-30T23:43:28.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732670+0000", 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035588900000000002 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732676+0000", 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1500000000000004e-06 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702608+0000", 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96993175200000004 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702619+0000", 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1192999999999999e-05 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86697 5.7 5.f642fc47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 76.612589829000001, 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97034427400000001, 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86697 2024-08-30T23:43:28.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:43:28.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8500000000000002e-07 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732323+0000", 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4320000000000002e-06 2024-08-30T23:43:28.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732344+0000", 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1539000000000002e-05 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732349+0000", 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0289999999999996e-06 2024-08-30T23:43:28.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732694+0000", 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034466299999999999 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732696+0000", 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6139999999999999e-06 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702655+0000", 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96995884499999996 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702664+0000", 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7669999999999992e-06 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:94589 5.5 5:a5e62c6a:::benchmark_data_smithi173_87639_object14349:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e532)", 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:26.708552+0000", 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 61.636357232999998, 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44460229200000001, 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.909 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 94589 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.708552+0000", 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.708552+0000", 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.708554+0000", 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.697e-06 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.708560+0000", 2024-08-30T23:43:28.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6339999999999998e-06 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.708571+0000", 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.096e-05 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.708578+0000", 2024-08-30T23:43:28.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.418e-06 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.713902+0000", 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0053244579999999998 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:26.717914+0000", 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040118819999999996 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.118153+0000", 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.400238749 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.120255+0000", 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021014419999999998 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.124854+0000", 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045994290000000004 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.138907+0000", 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014053128 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:28.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.139779+0000", 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00087160299999999998 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.143430+0000", 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036513880000000002 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:43:28.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.145222+0000", 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017922840000000001 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.153155+0000", 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0079322200000000002 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:95150 5.5 5:a3112e55:::benchmark_data_smithi173_87639_object14910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:27.757018+0000", 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.587891953000003, 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.43184395399999997, 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 95150 2024-08-30T23:43:28.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.757018+0000", 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.757018+0000", 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.757019+0000", 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5859999999999999e-06 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.757026+0000", 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.764e-06 2024-08-30T23:43:28.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.757037+0000", 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0774999999999999e-05 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.757043+0000", 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3160000000000002e-06 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.875369+0000", 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.118325791 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.878105+0000", 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027363050000000001 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.048039+0000", 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16993359399999999 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.050541+0000", 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025024029999999998 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.054482+0000", 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039405150000000003 2024-08-30T23:43:28.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.066225+0000", 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011743755 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.066645+0000", 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00041933500000000001 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.180596+0000", 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.113951344 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.183906+0000", 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003309789 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.188862+0000", 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049556820000000003 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:95915 5.5 5:a7a5f5b9:::benchmark_data_smithi173_87639_object15675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 59.053999871000002, 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.567720742, 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.926 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 95915 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:43:28.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290911+0000", 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6500000000000001e-06 2024-08-30T23:43:28.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290917+0000", 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3020000000000002e-06 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290930+0000", 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.305e-05 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290939+0000", 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0669999999999998e-06 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.371235+0000", 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080296505000000004 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.374504+0000", 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032687430000000002 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858349+0000", 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48384462 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858630+0000", 2024-08-30T23:43:28.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028180499999999999 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:95915 5.5 5:a7a5f5b9:::benchmark_data_smithi173_87639_object15675:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e532)", 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:29.445221+0000", 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 58.899688791000003, 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44213783000000001, 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:28.932 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 95915 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.445221+0000", 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.445221+0000", 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.445221+0000", 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6100000000000001e-07 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.445223+0000", 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.979e-06 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.445230+0000", 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8170000000000001e-06 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.445235+0000", 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1410000000000004e-06 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.467572+0000", 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.022336577 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.470619+0000", 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030472400000000001 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858950+0000", 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38833145400000002 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.861485+0000", 2024-08-30T23:43:28.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025342049999999999 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.865759+0000", 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042745960000000003 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.876674+0000", 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010914666 2024-08-30T23:43:28.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.877273+0000", 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059944100000000004 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.878921+0000", 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001647967 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.881347+0000", 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024255409999999998 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.887359+0000", 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060117449999999998 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:28.941 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:28.941 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:43:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:28.942 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:28.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:29.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:29.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:29.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:29.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:29.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:29.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:29.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:29.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:29.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:29.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:29.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:29.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:29.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:29.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:29.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:29.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:29.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:29.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:29 smithi136 ceph-mon[20207]: pgmap v1297: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 2 active+recovering, 32 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 435 op/s; 9469/110535 objects degraded (8.567%); 10908/110535 objects misplaced (9.868%); 28 MiB/s, 617 objects/s recovering 2024-08-30T23:43:30.041 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:29 smithi173 ceph-mon[20554]: pgmap v1297: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 2 active+recovering, 32 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 435 op/s; 9469/110535 objects degraded (8.567%); 10908/110535 objects misplaced (9.868%); 28 MiB/s, 617 objects/s recovering 2024-08-30T23:43:30.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:29 smithi106 ceph-mon[16603]: pgmap v1297: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 remapped+peering, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 2 active+recovering, 32 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 435 op/s; 9469/110535 objects degraded (8.567%); 10908/110535 objects misplaced (9.868%); 28 MiB/s, 617 objects/s recovering 2024-08-30T23:43:30.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:30.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:30.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:30.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:30.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:30.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:30.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:30.545 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:43:30.545 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:43:30.545 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:43:30.545 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-08-30T23:43:30.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-30T23:43:30.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:30.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:30.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:30.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:30.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:30.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:30.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:31.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:31.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:31.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:31.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:31.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:31.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:31.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:31.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:31.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:31.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:31.418 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:31.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:31.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:31.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:31.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.686 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:31.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:31.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:31 smithi173 ceph-mon[20554]: osdmap e569: 12 total, 12 up, 12 in 2024-08-30T23:43:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9469/110535 objects degraded (8.567%), 10 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:31 smithi173 ceph-mon[20554]: osdmap e570: 12 total, 12 up, 12 in 2024-08-30T23:43:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:31 smithi173 ceph-mon[20554]: pgmap v1300: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 2 active+recovering, 32 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 11568/105441 objects degraded (10.971%); 9914/105441 objects misplaced (9.402%); 23 MiB/s, 549 objects/s recovering 2024-08-30T23:43:31.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:31.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:31.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:31.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:31 smithi106 ceph-mon[16603]: osdmap e569: 12 total, 12 up, 12 in 2024-08-30T23:43:31.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9469/110535 objects degraded (8.567%), 10 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:31.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:31 smithi106 ceph-mon[16603]: osdmap e570: 12 total, 12 up, 12 in 2024-08-30T23:43:31.979 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:31 smithi106 ceph-mon[16603]: pgmap v1300: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 2 active+recovering, 32 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 11568/105441 objects degraded (10.971%); 9914/105441 objects misplaced (9.402%); 23 MiB/s, 549 objects/s recovering 2024-08-30T23:43:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:31 smithi136 ceph-mon[20207]: osdmap e569: 12 total, 12 up, 12 in 2024-08-30T23:43:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9469/110535 objects degraded (8.567%), 10 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:32.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:31 smithi136 ceph-mon[20207]: osdmap e570: 12 total, 12 up, 12 in 2024-08-30T23:43:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:31 smithi136 ceph-mon[20207]: pgmap v1300: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 2 active+recovering, 32 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 523 op/s; 11568/105441 objects degraded (10.971%); 9914/105441 objects misplaced (9.402%); 23 MiB/s, 549 objects/s recovering 2024-08-30T23:43:32.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:32.089 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:32.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:32.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:32.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:32.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:32.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:32.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:32.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:32.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:32.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:32.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:32.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:32.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:32.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:32.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:32.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:32.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:33.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:32 smithi173 ceph-mon[20554]: osdmap e571: 12 total, 12 up, 12 in 2024-08-30T23:43:33.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:32 smithi106 ceph-mon[16603]: osdmap e571: 12 total, 12 up, 12 in 2024-08-30T23:43:33.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:33.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:33.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:33.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:32 smithi136 ceph-mon[20207]: osdmap e571: 12 total, 12 up, 12 in 2024-08-30T23:43:33.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:33.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:33.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:33.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:33.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:33.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:33.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:33.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:33.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:33.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:33.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:33.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:34.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:34.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:33 smithi173 ceph-mon[20554]: osdmap e572: 12 total, 12 up, 12 in 2024-08-30T23:43:34.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:33 smithi173 ceph-mon[20554]: pgmap v1303: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering, 33 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 9243/100470 objects degraded (9.200%); 9261/100470 objects misplaced (9.218%); 32 MiB/s, 892 objects/s recovering 2024-08-30T23:43:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:33 smithi106 ceph-mon[16603]: osdmap e572: 12 total, 12 up, 12 in 2024-08-30T23:43:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:33 smithi106 ceph-mon[16603]: pgmap v1303: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering, 33 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 9243/100470 objects degraded (9.200%); 9261/100470 objects misplaced (9.218%); 32 MiB/s, 892 objects/s recovering 2024-08-30T23:43:34.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:34.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:34.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:34.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:33 smithi136 ceph-mon[20207]: osdmap e572: 12 total, 12 up, 12 in 2024-08-30T23:43:34.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:33 smithi136 ceph-mon[20207]: pgmap v1303: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering, 33 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 9243/100470 objects degraded (9.200%); 9261/100470 objects misplaced (9.218%); 32 MiB/s, 892 objects/s recovering 2024-08-30T23:43:34.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:34.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:34.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:34.421 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:34.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:34.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:34.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:34.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:34.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:34.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:34.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:34.923 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:34.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:35.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:35.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:34 smithi173 ceph-mon[20554]: osdmap e573: 12 total, 12 up, 12 in 2024-08-30T23:43:35.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:34 smithi106 ceph-mon[16603]: osdmap e573: 12 total, 12 up, 12 in 2024-08-30T23:43:35.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:35.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:35.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:35.297 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:34 smithi136 ceph-mon[20207]: osdmap e573: 12 total, 12 up, 12 in 2024-08-30T23:43:35.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:35.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:35.457 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:35.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:35.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:35.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:35.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:35.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:35.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:35.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:35.989 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:35.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:35 smithi173 ceph-mon[20554]: osdmap e574: 12 total, 12 up, 12 in 2024-08-30T23:43:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:35 smithi173 ceph-mon[20554]: 5.6 scrub starts 2024-08-30T23:43:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:35 smithi173 ceph-mon[20554]: pgmap v1306: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering, 33 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s; 9243/98646 objects degraded (9.370%); 8403/98646 objects misplaced (8.518%); 30 MiB/s, 857 objects/s recovering 2024-08-30T23:43:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:35 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9243/100470 objects degraded (9.200%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:35 smithi173 ceph-mon[20554]: osdmap e575: 12 total, 12 up, 12 in 2024-08-30T23:43:36.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:35 smithi106 ceph-mon[16603]: osdmap e574: 12 total, 12 up, 12 in 2024-08-30T23:43:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:35 smithi106 ceph-mon[16603]: 5.6 scrub starts 2024-08-30T23:43:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:35 smithi106 ceph-mon[16603]: pgmap v1306: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering, 33 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s; 9243/98646 objects degraded (9.370%); 8403/98646 objects misplaced (8.518%); 30 MiB/s, 857 objects/s recovering 2024-08-30T23:43:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:35 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9243/100470 objects degraded (9.200%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:35 smithi106 ceph-mon[16603]: osdmap e575: 12 total, 12 up, 12 in 2024-08-30T23:43:36.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:36.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:36.160 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:35 smithi136 ceph-mon[20207]: osdmap e574: 12 total, 12 up, 12 in 2024-08-30T23:43:36.160 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:35 smithi136 ceph-mon[20207]: 5.6 scrub starts 2024-08-30T23:43:36.160 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:35 smithi136 ceph-mon[20207]: pgmap v1306: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering, 33 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s; 9243/98646 objects degraded (9.370%); 8403/98646 objects misplaced (8.518%); 30 MiB/s, 857 objects/s recovering 2024-08-30T23:43:36.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:35 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9243/100470 objects degraded (9.200%), 9 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:36.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:35 smithi136 ceph-mon[20207]: osdmap e575: 12 total, 12 up, 12 in 2024-08-30T23:43:36.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:36.239 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:36.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:36.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:36.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:36.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:36.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:36.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:36.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:36.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:36.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:36.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:36.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:36.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:36.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:37.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:36 smithi173 ceph-mon[20554]: 5.6 scrub ok 2024-08-30T23:43:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:36 smithi173 ceph-mon[20554]: osdmap e576: 12 total, 12 up, 12 in 2024-08-30T23:43:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:36 smithi106 ceph-mon[16603]: 5.6 scrub ok 2024-08-30T23:43:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:36 smithi106 ceph-mon[16603]: osdmap e576: 12 total, 12 up, 12 in 2024-08-30T23:43:37.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:37.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:37.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:37.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:43:37.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:37.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:36 smithi136 ceph-mon[20207]: 5.6 scrub ok 2024-08-30T23:43:37.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:36 smithi136 ceph-mon[20207]: osdmap e576: 12 total, 12 up, 12 in 2024-08-30T23:43:37.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:37.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:37.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:37.545 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:37.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:37.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:37.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:37.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:37.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:37.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:37.947 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:37.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:38.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:38.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:38.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:37 smithi106 ceph-mon[16603]: pgmap v1309: 47 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovering, 35 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8097/94176 objects degraded (8.598%); 7184/94176 objects misplaced (7.628%); 29 MiB/s, 529 objects/s recovering 2024-08-30T23:43:38.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:38.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:38.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:37 smithi136 ceph-mon[20207]: pgmap v1309: 47 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovering, 35 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8097/94176 objects degraded (8.598%); 7184/94176 objects misplaced (7.628%); 29 MiB/s, 529 objects/s recovering 2024-08-30T23:43:38.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:37 smithi173 ceph-mon[20554]: pgmap v1309: 47 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 1 active+recovering, 35 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8097/94176 objects degraded (8.598%); 7184/94176 objects misplaced (7.628%); 29 MiB/s, 529 objects/s recovering 2024-08-30T23:43:38.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:38.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:38.465 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:38.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:38.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:38.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:38.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:38.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:38.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:38.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:38.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:38.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:38.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:39.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:38 smithi106 ceph-mon[16603]: 5.5 scrub starts 2024-08-30T23:43:39.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:39.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:39.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:38 smithi136 ceph-mon[20207]: 5.5 scrub starts 2024-08-30T23:43:39.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:43:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:38 smithi173 ceph-mon[20554]: 5.5 scrub starts 2024-08-30T23:43:39.328 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:39.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:39.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:39.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:39.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:39.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:39.688 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:39.726 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:43:39.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:39.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:39.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:39.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:39.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:40.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:40.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:40.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:39 smithi106 ceph-mon[16603]: 5.5 scrub ok 2024-08-30T23:43:40.102 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:39 smithi106 ceph-mon[16603]: pgmap v1310: 47 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering, 35 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8097/92898 objects degraded (8.716%); 7184/92898 objects misplaced (7.733%); 20 MiB/s, 399 objects/s recovering 2024-08-30T23:43:40.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:39 smithi136 ceph-mon[20207]: 5.5 scrub ok 2024-08-30T23:43:40.151 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:39 smithi136 ceph-mon[20207]: pgmap v1310: 47 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering, 35 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8097/92898 objects degraded (8.716%); 7184/92898 objects misplaced (7.733%); 20 MiB/s, 399 objects/s recovering 2024-08-30T23:43:40.198 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:40.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:40.312 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:43:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:43:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:132477 5.6 5:63fb6920:::benchmark_data_smithi173_87639_object52237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-08-30T23:43:40.312 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:40.309741+0000", 2024-08-30T23:43:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0010825380000000001, 2024-08-30T23:43:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011105329999999999, 2024-08-30T23:43:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:43:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:43:40.313 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:43:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:43:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:43:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:43:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 132477 2024-08-30T23:43:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.314 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:43:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:43:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309741+0000", 2024-08-30T23:43:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:43:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:43:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309741+0000", 2024-08-30T23:43:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:43:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:43:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309743+0000", 2024-08-30T23:43:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.68e-06 2024-08-30T23:43:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:43:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309752+0000", 2024-08-30T23:43:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8410000000000006e-06 2024-08-30T23:43:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:43:40.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309764+0000", 2024-08-30T23:43:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2187999999999999e-05 2024-08-30T23:43:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:43:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309771+0000", 2024-08-30T23:43:40.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.25e-06 2024-08-30T23:43:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:43:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309833+0000", 2024-08-30T23:43:40.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1394000000000002e-05 2024-08-30T23:43:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:43:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.309939+0000", 2024-08-30T23:43:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105828 2024-08-30T23:43:40.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:43:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:43:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:43:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:40.310052+0000", 2024-08-30T23:43:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011333200000000001 2024-08-30T23:43:40.322 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:43:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:43:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:43:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:43:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:43:40.323 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-08-30T23:43:40.323 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:43:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:39 smithi173 ceph-mon[20554]: 5.5 scrub ok 2024-08-30T23:43:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:39 smithi173 ceph-mon[20554]: pgmap v1310: 47 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering, 35 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8097/92898 objects degraded (8.716%); 7184/92898 objects misplaced (7.733%); 20 MiB/s, 399 objects/s recovering 2024-08-30T23:43:40.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:40.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:40.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:40.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:40.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:40.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:40.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:40.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:40.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:40.897 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:40.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:40.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:41.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:41.090 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:40 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8097/92898 objects degraded (8.716%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:41.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:41.165 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:41.268 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:40 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8097/92898 objects degraded (8.716%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:41.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:40 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8097/92898 objects degraded (8.716%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:41.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:41.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:41.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:41.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:43:41.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:41.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:41.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:41.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:41.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:41.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:41.891 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:43:41.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:41.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:41.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:41 smithi173 ceph-mon[20554]: osdmap e577: 12 total, 12 up, 12 in 2024-08-30T23:43:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:41 smithi173 ceph-mon[20554]: pgmap v1312: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering, 36 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6373/86505 objects degraded (7.367%); 6371/86505 objects misplaced (7.365%); 3.5 MiB/s, 153 objects/s recovering 2024-08-30T23:43:42.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:41 smithi106 ceph-mon[16603]: osdmap e577: 12 total, 12 up, 12 in 2024-08-30T23:43:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:41 smithi106 ceph-mon[16603]: pgmap v1312: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering, 36 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6373/86505 objects degraded (7.367%); 6371/86505 objects misplaced (7.365%); 3.5 MiB/s, 153 objects/s recovering 2024-08-30T23:43:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:42.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:42.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:42.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:42.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:41 smithi136 ceph-mon[20207]: osdmap e577: 12 total, 12 up, 12 in 2024-08-30T23:43:42.174 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:41 smithi136 ceph-mon[20207]: pgmap v1312: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering, 36 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6373/86505 objects degraded (7.367%); 6371/86505 objects misplaced (7.365%); 3.5 MiB/s, 153 objects/s recovering 2024-08-30T23:43:42.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:42.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:42.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:42.359 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:42.386 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:42.405 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:42.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:42.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:42.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:42.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:42.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:42.776 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:42.869 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:43:42.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:42.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:42.905 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:42.914 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2003591549' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:42.914 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:42 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:43.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:43.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:43.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:43.154 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2003591549' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:43.154 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:42 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:43.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:43.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:43.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2003591549' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:42 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:43:43.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:43.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:43.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:43.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:43.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:43.606 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:43:43.606 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:43:43.606 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:43:43.607 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:43:43.607 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:43:43.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:43.675 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:43.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:43.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:43.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:43.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:43.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:43:43.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:43.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:43.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:43.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:43.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:43.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:43.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:43.714 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:43.714 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:43.736 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:43.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:43.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:43.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:43.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:43.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:43.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:43.969 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:43 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:43:43.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:43 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:43.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:43 smithi136 ceph-mon[20207]: osdmap e578: 12 total, 12 up, 12 in 2024-08-30T23:43:43.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3393784142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:43.970 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:43 smithi136 ceph-mon[20207]: pgmap v1314: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering, 36 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6373/80586 objects degraded (7.908%); 5342/80586 objects misplaced (6.629%); 6.2 MiB/s, 97 objects/s recovering 2024-08-30T23:43:44.037 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.065 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:44.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:43 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:43:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:43 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:43 smithi106 ceph-mon[16603]: osdmap e578: 12 total, 12 up, 12 in 2024-08-30T23:43:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3393784142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:43 smithi106 ceph-mon[16603]: pgmap v1314: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering, 36 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6373/80586 objects degraded (7.908%); 5342/80586 objects misplaced (6.629%); 6.2 MiB/s, 97 objects/s recovering 2024-08-30T23:43:44.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:44.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:44.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:44.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:43:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:43 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:43:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:43 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:43:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:43 smithi173 ceph-mon[20554]: osdmap e578: 12 total, 12 up, 12 in 2024-08-30T23:43:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3393784142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:43 smithi173 ceph-mon[20554]: pgmap v1314: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 1 active+recovering, 36 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6373/80586 objects degraded (7.908%); 5342/80586 objects misplaced (6.629%); 6.2 MiB/s, 97 objects/s recovering 2024-08-30T23:43:44.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:44.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:44.461 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:44.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:44.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:44.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:44.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:44.740 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:43:44.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.858 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:44.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:44.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:44.986 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:44 smithi106 ceph-mon[16603]: osdmap e579: 12 total, 12 up, 12 in 2024-08-30T23:43:44.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:44.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:45.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:45.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:45.158 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:44 smithi136 ceph-mon[20207]: osdmap e579: 12 total, 12 up, 12 in 2024-08-30T23:43:45.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:45.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:45.314 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:43:45.314 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:43:45.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:44 smithi173 ceph-mon[20554]: osdmap e579: 12 total, 12 up, 12 in 2024-08-30T23:43:45.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:45.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:45.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:45.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:45.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:45.666 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:45.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:45.780 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-30T23:43:45.797 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:45.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:45.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:45.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:46.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:46.062 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:46.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:43:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2631744609' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:43:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:45 smithi106 ceph-mon[16603]: pgmap v1316: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 6013/77535 objects degraded (7.755%); 4483/77535 objects misplaced (5.782%); 9.1 MiB/s, 286 objects/s recovering 2024-08-30T23:43:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:45 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6373/80586 objects degraded (7.908%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2631744609' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:43:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:45 smithi106 ceph-mon[16603]: osdmap e580: 12 total, 12 up, 12 in 2024-08-30T23:43:46.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:46.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:46.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2631744609' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:43:46.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:45 smithi136 ceph-mon[20207]: pgmap v1316: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 6013/77535 objects degraded (7.755%); 4483/77535 objects misplaced (5.782%); 9.1 MiB/s, 286 objects/s recovering 2024-08-30T23:43:46.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:45 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6373/80586 objects degraded (7.908%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:46.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2631744609' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:43:46.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:45 smithi136 ceph-mon[20207]: osdmap e580: 12 total, 12 up, 12 in 2024-08-30T23:43:46.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:46.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2631744609' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:43:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:45 smithi173 ceph-mon[20554]: pgmap v1316: 47 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 37 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 149 op/s; 6013/77535 objects degraded (7.755%); 4483/77535 objects misplaced (5.782%); 9.1 MiB/s, 286 objects/s recovering 2024-08-30T23:43:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:45 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6373/80586 objects degraded (7.908%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2631744609' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:43:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:45 smithi173 ceph-mon[20554]: osdmap e580: 12 total, 12 up, 12 in 2024-08-30T23:43:46.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:46.449 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:46.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:46.574 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:46.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:46.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:46.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:43:46.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:46.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:46.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:46.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:46.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:47.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:47.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:47.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:47.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:47.213 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:47.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:47.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:47.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:47.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:47.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:43:47.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:47.628 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:47.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:47.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:47.785 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:47.834 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:47.835 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:47.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:47.865 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:47.865 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:47.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:47.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:47.889 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:47.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:47.915 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:47.915 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:47.930 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:47 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:47.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:47 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:47.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:47 smithi106 ceph-mon[16603]: osdmap e581: 12 total, 12 up, 12 in 2024-08-30T23:43:47.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:47 smithi106 ceph-mon[16603]: pgmap v1319: 47 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 6012/70776 objects degraded (8.494%); 3783/70776 objects misplaced (5.345%); 17 MiB/s, 643 objects/s recovering 2024-08-30T23:43:47.931 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4148496791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:47.943 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:47.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:47.971 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:43:47.972 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:43:47.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:48.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:48.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:48.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:48.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:48.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:48.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:47 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:47 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:47 smithi136 ceph-mon[20207]: osdmap e581: 12 total, 12 up, 12 in 2024-08-30T23:43:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:47 smithi136 ceph-mon[20207]: pgmap v1319: 47 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 6012/70776 objects degraded (8.494%); 3783/70776 objects misplaced (5.345%); 17 MiB/s, 643 objects/s recovering 2024-08-30T23:43:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4148496791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:48.318 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:138712 5.b e581/530 5:d868b7ba:::benchmark_data_smithi173_87639_object58472:head v 581'8850, mlcod=436'4762)", 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:48.318140+0000", 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00033758699999999998, 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035607, 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:43:48.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318140+0000", 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318140+0000", 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318140+0000", 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1800000000000005e-07 2024-08-30T23:43:48.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318146+0000", 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7660000000000004e-06 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318183+0000", 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6520999999999998e-05 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318190+0000", 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1930000000000004e-06 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318246+0000", 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6316000000000001e-05 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:48.318270+0000", 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3859999999999999e-05 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:48.322 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:43:48.323 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:48.323 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:43:48.323 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:43:48.323 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-08-30T23:43:48.323 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:43:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:47 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:47 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:47 smithi173 ceph-mon[20554]: osdmap e581: 12 total, 12 up, 12 in 2024-08-30T23:43:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:47 smithi173 ceph-mon[20554]: pgmap v1319: 47 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 831 op/s; 6012/70776 objects degraded (8.494%); 3783/70776 objects misplaced (5.345%); 17 MiB/s, 643 objects/s recovering 2024-08-30T23:43:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4148496791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:48.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:48.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:48.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:48.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:48.550 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:48.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:48.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:48.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:48.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:48.801 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:48.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:48.830 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:48.830 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:48.859 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:48.859 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:48.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:48.896 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:48.896 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:48.920 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:48.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:48.926 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:48.927 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:48.955 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:48.955 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:49.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:49.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:49.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:49.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.183 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:49.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:49.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:49.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:49.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:49.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:49.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:49.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:49.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:49.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:49.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:49.848 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:49.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:49.971 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:49.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:50.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:50.101 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:50.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:50.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:50.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:50.360 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:50.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:50.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:50.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:50.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:50.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:50.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:50 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:50 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:50 smithi173 ceph-mon[20554]: osdmap e582: 12 total, 12 up, 12 in 2024-08-30T23:43:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:50 smithi173 ceph-mon[20554]: pgmap v1321: 47 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 6032/69021 objects degraded (8.739%); 3783/69021 objects misplaced (5.481%); 14 MiB/s, 537 objects/s recovering 2024-08-30T23:43:50.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:50.851 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:50 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:50.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:50 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:50.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:50 smithi106 ceph-mon[16603]: osdmap e582: 12 total, 12 up, 12 in 2024-08-30T23:43:50.852 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:50 smithi106 ceph-mon[16603]: pgmap v1321: 47 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 6032/69021 objects degraded (8.739%); 3783/69021 objects misplaced (5.481%); 14 MiB/s, 537 objects/s recovering 2024-08-30T23:43:50.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:50.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:50.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:51.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:51.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:50 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:50 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:50 smithi136 ceph-mon[20207]: osdmap e582: 12 total, 12 up, 12 in 2024-08-30T23:43:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:50 smithi136 ceph-mon[20207]: pgmap v1321: 47 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+degraded+remapped, 38 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 672 op/s; 6032/69021 objects degraded (8.739%); 3783/69021 objects misplaced (5.481%); 14 MiB/s, 537 objects/s recovering 2024-08-30T23:43:51.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:51.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:51.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:51.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:51.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:51.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:51.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:51.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:51.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:51.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:51.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:51.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:51 smithi173 ceph-mon[20554]: osdmap e583: 12 total, 12 up, 12 in 2024-08-30T23:43:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6032/69021 objects degraded (8.739%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:51 smithi173 ceph-mon[20554]: osdmap e584: 12 total, 12 up, 12 in 2024-08-30T23:43:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:51 smithi173 ceph-mon[20554]: 5.7 scrub starts 2024-08-30T23:43:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:51 smithi106 ceph-mon[16603]: osdmap e583: 12 total, 12 up, 12 in 2024-08-30T23:43:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6032/69021 objects degraded (8.739%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:51 smithi106 ceph-mon[16603]: osdmap e584: 12 total, 12 up, 12 in 2024-08-30T23:43:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:51 smithi106 ceph-mon[16603]: 5.7 scrub starts 2024-08-30T23:43:51.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:51.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:51.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:52.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:52.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:51 smithi136 ceph-mon[20207]: osdmap e583: 12 total, 12 up, 12 in 2024-08-30T23:43:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6032/69021 objects degraded (8.739%), 8 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:52.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:51 smithi136 ceph-mon[20207]: osdmap e584: 12 total, 12 up, 12 in 2024-08-30T23:43:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:51 smithi136 ceph-mon[20207]: 5.7 scrub starts 2024-08-30T23:43:52.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:52.152 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:43:52.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:43:52.152 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.28076089959611494 2024-08-30T23:43:52.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 0.28076089959611494 2024-08-30T23:43:52.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:52.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:52.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:52.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:52.422 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:52.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:52.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:52.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:52.672 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:52.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:52.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:52.785 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: 5.7 scrub ok 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: pgmap v1324: 47 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 6009/62556 objects degraded (9.606%); 3166/62556 objects misplaced (5.061%); 13 MiB/s, 607 objects/s recovering 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: osdmap e585: 12 total, 12 up, 12 in 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: 5.a scrub starts 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1392779443' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: 5.14 scrub starts 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: 5.14 scrub ok 2024-08-30T23:43:52.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:52 smithi106 ceph-mon[16603]: 5.a scrub ok 2024-08-30T23:43:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.801 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:52.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: 5.7 scrub ok 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: pgmap v1324: 47 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 6009/62556 objects degraded (9.606%); 3166/62556 objects misplaced (5.061%); 13 MiB/s, 607 objects/s recovering 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: osdmap e585: 12 total, 12 up, 12 in 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: 5.a scrub starts 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1392779443' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: 5.14 scrub starts 2024-08-30T23:43:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:43:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:43:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: 5.14 scrub ok 2024-08-30T23:43:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:52 smithi173 ceph-mon[20554]: 5.a scrub ok 2024-08-30T23:43:52.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:52.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:52.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:53.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: 5.7 scrub ok 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: pgmap v1324: 47 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 828 op/s; 6009/62556 objects degraded (9.606%); 3166/62556 objects misplaced (5.061%); 13 MiB/s, 607 objects/s recovering 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: osdmap e585: 12 total, 12 up, 12 in 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: 5.a scrub starts 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1392779443' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: 5.14 scrub starts 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: 5.14 scrub ok 2024-08-30T23:43:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:52 smithi136 ceph-mon[20207]: 5.a scrub ok 2024-08-30T23:43:53.063 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:53.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:53.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:53.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:53.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:53.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:53.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:53.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:43:53.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:43:53.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:43:53.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.720 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:53.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:53.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:53 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1392779443' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:43:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:53 smithi173 ceph-mon[20554]: osdmap e586: 12 total, 12 up, 12 in 2024-08-30T23:43:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:53.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:53 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1392779443' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:43:53.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:53 smithi106 ceph-mon[16603]: osdmap e586: 12 total, 12 up, 12 in 2024-08-30T23:43:53.940 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:53 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:43:53.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1392779443' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:43:53.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:53 smithi136 ceph-mon[20207]: osdmap e586: 12 total, 12 up, 12 in 2024-08-30T23:43:53.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:53.972 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:53.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:54.016 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:54.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:54.095 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:54.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:54.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:54.323 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:54.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:54.446 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:54.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:54.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:54.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:54 smithi106 ceph-mon[16603]: pgmap v1327: 47 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 990 op/s; 4127/57138 objects degraded (7.223%); 3117/57138 objects misplaced (5.455%); 16 MiB/s, 868 objects/s recovering 2024-08-30T23:43:54.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:54 smithi106 ceph-mon[16603]: osdmap e587: 12 total, 12 up, 12 in 2024-08-30T23:43:54.647 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:54.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:54.715 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:54.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:54 smithi173 ceph-mon[20554]: pgmap v1327: 47 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 990 op/s; 4127/57138 objects degraded (7.223%); 3117/57138 objects misplaced (5.455%); 16 MiB/s, 868 objects/s recovering 2024-08-30T23:43:54.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:54 smithi173 ceph-mon[20554]: osdmap e587: 12 total, 12 up, 12 in 2024-08-30T23:43:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.841 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:54.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:54.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:54.979 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:54.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:55.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:43:55.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:54 smithi136 ceph-mon[20207]: pgmap v1327: 47 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 990 op/s; 4127/57138 objects degraded (7.223%); 3117/57138 objects misplaced (5.455%); 16 MiB/s, 868 objects/s recovering 2024-08-30T23:43:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:54 smithi136 ceph-mon[20207]: osdmap e587: 12 total, 12 up, 12 in 2024-08-30T23:43:55.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.116 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:55.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:55.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:43:55.233 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:43:55.233 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:43:55.233 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:43:55.233 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:43:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.254 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:55.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:55.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:55.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:55.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:55.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:55.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:55.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:55.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:55.770 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:55.793 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.0 to 0.280761 (47df) 2024-08-30T23:43:55.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:55.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:55.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:56.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:56.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:43:56.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:56.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:55 smithi106 ceph-mon[16603]: osdmap e588: 12 total, 12 up, 12 in 2024-08-30T23:43:56.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2455578624' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.28076089959611494} : dispatch 2024-08-30T23:43:56.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:55 smithi106 ceph-mon[16603]: pgmap v1330: 47 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 4127/55056 objects degraded (7.496%); 2696/55056 objects misplaced (4.897%); 13 MiB/s, 536 objects/s recovering 2024-08-30T23:43:56.060 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:55 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4127/57138 objects degraded (7.223%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:55 smithi173 ceph-mon[20554]: osdmap e588: 12 total, 12 up, 12 in 2024-08-30T23:43:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2455578624' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.28076089959611494} : dispatch 2024-08-30T23:43:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:55 smithi173 ceph-mon[20554]: pgmap v1330: 47 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 4127/55056 objects degraded (7.496%); 2696/55056 objects misplaced (4.897%); 13 MiB/s, 536 objects/s recovering 2024-08-30T23:43:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:55 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4127/57138 objects degraded (7.223%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:56.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:56.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:56.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:43:56.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:56.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:55 smithi136 ceph-mon[20207]: osdmap e588: 12 total, 12 up, 12 in 2024-08-30T23:43:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2455578624' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 0.28076089959611494} : dispatch 2024-08-30T23:43:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:55 smithi136 ceph-mon[20207]: pgmap v1330: 47 pgs: 1 active+recovering+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 39 active+clean; 1.1 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 624 op/s; 4127/55056 objects degraded (7.496%); 2696/55056 objects misplaced (4.897%); 13 MiB/s, 536 objects/s recovering 2024-08-30T23:43:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:55 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4127/57138 objects degraded (7.223%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:43:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:56.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:56.418 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:56.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:56.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:56.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:56.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:56.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:56.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:56.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2455578624' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.28076089959611494}]': finished 2024-08-30T23:43:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:56 smithi106 ceph-mon[16603]: osdmap e589: 12 total, 12 up, 12 in 2024-08-30T23:43:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:56 smithi106 ceph-mon[16603]: osdmap e590: 12 total, 12 up, 12 in 2024-08-30T23:43:56.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:56.917 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:56.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:57.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:57.037 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:57.049 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:57.064 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2455578624' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.28076089959611494}]': finished 2024-08-30T23:43:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:56 smithi173 ceph-mon[20554]: osdmap e589: 12 total, 12 up, 12 in 2024-08-30T23:43:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:56 smithi173 ceph-mon[20554]: osdmap e590: 12 total, 12 up, 12 in 2024-08-30T23:43:57.074 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:43:57.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:57.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:57.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:57.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2455578624' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.28076089959611494}]': finished 2024-08-30T23:43:57.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:56 smithi136 ceph-mon[20207]: osdmap e589: 12 total, 12 up, 12 in 2024-08-30T23:43:57.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:56 smithi136 ceph-mon[20207]: osdmap e590: 12 total, 12 up, 12 in 2024-08-30T23:43:57.304 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:57.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:57.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:57.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:57.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.553 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:57.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:43:57.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:57.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:43:57.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:43:57.804 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:43:57.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:43:57.927 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:57.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:43:58.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:58.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:57 smithi106 ceph-mon[16603]: pgmap v1333: 47 pgs: 1 active+remapped, 1 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2103/49569 objects degraded (4.243%); 2311/49569 objects misplaced (4.662%) 2024-08-30T23:43:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:57 smithi106 ceph-mon[16603]: osdmap e591: 12 total, 12 up, 12 in 2024-08-30T23:43:58.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:58.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:58.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:57 smithi136 ceph-mon[20207]: pgmap v1333: 47 pgs: 1 active+remapped, 1 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2103/49569 objects degraded (4.243%); 2311/49569 objects misplaced (4.662%) 2024-08-30T23:43:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:57 smithi136 ceph-mon[20207]: osdmap e591: 12 total, 12 up, 12 in 2024-08-30T23:43:58.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:57 smithi173 ceph-mon[20554]: pgmap v1333: 47 pgs: 1 active+remapped, 1 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 1.0 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2103/49569 objects degraded (4.243%); 2311/49569 objects misplaced (4.662%) 2024-08-30T23:43:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:57 smithi173 ceph-mon[20554]: osdmap e591: 12 total, 12 up, 12 in 2024-08-30T23:43:58.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:43:58.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:43:58.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:58.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:58.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:58.694 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:58.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:58.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:58.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:58.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:58.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:59.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:59.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:43:59.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:43:59.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:43:59.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:43:59.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:43:59.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:43:59.076 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:43:59.077 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:43:59.077 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:43:59.077 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:43:59.077 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:43:59.077 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:43:59.077 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:43:59.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:43:59.100 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:43:59.101 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:43:59.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1258870875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:59.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:58 smithi106 ceph-mon[16603]: osdmap e592: 12 total, 12 up, 12 in 2024-08-30T23:43:59.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:43:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.208 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:59.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:43:59.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1258870875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:59.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:58 smithi136 ceph-mon[20207]: osdmap e592: 12 total, 12 up, 12 in 2024-08-30T23:43:59.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:43:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1258870875' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:43:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:58 smithi173 ceph-mon[20554]: osdmap e592: 12 total, 12 up, 12 in 2024-08-30T23:43:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:59.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:43:59.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:43:59.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:43:59.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:43:59.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:43:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:43:59.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:43:59.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:43:59.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:43:59.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:43:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:43:59.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:43:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:43:59.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:43:59.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:59 smithi136 ceph-mon[20207]: 5.18 scrub starts 2024-08-30T23:44:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:43:59 smithi136 ceph-mon[20207]: pgmap v1336: 47 pgs: 1 active+remapped, 1 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 1013 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2234/48603 objects degraded (4.596%); 2311/48603 objects misplaced (4.755%) 2024-08-30T23:44:00.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:00.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:00.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:00.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:59 smithi173 ceph-mon[20554]: 5.18 scrub starts 2024-08-30T23:44:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:43:59 smithi173 ceph-mon[20554]: pgmap v1336: 47 pgs: 1 active+remapped, 1 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 1013 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2234/48603 objects degraded (4.596%); 2311/48603 objects misplaced (4.755%) 2024-08-30T23:44:00.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:59 smithi106 ceph-mon[16603]: 5.18 scrub starts 2024-08-30T23:44:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:43:59 smithi106 ceph-mon[16603]: pgmap v1336: 47 pgs: 1 active+remapped, 1 remapped+peering, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 1013 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2234/48603 objects degraded (4.596%); 2311/48603 objects misplaced (4.755%) 2024-08-30T23:44:00.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:00.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:00.510 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:00.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:00.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:00.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:00.771 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:00.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:00.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:00.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:00.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:00.907 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:00 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2234/48603 objects degraded (4.596%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:44:01.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:01.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.035 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:01.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:01.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:01.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:01.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:01.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:00 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2234/48603 objects degraded (4.596%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:44:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:00 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2234/48603 objects degraded (4.596%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:44:01.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:01.417 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:01.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:01.547 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:01.642 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:01.642 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:01.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:01.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:01.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:01.801 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:01.868 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:44:01.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:01.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:01.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:02.007 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:44:02.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:44:02.007 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:44:02.007 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-30T23:44:02.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5 2024-08-30T23:44:02.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:02.091 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:02.143 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:01 smithi106 ceph-mon[16603]: pgmap v1337: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 922 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2234/44235 objects degraded (5.050%); 2673/44235 objects misplaced (6.043%) 2024-08-30T23:44:02.143 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/15173770' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:02.143 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:02.143 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:02.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:02.217 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:02.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:02 smithi136 systemd[1]: Stopping Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:44:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:01 smithi136 ceph-mon[20207]: pgmap v1337: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 922 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2234/44235 objects degraded (5.050%); 2673/44235 objects misplaced (6.043%) 2024-08-30T23:44:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/15173770' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:02.319 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:01 smithi173 ceph-mon[20554]: pgmap v1337: 47 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 922 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2234/44235 objects degraded (5.050%); 2673/44235 objects misplaced (6.043%) 2024-08-30T23:44:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/15173770' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:02.350 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:02.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:02.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:02.583 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:02.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:02.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:02.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:02.789 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[107390]: 2024-08-30T23:44:02.484+0000 7f89e5bc4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-30T23:44:02.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[107390]: 2024-08-30T23:44:02.484+0000 7f89e5bc4640 -1 osd.5 593 *** Got signal Terminated *** 2024-08-30T23:44:02.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[107390]: 2024-08-30T23:44:02.484+0000 7f89e5bc4640 -1 osd.5 593 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-30T23:44:02.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:02.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:02.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:02.999 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:03.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:03.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:02 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/15173770' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:02 smithi106 ceph-mon[16603]: osdmap e593: 12 total, 12 up, 12 in 2024-08-30T23:44:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:02 smithi106 ceph-mon[16603]: osd.5 marked itself down and dead 2024-08-30T23:44:03.124 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:03.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.248 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:03.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/15173770' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-mon[20207]: osdmap e593: 12 total, 12 up, 12 in 2024-08-30T23:44:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:02 smithi136 ceph-mon[20207]: osd.5 marked itself down and dead 2024-08-30T23:44:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:02 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/15173770' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:02 smithi173 ceph-mon[20554]: osdmap e593: 12 total, 12 up, 12 in 2024-08-30T23:44:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:02 smithi173 ceph-mon[20554]: osd.5 marked itself down and dead 2024-08-30T23:44:03.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:03.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:03.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:03.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:03.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:03.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:03.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:44:03.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:03.742 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:03.789 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:03 smithi136 podman[110673]: 2024-08-30 23:44:03.368722614 +0000 UTC m=+1.020488186 container died ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-30T23:44:03.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:03.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:03.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:03.997 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:03.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:04.000 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:04.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:03 smithi173 ceph-mon[20554]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:44:04.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:03 smithi173 ceph-mon[20554]: osdmap e594: 12 total, 11 up, 12 in 2024-08-30T23:44:04.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:03 smithi173 ceph-mon[20554]: pgmap v1340: 47 pgs: 1 remapped+peering, 1 stale+active+clean, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 821 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 1840/39369 objects degraded (4.674%); 1283/39369 objects misplaced (3.259%); 6.0 MiB/s, 428 objects/s recovering 2024-08-30T23:44:04.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:03 smithi106 ceph-mon[16603]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:44:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:03 smithi106 ceph-mon[16603]: osdmap e594: 12 total, 11 up, 12 in 2024-08-30T23:44:04.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:03 smithi106 ceph-mon[16603]: pgmap v1340: 47 pgs: 1 remapped+peering, 1 stale+active+clean, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 821 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 1840/39369 objects degraded (4.674%); 1283/39369 objects misplaced (3.259%); 6.0 MiB/s, 428 objects/s recovering 2024-08-30T23:44:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:04.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:04.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:44:04.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:04.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:03 smithi136 ceph-mon[20207]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:44:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:03 smithi136 ceph-mon[20207]: osdmap e594: 12 total, 11 up, 12 in 2024-08-30T23:44:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:03 smithi136 ceph-mon[20207]: pgmap v1340: 47 pgs: 1 remapped+peering, 1 stale+active+clean, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 821 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 1840/39369 objects degraded (4.674%); 1283/39369 objects misplaced (3.259%); 6.0 MiB/s, 428 objects/s recovering 2024-08-30T23:44:04.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:04.381 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:04.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:04.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:04.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:04.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:04.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:04.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:04.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:04.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:04.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:04.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:05.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:05.017 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:05.017 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:05.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:05.129 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:05.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:44:05.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:05.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:04 smithi106 ceph-mon[16603]: osdmap e595: 12 total, 11 up, 12 in 2024-08-30T23:44:05.235 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:04 smithi106 ceph-mon[16603]: 5.18 scrub starts 2024-08-30T23:44:05.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:04 smithi136 ceph-mon[20207]: osdmap e595: 12 total, 11 up, 12 in 2024-08-30T23:44:05.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:04 smithi136 ceph-mon[20207]: 5.18 scrub starts 2024-08-30T23:44:05.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 podman[110673]: 2024-08-30 23:44:05.076268716 +0000 UTC m=+2.728034290 container cleanup ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-30T23:44:05.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 bash[110673]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5 2024-08-30T23:44:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:04 smithi173 ceph-mon[20554]: osdmap e595: 12 total, 11 up, 12 in 2024-08-30T23:44:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:04 smithi173 ceph-mon[20554]: 5.18 scrub starts 2024-08-30T23:44:05.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:05.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:05.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:05.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:05.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:05.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:05.657 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 podman[110704]: 2024-08-30 23:44:05.311715047 +0000 UTC m=+1.944145671 container remove ed09af7132494c9d6323966b33cac60178eccd9e5783c66fba9b6be2bfdaa8d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-30T23:44:05.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:44:05.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:05.749 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:05.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:05.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:05.875 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:05.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:05.909 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:05 smithi136 ceph-mon[20207]: osdmap e596: 12 total, 11 up, 12 in 2024-08-30T23:44:05.910 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 podman[110822]: 2024-08-30 23:44:05.560435153 +0000 UTC m=+0.016140269 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:44:05.910 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 podman[110822]: 2024-08-30 23:44:05.684735204 +0000 UTC m=+0.140440311 container create 9aa262f8c9518c72923093a464b0d06e54db0f4cb858818b106c61aad6f683ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-08-30T23:44:05.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:05.985 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:05 smithi106 ceph-mon[16603]: osdmap e596: 12 total, 11 up, 12 in 2024-08-30T23:44:05.985 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:05 smithi106 ceph-mon[16603]: pgmap v1343: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 stale+active+clean, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 785 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 2592/37659 objects degraded (6.883%); 2713/37659 objects misplaced (7.204%); 5.4 MiB/s, 325 objects/s recovering 2024-08-30T23:44:05.985 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1840/39369 objects degraded (4.674%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:44:05.985 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:05 smithi106 ceph-mon[16603]: osdmap e597: 12 total, 11 up, 12 in 2024-08-30T23:44:06.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:06.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:06.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:06.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:06.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:05 smithi136 ceph-mon[20207]: pgmap v1343: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 stale+active+clean, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 785 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 2592/37659 objects degraded (6.883%); 2713/37659 objects misplaced (7.204%); 5.4 MiB/s, 325 objects/s recovering 2024-08-30T23:44:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1840/39369 objects degraded (4.674%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:44:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:05 smithi136 ceph-mon[20207]: osdmap e597: 12 total, 11 up, 12 in 2024-08-30T23:44:06.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 podman[110822]: 2024-08-30 23:44:05.96051635 +0000 UTC m=+0.416221482 container init 9aa262f8c9518c72923093a464b0d06e54db0f4cb858818b106c61aad6f683ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:44:06.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:05 smithi136 podman[110822]: 2024-08-30 23:44:05.967278993 +0000 UTC m=+0.422984108 container start 9aa262f8c9518c72923093a464b0d06e54db0f4cb858818b106c61aad6f683ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-30T23:44:06.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:06 smithi136 podman[110822]: 2024-08-30 23:44:06.057943403 +0000 UTC m=+0.513648529 container attach 9aa262f8c9518c72923093a464b0d06e54db0f4cb858818b106c61aad6f683ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-30T23:44:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:05 smithi173 ceph-mon[20554]: osdmap e596: 12 total, 11 up, 12 in 2024-08-30T23:44:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:05 smithi173 ceph-mon[20554]: pgmap v1343: 47 pgs: 1 active+recovery_wait+degraded+remapped, 1 stale+active+clean, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 785 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 2592/37659 objects degraded (6.883%); 2713/37659 objects misplaced (7.204%); 5.4 MiB/s, 325 objects/s recovering 2024-08-30T23:44:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1840/39369 objects degraded (4.674%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:44:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:05 smithi173 ceph-mon[20554]: osdmap e597: 12 total, 11 up, 12 in 2024-08-30T23:44:06.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:06.392 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:06.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:06.560 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:06.620 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:44:06 smithi136 podman[110822]: 2024-08-30 23:44:06.294921531 +0000 UTC m=+0.750626647 container died 9aa262f8c9518c72923093a464b0d06e54db0f4cb858818b106c61aad6f683ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-30T23:44:06.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:06.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:06.733 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:06.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:06.812 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:06.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:06.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:06.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:06.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service' 2024-08-30T23:44:07.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-30T23:44:07.030 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-30T23:44:07.030 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-08-30T23:44:07.030 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi136.front.sepia.ceph.com 2024-08-30T23:44:07.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.5 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-08-30T23:44:07.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:07.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/454444383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:07.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:07.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:07.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:07.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/454444383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:07.311 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.5/config 2024-08-30T23:44:07.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:07.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/454444383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:07.356 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:07.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.438 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:07.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:07.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:07.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:07.592 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:07.593 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:07.614 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:07.614 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:07.636 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:07.636 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:07.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:07.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:07.668 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:07.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:07.689 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:07.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:07.689 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:07.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:07.711 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:07.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:07.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:07.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:07.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:07.774 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:07.774 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:07.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:07.796 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:07.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:07.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:07.817 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:07.818 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:07.899 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:44:07.899 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:44:07.899 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:44:07.899 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:44:07.900 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:44:07.913 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:44:07.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:07.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:07.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:08.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1993932863' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:07 smithi136 ceph-mon[20207]: pgmap v1345: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 707 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 3106/33912 objects degraded (9.159%); 2198/33912 objects misplaced (6.481%); 8.0 MiB/s, 353 objects/s recovering 2024-08-30T23:44:08.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:08.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1993932863' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:07 smithi106 ceph-mon[16603]: pgmap v1345: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 707 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 3106/33912 objects degraded (9.159%); 2198/33912 objects misplaced (6.481%); 8.0 MiB/s, 353 objects/s recovering 2024-08-30T23:44:08.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:08.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:08.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:08.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1993932863' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:07 smithi173 ceph-mon[20554]: pgmap v1345: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 707 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 3106/33912 objects degraded (9.159%); 2198/33912 objects misplaced (6.481%); 8.0 MiB/s, 353 objects/s recovering 2024-08-30T23:44:08.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:08.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:08.494 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:44:08.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:08.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:08.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:08.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:08.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:08.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:08.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:08.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:08.933 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:08.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:09.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:08 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1993932863' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:08 smithi136 ceph-mon[20207]: osdmap e598: 12 total, 11 up, 12 in 2024-08-30T23:44:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:09.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:08 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1993932863' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:08 smithi106 ceph-mon[16603]: osdmap e598: 12 total, 11 up, 12 in 2024-08-30T23:44:09.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:09.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:09.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:09.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:08 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1993932863' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:08 smithi173 ceph-mon[20554]: osdmap e598: 12 total, 11 up, 12 in 2024-08-30T23:44:09.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:09.430 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:09.522 INFO:teuthology.orchestra.run.smithi136.stdout:5.f 2024-08-30T23:44:09.522 INFO:teuthology.orchestra.run.smithi136.stdout:5.b 2024-08-30T23:44:09.522 INFO:teuthology.orchestra.run.smithi136.stdout:5.4 2024-08-30T23:44:09.522 INFO:teuthology.orchestra.run.smithi136.stdout:5.6 2024-08-30T23:44:09.522 INFO:teuthology.orchestra.run.smithi136.stdout:5.19 2024-08-30T23:44:09.523 INFO:teuthology.orchestra.run.smithi136.stdout:5.12 2024-08-30T23:44:09.523 INFO:teuthology.orchestra.run.smithi136.stdout:1.f 2024-08-30T23:44:09.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:09.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:09.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:09.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:09.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:09.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:09.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:09.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:10.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:10.049 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:10.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:10.170 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:09 smithi136 ceph-mon[20207]: 5.18 scrub starts 2024-08-30T23:44:10.170 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:09 smithi136 ceph-mon[20207]: 5.18 scrub ok 2024-08-30T23:44:10.170 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:09 smithi136 ceph-mon[20207]: pgmap v1347: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 702 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 330 op/s; 3106/33696 objects degraded (9.218%); 2198/33696 objects misplaced (6.523%); 6.6 MiB/s, 239 objects/s recovering 2024-08-30T23:44:10.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:10.193 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:09 smithi106 ceph-mon[16603]: 5.18 scrub starts 2024-08-30T23:44:10.193 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:09 smithi106 ceph-mon[16603]: 5.18 scrub ok 2024-08-30T23:44:10.193 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:09 smithi106 ceph-mon[16603]: pgmap v1347: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 702 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 330 op/s; 3106/33696 objects degraded (9.218%); 2198/33696 objects misplaced (6.523%); 6.6 MiB/s, 239 objects/s recovering 2024-08-30T23:44:10.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:44:10.274 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:10.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:09 smithi173 ceph-mon[20554]: 5.18 scrub starts 2024-08-30T23:44:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:09 smithi173 ceph-mon[20554]: 5.18 scrub ok 2024-08-30T23:44:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:09 smithi173 ceph-mon[20554]: pgmap v1347: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 702 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 330 op/s; 3106/33696 objects degraded (9.218%); 2198/33696 objects misplaced (6.523%); 6.6 MiB/s, 239 objects/s recovering 2024-08-30T23:44:10.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:10.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:10.453 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:10.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.5 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 1.f --file /var/log/ceph/exp.1.f.5 2024-08-30T23:44:10.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:10.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:10.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:44:10.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:10.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:10.743 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.5/config 2024-08-30T23:44:10.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:10.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:10.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:10.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:10.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:10.975 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.25654.0:168798 4.6 e375/369 4:68592339:::benchmark_data_smithi173_74914_object70556:head v 375'10605, mlcod=375'10602)", 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 443.44723472499999, 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.208871322, 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:10.976 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527109+0000", 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527110+0000", 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5499999999999997e-07 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.527116+0000", 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0209999999999996e-06 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732384+0000", 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.205268487 2024-08-30T23:44:10.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732390+0000", 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0419999999999999e-06 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732440+0000", 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9663000000000002e-05 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.732470+0000", 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0412e-05 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735945+0000", 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034745869999999999 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:36:47.735980+0000", 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5255e-05 2024-08-30T23:44:10.980 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:59001 5.6 e419/389 5:6208a760:::benchmark_data_smithi173_87639_object59000:head v 419'3623, mlcod=418'3620)", 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 338.83127627599998, 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.314356937, 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:10.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143067+0000", 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:10.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143070+0000", 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5739999999999998e-06 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.143128+0000", 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7890000000000003e-05 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455493+0000", 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31236522100000003 2024-08-30T23:44:10.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455506+0000", 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3655999999999999e-05 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455662+0000", 2024-08-30T23:44:10.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015594900000000001 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.455681+0000", 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8301e-05 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:10.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.457390+0000", 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017094860000000001 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:32.457424+0000", 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3859999999999998e-05 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 424/420 [PushReplyOp(5:11696329:::benchmark_data_smithi173_87639_object5188:head)])", 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:44:10.986 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 331.15301891000001, 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45081538799999998, 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:10.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821324+0000", 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821326+0000", 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3340000000000001e-06 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:39.821332+0000", 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0360000000000003e-06 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269704+0000", 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44837247699999999 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269713+0000", 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.269806+0000", 2024-08-30T23:44:10.989 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3058000000000002e-05 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.272140+0000", 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023334639999999999 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.990 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 425/420 [PushReplyOp(5:10493f51:::benchmark_data_smithi173_87639_object5215:head)])", 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 330.61215214800001, 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318688992, 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:44:10.991 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:10.992 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362191+0000", 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362192+0000", 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.141e-06 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.362198+0000", 2024-08-30T23:44:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6130000000000003e-06 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678705+0000", 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31650737299999998 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678716+0000", 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0288e-05 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.678809+0000", 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3018999999999998e-05 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:40.680880+0000", 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020715579999999998 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:10.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.8 427/420 [PushReplyOp(5:1ccb0413:::benchmark_data_smithi173_87639_object8971:head)])", 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 326.47065248000001, 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60307924400000001, 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.996 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503691+0000", 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503692+0000", 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.02e-07 2024-08-30T23:44:10.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:44.503698+0000", 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9819999999999996e-06 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104246+0000", 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.60054844500000004 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.998 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104278+0000", 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.18e-05 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.104366+0000", 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7817000000000003e-05 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:10.999 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:38:45.106770+0000", 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002404298 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:84105 5.8 5:17a91613:::benchmark_data_smithi173_87639_object3865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 293.802017197, 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22008060500000001, 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 84105 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.001 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172326+0000", 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.002 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172327+0000", 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.124e-06 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172332+0000", 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9910000000000002e-06 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172344+0000", 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1755e-05 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172351+0000", 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5699999999999998e-06 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172564+0000", 2024-08-30T23:44:11.004 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000213317 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172648+0000", 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3610999999999996e-05 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.005 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,9,10", 2024-08-30T23:44:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.172745+0000", 2024-08-30T23:44:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7783999999999998e-05 2024-08-30T23:44:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:11.006 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.174806+0000", 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020607030000000001 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175526+0000", 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00071991399999999999 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.176022+0000", 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049577499999999995 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.008 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391996+0000", 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21597425100000001 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392038+0000", 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2058999999999998e-05 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.009 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392407+0000", 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036875100000000001 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.010 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:84105 5.8 e443/420 ondisk, result = 0)", 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 293.799181965, 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21725276199999999, 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175161+0000", 2024-08-30T23:44:11.012 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175162+0000", 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.013 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.175166+0000", 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2189999999999998e-06 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391931+0000", 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21676427100000001 2024-08-30T23:44:11.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391937+0000", 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0270000000000001e-06 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391986+0000", 2024-08-30T23:44:11.015 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9175999999999998e-05 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.391992+0000", 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0929999999999999e-06 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.016 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:17.392414+0000", 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042228600000000002 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:95258 5.f e532/447 5:f9703d87:::benchmark_data_smithi173_87639_object15018:head v 532'5941, mlcod=532'5938)", 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:27.981612+0000", 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 102.99273111799999, 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24440374600000001, 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981612+0000", 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981612+0000", 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981613+0000", 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8100000000000001e-07 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981621+0000", 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9160000000000001e-06 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.222924+0000", 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24130264400000001 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.222931+0000", 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1339999999999998e-06 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.223002+0000", 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1369000000000001e-05 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.223030+0000", 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7460000000000001e-05 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.225981+0000", 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029507489999999999 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.226016+0000", 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5493000000000001e-05 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.14 542/530 [PushReplyOp(5:2b316a56:::benchmark_data_smithi173_87639_object66576:head)])", 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:41.484161+0000", 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 89.490182305999994, 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54557016300000005, 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484161+0000", 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484161+0000", 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484162+0000", 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5799999999999998e-07 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484168+0000", 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7230000000000001e-06 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.028873+0000", 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54470516400000002 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.028881+0000", 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8560000000000007e-06 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.028935+0000", 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4759999999999997e-05 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.029731+0000", 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00079590200000000002 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.027 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.14 542/530 [PushReplyOp(5:2fcba5e8:::benchmark_data_smithi173_87639_object68452:head)])", 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 88.415321840000004, 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23938505400000001, 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7800000000000001e-07 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559028+0000", 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1919999999999996e-06 2024-08-30T23:44:11.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.797504+0000", 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23847607300000001 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.797512+0000", 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8810000000000002e-06 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.797579+0000", 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6188999999999999e-05 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.798407+0000", 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082794100000000001 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(5.14 e544,530 5:2bed0bb1:::benchmark_data_smithi173_87639_object1178:head,440'5058)", 2024-08-30T23:44:11.031 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:47.088088+0000", 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 83.886255568999999, 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23303040999999999, 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.088088+0000", 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.088088+0000", 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.088089+0000", 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8800000000000001e-07 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.088095+0000", 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3319999999999997e-06 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.320708+0000", 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23261291000000001 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.320714+0000", 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9209999999999997e-06 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.320776+0000", 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2008000000000007e-05 2024-08-30T23:44:11.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.321118+0000", 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034235100000000002 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:103089 5.6 e546/488 5:612727a4:::benchmark_data_smithi173_87639_object22849:head v 546'5861, mlcod=545'5858)", 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:49.898028+0000", 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 81.076315230999995, 2024-08-30T23:44:11.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32197586099999997, 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898028+0000", 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898028+0000", 2024-08-30T23:44:11.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898029+0000", 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.322e-06 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898036+0000", 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1040000000000003e-06 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218477+0000", 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32044120700000001 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218485+0000", 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9869999999999999e-06 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218569+0000", 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4227000000000005e-05 2024-08-30T23:44:11.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218588+0000", 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9358999999999999e-05 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.219971+0000", 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001382882 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.220004+0000", 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2772999999999999e-05 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:103106 5.a e546/457 5:5e3a0b2f:::benchmark_data_smithi173_87639_object22866:head v 546'6510, mlcod=546'6507)", 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:49.972574+0000", 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 81.001769671999995, 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45312410199999997, 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.040 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972574+0000", 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.041 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972574+0000", 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972575+0000", 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.093e-06 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972582+0000", 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1060000000000002e-06 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422561+0000", 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44997882300000003 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422565+0000", 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2470000000000002e-06 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422614+0000", 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8711999999999997e-05 2024-08-30T23:44:11.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422637+0000", 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3771999999999999e-05 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.425662+0000", 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030249209999999999 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.047 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.425698+0000", 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5428e-05 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.f 554/536 [PushReplyOp(5:ffa89edb:::benchmark_data_smithi173_87639_object39711:head)])", 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:03.574448+0000", 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 67.399894950999993, 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23876501999999999, 2024-08-30T23:44:11.048 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574448+0000", 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574448+0000", 2024-08-30T23:44:11.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574449+0000", 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9299999999999997e-07 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574456+0000", 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6340000000000003e-06 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.810549+0000", 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23609303500000001 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.810555+0000", 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.896e-06 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.810616+0000", 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1427000000000007e-05 2024-08-30T23:44:11.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.813213+0000", 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025972349999999998 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:123150 5.f 5:fdac4e98:::benchmark_data_smithi173_87639_object42910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e568)", 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:26.004606+0000", 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 44.969736916000002, 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.248495243, 2024-08-30T23:44:11.052 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 123150 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004606+0000", 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004606+0000", 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004607+0000", 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.054 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004612+0000", 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3320000000000002e-06 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004623+0000", 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1623999999999999e-05 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004629+0000", 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.507e-06 2024-08-30T23:44:11.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.007575+0000", 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029462260000000001 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.007731+0000", 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155818 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.056 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,5,11", 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.007860+0000", 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012949500000000001 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.011684+0000", 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038234950000000001 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.021736+0000", 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01005195 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.024827+0000", 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030912790000000002 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251505+0000", 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22667806800000001 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251564+0000", 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8959000000000003e-05 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.253102+0000", 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015375199999999999 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.059 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:123150 5.f e568/566 ondisk, result = 0)", 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:26.012520+0000", 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 44.961823432000003, 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24060188699999999, 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012520+0000", 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012520+0000", 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012521+0000", 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0079999999999999e-06 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012527+0000", 2024-08-30T23:44:11.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0700000000000003e-06 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.245180+0000", 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23265303200000001 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.245187+0000", 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3300000000000001e-06 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251488+0000", 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063001519999999998 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251498+0000", 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0362e-05 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.253122+0000", 2024-08-30T23:44:11.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016239329999999999 2024-08-30T23:44:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.064 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:127331 5.14 e571/546 5:2fb0fe1d:::benchmark_data_smithi173_87639_object47091:head v 571'6632, mlcod=571'6628)", 2024-08-30T23:44:11.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:32.168977+0000", 2024-08-30T23:44:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 38.805366221, 2024-08-30T23:44:11.065 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.356479241, 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168977+0000", 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168977+0000", 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168978+0000", 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6000000000000003e-07 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168984+0000", 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8270000000000003e-06 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524130+0000", 2024-08-30T23:44:11.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.355145934 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524137+0000", 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.075e-06 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524181+0000", 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4292999999999998e-05 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524198+0000", 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.677e-05 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525419+0000", 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012211699999999999 2024-08-30T23:44:11.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525456+0000", 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7412e-05 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(5.f e574,566 5:f2f1486a:::benchmark_data_smithi173_87639_object13299:head,531'5830)", 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:34.873839+0000", 2024-08-30T23:44:11.070 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 36.100504282000003, 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26355642099999999, 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873839+0000", 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873839+0000", 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873840+0000", 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873844+0000", 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9099999999999998e-06 2024-08-30T23:44:11.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.133925+0000", 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26008125799999998 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.133931+0000", 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5890000000000002e-06 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.135718+0000", 2024-08-30T23:44:11.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017873800000000001 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.137396+0000", 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001677621 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:143169 5.f 5:f2854a5a:::benchmark_data_smithi173_87639_object62929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:53.894369+0000", 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 17.07997456, 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.333810253, 2024-08-30T23:44:11.074 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143169 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894369+0000", 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894369+0000", 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894370+0000", 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3549999999999999e-06 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894376+0000", 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.575e-06 2024-08-30T23:44:11.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894388+0000", 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2132e-05 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894393+0000", 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1880000000000001e-06 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894439+0000", 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5571000000000001e-05 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894549+0000", 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011027 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,5,11", 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894646+0000", 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7369999999999998e-05 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.078 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.897891+0000", 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032451839999999999 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.902231+0000", 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043393670000000002 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.902272+0000", 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1587e-05 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.079 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:54.227011+0000", 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32473906899999999 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:54.227056+0000", 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4805000000000002e-05 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:54.228179+0000", 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011227800000000001 2024-08-30T23:44:11.080 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPull(5.f e600/595 cost 1049576)", 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:44:10.837985+0000", 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.136358691, 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030881620000000002, 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:11.081 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.837985+0000", 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.837985+0000", 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.837986+0000", 2024-08-30T23:44:11.082 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1370000000000001e-06 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.837998+0000", 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2150999999999999e-05 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.838032+0000", 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4124000000000001e-05 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.083 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.838041+0000", 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2240000000000001e-06 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.838097+0000", 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5424999999999999e-05 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:10.841073+0000", 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029761010000000001 2024-08-30T23:44:11.084 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.085 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.085 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.085 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:11.085 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:11.085 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:44:11.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:10 smithi106 ceph-mon[16603]: osdmap e599: 12 total, 11 up, 12 in 2024-08-30T23:44:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3106/33696 objects degraded (9.218%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:10 smithi106 ceph-mon[16603]: osdmap e600: 12 total, 11 up, 12 in 2024-08-30T23:44:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:11.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:11.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:11.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:10 smithi136 ceph-mon[20207]: osdmap e599: 12 total, 11 up, 12 in 2024-08-30T23:44:11.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3106/33696 objects degraded (9.218%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:11.150 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:10 smithi136 ceph-mon[20207]: osdmap e600: 12 total, 11 up, 12 in 2024-08-30T23:44:11.209 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:11.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:10 smithi173 ceph-mon[20554]: osdmap e599: 12 total, 11 up, 12 in 2024-08-30T23:44:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3106/33696 objects degraded (9.218%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:10 smithi173 ceph-mon[20554]: osdmap e600: 12 total, 11 up, 12 in 2024-08-30T23:44:11.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:11.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:11.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:11.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:11.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:11.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:11.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:44:11.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:11.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:11.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:11.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:11.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:11.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:11.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:12.071 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:12.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:44:12.094 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:12.131 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:12.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:12.219 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:12.254 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:11 smithi106 ceph-mon[16603]: pgmap v1350: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3051/32967 objects degraded (9.255%); 486/32967 objects misplaced (1.474%) 2024-08-30T23:44:12.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:11 smithi136 ceph-mon[20207]: pgmap v1350: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3051/32967 objects degraded (9.255%); 486/32967 objects misplaced (1.474%) 2024-08-30T23:44:12.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:11 smithi173 ceph-mon[20554]: pgmap v1350: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3051/32967 objects degraded (9.255%); 486/32967 objects misplaced (1.474%) 2024-08-30T23:44:12.342 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:12.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:12.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:12.567 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:12.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:12.665 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:12.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:12.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:12.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:12.693 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:12.697 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:12.697 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:12.712 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:12.724 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:12.724 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:12.744 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:12.744 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:12.760 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:44:12.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:12.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:12.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:12.787 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:12.787 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:12.808 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:12.808 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:12.814 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:12.829 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:12.830 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:12.851 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:12.851 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:12.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:12.872 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:12.893 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:12.893 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:12.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:12.960 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:12.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:12 smithi136 ceph-mon[20207]: 5.0 scrub starts 2024-08-30T23:44:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:12 smithi136 ceph-mon[20207]: 5.0 scrub ok 2024-08-30T23:44:13.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2711307077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:13.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:13.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:12 smithi106 ceph-mon[16603]: 5.0 scrub starts 2024-08-30T23:44:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:12 smithi106 ceph-mon[16603]: 5.0 scrub ok 2024-08-30T23:44:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2711307077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:13.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:13.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:13.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:12 smithi173 ceph-mon[20554]: 5.0 scrub starts 2024-08-30T23:44:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:12 smithi173 ceph-mon[20554]: 5.0 scrub ok 2024-08-30T23:44:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2711307077' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:13.331 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:13.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:13.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:13.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:13.578 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:13.659 INFO:teuthology.orchestra.run.smithi136.stderr:Exporting 1.f info 1.f( v 23'92 (0'0,23'92] local-lis/les=567/568 n=0 ec=507/20 lis/c=567/567 les/c/f=568/568/0 sis=567) 2024-08-30T23:44:13.659 INFO:teuthology.orchestra.run.smithi136.stderr:Export successful 2024-08-30T23:44:13.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:13.702 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:13.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:13.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:13.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:13.927 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:44:13.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:13.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:13.951 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:14.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:14.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:13 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:44:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:13 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:44:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:13 smithi106 ceph-mon[16603]: 5.b scrub starts 2024-08-30T23:44:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:13 smithi106 ceph-mon[16603]: 5.b scrub ok 2024-08-30T23:44:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4287038816' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:13 smithi106 ceph-mon[16603]: pgmap v1351: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2741/32952 objects degraded (8.318%); 796/32952 objects misplaced (2.416%) 2024-08-30T23:44:14.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:14.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:13 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:44:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:13 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:44:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:13 smithi136 ceph-mon[20207]: 5.b scrub starts 2024-08-30T23:44:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:13 smithi136 ceph-mon[20207]: 5.b scrub ok 2024-08-30T23:44:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4287038816' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:13 smithi136 ceph-mon[20207]: pgmap v1351: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2741/32952 objects degraded (8.318%); 796/32952 objects misplaced (2.416%) 2024-08-30T23:44:14.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:14.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.321 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:13 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:44:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:13 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:44:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:13 smithi173 ceph-mon[20554]: 5.b scrub starts 2024-08-30T23:44:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:13 smithi173 ceph-mon[20554]: 5.b scrub ok 2024-08-30T23:44:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4287038816' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:13 smithi173 ceph-mon[20554]: pgmap v1351: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+undersized, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2741/32952 objects degraded (8.318%); 796/32952 objects misplaced (2.416%) 2024-08-30T23:44:14.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:14.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:14.523 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:14.523 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:14.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:14.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:14.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.5 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 1.f 2024-08-30T23:44:14.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:14.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:14.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:14.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:14.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:14.848 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.5/config 2024-08-30T23:44:14.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:14.927 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:14 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:44:14.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:14.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:15.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:15.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:15.154 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:14 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:44:15.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:14 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:44:15.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:14 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:44:15.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4287038816' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:15.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:14 smithi106 ceph-mon[16603]: osdmap e601: 12 total, 11 up, 12 in 2024-08-30T23:44:15.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:15.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:44:15.189 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:15.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:15.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:14 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:44:15.233 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:14 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:44:15.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4287038816' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:15.234 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:14 smithi136 ceph-mon[20207]: osdmap e601: 12 total, 11 up, 12 in 2024-08-30T23:44:15.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:15.313 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:14 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:44:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:14 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:44:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:14 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:44:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4287038816' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:14 smithi173 ceph-mon[20554]: osdmap e601: 12 total, 11 up, 12 in 2024-08-30T23:44:15.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:15.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:15.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:15.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:15.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:15.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:15.770 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:15.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:15.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:15.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:44:15.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:15.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:15.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:16.042 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:44:16.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:16.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:16.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:15 smithi106 ceph-mon[16603]: 5.e scrub starts 2024-08-30T23:44:16.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:15 smithi106 ceph-mon[16603]: 5.e scrub ok 2024-08-30T23:44:16.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:15 smithi106 ceph-mon[16603]: pgmap v1353: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%) 2024-08-30T23:44:16.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2741/32952 objects degraded (8.318%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:16.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:15 smithi106 ceph-mon[16603]: osdmap e602: 12 total, 11 up, 12 in 2024-08-30T23:44:16.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:16.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:15 smithi136 ceph-mon[20207]: 5.e scrub starts 2024-08-30T23:44:16.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:15 smithi136 ceph-mon[20207]: 5.e scrub ok 2024-08-30T23:44:16.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:15 smithi136 ceph-mon[20207]: pgmap v1353: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%) 2024-08-30T23:44:16.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2741/32952 objects degraded (8.318%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:16.197 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:15 smithi136 ceph-mon[20207]: osdmap e602: 12 total, 11 up, 12 in 2024-08-30T23:44:16.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:16.266 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:16.266 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:16.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:15 smithi173 ceph-mon[20554]: 5.e scrub starts 2024-08-30T23:44:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:15 smithi173 ceph-mon[20554]: 5.e scrub ok 2024-08-30T23:44:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:15 smithi173 ceph-mon[20554]: pgmap v1353: 47 pgs: 1 active+recovering+degraded+remapped, 1 active+clean+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 3 active+undersized+degraded, 39 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%) 2024-08-30T23:44:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2741/32952 objects degraded (8.318%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:15 smithi173 ceph-mon[20554]: osdmap e602: 12 total, 11 up, 12 in 2024-08-30T23:44:16.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:16.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:16.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:16.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:16.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:16.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:16.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:16.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:16.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:16.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:16.957 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:16.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:17.059 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:17.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:44:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.082 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:17.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:17.151 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:17.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:17.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:17.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.334 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:17.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:17.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:17.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:17.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: marking collection for removal 2024-08-30T23:44:17.494 INFO:teuthology.orchestra.run.smithi136.stdout:setting '_remove' omap key 2024-08-30T23:44:17.494 INFO:teuthology.orchestra.run.smithi136.stdout:finish_remove_pgs 1.f_head removing 1.f 2024-08-30T23:44:17.496 INFO:teuthology.orchestra.run.smithi136.stdout:Remove successful 2024-08-30T23:44:17.531 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:17 smithi106 ceph-mon[16603]: 5.9 scrub starts 2024-08-30T23:44:17.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:17 smithi106 ceph-mon[16603]: 5.9 scrub ok 2024-08-30T23:44:17.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:17 smithi106 ceph-mon[16603]: 5.d scrub starts 2024-08-30T23:44:17.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:17 smithi106 ceph-mon[16603]: 5.d scrub ok 2024-08-30T23:44:17.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:17.582 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:17.641 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:17.641 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:17.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:44:17.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:17.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:17.666 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:17.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:17.684 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:17.685 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:17.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:17.712 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:17.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:17.734 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:17.734 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:17.734 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:17.756 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:17.756 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:17.777 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:17.777 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:17 smithi136 ceph-mon[20207]: 5.9 scrub starts 2024-08-30T23:44:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:17 smithi136 ceph-mon[20207]: 5.9 scrub ok 2024-08-30T23:44:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:17 smithi136 ceph-mon[20207]: 5.d scrub starts 2024-08-30T23:44:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:17 smithi136 ceph-mon[20207]: 5.d scrub ok 2024-08-30T23:44:17.799 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:17.799 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:17.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:17.821 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:17.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:17 smithi173 ceph-mon[20554]: 5.9 scrub starts 2024-08-30T23:44:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:17 smithi173 ceph-mon[20554]: 5.9 scrub ok 2024-08-30T23:44:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:17 smithi173 ceph-mon[20554]: 5.d scrub starts 2024-08-30T23:44:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:17 smithi173 ceph-mon[20554]: 5.d scrub ok 2024-08-30T23:44:17.830 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:17.843 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:17.843 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:17.866 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:17.866 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:17.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:17.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:17.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:18.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.080 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:18.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:18.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:18.203 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:18.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:18.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:18.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:18.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:18.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:18.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:18 smithi106 ceph-mon[16603]: pgmap v1355: 47 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 37 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 13 op/s; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%); 5.4 MiB/s, 414 objects/s recovering 2024-08-30T23:44:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3871963869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:18 smithi106 ceph-mon[16603]: 5.8 scrub starts 2024-08-30T23:44:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:18 smithi106 ceph-mon[16603]: 5.8 scrub ok 2024-08-30T23:44:18.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:18.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:18.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:18.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:18 smithi136 ceph-mon[20207]: pgmap v1355: 47 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 37 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 13 op/s; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%); 5.4 MiB/s, 414 objects/s recovering 2024-08-30T23:44:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3871963869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:18 smithi136 ceph-mon[20207]: 5.8 scrub starts 2024-08-30T23:44:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:18 smithi136 ceph-mon[20207]: 5.8 scrub ok 2024-08-30T23:44:18.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:18.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:18.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:18 smithi173 ceph-mon[20554]: pgmap v1355: 47 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 37 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 13 op/s; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%); 5.4 MiB/s, 414 objects/s recovering 2024-08-30T23:44:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3871963869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:18 smithi173 ceph-mon[20554]: 5.8 scrub starts 2024-08-30T23:44:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:18 smithi173 ceph-mon[20554]: 5.8 scrub ok 2024-08-30T23:44:18.881 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:44:18.881 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62091 4.4 4:2f9ffd79:::benchmark_data_smithi173_74914_object62090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 561.71442546399999, 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.318614905, 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.882 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62091 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165668+0000", 2024-08-30T23:44:18.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165670+0000", 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165713+0000", 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3368000000000002e-05 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165732+0000", 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9031000000000001e-05 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165740+0000", 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8050000000000007e-06 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165802+0000", 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2279999999999993e-05 2024-08-30T23:44:18.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.165946+0000", 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014352899999999999 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.166046+0000", 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100046 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169317+0000", 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003271267 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.169909+0000", 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059201200000000005 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484178+0000", 2024-08-30T23:44:18.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31426850299999998 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484234+0000", 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6678000000000002e-05 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.484283+0000", 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8457e-05 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62105 4.4 4:24e8c307:::benchmark_data_smithi173_74914_object62104:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 561.70084113200005, 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34846845199999998, 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62105 2024-08-30T23:44:18.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179252+0000", 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179254+0000", 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.821e-06 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179297+0000", 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3108000000000001e-05 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179311+0000", 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.345e-05 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179316+0000", 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5130000000000004e-06 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179371+0000", 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5316999999999999e-05 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179508+0000", 2024-08-30T23:44:18.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013643600000000001 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.179665+0000", 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015680499999999999 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182071+0000", 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024068090000000002 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.182226+0000", 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015503899999999999 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527606+0000", 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34537996300000001 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527669+0000", 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2966999999999995e-05 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527721+0000", 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1224000000000003e-05 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62174 4.4 4:286e2c16:::benchmark_data_smithi173_74914_object62173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 561.35562935200005, 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57863658799999995, 2024-08-30T23:44:18.895 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.896 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.896 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62174 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524464+0000", 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524466+0000", 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0600000000000002e-06 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524501+0000", 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4635999999999998e-05 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524521+0000", 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0193000000000001e-05 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524531+0000", 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7780000000000002e-06 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524604+0000", 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3646000000000006e-05 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524763+0000", 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015852999999999999 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.524940+0000", 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017771799999999999 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.527733+0000", 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027928789999999998 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.528529+0000", 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000795487 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103030+0000", 2024-08-30T23:44:18.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57450106899999998 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103068+0000", 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8398000000000001e-05 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.103100+0000", 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2193999999999997e-05 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:62186 4.4 4:257aab52:::benchmark_data_smithi173_74914_object62185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 561.32651723100003, 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55186797300000001, 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 62186 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553576+0000", 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553578+0000", 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7719999999999999e-06 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553612+0000", 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3773999999999999e-05 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553627+0000", 2024-08-30T23:44:18.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5410999999999999e-05 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553637+0000", 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0101e-05 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553709+0000", 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2365000000000003e-05 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.553885+0000", 2024-08-30T23:44:18.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017522699999999999 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.554065+0000", 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000180101 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556530+0000", 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024650240000000001 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:57.556570+0000", 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9975999999999998e-05 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105343+0000", 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54877291299999997 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105399+0000", 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6648999999999999e-05 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:34:58.105444+0000", 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4660000000000003e-05 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.25654.0:196465 4.4 4:242a1b19:::benchmark_data_smithi173_74914_object98223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 420.863982679, 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45786220799999999, 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.909 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.25654", 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196465 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016111+0000", 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9200000000000001e-07 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016114+0000", 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2799999999999999e-06 2024-08-30T23:44:18.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016121+0000", 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9240000000000001e-06 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016125+0000", 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.704e-06 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016162+0000", 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6528e-05 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016238+0000", 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6246000000000001e-05 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.016310+0000", 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2475e-05 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020389+0000", 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040785120000000003 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.020517+0000", 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012816600000000001 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473901+0000", 2024-08-30T23:44:18.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45338352300000001 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473940+0000", 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9224000000000003e-05 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:37:18.473973+0000", 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3034000000000003e-05 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "rep_scrubmap(5.1 e389 from shard 9)", 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:25.986087+0000", 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 232.89400629599999, 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.935597817, 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986087+0000", 2024-08-30T23:44:18.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986087+0000", 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986088+0000", 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5239999999999999e-06 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.986137+0000", 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8992000000000001e-05 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921482+0000", 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93534420799999995 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921488+0000", 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7340000000000002e-06 2024-08-30T23:44:18.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921558+0000", 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9606999999999999e-05 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921568+0000", 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7340000000000003e-06 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:26.921685+0000", 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117018 2024-08-30T23:44:18.919 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "rep_scrubmap(5.1 e488 from shard 9)", 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:09.042611+0000", 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 189.83748273399999, 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.76847709799999997, 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.042611+0000", 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.042611+0000", 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.042611+0000", 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3499999999999995e-07 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.042634+0000", 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2323e-05 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.810872+0000", 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.76823795800000005 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.810880+0000", 2024-08-30T23:44:18.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6200000000000005e-06 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.810943+0000", 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3291000000000001e-05 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.810954+0000", 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0200999999999999e-05 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:09.811088+0000", 2024-08-30T23:44:18.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013396999999999999 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(5.17 506/491 [PushOp(5:ef3eaecd:::benchmark_data_smithi173_87639_object47307:head, version: 414'3013, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ef3eaecd:::benchmark_data_smithi173_87639_object47307:head@414'3013, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:37.103669+0000", 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 161.77642468900001, 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45059095900000001, 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.924 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.103669+0000", 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.103669+0000", 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.103670+0000", 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5820000000000001e-06 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.103751+0000", 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0909999999999996e-05 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.553115+0000", 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44936431399999999 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.553132+0000", 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6286999999999998e-05 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.553246+0000", 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114418 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.553257+0000", 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0742e-05 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:37.554260+0000", 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010027059999999999 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:e928c73b:::benchmark_data_smithi173_87639_object59838:head, version: 421'3786, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:e928c73b:::benchmark_data_smithi173_87639_object59838:head@421'3786, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:43.170078+0000", 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 155.71001480699999, 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.65726012899999997, 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.170078+0000", 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.170078+0000", 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.170079+0000", 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8000000000000004e-07 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.170122+0000", 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2394000000000001e-05 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.826050+0000", 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.65592865300000003 2024-08-30T23:44:18.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.826080+0000", 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9390000000000002e-05 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.826284+0000", 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000204229 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.826308+0000", 2024-08-30T23:44:18.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4022000000000001e-05 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:43.827339+0000", 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010305609999999999 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:e9b7fb9e:::benchmark_data_smithi173_87639_object60566:head, version: 422'3843, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:e9b7fb9e:::benchmark_data_smithi173_87639_object60566:head@422'3843, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:44.039143+0000", 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 154.84095013199999, 2024-08-30T23:44:18.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.43180438199999999, 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.039143+0000", 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.039143+0000", 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.039144+0000", 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.85e-07 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.039208+0000", 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3979000000000005e-05 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.469841+0000", 2024-08-30T23:44:18.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.43063275299999998 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.469875+0000", 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4780999999999998e-05 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.469950+0000", 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4573000000000006e-05 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.469960+0000", 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7100000000000002e-06 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:44.470948+0000", 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00098780099999999996 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:ea741e7d:::benchmark_data_smithi173_87639_object61809:head, version: 422'3927, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ea741e7d:::benchmark_data_smithi173_87639_object61809:head@422'3927, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:18.936 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:45.213263+0000", 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 153.666830226, 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.83889809400000004, 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:45.213263+0000", 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:45.213263+0000", 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:45.213264+0000", 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3300000000000001e-07 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:45.213331+0000", 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7107000000000006e-05 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.050728+0000", 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.83739729100000004 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.050759+0000", 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1130000000000002e-05 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.051022+0000", 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026262399999999998 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.051066+0000", 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3961999999999997e-05 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.052161+0000", 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010952469999999999 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(5.14 541/530 [PushOp(5:2ce5ad1b:::benchmark_data_smithi173_87639_object62371:head, version: 430'3877, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:2ce5ad1b:::benchmark_data_smithi173_87639_object62371:head@430'3877, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:40.176578+0000", 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 98.703515289999999, 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.447586504, 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.176578+0000", 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.176578+0000", 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.176579+0000", 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8599999999999997e-07 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.176607+0000", 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8064000000000001e-05 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.622898+0000", 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44629086200000001 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.622915+0000", 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7774999999999998e-05 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.622987+0000", 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1400000000000001e-05 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.622997+0000", 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0122000000000001e-05 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:40.624164+0000", 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011674949999999999 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.944 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:101081 5.16 5:68c04dd7:::benchmark_data_smithi173_87639_object20841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:45.552737+0000", 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 93.327355956999995, 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34279844999999998, 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 101081 2024-08-30T23:44:18.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552737+0000", 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552737+0000", 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552739+0000", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.316e-06 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552744+0000", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6679999999999998e-06 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552757+0000", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.295e-05 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552765+0000", 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5639999999999999e-06 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.552851+0000", 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6285000000000005e-05 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.553039+0000", 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000187815 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:44:18.949 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:18.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.553221+0000", 2024-08-30T23:44:18.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018174099999999999 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.555034+0000", 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001813397 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.559199+0000", 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041651140000000001 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.893854+0000", 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.334654954 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.893918+0000", 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3590999999999995e-05 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:45.895536+0000", 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001618055 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.951 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(5.14 e544,530 5:2f222811:::benchmark_data_smithi173_87639_object5625:head,445'5333)", 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:47.907743+0000", 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.972350340000006, 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48616630599999999, 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.907743+0000", 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.907743+0000", 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.907744+0000", 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.017e-06 2024-08-30T23:44:18.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:47.907750+0000", 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0039999999999996e-06 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:48.393797+0000", 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48604755500000002 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:48.393804+0000", 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7630000000000003e-06 2024-08-30T23:44:18.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:48.393855+0000", 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1066999999999997e-05 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:48.393869+0000", 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3550000000000001e-05 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:48.393909+0000", 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0349999999999998e-05 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(5.16 e567,540 5:6ba5c810:::benchmark_data_smithi173_87639_object6546:head,527'5360)", 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:24.571812+0000", 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 54.308280830999998, 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45848745699999999, 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:18.956 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.571812+0000", 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.571812+0000", 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.571813+0000", 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8899999999999998e-07 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.571818+0000", 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9169999999999997e-06 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.029775+0000", 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45795696499999999 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.029782+0000", 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4710000000000004e-06 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.029830+0000", 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.85e-05 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.030300+0000", 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000469615 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.959 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(5.16 e567,540 5:6f949103:::benchmark_data_smithi173_87639_object6562:head,527'5361)", 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:24.574926+0000", 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 54.305167157, 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45967450100000001, 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:44:18.960 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.574926+0000", 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.574926+0000", 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.574927+0000", 2024-08-30T23:44:18.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7999999999999995e-07 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:24.574931+0000", 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6800000000000001e-06 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.034257+0000", 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45932600800000001 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.034265+0000", 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.695e-06 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.034311+0000", 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5825999999999998e-05 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:25.034601+0000", 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028961200000000002 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:127331 5.14 5:2fb0fe1d:::benchmark_data_smithi173_87639_object47091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:32.162228+0000", 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 46.717864976999998, 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.36343651100000002, 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 127331 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.964 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.162228+0000", 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.162228+0000", 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.162230+0000", 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5400000000000001e-06 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.162235+0000", 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6080000000000003e-06 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168279+0000", 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060434900000000003 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168296+0000", 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7002000000000001e-05 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168687+0000", 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039154199999999999 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168788+0000", 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100367 2024-08-30T23:44:18.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168906+0000", 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118657 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.170646+0000", 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017398520000000001 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.170725+0000", 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8515000000000003e-05 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525593+0000", 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35486864200000001 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525633+0000", 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9709000000000002e-05 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525665+0000", 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1587000000000001e-05 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:140188 5.16 5:6884968c:::benchmark_data_smithi173_87639_object59948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e582)", 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:49.862565+0000", 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 29.017528256999999, 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.138147872, 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.970 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 140188 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862565+0000", 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862565+0000", 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862567+0000", 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8339999999999999e-06 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862574+0000", 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6869999999999997e-06 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862590+0000", 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6030000000000001e-05 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862598+0000", 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4880000000000007e-06 2024-08-30T23:44:18.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862662+0000", 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4247999999999999e-05 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862786+0000", 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000123628 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.862897+0000", 2024-08-30T23:44:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111173 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.998334+0000", 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.135437154 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:49.998357+0000", 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2373000000000001e-05 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.000608+0000", 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022512349999999999 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.000668+0000", 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0423000000000001e-05 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.000713+0000", 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4598999999999998e-05 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.976 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:140768 5.14 5:2cd08cec:::benchmark_data_smithi173_87639_object60528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e583)", 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:50.630029+0000", 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 28.250063803, 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14457207699999999, 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:18.977 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 140768 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630029+0000", 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630029+0000", 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630031+0000", 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3740000000000001e-06 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630038+0000", 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2150000000000004e-06 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630055+0000", 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7311000000000001e-05 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630064+0000", 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8489999999999995e-06 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630123+0000", 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8912000000000001e-05 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630215+0000", 2024-08-30T23:44:18.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2287000000000006e-05 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.630305+0000", 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9656999999999999e-05 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.632389+0000", 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020840770000000002 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.632618+0000", 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00022884899999999999 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.774498+0000", 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.14187959999999999 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.774552+0000", 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4546999999999998e-05 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:50.774602+0000", 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9398999999999999e-05 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(5.15 e597,590 5:a86f4579:::benchmark_data_smithi173_87639_object867:head,439'5231)", 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:44:07.518209+0000", 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 11.36188404, 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34414273099999998, 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:44:18.983 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.518209+0000", 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.518209+0000", 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.518210+0000", 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2099999999999995e-07 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.518215+0000", 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0849999999999996e-06 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.862223+0000", 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34400787399999999 2024-08-30T23:44:18.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.862230+0000", 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.362e-06 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.862286+0000", 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5679e-05 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:44:18.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.862293+0000", 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4229999999999999e-06 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:07.862352+0000", 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8486999999999998e-05 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:18.987 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:44:18.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:19.011 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:19.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:19.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:19.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:19.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:19.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:19.384 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:19.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:19.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:19 smithi106 ceph-mon[16603]: osdmap e603: 12 total, 11 up, 12 in 2024-08-30T23:44:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:19 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:44:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:19 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:44:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:19 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:44:19.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:19.631 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:19.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:19.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:19 smithi136 ceph-mon[20207]: osdmap e603: 12 total, 11 up, 12 in 2024-08-30T23:44:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:19 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:44:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:19 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:44:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:19 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:44:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:19 smithi173 ceph-mon[20554]: osdmap e603: 12 total, 11 up, 12 in 2024-08-30T23:44:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:19 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:44:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:19 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:44:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:19 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:44:19.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:19.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:19.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:19.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:20.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:20.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:20.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.130 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:20.231 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:20.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:20.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:44:20.347 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:44:20.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:20.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:20.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:20.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:20.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:20 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:44:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:20 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:44:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:20 smithi106 ceph-mon[16603]: osdmap e604: 12 total, 11 up, 12 in 2024-08-30T23:44:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:20 smithi106 ceph-mon[16603]: pgmap v1358: 47 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 37 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%); 5.4 MiB/s, 345 objects/s recovering 2024-08-30T23:44:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2400470644' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:44:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:20 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:44:20.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:20.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:44:20.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.752 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:20.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:20 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:44:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:20 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:44:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:20 smithi136 ceph-mon[20207]: osdmap e604: 12 total, 11 up, 12 in 2024-08-30T23:44:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:20 smithi136 ceph-mon[20207]: pgmap v1358: 47 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 37 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%); 5.4 MiB/s, 345 objects/s recovering 2024-08-30T23:44:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2400470644' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:44:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:20 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:44:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:20 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:44:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:20 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:44:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:20 smithi173 ceph-mon[20554]: osdmap e604: 12 total, 11 up, 12 in 2024-08-30T23:44:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:20 smithi173 ceph-mon[20554]: pgmap v1358: 47 pgs: 1 active+clean+remapped, 2 active+clean+scrubbing, 1 active+recovering+degraded+remapped, 1 active+undersized, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 37 active+clean; 687 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2208/32952 objects degraded (6.701%); 310/32952 objects misplaced (0.941%); 5.4 MiB/s, 345 objects/s recovering 2024-08-30T23:44:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2400470644' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:44:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:20 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:44:20.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.875 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:20.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:20.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:20.998 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:20.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:21.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:21.109 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:21.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:21.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:21.246 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:21.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:21.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:21.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:21.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:21.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:21 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2400470644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:44:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:21 smithi106 ceph-mon[16603]: osdmap e605: 12 total, 11 up, 12 in 2024-08-30T23:44:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2208/32952 objects degraded (6.701%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:21 smithi106 ceph-mon[16603]: osdmap e606: 12 total, 11 up, 12 in 2024-08-30T23:44:21.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:21.673 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:21.674 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:21.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:21.711 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:21 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2400470644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:44:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:21 smithi173 ceph-mon[20554]: osdmap e605: 12 total, 11 up, 12 in 2024-08-30T23:44:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2208/32952 objects degraded (6.701%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:21.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:21 smithi173 ceph-mon[20554]: osdmap e606: 12 total, 11 up, 12 in 2024-08-30T23:44:21.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:21.743 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:21.743 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:21.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:21.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:21.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:21 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2400470644' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:44:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:21 smithi136 ceph-mon[20207]: osdmap e605: 12 total, 11 up, 12 in 2024-08-30T23:44:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2208/32952 objects degraded (6.701%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:44:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:21 smithi136 ceph-mon[20207]: osdmap e606: 12 total, 11 up, 12 in 2024-08-30T23:44:21.802 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:21.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:21.803 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:21.832 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:21.832 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:21.861 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:21.861 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:21.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:21.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:21.891 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:21.895 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:21.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:21.917 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:21.917 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:21.944 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:21.945 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:21.973 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:21.973 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:21.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:22.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.5 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.f.5 2024-08-30T23:44:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:22.024 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:22.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:22.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:22.163 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:22.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:22.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:22.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:22.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:22.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:22.542 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:22 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:44:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:22 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:44:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:22 smithi106 ceph-mon[16603]: pgmap v1361: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 1 active+undersized, 3 active+undersized+degraded, 37 active+clean; 686 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 3 op/s; 1004/32910 objects degraded (3.051%); 1017/32910 objects misplaced (3.090%); 0 B/s, 527 objects/s recovering 2024-08-30T23:44:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1682385578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:22.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:22.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:22 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:44:22.650 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:22 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:44:22.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:22 smithi136 ceph-mon[20207]: pgmap v1361: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 1 active+undersized, 3 active+undersized+degraded, 37 active+clean; 686 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 3 op/s; 1004/32910 objects degraded (3.051%); 1017/32910 objects misplaced (3.090%); 0 B/s, 527 objects/s recovering 2024-08-30T23:44:22.651 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1682385578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:22.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:22.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:22.779 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:22.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:22 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:44:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:22 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:44:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:22 smithi173 ceph-mon[20554]: pgmap v1361: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 1 active+undersized, 3 active+undersized+degraded, 37 active+clean; 686 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 3 op/s; 1004/32910 objects degraded (3.051%); 1017/32910 objects misplaced (3.090%); 0 B/s, 527 objects/s recovering 2024-08-30T23:44:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1682385578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:22.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:22.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:22.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:23.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:23.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:23.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:44:23.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:23.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:23.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:23.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:23.377 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:23.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:23.431 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:23.447 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:23.447 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.5/config 2024-08-30T23:44:23.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:23.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:23.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:23.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:23.703 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:23 smithi106 ceph-mon[16603]: osdmap e607: 12 total, 11 up, 12 in 2024-08-30T23:44:23.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:23 smithi136 ceph-mon[20207]: osdmap e607: 12 total, 11 up, 12 in 2024-08-30T23:44:23.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:23.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:23 smithi173 ceph-mon[20554]: osdmap e607: 12 total, 11 up, 12 in 2024-08-30T23:44:23.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:23.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:23.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:24.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:24.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:24.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:24.202 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:24.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:44:24.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:24.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:24.366 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:44:24.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:24.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:24 smithi136 ceph-mon[20207]: osdmap e608: 12 total, 11 up, 12 in 2024-08-30T23:44:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:24 smithi136 ceph-mon[20207]: pgmap v1364: 47 pgs: 1 active+undersized, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 3 active+undersized+degraded, 37 active+clean; 652 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 954/31281 objects degraded (3.050%); 980/31281 objects misplaced (3.133%); 0 B/s, 527 objects/s recovering 2024-08-30T23:44:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1090383332' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:24.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:24.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:24.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:24 smithi106 ceph-mon[16603]: osdmap e608: 12 total, 11 up, 12 in 2024-08-30T23:44:24.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:24 smithi106 ceph-mon[16603]: pgmap v1364: 47 pgs: 1 active+undersized, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 3 active+undersized+degraded, 37 active+clean; 652 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 954/31281 objects degraded (3.050%); 980/31281 objects misplaced (3.133%); 0 B/s, 527 objects/s recovering 2024-08-30T23:44:24.687 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1090383332' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:24.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:24.724 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:24.781 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:44:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:24 smithi173 ceph-mon[20554]: osdmap e608: 12 total, 11 up, 12 in 2024-08-30T23:44:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:24 smithi173 ceph-mon[20554]: pgmap v1364: 47 pgs: 1 active+undersized, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 3 active+undersized+degraded, 37 active+clean; 652 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 139 op/s; 954/31281 objects degraded (3.050%); 980/31281 objects misplaced (3.133%); 0 B/s, 527 objects/s recovering 2024-08-30T23:44:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1090383332' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:24.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:24.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:24.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:24.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:24.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:25.073 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:25.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.111 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:25.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:25.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:25.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:25.317 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:25.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:25.381 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:25.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1090383332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:25.448 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:25 smithi106 ceph-mon[16603]: osdmap e609: 12 total, 11 up, 12 in 2024-08-30T23:44:25.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:25.500 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1090383332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:25.501 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:25 smithi136 ceph-mon[20207]: osdmap e609: 12 total, 11 up, 12 in 2024-08-30T23:44:25.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:25.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:25.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:25.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:25.725 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:44:25.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:25.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1090383332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:25 smithi173 ceph-mon[20554]: osdmap e609: 12 total, 11 up, 12 in 2024-08-30T23:44:25.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:25.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:25.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:25.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:26.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:26.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:26.125 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:26.125 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:26.152 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:26.158 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:26.158 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:26.177 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:26.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:26.186 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:26.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:26.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:26.252 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:26.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:26.253 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:26.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:26.279 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:26.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:26.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:26.313 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:26.313 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:26.342 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:26.343 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:26.370 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:26.370 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:26.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:26.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:26.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:26.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:26.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:26.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:26.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:26.543 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:26 smithi106 ceph-mon[16603]: pgmap v1366: 47 pgs: 1 active+undersized, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 3 active+undersized+degraded, 37 active+clean; 591 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 318 op/s; 954/28356 objects degraded (3.364%); 724/28356 objects misplaced (2.553%); 0 B/s, 194 objects/s recovering 2024-08-30T23:44:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:26 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 954/31281 objects degraded (3.050%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:26 smithi106 ceph-mon[16603]: osdmap e610: 12 total, 11 up, 12 in 2024-08-30T23:44:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1199026803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:26 smithi106 ceph-mon[16603]: from='client.17526 172.21.15.106:0/1199026803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:26.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:26.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:26.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:26 smithi136 ceph-mon[20207]: pgmap v1366: 47 pgs: 1 active+undersized, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 3 active+undersized+degraded, 37 active+clean; 591 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 318 op/s; 954/28356 objects degraded (3.364%); 724/28356 objects misplaced (2.553%); 0 B/s, 194 objects/s recovering 2024-08-30T23:44:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:26 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 954/31281 objects degraded (3.050%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:26 smithi136 ceph-mon[20207]: osdmap e610: 12 total, 11 up, 12 in 2024-08-30T23:44:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1199026803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:26 smithi136 ceph-mon[20207]: from='client.17526 172.21.15.106:0/1199026803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:26.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:26 smithi173 ceph-mon[20554]: pgmap v1366: 47 pgs: 1 active+undersized, 1 active+recovering+undersized+remapped, 2 active+clean+remapped, 3 active+clean+scrubbing, 3 active+undersized+degraded, 37 active+clean; 591 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 318 op/s; 954/28356 objects degraded (3.364%); 724/28356 objects misplaced (2.553%); 0 B/s, 194 objects/s recovering 2024-08-30T23:44:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:26 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 954/31281 objects degraded (3.050%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:26 smithi173 ceph-mon[20554]: osdmap e610: 12 total, 11 up, 12 in 2024-08-30T23:44:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1199026803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:26 smithi173 ceph-mon[20554]: from='client.17526 172.21.15.106:0/1199026803' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:26.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:26.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:26.927 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:27.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:27.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:27.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:27.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:27.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:27.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:44:27.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.319 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:27.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:27.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:27.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:27.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.587 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:27.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:27.588 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:27.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:27.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:27.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:27.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:27.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:27.983 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:27.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:27.988 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:27 smithi106 ceph-mon[16603]: pgmap v1368: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 402 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s; 531/19281 objects degraded (2.754%); 581/19281 objects misplaced (3.013%); 0 B/s, 55 objects/s recovering 2024-08-30T23:44:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:27 smithi136 ceph-mon[20207]: pgmap v1368: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 402 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s; 531/19281 objects degraded (2.754%); 581/19281 objects misplaced (3.013%); 0 B/s, 55 objects/s recovering 2024-08-30T23:44:28.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:27 smithi173 ceph-mon[20554]: pgmap v1368: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 402 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s; 531/19281 objects degraded (2.754%); 581/19281 objects misplaced (3.013%); 0 B/s, 55 objects/s recovering 2024-08-30T23:44:28.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:28.113 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:28.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:28.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:44:28.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:28.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:28.388 INFO:teuthology.orchestra.run.smithi136.stdout:get_pg_num_history pg_num_history pg_num_history(e593 pg_nums {1={20=1,440=11,507=21},4={306=16},5={389=16,488=26}} deleted_pools 386,4) 2024-08-30T23:44:28.388 INFO:teuthology.orchestra.run.smithi136.stdout:Importing pgid 1.f 2024-08-30T23:44:28.388 INFO:teuthology.orchestra.run.smithi136.stdout:write_pg epoch 568 info 1.f( v 23'92 (0'0,23'92] local-lis/les=567/568 n=0 ec=507/20 lis/c=567/567 les/c/f=568/568/0 sis=567) 2024-08-30T23:44:28.389 INFO:teuthology.orchestra.run.smithi136.stdout:Import successful 2024-08-30T23:44:28.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:28.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:28.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:28.573 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:28.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:28.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:28.771 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:28.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:28.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:28.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:28.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:29.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:29.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:44553 5.c 5:323e8416:::benchmark_data_smithi173_87639_object44552:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 376.71009570899997, 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83468573899999998, 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44553 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448566+0000", 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448567+0000", 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.093e-06 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448624+0000", 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6285000000000001e-05 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448633+0000", 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9830000000000002e-06 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448639+0000", 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.725e-06 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448672+0000", 2024-08-30T23:44:29.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2815999999999999e-05 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448753+0000", 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1149999999999994e-05 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,6", 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.448841+0000", 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7267000000000003e-05 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:44:29.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.449924+0000", 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001083533 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450381+0000", 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045717400000000002 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283161+0000", 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83277994799999999 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283216+0000", 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5067e-05 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283252+0000", 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5698000000000003e-05 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:44553 5.c e410/389 ondisk, result = 0)", 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 376.708660979, 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.83325748899999996, 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450001+0000", 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450002+0000", 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:12.450005+0000", 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.58e-06 2024-08-30T23:44:29.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283028+0000", 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.833022924 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283038+0000", 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.713e-06 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283141+0000", 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102903 2024-08-30T23:44:29.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283154+0000", 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.289e-05 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:13.283259+0000", 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010485899999999999 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58987 5.0 5.b63d9900 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.04386821499997, 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99942602000000003, 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58987 2024-08-30T23:44:29.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114794+0000", 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114795+0000", 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114853+0000", 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7525000000000002e-05 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114863+0000", 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9899999999999992e-06 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114871+0000", 2024-08-30T23:44:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5709999999999998e-06 2024-08-30T23:44:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114907+0000", 2024-08-30T23:44:29.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5661000000000003e-05 2024-08-30T23:44:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:44:29.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.114912+0000", 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2809999999999999e-06 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114202+0000", 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99928982799999999 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114220+0000", 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7734e-05 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:58996 5.0 5.b98b8b60 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.031460466, 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.987150694, 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58996 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.178 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127202+0000", 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127203+0000", 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.347e-06 2024-08-30T23:44:29.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127240+0000", 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7403000000000001e-05 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127253+0000", 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2870999999999999e-05 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127260+0000", 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8800000000000002e-06 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127312+0000", 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1431000000000003e-05 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.127318+0000", 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6229999999999999e-06 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114337+0000", 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.987019172 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114352+0000", 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4966999999999999e-05 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59005 5.0 5.61e98650 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:44:29.182 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.00097224299998, 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95670884499999997, 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59005 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:44:29.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157690+0000", 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157691+0000", 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181e-06 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157717+0000", 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5612000000000001e-05 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157729+0000", 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157734+0000", 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1800000000000004e-06 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157780+0000", 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5779999999999999e-05 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.157787+0000", 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.815e-06 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114390+0000", 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.95660253100000003 2024-08-30T23:44:29.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114399+0000", 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1249999999999999e-06 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59006 5.0 5.aa303940 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 356.94036484700001, 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89614382599999998, 2024-08-30T23:44:29.187 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59006 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218297+0000", 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218298+0000", 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218324+0000", 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5857e-05 2024-08-30T23:44:29.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218335+0000", 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0716999999999999e-05 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218340+0000", 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9490000000000004e-06 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218423+0000", 2024-08-30T23:44:29.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3065000000000005e-05 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.218430+0000", 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1400000000000002e-06 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114428+0000", 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.89599851500000005 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.114441+0000", 2024-08-30T23:44:29.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2727e-05 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.c e389 from shard 0)", 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 219.98280803, 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.840740928, 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:29.192 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175855+0000", 2024-08-30T23:44:29.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175892+0000", 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7345000000000002e-05 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016334+0000", 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.84044191000000001 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016342+0000", 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0420000000000003e-06 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016459+0000", 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116371 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016469+0000", 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0713999999999999e-05 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016595+0000", 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012576500000000001 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.11 542/536 [PushOp(5:884f9bdc:::benchmark_data_smithi173_87639_object32045:head, version: 407'1909, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:884f9bdc:::benchmark_data_smithi173_87639_object32045:head@407'1909, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:40.779709+0000", 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.37895262000001, 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046096248, 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.196 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779709+0000", 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779709+0000", 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779711+0000", 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8509999999999999e-06 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779756+0000", 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5099999999999998e-05 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779818+0000", 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2032999999999994e-05 2024-08-30T23:44:29.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779841+0000", 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2836e-05 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.824706+0000", 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.044864397 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.824719+0000", 2024-08-30T23:44:29.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3073e-05 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.825806+0000", 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010869580000000001 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99099 5.1 5:87e96639:::benchmark_data_smithi173_87639_object18859:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038696+0000", 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.119965597, 2024-08-30T23:44:29.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63555977200000002, 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99099 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038696+0000", 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038696+0000", 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038698+0000", 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1000000000000001e-06 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038702+0000", 2024-08-30T23:44:29.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4279999999999998e-06 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038713+0000", 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0597e-05 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038720+0000", 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2729999999999997e-06 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.671475+0000", 2024-08-30T23:44:29.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63275495100000001 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.674256+0000", 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027814229999999999 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99121 5.1 5:865db40b:::benchmark_data_smithi173_87639_object18881:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:44:29.204 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.119926358, 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65404301300000001, 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99121 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.06e-07 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038744+0000", 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0180000000000001e-06 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038754+0000", 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3589999999999997e-06 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038760+0000", 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1e-06 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.689969+0000", 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65120950300000002 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.692779+0000", 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002809527 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99122 5.1 5:832b5c08:::benchmark_data_smithi173_87639_object18882:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.208 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038768+0000", 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.119893636, 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65762801900000001, 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99122 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038768+0000", 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038768+0000", 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038769+0000", 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3599999999999999e-07 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038771+0000", 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3549999999999999e-06 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038778+0000", 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.939e-06 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038784+0000", 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.693122+0000", 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65433889999999995 2024-08-30T23:44:29.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.696396+0000", 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003274019 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99195 5.1 5:812e864c:::benchmark_data_smithi173_87639_object18955:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038790+0000", 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.119871624, 2024-08-30T23:44:29.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68118318899999997, 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99195 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038790+0000", 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038790+0000", 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038791+0000", 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000001e-07 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038793+0000", 2024-08-30T23:44:29.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.261e-06 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038799+0000", 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.976e-06 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038804+0000", 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8779999999999997e-06 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.716446+0000", 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67764244299999998 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.719974+0000", 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035272369999999999 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99201 5.1 5:82ea3608:::benchmark_data_smithi173_87639_object18961:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:44:29.217 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.11985199999999, 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68427159100000001, 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99201 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.218 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9900000000000001e-07 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038813+0000", 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1509999999999998e-06 2024-08-30T23:44:29.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038820+0000", 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0430000000000002e-06 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038825+0000", 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9259999999999999e-06 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.720250+0000", 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68142502100000002 2024-08-30T23:44:29.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.723082+0000", 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028320509999999999 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99217 5.1 5:868d9725:::benchmark_data_smithi173_87639_object18977:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.11983140700001, 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68806426499999995, 2024-08-30T23:44:29.224 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99217 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038836+0000", 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1460000000000004e-06 2024-08-30T23:44:29.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038844+0000", 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8199999999999997e-06 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038849+0000", 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.955e-06 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.723373+0000", 2024-08-30T23:44:29.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68452376699999995 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.726895+0000", 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035221760000000001 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99248 5.1 5:83333dac:::benchmark_data_smithi173_87639_object19008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038855+0000", 2024-08-30T23:44:29.228 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.11980667500001, 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71883598999999998, 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99248 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038855+0000", 2024-08-30T23:44:29.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038855+0000", 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038856+0000", 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038858+0000", 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3190000000000002e-06 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038866+0000", 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3699999999999997e-06 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038870+0000", 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9019999999999998e-06 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.754877+0000", 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.716006633 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.757691+0000", 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028143650000000001 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99261 5.1 5:8451e403:::benchmark_data_smithi173_87639_object19021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.119784008, 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.72356547599999999, 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99261 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038881+0000", 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2539999999999999e-06 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038888+0000", 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1069999999999999e-06 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038893+0000", 2024-08-30T23:44:29.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9819999999999999e-06 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.758644+0000", 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71975094500000003 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.762444+0000", 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037997970000000002 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99284 5.1 5:817284a4:::benchmark_data_smithi173_87639_object19044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.119763315, 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.72655738000000003, 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99284 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038901+0000", 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0899999999999999e-06 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038907+0000", 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5910000000000003e-06 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038911+0000", 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4340000000000002e-06 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.762677+0000", 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.72376565299999995 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.765456+0000", 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027792110000000002 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.240 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.1 562/540 [PushOp(5:85a0f8be:::benchmark_data_smithi173_87639_object42058:head, version: 409'2536, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:85a0f8be:::benchmark_data_smithi173_87639_object42058:head@409'2536, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:16.142061+0000", 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.016601085999994, 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.69249434399999998, 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142061+0000", 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142061+0000", 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142062+0000", 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6599999999999996e-07 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142089+0000", 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7192999999999999e-05 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.819497+0000", 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67740837499999995 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.819514+0000", 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6929e-05 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.824723+0000", 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005208959 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.824928+0000", 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020489799999999999 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.834555+0000", 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096273239999999996 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.1 576/540 [PushOp(5:801bc841:::benchmark_data_smithi173_87639_object73320:head, version: 436'4483, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:801bc841:::benchmark_data_smithi173_87639_object73320:head@436'4483, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:36.896183+0000", 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.262479106999997, 2024-08-30T23:44:29.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88981909400000003, 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896183+0000", 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896183+0000", 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896184+0000", 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.052e-06 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896215+0000", 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0482000000000001e-05 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783063+0000", 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88684886399999996 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783077+0000", 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3526000000000001e-05 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783301+0000", 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022379299999999999 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783350+0000", 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9030000000000003e-05 2024-08-30T23:44:29.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.786002+0000", 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026523470000000002 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:144501 5.1 5:827f28b3:::benchmark_data_smithi173_87639_object64261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:57.229197+0000", 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 31.929464947, 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64186927699999996, 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:29.248 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 144501 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229197+0000", 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:29.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229197+0000", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229198+0000", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.235e-06 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229204+0000", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229217+0000", 2024-08-30T23:44:29.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3011e-05 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229223+0000", 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.404e-06 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229277+0000", 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3495000000000002e-05 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229412+0000", 2024-08-30T23:44:29.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013536900000000001 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229538+0000", 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125399 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.231599+0000", 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020610350000000001 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.231790+0000", 2024-08-30T23:44:29.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000191387 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.870974+0000", 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63918373100000003 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.871023+0000", 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9081e-05 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.871066+0000", 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3579999999999999e-05 2024-08-30T23:44:29.253 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:29.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:29.254 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:44:29.255 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:44:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.256 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:29.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:29.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:29.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:29.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:29.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:29.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/84995469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:44:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:44:29.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:44:29.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/84995469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:44:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:44:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:44:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:44:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:44:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:44:29.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.576 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.576 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.576 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:29.576 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/84995469' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:44:29.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:44:29.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:29.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:29.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:29.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:29.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:29.901 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:29.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:30.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:30.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:30.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:30.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:30.243 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:30.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:30.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:30.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:30.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/84995469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: osdmap e611: 12 total, 11 up, 12 in 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: 5.c scrub starts 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: 5.c scrub ok 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: pgmap v1370: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 371 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 531/17766 objects degraded (2.989%); 581/17766 objects misplaced (3.270%); 0 B/s, 48 objects/s recovering 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:44:30.396 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:30 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:44:30.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:30.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.1.f.5 2024-08-30T23:44:30.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/84995469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: osdmap e611: 12 total, 11 up, 12 in 2024-08-30T23:44:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: 5.c scrub starts 2024-08-30T23:44:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: 5.c scrub ok 2024-08-30T23:44:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: pgmap v1370: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 371 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 531/17766 objects degraded (2.989%); 581/17766 objects misplaced (3.270%); 0 B/s, 48 objects/s recovering 2024-08-30T23:44:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:44:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:30 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:44:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.541 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:30.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:30.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/84995469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: osdmap e611: 12 total, 11 up, 12 in 2024-08-30T23:44:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: 5.c scrub starts 2024-08-30T23:44:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: 5.c scrub ok 2024-08-30T23:44:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: pgmap v1370: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 371 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 750 op/s; 531/17766 objects degraded (2.989%); 581/17766 objects misplaced (3.270%); 0 B/s, 48 objects/s recovering 2024-08-30T23:44:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:44:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:30 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:44:30.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:30.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:30.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:30.794 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:30.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:30.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:30.923 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:30.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:30.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:44:31.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:31.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:31.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:31.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:31.238 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:31.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:31.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 5.3 scrub starts 2024-08-30T23:44:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 5.3 scrub ok 2024-08-30T23:44:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:44:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 5.1 scrub starts 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: 5.1 scrub ok 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 531/17766 objects degraded (2.989%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:31 smithi106 ceph-mon[16603]: osdmap e612: 12 total, 11 up, 12 in 2024-08-30T23:44:31.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:31.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:31.501 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 5.3 scrub starts 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 5.3 scrub ok 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 5.1 scrub starts 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: 5.1 scrub ok 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:31.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 531/17766 objects degraded (2.989%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:31.503 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:31 smithi136 ceph-mon[20207]: osdmap e612: 12 total, 11 up, 12 in 2024-08-30T23:44:31.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:31.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 5.3 scrub starts 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 5.3 scrub ok 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 5.1 scrub starts 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: 5.1 scrub ok 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 531/17766 objects degraded (2.989%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:31.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:31 smithi173 ceph-mon[20554]: osdmap e612: 12 total, 11 up, 12 in 2024-08-30T23:44:31.598 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:31.599 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:31.628 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:31.628 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:31.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:31.656 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:31.656 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:31.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:31.683 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:31.683 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:31.710 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:31.710 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:31.739 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:31.739 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:31.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:31.766 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:31.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:31.791 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:31.791 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:31.794 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:31.794 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:31.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:31.828 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:31.829 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:31.858 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:31.858 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:31.890 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:31.890 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:31.913 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:31.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:31.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:32.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:32.063 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:32.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:44:32.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:32.272 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:32.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:32.394 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:32.463 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:32.463 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:32.463 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: 5.11 scrub starts 2024-08-30T23:44:32.463 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: 5.11 scrub ok 2024-08-30T23:44:32.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: from='client.26921 172.21.15.106:0/3787406767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:32.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: pgmap v1372: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 172 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 205/8238 objects degraded (2.488%); 232/8238 objects misplaced (2.816%); 0 B/s, 48 objects/s recovering 2024-08-30T23:44:32.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: 5.2 scrub starts 2024-08-30T23:44:32.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: 5.2 scrub ok 2024-08-30T23:44:32.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:32.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:32.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.519 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:32.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: 5.11 scrub starts 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: 5.11 scrub ok 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: from='client.26921 172.21.15.106:0/3787406767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: pgmap v1372: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 172 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 205/8238 objects degraded (2.488%); 232/8238 objects misplaced (2.816%); 0 B/s, 48 objects/s recovering 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: 5.2 scrub starts 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: 5.2 scrub ok 2024-08-30T23:44:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:32.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: 5.11 scrub starts 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: 5.11 scrub ok 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: from='client.26921 172.21.15.106:0/3787406767' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: pgmap v1372: 47 pgs: 1 active+undersized, 2 active+clean+remapped, 3 active+undersized+degraded, 41 active+clean; 172 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.12k op/s; 205/8238 objects degraded (2.488%); 232/8238 objects misplaced (2.816%); 0 B/s, 48 objects/s recovering 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: 5.2 scrub starts 2024-08-30T23:44:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: 5.2 scrub ok 2024-08-30T23:44:32.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:32.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:32.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:32.641 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:32.704 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 80237 objects 2024-08-30T23:44:32.705 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :319.692 2024-08-30T23:44:32.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:32.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:32.867 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:32.867 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:32.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:32.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:32.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:32.969 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:32.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:33.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:33.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.143 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:33.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:33.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:33.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:33.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:44:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:44:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: osdmap e613: 12 total, 11 up, 12 in 2024-08-30T23:44:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:44:33.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:33 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:44:33.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: osdmap e613: 12 total, 11 up, 12 in 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:44:33.376 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:33 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:44:33.395 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:33.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:44:33.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:33.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: osdmap e613: 12 total, 11 up, 12 in 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:44:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:33 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:44:33.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:33.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:33.727 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:33.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:33.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:33.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:33.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:33.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:33.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:34.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:34.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:34.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:34.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:34.228 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:34.228 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:34.264 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:44:34.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:34.288 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:34.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:34.413 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:34.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: 5.15 scrub starts 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: 5.15 scrub ok 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: 5.16 scrub starts 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: 5.16 scrub ok 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: osdmap e614: 12 total, 11 up, 12 in 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: pgmap v1375: 47 pgs: 2 active+undersized, 2 active+clean+remapped, 2 active+undersized+degraded, 41 active+clean; 40 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 46/1908 objects degraded (2.411%); 102/1908 objects misplaced (5.346%) 2024-08-30T23:44:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3142739949' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:34.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: 5.15 scrub starts 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: 5.15 scrub ok 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: 5.16 scrub starts 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: 5.16 scrub ok 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: osdmap e614: 12 total, 11 up, 12 in 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: pgmap v1375: 47 pgs: 2 active+undersized, 2 active+clean+remapped, 2 active+undersized+degraded, 41 active+clean; 40 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 46/1908 objects degraded (2.411%); 102/1908 objects misplaced (5.346%) 2024-08-30T23:44:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3142739949' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: 5.15 scrub starts 2024-08-30T23:44:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: 5.15 scrub ok 2024-08-30T23:44:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:44:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:44:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: 5.16 scrub starts 2024-08-30T23:44:34.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: 5.16 scrub ok 2024-08-30T23:44:34.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: osdmap e614: 12 total, 11 up, 12 in 2024-08-30T23:44:34.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: pgmap v1375: 47 pgs: 2 active+undersized, 2 active+clean+remapped, 2 active+undersized+degraded, 41 active+clean; 40 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 46/1908 objects degraded (2.411%); 102/1908 objects misplaced (5.346%) 2024-08-30T23:44:34.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3142739949' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:34.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:34.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:34.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:34.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:34.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:44:34.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:44:34.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:34.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:34.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:35.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:35.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:35.115 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:35.131 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:35.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:35.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:35.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:35.293 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:35.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:35.447 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:35.477 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:44:35.477 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:44:35.477 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-08-30T23:44:35.477 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:44:35.478 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:44:35.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-08-30T23:44:35.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:44:35.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:44:35.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:44:35.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:44:35.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:44:35.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3142739949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:35.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:35 smithi106 ceph-mon[16603]: osdmap e615: 12 total, 11 up, 12 in 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3142739949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:35 smithi136 ceph-mon[20207]: osdmap e615: 12 total, 11 up, 12 in 2024-08-30T23:44:35.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:35.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3142739949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:35 smithi173 ceph-mon[20554]: osdmap e615: 12 total, 11 up, 12 in 2024-08-30T23:44:35.597 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:35.597 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:35.634 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:35.635 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:35.637 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:35.637 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:35.658 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:35.658 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:35.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:35.678 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:35.679 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:35.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:35.705 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:35.705 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:35.705 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:35.733 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:35.734 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:35.755 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:35.756 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:35.758 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:35.777 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:35.778 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:35.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:35.800 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:35.801 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:35.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:35.822 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:35.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:35.845 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:35.846 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:35.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:35.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:35.947 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:36.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:36.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:36.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:36.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:36.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:44:36.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:36.318 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:36.408 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:44:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:44:36.408 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:44:36.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:36.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:36.508 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 5.13 scrub starts 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 5.13 scrub ok 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 5.19 scrub starts 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 5.19 scrub ok 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 5.17 scrub starts 2024-08-30T23:44:36.534 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: 5.17 scrub ok 2024-08-30T23:44:36.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1916822991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:36.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: pgmap v1377: 47 pgs: 2 active+undersized, 2 active+clean+remapped, 2 active+undersized+degraded, 41 active+clean; 16 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 46/771 objects degraded (5.966%) 2024-08-30T23:44:36.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 46/1908 objects degraded (2.411%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:36.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:36 smithi106 ceph-mon[16603]: osdmap e616: 12 total, 11 up, 12 in 2024-08-30T23:44:36.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 5.13 scrub starts 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 5.13 scrub ok 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 5.19 scrub starts 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 5.19 scrub ok 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 5.17 scrub starts 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: 5.17 scrub ok 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1916822991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: pgmap v1377: 47 pgs: 2 active+undersized, 2 active+clean+remapped, 2 active+undersized+degraded, 41 active+clean; 16 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 46/771 objects degraded (5.966%) 2024-08-30T23:44:36.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 46/1908 objects degraded (2.411%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:36.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:36 smithi136 ceph-mon[20207]: osdmap e616: 12 total, 11 up, 12 in 2024-08-30T23:44:36.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:36.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 5.13 scrub starts 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 5.13 scrub ok 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 5.19 scrub starts 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 5.19 scrub ok 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 5.17 scrub starts 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: 5.17 scrub ok 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1916822991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: pgmap v1377: 47 pgs: 2 active+undersized, 2 active+clean+remapped, 2 active+undersized+degraded, 41 active+clean; 16 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 46/771 objects degraded (5.966%) 2024-08-30T23:44:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 46/1908 objects degraded (2.411%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:36 smithi173 ceph-mon[20554]: osdmap e616: 12 total, 11 up, 12 in 2024-08-30T23:44:36.669 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:36.790 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:36.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.816 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:36.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:36.891 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-08-30T23:44:36.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-08-30T23:44:36.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:36.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:44:36.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:36.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:37.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:37.078 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:44:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:44:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:44:37.078 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:44:37.080 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:37.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:37.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:37.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:37.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:37.407 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:37 smithi106 ceph-mon[16603]: 5.f scrub starts 2024-08-30T23:44:37.407 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:37 smithi106 ceph-mon[16603]: 5.f scrub ok 2024-08-30T23:44:37.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:37.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:37.473 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:37 smithi136 ceph-mon[20207]: 5.f scrub starts 2024-08-30T23:44:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:37 smithi136 ceph-mon[20207]: 5.f scrub ok 2024-08-30T23:44:37.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:37 smithi173 ceph-mon[20554]: 5.f scrub starts 2024-08-30T23:44:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:37 smithi173 ceph-mon[20554]: 5.f scrub ok 2024-08-30T23:44:37.578 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:37.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:37.701 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:37.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:44:37.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:37.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:37.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:37.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:37.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:38.013 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:38.044 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:38.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.080 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:38.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:38.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:38.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:38.281 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_3' removed 2024-08-30T23:44:38.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:38.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:38.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:38 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:44:38.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:38 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:44:38.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:38 smithi136 ceph-mon[20207]: pgmap v1379: 47 pgs: 4 active+undersized, 2 active+clean+remapped, 41 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 356 op/s 2024-08-30T23:44:38.394 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/466678259' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:44:38.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:38.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:38.502 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:44:38.502 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:44:38.502 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:44:38.502 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:44:38.502 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:44:38.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:38 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:44:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:38 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:44:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:38 smithi173 ceph-mon[20554]: pgmap v1379: 47 pgs: 4 active+undersized, 2 active+clean+remapped, 41 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 356 op/s 2024-08-30T23:44:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/466678259' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:44:38.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:38 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:44:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:38 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:44:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:38 smithi106 ceph-mon[16603]: pgmap v1379: 47 pgs: 4 active+undersized, 2 active+clean+remapped, 41 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 356 op/s 2024-08-30T23:44:38.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/466678259' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:44:38.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:38.705 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:38.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:38.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:44:38.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:38.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:38.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:38.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:38.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:39.031 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:44:39.032 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:44:39.033 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-08-30T23:44:39.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-08-30T23:44:39.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:39.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:39.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:44:39.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:39.204 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:39.271 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:39.302 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:39.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:39.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:39.385 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:44:39.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:39.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:39.505 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 46/771 objects degraded (5.966%), 2 pgs degraded) 2024-08-30T23:44:39.505 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/466678259' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:44:39.505 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: osdmap e617: 12 total, 11 up, 12 in 2024-08-30T23:44:39.506 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:44:39.506 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:44:39.506 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:44:39.506 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:39 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:44:39.545 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 46/771 objects degraded (5.966%), 2 pgs degraded) 2024-08-30T23:44:39.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/466678259' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:44:39.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: osdmap e617: 12 total, 11 up, 12 in 2024-08-30T23:44:39.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:44:39.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:44:39.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:44:39.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:39 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:44:39.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 46/771 objects degraded (5.966%), 2 pgs degraded) 2024-08-30T23:44:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/466678259' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:44:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: osdmap e617: 12 total, 11 up, 12 in 2024-08-30T23:44:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:44:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:44:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:44:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:39 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:44:39.612 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:39.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:39.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:39.834 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61137 4.0 4:05184379:::benchmark_data_smithi173_74914_object61136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 585.42046378099997, 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0987384589999998, 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:39.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61137 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413066+0000", 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413067+0000", 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2640000000000001e-06 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413084+0000", 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6407999999999999e-05 2024-08-30T23:44:39.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413095+0000", 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1277999999999999e-05 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413117+0000", 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1498e-05 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413168+0000", 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1266e-05 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413265+0000", 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6620000000000007e-05 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.413359+0000", 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4003000000000001e-05 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415078+0000", 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001719478 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:54.415468+0000", 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038999199999999998 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511679+0000", 2024-08-30T23:44:39.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0962105000000002 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511758+0000", 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9426999999999997e-05 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511805+0000", 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6724999999999997e-05 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:61138 4.0 4:0601b2bd:::benchmark_data_smithi173_74914_object61137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 584.67980902199997, 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582538660000001, 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61138 2024-08-30T23:44:39.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153721+0000", 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153722+0000", 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-06 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153826+0000", 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103683 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153839+0000", 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3169e-05 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153863+0000", 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4326000000000001e-05 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.153921+0000", 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7283e-05 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154051+0000", 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012986399999999999 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.154154+0000", 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103188 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156034+0000", 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018801009999999999 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:55.156113+0000", 2024-08-30T23:44:39.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8807000000000006e-05 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511881+0000", 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3557679949999999 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511932+0000", 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1134999999999997e-05 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:34:56.511975+0000", 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2973999999999999e-05 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:196473 4.0 4:0415f716:::benchmark_data_smithi173_74914_object98231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 441.80926938099998, 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49994438299999999, 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:44:39.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 196473 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024266+0000", 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1779999999999996e-06 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024277+0000", 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1567e-05 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024284+0000", 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4389999999999997e-06 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024336+0000", 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2145000000000003e-05 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024443+0000", 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107633 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024556+0000", 2024-08-30T23:44:39.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011263 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029319+0000", 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047626949999999999 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029471+0000", 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015204800000000001 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524119+0000", 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49464797199999999 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524156+0000", 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7716000000000003e-05 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524205+0000", 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8591000000000003e-05 2024-08-30T23:44:39.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 451/432 [PushOp(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head, version: 400'576, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head@400'576, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 312.234000993, 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44893841800000001, 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599530+0000", 2024-08-30T23:44:39.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4339999999999999e-06 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599595+0000", 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4184999999999996e-05 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046675+0000", 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.447080796 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046706+0000", 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0505999999999999e-05 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046916+0000", 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021055600000000001 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046944+0000", 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7492999999999999e-05 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.048467+0000", 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015234479999999999 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 464/432 [PushOp(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head, version: 408'2477, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head@408'2477, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 266.756246992, 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52881942000000004, 2024-08-30T23:44:39.859 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077310+0000", 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6896000000000001e-05 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077371+0000", 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1471999999999997e-05 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604348+0000", 2024-08-30T23:44:39.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52697629300000004 2024-08-30T23:44:39.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:39.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604378+0000", 2024-08-30T23:44:39.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0654999999999997e-05 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604592+0000", 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021338600000000001 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604609+0000", 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7244999999999998e-05 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.606102+0000", 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014934729999999999 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "replica_scrub(pg: 5.1,from:0'0,to:445'5203,epoch:470/389,start:5:8a8f2f50:::benchmark_data_smithi173_87639_object56981:0,end:5:8ae6a8d1:::benchmark_data_smithi173_87639_object64309:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 252.71070966900001, 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74865697399999998, 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.865 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122821+0000", 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7599999999999997e-07 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122828+0000", 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-06 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871217+0000", 2024-08-30T23:44:39.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74838935200000001 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871235+0000", 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7708999999999999e-05 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871407+0000", 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172153 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871447+0000", 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9582999999999997e-05 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871477+0000", 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0501000000000002e-05 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.869 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.9 e488 from shard 6)", 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 208.95618146199999, 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49092997399999999, 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.34e-07 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877368+0000", 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8964999999999999e-05 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368058+0000", 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.490690554 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368065+0000", 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.72e-06 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368139+0000", 2024-08-30T23:44:39.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3943000000000007e-05 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368148+0000", 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.67e-06 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368278+0000", 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013068800000000001 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e502,491 5:9f5e9ce2:::benchmark_data_smithi173_87639_object5778:head,446'5327)", 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 186.85869485500001, 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80846543500000001, 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:44:39.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9400000000000001e-07 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974839+0000", 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1080000000000001e-06 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783082+0000", 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80824320199999999 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783093+0000", 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1122e-05 2024-08-30T23:44:39.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783147+0000", 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4397000000000001e-05 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783301+0000", 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015331200000000001 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 505/491 [PushOp(5:0a8a0c4f:::benchmark_data_smithi173_87639_object7264:head, version: 399'488, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0a8a0c4f:::benchmark_data_smithi173_87639_object7264:head@399'488, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 183.572524724, 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76577126500000003, 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:44:39.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261006+0000", 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9699999999999996e-07 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261041+0000", 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4728999999999999e-05 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025849+0000", 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76480819099999997 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025877+0000", 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8016000000000001e-05 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025956+0000", 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9369000000000005e-05 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025971+0000", 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4516e-05 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.026777+0000", 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080584700000000005 2024-08-30T23:44:39.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 511/491 [PushOp(5:0e383089:::benchmark_data_smithi173_87639_object11978:head, version: 401'782, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0e383089:::benchmark_data_smithi173_87639_object11978:head@401'782, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 177.51851029599999, 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88692451299999997, 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315021+0000", 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3859999999999999e-06 2024-08-30T23:44:39.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315100+0000", 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9388000000000006e-05 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200480+0000", 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88538003499999995 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200526+0000", 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5198999999999999e-05 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200744+0000", 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021846300000000001 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200767+0000", 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2996000000000001e-05 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.201944+0000", 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001177046 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 511/491 [PushOp(5:1e005da2:::benchmark_data_smithi173_87639_object35084:head, version: 407'2211, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1e005da2:::benchmark_data_smithi173_87639_object35084:head@407'2211, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 174.45690374200001, 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42476207799999999, 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376636+0000", 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0217000000000001e-05 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376737+0000", 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100571 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800459+0000", 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.423722339 2024-08-30T23:44:39.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800470+0000", 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0519e-05 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800543+0000", 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2799999999999994e-05 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800557+0000", 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4208e-05 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.801388+0000", 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083142399999999999 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.890 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 518/491 [PushOp(5:1ade8763:::benchmark_data_smithi173_87639_object73716:head, version: 436'4607, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1ade8763:::benchmark_data_smithi173_87639_object73716:head@436'4607, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.77050541, 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59454703399999997, 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8400000000000004e-07 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063054+0000", 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9164999999999999e-05 2024-08-30T23:44:39.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656490+0000", 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59343594399999999 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656518+0000", 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8041000000000001e-05 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656611+0000", 2024-08-30T23:44:39.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3068999999999999e-05 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656622+0000", 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0903999999999999e-05 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.657572+0000", 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094932700000000003 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 519/491 [PushOp(5:092b6415:::benchmark_data_smithi173_87639_object36917:head, version: 408'2349, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:092b6415:::benchmark_data_smithi173_87639_object36917:head@408'2349, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 157.27641286100001, 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68386947499999995, 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:44:39.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557119+0000", 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4670000000000001e-06 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557185+0000", 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6804999999999995e-05 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239646+0000", 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68246094099999999 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239664+0000", 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8122999999999999e-05 2024-08-30T23:44:39.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239802+0000", 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013746299999999999 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239814+0000", 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2029e-05 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.240987+0000", 2024-08-30T23:44:39.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001172647 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(5.18 e520,491 5:1cf428d8:::benchmark_data_smithi173_87639_object1164:head,440'5076)", 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.65224204099999, 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54704231800000003, 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181289+0000", 2024-08-30T23:44:39.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.387e-06 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181305+0000", 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5756e-05 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728218+0000", 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54691292999999996 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728224+0000", 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3759999999999996e-06 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728270+0000", 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6016000000000001e-05 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728277+0000", 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728330+0000", 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3631000000000002e-05 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.b 542/530 [PushOp(5:d3ac1e69:::benchmark_data_smithi173_87639_object22833:head, version: 405'1449, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:d3ac1e69:::benchmark_data_smithi173_87639_object22833:head@405'1449, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:40.832451+0000", 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 119.001078896, 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.75528995899999996, 2024-08-30T23:44:39.903 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832451+0000", 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832451+0000", 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832452+0000", 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7599999999999997e-07 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832487+0000", 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5097e-05 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585781+0000", 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.75329446700000002 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585798+0000", 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6438000000000001e-05 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585874+0000", 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6730000000000006e-05 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585883+0000", 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7290000000000006e-06 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.587741+0000", 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001858022 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.907 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e590,567 5:9ca44d88:::benchmark_data_smithi173_87639_object2291:head,441'5121)", 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.665602462999999, 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63675648500000004, 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4499999999999995e-07 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167933+0000", 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9579999999999998e-06 2024-08-30T23:44:39.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.803817+0000", 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63588312800000002 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.803827+0000", 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148e-05 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.803915+0000", 2024-08-30T23:44:39.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8802999999999997e-05 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.804684+0000", 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076850300000000002 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e597,595 5:9ded4f6f:::benchmark_data_smithi173_87639_object57150:head,580'6894)", 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:07.363461+0000", 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.470069008999999, 2024-08-30T23:44:39.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61280746100000005, 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363461+0000", 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363461+0000", 2024-08-30T23:44:39.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363462+0000", 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2699999999999999e-07 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363466+0000", 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0899999999999998e-06 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976023+0000", 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61255709400000002 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976033+0000", 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.005e-05 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976086+0000", 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3375999999999999e-05 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976268+0000", 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018222400000000001 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e598,595 5:9ccfef18:::benchmark_data_smithi173_87639_object59658:head,582'6977)", 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:08.643616+0000", 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 31.189913876999999, 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60924447699999995, 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.915 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643616+0000", 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643616+0000", 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643617+0000", 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643620+0000", 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5159999999999999e-06 2024-08-30T23:44:39.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252614+0000", 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60899351999999995 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252638+0000", 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4556999999999999e-05 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252689+0000", 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0689000000000002e-05 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252861+0000", 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000171609 2024-08-30T23:44:39.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e598,595 5:988025b4:::benchmark_data_smithi173_87639_object59776:head,582'6978)", 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:08.645761+0000", 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 31.187768905999999, 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60965304099999995, 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:39.921 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645761+0000", 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645761+0000", 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645762+0000", 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8700000000000005e-07 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645766+0000", 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1339999999999997e-06 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255226+0000", 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.609460059 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255234+0000", 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7519999999999996e-06 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255277+0000", 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3739999999999998e-05 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255414+0000", 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013666899999999999 2024-08-30T23:44:39.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e599,595 5:9f3d16fd:::benchmark_data_smithi173_87639_object64699:head,592'7133)", 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.479004156999999, 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.420813033, 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:44:39.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354527+0000", 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:44:39.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354530+0000", 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8600000000000003e-06 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775055+0000", 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42052456799999999 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775063+0000", 2024-08-30T23:44:39.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775123+0000", 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9885e-05 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775339+0000", 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000216188 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:44:39.929 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:44:39.929 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:44:39.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:39.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:39.965 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:39.965 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:39.986 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:39.986 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:40.006 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:40.006 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:40.027 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:40.048 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:40.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:40.049 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:40.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:40.070 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:40.070 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:40.091 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:40.113 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:40.135 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:40.156 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:40.156 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:40.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:40.183 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:40.183 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:40.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:40.288 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:44:40.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:40.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/675899664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:44:40.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:40 smithi106 ceph-mon[16603]: pgmap v1381: 21 pgs: 1 active+undersized, 20 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:40.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2065078284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:40.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:40.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/675899664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:44:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:40 smithi136 ceph-mon[20207]: pgmap v1381: 21 pgs: 1 active+undersized, 20 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2065078284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:40.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:40.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/675899664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:44:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:40 smithi173 ceph-mon[20554]: pgmap v1381: 21 pgs: 1 active+undersized, 20 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2065078284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:40.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:40.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:44:40.700 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:40.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:40.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:40.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:40.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:40.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:40.969 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:41.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:41.074 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:41.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:41.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:41.291 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_4' created 2024-08-30T23:44:41.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:41.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:41.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:41 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:41.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/675899664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:44:41.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:41 smithi173 ceph-mon[20554]: osdmap e618: 12 total, 11 up, 12 in 2024-08-30T23:44:41.390 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3897441237' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-30T23:44:41.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:41.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:41.491 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:41.491 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:41 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/675899664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:44:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:41 smithi136 ceph-mon[20207]: osdmap e618: 12 total, 11 up, 12 in 2024-08-30T23:44:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3897441237' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-30T23:44:41.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:41.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:41 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/675899664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:44:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:41 smithi106 ceph-mon[16603]: osdmap e618: 12 total, 11 up, 12 in 2024-08-30T23:44:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3897441237' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-30T23:44:41.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:41.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:41.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:41.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.816 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:41.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:41.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:41.939 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:41.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:42.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:42.062 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:42.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:42.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:44:42.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-08-30T23:44:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:42.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:42.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:42.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:42.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:42.416 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:42.433 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:42.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:42.441 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:44:42.441 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:44:42.441 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T23:44:42.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:44:42.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3897441237' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-30T23:44:42.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:42 smithi106 ceph-mon[16603]: osdmap e619: 12 total, 11 up, 12 in 2024-08-30T23:44:42.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:42 smithi106 ceph-mon[16603]: pgmap v1384: 37 pgs: 2 creating+peering, 14 unknown, 1 active+undersized, 20 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:42.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:42.538 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3897441237' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-30T23:44:42.538 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:42 smithi173 ceph-mon[20554]: osdmap e619: 12 total, 11 up, 12 in 2024-08-30T23:44:42.538 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:42 smithi173 ceph-mon[20554]: pgmap v1384: 37 pgs: 2 creating+peering, 14 unknown, 1 active+undersized, 20 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:42.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:42.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:42.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:42.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3897441237' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-30T23:44:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:42 smithi136 ceph-mon[20207]: osdmap e619: 12 total, 11 up, 12 in 2024-08-30T23:44:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:42 smithi136 ceph-mon[20207]: pgmap v1384: 37 pgs: 2 creating+peering, 14 unknown, 1 active+undersized, 20 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.807 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:42.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:42.898 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:42.899 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:42.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:42.921 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:42.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:42.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:42.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:44:43.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:43.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:43.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:43.181 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:43.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:43.304 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-08-30T23:44:43.305 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:43.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:43.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:43.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:43.548 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 621, 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:44:43.298107+0000", 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:44:02.874029+0000", 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:38:32.101509+0000", 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5801984, 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:44:43.549 INFO:teuthology.orchestra.run.smithi106.stdout: "noscrub", 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 28, 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 6, 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:44:43.550 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "jewel", 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:44:43.551 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 21, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 21, 2024-08-30T23:44:43.552 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 21, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 21, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 21, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "509", 2024-08-30T23:44:43.553 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "507", 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:44:43.554 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:44:43.555 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:44:43.556 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.9099999666213989, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.9099999666213989, 2024-08-30T23:44:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.0999999046325684, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.0999999046325684, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.71190768480300903, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.85000002384185791, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.70999997854232788 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 6, 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_4", 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:44:40.370699+0000", 2024-08-30T23:44:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:44:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "621", 2024-08-30T23:44:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:44:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:44:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.130000114440918, 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 3.130000114440918, 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.440000057220459, 2024-08-30T23:44:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3.440000057220459, 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.71190768480300903, 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.85000002384185791, 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.70999997854232788 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.2807464599609375, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 539, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:43 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:44:43.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:43 smithi106 ceph-mon[16603]: osdmap e620: 12 total, 11 up, 12 in 2024-08-30T23:44:43.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3307260718' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:44:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:44:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:44:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-30T23:44:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 594, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:44:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:44:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:44:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.426727294921875, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 39, 2024-08-30T23:44:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 446, 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 566, 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 573, 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/1428551692", 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/1428551692", 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/1428551692", 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/1428551692", 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7101593017578125, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7503204345703125, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:44:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:44:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:44:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.2454376220703125, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 595, 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:44:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:44:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:44:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:44:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:44:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 587, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:44:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:44:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:01.788525+0000", 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:31.939564+0000", 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:09.378080+0000", 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:52.250956+0000", 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:51.029648+0000", 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:22.725412+0000", 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:49.490054+0000", 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:46.410922+0000", 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:44:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:28.335352+0000", 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:40.594790+0000", 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:32.327364+0000", 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:30.306781+0000", 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [], 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:44:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:44:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:44:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.655 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:43.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:43.656 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:43 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:44:43.656 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:43 smithi173 ceph-mon[20554]: osdmap e620: 12 total, 11 up, 12 in 2024-08-30T23:44:43.656 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3307260718' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:44:43.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:44:43.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:44:43.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:43.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:43 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:44:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:43 smithi136 ceph-mon[20207]: osdmap e620: 12 total, 11 up, 12 in 2024-08-30T23:44:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3307260718' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:44:43.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:43.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:43.902 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:43.940 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:44.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:44.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:44.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:44.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:44.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:44.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:44.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3307260718' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:44:44.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:44 smithi173 ceph-mon[20554]: osdmap e621: 12 total, 11 up, 12 in 2024-08-30T23:44:44.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3588514072' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:44:44.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:44.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:44.408 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:44.409 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:44.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:44.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:44.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3307260718' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:44:44.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:44 smithi106 ceph-mon[16603]: osdmap e621: 12 total, 11 up, 12 in 2024-08-30T23:44:44.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3588514072' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:44:44.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:44.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:44.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:44.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3307260718' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:44:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:44 smithi136 ceph-mon[20207]: osdmap e621: 12 total, 11 up, 12 in 2024-08-30T23:44:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3588514072' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:44:44.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:44.910 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:44.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:44.931 INFO:tasks.thrashosds.thrasher:j is {'epoch': 621, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:44:43.298107+0000', 'last_up_change': '2024-08-30T23:44:02.874029+0000', 'last_in_change': '2024-08-30T23:38:32.101509+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '509', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '507', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.0999999046325684, 'raw_score_stable': 2.0999999046325684, 'primary_affinity_weighted': 0.711907684803009, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7099999785423279}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-08-30T23:44:40.370699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '621', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.130000114440918, 'score_stable': 3.130000114440918, 'optimal_score': 1.100000023841858, 'raw_score_acting': 3.440000057220459, 'raw_score_stable': 3.440000057220459, 'primary_affinity_weighted': 0.711907684803009, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7099999785423279}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 0.2807464599609375, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 619, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 594, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 0, 'in': 1, 'weight': 0.426727294921875, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 446, 'up_from': 566, 'up_thru': 573, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 1428551692}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 1428551692}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 1428551692}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 1428551692}]}, 'public_addr': '172.21.15.136:6809/1428551692', 'cluster_addr': '172.21.15.136:6811/1428551692', 'heartbeat_back_addr': '172.21.15.136:6815/1428551692', 'heartbeat_front_addr': '172.21.15.136:6813/1428551692', 'state': ['exists']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 0.7101593017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 0.7503204345703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2454376220703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 595, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 587, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:01.788525+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:31.939564+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:09.378080+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:52.250956+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:51.029648+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:22.725412+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:49.490054+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:46.410922+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:28.335352+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:40.594790+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:32.327364+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:30.306781+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:44:44.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:44:44.945 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:44.956 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:45.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:45.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:45.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:44:45.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:45.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:45.248 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-08-30T23:44:45.251 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:44:45.251 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:44:45.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:45.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:45.316 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:45.368 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:44:45.368 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:44:45.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_114000 2024-08-30T23:44:45.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:44:45.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:44:45.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:45.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:45.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:45.546 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: pgmap v1387: 37 pgs: 2 active+undersized, 3 creating+activating, 3 creating+peering, 5 unknown, 24 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:44:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=53,logm=1} crc {auth=250844572,config=1460075751,health=3191995951,kv=1091147716,logm=1930437604}) 2024-08-30T23:44:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586615471}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462695514}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=26241341}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3683691017}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295365647}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452336558}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=995396322}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667772139}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859183966}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559456010}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772743272}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2686031307}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2040345007}) 2024-08-30T23:44:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=470911063}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3355303277}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074355462}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4122127040}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1588993151}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1541040642}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279144253}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359098647}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085778221}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1772465508}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3101970132}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1731380643}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826645911}) 2024-08-30T23:44:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679656024}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2549271812}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371814138}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877423263}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2133612527}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=798528747}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3417227620}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811418229}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440852580}) 2024-08-30T23:44:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644089158}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2709821046}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3035606184}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374030433}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248472651}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705375252}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519218550}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758265663}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2738363700}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027341331}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960405608}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1285744694}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2947646866}) 2024-08-30T23:44:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427664610}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971220639}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=812683281}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145234969}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=3418899150,mdsmap=425822562,mgr=2437212703}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=79} crc {mgr=3510304847,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=2009673873,mon_config_key=3073340454}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1393863943}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3394839728}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=700201789,monmap=1193225519,osd_metadata=3035196017,osd_pg_creating=2719859235,osdmap=3948633623}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2880553573}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1116486778}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=515263393}) 2024-08-30T23:44:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=12} crc {osdmap=2037548287}) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: pgmap v1387: 37 pgs: 2 active+undersized, 3 creating+activating, 3 creating+peering, 5 unknown, 24 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=53,logm=1} crc {auth=250844572,config=1460075751,health=3191995951,kv=1091147716,logm=1930437604}) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586615471}) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462695514}) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=26241341}) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3683691017}) 2024-08-30T23:44:45.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295365647}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452336558}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=995396322}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667772139}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859183966}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559456010}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772743272}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2686031307}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2040345007}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=470911063}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3355303277}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074355462}) 2024-08-30T23:44:45.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4122127040}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1588993151}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1541040642}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279144253}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359098647}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085778221}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1772465508}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3101970132}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1731380643}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826645911}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679656024}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2549271812}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371814138}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877423263}) 2024-08-30T23:44:45.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2133612527}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=798528747}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3417227620}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811418229}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440852580}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644089158}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2709821046}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3035606184}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374030433}) 2024-08-30T23:44:45.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248472651}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705375252}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519218550}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758265663}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2738363700}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027341331}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960405608}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1285744694}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2947646866}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427664610}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971220639}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=812683281}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145234969}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=3418899150,mdsmap=425822562,mgr=2437212703}) 2024-08-30T23:44:45.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=79} crc {mgr=3510304847,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=2009673873,mon_config_key=3073340454}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1393863943}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3394839728}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=700201789,monmap=1193225519,osd_metadata=3035196017,osd_pg_creating=2719859235,osdmap=3948633623}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2880553573}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1116486778}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=515263393}) 2024-08-30T23:44:45.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=12} crc {osdmap=2037548287}) 2024-08-30T23:44:45.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:45.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:45.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: pgmap v1387: 37 pgs: 2 active+undersized, 3 creating+activating, 3 creating+peering, 5 unknown, 24 active+clean; 577 KiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:44:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:44:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=53,logm=1} crc {auth=250844572,config=1460075751,health=3191995951,kv=1091147716,logm=1930437604}) 2024-08-30T23:44:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1586615471}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2462695514}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=26241341}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3683691017}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3295365647}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=452336558}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=995396322}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667772139}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1859183966}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559456010}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3772743272}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2686031307}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2040345007}) 2024-08-30T23:44:45.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=470911063}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3355303277}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2074355462}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4122127040}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1588993151}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1541040642}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279144253}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359098647}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3085778221}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1772465508}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3101970132}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1731380643}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3826645911}) 2024-08-30T23:44:45.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679656024}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2549271812}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371814138}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877423263}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2133612527}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=798528747}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3417227620}) 2024-08-30T23:44:45.793 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811418229}) 2024-08-30T23:44:45.794 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1440852580}) 2024-08-30T23:44:45.794 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2644089158}) 2024-08-30T23:44:45.794 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2709821046}) 2024-08-30T23:44:45.794 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3035606184}) 2024-08-30T23:44:45.794 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1374030433}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1248472651}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705375252}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2519218550}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3758265663}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2738363700}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3027341331}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960405608}) 2024-08-30T23:44:45.795 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1285744694}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2947646866}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427664610}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1971220639}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=812683281}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4145234969}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=87,mdsmap=3,mgr=10} crc {logm=3418899150,mdsmap=425822562,mgr=2437212703}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=79} crc {mgr=3510304847,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=2009673873,mon_config_key=3073340454}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1393863943}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3394839728}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=700201789,monmap=1193225519,osd_metadata=3035196017,osd_pg_creating=2719859235,osdmap=3948633623}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2880553573}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1116486778}) 2024-08-30T23:44:45.796 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=515263393}) 2024-08-30T23:44:45.797 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=12} crc {osdmap=2037548287}) 2024-08-30T23:44:45.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:45.852 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:45.852 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:45.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:45.930 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:45.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:46.032 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.062 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:46.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:46.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:46.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:46.202 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:46.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:46.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:46.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 2070 2054 128.368 128.375 0.0209839 0.00767958 2024-08-30T23:44:46.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:46.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:46.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:46.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:44:46.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:46.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:46.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:46.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:46.811 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:44:46.818 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:46.876 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:46.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:46.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:46.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:46.999 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:44:47.000 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:44:47.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:47.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:47.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:47.047 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:47.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:47.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:46 smithi106 ceph-mon[16603]: pgmap v1388: 37 pgs: 2 active+undersized, 3 creating+activating, 3 creating+peering, 3 unknown, 26 active+clean; 2.4 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s wr, 5 op/s 2024-08-30T23:44:47.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:46 smithi106 ceph-mon[16603]: osdmap e622: 12 total, 11 up, 12 in 2024-08-30T23:44:47.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2162412897' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:47.068 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/548952394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:47.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:47.072 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:47.072 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:46 smithi173 ceph-mon[20554]: pgmap v1388: 37 pgs: 2 active+undersized, 3 creating+activating, 3 creating+peering, 3 unknown, 26 active+clean; 2.4 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s wr, 5 op/s 2024-08-30T23:44:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:46 smithi173 ceph-mon[20554]: osdmap e622: 12 total, 11 up, 12 in 2024-08-30T23:44:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2162412897' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:47.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/548952394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:47.097 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:47.097 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:47.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:47.121 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:47.122 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:47.145 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:47.145 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:47.167 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:47.168 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:47.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:47.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:47.188 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:47.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:47.208 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:47.208 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:47.229 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:47.229 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:47.250 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:47.250 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:46 smithi136 ceph-mon[20207]: pgmap v1388: 37 pgs: 2 active+undersized, 3 creating+activating, 3 creating+peering, 3 unknown, 26 active+clean; 2.4 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s wr, 5 op/s 2024-08-30T23:44:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:46 smithi136 ceph-mon[20207]: osdmap e622: 12 total, 11 up, 12 in 2024-08-30T23:44:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2162412897' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:44:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/548952394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:47.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:47.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:47.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3641 3625 113.274 98.1875 0.00763085 0.00879891 2024-08-30T23:44:47.402 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:44:47.402 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:44:47.402 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:44:47.403 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:44:47.403 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:44:47.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:47.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:47.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:47.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:47.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:47.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:47.709 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1390,"stamp":"2024-08-30T23:44:46.812377+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":41157152,"num_objects":621,"num_object_clones":0,"num_object_copies":1863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":619,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2551,"ondisk_log_size":2551,"up":108,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5079416,"kb_used_data":755384,"kb_used_omap":107,"kb_used_meta":4321748,"kb_avail":1119862408,"statfs":{"total":1151940427776,"available":1146739105792,"internally_reserved":0,"allocated":773513216,"data_stored":760275417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110317,"internal_metadata":4425470227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":619,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.201379"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":255,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-08-30T23:44:41.296312+0000","last_change":"2024-08-30T23:44:30.071083+0000","last_active":"2024-08-30T23:44:41.296312+0000","last_peered":"2024-08-30T23:44:41.296312+0000","last_clean":"2024-08-30T23:44:41.296312+0000","last_became_active":"2024-08-30T23:42:13.920647+0000","last_became_peered":"2024-08-30T23:42:13.920647+0000","last_unstale":"2024-08-30T23:44:41.296312+0000","last_undegraded":"2024-08-30T23:44:41.296312+0000","last_fullsized":"2024-08-30T23:44:41.296312+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:30.071017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:30.071017+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:32.648433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":224,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.812577+0000","last_change":"2024-08-30T23:44:29.831279+0000","last_active":"2024-08-30T23:44:45.812577+0000","last_peered":"2024-08-30T23:44:45.812577+0000","last_clean":"2024-08-30T23:44:45.812577+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:44:45.812577+0000","last_undegraded":"2024-08-30T23:44:45.812577+0000","last_fullsized":"2024-08-30T23:44:45.812577+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.831212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.831212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:57.676072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":266,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.807152+0000","last_change":"2024-08-30T23:44:31.878852+0000","last_active":"2024-08-30T23:44:45.807152+0000","last_peered":"2024-08-30T23:44:45.807152+0000","last_clean":"2024-08-30T23:44:45.807152+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:44:45.807152+0000","last_undegraded":"2024-08-30T23:44:45.807152+0000","last_fullsized":"2024-08-30T23:44:45.807152+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:31.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:31.878785+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:33.191051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":1295,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.304131+0000","last_change":"2024-08-30T23:44:29.953063+0000","last_active":"2024-08-30T23:44:42.304131+0000","last_peered":"2024-08-30T23:44:42.304131+0000","last_clean":"2024-08-30T23:44:42.304131+0000","last_became_active":"2024-08-30T23:42:40.017555+0000","last_became_peered":"2024-08-30T23:42:40.017555+0000","last_unstale":"2024-08-30T23:44:42.304131+0000","last_undegraded":"2024-08-30T23:44:42.304131+0000","last_fullsized":"2024-08-30T23:44:42.304131+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.953019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.953019+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.856722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":8,"reported_epoch":620,"state":"active+undersized","last_fresh":"2024-08-30T23:44:42.310924+0000","last_change":"2024-08-30T23:44:42.310484+0000","last_active":"2024-08-30T23:44:42.310924+0000","last_peered":"2024-08-30T23:44:42.310924+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"2024-08-30T23:44:42.310484+0000","last_became_peered":"2024-08-30T23:44:42.310484+0000","last_unstale":"2024-08-30T23:44:42.310924+0000","last_undegraded":"2024-08-30T23:44:42.310924+0000","last_fullsized":"2024-08-30T23:44:42.303428+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.191536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":320,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-08-30T23:44:41.296333+0000","last_change":"2024-08-30T23:44:33.929023+0000","last_active":"2024-08-30T23:44:41.296333+0000","last_peered":"2024-08-30T23:44:41.296333+0000","last_clean":"2024-08-30T23:44:41.296333+0000","last_became_active":"2024-08-30T23:42:38.914560+0000","last_became_peered":"2024-08-30T23:42:38.914560+0000","last_unstale":"2024-08-30T23:44:41.296333+0000","last_undegraded":"2024-08-30T23:44:41.296333+0000","last_fullsized":"2024-08-30T23:44:41.296333+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:33.928947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:33.928947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.006533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":4,"reported_epoch":620,"state":"creating+activating","last_fresh":"2024-08-30T23:44:42.313886+0000","last_change":"2024-08-30T23:44:42.303935+0000","last_active":"2024-08-30T23:44:41.288579+0000","last_peered":"2024-08-30T23:44:41.288579+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:44:42.313886+0000","last_undegraded":"2024-08-30T23:44:42.313886+0000","last_fullsized":"2024-08-30T23:44:42.313886+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":["7","6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":555,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-08-30T23:44:41.296313+0000","last_change":"2024-08-30T23:44:19.130985+0000","last_active":"2024-08-30T23:44:41.296313+0000","last_peered":"2024-08-30T23:44:41.296313+0000","last_clean":"2024-08-30T23:44:41.296313+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:44:41.296313+0000","last_undegraded":"2024-08-30T23:44:41.296313+0000","last_fullsized":"2024-08-30T23:44:41.296313+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:19.130919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:19.130919+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:26.399208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"621'18","reported_seq":31,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.442948+0000","last_change":"2024-08-30T23:44:42.456106+0000","last_active":"2024-08-30T23:44:45.442948+0000","last_peered":"2024-08-30T23:44:45.442948+0000","last_clean":"2024-08-30T23:44:45.442948+0000","last_became_active":"2024-08-30T23:44:42.455726+0000","last_became_peered":"2024-08-30T23:44:42.455726+0000","last_unstale":"2024-08-30T23:44:45.442948+0000","last_undegraded":"2024-08-30T23:44:45.442948+0000","last_fullsized":"2024-08-30T23:44:45.442948+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:08.436990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":439,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.812685+0000","last_change":"2024-08-30T23:44:38.508452+0000","last_active":"2024-08-30T23:44:45.812685+0000","last_peered":"2024-08-30T23:44:45.812685+0000","last_clean":"2024-08-30T23:44:45.812685+0000","last_became_active":"2024-08-30T23:42:34.874330+0000","last_became_peered":"2024-08-30T23:42:34.874330+0000","last_unstale":"2024-08-30T23:44:45.812685+0000","last_undegraded":"2024-08-30T23:44:45.812685+0000","last_fullsized":"2024-08-30T23:44:45.812685+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.508393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.508393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:50.318741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"622'98","reported_seq":114,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.907172+0000","last_change":"2024-08-30T23:44:43.028992+0000","last_active":"2024-08-30T23:44:45.907172+0000","last_peered":"2024-08-30T23:44:45.907172+0000","last_clean":"2024-08-30T23:44:45.907172+0000","last_became_active":"2024-08-30T23:44:43.028669+0000","last_became_peered":"2024-08-30T23:44:43.028669+0000","last_unstale":"2024-08-30T23:44:45.907172+0000","last_undegraded":"2024-08-30T23:44:45.907172+0000","last_fullsized":"2024-08-30T23:44:45.907172+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:01.890978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":6272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":419,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.812516+0000","last_change":"2024-08-30T23:44:19.298260+0000","last_active":"2024-08-30T23:44:45.812516+0000","last_peered":"2024-08-30T23:44:45.812516+0000","last_clean":"2024-08-30T23:44:45.812516+0000","last_became_active":"2024-08-30T23:41:38.809561+0000","last_became_peered":"2024-08-30T23:41:38.809561+0000","last_unstale":"2024-08-30T23:44:45.812516+0000","last_undegraded":"2024-08-30T23:44:45.812516+0000","last_fullsized":"2024-08-30T23:44:45.812516+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:19.298203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:19.298203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:46.925422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.457242+0000","last_change":"2024-08-30T23:44:42.456851+0000","last_active":"2024-08-30T23:44:42.457242+0000","last_peered":"2024-08-30T23:44:42.457242+0000","last_clean":"2024-08-30T23:44:42.457242+0000","last_became_active":"2024-08-30T23:44:42.455669+0000","last_became_peered":"2024-08-30T23:44:42.455669+0000","last_unstale":"2024-08-30T23:44:42.457242+0000","last_undegraded":"2024-08-30T23:44:42.457242+0000","last_fullsized":"2024-08-30T23:44:42.457242+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:57.856562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":435,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-08-30T23:44:41.296782+0000","last_change":"2024-08-30T23:44:34.916024+0000","last_active":"2024-08-30T23:44:41.296782+0000","last_peered":"2024-08-30T23:44:41.296782+0000","last_clean":"2024-08-30T23:44:41.296782+0000","last_became_active":"2024-08-30T23:42:20.761486+0000","last_became_peered":"2024-08-30T23:42:20.761486+0000","last_unstale":"2024-08-30T23:44:41.296782+0000","last_undegraded":"2024-08-30T23:44:41.296782+0000","last_fullsized":"2024-08-30T23:44:41.296782+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:34.915971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:34.915971+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:40.704451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.311137+0000","last_change":"2024-08-30T23:44:42.310819+0000","last_active":"2024-08-30T23:44:42.311137+0000","last_peered":"2024-08-30T23:44:42.311137+0000","last_clean":"2024-08-30T23:44:42.311137+0000","last_became_active":"2024-08-30T23:44:42.310475+0000","last_became_peered":"2024-08-30T23:44:42.310475+0000","last_unstale":"2024-08-30T23:44:42.311137+0000","last_undegraded":"2024-08-30T23:44:42.311137+0000","last_fullsized":"2024-08-30T23:44:42.311137+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:01.737704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":424,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.812021+0000","last_change":"2024-08-30T23:44:32.434271+0000","last_active":"2024-08-30T23:44:45.812021+0000","last_peered":"2024-08-30T23:44:45.812021+0000","last_clean":"2024-08-30T23:44:45.812021+0000","last_became_active":"2024-08-30T23:42:38.913092+0000","last_became_peered":"2024-08-30T23:42:38.913092+0000","last_unstale":"2024-08-30T23:44:45.812021+0000","last_undegraded":"2024-08-30T23:44:45.812021+0000","last_fullsized":"2024-08-30T23:44:45.812021+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:32.434222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:32.434222+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:37.108657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,2],"acting":[10,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":4,"reported_epoch":620,"state":"creating+activating","last_fresh":"2024-08-30T23:44:42.455829+0000","last_change":"2024-08-30T23:44:42.303951+0000","last_active":"2024-08-30T23:44:41.288579+0000","last_peered":"2024-08-30T23:44:41.288579+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:44:42.455829+0000","last_undegraded":"2024-08-30T23:44:42.455829+0000","last_fullsized":"2024-08-30T23:44:42.455829+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":["7","2","9"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1288,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:43.308260+0000","last_change":"2024-08-30T23:42:38.912360+0000","last_active":"2024-08-30T23:44:43.308260+0000","last_peered":"2024-08-30T23:44:43.308260+0000","last_clean":"2024-08-30T23:44:43.308260+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:44:43.308260+0000","last_undegraded":"2024-08-30T23:44:43.308260+0000","last_fullsized":"2024-08-30T23:44:43.308260+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:55.978214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:55.978214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"622'107","reported_seq":120,"reported_epoch":622,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:44:45.924171+0000","last_change":"2024-08-30T23:44:45.386164+0000","last_active":"2024-08-30T23:44:45.924171+0000","last_peered":"2024-08-30T23:44:45.924171+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"2024-08-30T23:44:42.306853+0000","last_became_peered":"2024-08-30T23:44:42.306853+0000","last_unstale":"2024-08-30T23:44:45.924171+0000","last_undegraded":"2024-08-30T23:44:45.385784+0000","last_fullsized":"2024-08-30T23:44:42.304842+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.220393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":6848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":107}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":183,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:43.308186+0000","last_change":"2024-08-30T23:42:34.865291+0000","last_active":"2024-08-30T23:44:43.308186+0000","last_peered":"2024-08-30T23:44:43.308186+0000","last_clean":"2024-08-30T23:44:43.308186+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:44:43.308186+0000","last_undegraded":"2024-08-30T23:44:43.308186+0000","last_fullsized":"2024-08-30T23:44:43.308186+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.830758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.830758+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:44:45.531639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":11,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.314670+0000","last_change":"2024-08-30T23:44:42.314402+0000","last_active":"2024-08-30T23:44:42.314670+0000","last_peered":"2024-08-30T23:44:42.314670+0000","last_clean":"2024-08-30T23:44:42.314670+0000","last_became_active":"2024-08-30T23:44:42.314051+0000","last_became_peered":"2024-08-30T23:44:42.314051+0000","last_unstale":"2024-08-30T23:44:42.314670+0000","last_undegraded":"2024-08-30T23:44:42.314670+0000","last_fullsized":"2024-08-30T23:44:42.314670+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.885555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":273,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.303911+0000","last_change":"2024-08-30T23:44:30.127625+0000","last_active":"2024-08-30T23:44:42.303911+0000","last_peered":"2024-08-30T23:44:42.303911+0000","last_clean":"2024-08-30T23:44:42.303911+0000","last_became_active":"2024-08-30T23:43:56.833115+0000","last_became_peered":"2024-08-30T23:43:56.833115+0000","last_unstale":"2024-08-30T23:44:42.303911+0000","last_undegraded":"2024-08-30T23:44:42.303911+0000","last_fullsized":"2024-08-30T23:44:42.303911+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:30.127565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:30.127565+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.812310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":619,"state":"creating+peering","last_fresh":"2024-08-30T23:44:41.296436+0000","last_change":"2024-08-30T23:44:41.296436+0000","last_active":"2024-08-30T23:44:41.288579+0000","last_peered":"2024-08-30T23:44:41.288579+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:44:41.296436+0000","last_undegraded":"2024-08-30T23:44:41.296436+0000","last_fullsized":"2024-08-30T23:44:41.296436+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":260,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:43.308183+0000","last_change":"2024-08-30T23:44:38.472833+0000","last_active":"2024-08-30T23:44:43.308183+0000","last_peered":"2024-08-30T23:44:43.308183+0000","last_clean":"2024-08-30T23:44:43.308183+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:44:43.308183+0000","last_undegraded":"2024-08-30T23:44:43.308183+0000","last_fullsized":"2024-08-30T23:44:43.308183+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.472767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.472767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.283833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"622'102","reported_seq":116,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.939093+0000","last_change":"2024-08-30T23:44:43.029448+0000","last_active":"2024-08-30T23:44:45.939093+0000","last_peered":"2024-08-30T23:44:45.939093+0000","last_clean":"2024-08-30T23:44:45.939093+0000","last_became_active":"2024-08-30T23:44:43.029122+0000","last_became_peered":"2024-08-30T23:44:43.029122+0000","last_unstale":"2024-08-30T23:44:45.939093+0000","last_undegraded":"2024-08-30T23:44:45.939093+0000","last_fullsized":"2024-08-30T23:44:45.939093+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.949159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":190,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:43.308360+0000","last_change":"2024-08-30T23:42:34.888040+0000","last_active":"2024-08-30T23:44:43.308360+0000","last_peered":"2024-08-30T23:44:43.308360+0000","last_clean":"2024-08-30T23:44:43.308360+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:44:43.308360+0000","last_undegraded":"2024-08-30T23:44:43.308360+0000","last_fullsized":"2024-08-30T23:44:43.308360+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:34.887995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:34.887995+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":4,"reported_epoch":620,"state":"creating+activating","last_fresh":"2024-08-30T23:44:42.306239+0000","last_change":"2024-08-30T23:44:42.303898+0000","last_active":"2024-08-30T23:44:41.288579+0000","last_peered":"2024-08-30T23:44:41.288579+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:44:42.306239+0000","last_undegraded":"2024-08-30T23:44:42.306239+0000","last_fullsized":"2024-08-30T23:44:42.306239+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":["7","3","11"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":444,"reported_epoch":619,"state":"active+undersized","last_fresh":"2024-08-30T23:44:41.296776+0000","last_change":"2024-08-30T23:44:03.894450+0000","last_active":"2024-08-30T23:44:41.296776+0000","last_peered":"2024-08-30T23:44:41.296776+0000","last_clean":"2024-08-30T23:44:01.872560+0000","last_became_active":"2024-08-30T23:44:03.894450+0000","last_became_peered":"2024-08-30T23:44:03.894450+0000","last_unstale":"2024-08-30T23:44:41.296776+0000","last_undegraded":"2024-08-30T23:44:41.296776+0000","last_fullsized":"2024-08-30T23:44:03.891268+0000","mapping_epoch":594,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.288446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.288446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"622'146","reported_seq":161,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:46.479319+0000","last_change":"2024-08-30T23:44:43.028106+0000","last_active":"2024-08-30T23:44:46.479319+0000","last_peered":"2024-08-30T23:44:46.479319+0000","last_clean":"2024-08-30T23:44:46.479319+0000","last_became_active":"2024-08-30T23:44:43.027789+0000","last_became_peered":"2024-08-30T23:44:43.027789+0000","last_unstale":"2024-08-30T23:44:46.479319+0000","last_undegraded":"2024-08-30T23:44:46.479319+0000","last_fullsized":"2024-08-30T23:44:46.479319+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.782764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":146,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":386,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.304060+0000","last_change":"2024-08-30T23:44:37.215950+0000","last_active":"2024-08-30T23:44:42.304060+0000","last_peered":"2024-08-30T23:44:42.304060+0000","last_clean":"2024-08-30T23:44:42.304060+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:44:42.304060+0000","last_undegraded":"2024-08-30T23:44:42.304060+0000","last_fullsized":"2024-08-30T23:44:42.304060+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:37.215896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:37.215896+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.523982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"621'12","reported_seq":26,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:45.453237+0000","last_change":"2024-08-30T23:44:42.456265+0000","last_active":"2024-08-30T23:44:45.453237+0000","last_peered":"2024-08-30T23:44:45.453237+0000","last_clean":"2024-08-30T23:44:45.453237+0000","last_became_active":"2024-08-30T23:44:42.455981+0000","last_became_peered":"2024-08-30T23:44:42.455981+0000","last_unstale":"2024-08-30T23:44:45.453237+0000","last_undegraded":"2024-08-30T23:44:45.453237+0000","last_fullsized":"2024-08-30T23:44:45.453237+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.062797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1272,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.303599+0000","last_change":"2024-08-30T23:44:13.869717+0000","last_active":"2024-08-30T23:44:42.303599+0000","last_peered":"2024-08-30T23:44:42.303599+0000","last_clean":"2024-08-30T23:44:42.303599+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:44:42.303599+0000","last_undegraded":"2024-08-30T23:44:42.303599+0000","last_fullsized":"2024-08-30T23:44:42.303599+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:13.869671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:13.869671+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.045381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"622'136","reported_seq":152,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-08-30T23:44:46.509171+0000","last_change":"2024-08-30T23:44:42.930674+0000","last_active":"2024-08-30T23:44:46.509171+0000","last_peered":"2024-08-30T23:44:46.509171+0000","last_clean":"2024-08-30T23:44:46.509171+0000","last_became_active":"2024-08-30T23:44:42.930380+0000","last_became_peered":"2024-08-30T23:44:42.930380+0000","last_unstale":"2024-08-30T23:44:46.509171+0000","last_undegraded":"2024-08-30T23:44:46.509171+0000","last_fullsized":"2024-08-30T23:44:46.509171+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:09.285121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8912896,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":8704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,11],"acting":[10,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":396,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-08-30T23:44:44.070303+0000","last_change":"2024-08-30T23:44:12.112535+0000","last_active":"2024-08-30T23:44:44.070303+0000","last_peered":"2024-08-30T23:44:44.070303+0000","last_clean":"2024-08-30T23:44:44.070303+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:44:44.070303+0000","last_undegraded":"2024-08-30T23:44:44.070303+0000","last_fullsized":"2024-08-30T23:44:44.070303+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:12.112429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:12.112429+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:38.348925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.306432+0000","last_change":"2024-08-30T23:44:42.305791+0000","last_active":"2024-08-30T23:44:42.306432+0000","last_peered":"2024-08-30T23:44:42.306432+0000","last_clean":"2024-08-30T23:44:42.306432+0000","last_became_active":"2024-08-30T23:44:42.305499+0000","last_became_peered":"2024-08-30T23:44:42.305499+0000","last_unstale":"2024-08-30T23:44:42.306432+0000","last_undegraded":"2024-08-30T23:44:42.306432+0000","last_fullsized":"2024-08-30T23:44:42.306432+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:56.085891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,7],"acting":[3,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":172,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-08-30T23:44:41.296321+0000","last_change":"2024-08-30T23:44:32.963615+0000","last_active":"2024-08-30T23:44:41.296321+0000","last_peered":"2024-08-30T23:44:41.296321+0000","last_clean":"2024-08-30T23:44:41.296321+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:44:41.296321+0000","last_undegraded":"2024-08-30T23:44:41.296321+0000","last_fullsized":"2024-08-30T23:44:41.296321+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:32.963552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:32.963552+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.292206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1129,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-08-30T23:44:42.303642+0000","last_change":"2024-08-30T23:44:33.986377+0000","last_active":"2024-08-30T23:44:42.303642+0000","last_peered":"2024-08-30T23:44:42.303642+0000","last_clean":"2024-08-30T23:44:42.303642+0000","last_became_active":"2024-08-30T23:41:40.752006+0000","last_became_peered":"2024-08-30T23:41:40.752006+0000","last_unstale":"2024-08-30T23:44:42.303642+0000","last_undegraded":"2024-08-30T23:44:42.303642+0000","last_fullsized":"2024-08-30T23:44:42.303642+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:33.986333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:33.986333+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.206532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":619,"num_write_kb":39616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":74842112,"data_stored":74842112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":619,"ondisk_log_size":619,"up":46,"acting":46,"num_store_stats":10},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187698,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435852,"kb_used_data":3264,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93309300,"statfs":{"total":95995035648,"available":95548723200,"internally_reserved":0,"allocated":3342336,"data_stored":2168611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":442759620},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449339,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":452624,"kb_used_data":106180,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93292528,"statfs":{"total":95995035648,"available":95531548672,"internally_reserved":0,"allocated":108728320,"data_stored":107722738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440176,"kb_used_data":3248,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93304976,"statfs":{"total":95995035648,"available":95544295424,"internally_reserved":0,"allocated":3325952,"data_stored":2278972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972611,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361224,"kb_used_data":19064,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93383928,"statfs":{"total":95995035648,"available":95625142272,"internally_reserved":0,"allocated":19521536,"data_stored":18471004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234245,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590476,"kb_used_data":107904,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93154676,"statfs":{"total":95995035648,"available":95390388224,"internally_reserved":0,"allocated":110493696,"data_stored":109319979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528588,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359820,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93385332,"statfs":{"total":95995035648,"available":95626579968,"internally_reserved":0,"allocated":3670016,"data_stored":2496299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":566,"seq":2430951489545,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552508,"kb_used_data":479660,"kb_used_omap":4,"kb_used_meta":72827,"kb_avail":93192644,"statfs":{"total":95995035648,"available":95429267456,"internally_reserved":0,"allocated":491171840,"data_stored":490053979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4411,"internal_metadata":74575557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953750,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421624,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93323528,"statfs":{"total":95995035648,"available":95563292672,"internally_reserved":0,"allocated":2998272,"data_stored":1951292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149978,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":450088,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93295064,"statfs":{"total":95995035648,"available":95534145536,"internally_reserved":0,"allocated":3702784,"data_stored":2588780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":456980935},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411616,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309716,"kb_used_data":3464,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93435436,"statfs":{"total":95995035648,"available":95677886464,"internally_reserved":0,"allocated":3547136,"data_stored":2369867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830432,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":197248,"kb_used_data":19376,"kb_used_omap":87,"kb_used_meta":177768,"kb_avail":93547904,"statfs":{"total":95995035648,"available":95793053696,"internally_reserved":0,"allocated":19841024,"data_stored":18794044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":182034591},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508060,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93237092,"statfs":{"total":95995035648,"available":95474782208,"internally_reserved":0,"allocated":3170304,"data_stored":2059852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24051712,"data_stored":24051712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4456448,"data_stored":4456448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":19070976,"data_stored":19070976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":27262976,"data_stored":27262976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:44:47.711 INFO:tasks.thrashosds.thrasher:Setting 1.1 to [2, 0, 10] 2024-08-30T23:44:47.711 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1', '2', '0', '10'] 2024-08-30T23:44:47.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 1.1 2 0 10 2024-08-30T23:44:47.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:47.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:47.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:47.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:47.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:48.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:47 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2162412897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:48.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:47 smithi173 ceph-mon[20554]: osdmap e623: 12 total, 11 up, 12 in 2024-08-30T23:44:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:47 smithi173 ceph-mon[20554]: from='client.17610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:44:48.091 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:47 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2162412897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:47 smithi106 ceph-mon[16603]: osdmap e623: 12 total, 11 up, 12 in 2024-08-30T23:44:48.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:47 smithi106 ceph-mon[16603]: from='client.17610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:44:48.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:48.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:44:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:48.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:48.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:47 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2162412897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:44:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:47 smithi136 ceph-mon[20207]: osdmap e623: 12 total, 11 up, 12 in 2024-08-30T23:44:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:47 smithi136 ceph-mon[20207]: from='client.17610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:44:48.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:48.354 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:48.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5110 5094 106.118 91.8125 0.0149987 0.00931241 2024-08-30T23:44:48.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:48.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:48.494 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:48.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:48.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:48.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:48.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:48.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:48.842 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:48.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:48.880 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:48.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:48 smithi173 ceph-mon[20554]: pgmap v1391: 37 pgs: 2 active+undersized, 1 active+undersized+degraded, 34 active+clean; 101 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 302 op/s; 107/4821 objects degraded (2.219%) 2024-08-30T23:44:48.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:44:48.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:44:48.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:48 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 107/4821 objects degraded (2.219%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:44:48.909 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:48 smithi173 ceph-mon[20554]: osdmap e624: 12 total, 11 up, 12 in 2024-08-30T23:44:48.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:48.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:48.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:49.016 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:48 smithi106 ceph-mon[16603]: pgmap v1391: 37 pgs: 2 active+undersized, 1 active+undersized+degraded, 34 active+clean; 101 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 302 op/s; 107/4821 objects degraded (2.219%) 2024-08-30T23:44:49.016 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:44:49.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:44:49.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:48 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 107/4821 objects degraded (2.219%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:44:49.017 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:48 smithi106 ceph-mon[16603]: osdmap e624: 12 total, 11 up, 12 in 2024-08-30T23:44:49.018 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:44:49.018 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:44:49.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:44:49.018 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:44:49.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.018 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:143663 4.6 4:61ba4f89:::benchmark_data_smithi173_74914_object45421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:36:18.874694+0000", 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 510.14276439700001, 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55230828300000001, 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 143663 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874694+0000", 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874694+0000", 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874696+0000", 2024-08-30T23:44:49.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.294e-06 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874701+0000", 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3789999999999997e-06 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874714+0000", 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3351e-05 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874722+0000", 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8380000000000002e-06 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874772+0000", 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9506999999999999e-05 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874852+0000", 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9901000000000006e-05 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.874918+0000", 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5838000000000002e-05 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.879162+0000", 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042439879999999998 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:18.879171+0000", 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3000000000000007e-06 2024-08-30T23:44:49.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:19.426920+0000", 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54774905600000001 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:19.426953+0000", 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2602999999999999e-05 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:36:19.427003+0000", 2024-08-30T23:44:49.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0228000000000001e-05 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86189 5.17 5.7a10c297 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.042890836, 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83557530000000002, 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.025 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86189 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974569+0000", 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8599999999999998e-07 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974571+0000", 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6599999999999999e-06 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974579+0000", 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4259999999999997e-06 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974584+0000", 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6740000000000002e-06 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974844+0000", 2024-08-30T23:44:49.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025967600000000003 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974852+0000", 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4200000000000001e-06 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810133+0000", 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83528164100000002 2024-08-30T23:44:49.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810143+0000", 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0117e-05 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86191 5.17 5.b68bebd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.04286514099999, 2024-08-30T23:44:49.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83558337400000005, 2024-08-30T23:44:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86191 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4799999999999999e-07 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974596+0000", 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9589999999999998e-06 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974602+0000", 2024-08-30T23:44:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:44:49.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974607+0000", 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.977e-06 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974875+0000", 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026775900000000001 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974880+0000", 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2220000000000001e-06 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810168+0000", 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83528822599999997 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810177+0000", 2024-08-30T23:44:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1549999999999996e-06 2024-08-30T23:44:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86212 5.17 5.7a86b997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.04284618099999, 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83559074600000005, 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.038 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86212 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0499999999999999e-07 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974615+0000", 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6929999999999999e-06 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974620+0000", 2024-08-30T23:44:49.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6359999999999999e-06 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974625+0000", 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4349999999999999e-06 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974930+0000", 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030525099999999998 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974938+0000", 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3980000000000002e-06 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810196+0000", 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83525841000000001 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810203+0000", 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.042 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86322 5.17 5.11e87097 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.04273583200001, 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83550676000000001, 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.043 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86322 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974724+0000", 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974728+0000", 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1509999999999997e-06 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974739+0000", 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0575e-05 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974746+0000", 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8929999999999999e-06 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974986+0000", 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000240607 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974992+0000", 2024-08-30T23:44:49.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3430000000000003e-06 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810222+0000", 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83523082599999998 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810230+0000", 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3459999999999996e-06 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86384 5.17 5.e2d2ce77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.04263104099999, 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83542769299999997, 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86384 2024-08-30T23:44:49.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1300000000000001e-07 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974845+0000", 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6341000000000002e-05 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974850+0000", 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6359999999999999e-06 2024-08-30T23:44:49.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974855+0000", 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7530000000000001e-06 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975038+0000", 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018276300000000001 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975044+0000", 2024-08-30T23:44:49.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9830000000000001e-06 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810248+0000", 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83520457400000003 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810256+0000", 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2300000000000002e-06 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86434 5.17 5.dc1639f7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.04258806600001, 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.835407809, 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.053 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86434 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6899999999999998e-07 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974874+0000", 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3309999999999998e-06 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974879+0000", 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.699e-06 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974883+0000", 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2600000000000001e-06 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975066+0000", 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018369799999999999 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975071+0000", 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8930000000000004e-06 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810272+0000", 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83520088199999998 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810279+0000", 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5769999999999999e-06 2024-08-30T23:44:49.057 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86189 5.17 5.7a10c297 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.036810613, 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82969184399999996, 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.058 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86189 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980649+0000", 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2300000000000002e-07 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980651+0000", 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980658+0000", 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7800000000000003e-06 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980662+0000", 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6399999999999996e-06 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980723+0000", 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0745999999999997e-05 2024-08-30T23:44:49.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980732+0000", 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9379999999999998e-06 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810304+0000", 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82957220200000004 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810340+0000", 2024-08-30T23:44:49.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5905000000000003e-05 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86191 5.17 5.b68bebd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03677446899999, 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968160300000005, 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.063 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86191 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:44:49.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980685+0000", 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7899999999999999e-07 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980687+0000", 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1270000000000001e-06 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980694+0000", 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.703e-06 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980699+0000", 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4229999999999996e-06 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980764+0000", 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5363000000000004e-05 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980771+0000", 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1990000000000003e-06 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810359+0000", 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82958857900000005 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810366+0000", 2024-08-30T23:44:49.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8299999999999998e-06 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86212 5.17 5.7a86b997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03674656199999, 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967699800000005, 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.068 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86212 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:44:49.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980713+0000", 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5699999999999998e-07 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980715+0000", 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1629999999999999e-06 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980720+0000", 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4920000000000001e-06 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980725+0000", 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.493e-06 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980812+0000", 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7479999999999996e-05 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980819+0000", 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810384+0000", 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956448100000002 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810389+0000", 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5269999999999998e-06 2024-08-30T23:44:49.072 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86322 5.17 5.11e87097 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03666910800001, 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82972026099999996, 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.073 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86322 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2599999999999998e-07 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980792+0000", 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.894e-06 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980798+0000", 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980802+0000", 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3100000000000002e-06 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980870+0000", 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7617e-05 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980877+0000", 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4309999999999997e-06 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810498+0000", 2024-08-30T23:44:49.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82962015499999997 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810510+0000", 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2413e-05 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86384 5.17 5.e2d2ce77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03659671700001, 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967802800000001, 2024-08-30T23:44:49.078 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86384 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:44:49.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3000000000000002e-07 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980872+0000", 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2709999999999998e-06 2024-08-30T23:44:49.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980877+0000", 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6999999999999996e-06 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980883+0000", 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2270000000000001e-06 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980958+0000", 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4972e-05 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980972+0000", 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4389999999999999e-05 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810534+0000", 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956191800000001 2024-08-30T23:44:49.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810540+0000", 2024-08-30T23:44:49.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199999999999997e-06 2024-08-30T23:44:49.084 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.084 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86434 5.17 5.dc1639f7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.036568854, 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967796100000002, 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.085 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86434 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2000000000000001e-07 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980896+0000", 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980902+0000", 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980908+0000", 2024-08-30T23:44:49.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2260000000000002e-06 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980995+0000", 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6890000000000003e-05 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.604e-06 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810562+0000", 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956157900000005 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810568+0000", 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7749999999999998e-06 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86440 5.17 5.705d7d57 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:44:49.090 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03654308500001, 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968054700000005, 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86440 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5700000000000002e-07 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980918+0000", 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9819999999999998e-06 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980924+0000", 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7259999999999999e-06 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980929+0000", 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1749999999999996e-06 2024-08-30T23:44:49.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981037+0000", 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107353 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981044+0000", 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.272e-06 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810590+0000", 2024-08-30T23:44:49.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82954588500000004 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810596+0000", 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5969999999999997e-06 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86445 5.17 5.715d0bd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03652395099999, 2024-08-30T23:44:49.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968721000000001, 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86445 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8700000000000002e-07 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980937+0000", 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980942+0000", 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4739999999999996e-06 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980947+0000", 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8590000000000004e-06 2024-08-30T23:44:49.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981087+0000", 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139479 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981094+0000", 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.827e-06 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810616+0000", 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82952199699999996 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810622+0000", 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.494e-06 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86454 5.17 5.47a3f917 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 221.03645770099999, 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82964718199999998, 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86454 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.101 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981002+0000", 2024-08-30T23:44:49.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3499999999999996e-07 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981004+0000", 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9489999999999999e-06 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981010+0000", 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981014+0000", 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.617e-06 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981118+0000", 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103795 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981124+0000", 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1469999999999998e-06 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810642+0000", 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.829518333 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810648+0000", 2024-08-30T23:44:49.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9020000000000004e-06 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.17 505/491 [PushReplyOp(5:ed0feeac:::benchmark_data_smithi173_87639_object47011:head)])", 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 192.757293944, 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73458869599999999, 2024-08-30T23:44:49.106 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:44:49.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1500000000000005e-07 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260173+0000", 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7519999999999996e-06 2024-08-30T23:44:49.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993751+0000", 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73357760100000002 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993758+0000", 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8630000000000002e-06 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993850+0000", 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2544e-05 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.994754+0000", 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090342099999999995 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.112 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.17 505/491 [PushReplyOp(5:ef72d6c2:::benchmark_data_smithi173_87639_object47037:head)])", 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 192.74810571, 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73803978299999995, 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:44:49.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269354+0000", 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3099999999999998e-07 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269358+0000", 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169999999999996e-06 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.001419+0000", 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73206070400000001 2024-08-30T23:44:49.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.001426+0000", 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4939999999999997e-06 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.006768+0000", 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005341485 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.007393+0000", 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062535200000000003 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.118 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(5.b e587,530 5:d5260a3e:::benchmark_data_smithi173_87639_object2707:head,442'5272)", 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.930340934999997, 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71060140599999999, 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087119+0000", 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087123+0000", 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2479999999999998e-06 2024-08-30T23:44:49.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797124+0000", 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71000132900000001 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797133+0000", 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4440000000000008e-06 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797194+0000", 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0937999999999998e-05 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797719+0000", 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000525808 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:144501 5.1 e590/540 5:827f28b3:::benchmark_data_smithi173_87639_object64261:head v 590'7080, mlcod=589'7078)", 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:44:49.122 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 51.787767271, 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64106508200000001, 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:44:49.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5900000000000001e-07 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229695+0000", 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6009999999999999e-06 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:44:49.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868724+0000", 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.639028284 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868732+0000", 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3080000000000008e-06 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868794+0000", 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1963000000000003e-05 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868819+0000", 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5282999999999998e-05 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.870718+0000", 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001898582 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.870757+0000", 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8902000000000001e-05 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.126 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:44:49.127 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:44:49.127 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:44:49.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:49.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:49.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:49.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:48 smithi136 ceph-mon[20207]: pgmap v1391: 37 pgs: 2 active+undersized, 1 active+undersized+degraded, 34 active+clean; 101 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 302 op/s; 107/4821 objects degraded (2.219%) 2024-08-30T23:44:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:44:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:44:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:48 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 107/4821 objects degraded (2.219%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:44:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:48 smithi136 ceph-mon[20207]: osdmap e624: 12 total, 11 up, 12 in 2024-08-30T23:44:49.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6654 6638 103.711 96.5 0.00423074 0.00961778 2024-08-30T23:44:49.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:49.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:49.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:49.549 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:49.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:49.683 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:49.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:49.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:49.816 INFO:teuthology.orchestra.run.smithi106.stderr:set 1.1 pg_upmap mapping to [2,0,10] 2024-08-30T23:44:49.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:49.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:49.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:50.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:50.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2072110411' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]} : dispatch 2024-08-30T23:44:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:49 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]} : dispatch 2024-08-30T23:44:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2072110411' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]} : dispatch 2024-08-30T23:44:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:49 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]} : dispatch 2024-08-30T23:44:50.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:44:50.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:50.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:50.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:50.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:44:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2072110411' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]} : dispatch 2024-08-30T23:44:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:49 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]} : dispatch 2024-08-30T23:44:50.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:50.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 8422 8406 105.067 110.5 0.0379162 0.00949468 2024-08-30T23:44:50.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:50.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:50.460 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:50.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:50.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:50.605 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:50.664 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:50.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:50.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:50.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:44:50.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:50.893 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:50 smithi173 ceph-mon[20554]: pgmap v1393: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 201 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 533 op/s; 413/9603 objects degraded (4.301%) 2024-08-30T23:44:50.893 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:50 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]}]': finished 2024-08-30T23:44:50.893 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:50 smithi173 ceph-mon[20554]: osdmap e625: 12 total, 11 up, 12 in 2024-08-30T23:44:50.893 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:50 smithi173 ceph-mon[20554]: osdmap e626: 12 total, 11 up, 12 in 2024-08-30T23:44:50.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:50.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:50.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:50.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:51.022 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:44:51.023 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:44:51.023 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:44:51.023 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:44:51.032 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:50 smithi106 ceph-mon[16603]: pgmap v1393: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 201 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 533 op/s; 413/9603 objects degraded (4.301%) 2024-08-30T23:44:51.032 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:50 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]}]': finished 2024-08-30T23:44:51.032 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:50 smithi106 ceph-mon[16603]: osdmap e625: 12 total, 11 up, 12 in 2024-08-30T23:44:51.032 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:50 smithi106 ceph-mon[16603]: osdmap e626: 12 total, 11 up, 12 in 2024-08-30T23:44:51.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:51.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:51.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:51.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:50 smithi136 ceph-mon[20207]: pgmap v1393: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 201 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 MiB/s wr, 533 op/s; 413/9603 objects degraded (4.301%) 2024-08-30T23:44:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:50 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [2, 0, 10]}]': finished 2024-08-30T23:44:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:50 smithi136 ceph-mon[20207]: osdmap e625: 12 total, 11 up, 12 in 2024-08-30T23:44:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:50 smithi136 ceph-mon[20207]: osdmap e626: 12 total, 11 up, 12 in 2024-08-30T23:44:51.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:51.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:51.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 10047 10031 104.482 101.562 0.0120924 0.00955942 2024-08-30T23:44:51.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:51.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:51.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:51.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:51.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:51.703 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:51.764 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:51.766 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:51.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:51.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:51.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:51.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:51.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:51.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:44:51.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:51.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:51.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:51.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:51.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:52.038 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:52.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:52.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3346868939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:52.082 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3346868939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:52.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:52.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3346868939' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:52.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:52.334 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:52.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 11161 11145 99.5015 69.625 0.0159177 0.0100302 2024-08-30T23:44:52.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:52.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:52.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:52.669 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:52.669 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:52.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:52.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:52.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:52.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:52.939 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:52.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:53.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:53.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:53.104 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:52 smithi106 ceph-mon[16603]: pgmap v1396: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 421 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s; 903/20184 objects degraded (4.474%) 2024-08-30T23:44:53.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:53.180 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:52 smithi173 ceph-mon[20554]: pgmap v1396: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 421 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s; 903/20184 objects degraded (4.474%) 2024-08-30T23:44:53.190 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:52 smithi136 ceph-mon[20207]: pgmap v1396: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 421 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s; 903/20184 objects degraded (4.474%) 2024-08-30T23:44:53.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:53.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:53.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:53.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 12726 12710 99.2894 97.8125 0.00705144 0.0100617 2024-08-30T23:44:53.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:44:53.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:53.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.459 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:53.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:53.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:53.591 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:53.683 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:53.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:53.700 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:53.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.715 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:53.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:53.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:53.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:53.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:53.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:54.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:53 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1425/33042 objects degraded (4.313%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:54.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:54.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:53 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1425/33042 objects degraded (4.313%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:54.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:54.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:53 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1425/33042 objects degraded (4.313%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:44:54.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:54.348 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:54.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:54.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 14052 14036 97.4647 82.875 0.00158436 0.0100982 2024-08-30T23:44:54.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:54.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:54.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:54.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:54.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:54.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:54.829 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:54.830 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:54.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:54.890 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:44:54.893 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:54.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:54.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:44:55.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:44:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:54 smithi173 ceph-mon[20554]: pgmap v1397: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 689 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 1425/33042 objects degraded (4.313%) 2024-08-30T23:44:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4145053006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:54 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:55.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:55.143 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:55.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:55.268 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:54 smithi136 ceph-mon[20207]: pgmap v1397: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 689 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 1425/33042 objects degraded (4.313%) 2024-08-30T23:44:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4145053006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:54 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:54 smithi106 ceph-mon[16603]: pgmap v1397: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 689 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 1425/33042 objects degraded (4.313%) 2024-08-30T23:44:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4145053006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:54 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:44:55.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 15341 15325 95.7737 80.5625 0.00279684 0.0104249 2024-08-30T23:44:55.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:55.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:55.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:55.513 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:44:55.513 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:44:55.513 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T23:44:55.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:44:55.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:55.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:44:55.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:55.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:55.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:44:55.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:55.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:55.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:55.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:55.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:55.982 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:56.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:44:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:56.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:44:56.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:55 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:55 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:55 smithi173 ceph-mon[20554]: osdmap e627: 12 total, 11 up, 12 in 2024-08-30T23:44:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:55 smithi173 ceph-mon[20554]: osdmap e628: 12 total, 11 up, 12 in 2024-08-30T23:44:56.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:56.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:56.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:56.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:55 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:55 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:55 smithi136 ceph-mon[20207]: osdmap e627: 12 total, 11 up, 12 in 2024-08-30T23:44:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:55 smithi136 ceph-mon[20207]: osdmap e628: 12 total, 11 up, 12 in 2024-08-30T23:44:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:55 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:44:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:55 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:44:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:55 smithi106 ceph-mon[16603]: osdmap e627: 12 total, 11 up, 12 in 2024-08-30T23:44:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:55 smithi106 ceph-mon[16603]: osdmap e628: 12 total, 11 up, 12 in 2024-08-30T23:44:56.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 16618 16602 94.3214 79.8125 0.00992039 0.010569 2024-08-30T23:44:56.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:56.411 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:56.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:56.534 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:56.534 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:56.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:56.593 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:56.593 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:56.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:56.661 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:56.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:56.785 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:56.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:56.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:56.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:56.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:57.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:56 smithi106 ceph-mon[16603]: pgmap v1399: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 744 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 1425/35712 objects degraded (3.990%) 2024-08-30T23:44:57.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:56 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:44:57.007 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:56 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:44:57.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:57.048 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:57.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:57.171 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:56 smithi173 ceph-mon[20554]: pgmap v1399: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 744 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 1425/35712 objects degraded (3.990%) 2024-08-30T23:44:57.171 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:56 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:44:57.171 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:56 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:44:57.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:57.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:44:57.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:56 smithi136 ceph-mon[20207]: pgmap v1399: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 744 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s; 1425/35712 objects degraded (3.990%) 2024-08-30T23:44:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:56 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:44:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:56 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:44:57.296 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:57.369 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 18041 18025 93.872 88.9375 0.20919 0.0106116 2024-08-30T23:44:57.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:44:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:57.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 628, 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:44:55.803349+0000", 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:44:02.874029+0000", 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:38:32.101509+0000", 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5804032, 2024-08-30T23:44:57.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "nodeep-scrub", 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 28, 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 6, 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:44:57.467 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "luminous", 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:44:57.468 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 21, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 21, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 21, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 21, 2024-08-30T23:44:57.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 21, 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "509", 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "507", 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:44:57.470 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:44:57.471 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:44:57.472 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:44:57.473 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.9099999666213989, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.9099999666213989, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.0999999046325684, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.0999999046325684, 2024-08-30T23:44:57.474 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.71190768480300903, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.85000002384185791, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.70999997854232788 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 6, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_4", 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:44:40.370699+0000", 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:44:57.475 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:44:57.476 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:44:57.477 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:44:57.477 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:44:57.477 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:44:57.477 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:44:57.477 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:44:57.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "621", 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:44:57.478 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:44:57.479 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:44:57.480 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:44:57.481 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:44:57.482 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:44:57.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.482 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:44:57.482 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:44:57.482 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.130000114440918, 2024-08-30T23:44:57.482 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 3.130000114440918, 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.440000057220459, 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3.440000057220459, 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.71190768480300903, 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.85000002384185791, 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.70999997854232788 2024-08-30T23:44:57.483 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.484 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.484 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:57.484 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:44:57.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.484 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:44:57.484 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:44:57.485 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.485 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.485 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.2807464599609375, 2024-08-30T23:44:57.485 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:44:57.485 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.485 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.486 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:44:57.486 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 539, 2024-08-30T23:44:57.486 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.486 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.486 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.486 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.487 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.488 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:44:57.488 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.489 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:57.490 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.490 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.490 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.491 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.491 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.492 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.492 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:44:57.492 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.493 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.493 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:44:57.493 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.493 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.493 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.494 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.494 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.494 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.494 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:44:57.495 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.495 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.495 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:44:57.496 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.496 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.496 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.496 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.496 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.497 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.497 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:44:57.497 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.498 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.498 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:44:57.498 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:44:57.498 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.498 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.499 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:44:57.499 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:44:57.499 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:44:57.499 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:44:57.499 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.500 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.500 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.500 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.500 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:44:57.501 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:44:57.501 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.501 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.501 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.501 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:44:57.502 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-30T23:44:57.502 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-30T23:44:57.502 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-30T23:44:57.502 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:57.502 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:44:57.502 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.503 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.503 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.503 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.503 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:44:57.503 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.504 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.504 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:44:57.504 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.504 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.505 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.505 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.505 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.506 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.506 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:44:57.506 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.507 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.507 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:44:57.507 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.507 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.507 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.508 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.508 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.508 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.508 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:44:57.509 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.509 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.509 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:44:57.509 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.510 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.511 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 594, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.513 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.515 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.516 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:44:57.517 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.519 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:44:57.519 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:44:57.519 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.521 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.521 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:44:57.521 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.522 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.522 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:44:57.522 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.522 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.522 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.523 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.523 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.523 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.523 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:44:57.523 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.525 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.526 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.527 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.528 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.529 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:44:57.530 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.531 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.531 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.531 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.531 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.533 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:44:57.534 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.426727294921875, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 39, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 446, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 566, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 573, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:44:57.536 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.537 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/1428551692", 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/1428551692", 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/1428551692", 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/1428551692", 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.541 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7101593017578125, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.546 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.546 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:57.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.547 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.547 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:44:57.548 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:44:57.549 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7503204345703125, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.550 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.551 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.552 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.553 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:44:57.554 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:44:57.555 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:44:57.556 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.557 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.559 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:44:57.560 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:44:57.561 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.562 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.2454376220703125, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 595, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.564 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.564 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:44:57.564 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.565 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.565 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:44:57.565 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.565 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.566 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.567 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:44:57.568 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.569 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 625, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.571 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.572 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.573 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.575 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:44:57.576 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 587, 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.577 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.578 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.579 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.581 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:44:57.582 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:01.788525+0000", 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.583 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:31.939564+0000", 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:09.378080+0000", 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:52.250956+0000", 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.585 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:51.029648+0000", 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:22.725412+0000", 2024-08-30T23:44:57.586 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:49.490054+0000", 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:44:57.587 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:46.410922+0000", 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.588 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:28.335352+0000", 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:47.644167+0000", 2024-08-30T23:44:57.589 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:32.327364+0000", 2024-08-30T23:44:57.590 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:30.306781+0000", 2024-08-30T23:44:57.591 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [ 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1", 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-30T23:44:57.592 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:57.593 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:44:57.595 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:44:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:57.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:57.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:57.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:57.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:57.851 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:57.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:57.979 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:57.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:58.015 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4148119627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:44:58.015 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/812681382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:58.017 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:44:58.017 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:44:58.017 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:44:58.017 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:44:58.017 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:44:58.017 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:44:58.018 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:44:58.019 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:44:58.080 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:58.105 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:58.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:44:58.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:44:58.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4148119627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:44:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/812681382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4148119627' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:44:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/812681382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:44:58.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:58.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:58.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 19800 19784 95.1071 109.938 0.0114227 0.0105065 2024-08-30T23:44:58.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:58.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:44:58.491 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:58.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:44:58.620 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:44:58.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:44:58.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:58.748 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:58.849 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:58.873 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:58.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:44:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:44:58.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:44:58.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:58.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:44:59.057 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:44:59.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:59.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:59.184 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:58 smithi173 ceph-mon[20554]: pgmap v1401: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 985 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.69k op/s; 1861/47235 objects degraded (3.940%) 2024-08-30T23:44:59.184 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:58 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:44:59.184 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:58 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:44:59.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:44:59.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:44:59.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:58 smithi136 ceph-mon[20207]: pgmap v1401: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 985 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.69k op/s; 1861/47235 objects degraded (3.940%) 2024-08-30T23:44:59.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:58 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:44:59.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:58 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:44:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:58 smithi106 ceph-mon[16603]: pgmap v1401: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 985 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.69k op/s; 1861/47235 objects degraded (3.940%) 2024-08-30T23:44:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:58 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:44:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:58 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:44:59.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:59.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 21735 21719 96.9514 120.938 0.00575309 0.0103047 2024-08-30T23:44:59.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:59.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:44:59.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:44:59.567 INFO:tasks.thrashosds.thrasher:j is {'epoch': 628, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:44:55.803349+0000', 'last_up_change': '2024-08-30T23:44:02.874029+0000', 'last_in_change': '2024-08-30T23:38:32.101509+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '509', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '507', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.0999999046325684, 'raw_score_stable': 2.0999999046325684, 'primary_affinity_weighted': 0.711907684803009, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7099999785423279}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-08-30T23:44:40.370699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '621', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.130000114440918, 'score_stable': 3.130000114440918, 'optimal_score': 1.100000023841858, 'raw_score_acting': 3.440000057220459, 'raw_score_stable': 3.440000057220459, 'primary_affinity_weighted': 0.711907684803009, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7099999785423279}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 0.2807464599609375, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 619, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 594, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 0, 'in': 1, 'weight': 0.426727294921875, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 446, 'up_from': 566, 'up_thru': 573, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 1428551692}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 1428551692}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 1428551692}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 1428551692}]}, 'public_addr': '172.21.15.136:6809/1428551692', 'cluster_addr': '172.21.15.136:6811/1428551692', 'heartbeat_back_addr': '172.21.15.136:6815/1428551692', 'heartbeat_front_addr': '172.21.15.136:6813/1428551692', 'state': ['exists']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 0.7101593017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 0.7503204345703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2454376220703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 595, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 625, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 587, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:01.788525+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:31.939564+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:09.378080+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:52.250956+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:51.029648+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:22.725412+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:49.490054+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:46.410922+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:28.335352+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:47.644167+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:32.327364+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:30.306781+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [2, 0, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:44:59.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:44:59.605 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:44:59.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:44:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:44:59.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:44:59.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:44:59.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:44:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:44:59.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:44:59.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:44:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:44:59.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:44:59.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:00.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.017 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:00.117 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:00.118 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:00.140 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:00.237 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:59 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:00.237 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:59 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:00.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:44:59 smithi106 ceph-mon[16603]: osdmap e629: 12 total, 11 up, 12 in 2024-08-30T23:45:00.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:00.262 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:00.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:59 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:59 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:44:59 smithi136 ceph-mon[20207]: osdmap e629: 12 total, 11 up, 12 in 2024-08-30T23:45:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:59 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:59 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:44:59 smithi173 ceph-mon[20554]: osdmap e629: 12 total, 11 up, 12 in 2024-08-30T23:45:00.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:00.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 15 22569 22554 93.9669 52.1875 0.0078888 0.0106332 2024-08-30T23:45:00.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:00.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:00.527 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:00.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:00.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:00.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:00.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:00.806 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:45:00.833 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:45:00.834 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:45:00.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:00.906 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:00.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:00.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:45:01.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:01.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: pgmap v1403: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 2284/53949 objects degraded (4.234%) 2024-08-30T23:45:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2284/53949 objects degraded (4.234%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:45:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/958227101' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:01.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:45:01.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:45:01.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:45:01.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/958227101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:01.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:00 smithi106 ceph-mon[16603]: osdmap e630: 12 total, 11 up, 12 in 2024-08-30T23:45:01.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:01.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:01.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:01.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: pgmap v1403: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 2284/53949 objects degraded (4.234%) 2024-08-30T23:45:01.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2284/53949 objects degraded (4.234%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:45:01.263 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/958227101' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:01.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:45:01.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:45:01.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:45:01.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/958227101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:01.264 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:00 smithi173 ceph-mon[20554]: osdmap e630: 12 total, 11 up, 12 in 2024-08-30T23:45:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: pgmap v1403: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 2284/53949 objects degraded (4.234%) 2024-08-30T23:45:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2284/53949 objects degraded (4.234%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:45:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/958227101' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:45:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:45:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:45:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/958227101' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:00 smithi136 ceph-mon[20207]: osdmap e630: 12 total, 11 up, 12 in 2024-08-30T23:45:01.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:01.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 23838 23822 93.0467 79.25 0.00325177 0.0107401 2024-08-30T23:45:01.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:01.404 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:45:01.404 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:45:01.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:45:01.404 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:45:01.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.404 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.25654.0:196463 4.1 4:8f28ee96:::benchmark_data_smithi173_74914_object98221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.015837+0000", 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 463.38600712200002, 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55394642699999996, 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.25654", 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 196463 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015837+0000", 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015837+0000", 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015839+0000", 2024-08-30T23:45:01.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7290000000000001e-06 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015846+0000", 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7479999999999996e-06 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015863+0000", 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7431999999999998e-05 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.015872+0000", 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0659999999999993e-06 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016253+0000", 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000380987 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016363+0000", 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110047 2024-08-30T23:45:01.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,6", 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.016488+0000", 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124957 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.020434+0000", 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039450889999999997 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.022249+0000", 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018158779999999999 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569606+0000", 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54735697400000005 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569708+0000", 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010207100000000001 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569784+0000", 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5449000000000002e-05 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.25654.0:196463 4.1 e381/306 ondisk, result = 0)", 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 463.37986785200002, 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.547816529, 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.411 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021977+0000", 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7999999999999996e-07 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.021980+0000", 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.864e-06 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569398+0000", 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.54741768300000004 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569420+0000", 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2169999999999999e-05 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569578+0000", 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001585 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569596+0000", 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7725999999999999e-05 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:37:18.569793+0000", 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019710600000000001 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.415 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 502/491 [PushOp(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head, version: 408'2535, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head@408'2535, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 208.20265057200001, 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59550546699999996, 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199198+0000", 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5470000000000001e-06 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199226+0000", 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8228000000000001e-05 2024-08-30T23:45:01.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793558+0000", 2024-08-30T23:45:01.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59433249099999996 2024-08-30T23:45:01.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793576+0000", 2024-08-30T23:45:01.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8108999999999999e-05 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793661+0000", 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5122000000000003e-05 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793671+0000", 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.794699+0000", 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001028581 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 509/491 [PushOp(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head, version: 417'3530, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head@417'3530, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 201.06305729300001, 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42136218600000003, 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.423 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338788+0000", 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.33e-07 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338818+0000", 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9720999999999999e-05 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758562+0000", 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.419743904 2024-08-30T23:45:01.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758588+0000", 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5839000000000001e-05 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758731+0000", 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014296099999999999 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758750+0000", 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9196000000000001e-05 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.760149+0000", 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001399632 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 510/491 [PushOp(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head, version: 418'3647, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head@418'3647, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 200.273093671, 2024-08-30T23:45:01.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62881896699999995, 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128752+0000", 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2219999999999999e-06 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128784+0000", 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1445000000000001e-05 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756187+0000", 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.627403029 2024-08-30T23:45:01.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756208+0000", 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1382999999999998e-05 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756330+0000", 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121612 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756346+0000", 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5962000000000001e-05 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.757570+0000", 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001224314 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:ef154109:::benchmark_data_smithi173_87639_object59696:head, version: 421'3773, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ef154109:::benchmark_data_smithi173_87639_object59696:head@421'3773, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:45:01.431 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 199.313568256, 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60072882599999999, 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088277+0000", 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4600000000000003e-07 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088304+0000", 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6968999999999999e-05 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687871+0000", 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59956659400000001 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687891+0000", 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0307999999999999e-05 2024-08-30T23:45:01.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688018+0000", 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012653499999999999 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688034+0000", 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6728999999999999e-05 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.689005+0000", 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097084500000000002 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.436 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 512/491 [PushOp(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head, version: 430'4003, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head@430'4003, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 195.13801558200001, 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48614643400000002, 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263830+0000", 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8500000000000001e-07 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263854+0000", 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3934000000000002e-05 2024-08-30T23:45:01.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748876+0000", 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.485022907 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748889+0000", 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2785999999999999e-05 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748973+0000", 2024-08-30T23:45:01.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.331e-05 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748983+0000", 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.749975+0000", 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099270099999999991 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.d 542/530 [PushReplyOp(5:bce15829:::benchmark_data_smithi173_87639_object23219:head)])", 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 140.58503380299999, 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44671719199999999, 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816812+0000", 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1260000000000001e-06 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816821+0000", 2024-08-30T23:45:01.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5510000000000005e-06 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260989+0000", 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44416764800000003 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260997+0000", 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5750000000000003e-06 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.261062+0000", 2024-08-30T23:45:01.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5520999999999998e-05 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.263528+0000", 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024657709999999998 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.445 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99435 5.8 5:154d2bda:::benchmark_data_smithi173_87639_object19195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:45:01.445 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:45:01.445 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.043552697, 2024-08-30T23:45:01.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39149939, 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99435 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358293+0000", 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5540000000000001e-06 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358300+0000", 2024-08-30T23:45:01.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4250000000000003e-06 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358312+0000", 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2333e-05 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358321+0000", 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4160000000000004e-06 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.383891+0000", 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025570521999999998 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.387506+0000", 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036149160000000001 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.705150+0000", 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31764353000000001 2024-08-30T23:45:01.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.707627+0000", 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024771609999999999 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.712526+0000", 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048989070000000001 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740121+0000", 2024-08-30T23:45:01.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027595773000000001 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740604+0000", 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000482196 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.741372+0000", 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076844900000000004 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.743276+0000", 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019037590000000001 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.749791+0000", 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065154490000000004 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99448 5.8 5:1453319e:::benchmark_data_smithi173_87639_object19208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:45:01.452 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 139.02177867899999, 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.415457309, 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99448 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380068+0000", 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.637e-06 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380074+0000", 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3500000000000002e-06 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380092+0000", 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7768999999999999e-05 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380101+0000", 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.389216+0000", 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0091151319999999997 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.392658+0000", 2024-08-30T23:45:01.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034414720000000001 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.727246+0000", 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33458855300000001 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.729727+0000", 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024804380000000002 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.733843+0000", 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041159079999999997 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.750905+0000", 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017061951999999998 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.752106+0000", 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001200916 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.789800+0000", 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037693979000000002 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.790989+0000", 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001189563 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.795523+0000", 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045340220000000004 2024-08-30T23:45:01.462 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102045 5.8 5:13843233:::benchmark_data_smithi173_87639_object21805:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 134.190584223, 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41801196699999998, 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.463 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102045 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211262+0000", 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7630000000000001e-06 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211268+0000", 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3489999999999997e-06 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.465 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211283+0000", 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4834000000000001e-05 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211293+0000", 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0037999999999999e-05 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.235799+0000", 2024-08-30T23:45:01.466 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024505741000000001 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.239265+0000", 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034655129999999999 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.592550+0000", 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35328515199999999 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.594844+0000", 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002294164 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.599415+0000", 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045711010000000002 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.620491+0000", 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021075930999999999 2024-08-30T23:45:01.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.621208+0000", 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071667200000000001 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.622191+0000", 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098297100000000002 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.623817+0000", 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016259740000000001 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629272+0000", 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054557640000000001 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102049 5.8 5:1445c9f0:::benchmark_data_smithi173_87639_object21809:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:45:01.470 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 134.16756823599999, 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40420456799999999, 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102049 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234278+0000", 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.012e-06 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234286+0000", 2024-08-30T23:45:01.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2799999999999998e-06 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234301+0000", 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5590000000000002e-05 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234310+0000", 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.241629+0000", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073184249999999999 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.244833+0000", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032044450000000002 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.610115+0000", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.365282208 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.612362+0000", 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022467279999999999 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.616714+0000", 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043520410000000001 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629561+0000", 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012847305999999999 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.630448+0000", 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088642099999999998 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.631622+0000", 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011737430000000001 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.633508+0000", 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018869030000000001 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.638481+0000", 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049724469999999996 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102205 5.8 5:100735ae:::benchmark_data_smithi173_87639_object21965:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 133.72717412899999, 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41224830200000001, 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.477 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102205 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:45:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674673+0000", 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.063e-06 2024-08-30T23:45:01.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674679+0000", 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3380000000000001e-06 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674692+0000", 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3383e-05 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674702+0000", 2024-08-30T23:45:01.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4280000000000002e-06 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.691678+0000", 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016976577 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.694973+0000", 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032944049999999998 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.051542+0000", 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35656930599999997 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.054103+0000", 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025610979999999999 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.057950+0000", 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003847296 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.078442+0000", 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020491453 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.079182+0000", 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073979099999999995 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.080424+0000", 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00124194 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.081991+0000", 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015673 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.086919+0000", 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049279240000000002 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:127378 5.8 5:10a9409e:::benchmark_data_smithi173_87639_object47138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 89.112078947000001, 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41103042400000001, 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127378 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289767+0000", 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.575e-06 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289774+0000", 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8510000000000001e-06 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289788+0000", 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4056999999999999e-05 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289796+0000", 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5800000000000003e-06 2024-08-30T23:45:01.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290170+0000", 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000373829 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290389+0000", 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000219559 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290617+0000", 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022764500000000001 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.293291+0000", 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026742269999999999 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.293692+0000", 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040149700000000001 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700642+0000", 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40694962699999998 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700700+0000", 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7865000000000002e-05 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700796+0000", 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6112000000000004e-05 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:144420 5.d 5:b9edf439:::benchmark_data_smithi173_87639_object64180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 65.280412021999993, 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.397110827, 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144420 2024-08-30T23:45:01.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121434+0000", 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121440+0000", 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0909999999999998e-06 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121453+0000", 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2656e-05 2024-08-30T23:45:01.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121459+0000", 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1489999999999999e-06 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121516+0000", 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7333000000000001e-05 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121593+0000", 2024-08-30T23:45:01.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7107000000000006e-05 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121655+0000", 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2217000000000004e-05 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125135+0000", 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00347956 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125567+0000", 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043209299999999998 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518466+0000", 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39289916899999999 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518501+0000", 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4971000000000001e-05 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518543+0000", 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2261000000000001e-05 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:144420 5.d e589/587 ondisk, result = 0)", 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 65.276691451000005, 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39339614299999998, 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.498 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125154+0000", 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2699999999999999e-07 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125158+0000", 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9559999999999999e-06 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518369+0000", 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39321148299999997 2024-08-30T23:45:01.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518374+0000", 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8439999999999997e-06 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518457+0000", 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2750999999999994e-05 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518462+0000", 2024-08-30T23:45:01.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6069999999999998e-06 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518549+0000", 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6774999999999995e-05 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(1.b e543 from shard 1)", 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 41.238665660000002, 2024-08-30T23:45:01.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91109331999999998, 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7400000000000003e-07 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163183+0000", 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.19e-06 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074076+0000", 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91089344999999999 2024-08-30T23:45:01.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074089+0000", 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3113e-05 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074186+0000", 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7214999999999997e-05 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074207+0000", 2024-08-30T23:45:01.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0372000000000001e-05 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074272+0000", 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5406000000000004e-05 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:154656 5.8 5:11e505a0:::benchmark_data_smithi173_87639_object74416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e610)", 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:26.636768+0000", 2024-08-30T23:45:01.506 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 34.765076938, 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15370636400000001, 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 154656 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636768+0000", 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636768+0000", 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636769+0000", 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.497e-06 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636775+0000", 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6370000000000004e-06 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636786+0000", 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1089e-05 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636794+0000", 2024-08-30T23:45:01.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2460000000000003e-06 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636874+0000", 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9929e-05 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.636994+0000", 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011987199999999999 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.637110+0000", 2024-08-30T23:45:01.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115956 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.639730+0000", 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026200590000000001 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.641599+0000", 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018688209999999999 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:45:01.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.790257+0000", 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14865857499999999 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.790420+0000", 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016268399999999999 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:26.790474+0000", 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3999000000000003e-05 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27034.0:21773 6.d e629/619 6:b9cb6826:::benchmark_data_smithi173_114000_object21772:head v 629'1287, mlcod=629'1284)", 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:59.420504+0000", 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.981340758, 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.059786615000000001, 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.420504+0000", 2024-08-30T23:45:01.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.420504+0000", 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.420505+0000", 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3399999999999998e-07 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.420524+0000", 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9244000000000001e-05 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.479217+0000", 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.058692753 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.479225+0000", 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5580000000000001e-06 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.479295+0000", 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0209999999999994e-05 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.479316+0000", 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.052e-05 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.480247+0000", 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093153799999999998 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.480290+0000", 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3058e-05 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27034.0:22075 6.d e629/619 6:b941b280:::benchmark_data_smithi173_114000_object22074:head v 629'1302, mlcod=629'1299)", 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:59.648365+0000", 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.753479977, 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.046574283000000001, 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:45:01.517 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.648365+0000", 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:45:01.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.648365+0000", 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.648365+0000", 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.37e-07 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.648385+0000", 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0120999999999999e-05 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.694081+0000", 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.045695998000000002 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.694086+0000", 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6260000000000003e-06 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.694131+0000", 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4671e-05 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.694145+0000", 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3886999999999999e-05 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.694900+0000", 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00075589500000000003 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:59.694939+0000", 2024-08-30T23:45:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8448000000000002e-05 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:45:01.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:01.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:01.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:01.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:01.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:01.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:01.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:01.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:01.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:02.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:02.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:02.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:02.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:02.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:02.280 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:01 smithi136 ceph-mon[20207]: from='client.17646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:45:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:01 smithi173 ceph-mon[20554]: from='client.17646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:45:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:01 smithi106 ceph-mon[16603]: from='client.17646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:45:02.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 25350 25334 93.1317 94.5 0.0107434 0.0107247 2024-08-30T23:45:02.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:02.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:45:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.405 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:02.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:02.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:02.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:02.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:02.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:02.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.785 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:02.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:02.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:02.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:02.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:03.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:03.037 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:02 smithi106 ceph-mon[16603]: pgmap v1405: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s; 2712/63024 objects degraded (4.303%) 2024-08-30T23:45:03.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:03.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:03.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:03.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:02 smithi136 ceph-mon[20207]: pgmap v1405: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s; 2712/63024 objects degraded (4.303%) 2024-08-30T23:45:03.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:02 smithi173 ceph-mon[20554]: pgmap v1405: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.3 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.52k op/s; 2712/63024 objects degraded (4.303%) 2024-08-30T23:45:03.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 26554 26538 92.1379 75.25 0.00229778 0.0108461 2024-08-30T23:45:03.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:03.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:03.421 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1404,"stamp":"2024-08-30T23:45:00.806919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1236533792,"num_objects":18861,"num_object_clones":0,"num_object_copies":56583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18859,"num_write_kb":1206976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20791,"ondisk_log_size":20791,"up":108,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8357060,"kb_used_data":4027988,"kb_used_omap":107,"kb_used_meta":4328724,"kb_avail":1116584764,"statfs":{"total":1151940427776,"available":1143382798336,"internally_reserved":0,"allocated":4124659712,"data_stored":4112167655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110317,"internal_metadata":4432613651},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":514260992,"num_objects":7847,"num_object_clones":0,"num_object_copies":23541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":859,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7847,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.923634"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":273,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.822527+0000","last_change":"2024-08-30T23:44:30.071083+0000","last_active":"2024-08-30T23:44:55.822527+0000","last_peered":"2024-08-30T23:44:55.822527+0000","last_clean":"2024-08-30T23:44:55.822527+0000","last_became_active":"2024-08-30T23:42:13.920647+0000","last_became_peered":"2024-08-30T23:42:13.920647+0000","last_unstale":"2024-08-30T23:44:55.822527+0000","last_undegraded":"2024-08-30T23:44:55.822527+0000","last_fullsized":"2024-08-30T23:44:55.822527+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:30.071017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:30.071017+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:32.648433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":236,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.819887+0000","last_change":"2024-08-30T23:44:29.831279+0000","last_active":"2024-08-30T23:44:55.819887+0000","last_peered":"2024-08-30T23:44:55.819887+0000","last_clean":"2024-08-30T23:44:55.819887+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:44:55.819887+0000","last_undegraded":"2024-08-30T23:44:55.819887+0000","last_fullsized":"2024-08-30T23:44:55.819887+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.831212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.831212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:57.676072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":278,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.808149+0000","last_change":"2024-08-30T23:44:31.878852+0000","last_active":"2024-08-30T23:44:55.808149+0000","last_peered":"2024-08-30T23:44:55.808149+0000","last_clean":"2024-08-30T23:44:55.808149+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:44:55.808149+0000","last_undegraded":"2024-08-30T23:44:55.808149+0000","last_fullsized":"2024-08-30T23:44:55.808149+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:31.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:31.878785+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:33.191051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":1311,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.810792+0000","last_change":"2024-08-30T23:44:29.953063+0000","last_active":"2024-08-30T23:44:55.810792+0000","last_peered":"2024-08-30T23:44:55.810792+0000","last_clean":"2024-08-30T23:44:55.810792+0000","last_became_active":"2024-08-30T23:42:40.017555+0000","last_became_peered":"2024-08-30T23:42:40.017555+0000","last_unstale":"2024-08-30T23:44:55.810792+0000","last_undegraded":"2024-08-30T23:44:55.810792+0000","last_fullsized":"2024-08-30T23:44:55.810792+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.953019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.953019+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.856722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"628'1251","reported_seq":1276,"reported_epoch":628,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:44:58.463157+0000","last_change":"2024-08-30T23:44:45.378344+0000","last_active":"2024-08-30T23:44:58.463157+0000","last_peered":"2024-08-30T23:44:58.463157+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"2024-08-30T23:44:42.310484+0000","last_became_peered":"2024-08-30T23:44:42.310484+0000","last_unstale":"2024-08-30T23:44:58.463157+0000","last_undegraded":"2024-08-30T23:44:45.378214+0000","last_fullsized":"2024-08-30T23:44:42.303428+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:01.152599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81985536,"num_objects":1251,"num_object_clones":0,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1251,"num_write_kb":80064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":338,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.822368+0000","last_change":"2024-08-30T23:44:33.929023+0000","last_active":"2024-08-30T23:44:55.822368+0000","last_peered":"2024-08-30T23:44:55.822368+0000","last_clean":"2024-08-30T23:44:55.822368+0000","last_became_active":"2024-08-30T23:42:38.914560+0000","last_became_peered":"2024-08-30T23:42:38.914560+0000","last_unstale":"2024-08-30T23:44:55.822368+0000","last_undegraded":"2024-08-30T23:44:55.822368+0000","last_fullsized":"2024-08-30T23:44:55.822368+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:33.928947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:33.928947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.006533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,9],"acting":[11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"628'1144","reported_seq":1172,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:57.582369+0000","last_change":"2024-08-30T23:44:42.930968+0000","last_active":"2024-08-30T23:44:57.582369+0000","last_peered":"2024-08-30T23:44:57.582369+0000","last_clean":"2024-08-30T23:44:57.582369+0000","last_became_active":"2024-08-30T23:44:42.930551+0000","last_became_peered":"2024-08-30T23:44:42.930551+0000","last_unstale":"2024-08-30T23:44:57.582369+0000","last_undegraded":"2024-08-30T23:44:57.582369+0000","last_fullsized":"2024-08-30T23:44:57.582369+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:58.994565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74973184,"num_objects":1144,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1144,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":573,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.822451+0000","last_change":"2024-08-30T23:44:19.130985+0000","last_active":"2024-08-30T23:44:55.822451+0000","last_peered":"2024-08-30T23:44:55.822451+0000","last_clean":"2024-08-30T23:44:55.822451+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:44:55.822451+0000","last_undegraded":"2024-08-30T23:44:55.822451+0000","last_fullsized":"2024-08-30T23:44:55.822451+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:19.130919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:19.130919+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:26.399208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"629'1458","reported_seq":1488,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:45:00.407467+0000","last_change":"2024-08-30T23:44:42.456106+0000","last_active":"2024-08-30T23:45:00.407467+0000","last_peered":"2024-08-30T23:45:00.407467+0000","last_clean":"2024-08-30T23:45:00.407467+0000","last_became_active":"2024-08-30T23:44:42.455726+0000","last_became_peered":"2024-08-30T23:44:42.455726+0000","last_unstale":"2024-08-30T23:45:00.407467+0000","last_undegraded":"2024-08-30T23:45:00.407467+0000","last_fullsized":"2024-08-30T23:45:00.407467+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1458,"log_dups_size":0,"ondisk_log_size":1458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:08.436990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95551488,"num_objects":1458,"num_object_clones":0,"num_object_copies":4374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":93312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":451,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.827832+0000","last_change":"2024-08-30T23:44:38.508452+0000","last_active":"2024-08-30T23:44:55.827832+0000","last_peered":"2024-08-30T23:44:55.827832+0000","last_clean":"2024-08-30T23:44:55.827832+0000","last_became_active":"2024-08-30T23:42:34.874330+0000","last_became_peered":"2024-08-30T23:42:34.874330+0000","last_unstale":"2024-08-30T23:44:55.827832+0000","last_undegraded":"2024-08-30T23:44:55.827832+0000","last_fullsized":"2024-08-30T23:44:55.827832+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.508393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.508393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:50.318741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"628'1021","reported_seq":1048,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.918995+0000","last_change":"2024-08-30T23:44:43.028992+0000","last_active":"2024-08-30T23:44:55.918995+0000","last_peered":"2024-08-30T23:44:55.918995+0000","last_clean":"2024-08-30T23:44:55.918995+0000","last_became_active":"2024-08-30T23:44:43.028669+0000","last_became_peered":"2024-08-30T23:44:43.028669+0000","last_unstale":"2024-08-30T23:44:55.918995+0000","last_undegraded":"2024-08-30T23:44:55.918995+0000","last_fullsized":"2024-08-30T23:44:55.918995+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:01.890978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66912256,"num_objects":1021,"num_object_clones":0,"num_object_copies":3063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":65344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":431,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.827764+0000","last_change":"2024-08-30T23:44:19.298260+0000","last_active":"2024-08-30T23:44:55.827764+0000","last_peered":"2024-08-30T23:44:55.827764+0000","last_clean":"2024-08-30T23:44:55.827764+0000","last_became_active":"2024-08-30T23:41:38.809561+0000","last_became_peered":"2024-08-30T23:41:38.809561+0000","last_unstale":"2024-08-30T23:44:55.827764+0000","last_undegraded":"2024-08-30T23:44:55.827764+0000","last_fullsized":"2024-08-30T23:44:55.827764+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:19.298203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:19.298203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:46.925422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"628'1247","reported_seq":1275,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.485988+0000","last_change":"2024-08-30T23:44:42.456851+0000","last_active":"2024-08-30T23:44:58.485988+0000","last_peered":"2024-08-30T23:44:58.485988+0000","last_clean":"2024-08-30T23:44:58.485988+0000","last_became_active":"2024-08-30T23:44:42.455669+0000","last_became_peered":"2024-08-30T23:44:42.455669+0000","last_unstale":"2024-08-30T23:44:58.485988+0000","last_undegraded":"2024-08-30T23:44:58.485988+0000","last_fullsized":"2024-08-30T23:44:58.485988+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:44.006738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81723392,"num_objects":1247,"num_object_clones":0,"num_object_copies":3741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1247,"num_write_kb":79808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":453,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.822322+0000","last_change":"2024-08-30T23:44:34.916024+0000","last_active":"2024-08-30T23:44:55.822322+0000","last_peered":"2024-08-30T23:44:55.822322+0000","last_clean":"2024-08-30T23:44:55.822322+0000","last_became_active":"2024-08-30T23:42:20.761486+0000","last_became_peered":"2024-08-30T23:42:20.761486+0000","last_unstale":"2024-08-30T23:44:55.822322+0000","last_undegraded":"2024-08-30T23:44:55.822322+0000","last_fullsized":"2024-08-30T23:44:55.822322+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:34.915971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:34.915971+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:40.704451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"628'1245","reported_seq":1273,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.480953+0000","last_change":"2024-08-30T23:44:42.310819+0000","last_active":"2024-08-30T23:44:58.480953+0000","last_peered":"2024-08-30T23:44:58.480953+0000","last_clean":"2024-08-30T23:44:58.480953+0000","last_became_active":"2024-08-30T23:44:42.310475+0000","last_became_peered":"2024-08-30T23:44:42.310475+0000","last_unstale":"2024-08-30T23:44:58.480953+0000","last_undegraded":"2024-08-30T23:44:58.480953+0000","last_fullsized":"2024-08-30T23:44:58.480953+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1245,"log_dups_size":0,"ondisk_log_size":1245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:10.721902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81592320,"num_objects":1245,"num_object_clones":0,"num_object_copies":3735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1245,"num_write_kb":79680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":443,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.827711+0000","last_change":"2024-08-30T23:44:50.821242+0000","last_active":"2024-08-30T23:44:55.827711+0000","last_peered":"2024-08-30T23:44:55.827711+0000","last_clean":"2024-08-30T23:44:55.827711+0000","last_became_active":"2024-08-30T23:44:50.820959+0000","last_became_peered":"2024-08-30T23:44:50.820959+0000","last_unstale":"2024-08-30T23:44:55.827711+0000","last_undegraded":"2024-08-30T23:44:55.827711+0000","last_fullsized":"2024-08-30T23:44:55.827711+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:32.434222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:32.434222+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.246049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"628'1196","reported_seq":1224,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:57.582430+0000","last_change":"2024-08-30T23:44:43.028344+0000","last_active":"2024-08-30T23:44:57.582430+0000","last_peered":"2024-08-30T23:44:57.582430+0000","last_clean":"2024-08-30T23:44:57.582430+0000","last_became_active":"2024-08-30T23:44:43.027945+0000","last_became_peered":"2024-08-30T23:44:43.027945+0000","last_unstale":"2024-08-30T23:44:57.582430+0000","last_undegraded":"2024-08-30T23:44:57.582430+0000","last_fullsized":"2024-08-30T23:44:57.582430+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.478496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":76544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1312,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.895329+0000","last_change":"2024-08-30T23:44:55.636789+0000","last_active":"2024-08-30T23:44:58.895329+0000","last_peered":"2024-08-30T23:44:58.895329+0000","last_clean":"2024-08-30T23:44:58.895329+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:44:58.895329+0000","last_undegraded":"2024-08-30T23:44:58.895329+0000","last_fullsized":"2024-08-30T23:44:58.895329+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:55.636740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:55.636740+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.637189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"628'1033","reported_seq":1058,"reported_epoch":628,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:44:55.891680+0000","last_change":"2024-08-30T23:44:45.386164+0000","last_active":"2024-08-30T23:44:55.891680+0000","last_peered":"2024-08-30T23:44:55.891680+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"2024-08-30T23:44:42.306853+0000","last_became_peered":"2024-08-30T23:44:42.306853+0000","last_unstale":"2024-08-30T23:44:55.891680+0000","last_undegraded":"2024-08-30T23:44:45.385784+0000","last_fullsized":"2024-08-30T23:44:42.304842+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.220393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67698688,"num_objects":1033,"num_object_clones":0,"num_object_copies":3099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1033,"num_write_kb":66112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":1033}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":207,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.893744+0000","last_change":"2024-08-30T23:44:58.589043+0000","last_active":"2024-08-30T23:44:58.893744+0000","last_peered":"2024-08-30T23:44:58.893744+0000","last_clean":"2024-08-30T23:44:58.893744+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:44:58.893744+0000","last_undegraded":"2024-08-30T23:44:58.893744+0000","last_fullsized":"2024-08-30T23:44:58.893744+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:58.588952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:58.588952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:25.850153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"628'1250","reported_seq":1276,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.485652+0000","last_change":"2024-08-30T23:44:42.314402+0000","last_active":"2024-08-30T23:44:58.485652+0000","last_peered":"2024-08-30T23:44:58.485652+0000","last_clean":"2024-08-30T23:44:58.485652+0000","last_became_active":"2024-08-30T23:44:42.314051+0000","last_became_peered":"2024-08-30T23:44:42.314051+0000","last_unstale":"2024-08-30T23:44:58.485652+0000","last_undegraded":"2024-08-30T23:44:58.485652+0000","last_fullsized":"2024-08-30T23:44:58.485652+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:03.425147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":289,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.809752+0000","last_change":"2024-08-30T23:44:30.127625+0000","last_active":"2024-08-30T23:44:55.809752+0000","last_peered":"2024-08-30T23:44:55.809752+0000","last_clean":"2024-08-30T23:44:55.809752+0000","last_became_active":"2024-08-30T23:43:56.833115+0000","last_became_peered":"2024-08-30T23:43:56.833115+0000","last_unstale":"2024-08-30T23:44:55.809752+0000","last_undegraded":"2024-08-30T23:44:55.809752+0000","last_fullsized":"2024-08-30T23:44:55.809752+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:30.127565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:30.127565+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.812310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"628'1090","reported_seq":1118,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:57.127256+0000","last_change":"2024-08-30T23:44:43.028476+0000","last_active":"2024-08-30T23:44:57.127256+0000","last_peered":"2024-08-30T23:44:57.127256+0000","last_clean":"2024-08-30T23:44:57.127256+0000","last_became_active":"2024-08-30T23:44:43.028019+0000","last_became_peered":"2024-08-30T23:44:43.028019+0000","last_unstale":"2024-08-30T23:44:57.127256+0000","last_undegraded":"2024-08-30T23:44:57.127256+0000","last_fullsized":"2024-08-30T23:44:57.127256+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.206952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71434240,"num_objects":1090,"num_object_clones":0,"num_object_copies":3270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":69760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":277,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.893748+0000","last_change":"2024-08-30T23:44:38.472833+0000","last_active":"2024-08-30T23:44:58.893748+0000","last_peered":"2024-08-30T23:44:58.893748+0000","last_clean":"2024-08-30T23:44:58.893748+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:44:58.893748+0000","last_undegraded":"2024-08-30T23:44:58.893748+0000","last_fullsized":"2024-08-30T23:44:58.893748+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.472767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.472767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.283833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"628'1063","reported_seq":1091,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.931767+0000","last_change":"2024-08-30T23:44:43.029448+0000","last_active":"2024-08-30T23:44:55.931767+0000","last_peered":"2024-08-30T23:44:55.931767+0000","last_clean":"2024-08-30T23:44:55.931767+0000","last_became_active":"2024-08-30T23:44:43.029122+0000","last_became_peered":"2024-08-30T23:44:43.029122+0000","last_unstale":"2024-08-30T23:44:55.931767+0000","last_undegraded":"2024-08-30T23:44:55.931767+0000","last_fullsized":"2024-08-30T23:44:55.931767+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1063,"log_dups_size":0,"ondisk_log_size":1063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.949159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69664768,"num_objects":1063,"num_object_clones":0,"num_object_copies":3189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1063,"num_write_kb":68032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":207,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.893859+0000","last_change":"2024-08-30T23:42:34.888040+0000","last_active":"2024-08-30T23:44:58.893859+0000","last_peered":"2024-08-30T23:44:58.893859+0000","last_clean":"2024-08-30T23:44:58.893859+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:44:58.893859+0000","last_undegraded":"2024-08-30T23:44:58.893859+0000","last_fullsized":"2024-08-30T23:44:58.893859+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:42:34.887995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:42:34.887995+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:44:59.627487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"628'1170","reported_seq":1198,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:57.580962+0000","last_change":"2024-08-30T23:44:42.930929+0000","last_active":"2024-08-30T23:44:57.580962+0000","last_peered":"2024-08-30T23:44:57.580962+0000","last_clean":"2024-08-30T23:44:57.580962+0000","last_became_active":"2024-08-30T23:44:42.930567+0000","last_became_peered":"2024-08-30T23:44:42.930567+0000","last_unstale":"2024-08-30T23:44:57.580962+0000","last_undegraded":"2024-08-30T23:44:57.580962+0000","last_fullsized":"2024-08-30T23:44:57.580962+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1170,"log_dups_size":0,"ondisk_log_size":1170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.461680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1170,"num_write_kb":74880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":462,"reported_epoch":628,"state":"active+undersized","last_fresh":"2024-08-30T23:44:55.822304+0000","last_change":"2024-08-30T23:44:03.894450+0000","last_active":"2024-08-30T23:44:55.822304+0000","last_peered":"2024-08-30T23:44:55.822304+0000","last_clean":"2024-08-30T23:44:01.872560+0000","last_became_active":"2024-08-30T23:44:03.894450+0000","last_became_peered":"2024-08-30T23:44:03.894450+0000","last_unstale":"2024-08-30T23:44:55.822304+0000","last_undegraded":"2024-08-30T23:44:55.822304+0000","last_fullsized":"2024-08-30T23:44:03.891268+0000","mapping_epoch":594,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.288446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.288446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:44:59.975534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"628'1042","reported_seq":1070,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:56.528722+0000","last_change":"2024-08-30T23:44:43.028106+0000","last_active":"2024-08-30T23:44:56.528722+0000","last_peered":"2024-08-30T23:44:56.528722+0000","last_clean":"2024-08-30T23:44:56.528722+0000","last_became_active":"2024-08-30T23:44:43.027789+0000","last_became_peered":"2024-08-30T23:44:43.027789+0000","last_unstale":"2024-08-30T23:44:56.528722+0000","last_undegraded":"2024-08-30T23:44:56.528722+0000","last_fullsized":"2024-08-30T23:44:56.528722+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1042,"log_dups_size":0,"ondisk_log_size":1042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.782764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68288512,"num_objects":1042,"num_object_clones":0,"num_object_copies":3126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1042,"num_write_kb":66688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":402,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.809724+0000","last_change":"2024-08-30T23:44:37.215950+0000","last_active":"2024-08-30T23:44:55.809724+0000","last_peered":"2024-08-30T23:44:55.809724+0000","last_clean":"2024-08-30T23:44:55.809724+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:44:55.809724+0000","last_undegraded":"2024-08-30T23:44:55.809724+0000","last_fullsized":"2024-08-30T23:44:55.809724+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:37.215896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:37.215896+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.523982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"629'1403","reported_seq":1433,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:45:00.370783+0000","last_change":"2024-08-30T23:44:42.456265+0000","last_active":"2024-08-30T23:45:00.370783+0000","last_peered":"2024-08-30T23:45:00.370783+0000","last_clean":"2024-08-30T23:45:00.370783+0000","last_became_active":"2024-08-30T23:44:42.455981+0000","last_became_peered":"2024-08-30T23:44:42.455981+0000","last_unstale":"2024-08-30T23:45:00.370783+0000","last_undegraded":"2024-08-30T23:45:00.370783+0000","last_fullsized":"2024-08-30T23:45:00.370783+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1403,"log_dups_size":0,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.062797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91947008,"num_objects":1403,"num_object_clones":0,"num_object_copies":4209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1403,"num_write_kb":89792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1288,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.810837+0000","last_change":"2024-08-30T23:44:13.869717+0000","last_active":"2024-08-30T23:44:55.810837+0000","last_peered":"2024-08-30T23:44:55.810837+0000","last_clean":"2024-08-30T23:44:55.810837+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:44:55.810837+0000","last_undegraded":"2024-08-30T23:44:55.810837+0000","last_fullsized":"2024-08-30T23:44:55.810837+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:13.869671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:13.869671+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.045381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"628'1008","reported_seq":1036,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:56.512134+0000","last_change":"2024-08-30T23:44:42.930674+0000","last_active":"2024-08-30T23:44:56.512134+0000","last_peered":"2024-08-30T23:44:56.512134+0000","last_clean":"2024-08-30T23:44:56.512134+0000","last_became_active":"2024-08-30T23:44:42.930380+0000","last_became_peered":"2024-08-30T23:44:42.930380+0000","last_unstale":"2024-08-30T23:44:56.512134+0000","last_undegraded":"2024-08-30T23:44:56.512134+0000","last_fullsized":"2024-08-30T23:44:56.512134+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1008,"log_dups_size":0,"ondisk_log_size":1008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:09.285121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66060288,"num_objects":1008,"num_object_clones":0,"num_object_copies":3024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1008,"num_write_kb":64512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,11],"acting":[10,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":412,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.901190+0000","last_change":"2024-08-30T23:44:12.112535+0000","last_active":"2024-08-30T23:44:58.901190+0000","last_peered":"2024-08-30T23:44:58.901190+0000","last_clean":"2024-08-30T23:44:58.901190+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:44:58.901190+0000","last_undegraded":"2024-08-30T23:44:58.901190+0000","last_fullsized":"2024-08-30T23:44:58.901190+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:12.112429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:12.112429+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:38.348925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"628'1238","reported_seq":1266,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:58.491637+0000","last_change":"2024-08-30T23:44:42.305791+0000","last_active":"2024-08-30T23:44:58.491637+0000","last_peered":"2024-08-30T23:44:58.491637+0000","last_clean":"2024-08-30T23:44:58.491637+0000","last_became_active":"2024-08-30T23:44:42.305499+0000","last_became_peered":"2024-08-30T23:44:42.305499+0000","last_unstale":"2024-08-30T23:44:58.491637+0000","last_undegraded":"2024-08-30T23:44:58.491637+0000","last_fullsized":"2024-08-30T23:44:58.491637+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":1238,"log_dups_size":0,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:00.262210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81133568,"num_objects":1238,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1238,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,7],"acting":[3,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":190,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.822436+0000","last_change":"2024-08-30T23:44:32.963615+0000","last_active":"2024-08-30T23:44:55.822436+0000","last_peered":"2024-08-30T23:44:55.822436+0000","last_clean":"2024-08-30T23:44:55.822436+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:44:55.822436+0000","last_undegraded":"2024-08-30T23:44:55.822436+0000","last_fullsized":"2024-08-30T23:44:55.822436+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:32.963552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:32.963552+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.292206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1145,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-08-30T23:44:55.810939+0000","last_change":"2024-08-30T23:44:33.986377+0000","last_active":"2024-08-30T23:44:55.810939+0000","last_peered":"2024-08-30T23:44:55.810939+0000","last_clean":"2024-08-30T23:44:55.810939+0000","last_became_active":"2024-08-30T23:41:40.752006+0000","last_became_peered":"2024-08-30T23:41:40.752006+0000","last_unstale":"2024-08-30T23:44:55.810939+0000","last_undegraded":"2024-08-30T23:44:55.810939+0000","last_fullsized":"2024-08-30T23:44:55.810939+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:33.986333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:33.986333+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.206532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1235943424,"num_objects":18859,"num_object_clones":0,"num_object_copies":56577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18859,"num_write_kb":1206976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3635216384,"data_stored":3635216384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18859,"ondisk_log_size":18859,"up":46,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187701,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635364,"kb_used_data":207116,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93109788,"statfs":{"total":95995035648,"available":95344422912,"internally_reserved":0,"allocated":212086784,"data_stored":211054129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":438499780},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449341,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":580772,"kb_used_data":238604,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93164380,"statfs":{"total":95995035648,"available":95400325120,"internally_reserved":0,"allocated":244330496,"data_stored":243297849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":350353856},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710976,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":875756,"kb_used_data":447512,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92869396,"statfs":{"total":95995035648,"available":95098261504,"internally_reserved":0,"allocated":458252288,"data_stored":457215176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972613,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538156,"kb_used_data":200148,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93206996,"statfs":{"total":95995035648,"available":95443963904,"internally_reserved":0,"allocated":204951552,"data_stored":203915353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234248,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":861212,"kb_used_data":374668,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92883940,"statfs":{"total":95995035648,"available":95113154560,"internally_reserved":0,"allocated":383660032,"data_stored":382627385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528591,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564580,"kb_used_data":212684,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93180572,"statfs":{"total":95995035648,"available":95416905728,"internally_reserved":0,"allocated":217788416,"data_stored":216755769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":566,"seq":2430951489545,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552508,"kb_used_data":479660,"kb_used_omap":4,"kb_used_meta":72827,"kb_avail":93192644,"statfs":{"total":95995035648,"available":95429267456,"internally_reserved":0,"allocated":491171840,"data_stored":490053979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4411,"internal_metadata":74575557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953753,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764268,"kb_used_data":358232,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92980884,"statfs":{"total":95995035648,"available":95212425216,"internally_reserved":0,"allocated":366829568,"data_stored":365792456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149981,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112868,"kb_used_data":626324,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92632284,"statfs":{"total":95995035648,"available":94855458816,"internally_reserved":0,"allocated":641355776,"data_stored":640319577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":498203079},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411619,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":782564,"kb_used_data":480660,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92962588,"statfs":{"total":95995035648,"available":95193690112,"internally_reserved":0,"allocated":492195840,"data_stored":491159641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830434,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":449364,"kb_used_data":259008,"kb_used_omap":87,"kb_used_meta":190248,"kb_avail":93295788,"statfs":{"total":95995035648,"available":95534886912,"internally_reserved":0,"allocated":265224192,"data_stored":264196060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":194814111},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639648,"kb_used_data":143372,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93105504,"statfs":{"total":95995035648,"available":95340036096,"internally_reserved":0,"allocated":146812928,"data_stored":145780281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158007296,"data_stored":158007296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265289728,"data_stored":265289728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":496631808,"data_stored":496631808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":658374656,"data_stored":658374656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":372834304,"data_stored":372834304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":217972736,"data_stored":217972736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":380436480,"data_stored":380436480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":202702848,"data_stored":202702848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":461570048,"data_stored":461570048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204406784,"data_stored":204406784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":216989696,"data_stored":216989696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:45:03.423 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [6, 7, 11] 2024-08-30T23:45:03.423 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '6', '7', '11'] 2024-08-30T23:45:03.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 1.0 6 7 11 2024-08-30T23:45:03.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:03.526 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:03.531 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:03.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:03.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:03.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:03.772 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:03.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:03.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:03.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:03.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:04.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:04.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:04.139 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:04.139 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:45:04.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.139 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:27723 6.d e630/619 6:be710a39:::benchmark_data_smithi173_114000_object27722:head v 630'1638, mlcod=630'1634)", 2024-08-30T23:45:04.139 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:04.122260+0000", 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.016313217000000001, 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016341635, 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.122260+0000", 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:04.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.122260+0000", 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.122261+0000", 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.088e-06 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.122336+0000", 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5006999999999995e-05 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137440+0000", 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015103728 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137447+0000", 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8369999999999999e-06 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137504+0000", 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7327000000000002e-05 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137520+0000", 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6543000000000001e-05 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.138576+0000", 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001055636 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:27757 6.4 6:222a2eb7:::benchmark_data_smithi173_114000_object27756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e630)", 2024-08-30T23:45:04.143 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:04.137481+0000", 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.001092289, 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011980579999999999, 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 27757 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137481+0000", 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137481+0000", 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137482+0000", 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3540000000000001e-06 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137504+0000", 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2268000000000001e-05 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137513+0000", 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1280000000000008e-06 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137520+0000", 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9789999999999996e-06 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137582+0000", 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2407000000000001e-05 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137707+0000", 2024-08-30T23:45:04.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000125506 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3", 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.137842+0000", 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013456199999999999 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:27725 6.d e630/619 6:b053ad9c:::benchmark_data_smithi173_114000_object27724:head v 630'1639, mlcod=630'1634)", 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:04.123382+0000", 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015191444, 2024-08-30T23:45:04.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015378737, 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.123382+0000", 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.123382+0000", 2024-08-30T23:45:04.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.123382+0000", 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9500000000000002e-07 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.123473+0000", 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1081000000000004e-05 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.138618+0000", 2024-08-30T23:45:04.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015145031 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.138625+0000", 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.365e-06 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.138696+0000", 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1398999999999999e-05 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:04.138746+0000", 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9502999999999997e-05 2024-08-30T23:45:04.151 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.152 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:04.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:04.152 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:04.152 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 3 2024-08-30T23:45:04.152 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:04.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:04.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:04.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:04.309 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:04.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 28163 28147 92.5809 100.562 0.00131626 0.0107884 2024-08-30T23:45:04.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:04.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:04.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:04.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:04.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:04.675 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:04.676 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:04.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:04.704 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:04.704 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:04.733 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:04.734 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:04.759 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:04.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:04.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:04.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:04.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:45:04.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:04.943 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:04.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:04.989 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:04.990 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:04.990 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:05.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:04 smithi136 ceph-mon[20207]: pgmap v1406: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s; 3103/74664 objects degraded (4.156%) 2024-08-30T23:45:05.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2067894610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:05.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:05.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:05.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:05.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:05.212 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:05.212 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:05.213 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:04 smithi106 ceph-mon[16603]: pgmap v1406: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s; 3103/74664 objects degraded (4.156%) 2024-08-30T23:45:05.213 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2067894610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:05.233 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:05.233 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:05.256 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:05.256 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:05.283 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:05.283 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:05.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:05.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:45:05.303 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:05.315 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:05.316 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:05.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.317 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:05.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:04 smithi173 ceph-mon[20554]: pgmap v1406: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s; 3103/74664 objects degraded (4.156%) 2024-08-30T23:45:05.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2067894610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:05.348 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:05.349 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:05.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:45:05.371292+0000 min lat: 0.00104751 max lat: 0.297715 avg lat: 0.0109749 2024-08-30T23:45:05.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:45:05.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 29155 29139 91.0517 62 0.0202596 0.0109749 2024-08-30T23:45:05.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:05.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:05.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:05.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:05.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:05.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:05.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:05.822 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:05.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:05.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:05.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:06.040 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3103/74664 objects degraded (4.156%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:45:06.045 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:06.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:06.070 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:06.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:06.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:06.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3103/74664 objects degraded (4.156%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:45:06.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3103/74664 objects degraded (4.156%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:45:06.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:06.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 30261 30245 90.0071 69.125 0.00497602 0.0110959 2024-08-30T23:45:06.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:06.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:06.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:06.613 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:06.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.749 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:06.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:06.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:06.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:06.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:06.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:06.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:07.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:07.123 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:07.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:07.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:07.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:06 smithi136 ceph-mon[20207]: pgmap v1407: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 3103/77310 objects degraded (4.014%) 2024-08-30T23:45:07.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:06 smithi173 ceph-mon[20554]: pgmap v1407: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 3103/77310 objects degraded (4.014%) 2024-08-30T23:45:07.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:07.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:06 smithi106 ceph-mon[16603]: pgmap v1407: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.6 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s; 3103/77310 objects degraded (4.014%) 2024-08-30T23:45:07.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 31640 31624 89.8332 86.1875 0.0111161 0.0111247 2024-08-30T23:45:07.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:07.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:07.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:07.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:07.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:07.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:07.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:07.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:07.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:07.886 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:07.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:07.968 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:45:07.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:08.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:08.024 INFO:teuthology.orchestra.run.smithi106.stderr:set 1.0 pg_upmap mapping to [6,7,11] 2024-08-30T23:45:08.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:45:08.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:08.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:08.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:08.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1801292581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:08.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/458786404' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 7, 11]} : dispatch 2024-08-30T23:45:08.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:08.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1801292581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/458786404' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 7, 11]} : dispatch 2024-08-30T23:45:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1801292581' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/458786404' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 7, 11]} : dispatch 2024-08-30T23:45:08.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 15 32975 32960 89.5576 83.5 0.0193488 0.0111594 2024-08-30T23:45:08.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:08.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:08.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:08.529 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:08.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:08.659 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:08.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:08.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:08.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:08.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:08.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:08.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:09.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:09.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.054 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:08 smithi106 ceph-mon[16603]: pgmap v1408: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 3500/88020 objects degraded (3.976%) 2024-08-30T23:45:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:08 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1801292581' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:45:09.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/458786404' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 7, 11]}]': finished 2024-08-30T23:45:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:08 smithi106 ceph-mon[16603]: osdmap e631: 12 total, 11 up, 12 in 2024-08-30T23:45:09.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:09.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:09.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:08 smithi136 ceph-mon[20207]: pgmap v1408: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 3500/88020 objects degraded (3.976%) 2024-08-30T23:45:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:08 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1801292581' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:45:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/458786404' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 7, 11]}]': finished 2024-08-30T23:45:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:08 smithi136 ceph-mon[20207]: osdmap e631: 12 total, 11 up, 12 in 2024-08-30T23:45:09.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:08 smithi173 ceph-mon[20554]: pgmap v1408: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.8 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 3500/88020 objects degraded (3.976%) 2024-08-30T23:45:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:08 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1801292581' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:45:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/458786404' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.0", "id": [6, 7, 11]}]': finished 2024-08-30T23:45:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:08 smithi173 ceph-mon[20554]: osdmap e631: 12 total, 11 up, 12 in 2024-08-30T23:45:09.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 34889 34873 90.8074 119.562 0.0235423 0.0110054 2024-08-30T23:45:09.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:09.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:09.506 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:09.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:09.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:09.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:09.683 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:09.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:09.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:09.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:09.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:09.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:10.026 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:10.026 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:10.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:09 smithi106 ceph-mon[16603]: osdmap e632: 12 total, 11 up, 12 in 2024-08-30T23:45:10.026 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:09 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:45:10.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:10.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:10.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:10.183 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:10.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:09 smithi136 ceph-mon[20207]: osdmap e632: 12 total, 11 up, 12 in 2024-08-30T23:45:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:09 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:45:10.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:09 smithi173 ceph-mon[20554]: osdmap e632: 12 total, 11 up, 12 in 2024-08-30T23:45:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:09 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:45:10.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 35679 35663 89.15 49.375 0.0139097 0.0112091 2024-08-30T23:45:10.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:10.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:10.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:10.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:10.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:10.673 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:10.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:45:10.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:10.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:10.824 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:10.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:10.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:10.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:11.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:11.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:11.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:11.214 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:11.236 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:11.237 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:11.237 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:10 smithi106 ceph-mon[16603]: pgmap v1411: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 3917/94128 objects degraded (4.161%) 2024-08-30T23:45:11.237 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3917/94128 objects degraded (4.161%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:11.271 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:11.275 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:11.275 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:10 smithi136 ceph-mon[20207]: pgmap v1411: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 3917/94128 objects degraded (4.161%) 2024-08-30T23:45:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3917/94128 objects degraded (4.161%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:11.305 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:11.306 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:11.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:10 smithi173 ceph-mon[20554]: pgmap v1411: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 1.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 3917/94128 objects degraded (4.161%) 2024-08-30T23:45:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3917/94128 objects degraded (4.161%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:11.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:11.338 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:11.343 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:11.365 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:11.365 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:11.366 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:11.370 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 37211 37195 89.4035 95.75 0.0106002 0.0111799 2024-08-30T23:45:11.396 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:11.396 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:11.425 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:11.425 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:11.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:11.454 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:11.455 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:11.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:11.484 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:11.484 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:11.510 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:11.510 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:11.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:45:11.543 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:11.543 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:11.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:11.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:11.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:11.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:11.819 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:11.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:11.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:11.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:11.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:12.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:12.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:12.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1228297665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:12.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:12.217 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86177 5.7 5.7fdfa7e7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.259007637, 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97022776200000005, 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.218 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86177 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731726+0000", 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.736e-06 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731731+0000", 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999998e-06 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731739+0000", 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6569999999999997e-06 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731746+0000", 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:45:12.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732139+0000", 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039219999999999999 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732147+0000", 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8910000000000001e-06 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701940+0000", 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979272699999997 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701952+0000", 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1629e-05 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86218 5.7 5.665a2e07 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.223 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25871313600001, 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96998314699999999, 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86218 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.82e-07 2024-08-30T23:45:12.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732023+0000", 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8060000000000001e-06 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732043+0000", 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9873000000000001e-05 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732050+0000", 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4950000000000002e-06 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732187+0000", 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013689799999999999 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732194+0000", 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6039999999999998e-06 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701991+0000", 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979669899999998 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702002+0000", 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.119e-05 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86288 5.7 5.4dc78ea7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.228 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25867259699999, 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96997772299999996, 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86288 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732060+0000", 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9799999999999999e-07 2024-08-30T23:45:12.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732062+0000", 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2019999999999998e-06 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732068+0000", 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6710000000000002e-06 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732072+0000", 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9770000000000002e-06 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732233+0000", 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016027700000000001 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732240+0000", 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9990000000000002e-06 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702030+0000", 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979006599999995 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702037+0000", 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:45:12.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86300 5.7 5.f6fa46a7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25865437900001, 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97000272399999998, 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.234 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86300 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732078+0000", 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4400000000000001e-07 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732079+0000", 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4839999999999999e-06 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732083+0000", 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2819999999999998e-06 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732094+0000", 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.075e-05 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732281+0000", 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000186517 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732286+0000", 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.553e-06 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702067+0000", 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96978114199999998 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702080+0000", 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2751999999999999e-05 2024-08-30T23:45:12.238 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86325 5.7 5.e2384ca7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.258630659, 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97003113200000002, 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.239 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86325 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732102+0000", 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5499999999999998e-07 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732106+0000", 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4460000000000003e-06 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732113+0000", 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7989999999999996e-06 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732119+0000", 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4080000000000003e-06 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732342+0000", 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00022250899999999999 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732347+0000", 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7969999999999997e-06 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702121+0000", 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96977379200000002 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702132+0000", 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0926e-05 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.243 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86376 5.7 5.ad230c87 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25860599800001, 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97006071599999999, 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.244 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86376 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4799999999999999e-07 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732129+0000", 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6529999999999998e-06 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732135+0000", 2024-08-30T23:45:12.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2210000000000002e-06 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732139+0000", 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.211e-06 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732389+0000", 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00024981999999999999 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732396+0000", 2024-08-30T23:45:12.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1189999999999999e-06 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702177+0000", 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96978100300000003 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702186+0000", 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2410000000000001e-06 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86452 5.7 5.ed9357e7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25857826999999, 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97006287000000002, 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.249 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86452 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732154+0000", 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4200000000000002e-07 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732162+0000", 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6659999999999997e-06 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732169+0000", 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6839999999999999e-06 2024-08-30T23:45:12.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732173+0000", 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1049999999999997e-06 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732425+0000", 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000251508 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732428+0000", 2024-08-30T23:45:12.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.202e-06 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702206+0000", 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96977835099999998 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702216+0000", 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0012e-05 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86465 5.7 5.831aa467 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25855308300001, 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97009033300000003, 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86465 2024-08-30T23:45:12.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6199999999999999e-07 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732180+0000", 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1510000000000001e-06 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732183+0000", 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3289999999999998e-06 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732189+0000", 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732442+0000", 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025276900000000002 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732445+0000", 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.638e-06 2024-08-30T23:45:12.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702257+0000", 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96981182700000002 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702269+0000", 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2242000000000001e-05 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.258 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86565 5.7 5.628bf187 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25853875199999, 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97012332999999995, 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86565 2024-08-30T23:45:12.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.260 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:45:12.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:45:12.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2499999999999999e-07 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732194+0000", 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.06e-06 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732198+0000", 2024-08-30T23:45:12.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2650000000000001e-06 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732202+0000", 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4150000000000001e-06 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732476+0000", 2024-08-30T23:45:12.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027440400000000001 2024-08-30T23:45:12.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732482+0000", 2024-08-30T23:45:12.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1539999999999999e-06 2024-08-30T23:45:12.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702307+0000", 2024-08-30T23:45:12.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982474900000004 2024-08-30T23:45:12.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702316+0000", 2024-08-30T23:45:12.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0580000000000004e-06 2024-08-30T23:45:12.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86603 5.7 5.32d9a587 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25852402199999, 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97014088799999998, 2024-08-30T23:45:12.268 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86603 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4299999999999999e-07 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732210+0000", 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.655e-06 2024-08-30T23:45:12.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732215+0000", 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6910000000000002e-06 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732220+0000", 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6489999999999999e-06 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732509+0000", 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028865599999999998 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732511+0000", 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7049999999999999e-06 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702338+0000", 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.969826246 2024-08-30T23:45:12.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702349+0000", 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0943e-05 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.273 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86625 5.7 5.9d66eb87 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.258505711, 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97015716600000002, 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.274 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86625 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:45:12.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:45:12.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:45:12.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5199999999999998e-07 2024-08-30T23:45:12.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732228+0000", 2024-08-30T23:45:12.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0389999999999999e-06 2024-08-30T23:45:12.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732234+0000", 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3979999999999998e-06 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732239+0000", 2024-08-30T23:45:12.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3029999999999999e-06 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732541+0000", 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000301582 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732548+0000", 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8709999999999999e-06 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702376+0000", 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982850899999995 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702383+0000", 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1119999999999998e-06 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86638 5.7 5.ef24e247 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.258486734, 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97016687199999996, 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86638 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.282 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8000000000000001e-07 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732247+0000", 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8500000000000001e-06 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732253+0000", 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5600000000000001e-06 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732257+0000", 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4939999999999997e-06 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732573+0000", 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031563199999999999 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732576+0000", 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6120000000000001e-06 2024-08-30T23:45:12.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702405+0000", 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982966100000001 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702412+0000", 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6830000000000002e-06 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86682 5.7 5.ad2d3f47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.258458252, 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97017973300000004, 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86682 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.287 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732274+0000", 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2000000000000001e-07 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732275+0000", 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.257e-06 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732279+0000", 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7469999999999999e-06 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732282+0000", 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1089999999999998e-06 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732606+0000", 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032443800000000003 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732613+0000", 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3810000000000004e-06 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702433+0000", 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982075199999995 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702453+0000", 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9729e-05 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.291 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86686 5.7 5.c68ec547 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25844634800001, 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97021215299999997, 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86686 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732286+0000", 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.28e-07 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732287+0000", 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.065e-06 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732290+0000", 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5499999999999999e-06 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732294+0000", 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2649999999999998e-06 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732637+0000", 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034266199999999998 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732640+0000", 2024-08-30T23:45:12.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5790000000000002e-06 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702487+0000", 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96984751400000002 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702498+0000", 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.029e-05 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.296 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86696 5.7 5.36b4ca47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.258432863, 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.970319865, 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.297 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86696 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.53e-07 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732301+0000", 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.618e-06 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732308+0000", 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6399999999999997e-06 2024-08-30T23:45:12.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732314+0000", 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2700000000000004e-06 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732670+0000", 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035588900000000002 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732676+0000", 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1500000000000004e-06 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702608+0000", 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96993175200000004 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702619+0000", 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1192999999999999e-05 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.301 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86697 5.7 5.f642fc47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 180.25841200599999, 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97034427400000001, 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.302 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86697 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8500000000000002e-07 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732323+0000", 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4320000000000002e-06 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732344+0000", 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1539000000000002e-05 2024-08-30T23:45:12.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732349+0000", 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0289999999999996e-06 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732694+0000", 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034466299999999999 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732696+0000", 2024-08-30T23:45:12.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6139999999999999e-06 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702655+0000", 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96995884499999996 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702664+0000", 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7669999999999992e-06 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:95915 5.5 5:a7a5f5b9:::benchmark_data_smithi173_87639_object15675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 162.69982204799999, 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.567720742, 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 95915 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.307 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290911+0000", 2024-08-30T23:45:12.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6500000000000001e-06 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290917+0000", 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3020000000000002e-06 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290930+0000", 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.305e-05 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290939+0000", 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0669999999999998e-06 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.371235+0000", 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080296505000000004 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.374504+0000", 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032687430000000002 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858349+0000", 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48384462 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858630+0000", 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028180499999999999 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:128523 5.15 e573/539 5:ab7a4302:::benchmark_data_smithi173_87639_object48283:head v 573'6948, mlcod=410'2803)", 2024-08-30T23:45:12.311 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:34.346043+0000", 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.644689033000006, 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55079527699999997, 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346043+0000", 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346043+0000", 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346044+0000", 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6139999999999999e-06 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346053+0000", 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8829999999999995e-06 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894635+0000", 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54858180999999995 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894641+0000", 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5949999999999998e-06 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894717+0000", 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6699000000000005e-05 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894736+0000", 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8321999999999998e-05 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.896812+0000", 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020768509999999998 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.896838+0000", 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5503e-05 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.315 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(5.c e600,530 5:30fc877c:::benchmark_data_smithi173_87639_object13490:head,531'6056)", 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:44:11.273195+0000", 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 60.717536285999998, 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48882419999999999, 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.273195+0000", 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.273195+0000", 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.273196+0000", 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9700000000000002e-07 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.273199+0000", 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0469999999999998e-06 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.761892+0000", 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48869317400000001 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.761902+0000", 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.028e-05 2024-08-30T23:45:12.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.761957+0000", 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4493999999999997e-05 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.761964+0000", 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5490000000000001e-06 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:11.762020+0000", 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5259000000000001e-05 2024-08-30T23:45:12.319 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(5.15 e602,590 5:ae155e6a:::benchmark_data_smithi173_87639_object50280:head,576'7007)", 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 54.617587794999999, 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54375525700000005, 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:12.320 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8999999999999998e-07 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373147+0000", 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3369999999999998e-06 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916687+0000", 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54354048600000004 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916702+0000", 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5315000000000002e-05 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916802+0000", 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9502999999999999e-05 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916815+0000", 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2757e-05 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:12.323 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916899+0000", 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4568999999999996e-05 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:12.324 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:12.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1228297665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:12.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:12.326 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1228297665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:12.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 38562 38546 89.2194 84.4375 0.0401627 0.0112 2024-08-30T23:45:12.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:12.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:12.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:12.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:12.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:12.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.696 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:12.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:12.825 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:12.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:12.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:12.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:13.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:13.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:13.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:13.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:12 smithi136 ceph-mon[20207]: pgmap v1412: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 4376/103050 objects degraded (4.246%) 2024-08-30T23:45:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:13.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:12 smithi173 ceph-mon[20554]: pgmap v1412: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 4376/103050 objects degraded (4.246%) 2024-08-30T23:45:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:13.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:12 smithi106 ceph-mon[16603]: pgmap v1412: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 4376/103050 objects degraded (4.246%) 2024-08-30T23:45:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:13.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 39846 39830 88.8988 80.25 0.0034678 0.0112415 2024-08-30T23:45:13.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:13.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:13.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:13.583 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:13.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:13.713 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:13.813 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:45:13.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:45:13.813 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-30T23:45:13.813 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:45:13.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:13.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:13.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:13.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:13.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:14.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:14.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:13 smithi106 ceph-mon[16603]: osdmap e633: 12 total, 11 up, 12 in 2024-08-30T23:45:14.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:14.209 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:13 smithi136 ceph-mon[20207]: osdmap e633: 12 total, 11 up, 12 in 2024-08-30T23:45:14.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:13 smithi173 ceph-mon[20554]: osdmap e633: 12 total, 11 up, 12 in 2024-08-30T23:45:14.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:14.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 15 40936 40921 88.1846 68.1875 0.0240824 0.0113331 2024-08-30T23:45:14.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:14.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:45:14.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:14.512 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:14.574 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:14.606 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:14.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:14.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:14.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:14.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:14.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:14.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:14.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:15.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:15.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:15.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:15.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:15.227 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:15 smithi106 ceph-mon[16603]: pgmap v1414: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 93 MiB/s wr, 1.48k op/s; 4779/114684 objects degraded (4.167%); 3.9 KiB/s, 0 objects/s recovering 2024-08-30T23:45:15.227 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:15.228 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:15.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:15.269 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.270 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:15 smithi136 ceph-mon[20207]: pgmap v1414: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 93 MiB/s wr, 1.48k op/s; 4779/114684 objects degraded (4.167%); 3.9 KiB/s, 0 objects/s recovering 2024-08-30T23:45:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:15 smithi173 ceph-mon[20554]: pgmap v1414: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 93 MiB/s wr, 1.48k op/s; 4779/114684 objects degraded (4.167%); 3.9 KiB/s, 0 objects/s recovering 2024-08-30T23:45:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:15.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:15.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 42749 42733 89.0198 113.25 0.00495881 0.0112272 2024-08-30T23:45:15.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:15.472 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:15.505 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:15.527 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:15.543 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:15.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:15.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:15.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:15.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:15.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:15.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:15.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:16.011 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:45:16.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:16.043 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:16.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:16.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:16.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:16.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:16 smithi136 ceph-mon[20207]: osdmap e634: 12 total, 11 up, 12 in 2024-08-30T23:45:16.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:16 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:45:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:16 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:16 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1776532604' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4779/114684 objects degraded (4.167%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:16.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:16 smithi173 ceph-mon[20554]: osdmap e634: 12 total, 11 up, 12 in 2024-08-30T23:45:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:16 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:45:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:16 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:16 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1776532604' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4779/114684 objects degraded (4.167%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:16.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:16 smithi106 ceph-mon[16603]: osdmap e634: 12 total, 11 up, 12 in 2024-08-30T23:45:16.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:16 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:45:16.356 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:16 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:16 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1776532604' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4779/114684 objects degraded (4.167%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:16.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 43837 43821 88.3416 68 0.00702753 0.0113134 2024-08-30T23:45:16.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:16.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:16.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:16.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:16.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:16.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:16.759 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:16.759 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:45:16.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:44412 6.a e635/619 6:57851176:::benchmark_data_smithi173_114000_object44411:head v 635'2728, mlcod=635'2725)", 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:16.753793+0000", 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.004597683, 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046266429999999997, 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.753793+0000", 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:16.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.753793+0000", 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.753794+0000", 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.088e-06 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.753901+0000", 2024-08-30T23:45:16.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010733799999999999 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.756261+0000", 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023600700000000001 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.756267+0000", 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2040000000000002e-06 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.756321+0000", 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3383e-05 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:16.756359+0000", 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8671000000000003e-05 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:16.763 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-08-30T23:45:16.764 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:16.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:16.792 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:45:16.792 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:16.792 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 635, 2024-08-30T23:45:16.792 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:45:16.792 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:45:16.792 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:45:16.006965+0000", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:44:02.874029+0000", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:38:32.101509+0000", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5806080, 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "nodeep-scrub", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "noscrub", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:16.793 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 28, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 6, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "luminous", 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.794 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:45:16.795 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:45:16.796 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 21, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 21, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 21, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 21, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 21, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:45:16.797 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "509", 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "507", 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:45:16.798 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:45:16.799 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:45:16.799 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:45:16.799 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:45:16.799 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:45:16.799 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:45:16.799 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:45:16.800 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:45:16.801 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:45:16.802 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.9099999666213989, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.9099999666213989, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.0999999046325684, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.0999999046325684, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.71190768480300903, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.85000002384185791, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.70999997854232788 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 6, 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_4", 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:44:40.370699+0000", 2024-08-30T23:45:16.803 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:45:16.804 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:45:16.805 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:45:16.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "621", 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:45:16.807 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:45:16.808 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:45:16.809 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.810 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.130000114440918, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 3.130000114440918, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.440000057220459, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3.440000057220459, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.71190768480300903, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.85000002384185791, 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.70999997854232788 2024-08-30T23:45:16.811 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.2807464599609375, 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:45:16.812 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 539, 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:45:16.813 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.814 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.814 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:45:16.816 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:45:16.819 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.820 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-30T23:45:16.821 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:45:16.823 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.824 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:45:16.826 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.827 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:45:16.828 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 594, 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.830 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.831 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.832 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.833 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.834 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:45:16.835 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.836 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:45:16.837 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.838 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.839 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:45:16.841 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.842 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.844 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.845 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:45:16.846 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:45:16.847 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.426727294921875, 2024-08-30T23:45:16.848 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 39, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 446, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 566, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 573, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.849 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.850 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.851 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.852 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1428551692 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/1428551692", 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/1428551692", 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/1428551692", 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/1428551692", 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.854 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7101593017578125, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.855 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 631, 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.857 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.858 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.859 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.860 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.861 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7503204345703125, 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.862 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:45:16.863 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.864 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.865 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.866 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:45:16.867 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:45:16.868 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.869 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 619, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:45:16.870 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.871 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.872 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.873 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:45:16.874 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.2454376220703125, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 595, 2024-08-30T23:45:16.876 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.877 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.880 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:45:16.881 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.882 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.882 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:45:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:16.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:16.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:16.883 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:45:16.884 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 625, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.885 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.886 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:45:16.887 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:45:16.889 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:45:16.890 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:45:16.891 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 587, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.892 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:45:16.893 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:45:16.895 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:45:16.897 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.898 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:01.788525+0000", 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.899 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:31.939564+0000", 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.900 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:09.378080+0000", 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.901 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:45:15.389191+0000", 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:45:14.089021+0000", 2024-08-30T23:45:16.902 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:43:22.725412+0000", 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-30T23:45:16.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:45:12.419468+0000", 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:58.405168+0000", 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.905 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:28.335352+0000", 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:45:16.906 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:47.644167+0000", 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.907 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:32.327364+0000", 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:45:16.908 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:44:30.306781+0000", 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [ 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.0", 2024-08-30T23:45:16.909 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: 7, 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: 11 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1", 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:45:16.910 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:45:16.911 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:45:16.912 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:45:16.913 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:16.914 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:16.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:17.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:17.013 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:17 smithi106 ceph-mon[16603]: pgmap v1416: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 308 B/s rd, 73 MiB/s wr, 1.16k op/s; 4779/117303 objects degraded (4.074%); 3.5 KiB/s, 0 objects/s recovering 2024-08-30T23:45:17.013 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:17 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:17.014 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1776532604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:17.014 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:17 smithi106 ceph-mon[16603]: osdmap e635: 12 total, 11 up, 12 in 2024-08-30T23:45:17.014 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3684272117' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:45:17.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:45:17.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:17.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:17.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:17.279 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:17.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:17 smithi136 ceph-mon[20207]: pgmap v1416: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 308 B/s rd, 73 MiB/s wr, 1.16k op/s; 4779/117303 objects degraded (4.074%); 3.5 KiB/s, 0 objects/s recovering 2024-08-30T23:45:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:17 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1776532604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:17 smithi136 ceph-mon[20207]: osdmap e635: 12 total, 11 up, 12 in 2024-08-30T23:45:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3684272117' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:45:17.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:17 smithi173 ceph-mon[20554]: pgmap v1416: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 308 B/s rd, 73 MiB/s wr, 1.16k op/s; 4779/117303 objects degraded (4.074%); 3.5 KiB/s, 0 objects/s recovering 2024-08-30T23:45:17.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:17 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1776532604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:17 smithi173 ceph-mon[20554]: osdmap e635: 12 total, 11 up, 12 in 2024-08-30T23:45:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3684272117' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:45:17.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 45482 45466 88.7936 102.812 0.00817022 0.0112553 2024-08-30T23:45:17.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:17.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:17.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:17.536 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:17.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:17.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:17.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:17.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:17.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:17.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:17.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:17.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:18.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:18.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:18.145 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:18.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:18.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:18.297 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:18.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 46998 46982 88.9739 94.75 0.00530584 0.0112319 2024-08-30T23:45:18.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:18.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:18.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:18.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:18.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:18.681 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:18.717 INFO:tasks.thrashosds.thrasher:j is {'epoch': 635, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:45:16.006965+0000', 'last_up_change': '2024-08-30T23:44:02.874029+0000', 'last_in_change': '2024-08-30T23:38:32.101509+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '509', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '507', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.0999999046325684, 'raw_score_stable': 2.0999999046325684, 'primary_affinity_weighted': 0.711907684803009, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7099999785423279}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-08-30T23:44:40.370699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '621', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.130000114440918, 'score_stable': 3.130000114440918, 'optimal_score': 1.100000023841858, 'raw_score_acting': 3.440000057220459, 'raw_score_stable': 3.440000057220459, 'primary_affinity_weighted': 0.711907684803009, 'average_primary_affinity': 0.8500000238418579, 'average_primary_affinity_weighted': 0.7099999785423279}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 0.2807464599609375, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 619, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 594, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 0, 'in': 1, 'weight': 0.426727294921875, 'primary_affinity': 1, 'last_clean_begin': 39, 'last_clean_end': 446, 'up_from': 566, 'up_thru': 573, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 1428551692}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 1428551692}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 1428551692}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 1428551692}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 1428551692}]}, 'public_addr': '172.21.15.136:6809/1428551692', 'cluster_addr': '172.21.15.136:6811/1428551692', 'heartbeat_back_addr': '172.21.15.136:6815/1428551692', 'heartbeat_front_addr': '172.21.15.136:6813/1428551692', 'state': ['exists']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 0.7101593017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 631, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 0.7503204345703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 619, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2454376220703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 595, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 625, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 587, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:01.788525+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:31.939564+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:09.378080+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:45:15.389191+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:45:14.089021+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:43:22.725412+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:45:12.419468+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:58.405168+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:28.335352+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:47.644167+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:32.327364+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:44:30.306781+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 7, 11]}, {'pgid': '1.1', 'osds': [2, 0, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:45:18.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:45:18.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:18.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:18.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:45:18.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:18.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:18.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:18.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:19.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.057 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:19.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:19.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:19.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:19.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:19.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:19 smithi136 ceph-mon[20207]: pgmap v1418: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 90 MiB/s wr, 1.43k op/s; 5182/128844 objects degraded (4.022%); 3.9 KiB/s, 0 objects/s recovering 2024-08-30T23:45:19.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:19 smithi173 ceph-mon[20554]: pgmap v1418: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 90 MiB/s wr, 1.43k op/s; 5182/128844 objects degraded (4.022%); 3.9 KiB/s, 0 objects/s recovering 2024-08-30T23:45:19.337 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:19 smithi106 ceph-mon[16603]: pgmap v1418: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 90 MiB/s wr, 1.43k op/s; 5182/128844 objects degraded (4.022%); 3.9 KiB/s, 0 objects/s recovering 2024-08-30T23:45:19.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 48152 48136 88.4781 72.125 0.00265098 0.0112887 2024-08-30T23:45:19.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:19.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:45:19.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.432 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:19.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:19.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:19.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:19.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.685 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:19.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:19.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:19.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:19.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:19.943 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:19.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:20.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:20.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:20.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:20.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:20.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:20.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:20.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 49664 49648 88.65 94.5 0.00543184 0.0112746 2024-08-30T23:45:20.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:20.458 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:20.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:20.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:20.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:20.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:20.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:20.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:20.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:20.972 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:20.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:21.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:21.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.099 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:21.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:21.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:21.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:21.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:21.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:21.187 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:21.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:21.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:21.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:21.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:21.188 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:21.190 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:21 smithi106 ceph-mon[16603]: pgmap v1419: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 5628/135471 objects degraded (4.154%) 2024-08-30T23:45:21.190 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2410993904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:21.190 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5628/135471 objects degraded (4.154%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:21.190 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:21 smithi106 ceph-mon[16603]: osdmap e636: 12 total, 11 up, 12 in 2024-08-30T23:45:21.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:21.223 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:21.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:21.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:21 smithi136 ceph-mon[20207]: pgmap v1419: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 5628/135471 objects degraded (4.154%) 2024-08-30T23:45:21.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2410993904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:21.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5628/135471 objects degraded (4.154%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:21.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:21 smithi136 ceph-mon[20207]: osdmap e636: 12 total, 11 up, 12 in 2024-08-30T23:45:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:21 smithi173 ceph-mon[20554]: pgmap v1419: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 5628/135471 objects degraded (4.154%) 2024-08-30T23:45:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2410993904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5628/135471 objects degraded (4.154%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:21 smithi173 ceph-mon[20554]: osdmap e636: 12 total, 11 up, 12 in 2024-08-30T23:45:21.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:21.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:21.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 50926 50910 88.3783 78.875 0.00248083 0.0113071 2024-08-30T23:45:21.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:21.479 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:21.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:21.607 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:21.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:21.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:21.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:21.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:21.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:21.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:21.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:21.977 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:21.989 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:21.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:22.024 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:22.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:22.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:22.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:22.238 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:22.274 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:22 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:45:22.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:22 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:45:22.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:22 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:45:22.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:22.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:22.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 51788 51772 87.4456 53.875 0.00751799 0.0114289 2024-08-30T23:45:22.465 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:22.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:22.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:22.620 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:22.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:22.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:22.796 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:22.796 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:22.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:22.877 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:22.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:22.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:23.001 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:23.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:23.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:23 smithi106 ceph-mon[16603]: pgmap v1421: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s; 6052/144777 objects degraded (4.180%) 2024-08-30T23:45:23.127 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:23.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:23.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:23.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:23.258 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:23 smithi136 ceph-mon[20207]: pgmap v1421: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s; 6052/144777 objects degraded (4.180%) 2024-08-30T23:45:23.349 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:23 smithi173 ceph-mon[20554]: pgmap v1421: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 2.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s; 6052/144777 objects degraded (4.180%) 2024-08-30T23:45:23.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:23.371 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 52905 52889 86.9815 69.8125 0.0215036 0.0114901 2024-08-30T23:45:23.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:23.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:23.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:23.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:23.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:23.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:23.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:23.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:23.814 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:45:23.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:23.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:23.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:23.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:45:23.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:24.039 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:45:24.052 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:24.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:24.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:24.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2493442358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:45:24.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2493442358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:45:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2493442358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:45:24.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 54172 54156 86.7815 79.1875 0.00546503 0.0115163 2024-08-30T23:45:24.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:24.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:24.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:24.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:24.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.685 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:24.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:24.787 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:24.815 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:24.852 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:45:24.853 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:45:24.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:24.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:24.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:25.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:25.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:25.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:25.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:25 smithi106 ceph-mon[16603]: pgmap v1422: 37 pgs: 1 active+undersized, 1 active+clean+scrubbing, 2 active+undersized+degraded, 33 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 6425/154905 objects degraded (4.148%) 2024-08-30T23:45:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:25 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2493442358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:45:25.288 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:25 smithi106 ceph-mon[16603]: osdmap e637: 12 total, 11 up, 12 in 2024-08-30T23:45:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:25 smithi136 ceph-mon[20207]: pgmap v1422: 37 pgs: 1 active+undersized, 1 active+clean+scrubbing, 2 active+undersized+degraded, 33 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 6425/154905 objects degraded (4.148%) 2024-08-30T23:45:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:25 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2493442358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:45:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:25 smithi136 ceph-mon[20207]: osdmap e637: 12 total, 11 up, 12 in 2024-08-30T23:45:25.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:25.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:25.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:25 smithi173 ceph-mon[20554]: pgmap v1422: 37 pgs: 1 active+undersized, 1 active+clean+scrubbing, 2 active+undersized+degraded, 33 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s; 6425/154905 objects degraded (4.148%) 2024-08-30T23:45:25.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:25 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:25.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2493442358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:45:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:25 smithi173 ceph-mon[20554]: osdmap e637: 12 total, 11 up, 12 in 2024-08-30T23:45:25.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:45:25.372812+0000 min lat: 0.00104751 max lat: 0.297715 avg lat: 0.0115627 2024-08-30T23:45:25.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:45:25.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 55347 55331 86.4477 73.4375 0.0138957 0.0115627 2024-08-30T23:45:25.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:25.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:25.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:25.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:25.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:25.705 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:25.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:25.835 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:25.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:25.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:25.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:26.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:26.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:26.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:26.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:26.228 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:26 smithi106 ceph-mon[16603]: from='client.27130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:45:26.228 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6425/154905 objects degraded (4.148%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:26.228 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:26 smithi106 ceph-mon[16603]: osdmap e638: 12 total, 11 up, 12 in 2024-08-30T23:45:26.266 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1423,"stamp":"2024-08-30T23:45:24.046753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3384437490,"num_objects":51635,"num_object_clones":0,"num_object_copies":154905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51635,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":51638,"num_write_kb":3304609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":351722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53565,"ondisk_log_size":53565,"up":108,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14535420,"kb_used_data":9909304,"kb_used_omap":107,"kb_used_meta":4625812,"kb_avail":1110406404,"statfs":{"total":1151940427776,"available":1137056157696,"internally_reserved":0,"allocated":10147127296,"data_stored":10134265091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110317,"internal_metadata":4736831763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":737411072,"num_objects":11252,"num_object_clones":0,"num_object_copies":33756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":720128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.031944"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":289,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.826744+0000","last_change":"2024-08-30T23:44:30.071083+0000","last_active":"2024-08-30T23:45:20.826744+0000","last_peered":"2024-08-30T23:45:20.826744+0000","last_clean":"2024-08-30T23:45:20.826744+0000","last_became_active":"2024-08-30T23:42:13.920647+0000","last_became_peered":"2024-08-30T23:42:13.920647+0000","last_unstale":"2024-08-30T23:45:20.826744+0000","last_undegraded":"2024-08-30T23:45:20.826744+0000","last_fullsized":"2024-08-30T23:45:20.826744+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:30.071017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:30.071017+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:32.648433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":252,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.829765+0000","last_change":"2024-08-30T23:44:29.831279+0000","last_active":"2024-08-30T23:45:20.829765+0000","last_peered":"2024-08-30T23:45:20.829765+0000","last_clean":"2024-08-30T23:45:20.829765+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:45:20.829765+0000","last_undegraded":"2024-08-30T23:45:20.829765+0000","last_fullsized":"2024-08-30T23:45:20.829765+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.831212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.831212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:57.676072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":294,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.814826+0000","last_change":"2024-08-30T23:44:31.878852+0000","last_active":"2024-08-30T23:45:20.814826+0000","last_peered":"2024-08-30T23:45:20.814826+0000","last_clean":"2024-08-30T23:45:20.814826+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:45:20.814826+0000","last_undegraded":"2024-08-30T23:45:20.814826+0000","last_fullsized":"2024-08-30T23:45:20.814826+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:31.878785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:31.878785+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:33.191051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":1327,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.815960+0000","last_change":"2024-08-30T23:44:29.953063+0000","last_active":"2024-08-30T23:45:20.815960+0000","last_peered":"2024-08-30T23:45:20.815960+0000","last_clean":"2024-08-30T23:45:20.815960+0000","last_became_active":"2024-08-30T23:42:40.017555+0000","last_became_peered":"2024-08-30T23:42:40.017555+0000","last_unstale":"2024-08-30T23:45:20.815960+0000","last_undegraded":"2024-08-30T23:45:20.815960+0000","last_fullsized":"2024-08-30T23:45:20.815960+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.953019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.953019+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.856722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,8],"acting":[7,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"636'3281","reported_seq":3322,"reported_epoch":636,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:45:23.474360+0000","last_change":"2024-08-30T23:44:45.378344+0000","last_active":"2024-08-30T23:45:23.474360+0000","last_peered":"2024-08-30T23:45:23.474360+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"2024-08-30T23:44:42.310484+0000","last_became_peered":"2024-08-30T23:44:42.310484+0000","last_unstale":"2024-08-30T23:45:23.474360+0000","last_undegraded":"2024-08-30T23:44:45.378214+0000","last_fullsized":"2024-08-30T23:44:42.303428+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3281,"log_dups_size":0,"ondisk_log_size":3281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:01.152599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215023616,"num_objects":3281,"num_object_clones":0,"num_object_copies":9843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3281,"num_write_kb":209984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":3281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":57,"reported_epoch":636,"state":"active+clean+scrubbing","last_fresh":"2024-08-30T23:45:20.975093+0000","last_change":"2024-08-30T23:45:20.974969+0000","last_active":"2024-08-30T23:45:20.975093+0000","last_peered":"2024-08-30T23:45:20.975093+0000","last_clean":"2024-08-30T23:45:20.975093+0000","last_became_active":"2024-08-30T23:45:08.975961+0000","last_became_peered":"2024-08-30T23:45:08.975961+0000","last_unstale":"2024-08-30T23:45:20.975093+0000","last_undegraded":"2024-08-30T23:45:20.975093+0000","last_fullsized":"2024-08-30T23:45:20.975093+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:33.928947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:33.928947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"636'3234","reported_seq":3278,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:22.564736+0000","last_change":"2024-08-30T23:44:42.930968+0000","last_active":"2024-08-30T23:45:22.564736+0000","last_peered":"2024-08-30T23:45:22.564736+0000","last_clean":"2024-08-30T23:45:22.564736+0000","last_became_active":"2024-08-30T23:44:42.930551+0000","last_became_peered":"2024-08-30T23:44:42.930551+0000","last_unstale":"2024-08-30T23:45:22.564736+0000","last_undegraded":"2024-08-30T23:45:22.564736+0000","last_fullsized":"2024-08-30T23:45:22.564736+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:58.994565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":206976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":589,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.826734+0000","last_change":"2024-08-30T23:44:19.130985+0000","last_active":"2024-08-30T23:45:20.826734+0000","last_peered":"2024-08-30T23:45:20.826734+0000","last_clean":"2024-08-30T23:45:20.826734+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:45:20.826734+0000","last_undegraded":"2024-08-30T23:45:20.826734+0000","last_fullsized":"2024-08-30T23:45:20.826734+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:19.130919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:19.130919+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:26.399208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"635'3169","reported_seq":3211,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.450187+0000","last_change":"2024-08-30T23:44:42.456106+0000","last_active":"2024-08-30T23:45:20.450187+0000","last_peered":"2024-08-30T23:45:20.450187+0000","last_clean":"2024-08-30T23:45:20.450187+0000","last_became_active":"2024-08-30T23:44:42.455726+0000","last_became_peered":"2024-08-30T23:44:42.455726+0000","last_unstale":"2024-08-30T23:45:20.450187+0000","last_undegraded":"2024-08-30T23:45:20.450187+0000","last_fullsized":"2024-08-30T23:45:20.450187+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3169,"log_dups_size":0,"ondisk_log_size":3169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:08.436990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":202816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":467,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.828625+0000","last_change":"2024-08-30T23:44:38.508452+0000","last_active":"2024-08-30T23:45:20.828625+0000","last_peered":"2024-08-30T23:45:20.828625+0000","last_clean":"2024-08-30T23:45:20.828625+0000","last_became_active":"2024-08-30T23:42:34.874330+0000","last_became_peered":"2024-08-30T23:42:34.874330+0000","last_unstale":"2024-08-30T23:45:20.828625+0000","last_undegraded":"2024-08-30T23:45:20.828625+0000","last_fullsized":"2024-08-30T23:45:20.828625+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.508393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.508393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:50.318741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"636'3145","reported_seq":3188,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.894395+0000","last_change":"2024-08-30T23:44:43.028992+0000","last_active":"2024-08-30T23:45:20.894395+0000","last_peered":"2024-08-30T23:45:20.894395+0000","last_clean":"2024-08-30T23:45:20.894395+0000","last_became_active":"2024-08-30T23:44:43.028669+0000","last_became_peered":"2024-08-30T23:44:43.028669+0000","last_unstale":"2024-08-30T23:45:20.894395+0000","last_undegraded":"2024-08-30T23:45:20.894395+0000","last_fullsized":"2024-08-30T23:45:20.894395+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3145,"log_dups_size":0,"ondisk_log_size":3145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:01.890978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206110720,"num_objects":3145,"num_object_clones":0,"num_object_copies":9435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":201280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":447,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.828594+0000","last_change":"2024-08-30T23:44:19.298260+0000","last_active":"2024-08-30T23:45:20.828594+0000","last_peered":"2024-08-30T23:45:20.828594+0000","last_clean":"2024-08-30T23:45:20.828594+0000","last_became_active":"2024-08-30T23:41:38.809561+0000","last_became_peered":"2024-08-30T23:41:38.809561+0000","last_unstale":"2024-08-30T23:45:20.828594+0000","last_undegraded":"2024-08-30T23:45:20.828594+0000","last_fullsized":"2024-08-30T23:45:20.828594+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:19.298203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:19.298203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:46.925422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"636'3336","reported_seq":3379,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:23.485431+0000","last_change":"2024-08-30T23:44:42.456851+0000","last_active":"2024-08-30T23:45:23.485431+0000","last_peered":"2024-08-30T23:45:23.485431+0000","last_clean":"2024-08-30T23:45:23.485431+0000","last_became_active":"2024-08-30T23:44:42.455669+0000","last_became_peered":"2024-08-30T23:44:42.455669+0000","last_unstale":"2024-08-30T23:45:23.485431+0000","last_undegraded":"2024-08-30T23:45:23.485431+0000","last_fullsized":"2024-08-30T23:45:23.485431+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3336,"log_dups_size":0,"ondisk_log_size":3336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:44.006738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218628096,"num_objects":3336,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3336,"num_write_kb":213504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":469,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.820437+0000","last_change":"2024-08-30T23:44:34.916024+0000","last_active":"2024-08-30T23:45:20.820437+0000","last_peered":"2024-08-30T23:45:20.820437+0000","last_clean":"2024-08-30T23:45:20.820437+0000","last_became_active":"2024-08-30T23:42:20.761486+0000","last_became_peered":"2024-08-30T23:42:20.761486+0000","last_unstale":"2024-08-30T23:45:20.820437+0000","last_undegraded":"2024-08-30T23:45:20.820437+0000","last_fullsized":"2024-08-30T23:45:20.820437+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:34.915971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:34.915971+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:40.704451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"636'3325","reported_seq":3369,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:23.488741+0000","last_change":"2024-08-30T23:44:42.310819+0000","last_active":"2024-08-30T23:45:23.488741+0000","last_peered":"2024-08-30T23:45:23.488741+0000","last_clean":"2024-08-30T23:45:23.488741+0000","last_became_active":"2024-08-30T23:44:42.310475+0000","last_became_peered":"2024-08-30T23:44:42.310475+0000","last_unstale":"2024-08-30T23:45:23.488741+0000","last_undegraded":"2024-08-30T23:45:23.488741+0000","last_fullsized":"2024-08-30T23:45:23.488741+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3325,"log_dups_size":0,"ondisk_log_size":3325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:10.721902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217907200,"num_objects":3325,"num_object_clones":0,"num_object_copies":9975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":212800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":459,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.828641+0000","last_change":"2024-08-30T23:44:50.821242+0000","last_active":"2024-08-30T23:45:20.828641+0000","last_peered":"2024-08-30T23:45:20.828641+0000","last_clean":"2024-08-30T23:45:20.828641+0000","last_became_active":"2024-08-30T23:44:50.820959+0000","last_became_peered":"2024-08-30T23:44:50.820959+0000","last_unstale":"2024-08-30T23:45:20.828641+0000","last_undegraded":"2024-08-30T23:45:20.828641+0000","last_fullsized":"2024-08-30T23:45:20.828641+0000","mapping_epoch":625,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:32.434222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:32.434222+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.246049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"636'3293","reported_seq":3337,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:22.554079+0000","last_change":"2024-08-30T23:44:43.028344+0000","last_active":"2024-08-30T23:45:22.554079+0000","last_peered":"2024-08-30T23:45:22.554079+0000","last_clean":"2024-08-30T23:45:22.554079+0000","last_became_active":"2024-08-30T23:44:43.027945+0000","last_became_peered":"2024-08-30T23:44:43.027945+0000","last_unstale":"2024-08-30T23:45:22.554079+0000","last_undegraded":"2024-08-30T23:45:22.554079+0000","last_fullsized":"2024-08-30T23:45:22.554079+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.478496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":210752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1324,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:16.025168+0000","last_change":"2024-08-30T23:44:55.636789+0000","last_active":"2024-08-30T23:45:16.025168+0000","last_peered":"2024-08-30T23:45:16.025168+0000","last_clean":"2024-08-30T23:45:16.025168+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:45:16.025168+0000","last_undegraded":"2024-08-30T23:45:16.025168+0000","last_fullsized":"2024-08-30T23:45:16.025168+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:55.636740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:55.636740+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.637189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"636'3144","reported_seq":3185,"reported_epoch":636,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:45:20.890565+0000","last_change":"2024-08-30T23:44:45.386164+0000","last_active":"2024-08-30T23:45:20.890565+0000","last_peered":"2024-08-30T23:45:20.890565+0000","last_clean":"2024-08-30T23:44:41.288579+0000","last_became_active":"2024-08-30T23:44:42.306853+0000","last_became_peered":"2024-08-30T23:44:42.306853+0000","last_unstale":"2024-08-30T23:45:20.890565+0000","last_undegraded":"2024-08-30T23:44:45.385784+0000","last_fullsized":"2024-08-30T23:44:42.304842+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.220393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206045184,"num_objects":3144,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3144,"num_write_kb":201216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":3144}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":219,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:16.025240+0000","last_change":"2024-08-30T23:44:58.589043+0000","last_active":"2024-08-30T23:45:16.025240+0000","last_peered":"2024-08-30T23:45:16.025240+0000","last_clean":"2024-08-30T23:45:16.025240+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:45:16.025240+0000","last_undegraded":"2024-08-30T23:45:16.025240+0000","last_fullsized":"2024-08-30T23:45:16.025240+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:58.588952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:58.588952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:25.850153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"636'3378","reported_seq":3422,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:23.483808+0000","last_change":"2024-08-30T23:44:42.314402+0000","last_active":"2024-08-30T23:45:23.483808+0000","last_peered":"2024-08-30T23:45:23.483808+0000","last_clean":"2024-08-30T23:45:23.483808+0000","last_became_active":"2024-08-30T23:44:42.314051+0000","last_became_peered":"2024-08-30T23:44:42.314051+0000","last_unstale":"2024-08-30T23:45:23.483808+0000","last_undegraded":"2024-08-30T23:45:23.483808+0000","last_fullsized":"2024-08-30T23:45:23.483808+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:03.425147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":216192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":305,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.817862+0000","last_change":"2024-08-30T23:44:30.127625+0000","last_active":"2024-08-30T23:45:20.817862+0000","last_peered":"2024-08-30T23:45:20.817862+0000","last_clean":"2024-08-30T23:45:20.817862+0000","last_became_active":"2024-08-30T23:43:56.833115+0000","last_became_peered":"2024-08-30T23:43:56.833115+0000","last_unstale":"2024-08-30T23:45:20.817862+0000","last_undegraded":"2024-08-30T23:45:20.817862+0000","last_fullsized":"2024-08-30T23:45:20.817862+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:30.127565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:30.127565+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.812310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"636'3177","reported_seq":3220,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:22.106011+0000","last_change":"2024-08-30T23:44:43.028476+0000","last_active":"2024-08-30T23:45:22.106011+0000","last_peered":"2024-08-30T23:45:22.106011+0000","last_clean":"2024-08-30T23:45:22.106011+0000","last_became_active":"2024-08-30T23:44:43.028019+0000","last_became_peered":"2024-08-30T23:44:43.028019+0000","last_unstale":"2024-08-30T23:45:22.106011+0000","last_undegraded":"2024-08-30T23:45:22.106011+0000","last_fullsized":"2024-08-30T23:45:22.106011+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3177,"log_dups_size":0,"ondisk_log_size":3177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.206952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208207872,"num_objects":3177,"num_object_clones":0,"num_object_copies":9531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":203328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":289,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:16.025361+0000","last_change":"2024-08-30T23:44:38.472833+0000","last_active":"2024-08-30T23:45:16.025361+0000","last_peered":"2024-08-30T23:45:16.025361+0000","last_clean":"2024-08-30T23:45:16.025361+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:45:16.025361+0000","last_undegraded":"2024-08-30T23:45:16.025361+0000","last_fullsized":"2024-08-30T23:45:16.025361+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.472767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.472767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.283833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"636'3274","reported_seq":3318,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.913503+0000","last_change":"2024-08-30T23:44:43.029448+0000","last_active":"2024-08-30T23:45:20.913503+0000","last_peered":"2024-08-30T23:45:20.913503+0000","last_clean":"2024-08-30T23:45:20.913503+0000","last_became_active":"2024-08-30T23:44:43.029122+0000","last_became_peered":"2024-08-30T23:44:43.029122+0000","last_unstale":"2024-08-30T23:45:20.913503+0000","last_undegraded":"2024-08-30T23:45:20.913503+0000","last_fullsized":"2024-08-30T23:45:20.913503+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.949159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214564864,"num_objects":3274,"num_object_clones":0,"num_object_copies":9822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3274,"num_write_kb":209536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":227,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:16.025378+0000","last_change":"2024-08-30T23:45:00.588948+0000","last_active":"2024-08-30T23:45:16.025378+0000","last_peered":"2024-08-30T23:45:16.025378+0000","last_clean":"2024-08-30T23:45:16.025378+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:45:16.025378+0000","last_undegraded":"2024-08-30T23:45:16.025378+0000","last_fullsized":"2024-08-30T23:45:16.025378+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:00.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:00.588903+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:27.889892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"636'3333","reported_seq":3375,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:22.578432+0000","last_change":"2024-08-30T23:44:42.930929+0000","last_active":"2024-08-30T23:45:22.578432+0000","last_peered":"2024-08-30T23:45:22.578432+0000","last_clean":"2024-08-30T23:45:22.578432+0000","last_became_active":"2024-08-30T23:44:42.930567+0000","last_became_peered":"2024-08-30T23:44:42.930567+0000","last_unstale":"2024-08-30T23:45:22.578432+0000","last_undegraded":"2024-08-30T23:45:22.578432+0000","last_fullsized":"2024-08-30T23:45:22.578432+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3333,"log_dups_size":0,"ondisk_log_size":3333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:53.461680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218431488,"num_objects":3333,"num_object_clones":0,"num_object_copies":9999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":213312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":478,"reported_epoch":636,"state":"active+undersized","last_fresh":"2024-08-30T23:45:20.820379+0000","last_change":"2024-08-30T23:44:03.894450+0000","last_active":"2024-08-30T23:45:20.820379+0000","last_peered":"2024-08-30T23:45:20.820379+0000","last_clean":"2024-08-30T23:44:01.872560+0000","last_became_active":"2024-08-30T23:44:03.894450+0000","last_became_peered":"2024-08-30T23:44:03.894450+0000","last_unstale":"2024-08-30T23:45:20.820379+0000","last_undegraded":"2024-08-30T23:45:20.820379+0000","last_fullsized":"2024-08-30T23:44:03.891268+0000","mapping_epoch":594,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":568,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:41:51.288446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:41:51.288446+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:24.996958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"636'3107","reported_seq":3150,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:21.480770+0000","last_change":"2024-08-30T23:44:43.028106+0000","last_active":"2024-08-30T23:45:21.480770+0000","last_peered":"2024-08-30T23:45:21.480770+0000","last_clean":"2024-08-30T23:45:21.480770+0000","last_became_active":"2024-08-30T23:44:43.027789+0000","last_became_peered":"2024-08-30T23:44:43.027789+0000","last_unstale":"2024-08-30T23:45:21.480770+0000","last_undegraded":"2024-08-30T23:45:21.480770+0000","last_fullsized":"2024-08-30T23:45:21.480770+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.782764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203620352,"num_objects":3107,"num_object_clones":0,"num_object_copies":9321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":198848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":418,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.818749+0000","last_change":"2024-08-30T23:44:37.215950+0000","last_active":"2024-08-30T23:45:20.818749+0000","last_peered":"2024-08-30T23:45:20.818749+0000","last_clean":"2024-08-30T23:45:20.818749+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:45:20.818749+0000","last_undegraded":"2024-08-30T23:45:20.818749+0000","last_fullsized":"2024-08-30T23:45:20.818749+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:37.215896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:37.215896+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:55.523982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"635'3215","reported_seq":3256,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.444777+0000","last_change":"2024-08-30T23:44:42.456265+0000","last_active":"2024-08-30T23:45:20.444777+0000","last_peered":"2024-08-30T23:45:20.444777+0000","last_clean":"2024-08-30T23:45:20.444777+0000","last_became_active":"2024-08-30T23:44:42.455981+0000","last_became_peered":"2024-08-30T23:44:42.455981+0000","last_unstale":"2024-08-30T23:45:20.444777+0000","last_undegraded":"2024-08-30T23:45:20.444777+0000","last_fullsized":"2024-08-30T23:45:20.444777+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:59.062797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1304,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.815878+0000","last_change":"2024-08-30T23:44:13.869717+0000","last_active":"2024-08-30T23:45:20.815878+0000","last_peered":"2024-08-30T23:45:20.815878+0000","last_clean":"2024-08-30T23:45:20.815878+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:45:20.815878+0000","last_undegraded":"2024-08-30T23:45:20.815878+0000","last_fullsized":"2024-08-30T23:45:20.815878+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:13.869671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:13.869671+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.045381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"636'3044","reported_seq":3088,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:21.493343+0000","last_change":"2024-08-30T23:44:42.930674+0000","last_active":"2024-08-30T23:45:21.493343+0000","last_peered":"2024-08-30T23:45:21.493343+0000","last_clean":"2024-08-30T23:45:21.493343+0000","last_became_active":"2024-08-30T23:44:42.930380+0000","last_became_peered":"2024-08-30T23:44:42.930380+0000","last_unstale":"2024-08-30T23:45:21.493343+0000","last_undegraded":"2024-08-30T23:45:21.493343+0000","last_fullsized":"2024-08-30T23:45:21.493343+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3044,"log_dups_size":0,"ondisk_log_size":3044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:09.285121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199491584,"num_objects":3044,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3044,"num_write_kb":194816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,11],"acting":[10,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":424,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-08-30T23:45:16.021231+0000","last_change":"2024-08-30T23:44:12.112535+0000","last_active":"2024-08-30T23:45:16.021231+0000","last_peered":"2024-08-30T23:45:16.021231+0000","last_clean":"2024-08-30T23:45:16.021231+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:45:16.021231+0000","last_undegraded":"2024-08-30T23:45:16.021231+0000","last_fullsized":"2024-08-30T23:45:16.021231+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:12.112429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:12.112429+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:38.348925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"636'3178","reported_seq":3222,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:23.499773+0000","last_change":"2024-08-30T23:44:42.305791+0000","last_active":"2024-08-30T23:45:23.499773+0000","last_peered":"2024-08-30T23:45:23.499773+0000","last_clean":"2024-08-30T23:45:23.499773+0000","last_became_active":"2024-08-30T23:44:42.305499+0000","last_became_peered":"2024-08-30T23:44:42.305499+0000","last_unstale":"2024-08-30T23:45:23.499773+0000","last_undegraded":"2024-08-30T23:45:23.499773+0000","last_fullsized":"2024-08-30T23:45:23.499773+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":3178,"log_dups_size":0,"ondisk_log_size":3178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:46:00.262210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208273408,"num_objects":3178,"num_object_clones":0,"num_object_copies":9534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3178,"num_write_kb":203392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,7],"acting":[3,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":207,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.826666+0000","last_change":"2024-08-30T23:44:32.963615+0000","last_active":"2024-08-30T23:45:20.826666+0000","last_peered":"2024-08-30T23:45:20.826666+0000","last_clean":"2024-08-30T23:45:20.826666+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:45:20.826666+0000","last_undegraded":"2024-08-30T23:45:20.826666+0000","last_fullsized":"2024-08-30T23:45:20.826666+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:32.963552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:32.963552+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:42.292206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1161,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-08-30T23:45:20.816020+0000","last_change":"2024-08-30T23:44:33.986377+0000","last_active":"2024-08-30T23:45:20.816020+0000","last_peered":"2024-08-30T23:45:20.816020+0000","last_clean":"2024-08-30T23:45:20.816020+0000","last_became_active":"2024-08-30T23:41:40.752006+0000","last_became_peered":"2024-08-30T23:41:40.752006+0000","last_unstale":"2024-08-30T23:45:20.816020+0000","last_undegraded":"2024-08-30T23:45:20.816020+0000","last_fullsized":"2024-08-30T23:45:20.816020+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:33.986333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:33.986333+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:45:43.206532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,9],"acting":[7,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3383820288,"num_objects":51633,"num_object_clones":0,"num_object_copies":154899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51633,"num_write_kb":3304512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9710534656,"data_stored":9710534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51633,"ondisk_log_size":51633,"up":46,"acting":46,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617202,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":351722,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":62,"acting":62,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187706,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1051004,"kb_used_data":614444,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92694148,"statfs":{"total":95995035648,"available":94918807552,"internally_reserved":0,"allocated":629190656,"data_stored":628122320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":447019460},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449346,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969980,"kb_used_data":600044,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92775172,"statfs":{"total":95995035648,"available":95001776128,"internally_reserved":0,"allocated":614445056,"data_stored":613376728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":378796480},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647710980,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434928,"kb_used_data":970592,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92310224,"statfs":{"total":95995035648,"available":94525669376,"internally_reserved":0,"allocated":993886208,"data_stored":992822362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":475462080},"hb_peers":[0,1,2,3,4,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972618,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":987832,"kb_used_data":605416,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92757320,"statfs":{"total":95995035648,"available":94983495680,"internally_reserved":0,"allocated":619945984,"data_stored":618878586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234253,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1542076,"kb_used_data":1027756,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92203076,"statfs":{"total":95995035648,"available":94415949824,"internally_reserved":0,"allocated":1052422144,"data_stored":1051353816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":526645706},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528596,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":981308,"kb_used_data":621100,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92763844,"statfs":{"total":95995035648,"available":94990176256,"internally_reserved":0,"allocated":636006400,"data_stored":634938072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":566,"seq":2430951489545,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552508,"kb_used_data":479660,"kb_used_omap":4,"kb_used_meta":72827,"kb_avail":93192644,"statfs":{"total":95995035648,"available":95429267456,"internally_reserved":0,"allocated":491171840,"data_stored":490053979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4411,"internal_metadata":74575557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953757,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1244656,"kb_used_data":790048,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":92500496,"statfs":{"total":95995035648,"available":94720507904,"internally_reserved":0,"allocated":809009152,"data_stored":807945306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149986,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2197572,"kb_used_data":1692980,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91547580,"statfs":{"total":95995035648,"available":93744721920,"internally_reserved":0,"allocated":1733611520,"data_stored":1732539640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":516684231},"hb_peers":[0,1,2,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411624,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1606276,"kb_used_data":1286324,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92138876,"statfs":{"total":95995035648,"available":94350209024,"internally_reserved":0,"allocated":1317195776,"data_stored":1316123896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830439,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1018672,"kb_used_data":804704,"kb_used_omap":87,"kb_used_meta":213864,"kb_avail":92726480,"statfs":{"total":95995035648,"available":94951915520,"internally_reserved":0,"allocated":824016896,"data_stored":822953050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":218996895},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738674,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948608,"kb_used_data":416236,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92796544,"statfs":{"total":95995035648,"available":95023661056,"internally_reserved":0,"allocated":426225664,"data_stored":425157336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":432078848,"data_stored":432078848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":826277888,"data_stored":826277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1317994496,"data_stored":1317994496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1742667776,"data_stored":1742667776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":825556992,"data_stored":825556992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":633274368,"data_stored":633274368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1064501248,"data_stored":1064501248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":619053056,"data_stored":619053056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1010434048,"data_stored":1010434048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":611647488,"data_stored":611647488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":627048448,"data_stored":627048448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:45:26.268 INFO:tasks.thrashosds.thrasher:Setting 1.9 to [11, 3, 5, 2, 1, 0] 2024-08-30T23:45:26.269 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.9', '11', '3', '5', '2', '1', '0'] 2024-08-30T23:45:26.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.9 11 3 5 2 1 0 2024-08-30T23:45:26.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:26 smithi136 ceph-mon[20207]: from='client.27130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:45:26.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6425/154905 objects degraded (4.148%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:26 smithi136 ceph-mon[20207]: osdmap e638: 12 total, 11 up, 12 in 2024-08-30T23:45:26.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:26.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:26 smithi173 ceph-mon[20554]: from='client.27130 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:45:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6425/154905 objects degraded (4.148%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:26 smithi173 ceph-mon[20554]: osdmap e638: 12 total, 11 up, 12 in 2024-08-30T23:45:26.333 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:26.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:26.359 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:26.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 56566 56550 86.1973 76.1875 0.023596 0.0115942 2024-08-30T23:45:26.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:26.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:26.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:26.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:26.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:26.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:26.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:26.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:26.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:26.961 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:26.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:27.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:27.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:27.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:27.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:27.321 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:27.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:27 smithi173 ceph-mon[20554]: pgmap v1424: 37 pgs: 1 active+undersized, 1 active+clean+scrubbing, 2 active+undersized+degraded, 33 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 6425/157020 objects degraded (4.092%) 2024-08-30T23:45:27.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:27 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:45:27.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:27 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:45:27.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:45:27.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:27.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:27 smithi106 ceph-mon[16603]: pgmap v1424: 37 pgs: 1 active+undersized, 1 active+clean+scrubbing, 2 active+undersized+degraded, 33 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 6425/157020 objects degraded (4.092%) 2024-08-30T23:45:27.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:27 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:45:27.354 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:27 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:45:27.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 57424 57408 85.4217 53.625 0.02373 0.0116982 2024-08-30T23:45:27.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:27.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:27 smithi136 ceph-mon[20207]: pgmap v1424: 37 pgs: 1 active+undersized, 1 active+clean+scrubbing, 2 active+undersized+degraded, 33 active+clean; 3.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s; 6425/157020 objects degraded (4.092%) 2024-08-30T23:45:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:27 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:45:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:27 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:45:27.570 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:27.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:27.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:27.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:27.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:27.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:27.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:27.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:27.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:28.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:28.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:28.161 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:28.161 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:28.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:28.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:28.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:28.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:28.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 58321 58305 84.7388 56.0625 0.081136 0.0117931 2024-08-30T23:45:28.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:28.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:28.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:28.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:28.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:28.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:28.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:28.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:28.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:28.986 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:28.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:29.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:45:29.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:29.112 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:29.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:29.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:29.263 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:29 smithi106 ceph-mon[16603]: pgmap v1426: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 6766/166515 objects degraded (4.063%) 2024-08-30T23:45:29.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:29 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:45:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:29 smithi173 ceph-mon[20554]: pgmap v1426: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 6766/166515 objects degraded (4.063%) 2024-08-30T23:45:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:29 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:45:29.331 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:29.331 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:29.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:29.348 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:29.358 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:29.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:29.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:29.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 59337 59321 84.256 63.5 0.0111342 0.0118624 2024-08-30T23:45:29.377 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:29.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:29.422 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:29.422 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:29.464 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:29.464 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:29.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:29.497 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:29.497 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:29.499 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:29.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:29.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:29 smithi136 ceph-mon[20207]: pgmap v1426: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s; 6766/166515 objects degraded (4.063%) 2024-08-30T23:45:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:29 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:45:29.540 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:29.541 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:29.580 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:29.580 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:29.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:29.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:29.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:29.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:29.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:29.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:29.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:29.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:30.027 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:30.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:30.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:30.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:30 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:45:30.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:30 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:45:30.155 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1097635672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:30.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:30.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:30 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:45:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:30 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:45:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1097635672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:30.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 60563 60547 84.0863 76.625 0.00380811 0.0118864 2024-08-30T23:45:30.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:30.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:30.421 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:30.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:30.459 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:30.459 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:30.459 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:30.501 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:30.501 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:30.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:30.537 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:30.537 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:30 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:45:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:30 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:45:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1097635672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:30.554 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:30.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:30.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:30.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:30.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:30.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:30.959 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:30.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:31.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:31.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:31.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:61669 6.e e638/619 6:7a0e97b5:::benchmark_data_smithi173_114000_object61668:head v 638'3925, mlcod=638'3923)", 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:31.190567+0000", 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00098347899999999999, 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001010359, 2024-08-30T23:45:31.192 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190567+0000", 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190567+0000", 2024-08-30T23:45:31.193 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190577+0000", 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3489999999999998e-06 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190611+0000", 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4227000000000003e-05 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:31.194 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190704+0000", 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3109999999999995e-05 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190719+0000", 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5310000000000001e-05 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190794+0000", 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4153999999999995e-05 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190816+0000", 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2772000000000002e-05 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:61671 6.2 e638/619 6:4ebee95d:::benchmark_data_smithi173_114000_object61670:head v 638'3926, mlcod=638'3924)", 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:31.190879+0000", 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00067234599999999999, 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00075246899999999997, 2024-08-30T23:45:31.196 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190879+0000", 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190879+0000", 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190879+0000", 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9800000000000003e-07 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190960+0000", 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0953999999999998e-05 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190986+0000", 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6191e-05 2024-08-30T23:45:31.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.190993+0000", 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7290000000000003e-06 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.191044+0000", 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0523999999999999e-05 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:45:31.199 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:31.191061+0000", 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7091e-05 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 2 2024-08-30T23:45:31.200 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:31.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:31.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:31.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:31.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 61887 61871 84.0571 82.75 0.0097112 0.0118899 2024-08-30T23:45:31.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:31.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:31.484 INFO:teuthology.orchestra.run.smithi106.stderr:set 1.9 pg_upmap_items mapping to [11->3,5->2,1->0] 2024-08-30T23:45:31.485 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:45:31.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:31.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:31.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:31.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: pgmap v1427: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s; 7071/171561 objects degraded (4.122%) 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/262480834' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]} : dispatch 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]} : dispatch 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3190640489' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7071/171561 objects degraded (4.122%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:45:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:31 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: pgmap v1427: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s; 7071/171561 objects degraded (4.122%) 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/262480834' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]} : dispatch 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]} : dispatch 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3190640489' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7071/171561 objects degraded (4.122%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:45:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:45:31.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:31 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: pgmap v1427: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s; 7071/171561 objects degraded (4.122%) 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/262480834' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]} : dispatch 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]} : dispatch 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3190640489' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7071/171561 objects degraded (4.122%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:45:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:45:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:31 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:45:31.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:31.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:32.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:45:32.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:32.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:32.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:32.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:32.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:32.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:32.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 63295 63279 84.1409 88 0.00515322 0.0118796 2024-08-30T23:45:32.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:32.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:32.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:32 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:45:32.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:32 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]}]': finished 2024-08-30T23:45:32.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:32 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:32.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:32 smithi106 ceph-mon[16603]: osdmap e639: 12 total, 11 up, 12 in 2024-08-30T23:45:32.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:32.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:32.590 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:32.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:32.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:32.748 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:32 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:45:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:32 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]}]': finished 2024-08-30T23:45:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:32 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:32 smithi136 ceph-mon[20207]: osdmap e639: 12 total, 11 up, 12 in 2024-08-30T23:45:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:32 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:45:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:32 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [11, 3, 5, 2, 1, 0]}]': finished 2024-08-30T23:45:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:32 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:32 smithi173 ceph-mon[20554]: osdmap e639: 12 total, 11 up, 12 in 2024-08-30T23:45:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:32.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:32.877 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:32.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:32.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:33.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:33.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:33.135 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:33.237 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:33.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:33.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:33.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 64794 64778 84.3396 93.6875 0.0284698 0.011852 2024-08-30T23:45:33.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:33.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:33.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:33.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:33.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:33.650 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:33.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:33.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:33 smithi136 ceph-mon[20207]: pgmap v1429: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s; 7393/178383 objects degraded (4.144%) 2024-08-30T23:45:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:33 smithi136 ceph-mon[20207]: osdmap e640: 12 total, 11 up, 12 in 2024-08-30T23:45:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:33 smithi173 ceph-mon[20554]: pgmap v1429: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s; 7393/178383 objects degraded (4.144%) 2024-08-30T23:45:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:33 smithi173 ceph-mon[20554]: osdmap e640: 12 total, 11 up, 12 in 2024-08-30T23:45:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:33 smithi106 ceph-mon[16603]: pgmap v1429: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s; 7393/178383 objects degraded (4.144%) 2024-08-30T23:45:33.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:33 smithi106 ceph-mon[16603]: osdmap e640: 12 total, 11 up, 12 in 2024-08-30T23:45:33.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:33.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:33.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:34.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:34.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:34.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:34.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:34.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:34.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:34.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 66197 66181 84.4078 87.6875 0.0108119 0.0118418 2024-08-30T23:45:34.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:34.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:34.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:34.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:34 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:45:34.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:34 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:45:34.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:34.678 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:34.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:34 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:45:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:34 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:45:34.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:34 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:45:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:34 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:45:34.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:34.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:34.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:35.032 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:35.056 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:35.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:35.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:35.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:35.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:35.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:35.372 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 67614 67598 84.4907 88.5625 0.0100346 0.0118299 2024-08-30T23:45:35.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:35.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:35.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:35.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:45:35.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:35.583 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:35.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:35.696 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:35.733 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:35 smithi106 ceph-mon[16603]: pgmap v1431: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s; 7801/188892 objects degraded (4.130%) 2024-08-30T23:45:35.733 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:35 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:35.733 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:35 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:35 smithi136 ceph-mon[20207]: pgmap v1431: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s; 7801/188892 objects degraded (4.130%) 2024-08-30T23:45:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:35 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:35 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:35.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:35 smithi173 ceph-mon[20554]: pgmap v1431: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s; 7801/188892 objects degraded (4.130%) 2024-08-30T23:45:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:35 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:35 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:35.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:35.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:35.952 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:35.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:36.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:36.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:36.156 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:36.156 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:45:36.156 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:45:36.156 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:45:36.156 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:36.165 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:36.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:36.205 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:36.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:36.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:36.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 68689 68673 84.1513 67.1875 0.00335711 0.0118471 2024-08-30T23:45:36.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:36.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:36.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:36.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:36 smithi106 ceph-mon[16603]: osdmap e641: 12 total, 11 up, 12 in 2024-08-30T23:45:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7801/188892 objects degraded (4.130%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:36 smithi106 ceph-mon[16603]: osdmap e642: 12 total, 11 up, 12 in 2024-08-30T23:45:36.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:36.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.705 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:36.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:36 smithi136 ceph-mon[20207]: osdmap e641: 12 total, 11 up, 12 in 2024-08-30T23:45:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7801/188892 objects degraded (4.130%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:36 smithi136 ceph-mon[20207]: osdmap e642: 12 total, 11 up, 12 in 2024-08-30T23:45:36.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:36.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:36 smithi173 ceph-mon[20554]: osdmap e641: 12 total, 11 up, 12 in 2024-08-30T23:45:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7801/188892 objects degraded (4.130%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:36 smithi173 ceph-mon[20554]: osdmap e642: 12 total, 11 up, 12 in 2024-08-30T23:45:36.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:36.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:36.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:45:36.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:36.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:37.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:37.086 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:37.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:37.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:37.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:37.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:37.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 15 69780 69765 83.8454 68.25 0.0227764 0.0119197 2024-08-30T23:45:37.430 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:37.430 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:37.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:37.453 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:37.453 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:37.462 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:37.477 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:37.477 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:37.506 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:37.506 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:37.512 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:37.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:45:37.540 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:37.540 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:37.541 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:37.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:37.573 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:37.574 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:37.603 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:37.609 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:37.609 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:37.645 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:37.645 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:37.679 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:37 smithi106 ceph-mon[16603]: pgmap v1433: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 7801/191469 objects degraded (4.074%) 2024-08-30T23:45:37.680 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:37 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:37.680 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:37 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:37.680 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2816783863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:37.680 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:37.680 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:37.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:37.714 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:37.715 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:37.729 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:37.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:37.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:37 smithi136 ceph-mon[20207]: pgmap v1433: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 7801/191469 objects degraded (4.074%) 2024-08-30T23:45:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:37 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:37 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2816783863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:37.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:37 smithi173 ceph-mon[20554]: pgmap v1433: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 3.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 7801/191469 objects degraded (4.074%) 2024-08-30T23:45:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:37 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:37 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2816783863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:37.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:37.857 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:37.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:37.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:37.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:38.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:38.115 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:38.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:38.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:38.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:38.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:38.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 70961 70945 83.6548 73.75 0.0276908 0.0119482 2024-08-30T23:45:38.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:38.496 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:38.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:38.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:38.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:38.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:38.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:38.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:38 smithi136 ceph-mon[20207]: osdmap e643: 12 total, 11 up, 12 in 2024-08-30T23:45:38.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:38 smithi173 ceph-mon[20554]: osdmap e643: 12 total, 11 up, 12 in 2024-08-30T23:45:38.834 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:38 smithi106 ceph-mon[16603]: osdmap e643: 12 total, 11 up, 12 in 2024-08-30T23:45:38.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:38.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:38.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:38.981 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:38.981 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:38.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:38.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:39.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:39.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:39.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:39.256 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:39.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:39.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 72217 72201 83.5593 78.5 0.0134269 0.0119619 2024-08-30T23:45:39.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:39.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:39.509 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:39.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:39.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:39.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:39.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:39 smithi106 ceph-mon[16603]: pgmap v1436: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 8251/203088 objects degraded (4.063%) 2024-08-30T23:45:39.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:39.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:45:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:39 smithi136 ceph-mon[20207]: pgmap v1436: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 8251/203088 objects degraded (4.063%) 2024-08-30T23:45:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:39 smithi173 ceph-mon[20554]: pgmap v1436: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s; 8251/203088 objects degraded (4.063%) 2024-08-30T23:45:39.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:39.866 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:39.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:39.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:39.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:40.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:40.031 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1] 2024-08-30T23:45:40.031 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:45:40.032 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-08-30T23:45:40.032 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-08-30T23:45:40.032 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service 2024-08-30T23:45:40.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl start ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5 2024-08-30T23:45:40.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:40.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:40.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:40.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:40.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 73279 73263 83.2467 66.375 0.00783777 0.0120065 2024-08-30T23:45:40.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:40.435 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:40.453 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 systemd[1]: Starting Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:45:40.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:40.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:40.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:45:40.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:40.681 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:40.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:40.733 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 podman[117781]: 2024-08-30 23:45:40.357030723 +0000 UTC m=+0.022154307 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:45:40.733 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 podman[117781]: 2024-08-30 23:45:40.479879819 +0000 UTC m=+0.145003393 container create 32a64d9bbb05bb34af900414811bc07cc6922d391440b8becb96e6d6ed6240a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-08-30T23:45:40.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:40.864 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:40.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:40.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:41.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 podman[117781]: 2024-08-30 23:45:40.732792246 +0000 UTC m=+0.397915838 container init 32a64d9bbb05bb34af900414811bc07cc6922d391440b8becb96e6d6ed6240a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 podman[117781]: 2024-08-30 23:45:40.737252472 +0000 UTC m=+0.402376047 container start 32a64d9bbb05bb34af900414811bc07cc6922d391440b8becb96e6d6ed6240a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 podman[117781]: 2024-08-30 23:45:40.857996999 +0000 UTC m=+0.523120588 container attach 32a64d9bbb05bb34af900414811bc07cc6922d391440b8becb96e6d6ed6240a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 bash[117781]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 bash[117781]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.041 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-30T23:45:41.041 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 bash[117781]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-30T23:45:41.041 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.041 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 bash[117781]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.041 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.041 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:40 smithi136 bash[117781]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-30T23:45:41.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:41.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:41.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:41.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:41.316 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:41.317 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:41.317 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:41.366 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:45:41.367 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:45:41.367 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-08-30T23:45:41.367 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2024-08-30T23:45:41.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:41.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 74474 74458 83.0937 74.6875 0.00474066 0.0120294 2024-08-30T23:45:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:41.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:41.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:41.531 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:41.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:41.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:41.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: pgmap v1437: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 8636/208710 objects degraded (4.138%) 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: osdmap e644: 12 total, 11 up, 12 in 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8636/208710 objects degraded (4.138%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4244577420' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:41.787 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate[117795]: --> ceph-volume lvm activate successful for osd ID: 5 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 bash[117781]: --> ceph-volume lvm activate successful for osd ID: 5 2024-08-30T23:45:41.788 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:41 smithi136 podman[117781]: 2024-08-30 23:45:41.409087532 +0000 UTC m=+1.074211108 container died 32a64d9bbb05bb34af900414811bc07cc6922d391440b8becb96e6d6ed6240a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-30T23:45:41.789 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:41.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: pgmap v1437: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 8636/208710 objects degraded (4.138%) 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: osdmap e644: 12 total, 11 up, 12 in 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8636/208710 objects degraded (4.138%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4244577420' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:41.815 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:41 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: pgmap v1437: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 8636/208710 objects degraded (4.138%) 2024-08-30T23:45:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: osdmap e644: 12 total, 11 up, 12 in 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8636/208710 objects degraded (4.138%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4244577420' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:41 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:45:41.845 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:45:41.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:41.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:41.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:42.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:42.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:42.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:42.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:42.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:42.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:42.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 75483 75467 82.7422 63.0625 0.00346862 0.0120802 2024-08-30T23:45:42.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:42.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:42.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:42.539 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 podman[117925]: 2024-08-30 23:45:42.280736479 +0000 UTC m=+0.864428489 container remove 32a64d9bbb05bb34af900414811bc07cc6922d391440b8becb96e6d6ed6240a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-activate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:45:42.539 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 podman[118014]: 2024-08-30 23:45:42.418378164 +0000 UTC m=+0.018831856 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:45:42.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:42.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:42.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:42.724 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:42 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:42.724 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:42 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:45:42.724 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:42 smithi106 ceph-mon[16603]: osdmap e645: 12 total, 11 up, 12 in 2024-08-30T23:45:42.725 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3484833136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:42.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:42.789 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 podman[118014]: 2024-08-30 23:45:42.540158021 +0000 UTC m=+0.140611708 container create be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-30T23:45:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:42 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:42 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:45:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:42 smithi136 ceph-mon[20207]: osdmap e645: 12 total, 11 up, 12 in 2024-08-30T23:45:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3484833136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:42.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:42 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:42 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:45:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:42 smithi173 ceph-mon[20554]: osdmap e645: 12 total, 11 up, 12 in 2024-08-30T23:45:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3484833136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:42.881 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:42.881 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:42.882 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:42.883 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:42.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:42.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:42.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:42.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:42.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:42.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:42.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:42.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:45:43.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:43.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.061 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:43.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:43.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:43.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:43.274 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:43.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:43.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 podman[118014]: 2024-08-30 23:45:42.810646295 +0000 UTC m=+0.411099991 container init be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-08-30T23:45:43.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 podman[118014]: 2024-08-30 23:45:42.815295645 +0000 UTC m=+0.415749336 container start be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-30T23:45:43.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 ceph-osd[118088]: -- 172.21.15.136:0/1805673419 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557dcd263a40 con 0x557dcd241c00 2024-08-30T23:45:43.291 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 bash[118014]: be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc 2024-08-30T23:45:43.291 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:42 smithi136 systemd[1]: Started Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-30T23:45:43.319 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:43.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 76405 76389 82.3091 57.625 0.00464992 0.0121426 2024-08-30T23:45:43.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:43.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:43.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:43.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:43.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:43.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:43.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:43.716 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:43 smithi136 ceph-mon[20207]: pgmap v1439: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 8993/216180 objects degraded (4.160%) 2024-08-30T23:45:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:43.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:43.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:43 smithi173 ceph-mon[20554]: pgmap v1439: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 8993/216180 objects degraded (4.160%) 2024-08-30T23:45:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.826 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:43.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:43 smithi106 ceph-mon[16603]: pgmap v1439: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s; 8993/216180 objects degraded (4.160%) 2024-08-30T23:45:43.865 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-30T23:45:43.865 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-30T23:45:43.865 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:config diff 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:config diff get 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:config get 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:config help [] 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:config set ... 2024-08-30T23:45:43.866 INFO:teuthology.orchestra.run.smithi136.stderr:admin_socket: invalid command 2024-08-30T23:45:43.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:43.957 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:43.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:44.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:44.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:44.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.223 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:44.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:44.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:45:44.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:44.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:44.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 77597 77581 82.1767 74.5 0.00820763 0.0121633 2024-08-30T23:45:44.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:44.479 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:44.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:44.585 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:44.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:44.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:44.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:45:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:45:44.728 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-08-30T23:45:44.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:44.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:44.863 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:44.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:44.997 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:44.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:45.039 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:44 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-30T23:45:44.724+0000 7fc7feef1640 -1 Falling back to public interface 2024-08-30T23:45:45.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:45.131 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:45.170 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:45:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:45:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:45:45.170 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:45:45.170 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:45:45.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:45.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.257 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:45.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:45.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:45:45.374451+0000 min lat: 0.00104751 max lat: 0.3611 avg lat: 0.0122047 2024-08-30T23:45:45.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:45:45.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 78632 78616 81.885 64.6875 0.00281444 0.0122047 2024-08-30T23:45:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:45.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:45.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:45.509 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:45.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:45.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:45.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:45.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.761 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:45.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:45.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:45:45.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:45 smithi173 ceph-mon[20554]: pgmap v1441: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 9321/225546 objects degraded (4.133%) 2024-08-30T23:45:45.831 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:45 smithi106 ceph-mon[16603]: pgmap v1441: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 9321/225546 objects degraded (4.133%) 2024-08-30T23:45:45.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:45.895 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:45.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:45.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:46.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:46.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:45 smithi136 ceph-mon[20207]: pgmap v1441: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 9321/225546 objects degraded (4.133%) 2024-08-30T23:45:46.055 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:46.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:46.154 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:46.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:46.282 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:46.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 79796 79780 81.7352 72.75 0.00883563 0.0122294 2024-08-30T23:45:46.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:46.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:46.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:46.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:46.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:45:46.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:46.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:46.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:46.803 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9321/225546 objects degraded (4.133%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:45:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:46 smithi173 ceph-mon[20554]: osdmap e646: 12 total, 11 up, 12 in 2024-08-30T23:45:46.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9321/225546 objects degraded (4.133%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:45:46.856 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:46 smithi106 ceph-mon[16603]: osdmap e646: 12 total, 11 up, 12 in 2024-08-30T23:45:46.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:46.928 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:46.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:47.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9321/225546 objects degraded (4.133%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:45:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:46 smithi136 ceph-mon[20207]: osdmap e646: 12 total, 11 up, 12 in 2024-08-30T23:45:47.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:47.144 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:47.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:47.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:47.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:47.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:47.373 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 80874 80858 81.5035 67.375 0.00693272 0.0122647 2024-08-30T23:45:47.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:47.449 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:47.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:47.554 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:45:47.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:47.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:47.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:47.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:47 smithi173 ceph-mon[20554]: pgmap v1442: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 9321/227547 objects degraded (4.096%) 2024-08-30T23:45:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3452863687' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:47.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:47.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:47 smithi106 ceph-mon[16603]: pgmap v1442: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 9321/227547 objects degraded (4.096%) 2024-08-30T23:45:47.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3452863687' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:47.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:47.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:47.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:48.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:47 smithi136 ceph-mon[20207]: pgmap v1442: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 9321/227547 objects degraded (4.096%) 2024-08-30T23:45:48.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3452863687' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:45:48.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:48.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:48.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:48.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.278 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:48.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 82178 82162 81.5033 81.5 0.00854105 0.0122643 2024-08-30T23:45:48.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:48.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:48.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:48.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:48.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:48.666 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:48.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:48.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:48 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3452863687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:48 smithi173 ceph-mon[20554]: osdmap e647: 12 total, 11 up, 12 in 2024-08-30T23:45:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3334289925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:48 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3452863687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:48 smithi106 ceph-mon[16603]: osdmap e647: 12 total, 11 up, 12 in 2024-08-30T23:45:48.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3334289925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:48.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:48.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:48.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:48.979 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:45:48.980 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:45:49.012 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:45:49.013 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:45:49.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:49.039 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:48 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-30T23:45:48.568+0000 7fc7feef1640 -1 osd.5 593 log_to_monitors true 2024-08-30T23:45:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:48 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3452863687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:45:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:48 smithi136 ceph-mon[20207]: osdmap e647: 12 total, 11 up, 12 in 2024-08-30T23:45:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3334289925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:45:49.044 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:45:49.044 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:45:49.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:49.076 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:45:49.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:45:49.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:45:49.114 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:45:49.115 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-30T23:45:49.146 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:45:49.146 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:45:49.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:49.173 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:45:49.173 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:45:49.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:49.199 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:45:49.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:45:49.229 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:45:49.229 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:45:49.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:45:49.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:45:49.285 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:45:49.285 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:45:49.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.312 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:49.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:49.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 83331 83315 81.3557 72.0625 0.0172671 0.0122862 2024-08-30T23:45:49.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:49.414 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:49.414 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:49.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:49.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:49.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:49.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:49.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:49.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:45:49.798 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:49.822 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.824 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:49.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:49 smithi106 ceph-mon[16603]: pgmap v1445: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 9671/236481 objects degraded (4.090%) 2024-08-30T23:45:49.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:49 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:45:49.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:49 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:45:49.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:49.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:49.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:50.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 30 23:45:49 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-30T23:45:49.610+0000 7fc7f6c9b640 -1 osd.5 648 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-30T23:45:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:49 smithi136 ceph-mon[20207]: pgmap v1445: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 9671/236481 objects degraded (4.090%) 2024-08-30T23:45:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:49 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:45:50.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:49 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:45:50.041 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:50.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:50.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:49 smithi173 ceph-mon[20554]: pgmap v1445: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 9671/236481 objects degraded (4.090%) 2024-08-30T23:45:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:49 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:45:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:49 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-30T23:45:50.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:50.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:50.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:50.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:50.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:50.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 84295 84279 81.0309 60.25 0.0211877 0.0123334 2024-08-30T23:45:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:45:50.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:45:50.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:50.464 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:50.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:50.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:50.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:50.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:50.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: from='osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: osdmap e648: 12 total, 11 up, 12 in 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:50.830 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:50.831 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:50 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:50.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.846 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:50.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: from='osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: osdmap e648: 12 total, 11 up, 12 in 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:50.885 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:50 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:50.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:50.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:50.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:51.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: from='osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: osdmap e648: 12 total, 11 up, 12 in 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:50 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:51.093 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:51.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:45:51.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:51.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:51.302 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:45:51.302 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:45:51.302 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:45:51.302 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:45:51.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:51.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:51.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 84701 84685 80.1876 25.375 0.180408 0.0123714 2024-08-30T23:45:51.386 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:51.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:51.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:51.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:51.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:51.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.743 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:51.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:51.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:51.786 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: pgmap v1447: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s; 10028/242061 objects degraded (4.143%) 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404] boot 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: osdmap e649: 12 total, 12 up, 12 in 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10028/242061 objects degraded (4.143%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: osdmap e650: 12 total, 12 up, 12 in 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:45:51.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:45:51.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:51.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.867 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:51.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: pgmap v1447: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s; 10028/242061 objects degraded (4.143%) 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404] boot 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: osdmap e649: 12 total, 12 up, 12 in 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10028/242061 objects degraded (4.143%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:45:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: osdmap e650: 12 total, 12 up, 12 in 2024-08-30T23:45:51.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:45:51.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:45:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:51.989 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:51.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: pgmap v1447: 37 pgs: 1 active+undersized, 2 active+undersized+degraded, 34 active+clean; 4.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 917 op/s; 10028/242061 objects degraded (4.143%) 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: osd.5 [v2:172.21.15.136:6808/2739515404,v1:172.21.15.136:6809/2739515404] boot 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: osdmap e649: 12 total, 12 up, 12 in 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-30T23:45:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10028/242061 objects degraded (4.143%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:45:52.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: osdmap e650: 12 total, 12 up, 12 in 2024-08-30T23:45:52.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:45:52.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:45:52.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:52.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:52.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:52.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:52.269 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:52.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:52.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 85081 85065 79.3452 23.75 0.0255793 0.0125977 2024-08-30T23:45:52.384 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:52.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:45:52.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:52.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:52.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:45:52.641 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:45:52.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:52.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:52.798 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:45:52.824 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:52.824 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:52.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:52 smithi106 ceph-mon[16603]: pgmap v1450: 37 pgs: 1 remapped+peering, 1 active+undersized, 1 active+undersized+degraded, 34 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 5064/247329 objects degraded (2.047%) 2024-08-30T23:45:52.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:52 smithi106 ceph-mon[16603]: osdmap e651: 12 total, 12 up, 12 in 2024-08-30T23:45:52.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1089611777' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:45:52.836 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:45:52.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:45:52.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:52.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:45:52.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:53.024 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:52 smithi173 ceph-mon[20554]: pgmap v1450: 37 pgs: 1 remapped+peering, 1 active+undersized, 1 active+undersized+degraded, 34 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 5064/247329 objects degraded (2.047%) 2024-08-30T23:45:53.024 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:52 smithi173 ceph-mon[20554]: osdmap e651: 12 total, 12 up, 12 in 2024-08-30T23:45:53.025 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1089611777' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:45:53.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:52 smithi136 ceph-mon[20207]: pgmap v1450: 37 pgs: 1 remapped+peering, 1 active+undersized, 1 active+undersized+degraded, 34 active+clean; 5.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.13k op/s; 5064/247329 objects degraded (2.047%) 2024-08-30T23:45:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:52 smithi136 ceph-mon[20207]: osdmap e651: 12 total, 12 up, 12 in 2024-08-30T23:45:53.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1089611777' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:45:53.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:53.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:53.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:53.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:53.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 86131 86115 79.1434 65.625 0.00444826 0.0126294 2024-08-30T23:45:53.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:53.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:53.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:53.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:45:53.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:53.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:53.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:53.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:53.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:53.850 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:53.893 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:53.921 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:53.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:54.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:54.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:54.080 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:53 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:54.080 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1089611777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:45:54.080 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:53 smithi136 ceph-mon[20207]: osdmap e652: 12 total, 12 up, 12 in 2024-08-30T23:45:54.080 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:53 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:45:54.080 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:53 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:45:54.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:53 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:54.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1089611777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:45:54.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:53 smithi173 ceph-mon[20554]: osdmap e652: 12 total, 12 up, 12 in 2024-08-30T23:45:54.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:53 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:45:54.091 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:53 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:45:54.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:53 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:45:54.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1089611777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:45:54.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:53 smithi106 ceph-mon[16603]: osdmap e652: 12 total, 12 up, 12 in 2024-08-30T23:45:54.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:53 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:45:54.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:53 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:45:54.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:54.182 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:54.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:54.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:54.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 87050 87034 78.8287 57.4375 0.012433 0.0126795 2024-08-30T23:45:54.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:54.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:54.439 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:54.439 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:54.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:54.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:54.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:54.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:54.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:54.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.821 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:54.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:54.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:54.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:54 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:45:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:54 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:45:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:54 smithi136 ceph-mon[20207]: pgmap v1453: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 35 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 5185/255858 objects degraded (2.027%); 608 KiB/s, 9 objects/s recovering 2024-08-30T23:45:55.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:55.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:55.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:54 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:45:55.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:54 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:45:55.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:54 smithi106 ceph-mon[16603]: pgmap v1453: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 35 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 5185/255858 objects degraded (2.027%); 608 KiB/s, 9 objects/s recovering 2024-08-30T23:45:55.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:45:55.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:55.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:55.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:54 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:45:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:54 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:45:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:54 smithi173 ceph-mon[20554]: pgmap v1453: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 35 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s; 5185/255858 objects degraded (2.027%); 608 KiB/s, 9 objects/s recovering 2024-08-30T23:45:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:55.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:55.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 87771 87755 78.3462 45.0625 0.0226766 0.0127321 2024-08-30T23:45:55.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:55.444 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:55.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:55.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:55.583 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:55.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:55.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:55.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:55.839 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:55.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:55.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:55.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:56.009 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:56.009 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:56.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:56.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:55 smithi106 ceph-mon[16603]: 6.9 scrub starts 2024-08-30T23:45:56.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:55 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5185/255858 objects degraded (2.027%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:45:56.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:56.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:56.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:55 smithi136 ceph-mon[20207]: 6.9 scrub starts 2024-08-30T23:45:56.232 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:55 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5185/255858 objects degraded (2.027%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:45:56.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:45:56.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:55 smithi173 ceph-mon[20554]: 6.9 scrub starts 2024-08-30T23:45:56.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:55 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5185/255858 objects degraded (2.027%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:45:56.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:45:56.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 88516 88500 77.8985 46.5625 0.0325651 0.0128302 2024-08-30T23:45:56.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:56.482 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:56.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:56.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:56.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:45:56.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:45:56.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:45:56.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:56.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:56.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:56.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:56.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:57.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:56 smithi136 ceph-mon[20207]: pgmap v1454: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 35 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 854 op/s; 5185/257136 objects degraded (2.016%); 482 KiB/s, 7 objects/s recovering 2024-08-30T23:45:57.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:56 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:45:57.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:56 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:45:57.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:56 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:45:57.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:56 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:45:57.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:56 smithi136 ceph-mon[20207]: 6.9 scrub ok 2024-08-30T23:45:57.047 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:57.097 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:56 smithi106 ceph-mon[16603]: pgmap v1454: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 35 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 854 op/s; 5185/257136 objects degraded (2.016%); 482 KiB/s, 7 objects/s recovering 2024-08-30T23:45:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:56 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:45:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:56 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:45:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:56 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:45:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:56 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:45:57.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:56 smithi106 ceph-mon[16603]: 6.9 scrub ok 2024-08-30T23:45:57.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:45:57.156 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:45:57.157 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:45:57.157 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-30T23:45:57.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd out 10 2024-08-30T23:45:57.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:57.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:57.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:56 smithi173 ceph-mon[20554]: pgmap v1454: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 35 active+clean; 5.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 854 op/s; 5185/257136 objects degraded (2.016%); 482 KiB/s, 7 objects/s recovering 2024-08-30T23:45:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:56 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:45:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:56 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:45:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:56 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:45:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:56 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:45:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:56 smithi173 ceph-mon[20554]: 6.9 scrub ok 2024-08-30T23:45:57.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:57.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 89409 89393 77.5917 55.8125 0.00407089 0.0128819 2024-08-30T23:45:57.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:57.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:45:57.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:45:57.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:45:57.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:57.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:45:57.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:57.766 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:57.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:57.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:57.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:58.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:45:58.018 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:58.018 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:58.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:45:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:57 smithi106 ceph-mon[16603]: 6.a scrub starts 2024-08-30T23:45:58.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:57 smithi106 ceph-mon[16603]: 6.f scrub starts 2024-08-30T23:45:58.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:45:58.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:45:58.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.285 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:58.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:58.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:57 smithi136 ceph-mon[20207]: 6.a scrub starts 2024-08-30T23:45:58.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:57 smithi136 ceph-mon[20207]: 6.f scrub starts 2024-08-30T23:45:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:57 smithi173 ceph-mon[20554]: 6.a scrub starts 2024-08-30T23:45:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:57 smithi173 ceph-mon[20554]: 6.f scrub starts 2024-08-30T23:45:58.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 90234 90218 77.2351 51.5625 0.00503811 0.0129419 2024-08-30T23:45:58.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:58.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:58.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:45:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:45:58.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:45:58.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:45:58.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:45:58.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:58.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:58.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:45:58.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:58.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:58.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:45:59.005 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:45:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:58 smithi136 ceph-mon[20207]: pgmap v1455: 37 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 966 op/s; 10368/263940 objects degraded (3.928%); 1.4 MiB/s, 22 objects/s recovering 2024-08-30T23:45:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:58 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:45:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:58 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:45:59.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:58 smithi136 ceph-mon[20207]: 6.f scrub ok 2024-08-30T23:45:59.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:59.064 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:58 smithi106 ceph-mon[16603]: pgmap v1455: 37 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 966 op/s; 10368/263940 objects degraded (3.928%); 1.4 MiB/s, 22 objects/s recovering 2024-08-30T23:45:59.064 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:58 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:45:59.065 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:58 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:45:59.065 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:58 smithi106 ceph-mon[16603]: 6.f scrub ok 2024-08-30T23:45:59.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:59.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:59.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:59.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:45:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:58 smithi173 ceph-mon[20554]: pgmap v1455: 37 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 966 op/s; 10368/263940 objects degraded (3.928%); 1.4 MiB/s, 22 objects/s recovering 2024-08-30T23:45:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:58 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:45:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:58 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:45:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:58 smithi173 ceph-mon[20554]: 6.f scrub ok 2024-08-30T23:45:59.335 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:45:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:45:59.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 91223 91207 77.0266 61.8125 0.00614433 0.0129772 2024-08-30T23:45:59.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:45:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:59.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:45:59.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:59.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:59.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:59.603 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:45:59.603 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:45:59.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:45:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:59.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:45:59.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:45:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:45:59.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:45:59.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:45:59.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:45:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:45:59.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:00.026 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:00.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:00.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:00.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:59 smithi106 ceph-mon[16603]: 6.a scrub ok 2024-08-30T23:46:00.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:45:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1608840328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:00.183 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:59 smithi136 ceph-mon[20207]: 6.a scrub ok 2024-08-30T23:46:00.183 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:45:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1608840328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:00.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:00.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:00.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:46:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:59 smithi173 ceph-mon[20554]: 6.a scrub ok 2024-08-30T23:46:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:45:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1608840328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:00.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:00.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:00.374 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 92166 92150 76.7854 58.9375 0.0249513 0.0130176 2024-08-30T23:46:00.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:00.494 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:46:00.494 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:46:00.498 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:00.525 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:46:00.525 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:46:00.558 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:46:00.558 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:46:00.594 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:46:00.594 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:46:00.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:00.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:00.625 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:46:00.625 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:46:00.630 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:46:00.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:46:00.665 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:46:00.700 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:46:00.701 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:46:00.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:00.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:00.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:46:00.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:00.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:00.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:00.942 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:46:00.942 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:46:00.969 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:46:00.969 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:46:00.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:01.003 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:46:01.003 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:46:01.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:01.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:00 smithi136 ceph-mon[20207]: pgmap v1456: 37 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 845 op/s; 10223/267627 objects degraded (3.820%); 2.4 MiB/s, 37 objects/s recovering 2024-08-30T23:46:01.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:00 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10223/267627 objects degraded (3.820%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:01.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:46:01.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:46:01.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:00 smithi106 ceph-mon[16603]: pgmap v1456: 37 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 845 op/s; 10223/267627 objects degraded (3.820%); 2.4 MiB/s, 37 objects/s recovering 2024-08-30T23:46:01.047 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:00 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10223/267627 objects degraded (3.820%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:01.093 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:46:01.093 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:46:01.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:01.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:01.210 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:46:01.210 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:46:01.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:46:01.210 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:46:01.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59003 5.2 5.20ed2842 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 449.06157768700001, 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96588689100000003, 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59003 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.211 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147195+0000", 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.429e-06 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147224+0000", 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9014999999999999e-05 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147237+0000", 2024-08-30T23:46:01.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.311e-05 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147244+0000", 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1160000000000004e-06 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147299+0000", 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5275000000000002e-05 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147305+0000", 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.302e-06 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113050+0000", 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96574470800000001 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113081+0000", 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0936000000000003e-05 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59011 5.2 5.db576df2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 448.75063252899997, 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65497856600000004, 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.216 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59011 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:46:01.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458149+0000", 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8779999999999993e-06 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458186+0000", 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6718000000000003e-05 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458199+0000", 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3811999999999999e-05 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458205+0000", 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6879999999999996e-06 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458260+0000", 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.516e-05 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458269+0000", 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3680000000000002e-06 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113109+0000", 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65484074999999997 2024-08-30T23:46:01.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113118+0000", 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1920000000000005e-06 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59016 5.2 5.92c60782 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 448.638802833, 2024-08-30T23:46:01.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54317872599999995, 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59016 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569970+0000", 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2449999999999999e-06 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570027+0000", 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7083999999999998e-05 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570039+0000", 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1664999999999999e-05 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570044+0000", 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6189999999999999e-06 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570095+0000", 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1085999999999998e-05 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570103+0000", 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5830000000000001e-06 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113141+0000", 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54303761500000003 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113148+0000", 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8290000000000002e-06 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59017 5.2 5.eccc78f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:01.227 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 448.48271887599998, 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38712284699999999, 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59017 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:46:01.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:46:01.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726060+0000", 2024-08-30T23:46:01.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2359999999999998e-06 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726121+0000", 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0909000000000002e-05 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726151+0000", 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0292999999999999e-05 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726165+0000", 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.428e-05 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726324+0000", 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015868300000000001 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726349+0000", 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4508e-05 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113170+0000", 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38682103699999998 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113176+0000", 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.2 426/420 [PushReplyOp(5:4213227c:::benchmark_data_smithi173_87639_object5797:head)])", 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 438.68579719000002, 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64882206499999995, 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522976+0000", 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136e-06 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522980+0000", 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3109999999999999e-06 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169121+0000", 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64614092499999998 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169129+0000", 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0099999999999995e-06 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169226+0000", 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7223000000000001e-05 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.171796+0000", 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025704600000000001 2024-08-30T23:46:01.239 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.2 432/420 [PushReplyOp(5:4f272be2:::benchmark_data_smithi173_87639_object27138:head)])", 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 426.33649611200002, 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43174907800000001, 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872277+0000", 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.674e-06 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872285+0000", 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4529999999999996e-06 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302111+0000", 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42982598799999999 2024-08-30T23:46:01.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302119+0000", 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1990000000000006e-06 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302204+0000", 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5348999999999999e-05 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.304025+0000", 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018204149999999999 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 508/491 [PushOp(5:19507caf:::benchmark_data_smithi173_87639_object19937:head, version: 403'1241, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:19507caf:::benchmark_data_smithi173_87639_object19937:head@403'1241, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:46:01.244 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 262.38402495000003, 2024-08-30T23:46:01.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:01.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32484032499999999, 2024-08-30T23:46:01.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7899999999999998e-07 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824799+0000", 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1171999999999997e-05 2024-08-30T23:46:01.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148648+0000", 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32384927299999999 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148664+0000", 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6316000000000001e-05 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148733+0000", 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8547000000000006e-05 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148741+0000", 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000007e-06 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.149587+0000", 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084573799999999996 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 508/491 [PushOp(5:0c44bbbd:::benchmark_data_smithi173_87639_object9387:head, version: 400'619, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0c44bbbd:::benchmark_data_smithi173_87639_object9387:head@400'619, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:39.158016+0000", 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 262.050755536, 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66108408900000004, 2024-08-30T23:46:01.251 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158016+0000", 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158016+0000", 2024-08-30T23:46:01.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158017+0000", 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4600000000000004e-07 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158100+0000", 2024-08-30T23:46:01.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3633000000000004e-05 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818107+0000", 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66000703199999999 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818125+0000", 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7620000000000001e-05 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818236+0000", 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011066200000000001 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818250+0000", 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3889e-05 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.819100+0000", 2024-08-30T23:46:01.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00085060699999999997 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 510/491 [PushOp(5:1f90ba55:::benchmark_data_smithi173_87639_object26324:head, version: 406'1649, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1f90ba55:::benchmark_data_smithi173_87639_object26324:head@406'1649, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:40.895932+0000", 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 260.31283974799999, 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.85628814200000003, 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.256 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.895932+0000", 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.895932+0000", 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.895933+0000", 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.896004+0000", 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1261999999999996e-05 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750707+0000", 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.85470286699999998 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750727+0000", 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0136000000000002e-05 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750825+0000", 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7887999999999995e-05 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750842+0000", 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7093000000000001e-05 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.752220+0000", 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001378167 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 511/491 [PushOp(5:192c7a90:::benchmark_data_smithi173_87639_object30798:head, version: 406'1932, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:192c7a90:::benchmark_data_smithi173_87639_object30798:head@406'1932, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:43.337860+0000", 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 257.87091142399998, 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64375286899999995, 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337860+0000", 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337860+0000", 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337862+0000", 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.48e-06 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337926+0000", 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4425999999999996e-05 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980456+0000", 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64252941900000005 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980483+0000", 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7291999999999999e-05 2024-08-30T23:46:01.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980716+0000", 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000232711 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980730+0000", 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4714e-05 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.981613+0000", 2024-08-30T23:46:01.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00088282699999999998 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 520/491 [PushOp(5:0bdf7ebf:::benchmark_data_smithi173_87639_object40617:head, version: 408'2576, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0bdf7ebf:::benchmark_data_smithi173_87639_object40617:head@408'2576, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:04.965652+0000", 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 236.243119294, 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.564411475, 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.265 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965652+0000", 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965652+0000", 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965653+0000", 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8299999999999996e-07 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965763+0000", 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109967 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529125+0000", 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.563361628 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529139+0000", 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4122e-05 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529258+0000", 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119528 2024-08-30T23:46:01.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529273+0000", 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5091e-05 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.530064+0000", 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00079045600000000004 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:109334 5.2 5:41d12184:::benchmark_data_smithi173_87639_object29094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:00.875893+0000", 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 180.33287813999999, 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64271831899999998, 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 109334 2024-08-30T23:46:01.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875893+0000", 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875893+0000", 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875895+0000", 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6619999999999999e-06 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875903+0000", 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2679999999999994e-06 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875915+0000", 2024-08-30T23:46:01.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.168e-05 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875923+0000", 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4959999999999999e-06 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875979+0000", 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6166999999999999e-05 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.876101+0000", 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012249400000000001 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.876215+0000", 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113727 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:01.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.877915+0000", 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001700072 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.878975+0000", 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010602820000000001 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:01.518471+0000", 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63949578799999995 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:01.518549+0000", 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000001e-05 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:01.518612+0000", 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2582999999999994e-05 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:110552 5.2 5:413513cd:::benchmark_data_smithi173_87639_object30312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:03.446594+0000", 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 177.76217718199999, 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35024460600000001, 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 110552 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.277 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446594+0000", 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446594+0000", 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446596+0000", 2024-08-30T23:46:01.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7969999999999999e-06 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446603+0000", 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7240000000000003e-06 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446618+0000", 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4856e-05 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446626+0000", 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0930000000000003e-06 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446692+0000", 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6222000000000004e-05 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446824+0000", 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013175099999999999 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446949+0000", 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012488899999999999 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.448987+0000", 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020381689999999998 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.449011+0000", 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4294000000000002e-05 2024-08-30T23:46:01.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.796773+0000", 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34776191000000001 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.796811+0000", 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8305e-05 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.796839+0000", 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7596000000000001e-05 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:123085 5.18 5:1e3a285a:::benchmark_data_smithi173_87639_object42845:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e568)", 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:25.945954+0000", 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.262817673, 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32379032200000002, 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.283 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 123085 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945954+0000", 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945954+0000", 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945955+0000", 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945959+0000", 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4970000000000003e-06 2024-08-30T23:46:01.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945969+0000", 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7710000000000001e-06 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945976+0000", 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0430000000000002e-06 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.946260+0000", 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028371000000000001 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.946361+0000", 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010116 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,11", 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.946500+0000", 2024-08-30T23:46:01.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138641 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.949021+0000", 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025208169999999998 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.949255+0000", 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000234315 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.954252+0000", 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049968130000000001 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268291+0000", 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.314038918 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:01.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268418+0000", 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012726400000000001 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.269744+0000", 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001326455 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:123085 5.18 e568/540 ondisk, result = 0)", 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:25.950853+0000", 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.257918668, 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31889914899999999, 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950853+0000", 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950853+0000", 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950854+0000", 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9299999999999996e-07 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950859+0000", 2024-08-30T23:46:01.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5729999999999998e-06 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268162+0000", 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31730273599999997 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268172+0000", 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0207000000000001e-05 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268253+0000", 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0877000000000005e-05 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268267+0000", 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4104000000000001e-05 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.269752+0000", 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001484759 2024-08-30T23:46:01.294 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.18 573/540 [PushReplyOp(5:1937383b:::benchmark_data_smithi173_87639_object49573:head)])", 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.87786001500001, 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34516386599999999, 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:46:01.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0699999999999996e-07 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330917+0000", 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9980000000000003e-06 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.674652+0000", 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34373422100000001 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.674662+0000", 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0212e-05 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.674726+0000", 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4642999999999998e-05 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.676075+0000", 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001348985 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:129319 5.18 5:1a97d0aa:::benchmark_data_smithi173_87639_object49079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e575)", 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:36.370255+0000", 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.83851669399999, 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30291225700000002, 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.299 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 129319 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370255+0000", 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370255+0000", 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370258+0000", 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9280000000000002e-06 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370266+0000", 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5120000000000008e-06 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370286+0000", 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0004999999999999e-05 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370296+0000", 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5580000000000006e-06 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370368+0000", 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2519000000000001e-05 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370495+0000", 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000126377 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,11", 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370601+0000", 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106165 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.372553+0000", 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019519629999999999 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:01.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.372633+0000", 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0439999999999996e-05 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.374178+0000", 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015449210000000001 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.672510+0000", 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29833189900000001 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:01.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.672578+0000", 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7566999999999999e-05 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.673167+0000", 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058940299999999995 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(5.19 e599,595 5:99c57c29:::benchmark_data_smithi173_87639_object64709:head,592'7134)", 2024-08-30T23:46:01.306 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 110.855548841, 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41795327700000001, 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:46:01.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3300000000000002e-07 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353228+0000", 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1930000000000003e-06 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771046+0000", 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41781854200000001 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771056+0000", 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771111+0000", 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5040999999999997e-05 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771118+0000", 2024-08-30T23:46:01.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771176+0000", 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8e-05 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:26169 6.b e630/619 6:d08a8ef8:::benchmark_data_smithi173_114000_object26168:head v 630'1641, mlcod=630'1639)", 2024-08-30T23:46:01.311 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:45:02.978452+0000", 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 58.230319543999997, 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17968193900000001, 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:02.978452+0000", 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:02.978452+0000", 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:02.978453+0000", 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.085e-06 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:02.978495+0000", 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2123000000000003e-05 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:03.157156+0000", 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17866037700000001 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:03.157160+0000", 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5879999999999999e-06 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:03.157205+0000", 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5009999999999998e-05 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:03.157225+0000", 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9559999999999999e-05 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:03.158095+0000", 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00087035699999999999 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:03.158134+0000", 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8838999999999999e-05 2024-08-30T23:46:01.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:68657 6.3 e642/619 6:c5908bac:::benchmark_data_smithi173_114000_object68656:head v 642'4236, mlcod=642'4233)", 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:45:36.206677+0000", 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.002094445000001, 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.210956951, 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:01.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.206677+0000", 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.206677+0000", 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.206678+0000", 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0410000000000001e-06 2024-08-30T23:46:01.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.206701+0000", 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2838000000000001e-05 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.415150+0000", 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20844907300000001 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.415158+0000", 2024-08-30T23:46:01.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3310000000000004e-06 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.415220+0000", 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1581999999999994e-05 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.415247+0000", 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6866e-05 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.417599+0000", 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023523210000000001 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:36.417634+0000", 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4898999999999999e-05 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:01.321 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:46:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:00 smithi173 ceph-mon[20554]: pgmap v1456: 37 pgs: 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 845 op/s; 10223/267627 objects degraded (3.820%); 2.4 MiB/s, 37 objects/s recovering 2024-08-30T23:46:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:00 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10223/267627 objects degraded (3.820%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:01.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:01.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 92941 92925 76.4123 48.4375 0.0854097 0.0130806 2024-08-30T23:46:01.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:01.503 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:01.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:46:01.298+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e652 definitely_dead 0 2024-08-30T23:46:01.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:01.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:01.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:01.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:01.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:01.885 INFO:teuthology.orchestra.run.smithi106.stderr:marked out osd.10. 2024-08-30T23:46:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:01.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:01.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:02.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:02.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:46:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:46:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:46:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:46:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2377208942' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:46:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:46:02.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:01 smithi136 ceph-mon[20207]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:46:02.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:02.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:02.160 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:02.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:02.257 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:02.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:46:02.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:46:02.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:46:02.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:46:02.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:46:02.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2377208942' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:46:02.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:46:02.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:01 smithi106 ceph-mon[16603]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2377208942' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-08-30T23:46:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:01 smithi173 ceph-mon[20554]: Client client.admin marked osd.10 out, while it was still marked up 2024-08-30T23:46:02.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:02.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 93467 93451 75.8469 32.875 0.0879421 0.0131154 2024-08-30T23:46:02.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:02.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:02.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:02.560 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:46:02.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:02.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:02.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:02.758 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:02.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:02.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:02.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:02.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:02.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:02 smithi136 ceph-mon[20207]: pgmap v1457: 37 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 780 op/s; 10048/271938 objects degraded (3.695%); 3.0 MiB/s, 48 objects/s recovering 2024-08-30T23:46:02.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:02 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:46:02.991 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:02 smithi136 ceph-mon[20207]: osdmap e653: 12 total, 12 up, 11 in 2024-08-30T23:46:03.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:03.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:03.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:03.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:02 smithi106 ceph-mon[16603]: pgmap v1457: 37 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 780 op/s; 10048/271938 objects degraded (3.695%); 3.0 MiB/s, 48 objects/s recovering 2024-08-30T23:46:03.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:02 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:46:03.144 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:02 smithi106 ceph-mon[16603]: osdmap e653: 12 total, 12 up, 11 in 2024-08-30T23:46:03.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:03.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:02 smithi173 ceph-mon[20554]: pgmap v1457: 37 pgs: 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 34 active+clean; 5.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 780 op/s; 10048/271938 objects degraded (3.695%); 3.0 MiB/s, 48 objects/s recovering 2024-08-30T23:46:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:02 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-08-30T23:46:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:02 smithi173 ceph-mon[20554]: osdmap e653: 12 total, 12 up, 11 in 2024-08-30T23:46:03.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:03.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 93481 93465 74.8857 0.875 0.014887 0.0131337 2024-08-30T23:46:03.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:03.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:03.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:03.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:03.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:03.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:03.760 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:03.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:03.876 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:46:03.886 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:03.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:03.914 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:46:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:46:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:46:03.915 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:46:03.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:46:03.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:04.011 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:04.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:04.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:04.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:04.159 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:03 smithi173 ceph-mon[20554]: osdmap e654: 12 total, 12 up, 11 in 2024-08-30T23:46:04.159 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:03 smithi173 ceph-mon[20554]: 6.5 scrub starts 2024-08-30T23:46:04.160 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2040172817' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:46:04.160 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:03 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:46:04.160 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2040172817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:46:04.160 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:03 smithi173 ceph-mon[20554]: osdmap e655: 12 total, 12 up, 11 in 2024-08-30T23:46:04.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:03 smithi136 ceph-mon[20207]: osdmap e654: 12 total, 12 up, 11 in 2024-08-30T23:46:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:03 smithi136 ceph-mon[20207]: 6.5 scrub starts 2024-08-30T23:46:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2040172817' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:46:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:03 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:46:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2040172817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:46:04.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:03 smithi136 ceph-mon[20207]: osdmap e655: 12 total, 12 up, 11 in 2024-08-30T23:46:04.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:03 smithi106 ceph-mon[16603]: osdmap e654: 12 total, 12 up, 11 in 2024-08-30T23:46:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:03 smithi106 ceph-mon[16603]: 6.5 scrub starts 2024-08-30T23:46:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2040172817' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:46:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:03 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:46:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2040172817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:46:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:03 smithi106 ceph-mon[16603]: osdmap e655: 12 total, 12 up, 11 in 2024-08-30T23:46:04.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 93565 93549 74.0042 5.25 0.00388927 0.0132994 2024-08-30T23:46:04.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:04.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:04.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:04.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:04.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:04.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:04.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:04.793 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:04.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:04.917 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:04.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:05.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:05.056 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:05.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:04 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:46:05.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:04 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:46:05.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:04 smithi136 ceph-mon[20207]: pgmap v1460: 37 pgs: 1 remapped+peering, 1 unknown, 4 peering, 2 active+recovering+undersized+degraded+remapped, 29 active+clean; 5.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 58 MiB/s wr, 930 op/s; 9858/279474 objects degraded (3.527%); 4.9 MiB/s, 78 objects/s recovering 2024-08-30T23:46:05.146 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:04 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:46:05.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:05.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:46:05.179 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.180 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:05.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:05.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:04 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:46:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:04 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:46:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:04 smithi173 ceph-mon[20554]: pgmap v1460: 37 pgs: 1 remapped+peering, 1 unknown, 4 peering, 2 active+recovering+undersized+degraded+remapped, 29 active+clean; 5.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 58 MiB/s wr, 930 op/s; 9858/279474 objects degraded (3.527%); 4.9 MiB/s, 78 objects/s recovering 2024-08-30T23:46:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:04 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:46:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:04 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:46:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:04 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:46:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:04 smithi106 ceph-mon[16603]: pgmap v1460: 37 pgs: 1 remapped+peering, 1 unknown, 4 peering, 2 active+recovering+undersized+degraded+remapped, 29 active+clean; 5.7 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 58 MiB/s wr, 930 op/s; 9858/279474 objects degraded (3.527%); 4.9 MiB/s, 78 objects/s recovering 2024-08-30T23:46:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:04 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:46:05.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:46:05.376155+0000 min lat: 0.00104751 max lat: 3.03568 avg lat: 0.0136227 2024-08-30T23:46:05.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:46:05.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 93800 93784 73.2627 14.6875 0.094016 0.0136227 2024-08-30T23:46:05.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:05.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:05.469 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:46:05.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:05.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:05.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:05.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:05.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:05.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:05.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:05.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:05.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:05 smithi136 ceph-mon[20207]: 6.5 scrub ok 2024-08-30T23:46:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9858/279474 objects degraded (3.527%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:06.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:05 smithi136 ceph-mon[20207]: osdmap e656: 12 total, 12 up, 11 in 2024-08-30T23:46:06.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:06.081 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:46:06.081 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:46:06.081 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:46:06.081 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:46:06.081 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:46:06.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:06.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:06.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:06.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:05 smithi173 ceph-mon[20554]: 6.5 scrub ok 2024-08-30T23:46:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9858/279474 objects degraded (3.527%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:05 smithi173 ceph-mon[20554]: osdmap e656: 12 total, 12 up, 11 in 2024-08-30T23:46:06.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:05 smithi106 ceph-mon[16603]: 6.5 scrub ok 2024-08-30T23:46:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9858/279474 objects degraded (3.527%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:05 smithi106 ceph-mon[16603]: osdmap e656: 12 total, 12 up, 11 in 2024-08-30T23:46:06.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 94531 94515 72.9223 45.6875 0.0248425 0.0137075 2024-08-30T23:46:06.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:06.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:06.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:06.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.592 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:06.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:06.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:06.795 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:46:06.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:06.848 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:06.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:06.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:06.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:07.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:06 smithi136 ceph-mon[20207]: pgmap v1462: 37 pgs: 1 remapped+peering, 1 unknown, 4 peering, 2 active+recovering+undersized+degraded+remapped, 29 active+clean; 5.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 43 MiB/s wr, 692 op/s; 9858/280095 objects degraded (3.520%); 3.8 MiB/s, 60 objects/s recovering 2024-08-30T23:46:07.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:07.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:07.117 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:46:07.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:07.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:07.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:06 smithi173 ceph-mon[20554]: pgmap v1462: 37 pgs: 1 remapped+peering, 1 unknown, 4 peering, 2 active+recovering+undersized+degraded+remapped, 29 active+clean; 5.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 43 MiB/s wr, 692 op/s; 9858/280095 objects degraded (3.520%); 3.8 MiB/s, 60 objects/s recovering 2024-08-30T23:46:07.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:07.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:07.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:06 smithi106 ceph-mon[16603]: pgmap v1462: 37 pgs: 1 remapped+peering, 1 unknown, 4 peering, 2 active+recovering+undersized+degraded+remapped, 29 active+clean; 5.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 43 MiB/s wr, 692 op/s; 9858/280095 objects degraded (3.520%); 3.8 MiB/s, 60 objects/s recovering 2024-08-30T23:46:07.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 95217 95201 72.5558 42.875 0.0273509 0.0137764 2024-08-30T23:46:07.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:07.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:07.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:07.599 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:46:07.599 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:46:07.599 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:46:07.599 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.163134 2024-08-30T23:46:07.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 7 0.16313400227029862 2024-08-30T23:46:07.606 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:07.695 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.25654.0:196473 4.0 4:0415f716:::benchmark_data_smithi173_74914_object98231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 529.669954512, 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49994438299999999, 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.696 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.25654", 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/1800184629", 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 196473 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:46:07.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024261+0000", 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6899999999999996e-07 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024266+0000", 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1779999999999996e-06 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024277+0000", 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1567e-05 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024284+0000", 2024-08-30T23:46:07.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4389999999999997e-06 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024336+0000", 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2145000000000003e-05 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024443+0000", 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107633 2024-08-30T23:46:07.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,8", 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.024556+0000", 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011263 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029319+0000", 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047626949999999999 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.029471+0000", 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015204800000000001 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524119+0000", 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49464797199999999 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524156+0000", 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7716000000000003e-05 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:37:18.524205+0000", 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8591000000000003e-05 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 451/432 [PushOp(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head, version: 400'576, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c5a46b1f:::benchmark_data_smithi173_87639_object9357:head@400'576, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 400.09468612400002, 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44893841800000001, 2024-08-30T23:46:07.703 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599529+0000", 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599530+0000", 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4339999999999999e-06 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:27.599595+0000", 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4184999999999996e-05 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046675+0000", 2024-08-30T23:46:07.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.447080796 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046706+0000", 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0505999999999999e-05 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046916+0000", 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021055600000000001 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.046944+0000", 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7492999999999999e-05 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:39:28.048467+0000", 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015234479999999999 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.3 464/432 [PushOp(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head, version: 408'2477, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cbcf07c3:::benchmark_data_smithi173_87639_object39288:head@408'2477, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.707 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 354.61693212300003, 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52881942000000004, 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077283+0000", 2024-08-30T23:46:07.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:07.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077310+0000", 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6896000000000001e-05 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.077371+0000", 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1471999999999997e-05 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604348+0000", 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52697629300000004 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604378+0000", 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0654999999999997e-05 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604592+0000", 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021338600000000001 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.604609+0000", 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7244999999999998e-05 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:13.606102+0000", 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014934729999999999 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.713 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "replica_scrub(pg: 5.1,from:0'0,to:445'5203,epoch:470/389,start:5:8a8f2f50:::benchmark_data_smithi173_87639_object56981:0,end:5:8ae6a8d1:::benchmark_data_smithi173_87639_object64309:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 340.57139480000001, 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74865697399999998, 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122820+0000", 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122821+0000", 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7599999999999997e-07 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.122828+0000", 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-06 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871217+0000", 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74838935200000001 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871235+0000", 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7708999999999999e-05 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871407+0000", 2024-08-30T23:46:07.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172153 2024-08-30T23:46:07.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871447+0000", 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9582999999999997e-05 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:27.871477+0000", 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0501000000000002e-05 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.9 e488 from shard 6)", 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 296.81686659299999, 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.49092997399999999, 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877349+0000", 2024-08-30T23:46:07.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.34e-07 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:10.877368+0000", 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8964999999999999e-05 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368058+0000", 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.490690554 2024-08-30T23:46:07.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368065+0000", 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.72e-06 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368139+0000", 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3943000000000007e-05 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368148+0000", 2024-08-30T23:46:07.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.67e-06 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:11.368278+0000", 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013068800000000001 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e502,491 5:9f5e9ce2:::benchmark_data_smithi173_87639_object5778:head,446'5327)", 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:46:07.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 274.71937998599998, 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80846543500000001, 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:46:07.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974835+0000", 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9400000000000001e-07 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:32.974839+0000", 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1080000000000001e-06 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783082+0000", 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.80824320199999999 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783093+0000", 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1122e-05 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783147+0000", 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4397000000000001e-05 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:33.783301+0000", 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015331200000000001 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 505/491 [PushOp(5:0a8a0c4f:::benchmark_data_smithi173_87639_object7264:head, version: 399'488, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0a8a0c4f:::benchmark_data_smithi173_87639_object7264:head@399'488, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 271.43320985499997, 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76577126500000003, 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261005+0000", 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261006+0000", 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9699999999999996e-07 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:36.261041+0000", 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4728999999999999e-05 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025849+0000", 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.76480819099999997 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025877+0000", 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8016000000000001e-05 2024-08-30T23:46:07.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025956+0000", 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9369000000000005e-05 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.025971+0000", 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4516e-05 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:37.026777+0000", 2024-08-30T23:46:07.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080584700000000005 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 511/491 [PushOp(5:0e383089:::benchmark_data_smithi173_87639_object11978:head, version: 401'782, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0e383089:::benchmark_data_smithi173_87639_object11978:head@401'782, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 265.37919542700001, 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88692451299999997, 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.732 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315020+0000", 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315021+0000", 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3859999999999999e-06 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:42.315100+0000", 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9388000000000006e-05 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200480+0000", 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88538003499999995 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200526+0000", 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5198999999999999e-05 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200744+0000", 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021846300000000001 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.200767+0000", 2024-08-30T23:46:07.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2996000000000001e-05 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.201944+0000", 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001177046 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 511/491 [PushOp(5:1e005da2:::benchmark_data_smithi173_87639_object35084:head, version: 407'2211, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1e005da2:::benchmark_data_smithi173_87639_object35084:head@407'2211, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:46:07.736 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 262.31758887299998, 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42476207799999999, 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376626+0000", 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376636+0000", 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0217000000000001e-05 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.376737+0000", 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100571 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800459+0000", 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.423722339 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800470+0000", 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0519e-05 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800543+0000", 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2799999999999994e-05 2024-08-30T23:46:07.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.800557+0000", 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4208e-05 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:45.801388+0000", 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083142399999999999 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 518/491 [PushOp(5:1ade8763:::benchmark_data_smithi173_87639_object73716:head, version: 436'4607, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1ade8763:::benchmark_data_smithi173_87639_object73716:head@436'4607, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 245.631190541, 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59454703399999997, 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:46:07.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063025+0000", 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8400000000000004e-07 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.063054+0000", 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9164999999999999e-05 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656490+0000", 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59343594399999999 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656518+0000", 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8041000000000001e-05 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656611+0000", 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3068999999999999e-05 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.656622+0000", 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0903999999999999e-05 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.657572+0000", 2024-08-30T23:46:07.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094932700000000003 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 519/491 [PushOp(5:092b6415:::benchmark_data_smithi173_87639_object36917:head, version: 408'2349, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:092b6415:::benchmark_data_smithi173_87639_object36917:head@408'2349, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 245.13709799200001, 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68386947499999995, 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.745 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557117+0000", 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557119+0000", 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4670000000000001e-06 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:02.557185+0000", 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6804999999999995e-05 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239646+0000", 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68246094099999999 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239664+0000", 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8122999999999999e-05 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239802+0000", 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013746299999999999 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.239814+0000", 2024-08-30T23:46:07.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2029e-05 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:03.240987+0000", 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001172647 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(5.18 e520,491 5:1cf428d8:::benchmark_data_smithi173_87639_object1164:head,440'5076)", 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:46:07.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.51292717199999, 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54704231800000003, 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181288+0000", 2024-08-30T23:46:07.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181289+0000", 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.387e-06 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.181305+0000", 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5756e-05 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728218+0000", 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54691292999999996 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728224+0000", 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3759999999999996e-06 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728270+0000", 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6016000000000001e-05 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728277+0000", 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.728330+0000", 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3631000000000002e-05 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.b 542/530 [PushOp(5:d3ac1e69:::benchmark_data_smithi173_87639_object22833:head, version: 405'1449, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:d3ac1e69:::benchmark_data_smithi173_87639_object22833:head@405'1449, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:40.832451+0000", 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 206.86176402699999, 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.75528995899999996, 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832451+0000", 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832451+0000", 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832452+0000", 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7599999999999997e-07 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.832487+0000", 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5097e-05 2024-08-30T23:46:07.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585781+0000", 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.75329446700000002 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585798+0000", 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6438000000000001e-05 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585874+0000", 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6730000000000006e-05 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.585883+0000", 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7290000000000006e-06 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.587741+0000", 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001858022 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e590,567 5:9ca44d88:::benchmark_data_smithi173_87639_object2291:head,441'5121)", 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.526287594, 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63675648500000004, 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167928+0000", 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4499999999999995e-07 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.167933+0000", 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9579999999999998e-06 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.803817+0000", 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63588312800000002 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.803827+0000", 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148e-05 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.803915+0000", 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8802999999999997e-05 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.804684+0000", 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076850300000000002 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e597,595 5:9ded4f6f:::benchmark_data_smithi173_87639_object57150:head,580'6894)", 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:07.363461+0000", 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 120.33075414, 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61280746100000005, 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363461+0000", 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363461+0000", 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363462+0000", 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2699999999999999e-07 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.363466+0000", 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0899999999999998e-06 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976023+0000", 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61255709400000002 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976033+0000", 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.005e-05 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976086+0000", 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3375999999999999e-05 2024-08-30T23:46:07.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:07.976268+0000", 2024-08-30T23:46:07.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018222400000000001 2024-08-30T23:46:07.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.766 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e598,595 5:9ccfef18:::benchmark_data_smithi173_87639_object59658:head,582'6977)", 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:08.643616+0000", 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 119.05059900800001, 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60924447699999995, 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643616+0000", 2024-08-30T23:46:07.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643616+0000", 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643617+0000", 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.643620+0000", 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5159999999999999e-06 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252614+0000", 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60899351999999995 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252638+0000", 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4556999999999999e-05 2024-08-30T23:46:07.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252689+0000", 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0689000000000002e-05 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.252861+0000", 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000171609 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e598,595 5:988025b4:::benchmark_data_smithi173_87639_object59776:head,582'6978)", 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:08.645761+0000", 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 119.048454037, 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60965304099999995, 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645761+0000", 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645761+0000", 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645762+0000", 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8700000000000005e-07 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:08.645766+0000", 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1339999999999997e-06 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255226+0000", 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.609460059 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255234+0000", 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7519999999999996e-06 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255277+0000", 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3739999999999998e-05 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:09.255414+0000", 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013666899999999999 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e599,595 5:9f3d16fd:::benchmark_data_smithi173_87639_object64699:head,592'7133)", 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.339689288, 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.420813033, 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354527+0000", 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354530+0000", 2024-08-30T23:46:07.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8600000000000003e-06 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775055+0000", 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42052456799999999 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775063+0000", 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775123+0000", 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9885e-05 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775339+0000", 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000216188 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.778 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:22484 6.3 6:c209b023:::benchmark_data_smithi173_114000_object22483:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:45:00.028514+0000", 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 67.665701306000003, 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29753540099999998, 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22484 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028514+0000", 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028514+0000", 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028514+0000", 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5799999999999999e-07 2024-08-30T23:46:07.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028572+0000", 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7707999999999997e-05 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028580+0000", 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8429999999999993e-06 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028584+0000", 2024-08-30T23:46:07.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1339999999999997e-06 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028633+0000", 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9394000000000002e-05 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028773+0000", 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139252 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6,11", 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.028928+0000", 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000154809 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.029631+0000", 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00070380599999999996 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.048926+0000", 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019295128000000002 2024-08-30T23:46:07.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.325980+0000", 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27705394 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.326023+0000", 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2209999999999997e-05 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:45:00.326049+0000", 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6619e-05 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:93583 6.8 6:14dca3a3:::benchmark_data_smithi173_114000_object93582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:46:07.785 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 3.191124995, 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34108081800000001, 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93583 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:46:07.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503091+0000", 2024-08-30T23:46:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503137+0000", 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6109000000000002e-05 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503151+0000", 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3179999999999999e-05 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503156+0000", 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5639999999999996e-06 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:46:07.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503212+0000", 2024-08-30T23:46:07.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5946999999999998e-05 2024-08-30T23:46:07.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:46:07.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503322+0000", 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011007500000000001 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,11", 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503414+0000", 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2143000000000002e-05 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.505854+0000", 2024-08-30T23:46:07.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002439887 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.506967+0000", 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001112341 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844102+0000", 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.33713554299999998 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844139+0000", 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6683000000000001e-05 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844171+0000", 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2095999999999999e-05 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:46:07.794 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:46:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:07.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:07.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:07.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:07.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:08.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:08.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:08.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:08.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:08.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:07 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:46:08.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:08.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:07 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:46:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:07 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:46:08.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 95944 95928 72.229 45.4375 0.0767111 0.0138386 2024-08-30T23:46:08.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:08.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:08.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:08.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:08.638 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:08.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:08.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:46:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.717 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:08.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:08.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:08.843 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:08.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:08.980 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:08.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:08.988 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:09.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:08 smithi173 ceph-mon[20554]: pgmap v1464: 37 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 33 MiB/s wr, 530 op/s; 9651/282747 objects degraded (3.413%); 11332/282747 objects misplaced (4.008%); 5.9 MiB/s, 95 objects/s recovering 2024-08-30T23:46:09.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:09.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:08 smithi106 ceph-mon[16603]: pgmap v1464: 37 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 33 MiB/s wr, 530 op/s; 9651/282747 objects degraded (3.413%); 11332/282747 objects misplaced (4.008%); 5.9 MiB/s, 95 objects/s recovering 2024-08-30T23:46:09.118 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:09.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:09.252 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:09.259 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:09.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:08 smithi136 ceph-mon[20207]: pgmap v1464: 37 pgs: 1 active+recovering+undersized+remapped, 1 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 33 MiB/s wr, 530 op/s; 9651/282747 objects degraded (3.413%); 11332/282747 objects misplaced (4.008%); 5.9 MiB/s, 95 objects/s recovering 2024-08-30T23:46:09.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:09.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 96525 96509 71.8014 36.3125 0.00463363 0.01392 2024-08-30T23:46:09.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:09.386 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:46:09.386 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:46:09.425 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:46:09.425 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:46:09.470 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:46:09.470 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:46:09.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:09.505 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:46:09.505 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:46:09.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:09.543 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:46:09.543 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:46:09.581 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:46:09.581 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:46:09.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:09.635 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:09.670 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:46:09.671 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:46:09.698 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:46:09.699 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:46:09.719 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:46:09.719 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:46:09.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:09.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:46:09.750 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:46:09.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:09.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:09.787 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:46:09.788 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:46:09.821 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:46:09.822 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:46:09.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:09.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:09.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:10.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:10.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:10.046 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:10.046 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:10.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:10.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:10.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/206686619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:10.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/206686619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/206686619' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:10.375 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 97281 97265 71.5125 47.25 0.019841 0.0139779 2024-08-30T23:46:10.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:10.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:10.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:10.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:10.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:46:10.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:10.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:10.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:10.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:10.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:46:10.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:10.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:10.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:11.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:11.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:10 smithi173 ceph-mon[20554]: pgmap v1465: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28 MiB/s wr, 447 op/s; 9458/285726 objects degraded (3.310%); 17006/285726 objects misplaced (5.952%); 7.9 MiB/s, 126 objects/s recovering 2024-08-30T23:46:11.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9458/285726 objects degraded (3.310%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:11.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:11.177 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:11.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:10 smithi136 ceph-mon[20207]: pgmap v1465: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28 MiB/s wr, 447 op/s; 9458/285726 objects degraded (3.310%); 17006/285726 objects misplaced (5.952%); 7.9 MiB/s, 126 objects/s recovering 2024-08-30T23:46:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9458/285726 objects degraded (3.310%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:11.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:11.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:10 smithi106 ceph-mon[16603]: pgmap v1465: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28 MiB/s wr, 447 op/s; 9458/285726 objects degraded (3.310%); 17006/285726 objects misplaced (5.952%); 7.9 MiB/s, 126 objects/s recovering 2024-08-30T23:46:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9458/285726 objects degraded (3.310%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:11.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 97853 97837 71.0966 35.75 0.0125083 0.0140577 2024-08-30T23:46:11.424 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:11.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:11.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:11.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:11.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:11.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:11.708 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:11.708 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:11.806 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:11.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:11.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:11.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:11.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:12.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:12.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:12.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:12.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1497224930' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.16313400227029862} : dispatch 2024-08-30T23:46:12.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1497224930' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.16313400227029862} : dispatch 2024-08-30T23:46:12.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:12.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1497224930' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.16313400227029862} : dispatch 2024-08-30T23:46:12.367 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:46:12.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 98304 98288 70.6034 28.1875 0.123352 0.0141144 2024-08-30T23:46:12.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:12.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:12.548 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.7 primary-affinity to 0.163134 (8106912) 2024-08-30T23:46:12.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:12.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:12.646 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:12.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:12.724 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:12.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:12.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:12.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:12.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:12.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:12 smithi173 ceph-mon[20554]: pgmap v1466: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 25 MiB/s wr, 392 op/s; 9275/288909 objects degraded (3.210%); 16821/288909 objects misplaced (5.822%); 8.0 MiB/s, 128 objects/s recovering 2024-08-30T23:46:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1497224930' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.16313400227029862}]': finished 2024-08-30T23:46:13.059 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:12 smithi173 ceph-mon[20554]: osdmap e657: 12 total, 12 up, 11 in 2024-08-30T23:46:13.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:13.110 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:13.181 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:13.181 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:13.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:13.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:12 smithi136 ceph-mon[20207]: pgmap v1466: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 25 MiB/s wr, 392 op/s; 9275/288909 objects degraded (3.210%); 16821/288909 objects misplaced (5.822%); 8.0 MiB/s, 128 objects/s recovering 2024-08-30T23:46:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1497224930' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.16313400227029862}]': finished 2024-08-30T23:46:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:12 smithi136 ceph-mon[20207]: osdmap e657: 12 total, 12 up, 11 in 2024-08-30T23:46:13.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:12 smithi106 ceph-mon[16603]: pgmap v1466: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.9 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 25 MiB/s wr, 392 op/s; 9275/288909 objects degraded (3.210%); 16821/288909 objects misplaced (5.822%); 8.0 MiB/s, 128 objects/s recovering 2024-08-30T23:46:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1497224930' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.16313400227029862}]': finished 2024-08-30T23:46:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:12 smithi106 ceph-mon[16603]: osdmap e657: 12 total, 12 up, 11 in 2024-08-30T23:46:13.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:13.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 98599 98583 70.0106 18.4375 0.0282428 0.0142724 2024-08-30T23:46:13.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:13.487 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:13.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:13.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:13.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:13.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:13.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:46:13.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:13.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:13.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:13.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:14.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:14.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:13 smithi173 ceph-mon[20554]: osdmap e658: 12 total, 12 up, 11 in 2024-08-30T23:46:14.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:14.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:14.172 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:14.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:14.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:13 smithi136 ceph-mon[20207]: osdmap e658: 12 total, 12 up, 11 in 2024-08-30T23:46:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:13 smithi106 ceph-mon[16603]: osdmap e658: 12 total, 12 up, 11 in 2024-08-30T23:46:14.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 99271 99255 69.6958 42 0.0140532 0.0143422 2024-08-30T23:46:14.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:14.411 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:14.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:14.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.543 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:14.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:14.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:14.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:14.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:14.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:14.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:14.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:15.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:15.050 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:15.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:15.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:14 smithi106 ceph-mon[16603]: pgmap v1469: 37 pgs: 2 peering, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 262 B/s rd, 32 MiB/s wr, 515 op/s; 9275/292131 objects degraded (3.175%); 16512/292131 objects misplaced (5.652%); 11 MiB/s, 171 objects/s recovering 2024-08-30T23:46:15.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:15 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:46:15.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:15 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:46:15.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:15.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:14 smithi136 ceph-mon[20207]: pgmap v1469: 37 pgs: 2 peering, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 262 B/s rd, 32 MiB/s wr, 515 op/s; 9275/292131 objects degraded (3.175%); 16512/292131 objects misplaced (5.652%); 11 MiB/s, 171 objects/s recovering 2024-08-30T23:46:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:14 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:46:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:14 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:46:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.315 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:15.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:14 smithi173 ceph-mon[20554]: pgmap v1469: 37 pgs: 2 peering, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 30 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 262 B/s rd, 32 MiB/s wr, 515 op/s; 9275/292131 objects degraded (3.175%); 16512/292131 objects misplaced (5.652%); 11 MiB/s, 171 objects/s recovering 2024-08-30T23:46:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:14 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:46:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:14 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:46:15.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 8 99795 99787 69.2909 33.25 0.0191964 0.0144268 2024-08-30T23:46:15.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:15.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:15.540 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:15.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:15.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:15.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:15.791 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:15.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:15.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:15.939 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:15.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:16.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:16.064 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:16.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:16.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:16.191 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:16.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9275/292131 objects degraded (3.175%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:16.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:15 smithi136 ceph-mon[20207]: 6.3 scrub starts 2024-08-30T23:46:16.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:16.317 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:16.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9275/292131 objects degraded (3.175%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:15 smithi173 ceph-mon[20554]: 6.3 scrub starts 2024-08-30T23:46:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9275/292131 objects degraded (3.175%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:15 smithi106 ceph-mon[16603]: 6.3 scrub starts 2024-08-30T23:46:16.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.7094 2024-08-30T23:46:16.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 99795 2024-08-30T23:46:16.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:46:16.376 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 68.7601 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 25.7269 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 128.375 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0.875 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1100 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 411.63 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2054 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 14 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0144417 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0417012 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 3.03568 2024-08-30T23:46:16.377 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00104751 2024-08-30T23:46:16.403 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:46:16.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:16.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:16.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:16.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:16.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.714 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:16.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:16.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:16.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:16.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:16.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:16.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:17.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:17.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:17.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:17.224 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.225 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:17.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:16 smithi136 ceph-mon[20207]: pgmap v1470: 37 pgs: 2 peering, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 255 B/s rd, 33 MiB/s wr, 530 op/s; 9090/295452 objects degraded (3.077%); 16512/295452 objects misplaced (5.589%); 9.1 MiB/s, 145 objects/s recovering 2024-08-30T23:46:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:16 smithi136 ceph-mon[20207]: osdmap e659: 12 total, 12 up, 11 in 2024-08-30T23:46:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:16 smithi136 ceph-mon[20207]: 6.3 scrub ok 2024-08-30T23:46:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:16 smithi173 ceph-mon[20554]: pgmap v1470: 37 pgs: 2 peering, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 255 B/s rd, 33 MiB/s wr, 530 op/s; 9090/295452 objects degraded (3.077%); 16512/295452 objects misplaced (5.589%); 9.1 MiB/s, 145 objects/s recovering 2024-08-30T23:46:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:17 smithi173 ceph-mon[20554]: osdmap e659: 12 total, 12 up, 11 in 2024-08-30T23:46:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:17 smithi173 ceph-mon[20554]: 6.3 scrub ok 2024-08-30T23:46:17.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:17.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:16 smithi106 ceph-mon[16603]: pgmap v1470: 37 pgs: 2 peering, 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 30 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 255 B/s rd, 33 MiB/s wr, 530 op/s; 9090/295452 objects degraded (3.077%); 16512/295452 objects misplaced (5.589%); 9.1 MiB/s, 145 objects/s recovering 2024-08-30T23:46:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:16 smithi106 ceph-mon[16603]: osdmap e659: 12 total, 12 up, 11 in 2024-08-30T23:46:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:16 smithi106 ceph-mon[16603]: 6.3 scrub ok 2024-08-30T23:46:17.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:17.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:17.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:17.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:17.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:17.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:17.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:17.852 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:17.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:17.990 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:17.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:18.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:18.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:18.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:18.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:18.275 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:18 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:18.276 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:18 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:18.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:17 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:17 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:18.352 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:18.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:17 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:18.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:17 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:18.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:18.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:18.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:18.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:18.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:18.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:18.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:18.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:18.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:18.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:18.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:19.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:18 smithi106 ceph-mon[16603]: pgmap v1472: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 511 B/s rd, 32 MiB/s wr, 553 op/s; 8841/297609 objects degraded (2.971%); 16107/297609 objects misplaced (5.412%); 12 MiB/s, 191 objects/s recovering 2024-08-30T23:46:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:18 smithi106 ceph-mon[16603]: osdmap e660: 12 total, 12 up, 11 in 2024-08-30T23:46:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:18 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:18 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:19.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:19.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:19.229 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:19.229 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:19.258 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:19.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:19.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:19 smithi136 ceph-mon[20207]: pgmap v1472: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 511 B/s rd, 32 MiB/s wr, 553 op/s; 8841/297609 objects degraded (2.971%); 16107/297609 objects misplaced (5.412%); 12 MiB/s, 191 objects/s recovering 2024-08-30T23:46:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:19 smithi136 ceph-mon[20207]: osdmap e660: 12 total, 12 up, 11 in 2024-08-30T23:46:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:18 smithi173 ceph-mon[20554]: pgmap v1472: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 511 B/s rd, 32 MiB/s wr, 553 op/s; 8841/297609 objects degraded (2.971%); 16107/297609 objects misplaced (5.412%); 12 MiB/s, 191 objects/s recovering 2024-08-30T23:46:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:18 smithi173 ceph-mon[20554]: osdmap e660: 12 total, 12 up, 11 in 2024-08-30T23:46:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:19 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:19 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:19.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:19.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:19.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:19.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:19.577 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:19.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:19.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.680 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:19.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:19.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:19.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:19.957 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:19.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:20.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:20.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:20.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:20.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:20.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:20 smithi136 ceph-mon[20207]: osdmap e661: 12 total, 12 up, 11 in 2024-08-30T23:46:20.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:20.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:20 smithi173 ceph-mon[20554]: osdmap e661: 12 total, 12 up, 11 in 2024-08-30T23:46:20.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:20 smithi106 ceph-mon[16603]: osdmap e661: 12 total, 12 up, 11 in 2024-08-30T23:46:20.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:20.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:20.566 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:20.597 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:20.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:20.721 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:20.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:20.833 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:46:20.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:20.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:20.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:20.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:21.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:21.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:21 smithi106 ceph-mon[16603]: pgmap v1475: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 25 MiB/s wr, 525 op/s; 8628/297192 objects degraded (2.903%); 15908/297192 objects misplaced (5.353%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:46:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2355631582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:46:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8628/297192 objects degraded (2.903%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:21 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2355631582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:46:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:21 smithi106 ceph-mon[16603]: osdmap e662: 12 total, 12 up, 11 in 2024-08-30T23:46:21.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:21.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:21.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:21 smithi136 ceph-mon[20207]: pgmap v1475: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 25 MiB/s wr, 525 op/s; 8628/297192 objects degraded (2.903%); 15908/297192 objects misplaced (5.353%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:46:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2355631582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:46:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8628/297192 objects degraded (2.903%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:21 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2355631582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:46:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:21 smithi136 ceph-mon[20207]: osdmap e662: 12 total, 12 up, 11 in 2024-08-30T23:46:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:21 smithi173 ceph-mon[20554]: pgmap v1475: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 25 MiB/s wr, 525 op/s; 8628/297192 objects degraded (2.903%); 15908/297192 objects misplaced (5.353%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:46:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2355631582' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:46:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8628/297192 objects degraded (2.903%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:21 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2355631582' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:46:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:21 smithi173 ceph-mon[20554]: osdmap e662: 12 total, 12 up, 11 in 2024-08-30T23:46:21.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:21.355 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:21.457 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:21.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:46:21.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:21.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:21.618 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:21.720 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:21.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:21.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:21.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:21.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:21.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:22.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:22.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:22.155 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:22.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:22.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:22.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:22.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:22.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:22.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:22.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:22.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.691 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:22.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:22.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:22.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:22.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:22.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:23.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:23.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:23.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:23.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:23.272 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:23 smithi106 ceph-mon[16603]: pgmap v1477: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 9.6 MiB/s wr, 447 op/s; 8429/294417 objects degraded (2.863%); 15674/294417 objects misplaced (5.324%); 12 MiB/s, 188 objects/s recovering 2024-08-30T23:46:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:23 smithi136 ceph-mon[20207]: pgmap v1477: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 9.6 MiB/s wr, 447 op/s; 8429/294417 objects degraded (2.863%); 15674/294417 objects misplaced (5.324%); 12 MiB/s, 188 objects/s recovering 2024-08-30T23:46:23.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:23.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.323 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:23.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:23 smithi173 ceph-mon[20554]: pgmap v1477: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 6.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 9.6 MiB/s wr, 447 op/s; 8429/294417 objects degraded (2.863%); 15674/294417 objects misplaced (5.324%); 12 MiB/s, 188 objects/s recovering 2024-08-30T23:46:23.356 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:23.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:23.458 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:23.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:23.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:23.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:23.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:46:23.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:23.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:23.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:23.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:23.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:46:23.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:46:23.902 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-30T23:46:23.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-08-30T23:46:23.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:23.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:23.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:24.050 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:24.094 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.134 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:24.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:24.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:24.264 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:24.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:24.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:24.493 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:24.521 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:24.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:24.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:24.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:24.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:24.857 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:24.857 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:24.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:24.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:24.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:24.971 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:25.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:25.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:25.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:25.073 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:46:25.073 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:46:25.106 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:46:25.107 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:46:25.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:25.144 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:46:25.145 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:46:25.145 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:25.145 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:25.145 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:25 smithi106 ceph-mon[16603]: pgmap v1478: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 4.0 MiB/s wr, 512 op/s; 8429/290703 objects degraded (2.900%); 15334/290703 objects misplaced (5.275%); 12 MiB/s, 197 objects/s recovering 2024-08-30T23:46:25.145 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:25 smithi106 ceph-mon[16603]: osdmap e663: 12 total, 12 up, 11 in 2024-08-30T23:46:25.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:25.180 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:46:25.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:46:25.210 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:46:25.211 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:46:25.244 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:46:25.244 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:46:25.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:25.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:46:25.274 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:46:25.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:25 smithi136 ceph-mon[20207]: pgmap v1478: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 4.0 MiB/s wr, 512 op/s; 8429/290703 objects degraded (2.900%); 15334/290703 objects misplaced (5.275%); 12 MiB/s, 197 objects/s recovering 2024-08-30T23:46:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:25 smithi136 ceph-mon[20207]: osdmap e663: 12 total, 12 up, 11 in 2024-08-30T23:46:25.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:25.298 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:46:25.298 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:46:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:25 smithi173 ceph-mon[20554]: pgmap v1478: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 4.0 MiB/s wr, 512 op/s; 8429/290703 objects degraded (2.900%); 15334/290703 objects misplaced (5.275%); 12 MiB/s, 197 objects/s recovering 2024-08-30T23:46:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:25 smithi173 ceph-mon[20554]: osdmap e663: 12 total, 12 up, 11 in 2024-08-30T23:46:25.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:46:25.327 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:46:25.327 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:46:25.362 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:46:25.362 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:46:25.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:25.399 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:46:25.399 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:46:25.438 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:46:25.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:46:25.443 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:25.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:25.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:25.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:46:25.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:25.722 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:25.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:25.854 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:25.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:25.962 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:25.991 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:25.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:26.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:26.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:26.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:26.253 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:26.281 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1623458949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:26.282 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8429/290703 objects degraded (2.900%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:26.282 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:26 smithi173 ceph-mon[20554]: osdmap e664: 12 total, 12 up, 11 in 2024-08-30T23:46:26.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1623458949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:26.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8429/290703 objects degraded (2.900%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:26.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:26 smithi136 ceph-mon[20207]: osdmap e664: 12 total, 12 up, 11 in 2024-08-30T23:46:26.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1623458949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8429/290703 objects degraded (2.900%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:26 smithi106 ceph-mon[16603]: osdmap e664: 12 total, 12 up, 11 in 2024-08-30T23:46:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:26.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:26.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:26.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:26.579 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:46:26.579 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:46:26.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:46:26.579 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86189 5.17 5.7a10c297 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.603367741, 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83557530000000002, 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86189 2024-08-30T23:46:26.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974569+0000", 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8599999999999998e-07 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974571+0000", 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6599999999999999e-06 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974579+0000", 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4259999999999997e-06 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974584+0000", 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6740000000000002e-06 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974844+0000", 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025967600000000003 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974852+0000", 2024-08-30T23:46:26.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4200000000000001e-06 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810133+0000", 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83528164100000002 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810143+0000", 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0117e-05 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.584 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86191 5.17 5.b68bebd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.60334204600002, 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83558337400000005, 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.585 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86191 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4799999999999999e-07 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974596+0000", 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9589999999999998e-06 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974602+0000", 2024-08-30T23:46:26.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974607+0000", 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.977e-06 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974875+0000", 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026775900000000001 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974880+0000", 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2220000000000001e-06 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810168+0000", 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83528822599999997 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810177+0000", 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1549999999999996e-06 2024-08-30T23:46:26.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86212 5.17 5.7a86b997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.60332308599999, 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83559074600000005, 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.590 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86212 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:46:26.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0499999999999999e-07 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974615+0000", 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6929999999999999e-06 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974620+0000", 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6359999999999999e-06 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974625+0000", 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4349999999999999e-06 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974930+0000", 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030525099999999998 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974938+0000", 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3980000000000002e-06 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810196+0000", 2024-08-30T23:46:26.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83525841000000001 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810203+0000", 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86322 5.17 5.11e87097 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.60321273699998, 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83550676000000001, 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.596 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86322 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974724+0000", 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974728+0000", 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1509999999999997e-06 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974739+0000", 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0575e-05 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974746+0000", 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8929999999999999e-06 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974986+0000", 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000240607 2024-08-30T23:46:26.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974992+0000", 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3430000000000003e-06 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810222+0000", 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83523082599999998 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810230+0000", 2024-08-30T23:46:26.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3459999999999996e-06 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86384 5.17 5.e2d2ce77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.60310794600002, 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83542769299999997, 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.601 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86384 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1300000000000001e-07 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974845+0000", 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6341000000000002e-05 2024-08-30T23:46:26.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974850+0000", 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6359999999999999e-06 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974855+0000", 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7530000000000001e-06 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975038+0000", 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018276300000000001 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975044+0000", 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9830000000000001e-06 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810248+0000", 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83520457400000003 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810256+0000", 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2300000000000002e-06 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86434 5.17 5.dc1639f7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.60306497099998, 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.835407809, 2024-08-30T23:46:26.606 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86434 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6899999999999998e-07 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974874+0000", 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3309999999999998e-06 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974879+0000", 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.699e-06 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974883+0000", 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2600000000000001e-06 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975066+0000", 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018369799999999999 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975071+0000", 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8930000000000004e-06 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810272+0000", 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83520088199999998 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810279+0000", 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5769999999999999e-06 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86189 5.17 5.7a10c297 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59728751799997, 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82969184399999996, 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.611 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86189 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980649+0000", 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2300000000000002e-07 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980651+0000", 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2024-08-30T23:46:26.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980658+0000", 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7800000000000003e-06 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980662+0000", 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6399999999999996e-06 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980723+0000", 2024-08-30T23:46:26.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0745999999999997e-05 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980732+0000", 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9379999999999998e-06 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810304+0000", 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82957220200000004 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810340+0000", 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5905000000000003e-05 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86191 5.17 5.b68bebd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.597251374, 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968160300000005, 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.616 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86191 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980685+0000", 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7899999999999999e-07 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980687+0000", 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1270000000000001e-06 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980694+0000", 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.703e-06 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980699+0000", 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4229999999999996e-06 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980764+0000", 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5363000000000004e-05 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980771+0000", 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1990000000000003e-06 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810359+0000", 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82958857900000005 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810366+0000", 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8299999999999998e-06 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.620 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86212 5.17 5.7a86b997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59722346699999, 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967699800000005, 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.621 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86212 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980713+0000", 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5699999999999998e-07 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980715+0000", 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1629999999999999e-06 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980720+0000", 2024-08-30T23:46:26.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4920000000000001e-06 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980725+0000", 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.493e-06 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980812+0000", 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7479999999999996e-05 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980819+0000", 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810384+0000", 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956448100000002 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810389+0000", 2024-08-30T23:46:26.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5269999999999998e-06 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86322 5.17 5.11e87097 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59714601299999, 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82972026099999996, 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86322 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:46:26.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2599999999999998e-07 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980792+0000", 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.894e-06 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980798+0000", 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980802+0000", 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3100000000000002e-06 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980870+0000", 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7617e-05 2024-08-30T23:46:26.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980877+0000", 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4309999999999997e-06 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810498+0000", 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82962015499999997 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810510+0000", 2024-08-30T23:46:26.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2413e-05 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86384 5.17 5.e2d2ce77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59707362199998, 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967802800000001, 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86384 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:46:26.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3000000000000002e-07 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980872+0000", 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2709999999999998e-06 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980877+0000", 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6999999999999996e-06 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980883+0000", 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2270000000000001e-06 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980958+0000", 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4972e-05 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980972+0000", 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4389999999999999e-05 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810534+0000", 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956191800000001 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810540+0000", 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199999999999997e-06 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.635 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86434 5.17 5.dc1639f7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59704575900003, 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967796100000002, 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86434 2024-08-30T23:46:26.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2000000000000001e-07 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980896+0000", 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980902+0000", 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980908+0000", 2024-08-30T23:46:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2260000000000002e-06 2024-08-30T23:46:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980995+0000", 2024-08-30T23:46:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6890000000000003e-05 2024-08-30T23:46:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.604e-06 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810562+0000", 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956157900000005 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810568+0000", 2024-08-30T23:46:26.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7749999999999998e-06 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86440 5.17 5.705d7d57 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59701998999998, 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968054700000005, 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.642 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86440 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5700000000000002e-07 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980918+0000", 2024-08-30T23:46:26.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9819999999999998e-06 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980924+0000", 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7259999999999999e-06 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980929+0000", 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1749999999999996e-06 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981037+0000", 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107353 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981044+0000", 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.272e-06 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810590+0000", 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82954588500000004 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810596+0000", 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5969999999999997e-06 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86445 5.17 5.715d0bd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59700085600002, 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968721000000001, 2024-08-30T23:46:26.647 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86445 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:46:26.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8700000000000002e-07 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980937+0000", 2024-08-30T23:46:26.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980942+0000", 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4739999999999996e-06 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980947+0000", 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8590000000000004e-06 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981087+0000", 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139479 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981094+0000", 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.827e-06 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810616+0000", 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82952199699999996 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810622+0000", 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.494e-06 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86454 5.17 5.47a3f917 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.59693460599999, 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82964718199999998, 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.652 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86454 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981002+0000", 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3499999999999996e-07 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981004+0000", 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9489999999999999e-06 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981010+0000", 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981014+0000", 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.617e-06 2024-08-30T23:46:26.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981118+0000", 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103795 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981124+0000", 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1469999999999998e-06 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810642+0000", 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.829518333 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810648+0000", 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9020000000000004e-06 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.657 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.17 505/491 [PushReplyOp(5:ed0feeac:::benchmark_data_smithi173_87639_object47011:head)])", 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 290.317770849, 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73458869599999999, 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1500000000000005e-07 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260173+0000", 2024-08-30T23:46:26.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7519999999999996e-06 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993751+0000", 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73357760100000002 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993758+0000", 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8630000000000002e-06 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993850+0000", 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2544e-05 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.994754+0000", 2024-08-30T23:46:26.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090342099999999995 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.17 505/491 [PushReplyOp(5:ef72d6c2:::benchmark_data_smithi173_87639_object47037:head)])", 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 290.30858261499998, 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73803978299999995, 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.662 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269354+0000", 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3099999999999998e-07 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269358+0000", 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169999999999996e-06 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.001419+0000", 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73206070400000001 2024-08-30T23:46:26.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.001426+0000", 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4939999999999997e-06 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.006768+0000", 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005341485 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.007393+0000", 2024-08-30T23:46:26.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062535200000000003 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(5.b e587,530 5:d5260a3e:::benchmark_data_smithi173_87639_object2707:head,442'5272)", 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 152.49081784000001, 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71060140599999999, 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:26.666 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087119+0000", 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-08-30T23:46:26.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087123+0000", 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2479999999999998e-06 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797124+0000", 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71000132900000001 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797133+0000", 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4440000000000008e-06 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797194+0000", 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0937999999999998e-05 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797719+0000", 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000525808 2024-08-30T23:46:26.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:144501 5.1 e590/540 5:827f28b3:::benchmark_data_smithi173_87639_object64261:head v 590'7080, mlcod=589'7078)", 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 149.34824417600001, 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64106508200000001, 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5900000000000001e-07 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229695+0000", 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6009999999999999e-06 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868724+0000", 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.639028284 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868732+0000", 2024-08-30T23:46:26.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3080000000000008e-06 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868794+0000", 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1963000000000003e-05 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868819+0000", 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5282999999999998e-05 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.870718+0000", 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001898582 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.870757+0000", 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8902000000000001e-05 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.674 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27034.0:105897 6.9 e663/619 6:937ff314:::benchmark_data_smithi173_114000_object6099:head v 663'6755, mlcod=663'6752)", 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:25.475982+0000", 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.101953703, 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.038538414999999999, 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.475982+0000", 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.475982+0000", 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.475984+0000", 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6440000000000001e-06 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.475993+0000", 2024-08-30T23:46:26.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5570000000000001e-06 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.511420+0000", 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035426920000000001 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.511428+0000", 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6629999999999993e-06 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:26.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.511493+0000", 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5190000000000004e-05 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.511510+0000", 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7039999999999999e-05 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.514487+0000", 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029765920000000001 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:26.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:25.514520+0000", 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3809000000000001e-05 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:26.679 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:46:26.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:26.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:26.836 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:26.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:26.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:26.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:27.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:27.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:27.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:27.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:27.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:27 smithi136 ceph-mon[20207]: pgmap v1480: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 545 op/s; 8225/286341 objects degraded (2.872%); 15334/286341 objects misplaced (5.355%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:46:27.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:27 smithi173 ceph-mon[20554]: pgmap v1480: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 545 op/s; 8225/286341 objects degraded (2.872%); 15334/286341 objects misplaced (5.355%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:46:27.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:27.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:27 smithi106 ceph-mon[16603]: pgmap v1480: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 545 op/s; 8225/286341 objects degraded (2.872%); 15334/286341 objects misplaced (5.355%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:46:27.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:27.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.480 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:27.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:46:27.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:27.614 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:27.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:27.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:27.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:27.847 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:27.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:27.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:27.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:27.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:28.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:28.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:28.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:28.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:28.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:28.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:28.390 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:28.425 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:46:28.425 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:46:28.426 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:46:28.426 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:46:28.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:28.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:28.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:28.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:28.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:28.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:28.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:28.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:28.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:28.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:29.018 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:29.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:46:29.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:29.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:29.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:29.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:29.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:29 smithi136 ceph-mon[20207]: pgmap v1482: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 8026/281058 objects degraded (2.856%); 14976/281058 objects misplaced (5.328%); 14 MiB/s, 225 objects/s recovering 2024-08-30T23:46:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:29 smithi173 ceph-mon[20554]: pgmap v1482: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 8026/281058 objects degraded (2.856%); 14976/281058 objects misplaced (5.328%); 14 MiB/s, 225 objects/s recovering 2024-08-30T23:46:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:29 smithi106 ceph-mon[16603]: pgmap v1482: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 8026/281058 objects degraded (2.856%); 14976/281058 objects misplaced (5.328%); 14 MiB/s, 225 objects/s recovering 2024-08-30T23:46:29.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:29.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:29.459 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:29.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:29.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:29.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:29.675 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:29.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:29.804 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:29.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:29.928 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:29.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:30.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:30.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:30.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:30.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:30.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:30.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:30.410 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:30.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:30.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:30.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:30.666 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:30.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.694 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:30.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:30.828 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:30.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:30.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:30.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:31.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:31.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:31.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:31.234 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:31.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:31 smithi173 ceph-mon[20554]: pgmap v1483: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 7856/277062 objects degraded (2.835%); 14819/277062 objects misplaced (5.349%); 11 MiB/s, 178 objects/s recovering 2024-08-30T23:46:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7856/277062 objects degraded (2.835%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:31.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:31 smithi106 ceph-mon[16603]: pgmap v1483: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 7856/277062 objects degraded (2.835%); 14819/277062 objects misplaced (5.349%); 11 MiB/s, 178 objects/s recovering 2024-08-30T23:46:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7856/277062 objects degraded (2.835%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:31.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:31.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:31.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:31 smithi136 ceph-mon[20207]: pgmap v1483: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 7856/277062 objects degraded (2.835%); 14819/277062 objects misplaced (5.349%); 11 MiB/s, 178 objects/s recovering 2024-08-30T23:46:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7856/277062 objects degraded (2.835%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:31.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:31.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:31.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:31.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:31.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:31.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:31.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:31.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:32.013 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:32.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:32.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:32.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:32.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:32.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:32.288 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:32.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:32.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:32.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:32.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:32.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:32.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:32.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:32.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:32.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:32.683 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:32.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:32.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:32.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:32.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:32.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:33.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:33.065 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:33.167 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:33.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:33.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:33.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:33 smithi106 ceph-mon[16603]: pgmap v1484: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 7641/273864 objects degraded (2.790%); 14612/273864 objects misplaced (5.335%); 12 MiB/s, 188 objects/s recovering 2024-08-30T23:46:33.403 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:33 smithi173 ceph-mon[20554]: pgmap v1484: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 7641/273864 objects degraded (2.790%); 14612/273864 objects misplaced (5.335%); 12 MiB/s, 188 objects/s recovering 2024-08-30T23:46:33.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:33.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:33.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:33 smithi136 ceph-mon[20207]: pgmap v1484: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.6 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 7641/273864 objects degraded (2.790%); 14612/273864 objects misplaced (5.335%); 12 MiB/s, 188 objects/s recovering 2024-08-30T23:46:33.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:33.580 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:33.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:33.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:33.714 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:33.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:33.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:33.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:33.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:33.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:34.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:34.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:34.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:34.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:34.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:46:34.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:34.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:34.412 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:46:34.412 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:46:34.412 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:46:34.412 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:46:34.412 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:46:34.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:34.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:34.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:34.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:34.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:34.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:34.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:34.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:34.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:34.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:34.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:35.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:35.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:35.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:35.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:35.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:35 smithi136 ceph-mon[20207]: pgmap v1485: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 601 op/s; 7641/270315 objects degraded (2.827%); 14281/270315 objects misplaced (5.283%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:46:35.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:35 smithi106 ceph-mon[16603]: pgmap v1485: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 601 op/s; 7641/270315 objects degraded (2.827%); 14281/270315 objects misplaced (5.283%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:46:35.381 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:35.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:35.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:35 smithi173 ceph-mon[20554]: pgmap v1485: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 601 op/s; 7641/270315 objects degraded (2.827%); 14281/270315 objects misplaced (5.283%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:46:35.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:35.642 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:35.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:35.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:35.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:35.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:35.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:36.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:36.028 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:36.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:46:36.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:36.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:36.191 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:36.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-08-30T23:46:36.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:36.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:36.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7641/270315 objects degraded (2.827%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:36.351 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7641/270315 objects degraded (2.827%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:36.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:36.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:36.523 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:36.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7641/270315 objects degraded (2.827%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:36.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:36.673 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:36.714 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:36.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:36.822 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:36.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:36.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:36.899 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:36.916 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:46:36.916 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:46:36.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58973 5.4 5.af96ba24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.114618+0000", 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.79954242899998, 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.997737031, 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.917 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58973 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114618+0000", 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114618+0000", 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114620+0000", 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.629e-06 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114646+0000", 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5905999999999999e-05 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114658+0000", 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2362e-05 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114667+0000", 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7819999999999999e-06 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114717+0000", 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9570000000000001e-05 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114726+0000", 2024-08-30T23:46:36.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1640000000000007e-06 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112346+0000", 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99762036799999998 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112355+0000", 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2499999999999995e-06 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58980 5.8 5.98770828 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.114757+0000", 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.799403627, 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99760046099999999, 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.922 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58980 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114757+0000", 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114757+0000", 2024-08-30T23:46:36.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114758+0000", 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6000000000000003e-07 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114778+0000", 2024-08-30T23:46:36.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9809e-05 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114787+0000", 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4450000000000002e-06 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114793+0000", 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6289999999999998e-06 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114842+0000", 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9650000000000001e-05 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114850+0000", 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3350000000000001e-06 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112343+0000", 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99749290499999999 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112358+0000", 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5028000000000001e-05 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58983 5.4 5.4902cd24 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.114807+0000", 2024-08-30T23:46:36.927 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.79935371400001, 2024-08-30T23:46:36.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99758265700000004, 2024-08-30T23:46:36.929 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.929 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.929 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.929 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.930 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.930 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.930 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58983 2024-08-30T23:46:36.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.930 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114807+0000", 2024-08-30T23:46:36.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114807+0000", 2024-08-30T23:46:36.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114808+0000", 2024-08-30T23:46:36.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2899999999999999e-07 2024-08-30T23:46:36.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114829+0000", 2024-08-30T23:46:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1347000000000001e-05 2024-08-30T23:46:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114836+0000", 2024-08-30T23:46:36.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2509999999999997e-06 2024-08-30T23:46:36.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114841+0000", 2024-08-30T23:46:36.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.933e-06 2024-08-30T23:46:36.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114882+0000", 2024-08-30T23:46:36.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1069000000000003e-05 2024-08-30T23:46:36.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.114894+0000", 2024-08-30T23:46:36.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1980999999999999e-05 2024-08-30T23:46:36.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112382+0000", 2024-08-30T23:46:36.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99748783600000002 2024-08-30T23:46:36.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112390+0000", 2024-08-30T23:46:36.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8110000000000003e-06 2024-08-30T23:46:36.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.941 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.941 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:58991 5.8 5.4658f208 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.116833+0000", 2024-08-30T23:46:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.79732810199999, 2024-08-30T23:46:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99558751199999995, 2024-08-30T23:46:36.943 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.943 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.943 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.943 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.943 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.944 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.944 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 58991 2024-08-30T23:46:36.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.944 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116833+0000", 2024-08-30T23:46:36.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116833+0000", 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116834+0000", 2024-08-30T23:46:36.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6339999999999999e-06 2024-08-30T23:46:36.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116859+0000", 2024-08-30T23:46:36.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4153999999999999e-05 2024-08-30T23:46:36.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116871+0000", 2024-08-30T23:46:36.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.234e-05 2024-08-30T23:46:36.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116879+0000", 2024-08-30T23:46:36.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.106e-06 2024-08-30T23:46:36.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116923+0000", 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4403999999999997e-05 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.116929+0000", 2024-08-30T23:46:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7690000000000002e-06 2024-08-30T23:46:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112412+0000", 2024-08-30T23:46:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99548259500000003 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112420+0000", 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5099999999999998e-06 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59002 5.8 5.40785bb8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.145317+0000", 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.76884421300002, 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96713300800000002, 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59002 2024-08-30T23:46:36.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145317+0000", 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145317+0000", 2024-08-30T23:46:36.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145319+0000", 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2589999999999999e-06 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145344+0000", 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4689e-05 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145362+0000", 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8624999999999999e-05 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145370+0000", 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.576e-06 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145432+0000", 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2663000000000006e-05 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.145442+0000", 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5049999999999996e-06 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112442+0000", 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96700018099999996 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112450+0000", 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5100000000000001e-06 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59008 5.4 5.ad204744 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.336873+0000", 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.57728795200001, 2024-08-30T23:46:36.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77569046100000005, 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59008 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336873+0000", 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336873+0000", 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336875+0000", 2024-08-30T23:46:36.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1289999999999999e-06 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336899+0000", 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4173e-05 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336918+0000", 2024-08-30T23:46:36.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8940000000000002e-05 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336927+0000", 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5720000000000003e-06 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336989+0000", 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2285000000000004e-05 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.336996+0000", 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2439999999999996e-06 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112555+0000", 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77555825599999995 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112563+0000", 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8619999999999992e-06 2024-08-30T23:46:36.963 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:59009 5.a 5.536bab9a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:32.341155+0000", 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.57300617300001, 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77138790199999996, 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:36.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 59009 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341155+0000", 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341155+0000", 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341157+0000", 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.695e-06 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341208+0000", 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.041e-05 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341230+0000", 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.247e-05 2024-08-30T23:46:36.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341240+0000", 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0069e-05 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341303+0000", 2024-08-30T23:46:36.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2879000000000006e-05 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:32.341312+0000", 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4100000000000008e-06 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112520+0000", 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.771208067 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:33.112543+0000", 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2901999999999999e-05 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.970 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.2 425/420 [PushOp(5:4ffcd3d6:::benchmark_data_smithi173_87639_object4590:head, version: 398'287, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:4ffcd3d6:::benchmark_data_smithi173_87639_object4590:head@398'287, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 476.52209133600002, 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94084964800000004, 2024-08-30T23:46:36.971 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392070+0000", 2024-08-30T23:46:36.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1600000000000001e-07 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.392115+0000", 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5108999999999998e-05 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331203+0000", 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93908753700000003 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331236+0000", 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2728999999999998e-05 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331454+0000", 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021814600000000001 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.331479+0000", 2024-08-30T23:46:36.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5789e-05 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.332919+0000", 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014397220000000001 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.977 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.4 425/420 [PushOp(5:2400e38c:::benchmark_data_smithi173_87639_object6909:head, version: 399'435, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:2400e38c:::benchmark_data_smithi173_87639_object6909:head@399'435, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:40.586744+0000", 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 476.32741695300001, 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84143205700000001, 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.978 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586744+0000", 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586744+0000", 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586745+0000", 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3799999999999999e-06 2024-08-30T23:46:36.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:40.586787+0000", 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1313999999999998e-05 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426748+0000", 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83996137699999995 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426778+0000", 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0380000000000001e-05 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426960+0000", 2024-08-30T23:46:36.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018135499999999999 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.426980+0000", 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0245999999999999e-05 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.428176+0000", 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001196005 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.8 426/420 [PushOp(5:1176d18c:::benchmark_data_smithi173_87639_object6158:head, version: 399'397, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1176d18c:::benchmark_data_smithi173_87639_object6158:head@399'397, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:41.768366+0000", 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 475.14579440900002, 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91116549400000002, 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768366+0000", 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768366+0000", 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768367+0000", 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8100000000000003e-07 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:41.768398+0000", 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0908000000000003e-05 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678320+0000", 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.90992154700000005 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678348+0000", 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8279000000000001e-05 2024-08-30T23:46:36.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678600+0000", 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025234299999999998 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.678614+0000", 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4294000000000001e-05 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:42.679532+0000", 2024-08-30T23:46:36.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091754200000000003 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.2 427/420 [PushOp(5:4ae76459:::benchmark_data_smithi173_87639_object8076:head, version: 400'527, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:4ae76459:::benchmark_data_smithi173_87639_object8076:head@400'527, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:38:44.354983+0000", 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 472.55917773099998, 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87486126600000003, 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.988 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.354983+0000", 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.354983+0000", 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.354985+0000", 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.463e-06 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:44.355050+0000", 2024-08-30T23:46:36.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5874e-05 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228350+0000", 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.87329974600000004 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228379+0000", 2024-08-30T23:46:36.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8702000000000001e-05 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228600+0000", 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022151199999999999 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:36.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.228616+0000", 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5291999999999999e-05 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:38:45.229844+0000", 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012286770000000001 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 449/432 [PushOp(5:c55d6b41:::benchmark_data_smithi173_87639_object8462:head, version: 400'521, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c55d6b41:::benchmark_data_smithi173_87639_object8462:head@400'521, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 431.93234516799998, 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73936645499999998, 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:46:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981816+0000", 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9900000000000001e-07 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:24.981923+0000", 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010642 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719510+0000", 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73758770799999995 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:36.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719527+0000", 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6744000000000001e-05 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719625+0000", 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7499999999999998e-05 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.719634+0000", 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2820000000000002e-06 2024-08-30T23:46:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.721182+0000", 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015483020000000001 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 450/432 [PushOp(5:cdb9415d:::benchmark_data_smithi173_87639_object8623:head, version: 400'532, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cdb9415d:::benchmark_data_smithi173_87639_object8623:head@400'532, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:25.931361+0000", 2024-08-30T23:46:36.998 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 430.98279976700002, 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72661587400000005, 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931361+0000", 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:36.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931361+0000", 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931366+0000", 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4739999999999999e-06 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.931429+0000", 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3728999999999999e-05 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656006+0000", 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.72457618999999995 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656019+0000", 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3563e-05 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656127+0000", 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108312 2024-08-30T23:46:37.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.656138+0000", 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0519999999999999e-05 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.657977+0000", 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001839086 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.4 450/448 [PushReplyOp(5:2985048c:::benchmark_data_smithi173_87639_object67330:head)])", 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 430.95472724500002, 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77832121700000001, 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959434+0000", 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9399999999999995e-07 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:25.959438+0000", 2024-08-30T23:46:37.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.715e-06 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737214+0000", 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77777609000000003 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737219+0000", 2024-08-30T23:46:37.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737291+0000", 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1145999999999999e-05 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:39:26.737755+0000", 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00046422199999999997 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.10 517/491 [PushReplyOp(5:0f68070c:::benchmark_data_smithi173_87639_object35798:head)])", 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:00.685200+0000", 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 276.22896043999998, 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97738226100000003, 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:00.685200+0000", 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:00.685200+0000", 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:00.685201+0000", 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6199999999999997e-07 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:00.685206+0000", 2024-08-30T23:46:37.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7149999999999997e-06 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:01.660988+0000", 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97578266899999999 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:01.660997+0000", 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8000000000000004e-06 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:01.661094+0000", 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6867999999999995e-05 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:01.662583+0000", 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001488547 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.10 520/491 [PushReplyOp(5:0930cc9a:::benchmark_data_smithi173_87639_object39623:head)])", 2024-08-30T23:46:37.011 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:03.858527+0000", 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 273.05563369999999, 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.84500469499999997, 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:03.858527+0000", 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:03.858527+0000", 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:03.858528+0000", 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:03.858532+0000", 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4429999999999996e-06 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:04.702023+0000", 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.843491031 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:04.702042+0000", 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8559999999999998e-05 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:04.702165+0000", 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000123058 2024-08-30T23:46:37.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:04.703532+0000", 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013668720000000001 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.10 524/491 [PushReplyOp(5:0b7c89aa:::benchmark_data_smithi173_87639_object52533:head)])", 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:10.963252+0000", 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 265.95090899500002, 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71143720799999999, 2024-08-30T23:46:37.015 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:10.963252+0000", 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:10.963252+0000", 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:10.963252+0000", 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5000000000000003e-07 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:10.963257+0000", 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0890000000000002e-06 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:11.673811+0000", 2024-08-30T23:46:37.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71055404600000005 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:11.673821+0000", 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0139999999999999e-05 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:11.673870+0000", 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8798000000000003e-05 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:11.674689+0000", 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00081958499999999995 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:22497 6.f 6:f81ba605:::benchmark_data_smithi173_114000_object22496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:45:00.049266+0000", 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.864894785000004, 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27151897699999999, 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:37.019 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22497 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049266+0000", 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049266+0000", 2024-08-30T23:46:37.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049268+0000", 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.494e-06 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049290+0000", 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1980999999999998e-05 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049306+0000", 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6776e-05 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049313+0000", 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0299999999999996e-06 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049366+0000", 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2923999999999997e-05 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049471+0000", 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104577 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.049558+0000", 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7244999999999996e-05 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.050407+0000", 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00084843500000000003 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:37.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.050668+0000", 2024-08-30T23:46:37.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000261196 2024-08-30T23:46:37.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:37.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:37.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320722+0000", 2024-08-30T23:46:37.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27005453099999999 2024-08-30T23:46:37.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320757+0000", 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4479e-05 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320785+0000", 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8309e-05 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.026 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27034.0:22497 6.f e629/619 ondisk, result = 0)", 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:45:00.050661+0000", 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 96.863500199000001, 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27013067000000002, 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.050661+0000", 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.050661+0000", 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.050661+0000", 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9200000000000002e-07 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.050663+0000", 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3800000000000001e-06 2024-08-30T23:46:37.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320679+0000", 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27001523199999999 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320682+0000", 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2930000000000001e-06 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320714+0000", 2024-08-30T23:46:37.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1628000000000003e-05 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320718+0000", 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.668e-06 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:00.320791+0000", 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3077000000000005e-05 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:47953 6.f 6:f96de401:::benchmark_data_smithi173_114000_object47952:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e635)", 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:45:19.013584+0000", 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 77.900577037000005, 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.272986004, 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 47953 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.031 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013584+0000", 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013584+0000", 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:46:37.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013585+0000", 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7000000000000003e-07 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013604+0000", 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9584000000000001e-05 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013614+0000", 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0139999999999999e-05 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013619+0000", 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.724e-06 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013659+0000", 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0299999999999997e-05 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013743+0000", 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3139999999999993e-05 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.013828+0000", 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5068999999999995e-05 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.015493+0000", 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001665536 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.268746+0000", 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.25325324700000001 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.286506+0000", 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017759885 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.286538+0000", 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1940000000000003e-05 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:45:19.286570+0000", 2024-08-30T23:46:37.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1569000000000002e-05 2024-08-30T23:46:37.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.037 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.037 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:46:37.037 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:46:37.037 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:46:37.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:37.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:37.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:37.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:37.270 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:37 smithi106 ceph-mon[16603]: pgmap v1486: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 7519/266088 objects degraded (2.826%); 14281/266088 objects misplaced (5.367%); 11 MiB/s, 175 objects/s recovering 2024-08-30T23:46:37.270 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:37.270 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:37.270 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:37.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:37.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:37 smithi173 ceph-mon[20554]: pgmap v1486: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 7519/266088 objects degraded (2.826%); 14281/266088 objects misplaced (5.367%); 11 MiB/s, 175 objects/s recovering 2024-08-30T23:46:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:37.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:37.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:37.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:37 smithi136 ceph-mon[20207]: pgmap v1486: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.4 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 674 op/s; 7519/266088 objects degraded (2.826%); 14281/266088 objects misplaced (5.367%); 11 MiB/s, 175 objects/s recovering 2024-08-30T23:46:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:37.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.562 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:37.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:37.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:37.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:37.826 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:37.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:46:37.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:37.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:37.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:38.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:38.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:38.121 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:46:38.192 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2734983748' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:46:38.192 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1220756464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:38.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:38.215 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:38.223 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:38.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2734983748' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:46:38.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1220756464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:38.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:38.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:38.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:38.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2734983748' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:46:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1220756464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:38.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:38.621 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:38.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:38.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:38.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:46:38.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:46:38.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:46:38.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:46:38.752 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:46:38.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:46:38.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:46:38.760 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:38.761 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:38.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:38.883 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:38.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:38.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:39.008 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:39.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:39.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:39.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:39.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:39.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:39.394 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:39.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.528 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:39.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:39 smithi136 ceph-mon[20207]: pgmap v1487: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 7292/261066 objects degraded (2.793%); 13935/261066 objects misplaced (5.338%); 12 MiB/s, 197 objects/s recovering 2024-08-30T23:46:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:39 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2734983748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:46:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:39 smithi136 ceph-mon[20207]: osdmap e665: 12 total, 12 up, 11 in 2024-08-30T23:46:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:39 smithi173 ceph-mon[20554]: pgmap v1487: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 7292/261066 objects degraded (2.793%); 13935/261066 objects misplaced (5.338%); 12 MiB/s, 197 objects/s recovering 2024-08-30T23:46:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:39 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2734983748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:46:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:39 smithi173 ceph-mon[20554]: osdmap e665: 12 total, 12 up, 11 in 2024-08-30T23:46:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:39 smithi106 ceph-mon[16603]: pgmap v1487: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 713 op/s; 7292/261066 objects degraded (2.793%); 13935/261066 objects misplaced (5.338%); 12 MiB/s, 197 objects/s recovering 2024-08-30T23:46:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:39 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2734983748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:46:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:39 smithi106 ceph-mon[16603]: osdmap e665: 12 total, 12 up, 11 in 2024-08-30T23:46:39.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:39.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:39.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:39.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:39.806 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:46:39.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:39.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:39.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:40.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:40.048 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:40.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-08-30T23:46:40.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:40.168 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:40.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:40.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:40.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:40.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:40.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:40.540 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:40.587 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:40.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:40.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:40.721 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:40.721 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:40.818 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:40.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:40.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:40.985 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:40.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:41.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:41.119 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:41.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:41.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:41.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:41.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:41.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:41 smithi173 ceph-mon[20554]: pgmap v1489: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 7089/257619 objects degraded (2.752%); 13769/257619 objects misplaced (5.345%); 11 MiB/s, 181 objects/s recovering 2024-08-30T23:46:41.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7089/257619 objects degraded (2.752%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:41.436 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:41 smithi173 ceph-mon[20554]: osdmap e666: 12 total, 12 up, 11 in 2024-08-30T23:46:41.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:41.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:46:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:41 smithi136 ceph-mon[20207]: pgmap v1489: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 7089/257619 objects degraded (2.752%); 13769/257619 objects misplaced (5.345%); 11 MiB/s, 181 objects/s recovering 2024-08-30T23:46:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7089/257619 objects degraded (2.752%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:41 smithi136 ceph-mon[20207]: osdmap e666: 12 total, 12 up, 11 in 2024-08-30T23:46:41.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:41 smithi106 ceph-mon[16603]: pgmap v1489: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 7089/257619 objects degraded (2.752%); 13769/257619 objects misplaced (5.345%); 11 MiB/s, 181 objects/s recovering 2024-08-30T23:46:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7089/257619 objects degraded (2.752%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:41 smithi106 ceph-mon[16603]: osdmap e666: 12 total, 12 up, 11 in 2024-08-30T23:46:41.644 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:41.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:41.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:41.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:41.853 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:46:41.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:41.925 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:41.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:41.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:42.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:42.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:42.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:42.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:42.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:42.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:42.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:42.439 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:46:42.440 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:46:42.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:42.543 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:42.574 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:42.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:46:42.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:42.708 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:42.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:42.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:42.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:42.962 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:42.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:43.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:43.093 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:43.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:46:43.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:43.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:43.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:43.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:43.344 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:43 smithi106 ceph-mon[16603]: pgmap v1491: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 6857/254244 objects degraded (2.697%); 13566/254244 objects misplaced (5.336%); 12 MiB/s, 187 objects/s recovering 2024-08-30T23:46:43.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:43.478 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:43 smithi136 ceph-mon[20207]: pgmap v1491: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 6857/254244 objects degraded (2.697%); 13566/254244 objects misplaced (5.336%); 12 MiB/s, 187 objects/s recovering 2024-08-30T23:46:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:43 smithi173 ceph-mon[20554]: pgmap v1491: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 6857/254244 objects degraded (2.697%); 13566/254244 objects misplaced (5.336%); 12 MiB/s, 187 objects/s recovering 2024-08-30T23:46:43.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:43.617 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:43.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:43.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:43.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:43.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:46:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:43.885 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:43.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:43.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:44.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:44.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.174 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:44.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:44.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:44.300 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:44.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:44.426 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:44.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:44.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:44.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:44.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:44.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:44.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:44.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:44.989 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:44.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:45.003 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:45.009 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:45.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:45.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:45.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:45.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:45.321 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:45.329 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:45 smithi106 ceph-mon[16603]: pgmap v1492: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 6857/250530 objects degraded (2.737%); 13214/250530 objects misplaced (5.274%); 14 MiB/s, 216 objects/s recovering 2024-08-30T23:46:45.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:45.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:45.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:45.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:45.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:45 smithi136 ceph-mon[20207]: pgmap v1492: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 6857/250530 objects degraded (2.737%); 13214/250530 objects misplaced (5.274%); 14 MiB/s, 216 objects/s recovering 2024-08-30T23:46:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:45 smithi173 ceph-mon[20554]: pgmap v1492: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 5.1 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 6857/250530 objects degraded (2.737%); 13214/250530 objects misplaced (5.274%); 14 MiB/s, 216 objects/s recovering 2024-08-30T23:46:45.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:45.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:45.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:45.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:45.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:45.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:45.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:45.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:45.877 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:45.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:45.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:46.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:46.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-08-30T23:46:46.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:46.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:46.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.275 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:46.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:46.325 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6857/250530 objects degraded (2.737%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:46.325 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:46:46.325 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:46:46.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:46:46.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:46.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:46.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:46.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:46 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:46.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:46.399 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.400 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:46.418 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:46:46.419 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:46:46.458 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:46:46.458 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:46:46.495 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:46:46.495 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:46:46.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:46.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:46.529 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:46:46.529 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6857/250530 objects degraded (2.737%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:46.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:46 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:46.558 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:46:46.559 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6857/250530 objects degraded (2.737%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:46 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:46.595 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:46:46.595 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:46:46.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:46.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:46.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:46.790 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:46.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:46.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:46.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:47.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:47.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:47.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:47.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:47.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:47.291 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:47.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:47.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:47.430 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:46:47.430 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:46:47.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3376208512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:47.462 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:46:47.462 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:46:47.496 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:46:47.496 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:46:47.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:47.534 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:46:47.534 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:46:47.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3376208512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:47.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:47.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:47.573 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:46:47.573 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:46:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3376208512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:46:47.606 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:46:47.606 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:46:47.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:47.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:47.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:47.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:47.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:47.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:47.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:48.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:48.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:48.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:48.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:48.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:48.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:48.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:48.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:48 smithi136 ceph-mon[20207]: osdmap e667: 12 total, 12 up, 11 in 2024-08-30T23:46:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1058997524' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:46:48.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:48.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:48 smithi173 ceph-mon[20554]: osdmap e667: 12 total, 12 up, 11 in 2024-08-30T23:46:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:48.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1058997524' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:46:48.582 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:46:48.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:48.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:48.582 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:48 smithi106 ceph-mon[16603]: osdmap e667: 12 total, 12 up, 11 in 2024-08-30T23:46:48.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:48.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:48.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1058997524' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:46:48.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:48.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:48.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:48.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:48.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:48.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:49.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:49.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:49.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:49.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:49.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:49.331 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:49.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:49.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:49.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:49.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:49.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:49 smithi136 ceph-mon[20207]: pgmap v1495: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 6504/240840 objects degraded (2.701%); 12787/240840 objects misplaced (5.309%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:46:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:49 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1058997524' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:46:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:49 smithi136 ceph-mon[20207]: osdmap e668: 12 total, 12 up, 11 in 2024-08-30T23:46:49.560 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:49.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:49 smithi173 ceph-mon[20554]: pgmap v1495: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 6504/240840 objects degraded (2.701%); 12787/240840 objects misplaced (5.309%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:46:49.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:49 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:49.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1058997524' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:46:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:49 smithi173 ceph-mon[20554]: osdmap e668: 12 total, 12 up, 11 in 2024-08-30T23:46:49.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:49.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:49 smithi106 ceph-mon[16603]: pgmap v1495: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.9 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 698 op/s; 6504/240840 objects degraded (2.701%); 12787/240840 objects misplaced (5.309%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:46:49.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:49 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:46:49.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1058997524' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:46:49.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:49 smithi106 ceph-mon[16603]: osdmap e668: 12 total, 12 up, 11 in 2024-08-30T23:46:49.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.711 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:49.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:49.813 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:49.843 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:49.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:49.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:49.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:50.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:50.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:50.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:50.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:50.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:50.366 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:50.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:50.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.495 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:46:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:46:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:46:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:50.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:50.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:50.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:50 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:46:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:46:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:46:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:50.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:50.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:50.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:50.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:46:50.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:46:50.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:46:50.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:46:50.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:46:50.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:46:50.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:46:50.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:50.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:50.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.754 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:50.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:50.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:50.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:50.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:51.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:51.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:51.145 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:51.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:51.278 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:51.380 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:51.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.406 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:51.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:51 smithi173 ceph-mon[20554]: pgmap v1497: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 6356/236868 objects degraded (2.683%); 12637/236868 objects misplaced (5.335%); 11 MiB/s, 178 objects/s recovering 2024-08-30T23:46:51.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6356/236868 objects degraded (2.683%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:51.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:51 smithi136 ceph-mon[20207]: pgmap v1497: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 6356/236868 objects degraded (2.683%); 12637/236868 objects misplaced (5.335%); 11 MiB/s, 178 objects/s recovering 2024-08-30T23:46:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6356/236868 objects degraded (2.683%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:51.542 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:51 smithi106 ceph-mon[16603]: pgmap v1497: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.8 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 6356/236868 objects degraded (2.683%); 12637/236868 objects misplaced (5.335%); 11 MiB/s, 178 objects/s recovering 2024-08-30T23:46:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6356/236868 objects degraded (2.683%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:46:51.644 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:51.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:51.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:51.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:51.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:51.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:51.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:52.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:52.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:52.131 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:46:52.131 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:46:52.131 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.131 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:123845 6.4 6:287b6900:::benchmark_data_smithi173_114000_object24047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-08-30T23:46:52.131 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.078662+0000", 2024-08-30T23:46:52.131 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.051584791999999997, 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.051616885000000001, 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 123845 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.132 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.078662+0000", 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.078662+0000", 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.078663+0000", 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.418e-06 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.133 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.078669+0000", 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7960000000000001e-06 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.078683+0000", 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3412e-05 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.078702+0000", 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9046e-05 2024-08-30T23:46:52.134 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.094839+0000", 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016137140000000001 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.096231+0000", 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001391864 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,5", 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.096663+0000", 2024-08-30T23:46:52.135 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00043233699999999998 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.106932+0000", 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010269051 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.106954+0000", 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2038000000000001e-05 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.136 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:123874 6.4 6:2329460e:::benchmark_data_smithi173_114000_object24076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.114705+0000", 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015541650000000001, 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015668080000000001, 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 123874 2024-08-30T23:46:52.137 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.114705+0000", 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.114705+0000", 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.138 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.114707+0000", 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5379999999999999e-06 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.114712+0000", 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5489999999999997e-06 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.114724+0000", 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1735e-05 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.139 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.114731+0000", 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8859999999999998e-06 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.120179+0000", 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005447833 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.120485+0000", 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030612199999999998 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.140 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 3,5", 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.120905+0000", 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000420715 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.125657+0000", 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047518439999999999 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.141 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:123879 6.e e668/619 6:7066b842:::benchmark_data_smithi173_114000_object24081:head v 668'7830, mlcod=668'7828)", 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.119721+0000", 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.01052617, 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010729648, 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119721+0000", 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.142 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119721+0000", 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119722+0000", 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5000000000000001e-07 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119727+0000", 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.75e-06 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.143 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119756+0000", 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8651000000000001e-05 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119763+0000", 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0570000000000004e-06 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119915+0000", 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000151609 2024-08-30T23:46:52.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.119959+0000", 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4481000000000003e-05 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.145 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:123880 6.e e668/619 6:7cb1a699:::benchmark_data_smithi173_114000_object24082:head v 668'7831, mlcod=668'7828)", 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.123507+0000", 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0067392520000000003, 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0069920989999999999, 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.123507+0000", 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.123507+0000", 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.123509+0000", 2024-08-30T23:46:52.147 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1710000000000001e-06 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.123515+0000", 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6760000000000001e-06 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.123552+0000", 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6690000000000003e-05 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.148 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.123560+0000", 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2940000000000006e-06 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.125921+0000", 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023606109999999999 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.125972+0000", 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1005999999999999e-05 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.149 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(6.4 668/650 [PushReplyOp(6:2cf7eca7:::benchmark_data_smithi173_114000_object39977:head)])", 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.127855+0000", 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.002391706, 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002689674, 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:46:52.150 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127855+0000", 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127855+0000", 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127856+0000", 2024-08-30T23:46:52.151 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.24e-06 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127869+0000", 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2248e-05 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127929+0000", 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0749999999999999e-05 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.152 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127939+0000", 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6129999999999993e-06 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27034.0:123874 6.4 e668/650)", 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.127874+0000", 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.002372248, 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027044019999999998, 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:46:52.153 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127874+0000", 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127874+0000", 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127875+0000", 2024-08-30T23:46:52.154 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.04e-07 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127878+0000", 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0369999999999999e-06 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127969+0000", 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0798999999999996e-05 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.155 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127973+0000", 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.549e-06 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:123881 6.e e668/619 6:7ff0d595:::benchmark_data_smithi173_114000_object24083:head v 668'7832, mlcod=668'7828)", 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.127885+0000", 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.002361765, 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027335770000000001, 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.156 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127885+0000", 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127885+0000", 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.157 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127885+0000", 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9799999999999999e-07 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127890+0000", 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3229999999999999e-06 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127985+0000", 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5147000000000002e-05 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.158 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.127989+0000", 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5950000000000001e-06 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.129797+0000", 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001807917 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.129815+0000", 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7929000000000001e-05 2024-08-30T23:46:52.159 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:123883 6.d e668/654 6:b1cea471:::benchmark_data_smithi173_114000_object24085:head v 668'7447, mlcod=630'1782)", 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:52.128103+0000", 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0021441009999999998, 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025651860000000001, 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.160 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128103+0000", 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128103+0000", 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128104+0000", 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7599999999999996e-07 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128110+0000", 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8619999999999997e-06 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128138+0000", 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7577e-05 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128145+0000", 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6340000000000003e-06 2024-08-30T23:46:52.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128203+0000", 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8217000000000002e-05 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:52.128227+0000", 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4218000000000001e-05 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:46:52.163 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 8 2024-08-30T23:46:52.164 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:46:52.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:52.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:52.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:52 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:52 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:52.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:52.450 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:52 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:52 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:52.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:52.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:52 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:52.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:52 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:52.591 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:52.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:52.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:52.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:52.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:52.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:52.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:52.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:53.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:53.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:53.198 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:53.232 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:53.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:53 smithi106 ceph-mon[16603]: pgmap v1498: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 6076/233781 objects degraded (2.599%); 12393/233781 objects misplaced (5.301%); 13 MiB/s, 200 objects/s recovering 2024-08-30T23:46:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:53 smithi106 ceph-mon[16603]: osdmap e669: 12 total, 12 up, 11 in 2024-08-30T23:46:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:53.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:53.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:53.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:53.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:53 smithi136 ceph-mon[20207]: pgmap v1498: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 6076/233781 objects degraded (2.599%); 12393/233781 objects misplaced (5.301%); 13 MiB/s, 200 objects/s recovering 2024-08-30T23:46:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:53 smithi136 ceph-mon[20207]: osdmap e669: 12 total, 12 up, 11 in 2024-08-30T23:46:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:53.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:53 smithi173 ceph-mon[20554]: pgmap v1498: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 6076/233781 objects degraded (2.599%); 12393/233781 objects misplaced (5.301%); 13 MiB/s, 200 objects/s recovering 2024-08-30T23:46:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:53 smithi173 ceph-mon[20554]: osdmap e669: 12 total, 12 up, 11 in 2024-08-30T23:46:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:53.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:53.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:53.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:53.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:53.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:53.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:53.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:53.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:54.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:54.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:54.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:54.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:54.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:54.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:54.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:54.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:54.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:54 smithi106 ceph-mon[16603]: osdmap e670: 12 total, 12 up, 11 in 2024-08-30T23:46:54.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:54 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:46:54.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:54 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:46:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:54 smithi136 ceph-mon[20207]: osdmap e670: 12 total, 12 up, 11 in 2024-08-30T23:46:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:54 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:46:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:54 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:46:54.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:54 smithi173 ceph-mon[20554]: osdmap e670: 12 total, 12 up, 11 in 2024-08-30T23:46:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:54 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:46:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:54 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:46:54.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:54.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:54.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:54.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:54.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:54.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:54.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:54.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:46:55.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:55.049 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:55.100 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:46:55.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:55.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:55.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:55.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:55.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:55.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.464 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:55.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:55.474 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:55 smithi106 ceph-mon[16603]: pgmap v1501: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 612 op/s; 6076/229812 objects degraded (2.644%); 12046/229812 objects misplaced (5.242%); 12 MiB/s, 194 objects/s recovering 2024-08-30T23:46:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:55 smithi136 ceph-mon[20207]: pgmap v1501: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 612 op/s; 6076/229812 objects degraded (2.644%); 12046/229812 objects misplaced (5.242%); 12 MiB/s, 194 objects/s recovering 2024-08-30T23:46:55.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:55.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:55 smithi173 ceph-mon[20554]: pgmap v1501: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.7 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 612 op/s; 6076/229812 objects degraded (2.644%); 12046/229812 objects misplaced (5.242%); 12 MiB/s, 194 objects/s recovering 2024-08-30T23:46:55.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:55.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:55.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:55.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:55.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:55.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:55.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:55.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:56.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:56.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:56.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:46:56.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:56.237 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:56.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:56.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.367 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:56.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:56.511 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:56.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6076/229812 objects degraded (2.644%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:46:56.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6076/229812 objects degraded (2.644%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:46:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6076/229812 objects degraded (2.644%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:46:56.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:56.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:56.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:46:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:56.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:46:56.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:56.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:56.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:57.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:46:57.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:57.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:46:57.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:57.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.292 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:57.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:57.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:46:57.428 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:46:57.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:57 smithi136 ceph-mon[20207]: pgmap v1502: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 5885/226644 objects degraded (2.597%); 12046/226644 objects misplaced (5.315%); 11 MiB/s, 181 objects/s recovering 2024-08-30T23:46:57.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:57.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:57 smithi173 ceph-mon[20554]: pgmap v1502: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 5885/226644 objects degraded (2.597%); 12046/226644 objects misplaced (5.315%); 11 MiB/s, 181 objects/s recovering 2024-08-30T23:46:57.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:57 smithi106 ceph-mon[16603]: pgmap v1502: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 632 op/s; 5885/226644 objects degraded (2.597%); 12046/226644 objects misplaced (5.315%); 11 MiB/s, 181 objects/s recovering 2024-08-30T23:46:57.660 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:57.684 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:57.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:57.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:57.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:57.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:57.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:58.038 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:58.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:58.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:58.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:58.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:58.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:58.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:46:58.440 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:46:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:58 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:46:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:58 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:46:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:58 smithi136 ceph-mon[20207]: osdmap e671: 12 total, 12 up, 11 in 2024-08-30T23:46:58.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:46:58.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:46:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:58 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:46:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:58 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:46:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:58 smithi173 ceph-mon[20554]: osdmap e671: 12 total, 12 up, 11 in 2024-08-30T23:46:58.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:58 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:46:58.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:58 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:46:58.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:58 smithi106 ceph-mon[16603]: osdmap e671: 12 total, 12 up, 11 in 2024-08-30T23:46:58.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:58.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:58.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:58.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:58.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:46:58.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:58.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:46:59.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:46:59.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:46:59.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:46:59.204 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:46:59.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:59.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:59.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:46:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:59.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:46:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:46:59 smithi136 ceph-mon[20207]: pgmap v1504: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 5635/220155 objects degraded (2.560%); 11595/220155 objects misplaced (5.267%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:46:59.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:46:59.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:46:59 smithi173 ceph-mon[20554]: pgmap v1504: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 5635/220155 objects degraded (2.560%); 11595/220155 objects misplaced (5.267%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:46:59.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:46:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:46:59 smithi106 ceph-mon[16603]: pgmap v1504: 37 pgs: 2 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 5635/220155 objects degraded (2.560%); 11595/220155 objects misplaced (5.267%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:46:59.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:46:59.716 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:46:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:46:59.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:46:59.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:46:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:46:59.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:46:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:46:59.978 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:46:59.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:00.080 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:00.112 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:00.214 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.245 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:00.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:00.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:00.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:00.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.515 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:00.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:00.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:00.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:00.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:00.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:00.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:00.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:00.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:00.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-08-30T23:47:01.002 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:01.005 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:01.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:01.027 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:01.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:01.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:01.177 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:01.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:01.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:01.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:01.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:01.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:01.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:01 smithi136 ceph-mon[20207]: pgmap v1505: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 5440/216195 objects degraded (2.516%); 11424/216195 objects misplaced (5.284%); 13 MiB/s, 215 objects/s recovering 2024-08-30T23:47:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5440/216195 objects degraded (2.516%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:01 smithi136 ceph-mon[20207]: osdmap e672: 12 total, 12 up, 11 in 2024-08-30T23:47:01.545 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:01.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:01 smithi173 ceph-mon[20554]: pgmap v1505: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 5440/216195 objects degraded (2.516%); 11424/216195 objects misplaced (5.284%); 13 MiB/s, 215 objects/s recovering 2024-08-30T23:47:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5440/216195 objects degraded (2.516%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:01 smithi173 ceph-mon[20554]: osdmap e672: 12 total, 12 up, 11 in 2024-08-30T23:47:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:01 smithi106 ceph-mon[16603]: pgmap v1505: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 5440/216195 objects degraded (2.516%); 11424/216195 objects misplaced (5.284%); 13 MiB/s, 215 objects/s recovering 2024-08-30T23:47:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5440/216195 objects degraded (2.516%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:01.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:01 smithi106 ceph-mon[16603]: osdmap e672: 12 total, 12 up, 11 in 2024-08-30T23:47:01.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:01.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.676 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:01.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:01.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:01.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:01.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:01.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:01.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:02.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:02.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:02.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:02.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:02.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:02.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:02 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:02 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:02.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:02 smithi106 ceph-mon[16603]: 6.a scrub starts 2024-08-30T23:47:02.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:02.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:02 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:02 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:02 smithi136 ceph-mon[20207]: 6.a scrub starts 2024-08-30T23:47:02.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:02.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:02.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:02.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:02 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:02.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:02 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:02 smithi173 ceph-mon[20554]: 6.a scrub starts 2024-08-30T23:47:02.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:02.701 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:02.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:02.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:02.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:02.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:02.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:03.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:03.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:03.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:03.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:03.242 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:47:03.243 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:47:03.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:47:03.285 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:47:03.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:03.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:03.344 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:47:03.344 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:47:03.387 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:47:03.388 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:47:03.425 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:47:03.425 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:47:03.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:03.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:03.463 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:47:03.463 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:47:03.497 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:47:03.497 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:47:03.498 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:03 smithi106 ceph-mon[16603]: pgmap v1507: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 5216/212970 objects degraded (2.449%); 11206/212970 objects misplaced (5.262%); 13 MiB/s, 212 objects/s recovering 2024-08-30T23:47:03.498 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3534260890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:03.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:03 smithi136 ceph-mon[20207]: pgmap v1507: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 5216/212970 objects degraded (2.449%); 11206/212970 objects misplaced (5.262%); 13 MiB/s, 212 objects/s recovering 2024-08-30T23:47:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3534260890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:03.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:03 smithi173 ceph-mon[20554]: pgmap v1507: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 701 op/s; 5216/212970 objects degraded (2.449%); 11206/212970 objects misplaced (5.262%); 13 MiB/s, 212 objects/s recovering 2024-08-30T23:47:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3534260890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:03.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:03.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:03.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:03.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:03.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:03.843 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:03.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:03.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:03.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:04.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:04.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:04.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:04.247 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:04.264 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:47:04.348 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:47:04.348 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:47:04.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:04.376 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.377 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:04.379 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:47:04.379 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:47:04.412 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:47:04.413 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:47:04.445 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:47:04.445 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:47:04.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:04.479 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:47:04.479 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:47:04.521 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:04.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:04 smithi136 ceph-mon[20207]: 6.a scrub ok 2024-08-30T23:47:04.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2995338294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:04.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:04 smithi173 ceph-mon[20554]: 6.a scrub ok 2024-08-30T23:47:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2995338294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:04.605 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:04 smithi106 ceph-mon[16603]: 6.a scrub ok 2024-08-30T23:47:04.605 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2995338294' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:04.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:04.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:04.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:04.776 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:04.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:04.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:04.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:05.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:05.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:05.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:05.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:05.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:05.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:05.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:05.428 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:05.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:05.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:05 smithi136 ceph-mon[20207]: pgmap v1508: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 5216/209901 objects degraded (2.485%); 10979/209901 objects misplaced (5.231%); 14 MiB/s, 216 objects/s recovering 2024-08-30T23:47:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:05 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:47:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2995338294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:05 smithi136 ceph-mon[20207]: osdmap e673: 12 total, 12 up, 11 in 2024-08-30T23:47:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:05.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:05 smithi173 ceph-mon[20554]: pgmap v1508: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 5216/209901 objects degraded (2.485%); 10979/209901 objects misplaced (5.231%); 14 MiB/s, 216 objects/s recovering 2024-08-30T23:47:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:05 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:47:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2995338294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:05 smithi173 ceph-mon[20554]: osdmap e673: 12 total, 12 up, 11 in 2024-08-30T23:47:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:05 smithi106 ceph-mon[16603]: pgmap v1508: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 697 op/s; 5216/209901 objects degraded (2.485%); 10979/209901 objects misplaced (5.231%); 14 MiB/s, 216 objects/s recovering 2024-08-30T23:47:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:05 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:47:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2995338294' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:05 smithi106 ceph-mon[16603]: osdmap e673: 12 total, 12 up, 11 in 2024-08-30T23:47:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:05.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:05.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:05.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:05.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:05.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:05.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:05.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:06.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:06.091 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:06.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:06.221 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:06.309 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:06.318 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:47:06.318 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:47:06.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:06.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.381 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:06.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:06.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:06.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:06 smithi136 ceph-mon[20207]: osdmap e674: 12 total, 12 up, 11 in 2024-08-30T23:47:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5216/209901 objects degraded (2.485%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:06 smithi173 ceph-mon[20554]: osdmap e674: 12 total, 12 up, 11 in 2024-08-30T23:47:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5216/209901 objects degraded (2.485%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:06 smithi106 ceph-mon[16603]: osdmap e674: 12 total, 12 up, 11 in 2024-08-30T23:47:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5216/209901 objects degraded (2.485%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:06.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:06.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:06.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:06.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:06.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:06.892 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:06.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:06.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:07.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.017 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:07.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:47:07.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:07.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:07.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:07.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:07.338 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:07 smithi106 ceph-mon[16603]: pgmap v1511: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 540 op/s; 5056/206469 objects degraded (2.449%); 10784/206469 objects misplaced (5.223%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:47:07.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:47:07.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:07.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:07.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:07.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:07.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:07 smithi136 ceph-mon[20207]: pgmap v1511: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 540 op/s; 5056/206469 objects degraded (2.449%); 10784/206469 objects misplaced (5.223%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:47:07.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:07 smithi173 ceph-mon[20554]: pgmap v1511: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 31 active+clean; 4.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 540 op/s; 5056/206469 objects degraded (2.449%); 10784/206469 objects misplaced (5.223%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:47:07.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:07.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:07.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:07.741 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:07.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:07.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:07.883 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:07.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:07.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:07.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:47:08.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:08.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:08.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:08.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:08.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:08.284 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:08.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:08.325 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:47:08.325 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:47:08.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:08.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:08.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:08.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:08.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:08.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:08.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:08.794 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:08.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:08.928 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:08.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:09.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:09.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:09.157 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:09.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:09.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:09.306 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:09.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:09.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:09.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:09.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:09 smithi136 ceph-mon[20207]: pgmap v1512: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 798 op/s; 4852/199890 objects degraded (2.427%); 10308/199890 objects misplaced (5.157%); 16 MiB/s, 250 objects/s recovering 2024-08-30T23:47:09.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:09.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:09 smithi173 ceph-mon[20554]: pgmap v1512: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 798 op/s; 4852/199890 objects degraded (2.427%); 10308/199890 objects misplaced (5.157%); 16 MiB/s, 250 objects/s recovering 2024-08-30T23:47:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:09 smithi106 ceph-mon[16603]: pgmap v1512: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 798 op/s; 4852/199890 objects degraded (2.427%); 10308/199890 objects misplaced (5.157%); 16 MiB/s, 250 objects/s recovering 2024-08-30T23:47:09.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:09.694 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:09.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:09.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:09.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:09.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:09.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:10.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:10.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:10.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:10.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:10.304 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:10.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:10.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:10.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:10.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:10.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:10.677 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:10.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:10.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:10.830 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:10.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:10.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:10.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:11.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:11.086 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:11.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:11.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:11.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:11.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.342 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:11.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:11.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:11.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:11 smithi136 ceph-mon[20207]: pgmap v1513: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 4665/195972 objects degraded (2.380%); 10130/195972 objects misplaced (5.169%); 13 MiB/s, 203 objects/s recovering 2024-08-30T23:47:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4665/195972 objects degraded (2.380%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:11.573 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:11 smithi173 ceph-mon[20554]: pgmap v1513: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 4665/195972 objects degraded (2.380%); 10130/195972 objects misplaced (5.169%); 13 MiB/s, 203 objects/s recovering 2024-08-30T23:47:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4665/195972 objects degraded (2.380%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:11.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:11 smithi106 ceph-mon[16603]: pgmap v1513: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 4.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 4665/195972 objects degraded (2.380%); 10130/195972 objects misplaced (5.169%); 13 MiB/s, 203 objects/s recovering 2024-08-30T23:47:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4665/195972 objects degraded (2.380%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:11.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:11.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:11.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:11.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:11.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:11.985 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:11.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:12.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:12.127 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:12.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:12.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:12.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:12.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:12.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:12.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:12.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:47:12.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:12.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:12.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:12.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:12.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:12.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:12.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:12.948 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:47:13.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:13.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-08-30T23:47:13.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:13.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:13.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:13.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:13.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:13.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:13.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:13.410 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:13 smithi136 ceph-mon[20207]: pgmap v1514: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 4473/192897 objects degraded (2.319%); 9846/192897 objects misplaced (5.104%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:47:13.423 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:13.442 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:13.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:13.526 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:47:13.527 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:47:13.549 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:47:13.549 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:47:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.562 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:13.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:13.563 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:47:13.564 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:47:13.564 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:13 smithi106 ceph-mon[16603]: pgmap v1514: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 4473/192897 objects degraded (2.319%); 9846/192897 objects misplaced (5.104%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:47:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:13 smithi173 ceph-mon[20554]: pgmap v1514: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 4473/192897 objects degraded (2.319%); 9846/192897 objects misplaced (5.104%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:47:13.599 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:47:13.599 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:47:13.637 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:47:13.637 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:47:13.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:13.673 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:47:13.674 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:47:13.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:13.700 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:47:13.701 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:47:13.730 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:47:13.730 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:47:13.762 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:13.767 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:47:13.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:47:13.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:13.809 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:47:13.809 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:47:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:13.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:13.886 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:47:13.887 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:47:13.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:47:13.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:47:13.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:13.962 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:47:13.963 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:47:13.981 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:13.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:14.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:14.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:14.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:14.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:14.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:14.366 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:14.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:14.498 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:14.510 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2998591844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:14.510 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:14 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:47:14.510 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:14 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:47:14.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:47:14.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2998591844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:14 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:47:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:14 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:47:14.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2998591844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:14 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:47:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:14 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:47:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:14.627 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:14.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:14.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:14.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:14.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:14.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:14.898 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:47:14.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:15.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:15.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:15.144 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:15.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.275 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:15.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:15.334 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:47:15.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:15.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:15.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:15.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:15.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:15 smithi106 ceph-mon[16603]: pgmap v1515: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 594 op/s; 4473/189726 objects degraded (2.358%); 9653/189726 objects misplaced (5.088%); 11 MiB/s, 182 objects/s recovering 2024-08-30T23:47:15.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3307091196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:47:15.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:15 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:47:15.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:15.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.664 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:15.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:15.778 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:15 smithi136 ceph-mon[20207]: pgmap v1515: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 594 op/s; 4473/189726 objects degraded (2.358%); 9653/189726 objects misplaced (5.088%); 11 MiB/s, 182 objects/s recovering 2024-08-30T23:47:15.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3307091196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:47:15.779 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:15 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:47:15.792 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:15 smithi173 ceph-mon[20554]: pgmap v1515: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 594 op/s; 4473/189726 objects degraded (2.358%); 9653/189726 objects misplaced (5.088%); 11 MiB/s, 182 objects/s recovering 2024-08-30T23:47:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3307091196' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:47:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:15 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:47:15.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:15.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:15.956 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:16.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.087 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:16.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:16.188 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:16.209 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:47:16.209 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:47:16.209 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:47:16.210 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:47:16.210 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:47:16.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:16.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:16.344 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:47:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: osdmap e675: 12 total, 12 up, 11 in 2024-08-30T23:47:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4473/189726 objects degraded (2.358%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: osdmap e676: 12 total, 12 up, 11 in 2024-08-30T23:47:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:16 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:16.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:16.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:16.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:16.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:16.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:16.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:16.734 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: osdmap e675: 12 total, 12 up, 11 in 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4473/189726 objects degraded (2.358%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: osdmap e676: 12 total, 12 up, 11 in 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:16.783 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:16 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: osdmap e675: 12 total, 12 up, 11 in 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4473/189726 objects degraded (2.358%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: osdmap e676: 12 total, 12 up, 11 in 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:16 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:16.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:16.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:16.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:16.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:47:16.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:17.001 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:17.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:17.141 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:17.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-08-30T23:47:17.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:17.263 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:47:17.278 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:17.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:17.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:17.406 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:17.423 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:17 smithi106 ceph-mon[16603]: pgmap v1517: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 688 op/s; 4341/185808 objects degraded (2.336%); 9424/185808 objects misplaced (5.072%); 13 MiB/s, 207 objects/s recovering 2024-08-30T23:47:17.423 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:17 smithi106 ceph-mon[16603]: osdmap e677: 12 total, 12 up, 11 in 2024-08-30T23:47:17.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:17.532 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:17.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:17.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:17.720 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:17 smithi136 ceph-mon[20207]: pgmap v1517: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 688 op/s; 4341/185808 objects degraded (2.336%); 9424/185808 objects misplaced (5.072%); 13 MiB/s, 207 objects/s recovering 2024-08-30T23:47:17.720 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:17 smithi136 ceph-mon[20207]: osdmap e677: 12 total, 12 up, 11 in 2024-08-30T23:47:17.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:17.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:17.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:17 smithi173 ceph-mon[20554]: pgmap v1517: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 688 op/s; 4341/185808 objects degraded (2.336%); 9424/185808 objects misplaced (5.072%); 13 MiB/s, 207 objects/s recovering 2024-08-30T23:47:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:17 smithi173 ceph-mon[20554]: osdmap e677: 12 total, 12 up, 11 in 2024-08-30T23:47:17.844 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:47:17.844 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:47:17.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:47:17.844 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:47:17.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.845 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.7 653/650 [PushOp(6:e8735af4:::benchmark_data_smithi173_114000_object5598:head, version: 624'324, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e8735af4:::benchmark_data_smithi173_114000_object5598:head@624'324, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.845 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:02.524306+0000", 2024-08-30T23:47:17.847 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 75.317731883999997, 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35381532799999998, 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524306+0000", 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524306+0000", 2024-08-30T23:47:17.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524308+0000", 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524353+0000", 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5572999999999999e-05 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875433+0000", 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35107914800000001 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875455+0000", 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2251000000000001e-05 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875559+0000", 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103646 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875575+0000", 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6943e-05 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.878122+0000", 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025463370000000001 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.4 655/650 [PushOp(6:28b12932:::benchmark_data_smithi173_114000_object6473:head, version: 624'432, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:28b12932:::benchmark_data_smithi173_114000_object6473:head@624'432, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:04.367318+0000", 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.474720493000007, 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.185390052, 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.853 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.367318+0000", 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.367318+0000", 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.367319+0000", 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3e-07 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.367373+0000", 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4264000000000001e-05 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.551102+0000", 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18372953 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.551114+0000", 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1456e-05 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.551183+0000", 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9195000000000004e-05 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.551191+0000", 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8650000000000001e-06 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.552708+0000", 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015170120000000001 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 655/654 [PushOp(6:f180e721:::benchmark_data_smithi173_114000_object484:head, version: 621'29, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f180e721:::benchmark_data_smithi173_114000_object484:head@621'29, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:04.555659+0000", 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.286379370000006, 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29762476500000001, 2024-08-30T23:47:17.857 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555659+0000", 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555659+0000", 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555661+0000", 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6670000000000001e-06 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555719+0000", 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8227999999999999e-05 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.849316+0000", 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29359667900000003 2024-08-30T23:47:17.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.849333+0000", 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7226000000000001e-05 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.852058+0000", 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027255059999999999 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.852067+0000", 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4780000000000008e-06 2024-08-30T23:47:17.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.853284+0000", 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001216981 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93681 6.f e655/654 6:faa4702b:::benchmark_data_smithi173_114000_object93680:head v 655'5802, mlcod=621'31)", 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.022099+0000", 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.819939563999995, 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.163216839, 2024-08-30T23:47:17.862 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.022099+0000", 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.022099+0000", 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.022099+0000", 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2600000000000002e-07 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.022196+0000", 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6427000000000004e-05 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.026472+0000", 2024-08-30T23:47:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042766130000000003 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.026478+0000", 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9050000000000002e-06 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.028851+0000", 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002372501 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.028874+0000", 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3292999999999999e-05 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.185294+0000", 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15641944299999999 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.185316+0000", 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2031e-05 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.866 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93683 6.f e655/654 6:ff6dc02e:::benchmark_data_smithi173_114000_object93682:head v 655'5803, mlcod=621'31)", 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.026412+0000", 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.815626148999996, 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183505627, 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.026412+0000", 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.026412+0000", 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.026413+0000", 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6200000000000006e-07 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.026453+0000", 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9659000000000001e-05 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.184060+0000", 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15760676800000001 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.184065+0000", 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8940000000000001e-06 2024-08-30T23:47:17.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.189479+0000", 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054142089999999997 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.189496+0000", 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7285000000000001e-05 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.209852+0000", 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020355860999999999 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.209918+0000", 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5988999999999994e-05 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93691 6.4 e655/650 6:2fdafc20:::benchmark_data_smithi173_114000_object93690:head v 655'5880, mlcod=624'430)", 2024-08-30T23:47:17.871 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.032353+0000", 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.809685006999999, 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15602527999999999, 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.032353+0000", 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.032353+0000", 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.032354+0000", 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8599999999999998e-07 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.032456+0000", 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102289 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.032471+0000", 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4426e-05 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.032477+0000", 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.891e-06 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.033867+0000", 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013906420000000001 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.033889+0000", 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1381000000000001e-05 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.188334+0000", 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15444566300000001 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.188379+0000", 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4301999999999997e-05 2024-08-30T23:47:17.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93693 6.7 e655/650 6:e935c630:::benchmark_data_smithi173_114000_object93692:head v 655'5762, mlcod=624'454)", 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.034158+0000", 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.807880838000003, 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.160136263, 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034158+0000", 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034158+0000", 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034159+0000", 2024-08-30T23:47:17.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1429999999999999e-06 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034203+0000", 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4515999999999999e-05 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034300+0000", 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6620000000000007e-05 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034310+0000", 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.047e-05 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034415+0000", 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104929 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034439+0000", 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3725000000000001e-05 2024-08-30T23:47:17.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.194254+0000", 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.159814821 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.194294+0000", 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0039000000000001e-05 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.7 655/650 [PushOp(6:e66924d3:::benchmark_data_smithi173_114000_object7622:head, version: 624'455, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e66924d3:::benchmark_data_smithi173_114000_object7622:head@624'455, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.034208+0000", 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.807830025000001, 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.151111194, 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034208+0000", 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034208+0000", 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034209+0000", 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5600000000000001e-07 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034251+0000", 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2236e-05 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034342+0000", 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1370000000000001e-05 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.034358+0000", 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5923e-05 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183732+0000", 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14937415100000001 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183748+0000", 2024-08-30T23:47:17.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5294e-05 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.185320+0000", 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015718640000000001 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93705 6.4 e655/650 6:2eea19b6:::benchmark_data_smithi173_114000_object93704:head v 655'5881, mlcod=624'430)", 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.037660+0000", 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.804378557999996, 2024-08-30T23:47:17.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15777217399999999, 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.037660+0000", 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.037660+0000", 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.037661+0000", 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0690000000000001e-06 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.037699+0000", 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8340000000000002e-05 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.037719+0000", 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9905999999999999e-05 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.037725+0000", 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.071e-06 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.039751+0000", 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020256670000000001 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183626+0000", 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14387472500000001 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.195382+0000", 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011756345 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.195432+0000", 2024-08-30T23:47:17.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0050999999999999e-05 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93709 6.4 e655/650 6:298451e5:::benchmark_data_smithi173_114000_object93708:head v 655'5882, mlcod=624'430)", 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.047438+0000", 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.794599876999996, 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14803080399999999, 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.890 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.047438+0000", 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.047438+0000", 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.047440+0000", 2024-08-30T23:47:17.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0100000000000001e-06 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.047492+0000", 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2142999999999999e-05 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183743+0000", 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13625157099999999 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183753+0000", 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0216e-05 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.188440+0000", 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046868259999999998 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.188469+0000", 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9036000000000001e-05 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.195438+0000", 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0069688459999999999 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.195469+0000", 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1155999999999998e-05 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93720 6.7 e655/650 6:e5085825:::benchmark_data_smithi173_114000_object93719:head v 655'5763, mlcod=624'454)", 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.056375+0000", 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.785662997000003, 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14341985199999999, 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.056375+0000", 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.056375+0000", 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.056376+0000", 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9099999999999991e-07 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.056452+0000", 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5154000000000006e-05 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183681+0000", 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12722969000000001 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.183694+0000", 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2337999999999999e-05 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.185274+0000", 2024-08-30T23:47:17.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015799659999999999 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.185295+0000", 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1807e-05 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.199739+0000", 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014443355 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.199795+0000", 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6551000000000001e-05 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:93759 6.f e655/654 6:fb742b04:::benchmark_data_smithi173_114000_object93758:head v 655'5807, mlcod=621'32)", 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:05.202304+0000", 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 72.639734258000004, 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.18264398200000001, 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.202304+0000", 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.202304+0000", 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.202305+0000", 2024-08-30T23:47:17.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0900000000000001e-07 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.202404+0000", 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8801999999999994e-05 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.378814+0000", 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.176410337 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.378823+0000", 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7669999999999992e-06 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.378903+0000", 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0476000000000007e-05 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.378943+0000", 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9839999999999998e-05 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.384916+0000", 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059725660000000003 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:05.384948+0000", 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2585000000000001e-05 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.903 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:17.905 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.905 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.4 656/650 [PushOp(6:21e02221:::benchmark_data_smithi173_114000_object8390:head, version: 625'541, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:21e02221:::benchmark_data_smithi173_114000_object8390:head@625'541, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.905 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:07.085670+0000", 2024-08-30T23:47:17.905 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.756368664000007, 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14682554, 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.085670+0000", 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.085670+0000", 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.085671+0000", 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.248e-06 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.085740+0000", 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9411000000000003e-05 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.230783+0000", 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.145042437 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.230795+0000", 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.230878+0000", 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3500999999999999e-05 2024-08-30T23:47:17.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.230887+0000", 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3429999999999996e-06 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:07.232495+0000", 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001608501 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.4 664/650 [PushOp(6:29718ccc:::benchmark_data_smithi173_114000_object22976:head, version: 629'1450, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:29718ccc:::benchmark_data_smithi173_114000_object22976:head@629'1450, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:28.982580+0000", 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.859458031000003, 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.130689103, 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:28.982580+0000", 2024-08-30T23:47:17.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:28.982580+0000", 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:28.982581+0000", 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0400000000000005e-07 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:28.982646+0000", 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5040000000000001e-05 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.111053+0000", 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12840709 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.111071+0000", 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7819e-05 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.111174+0000", 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102697 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.111188+0000", 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4504999999999999e-05 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.113269+0000", 2024-08-30T23:47:17.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020811480000000001 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 664/654 [PushOp(6:ffb0fea8:::benchmark_data_smithi173_114000_object13004:head, version: 626'799, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:ffb0fea8:::benchmark_data_smithi173_114000_object13004:head@626'799, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:29.520070+0000", 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.321968660000003, 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.183591749, 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.520070+0000", 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.520070+0000", 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.520071+0000", 2024-08-30T23:47:17.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.083e-06 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.520136+0000", 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5471999999999999e-05 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.701284+0000", 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.181148174 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.701295+0000", 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0794e-05 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.701384+0000", 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8481999999999998e-05 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.701397+0000", 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3393e-05 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:29.703661+0000", 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002264351 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:109292 6.7 e664/650 6:eb23d6a1:::benchmark_data_smithi173_114000_object9494:head v 664'6700, mlcod=629'1379)", 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:30.424514+0000", 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 47.417524194000002, 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.143258618, 2024-08-30T23:47:17.918 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.424514+0000", 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.424514+0000", 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.424516+0000", 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.325e-06 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.424524+0000", 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5140000000000001e-06 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.562325+0000", 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13780052300000001 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.562331+0000", 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.562388+0000", 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7398000000000002e-05 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.562407+0000", 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999999e-05 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.567724+0000", 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005316426 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.567773+0000", 2024-08-30T23:47:17.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8986999999999997e-05 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.7 664/650 [PushOp(6:eca2a086:::benchmark_data_smithi173_114000_object21786:head, version: 629'1381, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:eca2a086:::benchmark_data_smithi173_114000_object21786:head@629'1381, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:30.440867+0000", 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 47.401171359000003, 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13662960199999999, 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.440867+0000", 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.440867+0000", 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.440868+0000", 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.094e-06 2024-08-30T23:47:17.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.440927+0000", 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9354999999999997e-05 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.575117+0000", 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.134189685 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.575134+0000", 2024-08-30T23:47:17.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7167999999999999e-05 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.575241+0000", 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106266 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.575253+0000", 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2424e-05 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:30.577497+0000", 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00224361 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 670/654 [PushOp(6:fdb8253d:::benchmark_data_smithi173_114000_object25903:head, version: 630'1567, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:fdb8253d:::benchmark_data_smithi173_114000_object25903:head@630'1567, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:53.746237+0000", 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.095801715, 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12571206300000001, 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.927 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.746237+0000", 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.746237+0000", 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.746238+0000", 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.367e-06 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.746297+0000", 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8638999999999999e-05 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.869861+0000", 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12356419 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.869877+0000", 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6129000000000001e-05 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.869953+0000", 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5730999999999997e-05 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.869961+0000", 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7910000000000005e-06 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:53.871949+0000", 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019882160000000001 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.7 671/650 [PushOp(6:e7d2c9bc:::benchmark_data_smithi173_114000_object38848:head, version: 632'2388, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e7d2c9bc:::benchmark_data_smithi173_114000_object38848:head@632'2388, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:58.801800+0000", 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040238480999999, 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.150685607, 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.931 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.801800+0000", 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.801800+0000", 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.801801+0000", 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.801834+0000", 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3108000000000002e-05 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.950435+0000", 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14860121100000001 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.950446+0000", 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1163e-05 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.950505+0000", 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9182999999999999e-05 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.950514+0000", 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6130000000000004e-06 2024-08-30T23:47:17.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:58.952486+0000", 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019714609999999999 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:138605 6.f e674/654 6:f36aa5a8:::benchmark_data_smithi173_114000_object38807:head v 674'8508, mlcod=632'2342)", 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:13.941085+0000", 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 3.9009537029999999, 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.157301152, 2024-08-30T23:47:17.935 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:13.941085+0000", 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:13.941085+0000", 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:47:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:13.941086+0000", 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1090000000000001e-06 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:13.941096+0000", 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0001e-05 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:14.092816+0000", 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15171981800000001 2024-08-30T23:47:17.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:14.092825+0000", 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4860000000000003e-06 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:14.092889+0000", 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4333999999999998e-05 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:14.092919+0000", 2024-08-30T23:47:17.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9266999999999999e-05 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:14.098347+0000", 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054286780000000001 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:14.098386+0000", 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8458999999999999e-05 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:47:17.940 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:47:17.940 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:47:18.007 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:18.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:18.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:18.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:18.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:18.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:18.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:18.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:18.520 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:18.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:18.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:18.659 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:18.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:18.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:18.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:47:18.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:18.915 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:18.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:19.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:19.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:19.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:19.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:19.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:19.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:19.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:19.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:19.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:19.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.579 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:19.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:19 smithi106 ceph-mon[16603]: pgmap v1520: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 4068/179934 objects degraded (2.261%); 8846/179934 objects misplaced (4.916%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:47:19.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:19.710 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:19 smithi136 ceph-mon[20207]: pgmap v1520: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 4068/179934 objects degraded (2.261%); 8846/179934 objects misplaced (4.916%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:47:19.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:19 smithi173 ceph-mon[20554]: pgmap v1520: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 4068/179934 objects degraded (2.261%); 8846/179934 objects misplaced (4.916%); 15 MiB/s, 234 objects/s recovering 2024-08-30T23:47:19.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:19.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:19.938 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:19.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:47:19.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:19.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:20.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-08-30T23:47:20.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:20.084 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:20.175 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:20.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:20.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:20.358 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:20.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:20.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:47:20.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:20.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:20.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:20.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:20.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:20.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:20.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:20.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:20.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:20.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:20.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:20.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:20.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:21.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:21.036 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:21.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:21.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:21.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:21.290 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:21.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:21.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:21.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:21.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:21 smithi106 ceph-mon[16603]: pgmap v1521: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 3891/176277 objects degraded (2.207%); 8638/176277 objects misplaced (4.900%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:47:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3891/176277 objects degraded (2.207%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:21 smithi106 ceph-mon[16603]: osdmap e678: 12 total, 12 up, 11 in 2024-08-30T23:47:21.625 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:21 smithi173 ceph-mon[20554]: pgmap v1521: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 3891/176277 objects degraded (2.207%); 8638/176277 objects misplaced (4.900%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:47:21.625 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3891/176277 objects degraded (2.207%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:21.625 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:21 smithi173 ceph-mon[20554]: osdmap e678: 12 total, 12 up, 11 in 2024-08-30T23:47:21.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:21.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:21.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:21 smithi136 ceph-mon[20207]: pgmap v1521: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 746 op/s; 3891/176277 objects degraded (2.207%); 8638/176277 objects misplaced (4.900%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:47:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3891/176277 objects degraded (2.207%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:21 smithi136 ceph-mon[20207]: osdmap e678: 12 total, 12 up, 11 in 2024-08-30T23:47:21.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:21.901 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:21.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:21.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:22.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:22.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:22.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:22.182 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:22.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:22.320 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:22.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:22.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:22.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:22.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:22.638 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:47:22.638 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:47:22.638 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.638 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:143563 6.0 e678/619 6:0f1354d2:::benchmark_data_smithi173_114000_object43765:head v 678'9028, mlcod=678'9025)", 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:21.660784+0000", 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.016859546, 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016885196000000002, 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:47:22.639 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.660784+0000", 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.660784+0000", 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.660786+0000", 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9599999999999999e-06 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.660797+0000", 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0933e-05 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.676324+0000", 2024-08-30T23:47:22.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015527058999999999 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.676333+0000", 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1160000000000005e-06 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.676502+0000", 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016903799999999999 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:22.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:21.676575+0000", 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3367000000000004e-05 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-08-30T23:47:22.643 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:47:22.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:22.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:22.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:22.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:22.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:22.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:23.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:23.090 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:23.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:23.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:23.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:23.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:23.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:23.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:23.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:23.603 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:23 smithi136 ceph-mon[20207]: pgmap v1523: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 3656/172941 objects degraded (2.114%); 8365/172941 objects misplaced (4.837%); 18 MiB/s, 290 objects/s recovering 2024-08-30T23:47:23.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:23.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:23 smithi106 ceph-mon[16603]: pgmap v1523: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 3656/172941 objects degraded (2.114%); 8365/172941 objects misplaced (4.837%); 18 MiB/s, 290 objects/s recovering 2024-08-30T23:47:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:23.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:23.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:23.739 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:23.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:23 smithi173 ceph-mon[20554]: pgmap v1523: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.5 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 714 op/s; 3656/172941 objects degraded (2.114%); 8365/172941 objects misplaced (4.837%); 18 MiB/s, 290 objects/s recovering 2024-08-30T23:47:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:23.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.871 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:23.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:23.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:23.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:23.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:24.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:24.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:24.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:24.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:24.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:24.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:24.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:24.514 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:24.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:24.647 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:24.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:24.773 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:24.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:24 smithi136 ceph-mon[20207]: osdmap e679: 12 total, 12 up, 11 in 2024-08-30T23:47:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:24 smithi173 ceph-mon[20554]: osdmap e679: 12 total, 12 up, 11 in 2024-08-30T23:47:24.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:24 smithi106 ceph-mon[16603]: osdmap e679: 12 total, 12 up, 11 in 2024-08-30T23:47:24.875 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:24.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:24.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:25.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:25.022 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:25.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:25.149 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:25.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:25.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:25.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:25.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:25.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:25.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:25.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:25.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:25.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:25 smithi136 ceph-mon[20207]: pgmap v1525: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 3656/169953 objects degraded (2.151%); 8127/169953 objects misplaced (4.782%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:47:25.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:25.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:25 smithi173 ceph-mon[20554]: pgmap v1525: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 3656/169953 objects degraded (2.151%); 8127/169953 objects misplaced (4.782%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:47:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:25 smithi106 ceph-mon[16603]: pgmap v1525: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 3656/169953 objects degraded (2.151%); 8127/169953 objects misplaced (4.782%); 13 MiB/s, 208 objects/s recovering 2024-08-30T23:47:25.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:25.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:25.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:26.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:26.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:26.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:26.192 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:26.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:47:26.274 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:26.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.351 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:26.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:26.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:26.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:26 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:26.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:26 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:26.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:26 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:26.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:26 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:26.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3656/169953 objects degraded (2.151%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:26.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:26 smithi106 ceph-mon[16603]: osdmap e680: 12 total, 12 up, 11 in 2024-08-30T23:47:26.479 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:26.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:26.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:26.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:26.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:26.789 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:26 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:26 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:26 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:26 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3656/169953 objects degraded (2.151%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:26 smithi136 ceph-mon[20207]: osdmap e680: 12 total, 12 up, 11 in 2024-08-30T23:47:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:26 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:26 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:26 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:26 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3656/169953 objects degraded (2.151%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:26 smithi173 ceph-mon[20554]: osdmap e680: 12 total, 12 up, 11 in 2024-08-30T23:47:26.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:26.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:26.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:27.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.053 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:27.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:27.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.197 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:27.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:27.298 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.344 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:27.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:27.445 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:27.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:27.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:27.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:27 smithi106 ceph-mon[16603]: pgmap v1526: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 3431/166461 objects degraded (2.061%); 7922/166461 objects misplaced (4.759%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:47:27.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/698330386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:27.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:27.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:27.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:27.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:27.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:47:27.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:47:27.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:47:27.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:47:27.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:47:27.752 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:47:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:27 smithi136 ceph-mon[20207]: pgmap v1526: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 3431/166461 objects degraded (2.061%); 7922/166461 objects misplaced (4.759%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:47:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/698330386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:27 smithi173 ceph-mon[20554]: pgmap v1526: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 3431/166461 objects degraded (2.061%); 7922/166461 objects misplaced (4.759%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:47:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/698330386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:27.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:27.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:27.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:27.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:27.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:28.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:28.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:28.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:28.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:28.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:28.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:28.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:28.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:28.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:28.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:28.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:28.758 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:28.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:28.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:28.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:28.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:29.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:29.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:29.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:29.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:29.267 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:29.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:29.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:29.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:29 smithi106 ceph-mon[16603]: pgmap v1528: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 3020/160737 objects degraded (1.879%); 7322/160737 objects misplaced (4.555%); 20 MiB/s, 321 objects/s recovering 2024-08-30T23:47:29.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:29.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:29.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:29 smithi136 ceph-mon[20207]: pgmap v1528: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 3020/160737 objects degraded (1.879%); 7322/160737 objects misplaced (4.555%); 20 MiB/s, 321 objects/s recovering 2024-08-30T23:47:29.793 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:29.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:29 smithi173 ceph-mon[20554]: pgmap v1528: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 761 op/s; 3020/160737 objects degraded (1.879%); 7322/160737 objects misplaced (4.555%); 20 MiB/s, 321 objects/s recovering 2024-08-30T23:47:29.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:29.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:29.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:30.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:30.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:30.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:30.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:30.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:30.308 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:30.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:30.439 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:30.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:30.585 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:30.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:30.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:47:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:47:30.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:30.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:47:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:47:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:47:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:47:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:47:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:47:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:47:30.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:47:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:47:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:47:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:47:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:47:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:47:30.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:47:30.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:30.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:30.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:31.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:31.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:31.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:31.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:31.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:31.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:31.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:31.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:31.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:31.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:31.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:31.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:31 smithi136 ceph-mon[20207]: pgmap v1529: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 2763/157299 objects degraded (1.757%); 7101/157299 objects misplaced (4.514%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:47:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2763/157299 objects degraded (1.757%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:31 smithi173 ceph-mon[20554]: pgmap v1529: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 2763/157299 objects degraded (1.757%); 7101/157299 objects misplaced (4.514%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:47:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2763/157299 objects degraded (1.757%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:31.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:31 smithi106 ceph-mon[16603]: pgmap v1529: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 2763/157299 objects degraded (1.757%); 7101/157299 objects misplaced (4.514%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:47:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2763/157299 objects degraded (1.757%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:31.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:31.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:31.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:31.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:32.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:32.123 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:32.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:32.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:32.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:32.381 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:32.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-08-30T23:47:32.400 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:32.431 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:32.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.530 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:32.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:32.631 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:32.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:32.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:32.798 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:32.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:32.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:32.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:33.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:33.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:33.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:33.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:33.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:33.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:33.320 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:33.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:33.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:33.498 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:33 smithi106 ceph-mon[16603]: pgmap v1530: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2372/154023 objects degraded (1.540%); 6725/154023 objects misplaced (4.366%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:47:33.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:33.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:47:33.571 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:33.648 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:33 smithi136 ceph-mon[20207]: pgmap v1530: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2372/154023 objects degraded (1.540%); 6725/154023 objects misplaced (4.366%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:47:33.666 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:47:33.667 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:47:33.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:33.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:33.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:33.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:33 smithi173 ceph-mon[20554]: pgmap v1530: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 2372/154023 objects degraded (1.540%); 6725/154023 objects misplaced (4.366%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:47:33.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:33.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:33.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:33.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:34.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:34.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:34.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:34.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:34.316 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:34.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:34.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:34.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:34.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:34.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:34.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:34.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:47:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:47:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:34 smithi136 ceph-mon[20207]: osdmap e681: 12 total, 12 up, 11 in 2024-08-30T23:47:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:47:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:47:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:34 smithi173 ceph-mon[20554]: osdmap e681: 12 total, 12 up, 11 in 2024-08-30T23:47:34.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:34.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:47:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:47:34.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:34 smithi106 ceph-mon[16603]: osdmap e681: 12 total, 12 up, 11 in 2024-08-30T23:47:34.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:34.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:34.976 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:35.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:35.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:35.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:35.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:35.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:35.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:35.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:35.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:35.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:35.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:35.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:35.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:35.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:35 smithi136 ceph-mon[20207]: pgmap v1532: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 2372/150753 objects degraded (1.573%); 6507/150753 objects misplaced (4.316%); 19 MiB/s, 309 objects/s recovering 2024-08-30T23:47:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:35 smithi173 ceph-mon[20554]: pgmap v1532: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 2372/150753 objects degraded (1.573%); 6507/150753 objects misplaced (4.316%); 19 MiB/s, 309 objects/s recovering 2024-08-30T23:47:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:35 smithi106 ceph-mon[16603]: pgmap v1532: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 2372/150753 objects degraded (1.573%); 6507/150753 objects misplaced (4.316%); 19 MiB/s, 309 objects/s recovering 2024-08-30T23:47:35.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:35.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:35.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:36.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:36.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:36.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:36.165 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:36.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:36.302 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:36.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:36.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:36.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:36.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:36.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:36.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:47:36.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:36.670 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2372/150753 objects degraded (1.573%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:36.670 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:36 smithi106 ceph-mon[16603]: osdmap e682: 12 total, 12 up, 11 in 2024-08-30T23:47:36.679 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.710 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:36.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:36.739 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2372/150753 objects degraded (1.573%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:36 smithi136 ceph-mon[20207]: osdmap e682: 12 total, 12 up, 11 in 2024-08-30T23:47:36.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2372/150753 objects degraded (1.573%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:36 smithi173 ceph-mon[20554]: osdmap e682: 12 total, 12 up, 11 in 2024-08-30T23:47:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:36.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:36.938 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:36.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:36.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:37.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:37.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:37.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:37.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:37.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.399 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:37.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:37.420 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:47:37.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:37.525 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:37 smithi106 ceph-mon[16603]: pgmap v1533: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 2096/146775 objects degraded (1.428%); 6310/146775 objects misplaced (4.299%); 19 MiB/s, 300 objects/s recovering 2024-08-30T23:47:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1066980863' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:37.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:37.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:37.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-30T23:47:37.690+0000 7f210c9f8640 0 --2- 172.21.15.106:0/631704778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2108102960 0x7f2108198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-30T23:47:37.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:37.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:37 smithi136 ceph-mon[20207]: pgmap v1533: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 2096/146775 objects degraded (1.428%); 6310/146775 objects misplaced (4.299%); 19 MiB/s, 300 objects/s recovering 2024-08-30T23:47:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1066980863' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:37 smithi173 ceph-mon[20554]: pgmap v1533: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 3.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 669 op/s; 2096/146775 objects degraded (1.428%); 6310/146775 objects misplaced (4.299%); 19 MiB/s, 300 objects/s recovering 2024-08-30T23:47:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1066980863' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:37.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:37.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:37.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:38.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:38.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.049 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:38.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:38.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:38.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:38.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:38.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:38.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:38.459 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:38.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:38.594 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:38.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:38.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:38.766 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:38 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:38.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1066980863' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:38.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:38 smithi106 ceph-mon[16603]: osdmap e683: 12 total, 12 up, 11 in 2024-08-30T23:47:38.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1938290276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:38 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1066980863' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:38 smithi136 ceph-mon[20207]: osdmap e683: 12 total, 12 up, 11 in 2024-08-30T23:47:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1938290276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:38.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:38 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1066980863' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:38 smithi173 ceph-mon[20554]: osdmap e683: 12 total, 12 up, 11 in 2024-08-30T23:47:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1938290276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:38.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:38.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:38.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:38.994 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:38.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:39.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:39.109 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:47:39.109 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:47:39.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:39.142 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:47:39.142 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:47:39.166 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:47:39.167 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:47:39.206 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:47:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:47:39.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:39.240 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:47:39.241 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:47:39.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:39.276 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:47:39.276 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:47:39.309 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:39.314 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:47:39.314 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:47:39.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-08-30T23:47:39.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:39.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:39.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:39.503 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:39.551 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:47:39.551 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:47:39.577 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:47:39.577 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:47:39.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:39.612 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:47:39.612 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:47:39.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:39.642 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:47:39.642 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:47:39.670 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:47:39.671 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:47:39.686 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:39 smithi106 ceph-mon[16603]: pgmap v1536: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 763 op/s; 1661/140277 objects degraded (1.184%); 5825/140277 objects misplaced (4.152%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:47:39.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:39.762 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:39.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:39 smithi136 ceph-mon[20207]: pgmap v1536: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 763 op/s; 1661/140277 objects degraded (1.184%); 5825/140277 objects misplaced (4.152%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:47:39.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:39 smithi173 ceph-mon[20554]: pgmap v1536: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 763 op/s; 1661/140277 objects degraded (1.184%); 5825/140277 objects misplaced (4.152%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:47:39.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:39.875 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:47:39.875 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:80231 5.2 e439/420 5:4ddaa76e:::benchmark_data_smithi173_87639_object80230:head v 439'4926, mlcod=415'2985)", 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:11.991271+0000", 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 507.88347509200003, 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64341151600000002, 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991271+0000", 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991271+0000", 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991272+0000", 2024-08-30T23:47:39.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9699999999999996e-07 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:11.991419+0000", 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014760399999999999 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632382+0000", 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64096311900000003 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632392+0000", 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1400000000000006e-06 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632443+0000", 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1415000000000002e-05 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.632464+0000", 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0752999999999999e-05 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.634636+0000", 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021725220000000001 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:12.634683+0000", 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6366000000000003e-05 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.880 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:c8aed71c:::benchmark_data_smithi173_87639_object26550:head)])", 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 465.56356710099999, 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94060026299999999, 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311180+0000", 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5199999999999996e-07 2024-08-30T23:47:39.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311185+0000", 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8019999999999999e-06 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.249924+0000", 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93873898200000006 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.249936+0000", 2024-08-30T23:47:39.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2034e-05 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.250051+0000", 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115751 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.251779+0000", 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017280419999999999 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:cdfa4373:::benchmark_data_smithi173_87639_object26587:head)])", 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 464.58556644700002, 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64991660600000001, 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:47:39.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2300000000000001e-07 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289185+0000", 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2810000000000002e-06 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937334+0000", 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64814879999999997 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937343+0000", 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6199999999999994e-06 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937437+0000", 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3407999999999997e-05 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.939096+0000", 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001659874 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:c1c728e1:::benchmark_data_smithi173_87639_object28064:head)])", 2024-08-30T23:47:39.888 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 462.58336983700002, 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58924576699999998, 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291377+0000", 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8200000000000001e-07 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291379+0000", 2024-08-30T23:47:39.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.17e-06 2024-08-30T23:47:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.878910+0000", 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58753071199999995 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.878927+0000", 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7733000000000001e-05 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.879062+0000", 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013429699999999999 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.880622+0000", 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015604729999999999 2024-08-30T23:47:39.893 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:c3b8f5dc:::benchmark_data_smithi173_87639_object28564:head)])", 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 460.75307219899997, 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82014575300000003, 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8200000000000001e-07 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121677+0000", 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.807e-06 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939832+0000", 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81815479400000002 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939842+0000", 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9399999999999997e-06 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939959+0000", 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117068 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.941820+0000", 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001860762 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:ca3229d2:::benchmark_data_smithi173_87639_object28892:head)])", 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 459.69509221999999, 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62786811200000003, 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.898 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.35e-07 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179657+0000", 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4949999999999998e-06 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.805946+0000", 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62628936400000002 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.805956+0000", 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7599999999999997e-06 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.806029+0000", 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3069000000000001e-05 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.807522+0000", 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001493189 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 462/432 [PushReplyOp(5:c69c4cd1:::benchmark_data_smithi173_87639_object33999:head)])", 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 453.581068571, 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71283231999999996, 2024-08-30T23:47:39.902 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1300000000000001e-07 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293681+0000", 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6290000000000002e-06 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004558+0000", 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71087762700000001 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004576+0000", 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7363999999999998e-05 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004759+0000", 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018305599999999999 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.006510+0000", 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017513310000000001 2024-08-30T23:47:39.905 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 462/432 [PushReplyOp(5:c1f40d66:::benchmark_data_smithi173_87639_object34858:head)])", 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:08.049946+0000", 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 451.82479992899999, 2024-08-30T23:47:39.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72198392099999997, 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049946+0000", 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049946+0000", 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049947+0000", 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7299999999999996e-07 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049951+0000", 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7280000000000001e-06 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.770680+0000", 2024-08-30T23:47:39.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72072939499999999 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.770689+0000", 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8969999999999997e-06 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.770824+0000", 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013500599999999999 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.771930+0000", 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011063220000000001 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 464/432 [PushReplyOp(5:ccd525d4:::benchmark_data_smithi173_87639_object38189:head)])", 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:11.781189+0000", 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 448.093556761, 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70230110000000001, 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781189+0000", 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781189+0000", 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781190+0000", 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5499999999999998e-07 2024-08-30T23:47:39.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781193+0000", 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5410000000000001e-06 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.481528+0000", 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70033474699999998 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.481544+0000", 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.481728+0000", 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018372000000000001 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.483491+0000", 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017624050000000001 2024-08-30T23:47:39.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 469/432 [PushReplyOp(5:c90b87d7:::benchmark_data_smithi173_87639_object53091:head)])", 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 434.558135359, 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.59203615200000004, 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.58e-07 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316614+0000", 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0469999999999998e-06 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.907920+0000", 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.59130516200000005 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.907927+0000", 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.469e-06 2024-08-30T23:47:39.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.907990+0000", 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3126999999999994e-05 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.908647+0000", 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065688900000000002 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 470/432 [PushReplyOp(5:c2ca99af:::benchmark_data_smithi173_87639_object62935:head)])", 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 427.35103948, 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56332729199999998, 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4000000000000002e-07 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523711+0000", 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6780000000000002e-06 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.086521+0000", 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56280982599999996 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.086569+0000", 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8012e-05 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.086674+0000", 2024-08-30T23:47:39.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105009 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.087034+0000", 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036022700000000002 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.18 503/491 [PushReplyOp(5:1eedc20c:::benchmark_data_smithi173_87639_object10777:head)])", 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:33.998519+0000", 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 365.87622708499998, 2024-08-30T23:47:39.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.602415064, 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998519+0000", 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998519+0000", 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998520+0000", 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4499999999999997e-07 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998523+0000", 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7290000000000002e-06 2024-08-30T23:47:39.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.598369+0000", 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.59984580700000001 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.598377+0000", 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4919999999999996e-06 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.598450+0000", 2024-08-30T23:47:39.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2991999999999995e-05 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.600934+0000", 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024845990000000001 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.12 507/491 [PushReplyOp(5:483c8803:::benchmark_data_smithi173_87639_object15672:head)])", 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 361.630599727, 2024-08-30T23:47:39.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54423049400000001, 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244152+0000", 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0019999999999997e-06 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.786452+0000", 2024-08-30T23:47:39.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54229973399999998 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.786459+0000", 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3730000000000004e-06 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.786530+0000", 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1051999999999997e-05 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.788377+0000", 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018466159999999999 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.18 512/491 [PushReplyOp(5:1f0375e1:::benchmark_data_smithi173_87639_object35727:head)])", 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:46.007806+0000", 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 353.86693981299999, 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57058524300000002, 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.929 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007806+0000", 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007806+0000", 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007807+0000", 2024-08-30T23:47:39.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1900000000000002e-07 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007812+0000", 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9699999999999998e-06 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.576940+0000", 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56912784500000002 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.576947+0000", 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6969999999999996e-06 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.577003+0000", 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5945000000000001e-05 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.578392+0000", 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013890669999999999 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.12 518/491 [PushReplyOp(5:4b63bb01:::benchmark_data_smithi173_87639_object67919:head)])", 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:01.855351+0000", 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 338.01939497199999, 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80715373199999996, 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855351+0000", 2024-08-30T23:47:39.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855351+0000", 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855352+0000", 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999998e-07 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855355+0000", 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.112e-06 2024-08-30T23:47:39.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662036+0000", 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80668152599999998 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662044+0000", 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9400000000000002e-06 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662110+0000", 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6015999999999999e-05 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662505+0000", 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039469499999999997 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:109334 5.2 e552/488 5:41d12184:::benchmark_data_smithi173_87639_object29094:head v 552'6006, mlcod=551'6003)", 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:00.876275+0000", 2024-08-30T23:47:39.937 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 278.998471172, 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64198491599999996, 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876275+0000", 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876275+0000", 2024-08-30T23:47:39.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876277+0000", 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4610000000000001e-06 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876287+0000", 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0049e-05 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516376+0000", 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64008948300000001 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516383+0000", 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3540000000000002e-06 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516442+0000", 2024-08-30T23:47:39.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8616000000000003e-05 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516468+0000", 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5652000000000001e-05 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.518218+0000", 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017507009999999999 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.518260+0000", 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1600000000000002e-05 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:40058 6.0 e633/619 6:0cd7c34c:::benchmark_data_smithi173_114000_object40057:head v 633'2529, mlcod=633'2526)", 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:13.506735+0000", 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 146.36801116800001, 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15172578, 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.942 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.506735+0000", 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.506735+0000", 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.506736+0000", 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9500000000000001e-07 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.506780+0000", 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4261999999999997e-05 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.656949+0000", 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15016887900000001 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.656957+0000", 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5460000000000003e-06 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.657032+0000", 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5462000000000004e-05 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.657053+0000", 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0477999999999999e-05 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.658420+0000", 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013674679999999999 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:13.658461+0000", 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0790000000000001e-05 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.946 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:93442 6.a e653/619 6:562d5a8b:::benchmark_data_smithi173_114000_object93441:head v 653'5805, mlcod=652'5801)", 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 97.974213621999994, 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.287759343, 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6100000000000002e-07 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900581+0000", 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8588000000000003e-05 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185793+0000", 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28521170000000001 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185802+0000", 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3489999999999992e-06 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185856+0000", 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4540000000000003e-05 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185889+0000", 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2789000000000001e-05 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.188257+0000", 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002368448 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.188292+0000", 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4668000000000001e-05 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:93778 6.a e655/619 6:5b570e40:::benchmark_data_smithi173_114000_object93777:head v 655'5833, mlcod=655'5830)", 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:05.220365+0000", 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 94.654381435999994, 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.16863280899999999, 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.951 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:39.952 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.220365+0000", 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.220365+0000", 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.220366+0000", 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4759999999999999e-06 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.220399+0000", 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2570000000000002e-05 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.385671+0000", 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.165271895 2024-08-30T23:47:39.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.385681+0000", 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0068000000000001e-05 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.385753+0000", 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1829999999999995e-05 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.385775+0000", 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2785999999999998e-05 2024-08-30T23:47:39.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.388857+0000", 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003081777 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:05.388998+0000", 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140407 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:114039 6.0 e664/619 6:0dfb5516:::benchmark_data_smithi173_114000_object14241:head v 664'7136, mlcod=664'7131)", 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:37.847734+0000", 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 62.027012096, 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.201161276, 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:37.847734+0000", 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:37.847734+0000", 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:37.847735+0000", 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7199999999999997e-07 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:37.847742+0000", 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2049999999999996e-06 2024-08-30T23:47:39.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:38.045130+0000", 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.197387215 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:38.045138+0000", 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8389999999999996e-06 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:38.045200+0000", 2024-08-30T23:47:39.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1211999999999996e-05 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:38.045224+0000", 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4020999999999999e-05 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:38.048679+0000", 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034558409999999999 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:47:39.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:38.048895+0000", 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000215971 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:39.961 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:47:40.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:40.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.075 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:40.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:40.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:40.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:40.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:40.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:40.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:40.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:40.587 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:40.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:40.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.721 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:40.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:40 smithi136 ceph-mon[20207]: osdmap e684: 12 total, 12 up, 11 in 2024-08-30T23:47:40.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:40 smithi173 ceph-mon[20554]: osdmap e684: 12 total, 12 up, 11 in 2024-08-30T23:47:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:40.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:40 smithi106 ceph-mon[16603]: osdmap e684: 12 total, 12 up, 11 in 2024-08-30T23:47:40.957 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:40.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:40.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:41.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:41.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:41.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:47:41.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:41.306 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:41.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:41.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:41.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:41.564 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.565 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:41 smithi106 ceph-mon[16603]: pgmap v1538: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 1446/136305 objects degraded (1.061%); 5612/136305 objects misplaced (4.117%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:47:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:41 smithi106 ceph-mon[16603]: osdmap e685: 12 total, 12 up, 11 in 2024-08-30T23:47:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1446/136305 objects degraded (1.061%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:41 smithi106 ceph-mon[16603]: osdmap e686: 12 total, 12 up, 11 in 2024-08-30T23:47:41.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:41.694 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:41 smithi136 ceph-mon[20207]: pgmap v1538: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 1446/136305 objects degraded (1.061%); 5612/136305 objects misplaced (4.117%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:47:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:41 smithi136 ceph-mon[20207]: osdmap e685: 12 total, 12 up, 11 in 2024-08-30T23:47:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1446/136305 objects degraded (1.061%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:41 smithi136 ceph-mon[20207]: osdmap e686: 12 total, 12 up, 11 in 2024-08-30T23:47:41.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:41.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:41 smithi173 ceph-mon[20554]: pgmap v1538: 37 pgs: 1 active+recovering+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.8 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 802 op/s; 1446/136305 objects degraded (1.061%); 5612/136305 objects misplaced (4.117%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:47:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:41 smithi173 ceph-mon[20554]: osdmap e685: 12 total, 12 up, 11 in 2024-08-30T23:47:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1446/136305 objects degraded (1.061%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:41 smithi173 ceph-mon[20554]: osdmap e686: 12 total, 12 up, 11 in 2024-08-30T23:47:41.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:41.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:41.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:42.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:42.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:42.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:42.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:42.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:42.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:42.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:42.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:42.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:42.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:42.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:47:42.678 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:42.679 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:42.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:42.738 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:42.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:42.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:42.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:42.985 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:42.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:43.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:43.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:43.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.246 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:43.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:43.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:43.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:43.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:43.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:43.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:43 smithi106 ceph-mon[16603]: pgmap v1541: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 603 op/s; 1217/133275 objects degraded (0.913%); 5292/133275 objects misplaced (3.971%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:47:43.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:43.630 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:43.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:43.755 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:43 smithi136 ceph-mon[20207]: pgmap v1541: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 603 op/s; 1217/133275 objects degraded (0.913%); 5292/133275 objects misplaced (3.971%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:47:43.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:43 smithi173 ceph-mon[20554]: pgmap v1541: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 603 op/s; 1217/133275 objects degraded (0.913%); 5292/133275 objects misplaced (3.971%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:47:43.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:43.883 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:43.884 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:43.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:47:43.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:44.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:44.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:44.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:44.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:44.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:44.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:44.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:44.388 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:44.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-08-30T23:47:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:47:44.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:44.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:44.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:44.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:44.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:44.752 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3972416722' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:47:44.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3972416722' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:47:44.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3972416722' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:47:44.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:44.939 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:44.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:45.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:45.078 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:45.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:45.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:45.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:45.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:45.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:45.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:45.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:45.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:45.589 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:45.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:45.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:45 smithi136 ceph-mon[20207]: pgmap v1542: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 1217/130185 objects degraded (0.935%); 5044/130185 objects misplaced (3.874%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:47:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:45 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3972416722' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:47:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:45 smithi136 ceph-mon[20207]: osdmap e687: 12 total, 12 up, 11 in 2024-08-30T23:47:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:45 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:47:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:45 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:47:45.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:45 smithi173 ceph-mon[20554]: pgmap v1542: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 1217/130185 objects degraded (0.935%); 5044/130185 objects misplaced (3.874%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:47:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:45 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3972416722' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:47:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:45 smithi173 ceph-mon[20554]: osdmap e687: 12 total, 12 up, 11 in 2024-08-30T23:47:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:45 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:47:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:45 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:47:45.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:45 smithi106 ceph-mon[16603]: pgmap v1542: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 560 op/s; 1217/130185 objects degraded (0.935%); 5044/130185 objects misplaced (3.874%); 13 MiB/s, 204 objects/s recovering 2024-08-30T23:47:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:45 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:47:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3972416722' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:47:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:45 smithi106 ceph-mon[16603]: osdmap e687: 12 total, 12 up, 11 in 2024-08-30T23:47:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:45 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:47:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:45 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:47:45.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:45.984 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:45.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:46.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:46.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:46.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:46.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:46.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:46.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:46.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:46.363 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:46.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:46.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:46.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:46.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:46.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:46.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:46.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:159012 6.e e687/619 6:7989acdd:::benchmark_data_smithi173_114000_object59214:head v 687'10112, mlcod=687'10110)", 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.059009+0000", 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00028903799999999998, 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031430999999999999, 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.696 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059009+0000", 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059009+0000", 2024-08-30T23:47:46.697 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059010+0000", 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.235e-06 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059018+0000", 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:47:46.698 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059053+0000", 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5451999999999999e-05 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059061+0000", 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5950000000000001e-06 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.699 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059111+0000", 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9861000000000003e-05 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.059130+0000", 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9392e-05 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.700 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:158998 6.f 6:fd5d2d7f:::benchmark_data_smithi173_114000_object59200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.032027+0000", 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.027270577000000001, 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027355186, 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:46.701 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 158998 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.032027+0000", 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.702 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.032027+0000", 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.032029+0000", 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.528e-06 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.032035+0000", 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5529999999999998e-06 2024-08-30T23:47:46.703 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.032049+0000", 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3797e-05 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.032057+0000", 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6249999999999998e-06 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.704 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.035152+0000", 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030955610000000001 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.035416+0000", 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026356300000000002 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.705 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.048328+0000", 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012911917 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.048559+0000", 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00023125500000000001 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.049003+0000", 2024-08-30T23:47:46.706 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000443862 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:159001 6.f 6:fef9ed57:::benchmark_data_smithi173_114000_object59203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.034052+0000", 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.025246267999999999, 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025403802999999999, 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.707 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 159001 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.708 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.034052+0000", 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.034052+0000", 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.709 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.034053+0000", 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.46e-06 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.034059+0000", 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9529999999999996e-06 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.710 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.034070+0000", 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0815000000000001e-05 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.034077+0000", 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0990000000000001e-06 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.037908+0000", 2024-08-30T23:47:46.711 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038307580000000001 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.038071+0000", 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016353400000000001 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.038221+0000", 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014929599999999999 2024-08-30T23:47:46.712 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.044318+0000", 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060974549999999999 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:159007 6.5 e687/619 6:a38c78c0:::benchmark_data_smithi173_114000_object59209:head v 687'9985, mlcod=687'9983)", 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.050494+0000", 2024-08-30T23:47:46.713 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0088044260000000006, 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090283399999999993, 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050494+0000", 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050494+0000", 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050495+0000", 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0580000000000001e-06 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050501+0000", 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8759999999999999e-06 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050521+0000", 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9613e-05 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050529+0000", 2024-08-30T23:47:46.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8800000000000008e-06 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050644+0000", 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114795 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.050673+0000", 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8750000000000001e-05 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.717 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27034.0:159001 6.f e687/654)", 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.042728+0000", 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.016570196999999998, 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016839130000000001, 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:47:46.718 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.042728+0000", 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.042728+0000", 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.042728+0000", 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.44e-07 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.042733+0000", 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3839999999999999e-06 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055312+0000", 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012579664000000001 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055323+0000", 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0322e-05 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.721 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27034.0:158998 6.f e687/654)", 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.055508+0000", 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0037895709999999998, 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040945570000000004, 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055508+0000", 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055508+0000", 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055509+0000", 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3399999999999998e-07 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055515+0000", 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9880000000000001e-06 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055534+0000", 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8423e-05 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055541+0000", 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6690000000000006e-06 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(6.d 687/654 [PushOp(6:bf5d5d7c:::benchmark_data_smithi173_114000_object71230:head, version: 643'4308, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:bf5d5d7c:::benchmark_data_smithi173_114000_object71230:head@643'4308, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.036314+0000", 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.022984261999999998, 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.023353822999999999, 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.725 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.036314+0000", 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.036314+0000", 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.036314+0000", 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3300000000000002e-07 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.036352+0000", 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7728999999999997e-05 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055703+0000", 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019351229000000001 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055721+0000", 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7810999999999999e-05 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.056965+0000", 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001244069 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.056979+0000", 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4192999999999999e-05 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.729 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:159010 6.5 e687/619 6:a7e23c0b:::benchmark_data_smithi173_114000_object59212:head v 687'9986, mlcod=687'9983)", 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:45.055768+0000", 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0035296429999999998, 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0039509899999999997, 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055768+0000", 2024-08-30T23:47:46.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055768+0000", 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055769+0000", 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0699999999999997e-07 2024-08-30T23:47:46.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055774+0000", 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7899999999999999e-06 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055787+0000", 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3292000000000001e-05 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:46.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055793+0000", 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6269999999999997e-06 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055894+0000", 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101321 2024-08-30T23:47:46.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:45.055917+0000", 2024-08-30T23:47:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2912e-05 2024-08-30T23:47:46.736 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:47:46.737 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 8 2024-08-30T23:47:46.737 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:47:46.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:46.762 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:46.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1217/130185 objects degraded (0.935%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:46 smithi136 ceph-mon[20207]: osdmap e688: 12 total, 12 up, 11 in 2024-08-30T23:47:46.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1217/130185 objects degraded (0.935%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:46 smithi173 ceph-mon[20554]: osdmap e688: 12 total, 12 up, 11 in 2024-08-30T23:47:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1217/130185 objects degraded (0.935%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:46 smithi106 ceph-mon[16603]: osdmap e688: 12 total, 12 up, 11 in 2024-08-30T23:47:46.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:46.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:47.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:47.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:47.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:47.266 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:47.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:47.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:47.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:47.530 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:47.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:47 smithi106 ceph-mon[16603]: pgmap v1544: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 1025/126783 objects degraded (0.808%); 4816/126783 objects misplaced (3.799%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:47:47.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:47 smithi106 ceph-mon[16603]: 6.3 scrub starts 2024-08-30T23:47:47.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:47.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:47.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:47.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:47 smithi136 ceph-mon[20207]: pgmap v1544: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 1025/126783 objects degraded (0.808%); 4816/126783 objects misplaced (3.799%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:47:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:47 smithi136 ceph-mon[20207]: 6.3 scrub starts 2024-08-30T23:47:47.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:47 smithi173 ceph-mon[20554]: pgmap v1544: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 1025/126783 objects degraded (0.808%); 4816/126783 objects misplaced (3.799%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:47:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:47 smithi173 ceph-mon[20554]: 6.3 scrub starts 2024-08-30T23:47:47.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:47.907 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:47.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:48.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:48.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:48.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:48.163 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:48.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:48.302 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:48.404 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:48.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:48.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:48.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:48.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:48.678 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:48.709 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:48.780 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:48 smithi136 ceph-mon[20207]: 6.3 scrub ok 2024-08-30T23:47:48.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:48 smithi173 ceph-mon[20554]: 6.3 scrub ok 2024-08-30T23:47:48.843 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:48 smithi106 ceph-mon[16603]: 6.3 scrub ok 2024-08-30T23:47:48.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:48.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:48.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:49.038 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:49.066 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:49.168 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:49.201 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:49.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:47:49.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.383 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:49.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:49.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:49.514 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:49.548 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:49 smithi106 ceph-mon[16603]: pgmap v1546: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 1025/120363 objects degraded (0.852%); 4192/120363 objects misplaced (3.483%); 17 MiB/s, 270 objects/s recovering 2024-08-30T23:47:49.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.649 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:49.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:49.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:49.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:49.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:49 smithi136 ceph-mon[20207]: pgmap v1546: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 1025/120363 objects degraded (0.852%); 4192/120363 objects misplaced (3.483%); 17 MiB/s, 270 objects/s recovering 2024-08-30T23:47:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:49 smithi173 ceph-mon[20554]: pgmap v1546: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 1025/120363 objects degraded (0.852%); 4192/120363 objects misplaced (3.483%); 17 MiB/s, 270 objects/s recovering 2024-08-30T23:47:49.855 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:47:49.855 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:47:49.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:49.893 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:47:49.893 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:47:49.933 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:49.933 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:47:49.934 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:47:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:49.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:49.971 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:49.972 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:47:49.973 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:47:49.986 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:50.013 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:47:50.013 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:47:50.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:50.050 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:47:50.050 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:47:50.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:50.090 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:47:50.090 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:47:50.123 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:47:50.123 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:47:50.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:50.163 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:47:50.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:47:50.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:50.201 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:47:50.201 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:47:50.234 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:47:50.234 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:47:50.271 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:47:50.271 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:47:50.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:50.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:50.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:50.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:50.550 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:50.581 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:50.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:50.717 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:50.728 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2490322850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:50.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2490322850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:50.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:50.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2490322850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:50.862 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:50.944 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:47:50.945 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:47:50.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:50.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:50.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:51.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:51.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:51.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:51.259 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:51.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:51.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:51.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:51.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:51.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:51.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:51.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:51 smithi173 ceph-mon[20554]: pgmap v1547: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 741/116388 objects degraded (0.637%); 3954/116388 objects misplaced (3.397%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:47:51.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 741/116388 objects degraded (0.637%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:51.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:51.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:51 smithi136 ceph-mon[20207]: pgmap v1547: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 741/116388 objects degraded (0.637%); 3954/116388 objects misplaced (3.397%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:47:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 741/116388 objects degraded (0.637%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:51.796 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:51 smithi106 ceph-mon[16603]: pgmap v1547: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.4 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 703 op/s; 741/116388 objects degraded (0.637%); 3954/116388 objects misplaced (3.397%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:47:51.797 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 741/116388 objects degraded (0.637%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:51.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:51.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:51.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:51.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:47:52.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:52.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:52.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:52.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:52.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:52.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:52.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:52.403 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:47:52.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.426 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:52.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:52.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:52.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:52.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:52.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:52.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:52.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:52.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:52.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:53.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:53.086 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:53.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:53.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:53.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.367 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:53.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:53.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:53.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:53.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:53.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:53.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:53.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:53 smithi136 ceph-mon[20207]: pgmap v1548: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 741/113205 objects degraded (0.655%); 3561/113205 objects misplaced (3.146%); 15 MiB/s, 244 objects/s recovering 2024-08-30T23:47:53.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:53 smithi173 ceph-mon[20554]: pgmap v1548: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 741/113205 objects degraded (0.655%); 3561/113205 objects misplaced (3.146%); 15 MiB/s, 244 objects/s recovering 2024-08-30T23:47:53.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:53.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:53 smithi106 ceph-mon[16603]: pgmap v1548: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 741/113205 objects degraded (0.655%); 3561/113205 objects misplaced (3.146%); 15 MiB/s, 244 objects/s recovering 2024-08-30T23:47:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:53.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:53.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:53.893 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:53.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:54.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:54.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:54.165 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:54.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:54.293 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:54.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:54.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:54.459 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:54.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:54.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:54.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:54.686 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:54.788 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:54.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:54.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:54.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:54.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:55.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:55.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:47:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:55.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:55.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:55.258 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:55.298 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:47:55.298 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:47:55.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:55.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:55.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:55.511 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:47:55.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:55.613 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:55.650 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:55.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:55.777 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:55.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:55 smithi136 ceph-mon[20207]: pgmap v1549: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 741/109917 objects degraded (0.674%); 3252/109917 objects misplaced (2.959%); 13 MiB/s, 200 objects/s recovering 2024-08-30T23:47:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2186387698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:55 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:55 smithi173 ceph-mon[20554]: pgmap v1549: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 741/109917 objects degraded (0.674%); 3252/109917 objects misplaced (2.959%); 13 MiB/s, 200 objects/s recovering 2024-08-30T23:47:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2186387698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:55 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:55.844 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:55 smithi106 ceph-mon[16603]: pgmap v1549: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 610 op/s; 741/109917 objects degraded (0.674%); 3252/109917 objects misplaced (2.959%); 13 MiB/s, 200 objects/s recovering 2024-08-30T23:47:55.844 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2186387698' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:55.844 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:55 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:47:55.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:55.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:55.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:56.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:56.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:56.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:56.147 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:56.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:47:56.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:56.270 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.271 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:56.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:56.395 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:56.497 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:56.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:56.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:56.644 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:56.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:56.761 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:56.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:56.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:56 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:47:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:56 smithi136 ceph-mon[20207]: osdmap e689: 12 total, 12 up, 11 in 2024-08-30T23:47:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 741/109917 objects degraded (0.674%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:56 smithi136 ceph-mon[20207]: osdmap e690: 12 total, 12 up, 11 in 2024-08-30T23:47:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:56 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:47:56.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:56 smithi173 ceph-mon[20554]: osdmap e689: 12 total, 12 up, 11 in 2024-08-30T23:47:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 741/109917 objects degraded (0.674%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:56 smithi173 ceph-mon[20554]: osdmap e690: 12 total, 12 up, 11 in 2024-08-30T23:47:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:56 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:47:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:47:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:56 smithi106 ceph-mon[16603]: osdmap e689: 12 total, 12 up, 11 in 2024-08-30T23:47:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 741/109917 objects degraded (0.674%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:47:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:56 smithi106 ceph-mon[16603]: osdmap e690: 12 total, 12 up, 11 in 2024-08-30T23:47:56.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:56.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:56.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:56.993 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:57.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:57.119 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:57.141 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:57.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:47:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:57.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:47:57.370 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:47:57.371 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:47:57.374 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:57.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:57.395 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:47:57.395 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:47:57.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:57.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:47:57.420 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:47:57.444 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:47:57.445 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:47:57.465 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:47:57.466 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:47:57.486 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:47:57.487 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:47:57.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:57.507 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:47:57.507 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:47:57.526 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:57.529 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:47:57.529 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:47:57.549 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:47:57.550 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:47:57.570 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:47:57.571 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:47:57.593 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:47:57.594 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:47:57.618 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:47:57.618 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:47:57.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:57.651 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:57.753 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:57.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:57 smithi136 ceph-mon[20207]: pgmap v1551: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 503/105948 objects degraded (0.475%); 3013/105948 objects misplaced (2.844%); 15 MiB/s, 237 objects/s recovering 2024-08-30T23:47:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:57 smithi136 ceph-mon[20207]: osdmap e691: 12 total, 12 up, 11 in 2024-08-30T23:47:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2012436007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:57 smithi173 ceph-mon[20554]: pgmap v1551: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 503/105948 objects degraded (0.475%); 3013/105948 objects misplaced (2.844%); 15 MiB/s, 237 objects/s recovering 2024-08-30T23:47:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:57 smithi173 ceph-mon[20554]: osdmap e691: 12 total, 12 up, 11 in 2024-08-30T23:47:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2012436007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:57 smithi106 ceph-mon[16603]: pgmap v1551: 37 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 503/105948 objects degraded (0.475%); 3013/105948 objects misplaced (2.844%); 15 MiB/s, 237 objects/s recovering 2024-08-30T23:47:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:57 smithi106 ceph-mon[16603]: osdmap e691: 12 total, 12 up, 11 in 2024-08-30T23:47:57.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2012436007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:47:57.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:57.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:57.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:58.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:47:58.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:47:58.085 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167035 6.0 6:0e3a471e:::benchmark_data_smithi173_114000_object67237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.068340+0000", 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.01470661, 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.014738370000000001, 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.086 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167035 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068340+0000", 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068340+0000", 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.087 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068342+0000", 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1160000000000002e-06 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068350+0000", 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1009999999999992e-06 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068368+0000", 2024-08-30T23:47:58.088 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8179e-05 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068376+0000", 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4700000000000002e-06 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068636+0000", 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025940400000000003 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.089 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068754+0000", 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118077 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068880+0000", 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012598000000000001 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.077790+0000", 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0089105880000000005 2024-08-30T23:47:58.090 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167036 6.6 6:6459d4ee:::benchmark_data_smithi173_114000_object67238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.069455+0000", 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.013591196999999999, 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.013710214, 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.091 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167036 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.069455+0000", 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.069455+0000", 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.092 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.069456+0000", 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0500000000000002e-07 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.069460+0000", 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1670000000000001e-06 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.093 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.069468+0000", 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9419999999999995e-06 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.069473+0000", 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0379999999999999e-06 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.071690+0000", 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002216984 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.094 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.071812+0000", 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121532 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,7", 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.071933+0000", 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012131200000000001 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167037 6.b 6:d4e419ca:::benchmark_data_smithi173_114000_object67239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.095 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.078964+0000", 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0040817090000000002, 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042602150000000004, 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167037 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.096 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.078964+0000", 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.078964+0000", 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.078966+0000", 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5880000000000001e-06 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.097 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.078972+0000", 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0759999999999999e-06 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.078986+0000", 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3444e-05 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.078994+0000", 2024-08-30T23:47:58.098 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7860000000000002e-06 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079150+0000", 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015559699999999999 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079315+0000", 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016473200000000001 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:47:58.099 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079434+0000", 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000119271 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167040 6.6 6:6df9fc26:::benchmark_data_smithi173_114000_object67242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.079018+0000", 2024-08-30T23:47:58.100 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0040282019999999998, 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042641429999999998, 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167040 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.101 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079018+0000", 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079018+0000", 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079018+0000", 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2e-07 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.102 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079022+0000", 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8519999999999997e-06 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079038+0000", 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5865000000000001e-05 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.079045+0000", 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5189999999999998e-06 2024-08-30T23:47:58.103 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.081082+0000", 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020370789999999998 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.081192+0000", 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011039099999999999 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,7", 2024-08-30T23:47:58.104 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.081300+0000", 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010774500000000001 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167018 6.0 6:0b4146fd:::benchmark_data_smithi173_114000_object67220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.046578+0000", 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.036468576000000003, 2024-08-30T23:47:58.105 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.036761143000000003, 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167018 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.046578+0000", 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.106 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.046578+0000", 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.046578+0000", 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9599999999999999e-07 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.046580+0000", 2024-08-30T23:47:58.107 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.367e-06 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.046586+0000", 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6930000000000004e-06 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.046591+0000", 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7840000000000003e-06 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.108 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.051698+0000", 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0051073109999999998 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.051818+0000", 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000120197 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.051940+0000", 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121074 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.109 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.063036+0000", 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011096363 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.074239+0000", 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011203466 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.110 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167019 6.0 6:0f63fd67:::benchmark_data_smithi173_114000_object67221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.048576+0000", 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.034469721000000002, 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034827616999999998, 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167019 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.111 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.048576+0000", 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.048576+0000", 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.048578+0000", 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4130000000000001e-06 2024-08-30T23:47:58.112 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.048584+0000", 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0449999999999997e-06 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.048596+0000", 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.256e-05 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.048604+0000", 2024-08-30T23:47:58.113 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.17e-06 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.057836+0000", 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0092325389999999997 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.057967+0000", 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013105300000000001 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.114 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.058081+0000", 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011423 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.068607+0000", 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010525812000000001 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.074245+0000", 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0056375219999999998 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.115 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:167025 6.0 6:0741de46:::benchmark_data_smithi173_114000_object67227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e692)", 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:58.062952+0000", 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.020094279999999999, 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020520772999999999, 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:47:58.116 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 167025 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.062952+0000", 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.062952+0000", 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:47:58.117 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.062953+0000", 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1990000000000001e-06 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.062960+0000", 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0779999999999998e-06 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:47:58.118 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.062971+0000", 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0351e-05 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.062977+0000", 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3249999999999996e-06 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.063245+0000", 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026769000000000001 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.119 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.063349+0000", 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104775 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.063465+0000", 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011524800000000001 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.074249+0000", 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010784909000000001 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:47:58.120 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:58.077757+0000", 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035075929999999998 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 7 2024-08-30T23:47:58.121 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:47:58.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:47:58.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:58.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:47:58.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:47:58.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:47:58.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:58.425 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.426 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:58.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:58.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:58.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:58.688 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:58.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:47:58.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:47:58.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:47:58.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:47:58.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:47:58.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:47:58.940 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-08-30T23:47:58.940 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:47:58.940 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:47:58.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-08-30T23:47:58.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:58.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:47:59.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:59.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.069 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:59.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:59.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.197 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:47:59.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:58 smithi106 ceph-mon[16603]: pgmap v1554: 37 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 502/100446 objects degraded (0.500%); 2217/100446 objects misplaced (2.207%); 16 MiB/s, 263 objects/s recovering 2024-08-30T23:47:59.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:58 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:47:59.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:58 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:47:59.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:58 smithi106 ceph-mon[16603]: osdmap e692: 12 total, 12 up, 11 in 2024-08-30T23:47:59.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:47:59.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:58 smithi136 ceph-mon[20207]: pgmap v1554: 37 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 502/100446 objects degraded (0.500%); 2217/100446 objects misplaced (2.207%); 16 MiB/s, 263 objects/s recovering 2024-08-30T23:47:59.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:58 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:47:59.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:58 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:47:59.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:58 smithi136 ceph-mon[20207]: osdmap e692: 12 total, 12 up, 11 in 2024-08-30T23:47:59.291 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:47:59.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:47:59.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:47:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:58 smithi173 ceph-mon[20554]: pgmap v1554: 37 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 708 op/s; 502/100446 objects degraded (0.500%); 2217/100446 objects misplaced (2.207%); 16 MiB/s, 263 objects/s recovering 2024-08-30T23:47:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:58 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:47:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:58 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:47:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:58 smithi173 ceph-mon[20554]: osdmap e692: 12 total, 12 up, 11 in 2024-08-30T23:47:59.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:47:59.438 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:47:59.445 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:47:59.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:47:59.583 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:47:59.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:47:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:47:59.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:47:59.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:47:59.854 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:47:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:47:59.865 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:47:59.865 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:47:59.865 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:47:59.956 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:47:59.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:47:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:47:59.983 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:00.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:59 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:48:00.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:47:59 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:48:00.057 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:00.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:00.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:00.189 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:59 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:48:00.189 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:47:59 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:48:00.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:00.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:59 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:48:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:47:59 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:48:00.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:00.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:00.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:00.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:00.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:48:00.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:00.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:00.696 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:48:00.696 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:48:00.696 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:48:00.696 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:48:00.696 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:48:00.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:00.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:00.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:00.900 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:48:00.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:00.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:01.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:01.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:01.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:01.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:01.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:00 smithi136 ceph-mon[20207]: pgmap v1556: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 34 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 221/97614 objects degraded (0.226%); 2002/97614 objects misplaced (2.051%); 19 MiB/s, 311 objects/s recovering 2024-08-30T23:48:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/36423743' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:00 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 221/97614 objects degraded (0.226%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:48:01.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:00 smithi106 ceph-mon[16603]: pgmap v1556: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 34 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 221/97614 objects degraded (0.226%); 2002/97614 objects misplaced (2.051%); 19 MiB/s, 311 objects/s recovering 2024-08-30T23:48:01.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/36423743' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:01.310 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:00 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 221/97614 objects degraded (0.226%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:48:01.317 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:00 smithi173 ceph-mon[20554]: pgmap v1556: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 34 active+clean; 2.0 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 668 op/s; 221/97614 objects degraded (0.226%); 2002/97614 objects misplaced (2.051%); 19 MiB/s, 311 objects/s recovering 2024-08-30T23:48:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/36423743' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:00 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 221/97614 objects degraded (0.226%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:48:01.419 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:01.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:01.546 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:01.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:01.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:01.696 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:01.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:48:01.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:01.882 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:01.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:01.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:02.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:01 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/36423743' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:02.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:01 smithi106 ceph-mon[16603]: osdmap e693: 12 total, 12 up, 11 in 2024-08-30T23:48:02.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:02.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:02.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:01 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:02.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/36423743' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:02.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:01 smithi136 ceph-mon[20207]: osdmap e693: 12 total, 12 up, 11 in 2024-08-30T23:48:02.298 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:01 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/36423743' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:01 smithi173 ceph-mon[20554]: osdmap e693: 12 total, 12 up, 11 in 2024-08-30T23:48:02.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:02.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:02.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:02.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:02.595 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:02.652 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:02.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:02.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:02.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:02.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:02.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:02.939 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:02.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:03.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:03.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:02 smithi106 ceph-mon[16603]: pgmap v1558: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 221/94980 objects degraded (0.233%); 2002/94980 objects misplaced (2.108%); 14 MiB/s, 222 objects/s recovering 2024-08-30T23:48:03.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:03.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:03.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:02 smithi136 ceph-mon[20207]: pgmap v1558: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 221/94980 objects degraded (0.233%); 2002/94980 objects misplaced (2.108%); 14 MiB/s, 222 objects/s recovering 2024-08-30T23:48:03.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:02 smithi173 ceph-mon[20554]: pgmap v1558: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 221/94980 objects degraded (0.233%); 2002/94980 objects misplaced (2.108%); 14 MiB/s, 222 objects/s recovering 2024-08-30T23:48:03.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:03.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:03.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:03.512 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:03.512 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:03.534 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:03.535 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:03.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:03.577 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:03.577 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:03.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:03.609 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:03.609 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:03.643 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:03.644 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:03.678 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:03.679 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:03.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:03.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:03.711 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:03.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:03.738 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:03.739 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:03.773 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:03.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:03.812 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:03.812 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:03.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:03.851 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:03.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:03.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:03.891 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:03.892 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:03.948 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:03.948 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:03.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:03.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:03.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:04.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:04.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:04.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:04.237 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1839881536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1839881536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:04.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1839881536' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:04.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:04.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.489 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:04.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:04.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:48:04.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:04.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:04.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:04.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:04.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.873 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:04.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:04.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:04.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:04.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:05.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:04 smithi106 ceph-mon[16603]: pgmap v1559: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 221/91749 objects degraded (0.241%); 1683/91749 objects misplaced (1.834%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:48:05.121 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:05.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:48:05.204 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:04 smithi136 ceph-mon[20207]: pgmap v1559: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 221/91749 objects degraded (0.241%); 1683/91749 objects misplaced (1.834%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:48:05.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:05.244 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:05.289 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:05.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:04 smithi173 ceph-mon[20554]: pgmap v1559: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 35 active+clean; 1.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 221/91749 objects degraded (0.241%); 1683/91749 objects misplaced (1.834%); 11 MiB/s, 170 objects/s recovering 2024-08-30T23:48:05.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:05.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:05.436 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:05.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:05.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:05.580 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:48:05.581 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:48:05.581 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-30T23:48:05.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:48:05.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:05.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:05.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:05.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:05.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:05.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:05.860 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:05.957 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:05.957 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:05.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:05.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:05.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:05.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:05.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:05.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:05 smithi106 ceph-mon[16603]: osdmap e694: 12 total, 12 up, 11 in 2024-08-30T23:48:05.983 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 221/91749 objects degraded (0.241%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:48:06.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:06.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.105 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:06.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:06.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:05 smithi136 ceph-mon[20207]: osdmap e694: 12 total, 12 up, 11 in 2024-08-30T23:48:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 221/91749 objects degraded (0.241%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:48:06.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:05 smithi173 ceph-mon[20554]: osdmap e694: 12 total, 12 up, 11 in 2024-08-30T23:48:06.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 221/91749 objects degraded (0.241%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:48:06.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:06.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:06.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:06.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:06.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:06.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:06.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:48:06.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:06.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:06.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:48:06.796 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:48:06.799 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:48:06.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:06.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:06.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:06.932 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:48:06.955 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:06.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:07.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.003 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:07.056 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:06 smithi106 ceph-mon[16603]: pgmap v1561: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 221/87486 objects degraded (0.253%); 1460/87486 objects misplaced (1.669%); 8.3 MiB/s, 133 objects/s recovering 2024-08-30T23:48:07.056 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:06 smithi106 ceph-mon[16603]: osdmap e695: 12 total, 12 up, 11 in 2024-08-30T23:48:07.056 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:07.056 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:07.056 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3958174843' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:07.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:07.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.128 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:07.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:07.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:06 smithi136 ceph-mon[20207]: pgmap v1561: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 221/87486 objects degraded (0.253%); 1460/87486 objects misplaced (1.669%); 8.3 MiB/s, 133 objects/s recovering 2024-08-30T23:48:07.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:06 smithi136 ceph-mon[20207]: osdmap e695: 12 total, 12 up, 11 in 2024-08-30T23:48:07.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:07.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:07.237 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3958174843' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:07.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:06 smithi173 ceph-mon[20554]: pgmap v1561: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 35 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 221/87486 objects degraded (0.253%); 1460/87486 objects misplaced (1.669%); 8.3 MiB/s, 133 objects/s recovering 2024-08-30T23:48:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:06 smithi173 ceph-mon[20554]: osdmap e695: 12 total, 12 up, 11 in 2024-08-30T23:48:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3958174843' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:07.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:07.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:07.480 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:07.509 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:07.509 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:07.509 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:07.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:07.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:07.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:07.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:07.912 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:07.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:08.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:08.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:08.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:08.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:08.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:07 smithi136 ceph-mon[20207]: from='client.17940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:48:08.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:07 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:08.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3958174843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:08.185 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:07 smithi136 ceph-mon[20207]: osdmap e696: 12 total, 12 up, 11 in 2024-08-30T23:48:08.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:48:08.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:08.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:07 smithi173 ceph-mon[20554]: from='client.17940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:48:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:07 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3958174843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:07 smithi173 ceph-mon[20554]: osdmap e696: 12 total, 12 up, 11 in 2024-08-30T23:48:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:07 smithi106 ceph-mon[16603]: from='client.17940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:48:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:07 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3958174843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:07 smithi106 ceph-mon[16603]: osdmap e696: 12 total, 12 up, 11 in 2024-08-30T23:48:08.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:08.413 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:08.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:08.536 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:08.547 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:08.602 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1562,"stamp":"2024-08-30T23:48:05.922700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1911554616,"num_objects":29162,"num_object_clones":0,"num_object_copies":87486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":221,"num_objects_misplaced":1460,"num_objects_unfound":0,"num_objects_dirty":29162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170432,"num_write_kb":6386881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26112,"num_bytes_recovered":1711472640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162864,"ondisk_log_size":162864,"up":111,"acting":109,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":10500812,"kb_used_data":5612692,"kb_used_omap":107,"kb_used_meta":4887828,"kb_avail":1020695860,"statfs":{"total":1055945392128,"available":1045192560640,"internally_reserved":0,"allocated":5747396608,"data_stored":5732920392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110376,"internal_metadata":5005136088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,6,8,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-221249536,"num_objects":-3376,"num_object_clones":0,"num_object_copies":-10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-542,"num_objects_unfound":0,"num_objects_dirty":-3376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":35520512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.257942"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":419,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.900183+0000","last_change":"2024-08-30T23:46:03.153483+0000","last_active":"2024-08-30T23:48:00.900183+0000","last_peered":"2024-08-30T23:48:00.900183+0000","last_clean":"2024-08-30T23:48:00.900183+0000","last_became_active":"2024-08-30T23:46:03.153107+0000","last_became_peered":"2024-08-30T23:46:03.153107+0000","last_unstale":"2024-08-30T23:48:00.900183+0000","last_undegraded":"2024-08-30T23:48:00.900183+0000","last_fullsized":"2024-08-30T23:48:00.900183+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:56.069850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:56.069850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":367,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.902382+0000","last_change":"2024-08-30T23:44:29.831279+0000","last_active":"2024-08-30T23:48:00.902382+0000","last_peered":"2024-08-30T23:48:00.902382+0000","last_clean":"2024-08-30T23:48:00.902382+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:48:00.902382+0000","last_undegraded":"2024-08-30T23:48:00.902382+0000","last_fullsized":"2024-08-30T23:48:00.902382+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.831212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.831212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":417,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.907147+0000","last_change":"2024-08-30T23:45:33.691110+0000","last_active":"2024-08-30T23:48:00.907147+0000","last_peered":"2024-08-30T23:48:00.907147+0000","last_clean":"2024-08-30T23:48:00.907147+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:48:00.907147+0000","last_undegraded":"2024-08-30T23:48:00.907147+0000","last_fullsized":"2024-08-30T23:48:00.907147+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:33.691035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:33.691035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":1201,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.906941+0000","last_change":"2024-08-30T23:46:12.979778+0000","last_active":"2024-08-30T23:48:00.906941+0000","last_peered":"2024-08-30T23:48:00.906941+0000","last_clean":"2024-08-30T23:48:00.906941+0000","last_became_active":"2024-08-30T23:46:12.979459+0000","last_became_peered":"2024-08-30T23:46:12.979459+0000","last_unstale":"2024-08-30T23:48:00.906941+0000","last_undegraded":"2024-08-30T23:48:00.906941+0000","last_fullsized":"2024-08-30T23:48:00.906941+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:29.953019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:29.953019+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,4],"acting":[8,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.7","version":"693'10561","reported_seq":21018,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:04.648048+0000","last_change":"2024-08-30T23:47:57.893652+0000","last_active":"2024-08-30T23:48:04.648048+0000","last_peered":"2024-08-30T23:48:04.648048+0000","last_clean":"2024-08-30T23:48:04.648048+0000","last_became_active":"2024-08-30T23:47:57.869318+0000","last_became_peered":"2024-08-30T23:47:57.869318+0000","last_unstale":"2024-08-30T23:48:04.648048+0000","last_undegraded":"2024-08-30T23:48:04.648048+0000","last_fullsized":"2024-08-30T23:48:04.648048+0000","mapping_epoch":691,"log_start":"625'500","ondisk_log_start":"625'500","created":619,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110166016,"num_objects":1681,"num_object_clones":0,"num_object_copies":5043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":391744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5223,"num_bytes_recovered":342294528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":188,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.902538+0000","last_change":"2024-08-30T23:46:53.896596+0000","last_active":"2024-08-30T23:48:00.902538+0000","last_peered":"2024-08-30T23:48:00.902538+0000","last_clean":"2024-08-30T23:48:00.902538+0000","last_became_active":"2024-08-30T23:45:08.975961+0000","last_became_peered":"2024-08-30T23:45:08.975961+0000","last_unstale":"2024-08-30T23:48:00.902538+0000","last_undegraded":"2024-08-30T23:48:00.902538+0000","last_fullsized":"2024-08-30T23:48:00.902538+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:46:53.896552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:46:53.896552+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:07.274144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"693'10555","reported_seq":10822,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:02.186511+0000","last_change":"2024-08-30T23:47:47.467340+0000","last_active":"2024-08-30T23:48:02.186511+0000","last_peered":"2024-08-30T23:48:02.186511+0000","last_clean":"2024-08-30T23:48:02.186511+0000","last_became_active":"2024-08-30T23:46:12.977020+0000","last_became_peered":"2024-08-30T23:46:12.977020+0000","last_unstale":"2024-08-30T23:48:02.186511+0000","last_undegraded":"2024-08-30T23:48:02.186511+0000","last_fullsized":"2024-08-30T23:48:02.186511+0000","mapping_epoch":657,"log_start":"625'500","ondisk_log_start":"625'500","created":619,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'9979","last_scrub_stamp":"2024-08-30T23:47:47.467181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:47:47.467181+0000","objects_scrubbed":2468,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:50.765076+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122880000,"num_objects":1875,"num_object_clones":0,"num_object_copies":5625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10555,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":713,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.900065+0000","last_change":"2024-08-30T23:45:28.980455+0000","last_active":"2024-08-30T23:48:00.900065+0000","last_peered":"2024-08-30T23:48:00.900065+0000","last_clean":"2024-08-30T23:48:00.900065+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:48:00.900065+0000","last_undegraded":"2024-08-30T23:48:00.900065+0000","last_fullsized":"2024-08-30T23:48:00.900065+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:28.980388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:28.980388+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"693'10735","reported_seq":10894,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:01.465448+0000","last_change":"2024-08-30T23:44:42.456106+0000","last_active":"2024-08-30T23:48:01.465448+0000","last_peered":"2024-08-30T23:48:01.465448+0000","last_clean":"2024-08-30T23:48:01.465448+0000","last_became_active":"2024-08-30T23:44:42.455726+0000","last_became_peered":"2024-08-30T23:44:42.455726+0000","last_unstale":"2024-08-30T23:48:01.465448+0000","last_undegraded":"2024-08-30T23:48:01.465448+0000","last_fullsized":"2024-08-30T23:48:01.465448+0000","mapping_epoch":619,"log_start":"626'700","ondisk_log_start":"626'700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123928576,"num_objects":1891,"num_object_clones":0,"num_object_copies":5673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":404032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":336,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.900096+0000","last_change":"2024-08-30T23:46:02.884022+0000","last_active":"2024-08-30T23:48:00.900096+0000","last_peered":"2024-08-30T23:48:00.900096+0000","last_clean":"2024-08-30T23:48:00.900096+0000","last_became_active":"2024-08-30T23:46:02.883635+0000","last_became_peered":"2024-08-30T23:46:02.883635+0000","last_unstale":"2024-08-30T23:48:00.900096+0000","last_undegraded":"2024-08-30T23:48:00.900096+0000","last_fullsized":"2024-08-30T23:48:00.900096+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:38.508393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:38.508393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,4],"acting":[11,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.1","version":"693'10568","reported_seq":22462,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:01.054088+0000","last_change":"2024-08-30T23:47:56.890182+0000","last_active":"2024-08-30T23:48:01.054088+0000","last_peered":"2024-08-30T23:48:01.054088+0000","last_clean":"2024-08-30T23:48:01.054088+0000","last_became_active":"2024-08-30T23:47:56.868275+0000","last_became_peered":"2024-08-30T23:47:56.868275+0000","last_unstale":"2024-08-30T23:48:01.054088+0000","last_undegraded":"2024-08-30T23:48:01.054088+0000","last_fullsized":"2024-08-30T23:48:01.054088+0000","mapping_epoch":690,"log_start":"625'500","ondisk_log_start":"625'500","created":619,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129892352,"num_objects":1982,"num_object_clones":0,"num_object_copies":5946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5863,"num_bytes_recovered":384237568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":103,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-30T23:48:04.921692+0000","last_change":"2024-08-30T23:46:03.539515+0000","last_active":"2024-08-30T23:48:04.921692+0000","last_peered":"2024-08-30T23:48:04.921692+0000","last_clean":"2024-08-30T23:48:04.921692+0000","last_became_active":"2024-08-30T23:46:02.888245+0000","last_became_peered":"2024-08-30T23:46:02.888245+0000","last_unstale":"2024-08-30T23:48:04.921692+0000","last_undegraded":"2024-08-30T23:48:04.921692+0000","last_fullsized":"2024-08-30T23:48:04.921692+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:46:03.539472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:46:03.539472+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.0","version":"693'10722","reported_seq":10880,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:03.957823+0000","last_change":"2024-08-30T23:44:42.456851+0000","last_active":"2024-08-30T23:48:03.957823+0000","last_peered":"2024-08-30T23:48:03.957823+0000","last_clean":"2024-08-30T23:48:03.957823+0000","last_became_active":"2024-08-30T23:44:42.455669+0000","last_became_peered":"2024-08-30T23:44:42.455669+0000","last_unstale":"2024-08-30T23:48:03.957823+0000","last_undegraded":"2024-08-30T23:48:03.957823+0000","last_fullsized":"2024-08-30T23:48:03.957823+0000","mapping_epoch":619,"log_start":"626'700","ondisk_log_start":"626'700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118882304,"num_objects":1814,"num_object_clones":0,"num_object_copies":5442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":401152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":608,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.902831+0000","last_change":"2024-08-30T23:47:44.643691+0000","last_active":"2024-08-30T23:48:00.902831+0000","last_peered":"2024-08-30T23:48:00.902831+0000","last_clean":"2024-08-30T23:48:00.902831+0000","last_became_active":"2024-08-30T23:46:02.904175+0000","last_became_peered":"2024-08-30T23:46:02.904175+0000","last_unstale":"2024-08-30T23:48:00.902831+0000","last_undegraded":"2024-08-30T23:48:00.902831+0000","last_fullsized":"2024-08-30T23:48:00.902831+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:44.643614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:44.643614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:08.471210+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"693'10790","reported_seq":10949,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:03.847065+0000","last_change":"2024-08-30T23:44:42.310819+0000","last_active":"2024-08-30T23:48:03.847065+0000","last_peered":"2024-08-30T23:48:03.847065+0000","last_clean":"2024-08-30T23:48:03.847065+0000","last_became_active":"2024-08-30T23:44:42.310475+0000","last_became_peered":"2024-08-30T23:44:42.310475+0000","last_unstale":"2024-08-30T23:48:03.847065+0000","last_undegraded":"2024-08-30T23:48:03.847065+0000","last_fullsized":"2024-08-30T23:48:03.847065+0000","mapping_epoch":619,"log_start":"626'700","ondisk_log_start":"626'700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116260864,"num_objects":1774,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10790,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":92,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-30T23:48:04.921590+0000","last_change":"2024-08-30T23:46:02.889514+0000","last_active":"2024-08-30T23:48:04.921590+0000","last_peered":"2024-08-30T23:48:04.921590+0000","last_clean":"2024-08-30T23:48:04.921590+0000","last_became_active":"2024-08-30T23:46:02.889114+0000","last_became_peered":"2024-08-30T23:46:02.889114+0000","last_unstale":"2024-08-30T23:48:04.921590+0000","last_undegraded":"2024-08-30T23:48:04.921590+0000","last_fullsized":"2024-08-30T23:48:04.921590+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:46:00.551585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:46:00.551585+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.5","version":"693'10733","reported_seq":10958,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:02.564394+0000","last_change":"2024-08-30T23:46:05.011293+0000","last_active":"2024-08-30T23:48:02.564394+0000","last_peered":"2024-08-30T23:48:02.564394+0000","last_clean":"2024-08-30T23:48:02.564394+0000","last_became_active":"2024-08-30T23:44:43.027945+0000","last_became_peered":"2024-08-30T23:44:43.027945+0000","last_unstale":"2024-08-30T23:48:02.564394+0000","last_undegraded":"2024-08-30T23:48:02.564394+0000","last_fullsized":"2024-08-30T23:48:02.564394+0000","mapping_epoch":619,"log_start":"626'700","ondisk_log_start":"626'700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"655'5888","last_scrub_stamp":"2024-08-30T23:46:05.011113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:46:05.011113+0000","objects_scrubbed":5872,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115671040,"num_objects":1765,"num_object_clones":0,"num_object_copies":5295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":399936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1441,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.899848+0000","last_change":"2024-08-30T23:44:55.636789+0000","last_active":"2024-08-30T23:48:00.899848+0000","last_peered":"2024-08-30T23:48:00.899848+0000","last_clean":"2024-08-30T23:48:00.899848+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:48:00.899848+0000","last_undegraded":"2024-08-30T23:48:00.899848+0000","last_fullsized":"2024-08-30T23:48:00.899848+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:55.636740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:55.636740+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"693'10586","reported_seq":21354,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:01.050147+0000","last_change":"2024-08-30T23:47:40.461569+0000","last_active":"2024-08-30T23:48:01.050147+0000","last_peered":"2024-08-30T23:48:01.050147+0000","last_clean":"2024-08-30T23:48:01.050147+0000","last_became_active":"2024-08-30T23:47:40.439057+0000","last_became_peered":"2024-08-30T23:47:40.439057+0000","last_unstale":"2024-08-30T23:48:01.050147+0000","last_undegraded":"2024-08-30T23:48:01.050147+0000","last_fullsized":"2024-08-30T23:48:01.050147+0000","mapping_epoch":684,"log_start":"624'500","ondisk_log_start":"624'500","created":619,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126484480,"num_objects":1930,"num_object_clones":0,"num_object_copies":5790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10586,"num_write_kb":400512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5301,"num_bytes_recovered":347406336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":336,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.899840+0000","last_change":"2024-08-30T23:44:58.589043+0000","last_active":"2024-08-30T23:48:00.899840+0000","last_peered":"2024-08-30T23:48:00.899840+0000","last_clean":"2024-08-30T23:48:00.899840+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:48:00.899840+0000","last_undegraded":"2024-08-30T23:48:00.899840+0000","last_fullsized":"2024-08-30T23:48:00.899840+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:58.588952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:58.588952+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"693'10783","reported_seq":10942,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:03.910900+0000","last_change":"2024-08-30T23:44:42.314402+0000","last_active":"2024-08-30T23:48:03.910900+0000","last_peered":"2024-08-30T23:48:03.910900+0000","last_clean":"2024-08-30T23:48:03.910900+0000","last_became_active":"2024-08-30T23:44:42.314051+0000","last_became_peered":"2024-08-30T23:44:42.314051+0000","last_unstale":"2024-08-30T23:48:03.910900+0000","last_undegraded":"2024-08-30T23:48:03.910900+0000","last_fullsized":"2024-08-30T23:48:03.910900+0000","mapping_epoch":619,"log_start":"626'700","ondisk_log_start":"626'700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":400384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":428,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.904473+0000","last_change":"2024-08-30T23:47:59.712954+0000","last_active":"2024-08-30T23:48:00.904473+0000","last_peered":"2024-08-30T23:48:00.904473+0000","last_clean":"2024-08-30T23:48:00.904473+0000","last_became_active":"2024-08-30T23:43:56.833115+0000","last_became_peered":"2024-08-30T23:43:56.833115+0000","last_unstale":"2024-08-30T23:48:00.904473+0000","last_undegraded":"2024-08-30T23:48:00.904473+0000","last_fullsized":"2024-08-30T23:48:00.904473+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:59.712882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:59.712882+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:16.676177+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"693'10539","reported_seq":10814,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:02.170704+0000","last_change":"2024-08-30T23:47:03.354462+0000","last_active":"2024-08-30T23:48:02.170704+0000","last_peered":"2024-08-30T23:48:02.170704+0000","last_clean":"2024-08-30T23:48:02.170704+0000","last_became_active":"2024-08-30T23:44:43.028019+0000","last_became_peered":"2024-08-30T23:44:43.028019+0000","last_unstale":"2024-08-30T23:48:02.170704+0000","last_undegraded":"2024-08-30T23:48:02.170704+0000","last_fullsized":"2024-08-30T23:48:02.170704+0000","mapping_epoch":619,"log_start":"625'500","ondisk_log_start":"625'500","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"672'8124","last_scrub_stamp":"2024-08-30T23:47:03.354332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:47:03.354332+0000","objects_scrubbed":4291,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:09.179171+0000","scrub_duration":1604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121700352,"num_objects":1857,"num_object_clones":0,"num_object_copies":5571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10539,"num_write_kb":396672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":414,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.899874+0000","last_change":"2024-08-30T23:47:57.710542+0000","last_active":"2024-08-30T23:48:00.899874+0000","last_peered":"2024-08-30T23:48:00.899874+0000","last_clean":"2024-08-30T23:48:00.899874+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:48:00.899874+0000","last_undegraded":"2024-08-30T23:48:00.899874+0000","last_fullsized":"2024-08-30T23:48:00.899874+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:57.710491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:57.710491+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:05.529992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"693'10859","reported_seq":11080,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:01.083765+0000","last_change":"2024-08-30T23:45:56.318715+0000","last_active":"2024-08-30T23:48:01.083765+0000","last_peered":"2024-08-30T23:48:01.083765+0000","last_clean":"2024-08-30T23:48:01.083765+0000","last_became_active":"2024-08-30T23:44:43.029122+0000","last_became_peered":"2024-08-30T23:44:43.029122+0000","last_unstale":"2024-08-30T23:48:01.083765+0000","last_undegraded":"2024-08-30T23:48:01.083765+0000","last_fullsized":"2024-08-30T23:48:01.083765+0000","mapping_epoch":619,"log_start":"626'800","ondisk_log_start":"626'800","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"652'5669","last_scrub_stamp":"2024-08-30T23:45:56.318495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:45:56.318495+0000","objects_scrubbed":5640,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121962496,"num_objects":1861,"num_object_clones":0,"num_object_copies":5583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10859,"num_write_kb":407040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":344,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.900002+0000","last_change":"2024-08-30T23:45:00.588948+0000","last_active":"2024-08-30T23:48:00.900002+0000","last_peered":"2024-08-30T23:48:00.900002+0000","last_clean":"2024-08-30T23:48:00.900002+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:48:00.900002+0000","last_undegraded":"2024-08-30T23:48:00.900002+0000","last_fullsized":"2024-08-30T23:48:00.900002+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:00.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:00.588903+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"693'10833","reported_seq":10997,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:02.302815+0000","last_change":"2024-08-30T23:46:12.987104+0000","last_active":"2024-08-30T23:48:02.302815+0000","last_peered":"2024-08-30T23:48:02.302815+0000","last_clean":"2024-08-30T23:48:02.302815+0000","last_became_active":"2024-08-30T23:46:12.977124+0000","last_became_peered":"2024-08-30T23:46:12.977124+0000","last_unstale":"2024-08-30T23:48:02.302815+0000","last_undegraded":"2024-08-30T23:48:02.302815+0000","last_fullsized":"2024-08-30T23:48:02.302815+0000","mapping_epoch":657,"log_start":"626'800","ondisk_log_start":"626'800","created":619,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119603200,"num_objects":1825,"num_object_clones":0,"num_object_copies":5475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10833,"num_write_kb":405056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":616,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.902739+0000","last_change":"2024-08-30T23:47:13.689799+0000","last_active":"2024-08-30T23:48:00.902739+0000","last_peered":"2024-08-30T23:48:00.902739+0000","last_clean":"2024-08-30T23:48:00.902739+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:48:00.902739+0000","last_undegraded":"2024-08-30T23:48:00.902739+0000","last_fullsized":"2024-08-30T23:48:00.902739+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:13.689751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:13.689751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:14.311161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"693'10596","reported_seq":19422,"reported_epoch":693,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:48:03.723729+0000","last_change":"2024-08-30T23:46:58.188908+0000","last_active":"2024-08-30T23:48:03.723729+0000","last_peered":"2024-08-30T23:48:03.723729+0000","last_clean":"2024-08-30T23:46:01.825346+0000","last_became_active":"2024-08-30T23:46:04.016297+0000","last_became_peered":"2024-08-30T23:46:04.016297+0000","last_unstale":"2024-08-30T23:48:03.723729+0000","last_undegraded":"2024-08-30T23:46:58.185895+0000","last_fullsized":"2024-08-30T23:46:03.877675+0000","mapping_epoch":654,"log_start":"625'500","ondisk_log_start":"625'500","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"652'5594","last_scrub_stamp":"2024-08-30T23:45:58.431833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:45:58.431833+0000","objects_scrubbed":5563,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":948,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113770496,"num_objects":1736,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":221,"num_objects_misplaced":1263,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10596,"num_write_kb":394624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4301,"num_bytes_recovered":281804800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,9],"acting":[2,9],"avail_no_missing":["2","9","10"],"object_location_counts":[{"shards":"2,9","objects":252},{"shards":"2,9,10","objects":1263}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":534,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.903830+0000","last_change":"2024-08-30T23:44:37.215950+0000","last_active":"2024-08-30T23:48:00.903830+0000","last_peered":"2024-08-30T23:48:00.903830+0000","last_clean":"2024-08-30T23:48:00.903830+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:48:00.903830+0000","last_undegraded":"2024-08-30T23:48:00.903830+0000","last_fullsized":"2024-08-30T23:48:00.903830+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:44:37.215896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:44:37.215896+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"693'10760","reported_seq":10919,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:01.476408+0000","last_change":"2024-08-30T23:44:42.456265+0000","last_active":"2024-08-30T23:48:01.476408+0000","last_peered":"2024-08-30T23:48:01.476408+0000","last_clean":"2024-08-30T23:48:01.476408+0000","last_became_active":"2024-08-30T23:44:42.455981+0000","last_became_peered":"2024-08-30T23:44:42.455981+0000","last_unstale":"2024-08-30T23:48:01.476408+0000","last_undegraded":"2024-08-30T23:48:01.476408+0000","last_fullsized":"2024-08-30T23:48:01.476408+0000","mapping_epoch":619,"log_start":"626'700","ondisk_log_start":"626'700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125566976,"num_objects":1916,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":405632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1428,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.899430+0000","last_change":"2024-08-30T23:45:57.886763+0000","last_active":"2024-08-30T23:48:00.899430+0000","last_peered":"2024-08-30T23:48:00.899430+0000","last_clean":"2024-08-30T23:48:00.899430+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:48:00.899430+0000","last_undegraded":"2024-08-30T23:48:00.899430+0000","last_fullsized":"2024-08-30T23:48:00.899430+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:57.886697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:57.886697+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"693'10288","reported_seq":21296,"reported_epoch":693,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:48:02.275457+0000","last_change":"2024-08-30T23:46:04.075069+0000","last_active":"2024-08-30T23:48:02.275457+0000","last_peered":"2024-08-30T23:48:02.275457+0000","last_clean":"2024-08-30T23:46:01.844281+0000","last_became_active":"2024-08-30T23:46:04.043810+0000","last_became_peered":"2024-08-30T23:46:04.043810+0000","last_unstale":"2024-08-30T23:48:02.275457+0000","last_undegraded":"2024-08-30T23:48:02.275457+0000","last_fullsized":"2024-08-30T23:46:03.881181+0000","mapping_epoch":654,"log_start":"623'200","ondisk_log_start":"623'200","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115802136,"num_objects":1768,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":197,"num_objects_unfound":0,"num_objects_dirty":1768,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10288,"num_write_kb":385729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5423,"num_bytes_recovered":355401728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1],"avail_no_missing":["11","1","10"],"object_location_counts":[{"shards":"1,10,11","objects":197},{"shards":"1,11","objects":1571}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":552,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-08-30T23:48:04.921712+0000","last_change":"2024-08-30T23:45:40.727092+0000","last_active":"2024-08-30T23:48:04.921712+0000","last_peered":"2024-08-30T23:48:04.921712+0000","last_clean":"2024-08-30T23:48:04.921712+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:48:04.921712+0000","last_undegraded":"2024-08-30T23:48:04.921712+0000","last_fullsized":"2024-08-30T23:48:04.921712+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:40.727042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:40.727042+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"693'10524","reported_seq":10683,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:03.965751+0000","last_change":"2024-08-30T23:44:42.305791+0000","last_active":"2024-08-30T23:48:03.965751+0000","last_peered":"2024-08-30T23:48:03.965751+0000","last_clean":"2024-08-30T23:48:03.965751+0000","last_became_active":"2024-08-30T23:44:42.305499+0000","last_became_peered":"2024-08-30T23:44:42.305499+0000","last_unstale":"2024-08-30T23:48:03.965751+0000","last_undegraded":"2024-08-30T23:48:03.965751+0000","last_fullsized":"2024-08-30T23:48:03.965751+0000","mapping_epoch":619,"log_start":"625'500","ondisk_log_start":"625'500","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:44:41.288579+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115081216,"num_objects":1756,"num_object_clones":0,"num_object_copies":5268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10524,"num_write_kb":392960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,7],"acting":[3,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":330,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.900030+0000","last_change":"2024-08-30T23:45:53.048060+0000","last_active":"2024-08-30T23:48:00.900030+0000","last_peered":"2024-08-30T23:48:00.900030+0000","last_clean":"2024-08-30T23:48:00.900030+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:48:00.900030+0000","last_undegraded":"2024-08-30T23:48:00.900030+0000","last_fullsized":"2024-08-30T23:48:00.900030+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:53.047998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:53.047998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":969,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-08-30T23:48:00.901799+0000","last_change":"2024-08-30T23:46:13.714566+0000","last_active":"2024-08-30T23:48:00.901799+0000","last_peered":"2024-08-30T23:48:00.901799+0000","last_clean":"2024-08-30T23:48:00.901799+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:48:00.901799+0000","last_undegraded":"2024-08-30T23:48:00.901799+0000","last_fullsized":"2024-08-30T23:48:00.901799+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:46:13.714502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:46:13.714502+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1910964248,"num_objects":29160,"num_object_clones":0,"num_object_copies":87480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":221,"num_objects_misplaced":1460,"num_objects_unfound":0,"num_objects_dirty":29160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170432,"num_write_kb":6386881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26111,"num_bytes_recovered":1711144960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6700347392,"data_stored":6700335176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187738,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828332,"kb_used_data":363996,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92916820,"statfs":{"total":95995035648,"available":95146823680,"internally_reserved":0,"allocated":372731904,"data_stored":371413080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":475462084},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":335007449379,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711013,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1062632,"kb_used_data":553880,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92682520,"statfs":{"total":95995035648,"available":94906900480,"internally_reserved":0,"allocated":567173120,"data_stored":565858376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":520944064},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972650,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783028,"kb_used_data":372832,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92962124,"statfs":{"total":95995035648,"available":95193214976,"internally_reserved":0,"allocated":381779968,"data_stored":380461672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234285,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1351788,"kb_used_data":716696,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92393364,"statfs":{"total":95995035648,"available":94610804736,"internally_reserved":0,"allocated":733896704,"data_stored":732581960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":650312138},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528628,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750696,"kb_used_data":362712,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":92994456,"statfs":{"total":95995035648,"available":95226322944,"internally_reserved":0,"allocated":371417088,"data_stored":370102344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":397277638},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775132,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425960,"kb_used_data":304280,"kb_used_omap":5,"kb_used_meta":121658,"kb_avail":93319192,"statfs":{"total":95995035648,"available":95558852608,"internally_reserved":0,"allocated":311582720,"data_stored":310267976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":124577866},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953791,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973916,"kb_used_data":495692,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92771236,"statfs":{"total":95995035648,"available":94997745664,"internally_reserved":0,"allocated":507588608,"data_stored":506278288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150018,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1537584,"kb_used_data":910816,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92207568,"statfs":{"total":95995035648,"available":94420549632,"internally_reserved":0,"allocated":932675584,"data_stored":931357288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":641792455},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200436,"kb_used_data":794404,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92544716,"statfs":{"total":95995035648,"available":94765789184,"internally_reserved":0,"allocated":813469696,"data_stored":812147328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830471,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747296,"kb_used_data":497232,"kb_used_omap":87,"kb_used_meta":249960,"kb_avail":92997856,"statfs":{"total":95995035648,"available":95229804544,"internally_reserved":0,"allocated":509165568,"data_stored":507851176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":255959199},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738706,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839144,"kb_used_data":240152,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92906008,"statfs":{"total":95995035648,"available":95135752192,"internally_reserved":0,"allocated":245915648,"data_stored":244600904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238092288,"data_stored":238092288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":495259648,"data_stored":495255576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":800460800,"data_stored":800456728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":921960448,"data_stored":921960448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":492699648,"data_stored":492699648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":256311296,"data_stored":256311296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":363528192,"data_stored":363528192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":719912960,"data_stored":719912960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":376700928,"data_stored":376700928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":555941888,"data_stored":555941888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1121845248,"data_stored":1121845248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":357634048,"data_stored":357629976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:48:08.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-08-30T23:48:08.612 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:08.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:08.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:08.773 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:08.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:08.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:08.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:08.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:09.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:09.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:09.106 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:09.106 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:09.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:09.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:09.209 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:08 smithi106 ceph-mon[16603]: pgmap v1564: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.7 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 221/82026 objects degraded (0.269%); 1263/82026 objects misplaced (1.540%); 7.7 MiB/s, 123 objects/s recovering 2024-08-30T23:48:09.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:08 smithi136 ceph-mon[20207]: pgmap v1564: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.7 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 221/82026 objects degraded (0.269%); 1263/82026 objects misplaced (1.540%); 7.7 MiB/s, 123 objects/s recovering 2024-08-30T23:48:09.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:08 smithi173 ceph-mon[20554]: pgmap v1564: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.7 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 719 op/s; 221/82026 objects degraded (0.269%); 1263/82026 objects misplaced (1.540%); 7.7 MiB/s, 123 objects/s recovering 2024-08-30T23:48:09.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.461 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:09.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:09.562 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:09.595 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:09.697 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:09.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:09.740 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:09.741 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:09.741 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:09.741 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:09.741 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:09.741 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:09.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:48:09.797 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:09.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:09.863 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:09.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:09.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:09.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:10.076 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:10.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:10.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:48:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1882836313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:10.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:10.201 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1882836313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:10.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:10.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:10.253 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:10.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:10.290 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:10.291 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1882836313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:10.330 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:10.330 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:10.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:10.374 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:10.374 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:10.388 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:10.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:10.407 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:10.407 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:10.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:10.510 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:10.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:10.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:175402 6.3 6:c069f373:::benchmark_data_smithi173_114000_object75604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:10.688714+0000", 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0033086729999999998, 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003338504, 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:10.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 175402 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688714+0000", 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688714+0000", 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688716+0000", 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6419999999999999e-06 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688723+0000", 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1629999999999999e-06 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688739+0000", 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5659999999999999e-05 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688746+0000", 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7880000000000001e-06 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688809+0000", 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2874999999999997e-05 2024-08-30T23:48:10.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688899+0000", 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0557999999999996e-05 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.688972+0000", 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3114000000000005e-05 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:175403 6.b e697/619 6:dda6cd0d:::benchmark_data_smithi173_114000_object75605:head v 697'11022, mlcod=697'11020)", 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:10.690050+0000", 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0019727870000000002, 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020683279999999999, 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690050+0000", 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:10.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690050+0000", 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690051+0000", 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690058+0000", 2024-08-30T23:48:10.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5169999999999996e-06 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690091+0000", 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3348e-05 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690098+0000", 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9990000000000002e-06 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:10.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690144+0000", 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6140999999999998e-05 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.690163+0000", 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8661e-05 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:175392 6.3 6:c5dd84a2:::benchmark_data_smithi173_114000_object75594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:10.674129+0000", 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.017893737999999999, 2024-08-30T23:48:10.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018040003999999998, 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 175392 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674129+0000", 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:10.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674129+0000", 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674131+0000", 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.44e-06 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674138+0000", 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4739999999999999e-06 2024-08-30T23:48:10.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674155+0000", 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7041000000000002e-05 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674163+0000", 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4120000000000004e-06 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674230+0000", 2024-08-30T23:48:10.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6982000000000003e-05 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674381+0000", 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000150796 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.674498+0000", 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117625 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:10.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.679673+0000", 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051743509999999998 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:10.679712+0000", 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9578e-05 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 3 2024-08-30T23:48:10.706 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:10.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:10.772 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:10.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:10.898 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:10.899 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:11.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:11.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:10 smithi106 ceph-mon[16603]: pgmap v1565: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.6 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 221/78867 objects degraded (0.280%); 923/78867 objects misplaced (1.170%); 7.9 MiB/s, 126 objects/s recovering 2024-08-30T23:48:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:11.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:10 smithi106 ceph-mon[16603]: osdmap e697: 12 total, 12 up, 11 in 2024-08-30T23:48:11.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 221/78867 objects degraded (0.280%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:11.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:10 smithi106 ceph-mon[16603]: osdmap e698: 12 total, 12 up, 11 in 2024-08-30T23:48:11.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:11.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:11.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:11.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:11.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:10 smithi136 ceph-mon[20207]: pgmap v1565: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.6 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 221/78867 objects degraded (0.280%); 923/78867 objects misplaced (1.170%); 7.9 MiB/s, 126 objects/s recovering 2024-08-30T23:48:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:10 smithi136 ceph-mon[20207]: osdmap e697: 12 total, 12 up, 11 in 2024-08-30T23:48:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 221/78867 objects degraded (0.280%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:10 smithi136 ceph-mon[20207]: osdmap e698: 12 total, 12 up, 11 in 2024-08-30T23:48:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:10 smithi173 ceph-mon[20554]: pgmap v1565: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.6 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 715 op/s; 221/78867 objects degraded (0.280%); 923/78867 objects misplaced (1.170%); 7.9 MiB/s, 126 objects/s recovering 2024-08-30T23:48:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:10 smithi173 ceph-mon[20554]: osdmap e697: 12 total, 12 up, 11 in 2024-08-30T23:48:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 221/78867 objects degraded (0.280%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:11.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:10 smithi173 ceph-mon[20554]: osdmap e698: 12 total, 12 up, 11 in 2024-08-30T23:48:11.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:11.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:11.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:48:11.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:11.533 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:11.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:11.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.659 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:11.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:11.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:11.802 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:11.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:11.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:11.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:11.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:12.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:12.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:12.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:12.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:12.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:12.299 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:12.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:12.450 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:48:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:48:12.450 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:48:12.450 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:12.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:12.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:12.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:12.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.687 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:12.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:12.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:12.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:12.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:12.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:13.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:13.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:12 smithi106 ceph-mon[16603]: pgmap v1568: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.5 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 221/75633 objects degraded (0.292%); 923/75633 objects misplaced (1.220%); 5.8 MiB/s, 93 objects/s recovering 2024-08-30T23:48:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:13.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:13.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:48:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:13.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:12 smithi136 ceph-mon[20207]: pgmap v1568: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.5 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 221/75633 objects degraded (0.292%); 923/75633 objects misplaced (1.220%); 5.8 MiB/s, 93 objects/s recovering 2024-08-30T23:48:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:13.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:13.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:12 smithi173 ceph-mon[20554]: pgmap v1568: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.5 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 687 op/s; 221/75633 objects degraded (0.292%); 923/75633 objects misplaced (1.220%); 5.8 MiB/s, 93 objects/s recovering 2024-08-30T23:48:13.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:13.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:13.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:13.456 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:13.498 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:13.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:13.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:13.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:13.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:13.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:13.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:13.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:13.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:13.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:13.968 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pgp_num to 21 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59003 5.2 5.20ed2842 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 581.89624794999997, 2024-08-30T23:48:14.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96588689100000003, 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59003 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:48:14.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147194+0000", 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147195+0000", 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.429e-06 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147224+0000", 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9014999999999999e-05 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147237+0000", 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.311e-05 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147244+0000", 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1160000000000004e-06 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147299+0000", 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5275000000000002e-05 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.147305+0000", 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.302e-06 2024-08-30T23:48:14.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113050+0000", 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.96574470800000001 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113081+0000", 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0936000000000003e-05 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59011 5.2 5.db576df2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 581.58530279199999, 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65497856600000004, 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59011 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458139+0000", 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458149+0000", 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8779999999999993e-06 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458186+0000", 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6718000000000003e-05 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458199+0000", 2024-08-30T23:48:14.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3811999999999999e-05 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458205+0000", 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6879999999999996e-06 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458260+0000", 2024-08-30T23:48:14.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.516e-05 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.458269+0000", 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3680000000000002e-06 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113109+0000", 2024-08-30T23:48:14.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65484074999999997 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113118+0000", 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1920000000000005e-06 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59016 5.2 5.92c60782 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 581.47347309600002, 2024-08-30T23:48:14.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54317872599999995, 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59016 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569969+0000", 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.569970+0000", 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2449999999999999e-06 2024-08-30T23:48:14.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570027+0000", 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7083999999999998e-05 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570039+0000", 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1664999999999999e-05 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570044+0000", 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6189999999999999e-06 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570095+0000", 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1085999999999998e-05 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:48:14.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.570103+0000", 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5830000000000001e-06 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113141+0000", 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54303761500000003 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113148+0000", 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8290000000000002e-06 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:59017 5.2 5.eccc78f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 581.31738913900006, 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38712284699999999, 2024-08-30T23:48:14.063 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 59017 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726053+0000", 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726060+0000", 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2359999999999998e-06 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726121+0000", 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0909000000000002e-05 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726151+0000", 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0292999999999999e-05 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726165+0000", 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.428e-05 2024-08-30T23:48:14.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726324+0000", 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015868300000000001 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:32.726349+0000", 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4508e-05 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113170+0000", 2024-08-30T23:48:14.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38682103699999998 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:33.113176+0000", 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.2 426/420 [PushReplyOp(5:4213227c:::benchmark_data_smithi173_87639_object5797:head)])", 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:48:14.068 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 571.52046745300004, 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64882206499999995, 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522974+0000", 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522976+0000", 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.136e-06 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:42.522980+0000", 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3109999999999999e-06 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169121+0000", 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64614092499999998 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169129+0000", 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0099999999999995e-06 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.169226+0000", 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7223000000000001e-05 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:43.171796+0000", 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025704600000000001 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.072 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.2 432/420 [PushReplyOp(5:4f272be2:::benchmark_data_smithi173_87639_object27138:head)])", 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 559.17116637499998, 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43174907800000001, 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872276+0000", 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872277+0000", 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.674e-06 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:54.872285+0000", 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4529999999999996e-06 2024-08-30T23:48:14.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302111+0000", 2024-08-30T23:48:14.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42982598799999999 2024-08-30T23:48:14.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302119+0000", 2024-08-30T23:48:14.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1990000000000006e-06 2024-08-30T23:48:14.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.302204+0000", 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5348999999999999e-05 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:38:55.304025+0000", 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018204149999999999 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 508/491 [PushOp(5:19507caf:::benchmark_data_smithi173_87639_object19937:head, version: 403'1241, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:19507caf:::benchmark_data_smithi173_87639_object19937:head@403'1241, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:14.077 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 395.21869521299999, 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32484032499999999, 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824747+0000", 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7899999999999998e-07 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:38.824799+0000", 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1171999999999997e-05 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148648+0000", 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32384927299999999 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148664+0000", 2024-08-30T23:48:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6316000000000001e-05 2024-08-30T23:48:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148733+0000", 2024-08-30T23:48:14.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8547000000000006e-05 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.148741+0000", 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000007e-06 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.149587+0000", 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084573799999999996 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 508/491 [PushOp(5:0c44bbbd:::benchmark_data_smithi173_87639_object9387:head, version: 400'619, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0c44bbbd:::benchmark_data_smithi173_87639_object9387:head@400'619, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:39.158016+0000", 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 394.88542579900002, 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66108408900000004, 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158016+0000", 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158016+0000", 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158017+0000", 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4600000000000004e-07 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.158100+0000", 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3633000000000004e-05 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818107+0000", 2024-08-30T23:48:14.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66000703199999999 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818125+0000", 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7620000000000001e-05 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818236+0000", 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011066200000000001 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.818250+0000", 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3889e-05 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:39.819100+0000", 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00085060699999999997 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 510/491 [PushOp(5:1f90ba55:::benchmark_data_smithi173_87639_object26324:head, version: 406'1649, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:1f90ba55:::benchmark_data_smithi173_87639_object26324:head@406'1649, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:40.895932+0000", 2024-08-30T23:48:14.087 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 393.14751001100001, 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.85628814200000003, 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.895932+0000", 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.895932+0000", 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.895933+0000", 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2900000000000003e-07 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:40.896004+0000", 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1261999999999996e-05 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750707+0000", 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.85470286699999998 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750727+0000", 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0136000000000002e-05 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750825+0000", 2024-08-30T23:48:14.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7887999999999995e-05 2024-08-30T23:48:14.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.750842+0000", 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7093000000000001e-05 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:41.752220+0000", 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001378167 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.18 511/491 [PushOp(5:192c7a90:::benchmark_data_smithi173_87639_object30798:head, version: 406'1932, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:192c7a90:::benchmark_data_smithi173_87639_object30798:head@406'1932, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:41:43.337860+0000", 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 390.70558168700001, 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64375286899999995, 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.093 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337860+0000", 2024-08-30T23:48:14.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337860+0000", 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337862+0000", 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.48e-06 2024-08-30T23:48:14.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.337926+0000", 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4425999999999996e-05 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980456+0000", 2024-08-30T23:48:14.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64252941900000005 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980483+0000", 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7291999999999999e-05 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980716+0000", 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000232711 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.980730+0000", 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4714e-05 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:41:43.981613+0000", 2024-08-30T23:48:14.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00088282699999999998 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.10 520/491 [PushOp(5:0bdf7ebf:::benchmark_data_smithi173_87639_object40617:head, version: 408'2576, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:0bdf7ebf:::benchmark_data_smithi173_87639_object40617:head@408'2576, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:04.965652+0000", 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 369.07778955700002, 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.564411475, 2024-08-30T23:48:14.100 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965652+0000", 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965652+0000", 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965653+0000", 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8299999999999996e-07 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:04.965763+0000", 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109967 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529125+0000", 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.563361628 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529139+0000", 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4122e-05 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529258+0000", 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119528 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.529273+0000", 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5091e-05 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:05.530064+0000", 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00079045600000000004 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:109334 5.2 5:41d12184:::benchmark_data_smithi173_87639_object29094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e552)", 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:00.875893+0000", 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 313.16754840300001, 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64271831899999998, 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 109334 2024-08-30T23:48:14.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875893+0000", 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875893+0000", 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875895+0000", 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6619999999999999e-06 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875903+0000", 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2679999999999994e-06 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875915+0000", 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.168e-05 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875923+0000", 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4959999999999999e-06 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.875979+0000", 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6166999999999999e-05 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.876101+0000", 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012249400000000001 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.876215+0000", 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113727 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.877915+0000", 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001700072 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:00.878975+0000", 2024-08-30T23:48:14.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010602820000000001 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:01.518471+0000", 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63949578799999995 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:48:14.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:01.518549+0000", 2024-08-30T23:48:14.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000001e-05 2024-08-30T23:48:14.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:01.518612+0000", 2024-08-30T23:48:14.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2582999999999994e-05 2024-08-30T23:48:14.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:110552 5.2 5:413513cd:::benchmark_data_smithi173_87639_object30312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:03.446594+0000", 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 310.59684744499998, 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35024460600000001, 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 110552 2024-08-30T23:48:14.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446594+0000", 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446594+0000", 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446596+0000", 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7969999999999999e-06 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446603+0000", 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7240000000000003e-06 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446618+0000", 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4856e-05 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446626+0000", 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0930000000000003e-06 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446692+0000", 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6222000000000004e-05 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446824+0000", 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013175099999999999 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,10", 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.446949+0000", 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012488899999999999 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.448987+0000", 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020381689999999998 2024-08-30T23:48:14.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.449011+0000", 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4294000000000002e-05 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.796773+0000", 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34776191000000001 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.796811+0000", 2024-08-30T23:48:14.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8305e-05 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:03.796839+0000", 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7596000000000001e-05 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:123085 5.18 5:1e3a285a:::benchmark_data_smithi173_87639_object42845:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e568)", 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:25.945954+0000", 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 288.09748793599999, 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.32379032200000002, 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 123085 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945954+0000", 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945954+0000", 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945955+0000", 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945959+0000", 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4970000000000003e-06 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945969+0000", 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7710000000000001e-06 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.945976+0000", 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0430000000000002e-06 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.946260+0000", 2024-08-30T23:48:14.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028371000000000001 2024-08-30T23:48:14.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.946361+0000", 2024-08-30T23:48:14.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010116 2024-08-30T23:48:14.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,11", 2024-08-30T23:48:14.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.946500+0000", 2024-08-30T23:48:14.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138641 2024-08-30T23:48:14.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.949021+0000", 2024-08-30T23:48:14.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025208169999999998 2024-08-30T23:48:14.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.949255+0000", 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000234315 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.954252+0000", 2024-08-30T23:48:14.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049968130000000001 2024-08-30T23:48:14.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268291+0000", 2024-08-30T23:48:14.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.314038918 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268418+0000", 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012726400000000001 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.269744+0000", 2024-08-30T23:48:14.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001326455 2024-08-30T23:48:14.132 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.132 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.132 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.133 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.26173.0:123085 5.18 e568/540 ondisk, result = 0)", 2024-08-30T23:48:14.133 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:25.950853+0000", 2024-08-30T23:48:14.133 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 288.09258893100002, 2024-08-30T23:48:14.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31889914899999999, 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950853+0000", 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950853+0000", 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950854+0000", 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9299999999999996e-07 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:25.950859+0000", 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5729999999999998e-06 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268162+0000", 2024-08-30T23:48:14.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31730273599999997 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268172+0000", 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0207000000000001e-05 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268253+0000", 2024-08-30T23:48:14.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0877000000000005e-05 2024-08-30T23:48:14.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.268267+0000", 2024-08-30T23:48:14.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4104000000000001e-05 2024-08-30T23:48:14.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:26.269752+0000", 2024-08-30T23:48:14.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001484759 2024-08-30T23:48:14.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.141 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(5.18 573/540 [PushReplyOp(5:1937383b:::benchmark_data_smithi173_87639_object49573:head)])", 2024-08-30T23:48:14.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:48:14.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 279.71253027799997, 2024-08-30T23:48:14.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34516386599999999, 2024-08-30T23:48:14.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:48:14.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330912+0000", 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0699999999999996e-07 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.330917+0000", 2024-08-30T23:48:14.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9980000000000003e-06 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.674652+0000", 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34373422100000001 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.674662+0000", 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0212e-05 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.674726+0000", 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4642999999999998e-05 2024-08-30T23:48:14.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:34.676075+0000", 2024-08-30T23:48:14.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001348985 2024-08-30T23:48:14.148 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.148 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:129319 5.18 5:1a97d0aa:::benchmark_data_smithi173_87639_object49079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e575)", 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:36.370255+0000", 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 277.67318695699998, 2024-08-30T23:48:14.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30291225700000002, 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 129319 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370255+0000", 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370255+0000", 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370258+0000", 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9280000000000002e-06 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370266+0000", 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5120000000000008e-06 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370286+0000", 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0004999999999999e-05 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370296+0000", 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5580000000000006e-06 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370368+0000", 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2519000000000001e-05 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370495+0000", 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000126377 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,4,11", 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.370601+0000", 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106165 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.372553+0000", 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019519629999999999 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.372633+0000", 2024-08-30T23:48:14.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0439999999999996e-05 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.374178+0000", 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015449210000000001 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.672510+0000", 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29833189900000001 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:48:14.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.672578+0000", 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7566999999999999e-05 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.673167+0000", 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058940299999999995 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(5.19 e599,595 5:99c57c29:::benchmark_data_smithi173_87639_object64709:head,592'7134)", 2024-08-30T23:48:14.156 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 243.69021910399999, 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41795327700000001, 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3300000000000002e-07 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353228+0000", 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1930000000000003e-06 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771046+0000", 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41781854200000001 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771056+0000", 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771111+0000", 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5040999999999997e-05 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771118+0000", 2024-08-30T23:48:14.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771176+0000", 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8e-05 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.161 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:93442 6.a 6:562d5a8b:::benchmark_data_smithi173_114000_object93441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:01.898824+0000", 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 132.14461753399999, 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.289915007, 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:48:14.162 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93442 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898824+0000", 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898824+0000", 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898826+0000", 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.866e-06 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898869+0000", 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2790000000000002e-05 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898886+0000", 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7277000000000001e-05 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898895+0000", 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1419999999999999e-06 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.898951+0000", 2024-08-30T23:48:14.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5151000000000001e-05 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.899059+0000", 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108208 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.899146+0000", 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7169000000000005e-05 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:01.901661+0000", 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025145739999999999 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.030800+0000", 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12913902999999999 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.188663+0000", 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.15786315200000001 2024-08-30T23:48:14.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.188706+0000", 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3025000000000002e-05 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.188739+0000", 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3623e-05 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.168 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:155747 6.3 6:c7cf5f70:::benchmark_data_smithi173_114000_object55949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e684)", 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:39.838529+0000", 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 34.204913138000002, 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.33831727, 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:48:14.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155747 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838529+0000", 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838529+0000", 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:14.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838531+0000", 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.962e-06 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838539+0000", 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-06 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:14.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838558+0000", 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9082000000000001e-05 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838569+0000", 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1069e-05 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838628+0000", 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8347999999999998e-05 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838722+0000", 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4955000000000001e-05 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838799+0000", 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6344e-05 2024-08-30T23:48:14.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.842747+0000", 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039481250000000002 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.842865+0000", 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117719 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.176770+0000", 2024-08-30T23:48:14.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.333905754 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.176811+0000", 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0238999999999999e-05 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.176846+0000", 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5333000000000002e-05 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:14.176 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:14.176 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:14.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:14.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:14.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:13 smithi106 ceph-mon[16603]: osdmap e699: 12 total, 12 up, 11 in 2024-08-30T23:48:14.282 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1791697571' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:13 smithi136 ceph-mon[20207]: osdmap e699: 12 total, 12 up, 11 in 2024-08-30T23:48:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1791697571' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:14.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:13 smithi173 ceph-mon[20554]: osdmap e699: 12 total, 12 up, 11 in 2024-08-30T23:48:14.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1791697571' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:14.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:14.444 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:14.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:14.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:14.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:14.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:14.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:14.839 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:14.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:14.966 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:14.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:14.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:14.976 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:48:15.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:15.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:15.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:48:15.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:15.236 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:15.257 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:14 smithi106 ceph-mon[16603]: pgmap v1570: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 541 op/s; 221/72276 objects degraded (0.306%); 923/72276 objects misplaced (1.277%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:48:15.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1791697571' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-08-30T23:48:15.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:14 smithi106 ceph-mon[16603]: osdmap e700: 12 total, 12 up, 11 in 2024-08-30T23:48:15.258 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1581587880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:48:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:14 smithi136 ceph-mon[20207]: pgmap v1570: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 541 op/s; 221/72276 objects degraded (0.306%); 923/72276 objects misplaced (1.277%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:48:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1791697571' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-08-30T23:48:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:14 smithi136 ceph-mon[20207]: osdmap e700: 12 total, 12 up, 11 in 2024-08-30T23:48:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1581587880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:48:15.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:14 smithi173 ceph-mon[20554]: pgmap v1570: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.5 GiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 541 op/s; 221/72276 objects degraded (0.306%); 923/72276 objects misplaced (1.277%); 3.5 MiB/s, 56 objects/s recovering 2024-08-30T23:48:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1791697571' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-08-30T23:48:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:14 smithi173 ceph-mon[20554]: osdmap e700: 12 total, 12 up, 11 in 2024-08-30T23:48:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1581587880' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:48:15.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.389 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:15.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:15.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:15.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:15.621 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.642 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:15.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:15.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:15.767 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.768 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:15.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:48:15.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:15.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:15.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:15.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:16.010 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:15 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1581587880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:48:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:15 smithi136 ceph-mon[20207]: osdmap e701: 12 total, 12 up, 11 in 2024-08-30T23:48:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:15 smithi136 ceph-mon[20207]: 6.5 scrub starts 2024-08-30T23:48:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:15 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 221/72276 objects degraded (0.306%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:15 smithi136 ceph-mon[20207]: osdmap e702: 12 total, 12 up, 11 in 2024-08-30T23:48:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.052 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:16.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:16.113 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:16.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:16.156 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.199 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:16.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:16.254 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:15 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:16.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1581587880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:48:16.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:15 smithi106 ceph-mon[16603]: osdmap e701: 12 total, 12 up, 11 in 2024-08-30T23:48:16.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:15 smithi106 ceph-mon[16603]: 6.5 scrub starts 2024-08-30T23:48:16.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:15 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 221/72276 objects degraded (0.306%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:16.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:15 smithi106 ceph-mon[16603]: osdmap e702: 12 total, 12 up, 11 in 2024-08-30T23:48:16.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:16.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:15 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1581587880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:48:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:15 smithi173 ceph-mon[20554]: osdmap e701: 12 total, 12 up, 11 in 2024-08-30T23:48:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:15 smithi173 ceph-mon[20554]: 6.5 scrub starts 2024-08-30T23:48:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:15 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 221/72276 objects degraded (0.306%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:15 smithi173 ceph-mon[20554]: osdmap e702: 12 total, 12 up, 11 in 2024-08-30T23:48:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:16.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:16.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:16.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.591 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:16.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:16.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:179840 6.c e702/619 6:3f37e1bc:::benchmark_data_smithi173_114000_object80042:head v 702'11068, mlcod=702'11066)", 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:16.699929+0000", 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0052840930000000001, 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0053048239999999997, 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:16.706 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.699929+0000", 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.699929+0000", 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.699930+0000", 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0869999999999999e-06 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.699938+0000", 2024-08-30T23:48:16.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4870000000000002e-06 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.699977+0000", 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8043999999999997e-05 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.699985+0000", 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9819999999999997e-06 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.700053+0000", 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7675999999999994e-05 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.700076+0000", 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2602000000000001e-05 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.710 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:179841 6.1 e702/690 6:83c63289:::benchmark_data_smithi173_114000_object80043:head v 702'11258, mlcod=702'11255)", 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:16.701457+0000", 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0037561349999999999, 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038171860000000002, 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701457+0000", 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701457+0000", 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701458+0000", 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1099999999999995e-07 2024-08-30T23:48:16.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701463+0000", 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8989999999999998e-06 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:16.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701496+0000", 2024-08-30T23:48:16.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2971e-05 2024-08-30T23:48:16.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:16.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701504+0000", 2024-08-30T23:48:16.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4989999999999997e-06 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701561+0000", 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7136000000000003e-05 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.701580+0000", 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9204000000000001e-05 2024-08-30T23:48:16.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:179842 6.c e702/619 6:3b009345:::benchmark_data_smithi173_114000_object80044:head v 702'11069, mlcod=702'11066)", 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:16.704723+0000", 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00049012199999999995, 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059814000000000004, 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:48:16.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704723+0000", 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704723+0000", 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704724+0000", 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704730+0000", 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5119999999999997e-06 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704757+0000", 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7121999999999999e-05 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704763+0000", 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1070000000000004e-06 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704815+0000", 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2111000000000003e-05 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:48:16.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:16.704835+0000", 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0160999999999999e-05 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 3 2024-08-30T23:48:16.720 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:16.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:16.824 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:16.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:16.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:16.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:16.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:17.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:17.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:17.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:17.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:16 smithi136 ceph-mon[20207]: 6.5 scrub ok 2024-08-30T23:48:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:16 smithi136 ceph-mon[20207]: pgmap v1573: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 221/68196 objects degraded (0.324%); 593/68196 objects misplaced (0.870%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:48:17.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3159725250' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:16 smithi173 ceph-mon[20554]: 6.5 scrub ok 2024-08-30T23:48:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:16 smithi173 ceph-mon[20554]: pgmap v1573: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 221/68196 objects degraded (0.324%); 593/68196 objects misplaced (0.870%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:48:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3159725250' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:17.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:17.355 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:16 smithi106 ceph-mon[16603]: 6.5 scrub ok 2024-08-30T23:48:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:16 smithi106 ceph-mon[16603]: pgmap v1573: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.4 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 221/68196 objects degraded (0.324%); 593/68196 objects misplaced (0.870%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:48:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3159725250' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:17.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.479 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:17.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:17.515 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:17.515 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:17.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:48:17.539 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:17.539 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:17.570 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:17.570 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:17.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:17.601 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:17.601 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:17.633 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:17.642 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:17.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:17.672 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:17.672 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:17.701 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:17.701 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:17.744 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:17.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:17.833 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:17.833 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:17.863 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:17.863 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:17.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:17.892 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:17.892 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:17.902 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:17.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:17.904 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:17.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:17.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:17.950 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:17.950 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:17.998 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:48:18.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:18.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:17 smithi106 ceph-mon[16603]: 6.4 scrub starts 2024-08-30T23:48:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:17 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:18.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:17 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:17 smithi106 ceph-mon[16603]: osdmap e703: 12 total, 12 up, 11 in 2024-08-30T23:48:18.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:17 smithi106 ceph-mon[16603]: 6.4 scrub ok 2024-08-30T23:48:18.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:18.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:17 smithi136 ceph-mon[20207]: 6.4 scrub starts 2024-08-30T23:48:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:17 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:18.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:17 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:18.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:17 smithi136 ceph-mon[20207]: osdmap e703: 12 total, 12 up, 11 in 2024-08-30T23:48:18.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:17 smithi136 ceph-mon[20207]: 6.4 scrub ok 2024-08-30T23:48:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:17 smithi173 ceph-mon[20554]: 6.4 scrub starts 2024-08-30T23:48:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:17 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:17 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:17 smithi173 ceph-mon[20554]: osdmap e703: 12 total, 12 up, 11 in 2024-08-30T23:48:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:17 smithi173 ceph-mon[20554]: 6.4 scrub ok 2024-08-30T23:48:18.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:18.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:18.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:18.474 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:18.474 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:18.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:18.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:18.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:18.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:18.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:18.825 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:18.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:18.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:18.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:18.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:18.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:19.100 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:19.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: pgmap v1576: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 903 op/s; 221/61431 objects degraded (0.360%); 593/61431 objects misplaced (0.965%); 5.2 MiB/s, 82 objects/s recovering 2024-08-30T23:48:19.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:48:19.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:48:19.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:19.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:19.103 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: osdmap e704: 12 total, 12 up, 11 in 2024-08-30T23:48:19.104 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:48:19.104 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:48:19.104 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:18 smithi136 ceph-mon[20207]: 6.a scrub starts 2024-08-30T23:48:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.122 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:19.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:19.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:48:19.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:19.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: pgmap v1576: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 903 op/s; 221/61431 objects degraded (0.360%); 593/61431 objects misplaced (0.965%); 5.2 MiB/s, 82 objects/s recovering 2024-08-30T23:48:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:48:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:48:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: osdmap e704: 12 total, 12 up, 11 in 2024-08-30T23:48:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:48:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:48:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:18 smithi173 ceph-mon[20554]: 6.a scrub starts 2024-08-30T23:48:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: pgmap v1576: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.3 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 903 op/s; 221/61431 objects degraded (0.360%); 593/61431 objects misplaced (0.965%); 5.2 MiB/s, 82 objects/s recovering 2024-08-30T23:48:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:48:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:48:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: osdmap e704: 12 total, 12 up, 11 in 2024-08-30T23:48:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:48:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:48:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:18 smithi106 ceph-mon[16603]: 6.a scrub starts 2024-08-30T23:48:19.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:19.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.384 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:19.443 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:19.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:19.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:19.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:19.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:19.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:19.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:19.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:19.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:19.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:19.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:20.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:20.029 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:20.029 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:20.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:20.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:20.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:20.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:20 smithi136 ceph-mon[20207]: 6.d scrub starts 2024-08-30T23:48:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:20 smithi136 ceph-mon[20207]: 6.2 scrub starts 2024-08-30T23:48:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:20 smithi136 ceph-mon[20207]: 6.d scrub ok 2024-08-30T23:48:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:20 smithi136 ceph-mon[20207]: 6.2 scrub ok 2024-08-30T23:48:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:20 smithi136 ceph-mon[20207]: 6.a scrub ok 2024-08-30T23:48:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:20 smithi136 ceph-mon[20207]: 6.c scrub starts 2024-08-30T23:48:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:20.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:20.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:48:20.310 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:48:20.310 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-08-30T23:48:20.310 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:48:20.310 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:48:20.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-08-30T23:48:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:20 smithi173 ceph-mon[20554]: 6.d scrub starts 2024-08-30T23:48:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:20 smithi173 ceph-mon[20554]: 6.2 scrub starts 2024-08-30T23:48:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:20 smithi173 ceph-mon[20554]: 6.d scrub ok 2024-08-30T23:48:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:20 smithi173 ceph-mon[20554]: 6.2 scrub ok 2024-08-30T23:48:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:20 smithi173 ceph-mon[20554]: 6.a scrub ok 2024-08-30T23:48:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:20 smithi173 ceph-mon[20554]: 6.c scrub starts 2024-08-30T23:48:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:20 smithi106 ceph-mon[16603]: 6.d scrub starts 2024-08-30T23:48:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:20 smithi106 ceph-mon[16603]: 6.2 scrub starts 2024-08-30T23:48:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:20 smithi106 ceph-mon[16603]: 6.d scrub ok 2024-08-30T23:48:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:20 smithi106 ceph-mon[16603]: 6.2 scrub ok 2024-08-30T23:48:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:20 smithi106 ceph-mon[16603]: 6.a scrub ok 2024-08-30T23:48:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:20 smithi106 ceph-mon[16603]: 6.c scrub starts 2024-08-30T23:48:20.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:20.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:20.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:20.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:20.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:20.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:20.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:48:20.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:20.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:20.890 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:20.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:20.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:20.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:21.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:21.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:21.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:21 smithi106 ceph-mon[16603]: pgmap v1578: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 221/57183 objects degraded (0.386%); 263/57183 objects misplaced (0.460%); 4.4 MiB/s, 70 objects/s recovering 2024-08-30T23:48:21.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:21 smithi106 ceph-mon[16603]: 6.c scrub ok 2024-08-30T23:48:21.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:21 smithi106 ceph-mon[16603]: 6.0 scrub starts 2024-08-30T23:48:21.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 221/57183 objects degraded (0.386%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:21.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:21.197 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:21 smithi136 ceph-mon[20207]: pgmap v1578: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 221/57183 objects degraded (0.386%); 263/57183 objects misplaced (0.460%); 4.4 MiB/s, 70 objects/s recovering 2024-08-30T23:48:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:21 smithi136 ceph-mon[20207]: 6.c scrub ok 2024-08-30T23:48:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:21 smithi136 ceph-mon[20207]: 6.0 scrub starts 2024-08-30T23:48:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 221/57183 objects degraded (0.386%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:21.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:21.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:21.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:21 smithi173 ceph-mon[20554]: pgmap v1578: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.2 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 221/57183 objects degraded (0.386%); 263/57183 objects misplaced (0.460%); 4.4 MiB/s, 70 objects/s recovering 2024-08-30T23:48:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:21 smithi173 ceph-mon[20554]: 6.c scrub ok 2024-08-30T23:48:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:21 smithi173 ceph-mon[20554]: 6.0 scrub starts 2024-08-30T23:48:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 221/57183 objects degraded (0.386%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:21.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:21.498 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:21.498 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:48:21.498 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:21.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:21.578 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:21.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:21.707 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:21.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:21.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:21.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:21.956 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:21.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:21.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:22.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:22.086 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:48:22.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:48:22.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 6.e scrub starts 2024-08-30T23:48:22.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 6.e scrub ok 2024-08-30T23:48:22.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 6.0 scrub ok 2024-08-30T23:48:22.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:48:22.087 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:22 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:48:22.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:22.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:22.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:22.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:48:22.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:48:22.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 6.e scrub starts 2024-08-30T23:48:22.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 6.e scrub ok 2024-08-30T23:48:22.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 6.0 scrub ok 2024-08-30T23:48:22.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:48:22.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:22 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:48:22.320 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:22.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:48:22.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:48:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 6.e scrub starts 2024-08-30T23:48:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 6.e scrub ok 2024-08-30T23:48:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 6.0 scrub ok 2024-08-30T23:48:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:48:22.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:22 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:48:22.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:22.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:22.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:22.559 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:22.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:22.588 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:22.588 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:22.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:22.618 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:22.619 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:22.647 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:22.648 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:22.675 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:22.676 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:22.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:22.705 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:22.705 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:22.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:22.736 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:22.736 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:22.765 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:22.766 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:22.793 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:22.794 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:22.816 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:22.816 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:22.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:22.845 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:22.845 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:22.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:22.872 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:22.872 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:22.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:22.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:22.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:23.024 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:48:23.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:23.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:23.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:23.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: pgmap v1579: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 221/53778 objects degraded (0.411%); 263/53778 objects misplaced (0.489%); 3.4 MiB/s, 54 objects/s recovering 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3123232327' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:23.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3009401576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: pgmap v1579: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 221/53778 objects degraded (0.411%); 263/53778 objects misplaced (0.489%); 3.4 MiB/s, 54 objects/s recovering 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3123232327' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3009401576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:23.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:23.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: pgmap v1579: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.1 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 221/53778 objects degraded (0.411%); 263/53778 objects misplaced (0.489%); 3.4 MiB/s, 54 objects/s recovering 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3123232327' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:23.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3009401576' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:23.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.369 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:23.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:23.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:23.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:23.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:23.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:23.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:48:23.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:23.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:23.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:23.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:23.993 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:24.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:24.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:24.060 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:24.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:24.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.150 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:24.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:24.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:24 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:48:24.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:24 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:24.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:24 smithi106 ceph-mon[16603]: osdmap e705: 12 total, 12 up, 11 in 2024-08-30T23:48:24.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:24 smithi106 ceph-mon[16603]: 6.b scrub starts 2024-08-30T23:48:24.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:24 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:48:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:24 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:24 smithi136 ceph-mon[20207]: osdmap e705: 12 total, 12 up, 11 in 2024-08-30T23:48:24.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:24 smithi136 ceph-mon[20207]: 6.b scrub starts 2024-08-30T23:48:24.291 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:24 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:48:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:24 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:24 smithi173 ceph-mon[20554]: osdmap e705: 12 total, 12 up, 11 in 2024-08-30T23:48:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:24 smithi173 ceph-mon[20554]: 6.b scrub starts 2024-08-30T23:48:24.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:24.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:24.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:24.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:24.648 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:24.655 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:185426 6.9 6:9f1d77da:::benchmark_data_smithi173_114000_object85628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e705)", 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:24.614415+0000", 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.040817694000000002, 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.040835297, 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185426 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614415+0000", 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614415+0000", 2024-08-30T23:48:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614416+0000", 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2139999999999999e-06 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614423+0000", 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4239999999999998e-06 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614438+0000", 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4721e-05 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614445+0000", 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9369999999999998e-06 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614495+0000", 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9968e-05 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614586+0000", 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1439000000000003e-05 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.614666+0000", 2024-08-30T23:48:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0302000000000005e-05 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.623903+0000", 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0092365999999999993 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.623965+0000", 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2638999999999994e-05 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:185427 6.9 6:997681d6:::benchmark_data_smithi173_114000_object85629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e705)", 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:24.616856+0000", 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.038376642000000002, 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.038440895000000003, 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:48:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185427 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.616856+0000", 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.616856+0000", 2024-08-30T23:48:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.616857+0000", 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0800000000000004e-07 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.616861+0000", 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6959999999999998e-06 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.616871+0000", 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7939999999999997e-06 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.616876+0000", 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6999999999999996e-06 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.618536+0000", 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001659755 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.618682+0000", 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014579299999999999 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.618828+0000", 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000146491 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.624030+0000", 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052012880000000001 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.624043+0000", 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2897999999999999e-05 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:185437 6.9 6:9e7a3c47:::benchmark_data_smithi173_114000_object85639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e705)", 2024-08-30T23:48:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:24.645690+0000", 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0095425779999999995, 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0096480390000000006, 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185437 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645690+0000", 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645690+0000", 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645692+0000", 2024-08-30T23:48:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9290000000000001e-06 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645700+0000", 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7200000000000006e-06 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645716+0000", 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6418000000000001e-05 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645724+0000", 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3939999999999998e-06 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645787+0000", 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2891999999999993e-05 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.645935+0000", 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000148145 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.646082+0000", 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014751400000000001 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650151+0000", 2024-08-30T23:48:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040688180000000001 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650275+0000", 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012429499999999999 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:185440 6.1 6:85c7387f:::benchmark_data_smithi173_114000_object85642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e705)", 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:24.650606+0000", 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.004627394, 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047698749999999998, 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185440 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650606+0000", 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650606+0000", 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650607+0000", 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1480000000000001e-06 2024-08-30T23:48:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650611+0000", 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.087e-06 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650620+0000", 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4609999999999997e-06 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650625+0000", 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7419999999999997e-06 2024-08-30T23:48:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650665+0000", 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9867999999999998e-05 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650792+0000", 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012719599999999999 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.678 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.650916+0000", 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0001239 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:24.655055+0000", 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004138849 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.679 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:24.680 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:48:24.680 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 4 2024-08-30T23:48:24.680 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:24.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:24.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:24.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:24.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:24.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:24.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:25.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:25.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:25.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: pgmap v1581: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 700 op/s; 221/50661 objects degraded (0.436%); 263/50661 objects misplaced (0.519%); 3.1 MiB/s, 49 objects/s recovering 2024-08-30T23:48:25.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: 6.1 scrub starts 2024-08-30T23:48:25.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: 6.b scrub ok 2024-08-30T23:48:25.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:25.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:25.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: 6.1 scrub ok 2024-08-30T23:48:25.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:48:25.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:48:25.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:25 smithi106 ceph-mon[16603]: 6.7 scrub starts 2024-08-30T23:48:25.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:25.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:25.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:25.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:25.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: pgmap v1581: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 700 op/s; 221/50661 objects degraded (0.436%); 263/50661 objects misplaced (0.519%); 3.1 MiB/s, 49 objects/s recovering 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: 6.1 scrub starts 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: 6.b scrub ok 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: 6.1 scrub ok 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:48:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:25 smithi173 ceph-mon[20554]: 6.7 scrub starts 2024-08-30T23:48:25.383 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: pgmap v1581: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 36 active+clean; 1.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 700 op/s; 221/50661 objects degraded (0.436%); 263/50661 objects misplaced (0.519%); 3.1 MiB/s, 49 objects/s recovering 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: 6.1 scrub starts 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: 6.b scrub ok 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: 6.1 scrub ok 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:48:25.384 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:25 smithi136 ceph-mon[20207]: 6.7 scrub starts 2024-08-30T23:48:25.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:25.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:25.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:25.551 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:25.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:25.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:48:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:25.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:25.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:25.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:25.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:25.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:25.923 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:25.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:25.930 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:25.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:48:26.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:26.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:26.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:26.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:26.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:26.286 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:26.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:26 smithi173 ceph-mon[20554]: 6.8 scrub starts 2024-08-30T23:48:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:26 smithi173 ceph-mon[20554]: 6.8 scrub ok 2024-08-30T23:48:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:26 smithi173 ceph-mon[20554]: 6.7 scrub ok 2024-08-30T23:48:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:26 smithi173 ceph-mon[20554]: osdmap e706: 12 total, 12 up, 11 in 2024-08-30T23:48:26.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 221/50661 objects degraded (0.436%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:26 smithi106 ceph-mon[16603]: 6.8 scrub starts 2024-08-30T23:48:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:26 smithi106 ceph-mon[16603]: 6.8 scrub ok 2024-08-30T23:48:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:26 smithi106 ceph-mon[16603]: 6.7 scrub ok 2024-08-30T23:48:26.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:26 smithi106 ceph-mon[16603]: osdmap e706: 12 total, 12 up, 11 in 2024-08-30T23:48:26.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 221/50661 objects degraded (0.436%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:26.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:26.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:26.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:48:26.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:26 smithi136 ceph-mon[20207]: 6.8 scrub starts 2024-08-30T23:48:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:26 smithi136 ceph-mon[20207]: 6.8 scrub ok 2024-08-30T23:48:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:26 smithi136 ceph-mon[20207]: 6.7 scrub ok 2024-08-30T23:48:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:26 smithi136 ceph-mon[20207]: osdmap e706: 12 total, 12 up, 11 in 2024-08-30T23:48:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 221/50661 objects degraded (0.436%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:26.559 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:26.631 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:48:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:48:26.631 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:48:26.632 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:26.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:26.709 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:26.811 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:26.849 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:26.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:26.986 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:26.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:27.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:27.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:27.213 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:27.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:27.238 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: pgmap v1583: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 35 active+clean; 974 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 96/46743 objects degraded (0.205%); 4.8 MiB/s, 93 objects/s recovering 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: osdmap e707: 12 total, 12 up, 11 in 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: 6.6 scrub starts 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:48:27.297 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:27 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:48:27.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:48:27.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:27.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: pgmap v1583: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 35 active+clean; 974 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 96/46743 objects degraded (0.205%); 4.8 MiB/s, 93 objects/s recovering 2024-08-30T23:48:27.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:48:27.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: osdmap e707: 12 total, 12 up, 11 in 2024-08-30T23:48:27.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:27.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:27.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:48:27.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: 6.6 scrub starts 2024-08-30T23:48:27.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:48:27.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:27 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:48:27.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:27.396 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:48:27.397 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:48:27.397 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-08-30T23:48:27.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd in 10 2024-08-30T23:48:27.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:27.534 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: pgmap v1583: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 35 active+clean; 974 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 637 op/s; 96/46743 objects degraded (0.205%); 4.8 MiB/s, 93 objects/s recovering 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: osdmap e707: 12 total, 12 up, 11 in 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: 6.6 scrub starts 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:48:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:27 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:48:27.615 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:27.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:27.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:27.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:27.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:27.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:27.845 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:27.846 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:27.872 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:27.876 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:27.877 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:27.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:27.912 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:27.913 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:27.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:27.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:27.946 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:27.946 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:27.984 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:27.984 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:28.014 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:28.015 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:28.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:28.048 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:28.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.066 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:28.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:28.069 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:48:28.084 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:28.084 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:28.131 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:28.132 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:28.162 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:28.162 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:28.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:28.188 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:28.188 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:28.196 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: 6.6 scrub ok 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: osdmap e708: 12 total, 12 up, 11 in 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: 6.f scrub starts 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2200447869' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:28.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3971613098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:28.197 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:28.198 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:28.203 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:28.229 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:28.230 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:28.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: 6.6 scrub ok 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: osdmap e708: 12 total, 12 up, 11 in 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: 6.f scrub starts 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2200447869' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:28.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3971613098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:28.332 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:28.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:28.464 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: 6.6 scrub ok 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: osdmap e708: 12 total, 12 up, 11 in 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: 6.f scrub starts 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2200447869' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3971613098' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:28.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:28.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:28.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:28.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:28.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:28.862 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:28.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:48:28.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:28.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:28.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:29.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:29.120 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:29.208 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:29.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:29.253 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: pgmap v1586: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 34 active+clean; 855 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 96/41001 objects degraded (0.234%); 3.5 MiB/s, 83 objects/s recovering 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: 6.f scrub ok 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2200447869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:29.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:29 smithi173 ceph-mon[20554]: osdmap e709: 12 total, 12 up, 11 in 2024-08-30T23:48:29.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: pgmap v1586: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 34 active+clean; 855 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 96/41001 objects degraded (0.234%); 3.5 MiB/s, 83 objects/s recovering 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: 6.f scrub ok 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2200447869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-mon[16603]: osdmap e709: 12 total, 12 up, 11 in 2024-08-30T23:48:29.392 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:29.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:29.528 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: pgmap v1586: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 34 active+clean; 855 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 691 op/s; 96/41001 objects degraded (0.234%); 3.5 MiB/s, 83 objects/s recovering 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: 6.f scrub ok 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2200447869' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:29 smithi136 ceph-mon[20207]: osdmap e709: 12 total, 12 up, 11 in 2024-08-30T23:48:29.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:29.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:29.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:29.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:29.817 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:29 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:48:29.404+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e709 definitely_dead 0 2024-08-30T23:48:29.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:29.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:29.918 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:30.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:30.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:30.082 INFO:teuthology.orchestra.run.smithi106.stderr:marked in osd.10. 2024-08-30T23:48:30.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:30.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:30.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:30.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:30.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2051253489' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:48:30.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:30 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:48:30.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:30.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:30.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2051253489' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:48:30.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:30 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:48:30.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:30.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2051253489' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-08-30T23:48:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:30 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:48:30.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:30.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:30.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:30.760 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:30.760 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:30.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:30.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:30.895 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-08-30T23:48:30.907 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:30.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:31.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:31.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:31.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:31.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:31.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:31.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:31 smithi106 ceph-mon[16603]: pgmap v1588: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 34 active+clean; 803 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 589 op/s; 96/38541 objects degraded (0.249%) 2024-08-30T23:48:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2051253489' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:48:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:31 smithi106 ceph-mon[16603]: osdmap e710: 12 total, 12 up, 12 in 2024-08-30T23:48:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 96/38541 objects degraded (0.249%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:31.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:31 smithi106 ceph-mon[16603]: osdmap e711: 12 total, 12 up, 12 in 2024-08-30T23:48:31.381 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:31 smithi173 ceph-mon[20554]: pgmap v1588: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 34 active+clean; 803 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 589 op/s; 96/38541 objects degraded (0.249%) 2024-08-30T23:48:31.381 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2051253489' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:48:31.381 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:31 smithi173 ceph-mon[20554]: osdmap e710: 12 total, 12 up, 12 in 2024-08-30T23:48:31.381 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 96/38541 objects degraded (0.249%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:31.382 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:31 smithi173 ceph-mon[20554]: osdmap e711: 12 total, 12 up, 12 in 2024-08-30T23:48:31.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.405 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:31.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:31.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:48:31.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:31.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:31 smithi136 ceph-mon[20207]: pgmap v1588: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 34 active+clean; 803 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 589 op/s; 96/38541 objects degraded (0.249%) 2024-08-30T23:48:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2051253489' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-08-30T23:48:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:31 smithi136 ceph-mon[20207]: osdmap e710: 12 total, 12 up, 12 in 2024-08-30T23:48:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 96/38541 objects degraded (0.249%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-30T23:48:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:31 smithi136 ceph-mon[20207]: osdmap e711: 12 total, 12 up, 12 in 2024-08-30T23:48:31.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:31.652 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:31.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:31.677 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:31.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:31.775 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:31.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:48:31.878 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:31.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:31.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:32.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:32.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.034 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:32.125 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:32.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:32.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:48:32.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:32 smithi106 ceph-mon[16603]: osdmap e712: 12 total, 12 up, 12 in 2024-08-30T23:48:32.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:32.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:48:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.331 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:32.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:32.344 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:32.344 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:48:32.345 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:32 smithi173 ceph-mon[20554]: osdmap e712: 12 total, 12 up, 12 in 2024-08-30T23:48:32.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:32.459 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:32.503 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:32.503 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:48:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:48:32.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:32 smithi136 ceph-mon[20207]: osdmap e712: 12 total, 12 up, 12 in 2024-08-30T23:48:32.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:32.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:32.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:32.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:32.758 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:32.759 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:32.791 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:32.795 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:32.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:32.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:32.837 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:32.838 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:32.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:32.869 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:32.869 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:32.896 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:32.896 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:32.927 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:32.928 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:32.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:32.955 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:32.955 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:32.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:32.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:32.989 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:32.990 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:33.020 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:33.021 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:33.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:33.047 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:33.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:33.079 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:33.079 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:33.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:33.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:33.113 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:33.133 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: pgmap v1591: 37 pgs: 2 remapped+peering, 35 active+clean; 706 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 721 op/s; 0 B/s, 21 objects/s recovering 2024-08-30T23:48:33.134 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: 6.1 starting backfill to osd.10 from (0'0,652'5862] MIN to 709'11880 2024-08-30T23:48:33.134 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: 6.d starting backfill to osd.10 from (0'0,652'5620] MIN to 709'11384 2024-08-30T23:48:33.134 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: 6.f starting backfill to osd.10 from (0'0,652'5785] MIN to 709'11695 2024-08-30T23:48:33.134 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:48:33.134 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 96/38541 objects degraded (0.249%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:48:33.134 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1204756604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:33.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: pgmap v1591: 37 pgs: 2 remapped+peering, 35 active+clean; 706 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 721 op/s; 0 B/s, 21 objects/s recovering 2024-08-30T23:48:33.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: 6.1 starting backfill to osd.10 from (0'0,652'5862] MIN to 709'11880 2024-08-30T23:48:33.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: 6.d starting backfill to osd.10 from (0'0,652'5620] MIN to 709'11384 2024-08-30T23:48:33.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: 6.f starting backfill to osd.10 from (0'0,652'5785] MIN to 709'11695 2024-08-30T23:48:33.168 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:48:33.169 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 96/38541 objects degraded (0.249%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:48:33.169 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1204756604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:33.208 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:48:33.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:33.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:33.333 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:33.359 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:33.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:33.488 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: pgmap v1591: 37 pgs: 2 remapped+peering, 35 active+clean; 706 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 721 op/s; 0 B/s, 21 objects/s recovering 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: 6.1 starting backfill to osd.10 from (0'0,652'5862] MIN to 709'11880 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: 6.d starting backfill to osd.10 from (0'0,652'5620] MIN to 709'11384 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: 6.f starting backfill to osd.10 from (0'0,652'5785] MIN to 709'11695 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 96/38541 objects degraded (0.249%), 1 pg degraded, 1 pg undersized) 2024-08-30T23:48:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1204756604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:33.541 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:33.590 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:33.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:33.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:33.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:33.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:33.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:33.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:33.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:34.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:34.095 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:48:34.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86189 5.17 5.7a10c297 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13787440999999, 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83557530000000002, 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.115 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86189 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974568+0000", 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974569+0000", 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8599999999999998e-07 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974571+0000", 2024-08-30T23:48:34.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6599999999999999e-06 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974579+0000", 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4259999999999997e-06 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974584+0000", 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6740000000000002e-06 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974844+0000", 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025967600000000003 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974852+0000", 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4200000000000001e-06 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810133+0000", 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83528164100000002 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810143+0000", 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0117e-05 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86191 5.17 5.b68bebd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13784871500002, 2024-08-30T23:48:34.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83558337400000005, 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86191 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974594+0000", 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4799999999999999e-07 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974596+0000", 2024-08-30T23:48:34.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9589999999999998e-06 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974602+0000", 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6280000000000002e-06 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974607+0000", 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.977e-06 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974875+0000", 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026775900000000001 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974880+0000", 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2220000000000001e-06 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810168+0000", 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83528822599999997 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810177+0000", 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1549999999999996e-06 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.125 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86212 5.17 5.7a86b997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13782975499998, 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83559074600000005, 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86212 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.126 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974613+0000", 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0499999999999999e-07 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974615+0000", 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6929999999999999e-06 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974620+0000", 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6359999999999999e-06 2024-08-30T23:48:34.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974625+0000", 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4349999999999999e-06 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974930+0000", 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030525099999999998 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974938+0000", 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3980000000000002e-06 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810196+0000", 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83525841000000001 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810203+0000", 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:48:34.130 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86322 5.17 5.11e87097 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13771940599997, 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83550676000000001, 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.131 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86322 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974723+0000", 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974724+0000", 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0189999999999999e-06 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974728+0000", 2024-08-30T23:48:34.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1509999999999997e-06 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974739+0000", 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0575e-05 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974746+0000", 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8929999999999999e-06 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974986+0000", 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000240607 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974992+0000", 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3430000000000003e-06 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810222+0000", 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83523082599999998 2024-08-30T23:48:34.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810230+0000", 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3459999999999996e-06 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86384 5.17 5.e2d2ce77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13761461500002, 2024-08-30T23:48:34.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83542769299999997, 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86384 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974828+0000", 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1300000000000001e-07 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974845+0000", 2024-08-30T23:48:34.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6341000000000002e-05 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974850+0000", 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6359999999999999e-06 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974855+0000", 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7530000000000001e-06 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975038+0000", 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018276300000000001 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975044+0000", 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9830000000000001e-06 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810248+0000", 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83520457400000003 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810256+0000", 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2300000000000002e-06 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86434 5.17 5.dc1639f7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13757163999998, 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.835407809, 2024-08-30T23:48:34.141 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86434 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974871+0000", 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6899999999999998e-07 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974874+0000", 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3309999999999998e-06 2024-08-30T23:48:34.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974879+0000", 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.699e-06 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.974883+0000", 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2600000000000001e-06 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975066+0000", 2024-08-30T23:48:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018369799999999999 2024-08-30T23:48:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.975071+0000", 2024-08-30T23:48:34.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8930000000000004e-06 2024-08-30T23:48:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810272+0000", 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83520088199999998 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810279+0000", 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5769999999999999e-06 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86189 5.17 5.7a10c297 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13179418700003, 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82969184399999996, 2024-08-30T23:48:34.148 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86189 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980648+0000", 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980649+0000", 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2300000000000002e-07 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980651+0000", 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.21e-06 2024-08-30T23:48:34.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980658+0000", 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7800000000000003e-06 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980662+0000", 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6399999999999996e-06 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980723+0000", 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0745999999999997e-05 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980732+0000", 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9379999999999998e-06 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810304+0000", 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82957220200000004 2024-08-30T23:48:34.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810340+0000", 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5905000000000003e-05 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86191 5.17 5.b68bebd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13175804299999, 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968160300000005, 2024-08-30T23:48:34.153 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86191 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980684+0000", 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980685+0000", 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7899999999999999e-07 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980687+0000", 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1270000000000001e-06 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980694+0000", 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.703e-06 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980699+0000", 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4229999999999996e-06 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980764+0000", 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5363000000000004e-05 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980771+0000", 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1990000000000003e-06 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810359+0000", 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82958857900000005 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810366+0000", 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8299999999999998e-06 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86212 5.17 5.7a86b997 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13173013599999, 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967699800000005, 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86212 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980712+0000", 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980713+0000", 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5699999999999998e-07 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980715+0000", 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1629999999999999e-06 2024-08-30T23:48:34.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980720+0000", 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4920000000000001e-06 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980725+0000", 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.493e-06 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980812+0000", 2024-08-30T23:48:34.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7479999999999996e-05 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980819+0000", 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810384+0000", 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956448100000002 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810389+0000", 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5269999999999998e-06 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86322 5.17 5.11e87097 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13165268199998, 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82972026099999996, 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.163 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86322 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980790+0000", 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2599999999999998e-07 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980792+0000", 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.894e-06 2024-08-30T23:48:34.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980798+0000", 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980802+0000", 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3100000000000002e-06 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980870+0000", 2024-08-30T23:48:34.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7617e-05 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980877+0000", 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4309999999999997e-06 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810498+0000", 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82962015499999997 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810510+0000", 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2413e-05 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86384 5.17 5.e2d2ce77 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13158029099998, 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967802800000001, 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.168 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86384 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980862+0000", 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3000000000000002e-07 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980872+0000", 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2709999999999998e-06 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980877+0000", 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6999999999999996e-06 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980883+0000", 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2270000000000001e-06 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980958+0000", 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4972e-05 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980972+0000", 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4389999999999999e-05 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810534+0000", 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956191800000001 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810540+0000", 2024-08-30T23:48:34.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2199999999999997e-06 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86434 5.17 5.dc1639f7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13155242800002, 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82967796100000002, 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86434 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980890+0000", 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2000000000000001e-07 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980896+0000", 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3449999999999996e-06 2024-08-30T23:48:34.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980902+0000", 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980908+0000", 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2260000000000002e-06 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980995+0000", 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6890000000000003e-05 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.604e-06 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810562+0000", 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82956157900000005 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810568+0000", 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7749999999999998e-06 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86440 5.17 5.705d7d57 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13152665899997, 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968054700000005, 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.178 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86440 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980916+0000", 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5700000000000002e-07 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980918+0000", 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9819999999999998e-06 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980924+0000", 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7259999999999999e-06 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980929+0000", 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1749999999999996e-06 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981037+0000", 2024-08-30T23:48:34.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107353 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981044+0000", 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.272e-06 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810590+0000", 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82954588500000004 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810596+0000", 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5969999999999997e-06 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86445 5.17 5.715d0bd7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13150752500002, 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82968721000000001, 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.183 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86445 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980935+0000", 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8700000000000002e-07 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980937+0000", 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.793e-06 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980942+0000", 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4739999999999996e-06 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.980947+0000", 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8590000000000004e-06 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981087+0000", 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000139479 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981094+0000", 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.827e-06 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810616+0000", 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82952199699999996 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810622+0000", 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.494e-06 2024-08-30T23:48:34.187 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:86454 5.17 5.47a3f917 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e488)", 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 446.13144127499999, 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.82964718199999998, 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:34.188 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86454 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981001+0000", 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981002+0000", 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3499999999999996e-07 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981004+0000", 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9489999999999999e-06 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981010+0000", 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9039999999999997e-06 2024-08-30T23:48:34.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981014+0000", 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.617e-06 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981118+0000", 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103795 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:34.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:07.981124+0000", 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1469999999999998e-06 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810642+0000", 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.829518333 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:08.810648+0000", 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9020000000000004e-06 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.17 505/491 [PushReplyOp(5:ed0feeac:::benchmark_data_smithi173_87639_object47011:head)])", 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 417.85227751799999, 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73458869599999999, 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:48:34.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260165+0000", 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1500000000000005e-07 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.260173+0000", 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7519999999999996e-06 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993751+0000", 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73357760100000002 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993758+0000", 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8630000000000002e-06 2024-08-30T23:48:34.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.993850+0000", 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2544e-05 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.994754+0000", 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00090342099999999995 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.17 505/491 [PushReplyOp(5:ef72d6c2:::benchmark_data_smithi173_87639_object47037:head)])", 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 417.84308928399997, 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73803978299999995, 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:48:34.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269353+0000", 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269354+0000", 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3099999999999998e-07 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:36.269358+0000", 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2169999999999996e-06 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.001419+0000", 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.73206070400000001 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.001426+0000", 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4939999999999997e-06 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.006768+0000", 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005341485 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:37.007393+0000", 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062535200000000003 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.200 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(5.b e587,530 5:d5260a3e:::benchmark_data_smithi173_87639_object2707:head,442'5272)", 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 280.02532450899997, 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71060140599999999, 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087118+0000", 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087119+0000", 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.087123+0000", 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2479999999999998e-06 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797124+0000", 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.71000132900000001 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797133+0000", 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4440000000000008e-06 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797194+0000", 2024-08-30T23:48:34.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0937999999999998e-05 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:54.797719+0000", 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000525808 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.26173.0:144501 5.1 e590/540 5:827f28b3:::benchmark_data_smithi173_87639_object64261:head v 590'7080, mlcod=589'7078)", 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:48:34.204 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 276.88275084499998, 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64106508200000001, 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:48:34.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229692+0000", 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5900000000000001e-07 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.229695+0000", 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6009999999999999e-06 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868724+0000", 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.639028284 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868732+0000", 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3080000000000008e-06 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868794+0000", 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1963000000000003e-05 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.868819+0000", 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5282999999999998e-05 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.870718+0000", 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001898582 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:57.870757+0000", 2024-08-30T23:48:34.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8902000000000001e-05 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27034.0:155745 6.5 e684/619 6:a23d3df7:::benchmark_data_smithi173_114000_object55947:head v 684'9787, mlcod=684'9785)", 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:47:39.832338+0000", 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.280104371, 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67738270499999997, 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.832338+0000", 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.832338+0000", 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.832339+0000", 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.832345+0000", 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9290000000000003e-06 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.505836+0000", 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67349134300000002 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.505843+0000", 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1749999999999999e-06 2024-08-30T23:48:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.505953+0000", 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000109578 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.505981+0000", 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8532000000000001e-05 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.509687+0000", 2024-08-30T23:48:34.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037053720000000002 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.509721+0000", 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3924999999999998e-05 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:34.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:34.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:34.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/987893403' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:34.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/987893403' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:34.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:34.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/987893403' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:34.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:34.597 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:34.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:34.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:34.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:34.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.850 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:34.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:34.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:34.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:35.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:35.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:35.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:48:35.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:35.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:35.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:35 smithi106 ceph-mon[16603]: pgmap v1593: 37 pgs: 2 remapped+peering, 35 active+clean; 664 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 35 KiB/s wr, 546 op/s; 8.4 KiB/s, 17 objects/s recovering 2024-08-30T23:48:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:35 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/987893403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:35 smithi106 ceph-mon[16603]: osdmap e713: 12 total, 12 up, 12 in 2024-08-30T23:48:35.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:35.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:35.482 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:35 smithi136 ceph-mon[20207]: pgmap v1593: 37 pgs: 2 remapped+peering, 35 active+clean; 664 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 35 KiB/s wr, 546 op/s; 8.4 KiB/s, 17 objects/s recovering 2024-08-30T23:48:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:35 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/987893403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:35.483 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:35 smithi136 ceph-mon[20207]: osdmap e713: 12 total, 12 up, 12 in 2024-08-30T23:48:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.498 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:35.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:35.526 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:35 smithi173 ceph-mon[20554]: pgmap v1593: 37 pgs: 2 remapped+peering, 35 active+clean; 664 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 35 KiB/s wr, 546 op/s; 8.4 KiB/s, 17 objects/s recovering 2024-08-30T23:48:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:35 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/987893403' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:35 smithi173 ceph-mon[20554]: osdmap e713: 12 total, 12 up, 12 in 2024-08-30T23:48:35.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.626 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:35.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:35.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:35.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:35.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:35.887 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:35.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:35.896 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:48:35.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:48:35.896 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-30T23:48:35.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:48:35.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:36.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.019 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:36.097 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:36.097 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:48:36.097 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:48:36.097 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:36.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:36.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:36.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:36.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:36.312 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:36.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:36.398 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:36.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:36.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:36.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.670 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:36.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:36.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:48:36.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:36.799 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:36.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:36.938 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:36.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:37.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:37.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:36 smithi173 ceph-mon[20554]: pgmap v1595: 37 pgs: 2 remapped+peering, 1 active+remapped+backfill_toofull, 34 active+clean; 615 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 35 KiB/s wr, 465 op/s; 548/29487 objects misplaced (1.858%); 8.4 KiB/s, 0 objects/s recovering 2024-08-30T23:48:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:36 smithi173 ceph-mon[20554]: osdmap e714: 12 total, 12 up, 12 in 2024-08-30T23:48:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:36 smithi173 ceph-mon[20554]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-08-30T23:48:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:36 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:48:37.084 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:36 smithi106 ceph-mon[16603]: pgmap v1595: 37 pgs: 2 remapped+peering, 1 active+remapped+backfill_toofull, 34 active+clean; 615 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 35 KiB/s wr, 465 op/s; 548/29487 objects misplaced (1.858%); 8.4 KiB/s, 0 objects/s recovering 2024-08-30T23:48:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:36 smithi106 ceph-mon[16603]: osdmap e714: 12 total, 12 up, 12 in 2024-08-30T23:48:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:36 smithi106 ceph-mon[16603]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-08-30T23:48:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:36 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:48:37.154 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:48:37.155 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:48:37.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:37.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:37.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:48:37.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:36 smithi136 ceph-mon[20207]: pgmap v1595: 37 pgs: 2 remapped+peering, 1 active+remapped+backfill_toofull, 34 active+clean; 615 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 35 KiB/s wr, 465 op/s; 548/29487 objects misplaced (1.858%); 8.4 KiB/s, 0 objects/s recovering 2024-08-30T23:48:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:36 smithi136 ceph-mon[20207]: osdmap e714: 12 total, 12 up, 12 in 2024-08-30T23:48:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:36 smithi136 ceph-mon[20207]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-08-30T23:48:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:36 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:48:37.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:37.345 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:37.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:37.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:37.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:37.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:37.687 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:37.687 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:48:37.688 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:48:37.688 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:48:37.688 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:37.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.725 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:37.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:37.804 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2024-08-30T23:48:37.103656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":563860446,"num_objects":8597,"num_object_clones":0,"num_object_copies":25791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":8597,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":191007,"num_write_kb":6387076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28193,"num_bytes_recovered":1833155540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162929,"ondisk_log_size":162929,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7858136,"kb_used_data":2516720,"kb_used_omap":109,"kb_used_meta":5341010,"kb_avail":1117083688,"statfs":{"total":1151940427776,"available":1143893696512,"internally_reserved":0,"allocated":2577121280,"data_stored":2560300417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":5469194904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-176696922,"num_objects":-2697,"num_object_clones":0,"num_object_copies":-8091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":-2697,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":2707,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":396,"num_bytes_recovered":26000340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.428603"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":465,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.898562+0000","last_change":"2024-08-30T23:48:30.880809+0000","last_active":"2024-08-30T23:48:31.898562+0000","last_peered":"2024-08-30T23:48:31.898562+0000","last_clean":"2024-08-30T23:48:31.898562+0000","last_became_active":"2024-08-30T23:48:30.880410+0000","last_became_peered":"2024-08-30T23:48:30.880410+0000","last_unstale":"2024-08-30T23:48:31.898562+0000","last_undegraded":"2024-08-30T23:48:31.898562+0000","last_fullsized":"2024-08-30T23:48:31.898562+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:56.069850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:56.069850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":418,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.881968+0000","last_change":"2024-08-30T23:48:21.505243+0000","last_active":"2024-08-30T23:48:35.881968+0000","last_peered":"2024-08-30T23:48:35.881968+0000","last_clean":"2024-08-30T23:48:35.881968+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:48:35.881968+0000","last_undegraded":"2024-08-30T23:48:35.881968+0000","last_fullsized":"2024-08-30T23:48:35.881968+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:21.505170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:21.505170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:28.364776+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":468,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.882287+0000","last_change":"2024-08-30T23:48:27.943034+0000","last_active":"2024-08-30T23:48:35.882287+0000","last_peered":"2024-08-30T23:48:35.882287+0000","last_clean":"2024-08-30T23:48:35.882287+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:48:35.882287+0000","last_undegraded":"2024-08-30T23:48:35.882287+0000","last_fullsized":"2024-08-30T23:48:35.882287+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:27.942972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:27.942972+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:55.838934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":18,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.889829+0000","last_change":"2024-08-30T23:48:30.881979+0000","last_active":"2024-08-30T23:48:35.889829+0000","last_peered":"2024-08-30T23:48:35.889829+0000","last_clean":"2024-08-30T23:48:35.889829+0000","last_became_active":"2024-08-30T23:48:30.881538+0000","last_became_peered":"2024-08-30T23:48:30.881538+0000","last_unstale":"2024-08-30T23:48:35.889829+0000","last_undegraded":"2024-08-30T23:48:35.889829+0000","last_fullsized":"2024-08-30T23:48:35.889829+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:22.024520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:22.024520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:37.717608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.7","version":"713'11733","reported_seq":22248,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-08-30T23:48:34.627732+0000","last_change":"2024-08-30T23:48:24.687268+0000","last_active":"2024-08-30T23:48:34.627732+0000","last_peered":"2024-08-30T23:48:34.627732+0000","last_clean":"2024-08-30T23:48:34.627732+0000","last_became_active":"2024-08-30T23:47:57.869318+0000","last_became_peered":"2024-08-30T23:47:57.869318+0000","last_unstale":"2024-08-30T23:48:34.627732+0000","last_undegraded":"2024-08-30T23:48:34.627732+0000","last_fullsized":"2024-08-30T23:48:34.627732+0000","mapping_epoch":691,"log_start":"630'1700","ondisk_log_start":"630'1700","created":619,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'11400","last_scrub_stamp":"2024-08-30T23:48:24.687071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:24.687071+0000","objects_scrubbed":846,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:27.184281+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":391744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5223,"num_bytes_recovered":342294528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":235,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.899050+0000","last_change":"2024-08-30T23:48:26.872657+0000","last_active":"2024-08-30T23:48:31.899050+0000","last_peered":"2024-08-30T23:48:31.899050+0000","last_clean":"2024-08-30T23:48:31.899050+0000","last_became_active":"2024-08-30T23:45:08.975961+0000","last_became_peered":"2024-08-30T23:45:08.975961+0000","last_unstale":"2024-08-30T23:48:31.899050+0000","last_undegraded":"2024-08-30T23:48:31.899050+0000","last_fullsized":"2024-08-30T23:48:31.899050+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:26.872606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:26.872606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:47.518231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"712'11727","reported_seq":12032,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:32.209275+0000","last_change":"2024-08-30T23:47:47.467340+0000","last_active":"2024-08-30T23:48:32.209275+0000","last_peered":"2024-08-30T23:48:32.209275+0000","last_clean":"2024-08-30T23:48:32.209275+0000","last_became_active":"2024-08-30T23:46:12.977020+0000","last_became_peered":"2024-08-30T23:46:12.977020+0000","last_unstale":"2024-08-30T23:48:32.209275+0000","last_undegraded":"2024-08-30T23:48:32.209275+0000","last_fullsized":"2024-08-30T23:48:32.209275+0000","mapping_epoch":657,"log_start":"630'1700","ondisk_log_start":"630'1700","created":619,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'9979","last_scrub_stamp":"2024-08-30T23:47:47.467181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:47:47.467181+0000","objects_scrubbed":2468,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:50.765076+0000","scrub_duration":795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46071808,"num_objects":703,"num_object_clones":0,"num_object_copies":2109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":397760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":760,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.898438+0000","last_change":"2024-08-30T23:48:26.482500+0000","last_active":"2024-08-30T23:48:31.898438+0000","last_peered":"2024-08-30T23:48:31.898438+0000","last_clean":"2024-08-30T23:48:31.898438+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:48:31.898438+0000","last_undegraded":"2024-08-30T23:48:31.898438+0000","last_fullsized":"2024-08-30T23:48:31.898438+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:26.482450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:26.482450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:46.129058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"714'12193","reported_seq":12414,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:36.548610+0000","last_change":"2024-08-30T23:48:19.250120+0000","last_active":"2024-08-30T23:48:36.548610+0000","last_peered":"2024-08-30T23:48:36.548610+0000","last_clean":"2024-08-30T23:48:36.548610+0000","last_became_active":"2024-08-30T23:44:42.455726+0000","last_became_peered":"2024-08-30T23:44:42.455726+0000","last_unstale":"2024-08-30T23:48:36.548610+0000","last_undegraded":"2024-08-30T23:48:36.548610+0000","last_fullsized":"2024-08-30T23:48:36.548610+0000","mapping_epoch":619,"log_start":"630'2100","ondisk_log_start":"630'2100","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'11498","last_scrub_stamp":"2024-08-30T23:48:19.249800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:19.249800+0000","objects_scrubbed":1137,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:38.844785+0000","scrub_duration":476,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28377088,"num_objects":433,"num_object_clones":0,"num_object_copies":1299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12193,"num_write_kb":404032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":24,"reported_epoch":714,"state":"active+clean+scrubbing","last_fresh":"2024-08-30T23:48:36.434157+0000","last_change":"2024-08-30T23:48:36.433993+0000","last_active":"2024-08-30T23:48:36.434157+0000","last_peered":"2024-08-30T23:48:36.434157+0000","last_clean":"2024-08-30T23:48:36.434157+0000","last_became_active":"2024-08-30T23:48:30.881035+0000","last_became_peered":"2024-08-30T23:48:30.881035+0000","last_unstale":"2024-08-30T23:48:36.434157+0000","last_undegraded":"2024-08-30T23:48:36.434157+0000","last_fullsized":"2024-08-30T23:48:36.434157+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:20.438949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:20.438949+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"714'12065","reported_seq":24815,"reported_epoch":714,"state":"active+remapped+backfilling","last_fresh":"2024-08-30T23:48:36.112341+0000","last_change":"2024-08-30T23:48:32.076588+0000","last_active":"2024-08-30T23:48:36.112341+0000","last_peered":"2024-08-30T23:48:36.112341+0000","last_clean":"2024-08-30T23:48:30.076055+0000","last_became_active":"2024-08-30T23:48:32.031027+0000","last_became_peered":"2024-08-30T23:48:32.031027+0000","last_unstale":"2024-08-30T23:48:36.112341+0000","last_undegraded":"2024-08-30T23:48:36.112341+0000","last_fullsized":"2024-08-30T23:48:36.112341+0000","mapping_epoch":711,"log_start":"630'2000","ondisk_log_start":"630'2000","created":619,"last_epoch_clean":691,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'11612","last_scrub_stamp":"2024-08-30T23:48:24.240443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:24.240443+0000","objects_scrubbed":945,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":71,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12065,"num_write_kb":401600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6259,"num_bytes_recovered":410189824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,7],"avail_no_missing":["8","7","9"],"object_location_counts":[{"shards":"7,8,9","objects":485}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":21,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.889903+0000","last_change":"2024-08-30T23:48:34.095609+0000","last_active":"2024-08-30T23:48:35.889903+0000","last_peered":"2024-08-30T23:48:35.889903+0000","last_clean":"2024-08-30T23:48:35.889903+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:48:35.889903+0000","last_undegraded":"2024-08-30T23:48:35.889903+0000","last_fullsized":"2024-08-30T23:48:35.889903+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:27.521979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:27.521979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:48:37.386571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"712'11955","reported_seq":12169,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:34.025203+0000","last_change":"2024-08-30T23:48:21.246802+0000","last_active":"2024-08-30T23:48:34.025203+0000","last_peered":"2024-08-30T23:48:34.025203+0000","last_clean":"2024-08-30T23:48:34.025203+0000","last_became_active":"2024-08-30T23:44:42.455669+0000","last_became_peered":"2024-08-30T23:44:42.455669+0000","last_unstale":"2024-08-30T23:48:34.025203+0000","last_undegraded":"2024-08-30T23:48:34.025203+0000","last_fullsized":"2024-08-30T23:48:34.025203+0000","mapping_epoch":619,"log_start":"630'1900","ondisk_log_start":"630'1900","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'11483","last_scrub_stamp":"2024-08-30T23:48:21.246714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:21.246714+0000","objects_scrubbed":1062,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:48.893424+0000","scrub_duration":508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11955,"num_write_kb":401152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":654,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.899182+0000","last_change":"2024-08-30T23:48:30.880788+0000","last_active":"2024-08-30T23:48:31.899182+0000","last_peered":"2024-08-30T23:48:31.899182+0000","last_clean":"2024-08-30T23:48:31.899182+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:48:31.899182+0000","last_undegraded":"2024-08-30T23:48:31.899182+0000","last_fullsized":"2024-08-30T23:48:31.899182+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:44.643614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:44.643614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:01.882491+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"712'11980","reported_seq":12193,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:33.967798+0000","last_change":"2024-08-30T23:48:27.026389+0000","last_active":"2024-08-30T23:48:33.967798+0000","last_peered":"2024-08-30T23:48:33.967798+0000","last_clean":"2024-08-30T23:48:33.967798+0000","last_became_active":"2024-08-30T23:44:42.310475+0000","last_became_peered":"2024-08-30T23:44:42.310475+0000","last_unstale":"2024-08-30T23:48:33.967798+0000","last_undegraded":"2024-08-30T23:48:33.967798+0000","last_fullsized":"2024-08-30T23:48:33.967798+0000","mapping_epoch":619,"log_start":"630'1900","ondisk_log_start":"630'1900","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"707'11756","last_scrub_stamp":"2024-08-30T23:48:27.026296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:27.026296+0000","objects_scrubbed":812,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:32.451056+0000","scrub_duration":312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273024,"num_objects":584,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11980,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":18,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.889837+0000","last_change":"2024-08-30T23:48:30.881091+0000","last_active":"2024-08-30T23:48:35.889837+0000","last_peered":"2024-08-30T23:48:35.889837+0000","last_clean":"2024-08-30T23:48:35.889837+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:48:35.889837+0000","last_undegraded":"2024-08-30T23:48:35.889837+0000","last_fullsized":"2024-08-30T23:48:35.889837+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:46:00.551585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:46:00.551585+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"712'11895","reported_seq":12177,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:32.672105+0000","last_change":"2024-08-30T23:48:15.596665+0000","last_active":"2024-08-30T23:48:32.672105+0000","last_peered":"2024-08-30T23:48:32.672105+0000","last_clean":"2024-08-30T23:48:32.672105+0000","last_became_active":"2024-08-30T23:44:43.027945+0000","last_became_peered":"2024-08-30T23:44:43.027945+0000","last_unstale":"2024-08-30T23:48:32.672105+0000","last_undegraded":"2024-08-30T23:48:32.672105+0000","last_fullsized":"2024-08-30T23:48:32.672105+0000","mapping_epoch":619,"log_start":"630'1800","ondisk_log_start":"630'1800","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"701'11260","last_scrub_stamp":"2024-08-30T23:48:15.596574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:15.596574+0000","objects_scrubbed":1243,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:42.905546+0000","scrub_duration":269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":603,"num_object_clones":0,"num_object_copies":1809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11895,"num_write_kb":399936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1492,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.878750+0000","last_change":"2024-08-30T23:48:17.759711+0000","last_active":"2024-08-30T23:48:35.878750+0000","last_peered":"2024-08-30T23:48:35.878750+0000","last_clean":"2024-08-30T23:48:35.878750+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:48:35.878750+0000","last_undegraded":"2024-08-30T23:48:35.878750+0000","last_fullsized":"2024-08-30T23:48:35.878750+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:17.759639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:17.759639+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:42.961383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"714'12071","reported_seq":22901,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:36.052769+0000","last_change":"2024-08-30T23:48:17.027943+0000","last_active":"2024-08-30T23:48:36.052769+0000","last_peered":"2024-08-30T23:48:36.052769+0000","last_clean":"2024-08-30T23:48:36.052769+0000","last_became_active":"2024-08-30T23:47:40.439057+0000","last_became_peered":"2024-08-30T23:47:40.439057+0000","last_unstale":"2024-08-30T23:48:36.052769+0000","last_undegraded":"2024-08-30T23:48:36.052769+0000","last_fullsized":"2024-08-30T23:48:36.052769+0000","mapping_epoch":684,"log_start":"630'2000","ondisk_log_start":"630'2000","created":619,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'11279","last_scrub_stamp":"2024-08-30T23:48:17.027671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:17.027671+0000","objects_scrubbed":1249,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:19.436882+0000","scrub_duration":602,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29163520,"num_objects":445,"num_object_clones":0,"num_object_copies":1335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":400512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5301,"num_bytes_recovered":347406336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":388,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.878703+0000","last_change":"2024-08-30T23:48:26.562797+0000","last_active":"2024-08-30T23:48:35.878703+0000","last_peered":"2024-08-30T23:48:35.878703+0000","last_clean":"2024-08-30T23:48:35.878703+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:48:35.878703+0000","last_undegraded":"2024-08-30T23:48:35.878703+0000","last_fullsized":"2024-08-30T23:48:35.878703+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:26.562739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:26.562739+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:47.645825+0000","scrub_duration":746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"712'11944","reported_seq":12158,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:33.973529+0000","last_change":"2024-08-30T23:48:23.986305+0000","last_active":"2024-08-30T23:48:33.973529+0000","last_peered":"2024-08-30T23:48:33.973529+0000","last_clean":"2024-08-30T23:48:33.973529+0000","last_became_active":"2024-08-30T23:44:42.314051+0000","last_became_peered":"2024-08-30T23:44:42.314051+0000","last_unstale":"2024-08-30T23:48:33.973529+0000","last_undegraded":"2024-08-30T23:48:33.973529+0000","last_fullsized":"2024-08-30T23:48:33.973529+0000","mapping_epoch":619,"log_start":"630'1900","ondisk_log_start":"630'1900","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'11605","last_scrub_stamp":"2024-08-30T23:48:23.986117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:23.986117+0000","objects_scrubbed":910,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:31.590837+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37224448,"num_objects":568,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":400384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":468,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.898722+0000","last_change":"2024-08-30T23:47:59.712954+0000","last_active":"2024-08-30T23:48:31.898722+0000","last_peered":"2024-08-30T23:48:31.898722+0000","last_clean":"2024-08-30T23:48:31.898722+0000","last_became_active":"2024-08-30T23:43:56.833115+0000","last_became_peered":"2024-08-30T23:43:56.833115+0000","last_unstale":"2024-08-30T23:48:31.898722+0000","last_undegraded":"2024-08-30T23:48:31.898722+0000","last_fullsized":"2024-08-30T23:48:31.898722+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:59.712882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:59.712882+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:01.964110+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"712'11750","reported_seq":12082,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:32.183802+0000","last_change":"2024-08-30T23:48:19.286272+0000","last_active":"2024-08-30T23:48:32.183802+0000","last_peered":"2024-08-30T23:48:32.183802+0000","last_clean":"2024-08-30T23:48:32.183802+0000","last_became_active":"2024-08-30T23:44:43.028019+0000","last_became_peered":"2024-08-30T23:44:43.028019+0000","last_unstale":"2024-08-30T23:48:32.183802+0000","last_undegraded":"2024-08-30T23:48:32.183802+0000","last_fullsized":"2024-08-30T23:48:32.183802+0000","mapping_epoch":619,"log_start":"630'1700","ondisk_log_start":"630'1700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'11279","last_scrub_stamp":"2024-08-30T23:48:19.286136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:19.286136+0000","objects_scrubbed":1122,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:43.940263+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42336256,"num_objects":646,"num_object_clones":0,"num_object_copies":1938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11750,"num_write_kb":396672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":457,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.878714+0000","last_change":"2024-08-30T23:47:57.710542+0000","last_active":"2024-08-30T23:48:35.878714+0000","last_peered":"2024-08-30T23:48:35.878714+0000","last_clean":"2024-08-30T23:48:35.878714+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:48:35.878714+0000","last_undegraded":"2024-08-30T23:48:35.878714+0000","last_fullsized":"2024-08-30T23:48:35.878714+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:57.710491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:57.710491+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:12.946251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"714'12265","reported_seq":12530,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:36.052712+0000","last_change":"2024-08-30T23:45:56.318715+0000","last_active":"2024-08-30T23:48:36.052712+0000","last_peered":"2024-08-30T23:48:36.052712+0000","last_clean":"2024-08-30T23:48:36.052712+0000","last_became_active":"2024-08-30T23:44:43.029122+0000","last_became_peered":"2024-08-30T23:44:43.029122+0000","last_unstale":"2024-08-30T23:48:36.052712+0000","last_undegraded":"2024-08-30T23:48:36.052712+0000","last_fullsized":"2024-08-30T23:48:36.052712+0000","mapping_epoch":619,"log_start":"631'2200","ondisk_log_start":"631'2200","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"652'5669","last_scrub_stamp":"2024-08-30T23:45:56.318495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:45:56.318495+0000","objects_scrubbed":5640,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":407040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":387,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:35.882978+0000","last_change":"2024-08-30T23:45:00.588948+0000","last_active":"2024-08-30T23:48:35.882978+0000","last_peered":"2024-08-30T23:48:35.882978+0000","last_clean":"2024-08-30T23:48:35.882978+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:48:35.882978+0000","last_undegraded":"2024-08-30T23:48:35.882978+0000","last_fullsized":"2024-08-30T23:48:35.882978+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:00.588903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:00.588903+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:40.799929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"712'12054","reported_seq":12274,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:32.270102+0000","last_change":"2024-08-30T23:48:24.586688+0000","last_active":"2024-08-30T23:48:32.270102+0000","last_peered":"2024-08-30T23:48:32.270102+0000","last_clean":"2024-08-30T23:48:32.270102+0000","last_became_active":"2024-08-30T23:46:12.977124+0000","last_became_peered":"2024-08-30T23:46:12.977124+0000","last_unstale":"2024-08-30T23:48:32.270102+0000","last_undegraded":"2024-08-30T23:48:32.270102+0000","last_fullsized":"2024-08-30T23:48:32.270102+0000","mapping_epoch":657,"log_start":"630'2000","ondisk_log_start":"630'2000","created":619,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"705'11776","last_scrub_stamp":"2024-08-30T23:48:24.586460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:24.586460+0000","objects_scrubbed":883,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:44.323835+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":604,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12054,"num_write_kb":405056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":656,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.899117+0000","last_change":"2024-08-30T23:47:13.689799+0000","last_active":"2024-08-30T23:48:31.899117+0000","last_peered":"2024-08-30T23:48:31.899117+0000","last_clean":"2024-08-30T23:48:31.899117+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:48:31.899117+0000","last_undegraded":"2024-08-30T23:48:31.899117+0000","last_fullsized":"2024-08-30T23:48:31.899117+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:47:13.689751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:47:13.689751+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:34.875426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"712'11784","reported_seq":23651,"reported_epoch":712,"state":"active+remapped+backfill_toofull","last_fresh":"2024-08-30T23:48:33.742614+0000","last_change":"2024-08-30T23:48:32.123519+0000","last_active":"2024-08-30T23:48:33.742614+0000","last_peered":"2024-08-30T23:48:33.742614+0000","last_clean":"2024-08-30T23:48:30.031690+0000","last_became_active":"2024-08-30T23:48:32.068154+0000","last_became_peered":"2024-08-30T23:48:32.068154+0000","last_unstale":"2024-08-30T23:48:33.742614+0000","last_undegraded":"2024-08-30T23:48:33.742614+0000","last_fullsized":"2024-08-30T23:48:33.742614+0000","mapping_epoch":711,"log_start":"630'1700","ondisk_log_start":"630'1700","created":619,"last_epoch_clean":707,"parent":"0.0","parent_split_bits":0,"last_scrub":"708'11579","last_scrub_stamp":"2024-08-30T23:48:27.748738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:27.748738+0000","objects_scrubbed":761,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":548,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11784,"num_write_kb":394624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5787,"num_bytes_recovered":364576768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[2,9,5],"avail_no_missing":["2","5","9"],"object_location_counts":[{"shards":"2,5,9","objects":548}],"blocked_by":[],"up_primary":10,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":581,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.898491+0000","last_change":"2024-08-30T23:48:18.770498+0000","last_active":"2024-08-30T23:48:31.898491+0000","last_peered":"2024-08-30T23:48:31.898491+0000","last_clean":"2024-08-30T23:48:31.898491+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:48:31.898491+0000","last_undegraded":"2024-08-30T23:48:31.898491+0000","last_fullsized":"2024-08-30T23:48:31.898491+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:18.770199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:18.770199+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:21.006124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"714'12232","reported_seq":12451,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-08-30T23:48:36.515982+0000","last_change":"2024-08-30T23:48:21.091225+0000","last_active":"2024-08-30T23:48:36.515982+0000","last_peered":"2024-08-30T23:48:36.515982+0000","last_clean":"2024-08-30T23:48:36.515982+0000","last_became_active":"2024-08-30T23:44:42.455981+0000","last_became_peered":"2024-08-30T23:44:42.455981+0000","last_unstale":"2024-08-30T23:48:36.515982+0000","last_undegraded":"2024-08-30T23:48:36.515982+0000","last_fullsized":"2024-08-30T23:48:36.515982+0000","mapping_epoch":619,"log_start":"632'2200","ondisk_log_start":"632'2200","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'11601","last_scrub_stamp":"2024-08-30T23:48:21.091039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:21.091039+0000","objects_scrubbed":1078,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:30.881332+0000","scrub_duration":308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29097984,"num_objects":444,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12232,"num_write_kb":405632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1475,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.893523+0000","last_change":"2024-08-30T23:48:21.324664+0000","last_active":"2024-08-30T23:48:31.893523+0000","last_peered":"2024-08-30T23:48:31.893523+0000","last_clean":"2024-08-30T23:48:31.893523+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:48:31.893523+0000","last_undegraded":"2024-08-30T23:48:31.893523+0000","last_fullsized":"2024-08-30T23:48:31.893523+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:21.324583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:21.324583+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:23.787808+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"714'11595","reported_seq":23078,"reported_epoch":714,"state":"active+remapped+backfill_toofull","last_fresh":"2024-08-30T23:48:36.048656+0000","last_change":"2024-08-30T23:48:32.146337+0000","last_active":"2024-08-30T23:48:36.048656+0000","last_peered":"2024-08-30T23:48:36.048656+0000","last_clean":"2024-08-30T23:48:30.061724+0000","last_became_active":"2024-08-30T23:48:32.078986+0000","last_became_peered":"2024-08-30T23:48:32.078986+0000","last_unstale":"2024-08-30T23:48:36.048656+0000","last_undegraded":"2024-08-30T23:48:36.048656+0000","last_fullsized":"2024-08-30T23:48:36.048656+0000","mapping_epoch":711,"log_start":"630'1500","ondisk_log_start":"630'1500","created":619,"last_epoch_clean":695,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'10942","last_scrub_stamp":"2024-08-30T23:48:18.782704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:18.782704+0000","objects_scrubbed":1120,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146584,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":461,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11595,"num_write_kb":385729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5622,"num_bytes_recovered":368312320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,11],"acting":[1,11,2],"avail_no_missing":["1","2","11"],"object_location_counts":[{"shards":"1,2,11","objects":461}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":599,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-08-30T23:48:34.098328+0000","last_change":"2024-08-30T23:48:24.537429+0000","last_active":"2024-08-30T23:48:34.098328+0000","last_peered":"2024-08-30T23:48:34.098328+0000","last_clean":"2024-08-30T23:48:34.098328+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:48:34.098328+0000","last_undegraded":"2024-08-30T23:48:34.098328+0000","last_fullsized":"2024-08-30T23:48:34.098328+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:24.537372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:24.537372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:45.148033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"712'11754","reported_seq":11969,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:34.002497+0000","last_change":"2024-08-30T23:48:20.350257+0000","last_active":"2024-08-30T23:48:34.002497+0000","last_peered":"2024-08-30T23:48:34.002497+0000","last_clean":"2024-08-30T23:48:34.002497+0000","last_became_active":"2024-08-30T23:44:42.305499+0000","last_became_peered":"2024-08-30T23:44:42.305499+0000","last_unstale":"2024-08-30T23:48:34.002497+0000","last_undegraded":"2024-08-30T23:48:34.002497+0000","last_fullsized":"2024-08-30T23:48:34.002497+0000","mapping_epoch":619,"log_start":"630'1700","ondisk_log_start":"630'1700","created":619,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'11244","last_scrub_stamp":"2024-08-30T23:48:20.350003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:44:41.288579+0000","last_clean_scrub_stamp":"2024-08-30T23:48:20.350003+0000","objects_scrubbed":1043,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:36.932814+0000","scrub_duration":627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":392960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,7],"acting":[3,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":369,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.898357+0000","last_change":"2024-08-30T23:45:53.048060+0000","last_active":"2024-08-30T23:48:31.898357+0000","last_peered":"2024-08-30T23:48:31.898357+0000","last_clean":"2024-08-30T23:48:31.898357+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:48:31.898357+0000","last_undegraded":"2024-08-30T23:48:31.898357+0000","last_fullsized":"2024-08-30T23:48:31.898357+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:45:53.047998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:45:53.047998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:34.495644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1009,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-08-30T23:48:31.879476+0000","last_change":"2024-08-30T23:46:13.714566+0000","last_active":"2024-08-30T23:48:31.879476+0000","last_peered":"2024-08-30T23:48:31.879476+0000","last_clean":"2024-08-30T23:48:31.879476+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:48:31.879476+0000","last_undegraded":"2024-08-30T23:48:31.879476+0000","last_fullsized":"2024-08-30T23:48:31.879476+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:46:13.714502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:46:13.714502+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:48:35.210111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":563216408,"num_objects":8595,"num_object_clones":0,"num_object_copies":25785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":8595,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190997,"num_write_kb":6386881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28192,"num_bytes_recovered":1832779776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2441424896,"data_stored":2441412680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160997,"ondisk_log_size":160997,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":10,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":375764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187744,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596044,"kb_used_data":131700,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93149108,"statfs":{"total":95995035648,"available":95384686592,"internally_reserved":0,"allocated":134860800,"data_stored":133462969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":475462084},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449386,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1134692,"kb_used_data":720340,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92610460,"statfs":{"total":95995035648,"available":94833111040,"internally_reserved":0,"allocated":737628160,"data_stored":736221434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":424278464},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711019,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676188,"kb_used_data":175752,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93068964,"statfs":{"total":95995035648,"available":95302619136,"internally_reserved":0,"allocated":179970048,"data_stored":178567705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":512424384},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972657,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":504164,"kb_used_data":93904,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93240988,"statfs":{"total":95995035648,"available":95478771712,"internally_reserved":0,"allocated":96157696,"data_stored":94751801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234291,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":882516,"kb_used_data":251580,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92862636,"statfs":{"total":95995035648,"available":95091339264,"internally_reserved":0,"allocated":257617920,"data_stored":256219910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":646052298},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528634,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":536972,"kb_used_data":132336,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93208180,"statfs":{"total":95995035648,"available":95445176320,"internally_reserved":0,"allocated":135512064,"data_stored":134118313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":414316998},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775138,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249180,"kb_used_data":105288,"kb_used_omap":5,"kb_used_meta":143866,"kb_avail":93495972,"statfs":{"total":95995035648,"available":95739875328,"internally_reserved":0,"allocated":107814912,"data_stored":106412569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":147318858},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953798,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611608,"kb_used_data":125064,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93133544,"statfs":{"total":95995035648,"available":95368749056,"internally_reserved":0,"allocated":128065536,"data_stored":126663193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150024,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924564,"kb_used_data":293636,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92820588,"statfs":{"total":95995035648,"available":95048282112,"internally_reserved":0,"allocated":300683264,"data_stored":299281702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":646052295},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411662,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676632,"kb_used_data":270600,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93068520,"statfs":{"total":95995035648,"available":95302164480,"internally_reserved":0,"allocated":277094400,"data_stored":275688766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830478,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390684,"kb_used_data":132300,"kb_used_omap":87,"kb_used_meta":258280,"kb_avail":93354468,"statfs":{"total":95995035648,"available":95594975232,"internally_reserved":0,"allocated":135475200,"data_stored":134068785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":264478879},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738712,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":674892,"kb_used_data":84220,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93070260,"statfs":{"total":95995035648,"available":95303946240,"internally_reserved":0,"allocated":86241280,"data_stored":84843270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78184448,"data_stored":78184448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119083008,"data_stored":119078936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261165056,"data_stored":261160984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290652160,"data_stored":290652160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":114753536,"data_stored":114753536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":101711872,"data_stored":101711872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":130220032,"data_stored":130220032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":243335168,"data_stored":243335168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":89587712,"data_stored":89587712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":163315712,"data_stored":163315712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":721354752,"data_stored":721354752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":128061440,"data_stored":128057368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:48:37.806 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-08-30T23:48:37.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.875 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:37.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:37.877 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:37.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:48:37.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:37.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:37.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:38.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:38.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:38.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:38.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:38.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:38 smithi106 ceph-mon[16603]: osdmap e715: 12 total, 12 up, 12 in 2024-08-30T23:48:38.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:38 smithi106 ceph-mon[16603]: from='client.27418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:48:38.311 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:38 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:48:38.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:38.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:38.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:38.486 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:38.495 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:38.506 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:38.507 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:38.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:38 smithi136 ceph-mon[20207]: osdmap e715: 12 total, 12 up, 12 in 2024-08-30T23:48:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:38 smithi136 ceph-mon[20207]: from='client.27418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:48:38.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:38 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:48:38.545 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:38.546 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:38 smithi173 ceph-mon[20554]: osdmap e715: 12 total, 12 up, 12 in 2024-08-30T23:48:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:38 smithi173 ceph-mon[20554]: from='client.27418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:48:38.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:38 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:48:38.579 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:38.579 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:38.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:38.616 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:38.616 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:38.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:38.653 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:38.653 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:38.690 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:38.691 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:38.720 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:38.720 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:38.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:38.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:38.750 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:38.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:38.782 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:38.783 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:38.814 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:38.814 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:38.849 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:38.849 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:38.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:38.884 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:38.885 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:38.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:38.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:38.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:39.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:39.130 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:39.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:39.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:39.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:39.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:39.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:39.519 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:39 smithi136 ceph-mon[20207]: pgmap v1598: 37 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 33 active+clean; 477 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s; 1080/22881 objects misplaced (4.720%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:48:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:39 smithi136 ceph-mon[20207]: osdmap e716: 12 total, 12 up, 12 in 2024-08-30T23:48:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3446955334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:39.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:39.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:39 smithi106 ceph-mon[16603]: pgmap v1598: 37 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 33 active+clean; 477 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s; 1080/22881 objects misplaced (4.720%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:48:39.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:39 smithi106 ceph-mon[16603]: osdmap e716: 12 total, 12 up, 12 in 2024-08-30T23:48:39.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3446955334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:39 smithi173 ceph-mon[20554]: pgmap v1598: 37 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 33 active+clean; 477 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 632 op/s; 1080/22881 objects misplaced (4.720%); 4.3 MiB/s, 68 objects/s recovering 2024-08-30T23:48:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:39 smithi173 ceph-mon[20554]: osdmap e716: 12 total, 12 up, 12 in 2024-08-30T23:48:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3446955334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:39.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:39.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:39.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.826 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:39.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:39.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:39.952 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:39.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:40.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:40.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:40.128 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pgp_num to 21 2024-08-30T23:48:40.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:48:40.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.214 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:40.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:40.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:40.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2479276157' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:40 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:40.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:40.447 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:40.468 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:40.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2479276157' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:40 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:40.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:40.571 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2479276157' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:40.572 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:40 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-08-30T23:48:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:40.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:40.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:40.723 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:40.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:40.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:40.877 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:48:40.956 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:40.979 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:40.979 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:40.985 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:40.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:41.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:41.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:41.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:41.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:41.330 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: pgmap v1600: 37 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 33 active+clean; 412 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 873/19758 objects misplaced (4.418%); 4.4 MiB/s, 70 objects/s recovering 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: osdmap e717: 12 total, 12 up, 12 in 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3101299404' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3101299404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:48:41.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:41 smithi106 ceph-mon[16603]: osdmap e718: 12 total, 12 up, 12 in 2024-08-30T23:48:41.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:41.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:41.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:41.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:41.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: pgmap v1600: 37 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 33 active+clean; 412 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 873/19758 objects misplaced (4.418%); 4.4 MiB/s, 70 objects/s recovering 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: osdmap e717: 12 total, 12 up, 12 in 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3101299404' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3101299404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:48:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:41 smithi136 ceph-mon[20207]: osdmap e718: 12 total, 12 up, 12 in 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: pgmap v1600: 37 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 33 active+clean; 412 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 873/19758 objects misplaced (4.418%); 4.4 MiB/s, 70 objects/s recovering 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: osdmap e717: 12 total, 12 up, 12 in 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3101299404' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3101299404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:48:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:41 smithi173 ceph-mon[20554]: osdmap e718: 12 total, 12 up, 12 in 2024-08-30T23:48:41.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:41.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:41.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:48:41.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:41.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:41.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:41.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:41.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:42.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:42.056 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:42.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:42.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:42.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:42.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:42.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:42.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:42.432 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:42 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:48:42.432 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:42 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:48:42.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:42.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:42 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:48:42.486 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:42 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:48:42.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:42 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:48:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:42 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:48:42.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:42.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:42.619 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:42.619 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:42.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:48:42.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:42.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:42.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:42.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:42.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:42.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:42.957 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:43.013 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:43.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.090 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:43.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:43.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.226 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:43.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:43.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:43.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:43.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:43 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:48:43.440 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:43 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:48:43.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:43 smithi106 ceph-mon[16603]: pgmap v1603: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 34 active+clean; 322 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 755 op/s; 592/15414 objects misplaced (3.841%); 811 KiB/s, 12 objects/s recovering 2024-08-30T23:48:43.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:43 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:48:43.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:43 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:48:43.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:43.484 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.485 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:43 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:48:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:43 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:48:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:43 smithi136 ceph-mon[20207]: pgmap v1603: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 34 active+clean; 322 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 755 op/s; 592/15414 objects misplaced (3.841%); 811 KiB/s, 12 objects/s recovering 2024-08-30T23:48:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:43 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:48:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:43 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:48:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:43 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:48:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:43 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:48:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:43 smithi173 ceph-mon[20554]: pgmap v1603: 37 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 34 active+clean; 322 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 755 op/s; 592/15414 objects misplaced (3.841%); 811 KiB/s, 12 objects/s recovering 2024-08-30T23:48:43.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:43 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:48:43.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:43 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:48:43.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:43.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:43.622 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:43.622 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:43.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:48:43.661 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:43.661 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:43.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:43.695 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:43.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:43.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:43.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:43.749 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:43.757 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:43.758 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:43.789 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:43.789 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:43.818 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:43.818 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:43.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:48:43.847 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:43.848 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:43.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:43.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:43.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:43.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:43.886 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:43.886 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:43.915 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:43.915 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:43.954 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:43.955 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:43.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:43.994 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:43.994 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:44.003 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:44.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:44.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:44.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:44.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:44.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.261 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:44.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:44 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:48:44.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:44 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:48:44.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:44 smithi106 ceph-mon[16603]: osdmap e719: 12 total, 12 up, 12 in 2024-08-30T23:48:44.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4001700149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:44.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:44.388 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:44.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:44 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:48:44.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:44 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:48:44.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:44 smithi173 ceph-mon[20554]: osdmap e719: 12 total, 12 up, 12 in 2024-08-30T23:48:44.430 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4001700149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:44.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:44.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:44.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:44 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:48:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:44 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:48:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:44 smithi136 ceph-mon[20207]: osdmap e719: 12 total, 12 up, 12 in 2024-08-30T23:48:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4001700149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:44.552 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:44.552 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:48:44.552 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:48:44.552 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:48:44.552 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:44.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:44.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:44.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:44.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:44.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:44.844 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:44.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:44.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:44.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:45.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:45.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:45.130 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:45.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:45.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:45.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:48:45.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:45.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:45.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:45.489 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:45 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:48:45.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:45 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:48:45.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:45 smithi106 ceph-mon[16603]: pgmap v1605: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 270 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 592/12921 objects misplaced (4.582%); 662 KiB/s, 10 objects/s recovering 2024-08-30T23:48:45.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:45.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:45 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:48:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:45 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:48:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:45 smithi136 ceph-mon[20207]: pgmap v1605: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 270 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 592/12921 objects misplaced (4.582%); 662 KiB/s, 10 objects/s recovering 2024-08-30T23:48:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:45 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:48:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:45 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:48:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:45 smithi173 ceph-mon[20554]: pgmap v1605: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 270 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 592/12921 objects misplaced (4.582%); 662 KiB/s, 10 objects/s recovering 2024-08-30T23:48:45.599 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:48:45.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:45.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:45.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.815 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:45.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:45.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:45.941 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:45.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:46.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:46.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:46.107 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:48:46.107 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:48:46.107 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-30T23:48:46.107 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:48:46.107 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:48:46.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-08-30T23:48:46.130 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:48:46.130 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:48:46.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:48:46.130 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:48:46.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.131 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 502/491 [PushOp(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head, version: 408'2535, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head@408'2535, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:46.131 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:48:46.131 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 432.93001062100001, 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59550546699999996, 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199198+0000", 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5470000000000001e-06 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199226+0000", 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8228000000000001e-05 2024-08-30T23:48:46.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793558+0000", 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59433249099999996 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793576+0000", 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8108999999999999e-05 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793661+0000", 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5122000000000003e-05 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793671+0000", 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.794699+0000", 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001028581 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 509/491 [PushOp(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head, version: 417'3530, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head@417'3530, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 425.79041734200001, 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42136218600000003, 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338788+0000", 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.33e-07 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338818+0000", 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9720999999999999e-05 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758562+0000", 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.419743904 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758588+0000", 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5839000000000001e-05 2024-08-30T23:48:46.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758731+0000", 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014296099999999999 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758750+0000", 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9196000000000001e-05 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.760149+0000", 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001399632 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 510/491 [PushOp(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head, version: 418'3647, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head@418'3647, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 425.00045372, 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62881896699999995, 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.140 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128752+0000", 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2219999999999999e-06 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128784+0000", 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1445000000000001e-05 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756187+0000", 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.627403029 2024-08-30T23:48:46.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756208+0000", 2024-08-30T23:48:46.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1382999999999998e-05 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756330+0000", 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121612 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756346+0000", 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5962000000000001e-05 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.757570+0000", 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001224314 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:ef154109:::benchmark_data_smithi173_87639_object59696:head, version: 421'3773, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ef154109:::benchmark_data_smithi173_87639_object59696:head@421'3773, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 424.04092830500002, 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60072882599999999, 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.145 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088277+0000", 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4600000000000003e-07 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088304+0000", 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6968999999999999e-05 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687871+0000", 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59956659400000001 2024-08-30T23:48:46.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687891+0000", 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0307999999999999e-05 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688018+0000", 2024-08-30T23:48:46.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012653499999999999 2024-08-30T23:48:46.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688034+0000", 2024-08-30T23:48:46.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6728999999999999e-05 2024-08-30T23:48:46.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.689005+0000", 2024-08-30T23:48:46.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097084500000000002 2024-08-30T23:48:46.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.150 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 512/491 [PushOp(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head, version: 430'4003, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head@430'4003, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:48:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:48:46.151 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 419.86537563100001, 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48614643400000002, 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:48:46.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:48:46.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263830+0000", 2024-08-30T23:48:46.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8500000000000001e-07 2024-08-30T23:48:46.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263854+0000", 2024-08-30T23:48:46.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3934000000000002e-05 2024-08-30T23:48:46.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748876+0000", 2024-08-30T23:48:46.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.485022907 2024-08-30T23:48:46.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748889+0000", 2024-08-30T23:48:46.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2785999999999999e-05 2024-08-30T23:48:46.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748973+0000", 2024-08-30T23:48:46.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.331e-05 2024-08-30T23:48:46.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748983+0000", 2024-08-30T23:48:46.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-08-30T23:48:46.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.749975+0000", 2024-08-30T23:48:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099270099999999991 2024-08-30T23:48:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.160 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.161 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.d 542/530 [PushReplyOp(5:bce15829:::benchmark_data_smithi173_87639_object23219:head)])", 2024-08-30T23:48:46.161 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:48:46.161 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 365.31239385200001, 2024-08-30T23:48:46.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44671719199999999, 2024-08-30T23:48:46.162 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.162 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.162 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:46.162 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:48:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:48:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816812+0000", 2024-08-30T23:48:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1260000000000001e-06 2024-08-30T23:48:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816821+0000", 2024-08-30T23:48:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5510000000000005e-06 2024-08-30T23:48:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260989+0000", 2024-08-30T23:48:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44416764800000003 2024-08-30T23:48:46.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260997+0000", 2024-08-30T23:48:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5750000000000003e-06 2024-08-30T23:48:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.261062+0000", 2024-08-30T23:48:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5520999999999998e-05 2024-08-30T23:48:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.263528+0000", 2024-08-30T23:48:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024657709999999998 2024-08-30T23:48:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.169 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99435 5.8 5:154d2bda:::benchmark_data_smithi173_87639_object19195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 363.77091274600002, 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39149939, 2024-08-30T23:48:46.170 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99435 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358293+0000", 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5540000000000001e-06 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358300+0000", 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4250000000000003e-06 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358312+0000", 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2333e-05 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358321+0000", 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4160000000000004e-06 2024-08-30T23:48:46.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.383891+0000", 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025570521999999998 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.387506+0000", 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036149160000000001 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.705150+0000", 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31764353000000001 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.707627+0000", 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024771609999999999 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.712526+0000", 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048989070000000001 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740121+0000", 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027595773000000001 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740604+0000", 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000482196 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.741372+0000", 2024-08-30T23:48:46.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076844900000000004 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.743276+0000", 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019037590000000001 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.749791+0000", 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065154490000000004 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99448 5.8 5:1453319e:::benchmark_data_smithi173_87639_object19208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 363.74913872799999, 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.415457309, 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99448 2024-08-30T23:48:46.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380068+0000", 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.637e-06 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380074+0000", 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3500000000000002e-06 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380092+0000", 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7768999999999999e-05 2024-08-30T23:48:46.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380101+0000", 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.389216+0000", 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0091151319999999997 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:48:46.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.392658+0000", 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034414720000000001 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.727246+0000", 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33458855300000001 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.729727+0000", 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024804380000000002 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.733843+0000", 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041159079999999997 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.750905+0000", 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017061951999999998 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.752106+0000", 2024-08-30T23:48:46.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001200916 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.789800+0000", 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037693979000000002 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.790989+0000", 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001189563 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.795523+0000", 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045340220000000004 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102045 5.8 5:13843233:::benchmark_data_smithi173_87639_object21805:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 358.917944272, 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41801196699999998, 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.185 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102045 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211262+0000", 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7630000000000001e-06 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211268+0000", 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3489999999999997e-06 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.187 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211283+0000", 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4834000000000001e-05 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211293+0000", 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0037999999999999e-05 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.235799+0000", 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024505741000000001 2024-08-30T23:48:46.188 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.239265+0000", 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034655129999999999 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.592550+0000", 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35328515199999999 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.189 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.594844+0000", 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002294164 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.599415+0000", 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045711010000000002 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.620491+0000", 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021075930999999999 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.190 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.621208+0000", 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071667200000000001 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.622191+0000", 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098297100000000002 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.623817+0000", 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016259740000000001 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.191 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629272+0000", 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054557640000000001 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102049 5.8 5:1445c9f0:::benchmark_data_smithi173_87639_object21809:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 358.89492828499999, 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40420456799999999, 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.192 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102049 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234278+0000", 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.012e-06 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234286+0000", 2024-08-30T23:48:46.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2799999999999998e-06 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234301+0000", 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5590000000000002e-05 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234310+0000", 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.241629+0000", 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073184249999999999 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.244833+0000", 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032044450000000002 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.610115+0000", 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.365282208 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.612362+0000", 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022467279999999999 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.616714+0000", 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043520410000000001 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629561+0000", 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012847305999999999 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.630448+0000", 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088642099999999998 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.631622+0000", 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011737430000000001 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.633508+0000", 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018869030000000001 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.638481+0000", 2024-08-30T23:48:46.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049724469999999996 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102205 5.8 5:100735ae:::benchmark_data_smithi173_87639_object21965:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 358.45453417800002, 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41224830200000001, 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.201 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102205 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674673+0000", 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.063e-06 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674679+0000", 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3380000000000001e-06 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674692+0000", 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3383e-05 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674702+0000", 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4280000000000002e-06 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.691678+0000", 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016976577 2024-08-30T23:48:46.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.694973+0000", 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032944049999999998 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.051542+0000", 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35656930599999997 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.054103+0000", 2024-08-30T23:48:46.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025610979999999999 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.057950+0000", 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003847296 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.078442+0000", 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020491453 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.079182+0000", 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073979099999999995 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.080424+0000", 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00124194 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.081991+0000", 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015673 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.086919+0000", 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049279240000000002 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:127378 5.8 5:10a9409e:::benchmark_data_smithi173_87639_object47138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 313.83943899600001, 2024-08-30T23:48:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41103042400000001, 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127378 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:48:46.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289767+0000", 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.575e-06 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289774+0000", 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8510000000000001e-06 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289788+0000", 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4056999999999999e-05 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289796+0000", 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5800000000000003e-06 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290170+0000", 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000373829 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290389+0000", 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000219559 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290617+0000", 2024-08-30T23:48:46.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022764500000000001 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.293291+0000", 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026742269999999999 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.293692+0000", 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040149700000000001 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700642+0000", 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40694962699999998 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700700+0000", 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7865000000000002e-05 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700796+0000", 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6112000000000004e-05 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.214 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:144420 5.d 5:b9edf439:::benchmark_data_smithi173_87639_object64180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 290.00777207099998, 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.397110827, 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:48:46.215 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144420 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121434+0000", 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121440+0000", 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0909999999999998e-06 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121453+0000", 2024-08-30T23:48:46.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2656e-05 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121459+0000", 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1489999999999999e-06 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121516+0000", 2024-08-30T23:48:46.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7333000000000001e-05 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121593+0000", 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7107000000000006e-05 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121655+0000", 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2217000000000004e-05 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.219 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125135+0000", 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00347956 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125567+0000", 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043209299999999998 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518466+0000", 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39289916899999999 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518501+0000", 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4971000000000001e-05 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518543+0000", 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2261000000000001e-05 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:144420 5.d e589/587 ondisk, result = 0)", 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 290.0040515, 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39339614299999998, 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125154+0000", 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2699999999999999e-07 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125158+0000", 2024-08-30T23:48:46.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9559999999999999e-06 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518369+0000", 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39321148299999997 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518374+0000", 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8439999999999997e-06 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518457+0000", 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2750999999999994e-05 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518462+0000", 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6069999999999998e-06 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518549+0000", 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6774999999999995e-05 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.225 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(1.b e543 from shard 1)", 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 265.96602570900001, 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91109331999999998, 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7400000000000003e-07 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163183+0000", 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.19e-06 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074076+0000", 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91089344999999999 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074089+0000", 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3113e-05 2024-08-30T23:48:46.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074186+0000", 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7214999999999997e-05 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074207+0000", 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0372000000000001e-05 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074272+0000", 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5406000000000004e-05 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93436 6.d 6.1d3bdb7d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e653)", 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:01.884570+0000", 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 164.24463429299999, 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99829890499999996, 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:46.230 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93436 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884570+0000", 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884570+0000", 2024-08-30T23:48:46.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884572+0000", 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.406e-06 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884604+0000", 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1897000000000003e-05 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884614+0000", 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0118999999999999e-05 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884623+0000", 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0680000000000003e-06 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884675+0000", 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1788e-05 2024-08-30T23:48:46.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884683+0000", 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5010000000000004e-06 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.882832+0000", 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99814862000000004 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.882869+0000", 2024-08-30T23:48:46.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7506000000000003e-05 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93458 6.d 6.b13a355d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:02.203482+0000", 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 163.925722433, 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67964718800000001, 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:46.235 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93458 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203482+0000", 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203482+0000", 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203484+0000", 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0389999999999999e-06 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203598+0000", 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011348799999999999 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203613+0000", 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5554e-05 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203621+0000", 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4359999999999996e-06 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203691+0000", 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0599999999999995e-05 2024-08-30T23:48:46.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203704+0000", 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2490000000000001e-05 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.883120+0000", 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67941584099999996 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.883129+0000", 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999999e-06 2024-08-30T23:48:46.239 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(6.d 684/654 [PushReplyOp(6:b9e0903d:::benchmark_data_smithi173_114000_object66702:head)])", 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:47:39.972811+0000", 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 66.156393464000004, 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39577864600000001, 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972811+0000", 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972811+0000", 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972813+0000", 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:48:46.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972828+0000", 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5063e-05 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.366857+0000", 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39402893 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.366871+0000", 2024-08-30T23:48:46.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.45e-05 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.367015+0000", 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014346399999999999 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.368590+0000", 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015752089999999999 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.243 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:157517 6.8 6:14fb8c3a:::benchmark_data_smithi173_114000_object57719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:47:42.803699+0000", 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.325505575000001, 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28935165899999998, 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:46.244 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157517 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803699+0000", 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803699+0000", 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803700+0000", 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3090000000000001e-06 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803706+0000", 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8209999999999998e-06 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803719+0000", 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2679e-05 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803726+0000", 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8290000000000002e-06 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803776+0000", 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0138e-05 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803864+0000", 2024-08-30T23:48:46.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8298000000000001e-05 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803940+0000", 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5803999999999995e-05 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.807952+0000", 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040120879999999996 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.808487+0000", 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053511499999999996 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:43.092946+0000", 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28445894199999999 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:43.093010+0000", 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4331999999999993e-05 2024-08-30T23:48:46.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:43.093051+0000", 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0303999999999999e-05 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:193121 6.8 6:15eca3e2:::benchmark_data_smithi173_114000_object93323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e715)", 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:37.131569+0000", 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 8.9976360270000004, 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30851257300000001, 2024-08-30T23:48:46.250 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193121 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131569+0000", 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131569+0000", 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131570+0000", 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1200000000000001e-06 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131575+0000", 2024-08-30T23:48:46.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1580000000000004e-06 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131585+0000", 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0312999999999999e-05 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131593+0000", 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0879999999999995e-06 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:48:46.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131652+0000", 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9120999999999999e-05 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131792+0000", 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013964199999999999 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131931+0000", 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013943199999999999 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.136245+0000", 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043140009999999996 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.136820+0000", 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057477899999999998 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.440008+0000", 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30318730999999999 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.440044+0000", 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6702999999999997e-05 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.440081+0000", 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6906000000000002e-05 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:48:46.256 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:48:46.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: 6.9 scrub starts 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: 6.9 scrub ok 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: osdmap e720: 12 total, 12 up, 12 in 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/535255576' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:46 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:46.361 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:46.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:46.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:46.481 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: 6.9 scrub starts 2024-08-30T23:48:46.481 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: 6.9 scrub ok 2024-08-30T23:48:46.481 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:48:46.481 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:48:46.482 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-08-30T23:48:46.482 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: osdmap e720: 12 total, 12 up, 12 in 2024-08-30T23:48:46.482 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/535255576' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:46.482 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:46 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:46.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:46.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:46.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: 6.9 scrub starts 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: 6.9 scrub ok 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: osdmap e720: 12 total, 12 up, 12 in 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/535255576' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:46 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:46.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:46.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:46.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:46.768 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:46.822 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:48:46.822 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-30T23:48:46.822 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:48:46.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:48:46.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:46.890 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:48:46.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:46.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:46.937 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 99795 objects 2024-08-30T23:48:46.937 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :150.534 2024-08-30T23:48:46.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:47.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:47.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:48:47.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:47.142 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:47.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:47.266 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: pgmap v1606: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 178 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 370/8511 objects misplaced (4.347%); 614 KiB/s, 9 objects/s recovering 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: osdmap e721: 12 total, 12 up, 12 in 2024-08-30T23:48:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:48:47.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:47 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:48:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:48:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: pgmap v1606: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 178 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 370/8511 objects misplaced (4.347%); 614 KiB/s, 9 objects/s recovering 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: osdmap e721: 12 total, 12 up, 12 in 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:48:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:47 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:48:47.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:47.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:47.390 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:47.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:47.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: pgmap v1606: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 178 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 370/8511 objects misplaced (4.347%); 614 KiB/s, 9 objects/s recovering 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:48:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: osdmap e721: 12 total, 12 up, 12 in 2024-08-30T23:48:47.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:48:47.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:47 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:48:47.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:47.639 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:47.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:47.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:47.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:47.890 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:47.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:47.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:48.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:48.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:48.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:48.239 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:48.240 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:48.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:48.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:48.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:48.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:48.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:48.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:48 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:48:48.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:48.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:48.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:48 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:48:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:48 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:48:48.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:48.639 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:48.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:48.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:48.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:48.888 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:48.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:48.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:49.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:49.112 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:49.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:49.119 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:49.120 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:49.140 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:49.140 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:49.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:49.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:49.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:49.181 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:49.181 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:49.203 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:49.203 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:49.225 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:49.225 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:49.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:49.246 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:49.246 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:49.267 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:49.267 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:49.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:49.288 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:49.289 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:49.309 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:49.310 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:49.331 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:49.332 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:49.350 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:49 smithi106 ceph-mon[16603]: pgmap v1609: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 38 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 124/1815 objects misplaced (6.832%) 2024-08-30T23:48:49.350 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:49 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:48:49.350 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:49 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:48:49.350 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/24597012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:49.358 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-08-30T23:48:49.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-08-30T23:48:49.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:49.375 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:49.376 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:49.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.399 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:49.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:49.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:49.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:49 smithi136 ceph-mon[20207]: pgmap v1609: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 38 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 124/1815 objects misplaced (6.832%) 2024-08-30T23:48:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:49 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:48:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:49 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:48:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/24597012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:49 smithi173 ceph-mon[20554]: pgmap v1609: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 38 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 753 op/s; 124/1815 objects misplaced (6.832%) 2024-08-30T23:48:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:49 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:48:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:49 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:48:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/24597012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:49.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:49.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:49.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:49.776 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:49.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:48:49.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:49.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:49.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:49.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:50.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:50.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:50.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:50.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:50.257 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:50.287 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:50.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:50.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:50.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:50.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:50.644 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:50.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:50.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:48:50.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:50.784 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:50.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:50.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:50.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:50.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:51.026 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:51.053 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:51.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:51.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:51.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:51.306 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:51.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.309 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:51.337 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:51 smithi106 ceph-mon[16603]: pgmap v1610: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.1 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 5/33 objects misplaced (15.152%) 2024-08-30T23:48:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:51 smithi106 ceph-mon[16603]: osdmap e722: 12 total, 12 up, 12 in 2024-08-30T23:48:51.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:48:51.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:48:51.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:51 smithi136 ceph-mon[20207]: pgmap v1610: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.1 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 5/33 objects misplaced (15.152%) 2024-08-30T23:48:51.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:51 smithi136 ceph-mon[20207]: osdmap e722: 12 total, 12 up, 12 in 2024-08-30T23:48:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:51.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:51.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:51 smithi173 ceph-mon[20554]: pgmap v1610: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 1.1 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 655 op/s; 5/33 objects misplaced (15.152%) 2024-08-30T23:48:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:51 smithi173 ceph-mon[20554]: osdmap e722: 12 total, 12 up, 12 in 2024-08-30T23:48:51.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:51.683 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:51.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:51.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:51.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:51.930 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:51.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:52.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:48:52.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:52.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:52.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:52.175 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:48:52.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:52.195 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1254054565' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:52.279 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:52.279 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:52.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:52.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:52.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:52.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:52.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:52.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1254054565' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:52.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:52.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:52.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1254054565' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:48:52.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:52.682 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:52.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:52.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:52.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:52.940 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:52.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:53.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.065 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:53.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:53.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:53.180 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_4' removed 2024-08-30T23:48:53.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:53.191 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:48:53.191 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:48:53.191 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-08-30T23:48:53.191 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:48:53.191 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:48:53.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-08-30T23:48:53.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:48:53.291 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:53.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:53.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:53.440 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:53 smithi136 ceph-mon[20207]: pgmap v1612: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s 2024-08-30T23:48:53.440 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:53 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:53.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1254054565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:53.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:53 smithi136 ceph-mon[20207]: osdmap e723: 12 total, 12 up, 12 in 2024-08-30T23:48:53.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3724421396' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:48:53.454 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:53.486 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:53.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:53.558 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:53 smithi106 ceph-mon[16603]: pgmap v1612: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s 2024-08-30T23:48:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:53 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1254054565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:53 smithi106 ceph-mon[16603]: osdmap e723: 12 total, 12 up, 12 in 2024-08-30T23:48:53.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3724421396' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:48:53.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:53 smithi173 ceph-mon[20554]: pgmap v1612: 37 pgs: 2 active+remapped+backfill_toofull, 35 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s 2024-08-30T23:48:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:53 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:48:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1254054565' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:48:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:53 smithi173 ceph-mon[20554]: osdmap e723: 12 total, 12 up, 12 in 2024-08-30T23:48:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3724421396' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:48:53.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:53.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:53.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:53.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:53.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:53.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:53.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:53.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:53.964 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:48:53.964 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:48:53.965 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-08-30T23:48:53.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-08-30T23:48:54.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:54.076 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:54.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:54.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:54.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:54.313 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3724421396' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:48:54.314 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:54 smithi136 ceph-mon[20207]: osdmap e724: 12 total, 12 up, 12 in 2024-08-30T23:48:54.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:54.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:54.416 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:54.416 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "" 2024-08-30T23:48:54.416 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:54.426 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:54.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.452 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:54.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:54.458 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3724421396' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:48:54.458 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:54 smithi106 ceph-mon[16603]: osdmap e724: 12 total, 12 up, 12 in 2024-08-30T23:48:54.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3724421396' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:48:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:54 smithi173 ceph-mon[20554]: osdmap e724: 12 total, 12 up, 12 in 2024-08-30T23:48:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:54.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:54.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:54.698 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:54.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:54.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:54.822 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:54.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:54.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:54.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:55.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:55.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:55.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:55.201 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:55.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:55.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:55 smithi106 ceph-mon[16603]: pgmap v1615: 21 pgs: 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:55 smithi106 ceph-mon[16603]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-08-30T23:48:55.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/82997387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:55.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:55.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:55.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:55 smithi136 ceph-mon[20207]: pgmap v1615: 21 pgs: 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:55 smithi136 ceph-mon[20207]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-08-30T23:48:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/82997387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:55.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:55 smithi173 ceph-mon[20554]: pgmap v1615: 21 pgs: 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:55 smithi173 ceph-mon[20554]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-08-30T23:48:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/82997387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:48:55.575 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:48:55.575 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:48:55.581 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:55.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:55.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:55.718 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:48:55.719 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:48:55.738 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:48:55.738 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:48:55.758 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:48:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:48:55.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:48:55.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:48:55.806 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:48:55.807 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:48:55.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:55.827 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:48:55.828 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:48:55.833 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.834 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:55.848 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:48:55.848 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:48:55.873 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:48:55.873 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:48:55.895 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:48:55.895 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:48:55.918 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:48:55.919 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:48:55.934 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:55.939 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:48:55.939 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:48:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:55.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:55.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:55.960 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:48:55.961 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:48:56.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:56.083 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.084 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:56.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:56.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:56.310 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:56.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:56.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:56.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:56.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:48:56.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:56.586 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:56.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:56.714 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:56.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:56.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.842 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:56.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:56.964 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:56.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:57.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:57.087 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:57.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:57.195 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_5' created 2024-08-30T23:48:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.217 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:57.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:57.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:57.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:57.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:57 smithi106 ceph-mon[16603]: pgmap v1616: 21 pgs: 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:57.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1049646068' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-30T23:48:57.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:48:57.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:48:57.468 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:48:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:57 smithi136 ceph-mon[20207]: pgmap v1616: 21 pgs: 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1049646068' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-30T23:48:57.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:48:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:57 smithi173 ceph-mon[20554]: pgmap v1616: 21 pgs: 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1049646068' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-30T23:48:57.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:48:57.640 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:57.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:48:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:57.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:48:57.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:48:57.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:48:57.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:57.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:57.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:58.045 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-08-30T23:48:58.046 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:58.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:58.109 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:58.147 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:58.147 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:48:58.147 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:48:58.147 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:58.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:48:58.234 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:48:58.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1049646068' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-30T23:48:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:58 smithi136 ceph-mon[20207]: osdmap e725: 12 total, 12 up, 12 in 2024-08-30T23:48:58.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1049646068' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-30T23:48:58.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:58 smithi106 ceph-mon[16603]: osdmap e725: 12 total, 12 up, 12 in 2024-08-30T23:48:58.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:58.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:58.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:58.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:58.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:58.545 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1049646068' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-30T23:48:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:58 smithi173 ceph-mon[20554]: osdmap e725: 12 total, 12 up, 12 in 2024-08-30T23:48:58.590 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:58.611 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:58.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:58.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:48:58.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:58.789 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:48:58.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:48:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:58.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:48:58.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:58.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:58.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:59.036 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:48:59.090 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:48:59.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:59.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:59.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:48:59.238 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:48:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:48:59.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:48:59.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:48:59.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:48:59.447 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:59 smithi136 ceph-mon[20207]: pgmap v1618: 37 pgs: 4 creating+peering, 12 unknown, 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:59.447 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:59 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:48:59.447 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:59 smithi136 ceph-mon[20207]: osdmap e726: 12 total, 12 up, 12 in 2024-08-30T23:48:59.447 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:48:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1303711456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:59.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:59.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:59 smithi173 ceph-mon[20554]: pgmap v1618: 37 pgs: 4 creating+peering, 12 unknown, 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:59 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:48:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:59 smithi173 ceph-mon[20554]: osdmap e726: 12 total, 12 up, 12 in 2024-08-30T23:48:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:48:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1303711456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:59.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:48:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:59 smithi106 ceph-mon[16603]: pgmap v1618: 37 pgs: 4 creating+peering, 12 unknown, 21 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:48:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:59 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:48:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:59 smithi106 ceph-mon[16603]: osdmap e726: 12 total, 12 up, 12 in 2024-08-30T23:48:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:48:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1303711456' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:48:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:59.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:48:59.663 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:48:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:48:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:48:59.663 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:48:59.663 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:48:59.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:48:59.752 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:48:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:48:59.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:48:59.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:48:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:48:59.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:48:59.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:00.006 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:00.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:00.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:00.199 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:00.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:00.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:00.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:00.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:00.242 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:00.242 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:00.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:00 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:00.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1234333479' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:49:00.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1303711456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:00.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:00 smithi106 ceph-mon[16603]: osdmap e727: 12 total, 12 up, 12 in 2024-08-30T23:49:00.243 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2063190826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:00.244 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-08-30T23:49:00.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:00.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:00.264 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:00.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:00.286 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:00.286 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:00 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1234333479' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:49:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1303711456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:00 smithi136 ceph-mon[20207]: osdmap e727: 12 total, 12 up, 12 in 2024-08-30T23:49:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2063190826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:00.307 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:00.308 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:00.328 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:00.328 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:00.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:49:00.349 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:00.349 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:00.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:00.379 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:00.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:00.503 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:00.572 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:00.572 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:00.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:00 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1234333479' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:49:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1303711456' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:00 smithi173 ceph-mon[20554]: osdmap e727: 12 total, 12 up, 12 in 2024-08-30T23:49:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2063190826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:00.593 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:00.593 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:00.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:00.615 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:00.615 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:00.629 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:00.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:00.636 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:00.636 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:00.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:00.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:00.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:00.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:00.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:01.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:01.065 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:49:01.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:49:01.066 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:49:01.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:49:01.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:01.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:01.140 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:01.140 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:01.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:01.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:01 smithi106 ceph-mon[16603]: pgmap v1621: 37 pgs: 1 creating+activating, 4 creating+peering, 8 unknown, 24 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1234333479' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:49:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:01 smithi106 ceph-mon[16603]: osdmap e728: 12 total, 12 up, 12 in 2024-08-30T23:49:01.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:01.386 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:01.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:01 smithi136 ceph-mon[20207]: pgmap v1621: 37 pgs: 1 creating+activating, 4 creating+peering, 8 unknown, 24 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:01.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1234333479' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:49:01.452 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:01 smithi136 ceph-mon[20207]: osdmap e728: 12 total, 12 up, 12 in 2024-08-30T23:49:01.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:01.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:01 smithi173 ceph-mon[20554]: pgmap v1621: 37 pgs: 1 creating+activating, 4 creating+peering, 8 unknown, 24 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1234333479' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:49:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:01 smithi173 ceph-mon[20554]: osdmap e728: 12 total, 12 up, 12 in 2024-08-30T23:49:01.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.632 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:01.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:01.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:01.755 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:49:01.760 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:01.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:01.888 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:01.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:01.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:02.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:49:02.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:02.118 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:02.139 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:02.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:02.263 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:02.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:02.389 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:02.424 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:02.453 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:02.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:02.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:02.547 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:02 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:49:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:02 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:49:02.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:02.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:02.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:02.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:02 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:49:02.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:02.895 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:02.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:02.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:02.917 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:02.918 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:02.918 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:02.918 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:02.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:03.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:03.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:03.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:03.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:03.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:03.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:03.398 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:03.414 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-08-30T23:49:03.442 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:49:03.442 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:49:03.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:03.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:49:03.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:49:03.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_129296 2024-08-30T23:49:03.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:49:03.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:49:03.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:03.566 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:03 smithi106 ceph-mon[16603]: pgmap v1623: 37 pgs: 1 creating+activating, 4 creating+peering, 1 unknown, 31 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:03.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:03 smithi173 ceph-mon[20554]: pgmap v1623: 37 pgs: 1 creating+activating, 4 creating+peering, 1 unknown, 31 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:03.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:03.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:03.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:03.781 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:03.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:03 smithi136 ceph-mon[20207]: pgmap v1623: 37 pgs: 1 creating+activating, 4 creating+peering, 1 unknown, 31 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:03.812 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:49:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:03.858 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:03.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:49:03.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:03.929 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:03.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:04.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:04.054 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:04.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:04.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:04.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:04.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:04.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:04.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:04.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:04.502 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1260 1244 77.7459 77.75 0.0353002 0.0125464 2024-08-30T23:49:04.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:04.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:04.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:04.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:04.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:04.798 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:04.808 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:04.808 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:04.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:04.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:04.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:05.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:05.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:05.062 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:05.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:05.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:05.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:05.292 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:05.311 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:49:05.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:05.416 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:05.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 2278 2262 70.683 63.625 0.00172505 0.0141116 2024-08-30T23:49:05.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:05.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:05 smithi136 ceph-mon[20207]: pgmap v1624: 37 pgs: 1 creating+activating, 36 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3704154641' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.541 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:05.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:05 smithi173 ceph-mon[20554]: pgmap v1624: 37 pgs: 1 creating+activating, 36 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3704154641' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:05.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:05 smithi106 ceph-mon[16603]: pgmap v1624: 37 pgs: 1 creating+activating, 36 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-30T23:49:05.603 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3704154641' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:05.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:05.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:49:05.673 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:05.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:05.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:05.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:05.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:05.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:06.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:06.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:06.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:06.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:06.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:06.309 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:06.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:06.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:06.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:06.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 4231 4215 87.8065 122.062 0.00167451 0.0113533 2024-08-30T23:49:06.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:06.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:06 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3704154641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:49:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:06 smithi173 ceph-mon[20554]: osdmap e729: 12 total, 12 up, 12 in 2024-08-30T23:49:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:06 smithi173 ceph-mon[20554]: osdmap e730: 12 total, 12 up, 12 in 2024-08-30T23:49:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:06 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3704154641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:49:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:06 smithi106 ceph-mon[16603]: osdmap e729: 12 total, 12 up, 12 in 2024-08-30T23:49:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:06 smithi106 ceph-mon[16603]: osdmap e730: 12 total, 12 up, 12 in 2024-08-30T23:49:06.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:06.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:06.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:06 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3704154641' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:49:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:06 smithi136 ceph-mon[20207]: osdmap e729: 12 total, 12 up, 12 in 2024-08-30T23:49:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:06 smithi136 ceph-mon[20207]: osdmap e730: 12 total, 12 up, 12 in 2024-08-30T23:49:06.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:06.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:06.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:06.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:06.968 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:06.969 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:06.970 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:07.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:07.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:07.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:07.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:07.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:07.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:07.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:07.451 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:07.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6234 6218 97.1494 125.188 0.00146309 0.010267 2024-08-30T23:49:07.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:07 smithi173 ceph-mon[20554]: pgmap v1626: 37 pgs: 37 active+clean; 22 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s wr, 52 op/s 2024-08-30T23:49:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:07.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:07 smithi106 ceph-mon[16603]: pgmap v1626: 37 pgs: 37 active+clean; 22 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s wr, 52 op/s 2024-08-30T23:49:07.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:07.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:07.714 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:07.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:49:07.723 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:07.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:07 smithi136 ceph-mon[20207]: pgmap v1626: 37 pgs: 37 active+clean; 22 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s wr, 52 op/s 2024-08-30T23:49:07.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:07.826 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:07.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:07.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:07.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:08.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:08.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:08.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:08.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:08.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:08.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:08.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:08.462 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:08.467 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:08.467 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:08.492 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:08.492 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:08.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 15 7554 7539 94.2307 82.5625 0.0255289 0.0105625 2024-08-30T23:49:08.516 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:08.516 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:08.543 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:08.543 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:08.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:08.568 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:08.569 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:08.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:08.593 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:08.593 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:08.618 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:08.618 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:08.649 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:08.649 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:08.672 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:08.673 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:08.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:08.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:08.696 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:08.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:08.724 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:08.724 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:08.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:49:08.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:08.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:08.775 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:49:08.775 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:49:08.775 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:49:08.775 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:49:08.775 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:49:08.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:08.855 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:08.957 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:08.982 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:08.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:09.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:09.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:09.209 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:09.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:09.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:09.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:09.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:09.484 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:09.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 8815 8799 91.6496 78.75 0.00240836 0.0108992 2024-08-30T23:49:09.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:09 smithi106 ceph-mon[16603]: pgmap v1628: 37 pgs: 37 active+clean; 225 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 481 op/s 2024-08-30T23:49:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2586141675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:09.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:09.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:09.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:09.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:49:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:09 smithi136 ceph-mon[20207]: pgmap v1628: 37 pgs: 37 active+clean; 225 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 481 op/s 2024-08-30T23:49:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2586141675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:09.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:09 smithi173 ceph-mon[20554]: pgmap v1628: 37 pgs: 37 active+clean; 225 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 30 MiB/s wr, 481 op/s 2024-08-30T23:49:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2586141675' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:09.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:09.871 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:09.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:09.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:09.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:10.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:10.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:10.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:10.242 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:10.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:10.254 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:10.347 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:10.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:10.473 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:10.495 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:10.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 15 11123 11108 99.1714 144.312 0.00240472 0.0100712 2024-08-30T23:49:10.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:10 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:49:10.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:10 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:49:10.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:10.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:10.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:10.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:10 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:49:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:10 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:49:10.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:10 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:49:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:10 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:49:10.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:10.873 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:10.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:10.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:10.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:11.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:11.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:11.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:11.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:11.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:11.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:11.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:11.490 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:11.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 15 12754 12739 99.5166 101.938 0.00198347 0.0100286 2024-08-30T23:49:11.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:11 smithi106 ceph-mon[16603]: pgmap v1629: 37 pgs: 37 active+clean; 343 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 684 op/s 2024-08-30T23:49:11.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:11 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:49:11.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:11 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:49:11.520 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:11.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:11.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:11.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:11.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:11 smithi136 ceph-mon[20207]: pgmap v1629: 37 pgs: 37 active+clean; 343 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 684 op/s 2024-08-30T23:49:11.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:11 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:49:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:11 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:49:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:11 smithi173 ceph-mon[20554]: pgmap v1629: 37 pgs: 37 active+clean; 343 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 684 op/s 2024-08-30T23:49:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:11 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:49:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:11 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:49:11.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:11.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:11.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:11.994 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:11.994 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:11.994 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:11.994 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:11.994 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:11.994 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:11.995 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:11.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:11.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:11.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:11.995 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:11.995 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:11.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:12.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:12.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:12.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:12.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:12.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:49:12.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:12.347 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:49:12.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:12.396 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:12.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:12.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 14012 13996 97.1877 78.5625 0.0019107 0.0102566 2024-08-30T23:49:12.521 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1469011163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:12 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:12.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:12.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:12.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:12.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1469011163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:12.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:12 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1469011163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:12.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:12 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:12.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:12.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:12.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:13.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:13.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:13.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:13.145 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:13.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:13.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:13.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:13.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:13.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:13.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:13.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 15107 15091 94.3121 68.4375 0.0265458 0.010593 2024-08-30T23:49:13.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:13 smithi106 ceph-mon[16603]: pgmap v1630: 37 pgs: 37 active+clean; 580 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-08-30T23:49:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:13 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:49:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:13 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:49:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:13 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:13 smithi106 ceph-mon[16603]: osdmap e731: 12 total, 12 up, 12 in 2024-08-30T23:49:13.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:13.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:13.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:13.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:13 smithi136 ceph-mon[20207]: pgmap v1630: 37 pgs: 37 active+clean; 580 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-08-30T23:49:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:13 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:49:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:13 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:49:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:13 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:13 smithi136 ceph-mon[20207]: osdmap e731: 12 total, 12 up, 12 in 2024-08-30T23:49:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:13 smithi173 ceph-mon[20554]: pgmap v1630: 37 pgs: 37 active+clean; 580 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-08-30T23:49:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:13 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:49:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:13 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:49:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:13 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:13 smithi173 ceph-mon[20554]: osdmap e731: 12 total, 12 up, 12 in 2024-08-30T23:49:13.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:13.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:13.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:14.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:14.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:14.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:14.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:14.277 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:14.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:14.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:14.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:14.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 15 17167 17152 97.4476 128.812 0.00225412 0.010254 2024-08-30T23:49:14.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:14.567 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:14.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:14.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:14.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:14.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:14.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:14.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:15.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:15.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:15.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:15.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:15.229 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:15.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:49:15.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:15.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:15.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:15.429 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:15 smithi106 ceph-mon[16603]: pgmap v1632: 37 pgs: 37 active+clean; 732 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-30T23:49:15.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:15.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:15.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:15 smithi106 ceph-mon[16603]: osdmap e732: 12 total, 12 up, 12 in 2024-08-30T23:49:15.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:15 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:15.430 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:15 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:15.441 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:15.461 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:49:15.461 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:c8aed71c:::benchmark_data_smithi173_87639_object26550:head)])", 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 561.11814113200001, 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94060026299999999, 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311179+0000", 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.463 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311180+0000", 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5199999999999996e-07 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:54.311185+0000", 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8019999999999999e-06 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.249924+0000", 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93873898200000006 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.249936+0000", 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2034e-05 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.250051+0000", 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115751 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.251779+0000", 2024-08-30T23:49:15.465 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017280419999999999 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 458/432 [PushReplyOp(5:cdfa4373:::benchmark_data_smithi173_87639_object26587:head)])", 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 560.14014047800003, 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64991660600000001, 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.466 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.467 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289180+0000", 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2300000000000001e-07 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.289185+0000", 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2810000000000002e-06 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937334+0000", 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64814879999999997 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937343+0000", 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6199999999999994e-06 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.937437+0000", 2024-08-30T23:49:15.469 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3407999999999997e-05 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:55.939096+0000", 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001659874 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.470 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:c1c728e1:::benchmark_data_smithi173_87639_object28064:head)])", 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 558.13794386799998, 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58924576699999998, 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291376+0000", 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291377+0000", 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8200000000000001e-07 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.291379+0000", 2024-08-30T23:49:15.472 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.17e-06 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.878910+0000", 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58753071199999995 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.878927+0000", 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7733000000000001e-05 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.473 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.879062+0000", 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013429699999999999 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:57.880622+0000", 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015604729999999999 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:c3b8f5dc:::benchmark_data_smithi173_87639_object28564:head)])", 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:49:15.474 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 556.30764623000005, 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.82014575300000003, 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.475 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121674+0000", 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8200000000000001e-07 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.121677+0000", 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.807e-06 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.476 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939832+0000", 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81815479400000002 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939842+0000", 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9399999999999997e-06 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.477 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.478 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.478 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.939959+0000", 2024-08-30T23:49:15.478 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117068 2024-08-30T23:49:15.478 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.478 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:39:59.941820+0000", 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001860762 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 459/432 [PushReplyOp(5:ca3229d2:::benchmark_data_smithi173_87639_object28892:head)])", 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 555.24966625100001, 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62786811200000003, 2024-08-30T23:49:15.479 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.480 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179654+0000", 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.35e-07 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.179657+0000", 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4949999999999998e-06 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.805946+0000", 2024-08-30T23:49:15.481 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62628936400000002 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.805956+0000", 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7599999999999997e-06 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.806029+0000", 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3069000000000001e-05 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.482 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:00.807522+0000", 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001493189 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 462/432 [PushReplyOp(5:c69c4cd1:::benchmark_data_smithi173_87639_object33999:head)])", 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 549.13564260199996, 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71283231999999996, 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.483 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.484 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293678+0000", 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1300000000000001e-07 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:06.293681+0000", 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6290000000000002e-06 2024-08-30T23:49:15.485 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004558+0000", 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71087762700000001 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004576+0000", 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7363999999999998e-05 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.486 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.004759+0000", 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018305599999999999 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:07.006510+0000", 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017513310000000001 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.487 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 462/432 [PushReplyOp(5:c1f40d66:::benchmark_data_smithi173_87639_object34858:head)])", 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:08.049946+0000", 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 547.37937395999995, 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72198392099999997, 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049946+0000", 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.488 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.489 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049946+0000", 2024-08-30T23:49:15.489 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.489 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.489 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.489 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049947+0000", 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7299999999999996e-07 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.049951+0000", 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7280000000000001e-06 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.770680+0000", 2024-08-30T23:49:15.492 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72072939499999999 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.770689+0000", 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8969999999999997e-06 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.770824+0000", 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013500599999999999 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.493 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:08.771930+0000", 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011063220000000001 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 464/432 [PushReplyOp(5:ccd525d4:::benchmark_data_smithi173_87639_object38189:head)])", 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:11.781189+0000", 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 543.64813079199996, 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70230110000000001, 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.494 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781189+0000", 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781189+0000", 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.495 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781190+0000", 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5499999999999998e-07 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:11.781193+0000", 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5410000000000001e-06 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.481528+0000", 2024-08-30T23:49:15.496 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70033474699999998 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.481544+0000", 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6232e-05 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.481728+0000", 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018372000000000001 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.497 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:12.483491+0000", 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017624050000000001 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 469/432 [PushReplyOp(5:c90b87d7:::benchmark_data_smithi173_87639_object53091:head)])", 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 530.11270938999996, 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.59203615200000004, 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.498 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.499 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316611+0000", 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.58e-07 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.316614+0000", 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0469999999999998e-06 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.907920+0000", 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.59130516200000005 2024-08-30T23:49:15.500 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.907927+0000", 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.469e-06 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.907990+0000", 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3126999999999994e-05 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.501 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:25.908647+0000", 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00065688900000000002 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.3 470/432 [PushReplyOp(5:c2ca99af:::benchmark_data_smithi173_87639_object62935:head)])", 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 522.90561351099996, 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56332729199999998, 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.502 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.503 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523707+0000", 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4000000000000002e-07 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:32.523711+0000", 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6780000000000002e-06 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.086521+0000", 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56280982599999996 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.504 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.086569+0000", 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8012e-05 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.086674+0000", 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105009 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:40:33.087034+0000", 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00036022700000000002 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.505 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.18 503/491 [PushReplyOp(5:1eedc20c:::benchmark_data_smithi173_87639_object10777:head)])", 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:33.998519+0000", 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 461.430801116, 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.602415064, 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.506 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998519+0000", 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998519+0000", 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998520+0000", 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4499999999999997e-07 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.507 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:33.998523+0000", 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7290000000000002e-06 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.598369+0000", 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.59984580700000001 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.508 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.598377+0000", 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4919999999999996e-06 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.598450+0000", 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2991999999999995e-05 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:34.600934+0000", 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024845990000000001 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.509 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.12 507/491 [PushReplyOp(5:483c8803:::benchmark_data_smithi173_87639_object15672:head)])", 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 457.18517375800002, 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54423049400000001, 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.510 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244147+0000", 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1699999999999997e-07 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.511 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.244152+0000", 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0019999999999997e-06 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.786452+0000", 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54229973399999998 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.786459+0000", 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3730000000000004e-06 2024-08-30T23:49:15.512 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.786530+0000", 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1051999999999997e-05 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:38.788377+0000", 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018466159999999999 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.513 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.18 512/491 [PushReplyOp(5:1f0375e1:::benchmark_data_smithi173_87639_object35727:head)])", 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:41:46.007806+0000", 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 449.421513844, 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.57058524300000002, 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007806+0000", 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.514 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007806+0000", 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007807+0000", 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1900000000000002e-07 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.007812+0000", 2024-08-30T23:49:15.515 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9699999999999998e-06 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.576940+0000", 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56912784500000002 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.576947+0000", 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6969999999999996e-06 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.516 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.577003+0000", 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5945000000000001e-05 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:41:46.578392+0000", 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013890669999999999 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.12 518/491 [PushReplyOp(5:4b63bb01:::benchmark_data_smithi173_87639_object67919:head)])", 2024-08-30T23:49:15.517 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:01.855351+0000", 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 433.573969003, 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80715373199999996, 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855351+0000", 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.518 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855351+0000", 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855352+0000", 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4299999999999998e-07 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:01.855355+0000", 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.112e-06 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662036+0000", 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80668152599999998 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662044+0000", 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9400000000000002e-06 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662110+0000", 2024-08-30T23:49:15.520 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6015999999999999e-05 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:02.662505+0000", 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039469499999999997 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:109334 5.2 e552/488 5:41d12184:::benchmark_data_smithi173_87639_object29094:head v 552'6006, mlcod=551'6003)", 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:00.876275+0000", 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.55304520300001, 2024-08-30T23:49:15.521 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64198491599999996, 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876275+0000", 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876275+0000", 2024-08-30T23:49:15.522 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876277+0000", 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4610000000000001e-06 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:00.876287+0000", 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0049e-05 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.523 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516376+0000", 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.64008948300000001 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516383+0000", 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3540000000000002e-06 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516442+0000", 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8616000000000003e-05 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.524 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.516468+0000", 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5652000000000001e-05 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.518218+0000", 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017507009999999999 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:01.518260+0000", 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1600000000000002e-05 2024-08-30T23:49:15.525 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:93442 6.a e653/619 6:562d5a8b:::benchmark_data_smithi173_114000_object93441:head v 653'5805, mlcod=652'5801)", 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 193.52878765299999, 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.287759343, 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.526 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900533+0000", 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6100000000000002e-07 2024-08-30T23:49:15.527 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:01.900581+0000", 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8588000000000003e-05 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185793+0000", 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28521170000000001 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.528 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185802+0000", 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3489999999999992e-06 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185856+0000", 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4540000000000003e-05 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.185889+0000", 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2789000000000001e-05 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.529 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.188257+0000", 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002368448 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:02.188292+0000", 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4668000000000001e-05 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:158670 6.a e687/619 6:589e06ca:::benchmark_data_smithi173_114000_object58872:head v 687'9830, mlcod=687'9826)", 2024-08-30T23:49:15.530 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:44.592258+0000", 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 90.837062095999997, 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20296215300000001, 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.592258+0000", 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.531 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.592258+0000", 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.592260+0000", 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.79e-06 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.592268+0000", 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5959999999999998e-06 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.532 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.789566+0000", 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.197298315 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.789574+0000", 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0439999999999996e-06 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.789728+0000", 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000153583 2024-08-30T23:49:15.533 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.789774+0000", 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6646e-05 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.795068+0000", 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052937540000000003 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:44.795220+0000", 2024-08-30T23:49:15.534 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015242500000000001 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:185798 6.a e706/619 6:57949d66:::benchmark_data_smithi173_114000_object86000:head v 706'11520, mlcod=706'11517)", 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:25.389247+0000", 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 50.040073249999999, 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35204086899999998, 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.389247+0000", 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.389247+0000", 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.389248+0000", 2024-08-30T23:49:15.536 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.17e-06 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.389255+0000", 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5980000000000002e-06 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.736447+0000", 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34719201199999999 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.537 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.736456+0000", 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3759999999999997e-06 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.736530+0000", 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3913999999999997e-05 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.736558+0000", 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7441e-05 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.538 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.741242+0000", 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0046842949999999998 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.741288+0000", 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6063000000000003e-05 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:193158 6.0 e715/619 6:05148cb4:::benchmark_data_smithi173_114000_object93360:head v 715'12104, mlcod=714'12101)", 2024-08-30T23:49:15.539 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:37.198990+0000", 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 38.230330446000004, 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.248075302, 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.198990+0000", 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.540 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.198990+0000", 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.198991+0000", 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1990000000000001e-06 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.198999+0000", 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2169999999999994e-06 2024-08-30T23:49:15.541 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.442900+0000", 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24390107999999999 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.442910+0000", 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6940000000000007e-06 2024-08-30T23:49:15.542 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.442976+0000", 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5722999999999994e-05 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.443000+0000", 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4039000000000001e-05 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:15.543 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.447019+0000", 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040188699999999999 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.447065+0000", 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6480000000000002e-05 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:14756 7.d e731/725 7:bb9aa892:::benchmark_data_smithi173_129296_object14755:head v 731'934, mlcod=731'931)", 2024-08-30T23:49:15.544 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:13.200548+0000", 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.2287721989999998, 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.056980872000000002, 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.200548+0000", 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.545 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.200548+0000", 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.200549+0000", 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6799999999999999e-07 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.200594+0000", 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5519999999999998e-05 2024-08-30T23:49:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.256557+0000", 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.055962540999999998 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.256566+0000", 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5730000000000008e-06 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.256644+0000", 2024-08-30T23:49:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8194000000000004e-05 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.256665+0000", 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1061e-05 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.257486+0000", 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082148599999999996 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:15.548 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:13.257529+0000", 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2629000000000001e-05 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:15.549 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:49:15.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:15.550 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 18058 18042 93.962 55.625 0.104321 0.0105898 2024-08-30T23:49:15.571 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:15.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:15 smithi136 ceph-mon[20207]: pgmap v1632: 37 pgs: 37 active+clean; 732 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-30T23:49:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:15 smithi136 ceph-mon[20207]: osdmap e732: 12 total, 12 up, 12 in 2024-08-30T23:49:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:15 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:15 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:15.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:15 smithi173 ceph-mon[20554]: pgmap v1632: 37 pgs: 37 active+clean; 732 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-08-30T23:49:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:15 smithi173 ceph-mon[20554]: osdmap e732: 12 total, 12 up, 12 in 2024-08-30T23:49:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:15 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:15 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:15.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:15.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:15.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:15.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:16.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:16.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:16.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:16.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:16.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:16.331 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:16.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:16.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:16.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 19387 19371 93.1231 83.0625 0.00780065 0.01073 2024-08-30T23:49:16.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:16.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:16 smithi106 ceph-mon[16603]: osdmap e733: 12 total, 12 up, 12 in 2024-08-30T23:49:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:16 smithi106 ceph-mon[16603]: osdmap e734: 12 total, 12 up, 12 in 2024-08-30T23:49:16.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:16.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:16 smithi136 ceph-mon[20207]: osdmap e733: 12 total, 12 up, 12 in 2024-08-30T23:49:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:16 smithi136 ceph-mon[20207]: osdmap e734: 12 total, 12 up, 12 in 2024-08-30T23:49:16.807 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:16.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:16 smithi173 ceph-mon[20554]: osdmap e733: 12 total, 12 up, 12 in 2024-08-30T23:49:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:16 smithi173 ceph-mon[20554]: osdmap e734: 12 total, 12 up, 12 in 2024-08-30T23:49:16.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:16.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:16.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:49:16.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:16.959 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:16.982 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:17.038 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:17.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.089 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:17.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:17.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:17.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:17.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:17.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:17.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:17.463 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:17.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 20662 20646 92.1629 79.6875 0.00910303 0.0108215 2024-08-30T23:49:17.537 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:17.537 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:17.537 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:17 smithi106 ceph-mon[16603]: pgmap v1635: 37 pgs: 37 active+clean; 884 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-30T23:49:17.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:17.567 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:17.568 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:17.594 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:17.594 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.596 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:17.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:17.619 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:17.620 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:17.648 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:17.648 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:17.677 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:17.677 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:17.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:17.704 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:17.704 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:17.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:17.730 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:17.731 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:17.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:17.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:17.786 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:17.787 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:17.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:17 smithi136 ceph-mon[20207]: pgmap v1635: 37 pgs: 37 active+clean; 884 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-30T23:49:17.815 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:17.815 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:17 smithi173 ceph-mon[20554]: pgmap v1635: 37 pgs: 37 active+clean; 884 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-30T23:49:17.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:17.839 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:17.840 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:17.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:17.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:17.974 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:17.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:18.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:18.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:18.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.227 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:18.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:18.285 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:18.285 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:18.285 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:18.285 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:18.285 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:18.287 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:18.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:18.354 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:18.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:18.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:18.503 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 22119 22103 92.0891 91.0625 0.00289894 0.0108478 2024-08-30T23:49:18.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:49:18.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:18.604 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:18.612 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:18.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:18.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/815440775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/815440775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:18.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:18.862 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/815440775' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.870 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:18.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:18.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:18.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:18.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:19.065 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:19.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:49:19.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:19.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:19.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:19.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:19.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:19.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:19.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 23919 23903 93.3642 112.5 0.00490597 0.0107029 2024-08-30T23:49:19.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:19.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:19.553 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:19.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:19.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:19.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:19 smithi106 ceph-mon[16603]: pgmap v1637: 37 pgs: 37 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-30T23:49:19.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:19.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:19 smithi136 ceph-mon[20207]: pgmap v1637: 37 pgs: 37 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-30T23:49:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:19 smithi173 ceph-mon[20554]: pgmap v1637: 37 pgs: 37 active+clean; 1.1 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.30k op/s 2024-08-30T23:49:19.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:19.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:19.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:20.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:20.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:20.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:20.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:20.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:20.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:20.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:20.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:20.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 25297 25281 92.938 86.125 0.0187976 0.0107511 2024-08-30T23:49:20.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:20.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:20.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:20.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:20.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:20.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:20.796 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:20.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:20.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:20.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:21.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:21.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:21.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:21.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:21.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:21.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:21.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:21.408 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:21.422 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:21.438 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:21.438 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:21.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 26332 26316 91.3682 64.6875 0.00282598 0.01093 2024-08-30T23:49:21.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:21.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:49:21.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:21.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:21 smithi106 ceph-mon[16603]: pgmap v1638: 37 pgs: 37 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s 2024-08-30T23:49:21.625 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:21 smithi173 ceph-mon[20554]: pgmap v1638: 37 pgs: 37 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s 2024-08-30T23:49:21.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:21.670 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:21 smithi136 ceph-mon[20207]: pgmap v1638: 37 pgs: 37 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.45k op/s 2024-08-30T23:49:21.797 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:21.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:21.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:21.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:22.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:22.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:22.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.176 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:22.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:22.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:22.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:22.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:22.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:22.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 27576 27560 90.6511 77.75 0.0097769 0.0110204 2024-08-30T23:49:22.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:22 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:22.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:22 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:22.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:22 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:22.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:22 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:22.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:22 smithi106 ceph-mon[16603]: osdmap e735: 12 total, 12 up, 12 in 2024-08-30T23:49:22.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:22.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:22.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:22 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:22.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:22 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:22.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:22 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:22.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:22 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:22.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:22 smithi173 ceph-mon[20554]: osdmap e735: 12 total, 12 up, 12 in 2024-08-30T23:49:22.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:22.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.674 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:22.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:22.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:49:22.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:22 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:22 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:22 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:22 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:22 smithi136 ceph-mon[20207]: osdmap e735: 12 total, 12 up, 12 in 2024-08-30T23:49:22.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:22.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:22.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:23.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:23.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:23.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:23.183 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:23.202 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:23.207 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:23.208 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:23.208 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:23.208 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:23.208 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:23.208 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:23.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:23.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:23.381 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:49:23.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:23.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:23.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:49:23.504929+0000 min lat: 0.00104498 max lat: 0.307952 avg lat: 0.011112 2024-08-30T23:49:23.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:49:23.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 28794 28778 89.9245 76.125 0.0119178 0.011112 2024-08-30T23:49:23.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:23.564 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:23.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:23.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:23.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:23 smithi136 ceph-mon[20207]: pgmap v1639: 37 pgs: 37 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-30T23:49:23.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3778821739' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:23.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:23 smithi106 ceph-mon[16603]: pgmap v1639: 37 pgs: 37 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-30T23:49:23.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3778821739' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:23.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:23.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.813 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:23 smithi173 ceph-mon[20554]: pgmap v1639: 37 pgs: 37 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-30T23:49:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3778821739' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:23.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:23.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:23.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:24.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:24.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:24.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:24.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:24.242 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:24.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-30T23:49:24.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:24.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:24.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:24.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:24.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 29836 29820 88.743 65.125 0.00226016 0.0112164 2024-08-30T23:49:24.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:24.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:24 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3778821739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:49:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:24 smithi106 ceph-mon[16603]: osdmap e736: 12 total, 12 up, 12 in 2024-08-30T23:49:24.635 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:24 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:24.636 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3778821739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:49:24.636 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:24 smithi136 ceph-mon[20207]: osdmap e736: 12 total, 12 up, 12 in 2024-08-30T23:49:24.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:24.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:24.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:24.820 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:24.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:24 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:24.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3778821739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:49:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:24 smithi173 ceph-mon[20554]: osdmap e736: 12 total, 12 up, 12 in 2024-08-30T23:49:24.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:24.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:24.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:25.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:25.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:25.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:25.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:25.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:25.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:25.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:25.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:25.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:25.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 30950 30934 87.8737 69.625 0.0128236 0.0113733 2024-08-30T23:49:25.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:25.576 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:25.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:25 smithi106 ceph-mon[16603]: pgmap v1642: 37 pgs: 37 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-08-30T23:49:25.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:25.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:25.711 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:25.711 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:25 smithi136 ceph-mon[20207]: pgmap v1642: 37 pgs: 37 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-08-30T23:49:25.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:25 smithi173 ceph-mon[20554]: pgmap v1642: 37 pgs: 37 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-08-30T23:49:25.834 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:25.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:25.966 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:25.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:25.990 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:26.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:26.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:26.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.234 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:26.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:26.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:26.357 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:26.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:26.484 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:26.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 32615 32599 88.5772 104.062 0.00231069 0.0112819 2024-08-30T23:49:26.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:26.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:49:26.609 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:26.644 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:26.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:26.736 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:26.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:26.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:26.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:26.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:26.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:27.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:27.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:27.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:27.239 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:27.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:27.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:27.416 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:27 smithi106 ceph-mon[16603]: pgmap v1643: 37 pgs: 37 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:49:27.457 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:27.458 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:27.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:27.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:49:27.486 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:27.486 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:27.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:27.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 33858 33842 88.1232 77.6875 0.0106877 0.0113415 2024-08-30T23:49:27.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:27.519 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:27.558 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:27.559 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:27.597 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:27.597 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:27.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:27.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:27.633 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:27.633 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:27.652 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:27.659 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:27.659 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:27.692 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:27.692 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:27.719 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:27.720 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:27.720 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:27.720 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:27.720 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:27.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:27.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:27.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:27.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:27.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:27.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:27.775 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:27.775 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:27.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:27 smithi136 ceph-mon[20207]: pgmap v1643: 37 pgs: 37 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:49:27.799 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:27.799 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:27 smithi173 ceph-mon[20554]: pgmap v1643: 37 pgs: 37 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:49:27.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:27.873 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:27.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:27.997 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:27.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:28.099 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:28.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:28.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:28.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:28.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:28.375 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:28.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:28.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 15 34846 34831 87.0706 61.8125 0.00451251 0.0114751 2024-08-30T23:49:28.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:28.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:28.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:28.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:28.765 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:28.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/603372646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:28.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/603372646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:28.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/603372646' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:28.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:28.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:28.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:28.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:29.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:29.103 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:29.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-30T23:49:29.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:29.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:29.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:29.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:29.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:29.398 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:29.501 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:29.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 35925 35909 86.3129 67.375 0.044064 0.0115775 2024-08-30T23:49:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.526 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:29.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:29.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:29.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:29.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:29 smithi106 ceph-mon[16603]: pgmap v1644: 37 pgs: 37 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:95258 5.f e532/447 5:f9703d87:::benchmark_data_smithi173_87639_object15018:head v 532'5941, mlcod=532'5938)", 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:27.981612+0000", 2024-08-30T23:49:29.726 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 421.74234244600001, 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24440374600000001, 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981612+0000", 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981612+0000", 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981613+0000", 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8100000000000001e-07 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:27.981621+0000", 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9160000000000001e-06 2024-08-30T23:49:29.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.222924+0000", 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24130264400000001 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.222931+0000", 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1339999999999998e-06 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.223002+0000", 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1369000000000001e-05 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.223030+0000", 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7460000000000001e-05 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.225981+0000", 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029507489999999999 2024-08-30T23:49:29.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:28.226016+0000", 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5493000000000001e-05 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.14 542/530 [PushReplyOp(5:2b316a56:::benchmark_data_smithi173_87639_object66576:head)])", 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:41.484161+0000", 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 408.23979363400002, 2024-08-30T23:49:29.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54557016300000005, 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484161+0000", 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484161+0000", 2024-08-30T23:49:29.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484162+0000", 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5799999999999998e-07 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:41.484168+0000", 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7230000000000001e-06 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.028873+0000", 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54470516400000002 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.028881+0000", 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8560000000000007e-06 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.028935+0000", 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4759999999999997e-05 2024-08-30T23:49:29.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.029731+0000", 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00079590200000000002 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.14 542/530 [PushReplyOp(5:2fcba5e8:::benchmark_data_smithi173_87639_object68452:head)])", 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 407.164933168, 2024-08-30T23:49:29.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23938505400000001, 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559022+0000", 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7800000000000001e-07 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.559028+0000", 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1919999999999996e-06 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.797504+0000", 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23847607300000001 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.797512+0000", 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8810000000000002e-06 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.797579+0000", 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6188999999999999e-05 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:42.798407+0000", 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00082794100000000001 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.739 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:103089 5.6 e546/488 5:612727a4:::benchmark_data_smithi173_87639_object22849:head v 546'5861, mlcod=545'5858)", 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:49.898028+0000", 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 399.82592655899998, 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32197586099999997, 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898028+0000", 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898028+0000", 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898029+0000", 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.322e-06 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.898036+0000", 2024-08-30T23:49:29.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1040000000000003e-06 2024-08-30T23:49:29.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218477+0000", 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32044120700000001 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218485+0000", 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9869999999999999e-06 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218569+0000", 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4227000000000005e-05 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.218588+0000", 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9358999999999999e-05 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.219971+0000", 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001382882 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.220004+0000", 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2772999999999999e-05 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:103106 5.a e546/457 5:5e3a0b2f:::benchmark_data_smithi173_87639_object22866:head v 546'6510, mlcod=546'6507)", 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:49.972574+0000", 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 399.75138099999998, 2024-08-30T23:49:29.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45312410199999997, 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972574+0000", 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972574+0000", 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972575+0000", 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.093e-06 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:49.972582+0000", 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1060000000000002e-06 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422561+0000", 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44997882300000003 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422565+0000", 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2470000000000002e-06 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422614+0000", 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8711999999999997e-05 2024-08-30T23:49:29.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.422637+0000", 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3771999999999999e-05 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.425662+0000", 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030249209999999999 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:50.425698+0000", 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5428e-05 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(5.f 554/536 [PushReplyOp(5:ffa89edb:::benchmark_data_smithi173_87639_object39711:head)])", 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:03.574448+0000", 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 386.14950627899998, 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23876501999999999, 2024-08-30T23:49:29.750 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574448+0000", 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574448+0000", 2024-08-30T23:49:29.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574449+0000", 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9299999999999997e-07 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.574456+0000", 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6340000000000003e-06 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.810549+0000", 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23609303500000001 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.810555+0000", 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.896e-06 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.810616+0000", 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1427000000000007e-05 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:03.813213+0000", 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0025972349999999998 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:123150 5.f 5:fdac4e98:::benchmark_data_smithi173_87639_object42910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e568)", 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:26.004606+0000", 2024-08-30T23:49:29.754 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 363.719348244, 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.248495243, 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 123150 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004606+0000", 2024-08-30T23:49:29.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004606+0000", 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004607+0000", 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.756 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004612+0000", 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3320000000000002e-06 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004623+0000", 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1623999999999999e-05 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.004629+0000", 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.507e-06 2024-08-30T23:49:29.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.007575+0000", 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0029462260000000001 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.007731+0000", 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155818 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,5,11", 2024-08-30T23:49:29.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.007860+0000", 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012949500000000001 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.011684+0000", 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038234950000000001 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.021736+0000", 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01005195 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.024827+0000", 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030912790000000002 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251505+0000", 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22667806800000001 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251564+0000", 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8959000000000003e-05 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.253102+0000", 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015375199999999999 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.26173.0:123150 5.f e568/566 ondisk, result = 0)", 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:26.012520+0000", 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 363.71143475999997, 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24060188699999999, 2024-08-30T23:49:29.761 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012520+0000", 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012520+0000", 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012521+0000", 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0079999999999999e-06 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.012527+0000", 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0700000000000003e-06 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.245180+0000", 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23265303200000001 2024-08-30T23:49:29.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.245187+0000", 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3300000000000001e-06 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251488+0000", 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063001519999999998 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.251498+0000", 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0362e-05 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:26.253122+0000", 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0016239329999999999 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:127331 5.14 e571/546 5:2fb0fe1d:::benchmark_data_smithi173_87639_object47091:head v 571'6632, mlcod=571'6628)", 2024-08-30T23:49:29.765 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:32.168977+0000", 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 357.554977549, 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.356479241, 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168977+0000", 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168977+0000", 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168978+0000", 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6000000000000003e-07 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.168984+0000", 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8270000000000003e-06 2024-08-30T23:49:29.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524130+0000", 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.355145934 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524137+0000", 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.075e-06 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524181+0000", 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4292999999999998e-05 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.524198+0000", 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.677e-05 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525419+0000", 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012211699999999999 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:32.525456+0000", 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7412e-05 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(5.f e574,566 5:f2f1486a:::benchmark_data_smithi173_87639_object13299:head,531'5830)", 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:34.873839+0000", 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 354.85011560999999, 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26355642099999999, 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.770 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873839+0000", 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873839+0000", 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873840+0000", 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.873844+0000", 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9099999999999998e-06 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.133925+0000", 2024-08-30T23:49:29.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26008125799999998 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.133931+0000", 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5890000000000002e-06 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.135718+0000", 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0017873800000000001 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:35.137396+0000", 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001677621 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:143169 5.f 5:f2854a5a:::benchmark_data_smithi173_87639_object62929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:53.894369+0000", 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 335.829585888, 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.333810253, 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.774 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143169 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894369+0000", 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894369+0000", 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894370+0000", 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3549999999999999e-06 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894376+0000", 2024-08-30T23:49:29.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.575e-06 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894388+0000", 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2132e-05 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894393+0000", 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1880000000000001e-06 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894439+0000", 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5571000000000001e-05 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894549+0000", 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011027 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,5,11", 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.894646+0000", 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7369999999999998e-05 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.897891+0000", 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032451839999999999 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.902231+0000", 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043393670000000002 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:53.902272+0000", 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1587e-05 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:54.227011+0000", 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32473906899999999 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:54.227056+0000", 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4805000000000002e-05 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:54.228179+0000", 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011227800000000001 2024-08-30T23:49:29.780 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:47953 6.f e635/619 6:f96de401:::benchmark_data_smithi173_114000_object47952:head v 635'2915, mlcod=635'2912)", 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:45:19.013882+0000", 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 250.71007285799999, 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25471529900000001, 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.013882+0000", 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.013882+0000", 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.013883+0000", 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7100000000000011e-07 2024-08-30T23:49:29.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.013925+0000", 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1752999999999998e-05 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.267142+0000", 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25321717500000002 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.267145+0000", 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.534e-06 2024-08-30T23:49:29.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.267204+0000", 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8335000000000003e-05 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.267226+0000", 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2563999999999999e-05 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.268568+0000", 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001342017 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:45:19.268597+0000", 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8949999999999999e-05 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.785 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "replica_scrub(pg: 6.5,from:0'0,to:652'5738,epoch:654/619,start:6:abbfa831:::benchmark_data_smithi173_114000_object65616:0,end:6:ac128125:::benchmark_data_smithi173_114000_object47131:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:46:03.581016+0000", 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 206.14293882999999, 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.467085745, 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:03.581016+0000", 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:29.788 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:29.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:03.581016+0000", 2024-08-30T23:49:29.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:03.581017+0000", 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3820000000000001e-06 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:03.581026+0000", 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7220000000000005e-06 2024-08-30T23:49:29.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:04.047925+0000", 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46689942600000001 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:04.047935+0000", 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5740000000000002e-06 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:04.048042+0000", 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106823 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:04.048059+0000", 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6881999999999999e-05 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:46:04.048102+0000", 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2935999999999997e-05 2024-08-30T23:49:29.791 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(6.d 679/654 [PushOp(6:bb29f3e0:::benchmark_data_smithi173_114000_object56449:head, version: 638'3379, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:bb29f3e0:::benchmark_data_smithi173_114000_object56449:head@638'3379, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:24.863348+0000", 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 124.860606891, 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27540567799999999, 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:24.863348+0000", 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:24.863348+0000", 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:24.863349+0000", 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8599999999999997e-07 2024-08-30T23:49:29.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:24.863432+0000", 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3443999999999996e-05 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:25.135086+0000", 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27165360900000002 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:25.135101+0000", 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5231999999999999e-05 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:25.135195+0000", 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4556999999999996e-05 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:25.135207+0000", 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1491999999999999e-05 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:25.138753+0000", 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003546558 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(6.d 691/654 [PushOp(6:bc75eac3:::benchmark_data_smithi173_114000_object85166:head, version: 651'5133, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:bc75eac3:::benchmark_data_smithi173_114000_object85166:head@651'5133, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 92.249994982999993, 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.43877167099999997, 2024-08-30T23:49:29.796 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8999999999999996e-07 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.474010+0000", 2024-08-30T23:49:29.798 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9799000000000002e-05 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909183+0000", 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.435172475 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909195+0000", 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2727e-05 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.799 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909335+0000", 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014007399999999999 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909352+0000", 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6487999999999999e-05 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.800 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.912731+0000", 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033795180000000002 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27034.0:185803 6.9 e706/619 6:98dfff0c:::benchmark_data_smithi173_114000_object86005:head v 706'11889, mlcod=706'11886)", 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:25.412927+0000", 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 64.311028151000002, 2024-08-30T23:49:29.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29996426199999998, 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.412927+0000", 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.412927+0000", 2024-08-30T23:49:29.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.412928+0000", 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0440000000000001e-06 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.412932+0000", 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5399999999999997e-06 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.708835+0000", 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29590277700000001 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.708859+0000", 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3655e-05 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.708920+0000", 2024-08-30T23:49:29.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1507000000000006e-05 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.708942+0000", 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2167e-05 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.712841+0000", 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038989960000000001 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.805 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:25.712891+0000", 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9576000000000001e-05 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:193164 6.f 6:fab35f08:::benchmark_data_smithi173_114000_object93366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e715)", 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:37.221910+0000", 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 52.502044300999998, 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21345744699999999, 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.806 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 193164 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221910+0000", 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221910+0000", 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221912+0000", 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.265e-06 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221918+0000", 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9209999999999997e-06 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221930+0000", 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2559e-05 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221936+0000", 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5740000000000003e-06 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.809 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.221991+0000", 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5414999999999997e-05 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.222088+0000", 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7207999999999995e-05 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9,10", 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.222176+0000", 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7454999999999995e-05 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.225801+0000", 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036253539999999999 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.225850+0000", 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9109e-05 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.225895+0000", 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4916999999999997e-05 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.435255+0000", 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20935938500000001 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.435294+0000", 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.985e-05 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.812 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:37.435368+0000", 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3435000000000004e-05 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:199586 6.f 6:fec36aa3:::benchmark_data_smithi173_114000_object99788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.146463+0000", 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 43.577491983000002, 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.20229347, 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.813 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199586 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146463+0000", 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146463+0000", 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146464+0000", 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7740000000000001e-06 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146472+0000", 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9980000000000003e-06 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146491+0000", 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8238000000000001e-05 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146498+0000", 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6990000000000003e-06 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146572+0000", 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3807000000000007e-05 2024-08-30T23:49:29.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146726+0000", 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015353799999999999 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9,10", 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.146896+0000", 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016989800000000001 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.151969+0000", 2024-08-30T23:49:29.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050728379999999997 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152087+0000", 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011882 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152350+0000", 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026295000000000002 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.348632+0000", 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.19628189400000001 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.348678+0000", 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5626e-05 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.348756+0000", 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.839e-05 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.819 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:34182 7.a e736/725 7:5755cba8:::benchmark_data_smithi173_129296_object34181:head v 736'2107, mlcod=736'2104)", 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:27.827183+0000", 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.896771864, 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26242645599999997, 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.827183+0000", 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.827183+0000", 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.827184+0000", 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2699999999999998e-07 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.827246+0000", 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2075000000000005e-05 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.087165+0000", 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.259919495 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.087172+0000", 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4139999999999999e-06 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.087209+0000", 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7274999999999998e-05 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.087222+0000", 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2679e-05 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.089576+0000", 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023538560000000001 2024-08-30T23:49:29.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.089609+0000", 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3735000000000001e-05 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:34187 7.2 e736/725 7:423e434c:::benchmark_data_smithi173_129296_object34186:head v 736'2116, mlcod=736'2113)", 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:27.830520+0000", 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 1.8934345590000001, 2024-08-30T23:49:29.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25627948299999997, 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830520+0000", 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830520+0000", 2024-08-30T23:49:29.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830521+0000", 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8999999999999995e-07 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830545+0000", 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3988000000000001e-05 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830560+0000", 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5277e-05 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830567+0000", 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7000000000000002e-06 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830617+0000", 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0062999999999998e-05 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:27.830636+0000", 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8808e-05 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.086752+0000", 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25611659599999997 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:28.086800+0000", 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7261e-05 2024-08-30T23:49:29.828 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.829 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.829 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:29.829 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:29.829 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:49:29.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:29 smithi173 ceph-mon[20554]: pgmap v1644: 37 pgs: 37 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-30T23:49:29.830 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:29 smithi136 ceph-mon[20207]: pgmap v1644: 37 pgs: 37 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-08-30T23:49:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:29.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:29.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:29.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:29.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:30.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:30.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:30.179 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:30.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:30.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:30.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:30.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:30.463 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:30.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:30.504 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 37097 37081 85.8289 73.25 0.0229629 0.0116403 2024-08-30T23:49:30.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:30.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:30.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:30.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:30.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:30.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:30.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:30.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:30.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:49:30.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:30.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:30.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:31.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:31.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:31.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:31.209 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:31.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:31.231 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:31.231 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:31.231 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:31.231 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:31.231 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:31.231 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:31.232 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:31.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:31.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:31.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:31.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:31.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 38574 38558 86.0602 92.3125 0.00216396 0.0116103 2024-08-30T23:49:31.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:31 smithi106 ceph-mon[16603]: pgmap v1645: 37 pgs: 37 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-08-30T23:49:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:31 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:31 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:31 smithi106 ceph-mon[16603]: osdmap e737: 12 total, 12 up, 12 in 2024-08-30T23:49:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:31 smithi106 ceph-mon[16603]: osdmap e738: 12 total, 12 up, 12 in 2024-08-30T23:49:31.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:31.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:31 smithi136 ceph-mon[20207]: pgmap v1645: 37 pgs: 37 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-08-30T23:49:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:31 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:31 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:31 smithi136 ceph-mon[20207]: osdmap e737: 12 total, 12 up, 12 in 2024-08-30T23:49:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:31 smithi136 ceph-mon[20207]: osdmap e738: 12 total, 12 up, 12 in 2024-08-30T23:49:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:31 smithi173 ceph-mon[20554]: pgmap v1645: 37 pgs: 37 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-08-30T23:49:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:31 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:31 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:31 smithi173 ceph-mon[20554]: osdmap e737: 12 total, 12 up, 12 in 2024-08-30T23:49:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:31 smithi173 ceph-mon[20554]: osdmap e738: 12 total, 12 up, 12 in 2024-08-30T23:49:31.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:31.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:31.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:31.980 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:31.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:32.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:32.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:32.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:32.231 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:32.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:32.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:32.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:32.464 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:32.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-30T23:49:32.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:32.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 39236 39220 84.5192 41.375 0.00471305 0.0118208 2024-08-30T23:49:32.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:32.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:32.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:32.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:32.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:32.759 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:32.759 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:49:32.759 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:49:32.759 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:32 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:49:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:49:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:49:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:32 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:49:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:32.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:49:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:49:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:32 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:49:32.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:32.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:32.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:32.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:33.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:33.082 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:33.099 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:33.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:33.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:33.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:33.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:33.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:33.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:33.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 40666 40650 84.6808 89.375 0.00373456 0.0118024 2024-08-30T23:49:33.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:33.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:33.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:33.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:33.660 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:49:33.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:33.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:49:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:33.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: pgmap v1648: 37 pgs: 37 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:33.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:33.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:33.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2928466666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:33.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:33.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: pgmap v1648: 37 pgs: 37 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:49:33.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2928466666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:33.801 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: pgmap v1648: 37 pgs: 37 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:33.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:33.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:33.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:49:33.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2928466666' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:33.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:33.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:33.910 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:33.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:34.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:34.038 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:34.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:34.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:34.266 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:34.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:34.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:34.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:34.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 41836 41820 84.3078 73.125 0.0133077 0.0118542 2024-08-30T23:49:34.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:34.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:34 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:34 smithi106 ceph-mon[16603]: osdmap e739: 12 total, 12 up, 12 in 2024-08-30T23:49:34.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:34.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:34.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:34 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:34.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:34 smithi136 ceph-mon[20207]: osdmap e739: 12 total, 12 up, 12 in 2024-08-30T23:49:34.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:34.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:34 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:34 smithi173 ceph-mon[20554]: osdmap e739: 12 total, 12 up, 12 in 2024-08-30T23:49:34.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:34.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:34.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:35.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:35.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:35.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:35.175 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:35.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:35.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:35.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:35.429 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:35.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 43013 42997 83.9718 73.5625 0.00509759 0.0119014 2024-08-30T23:49:35.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:35.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:35.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:35.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:35.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:35.839 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:35.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:35 smithi106 ceph-mon[16603]: pgmap v1650: 37 pgs: 37 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 995 op/s 2024-08-30T23:49:35.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:35 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:35.910 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:35 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:35.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:35.970 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:35.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:35 smithi136 ceph-mon[20207]: pgmap v1650: 37 pgs: 37 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 995 op/s 2024-08-30T23:49:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:35 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:35 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:43812 7.2 7:43c89ba9:::benchmark_data_smithi173_129296_object43811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:36.027737+0000", 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.012137584, 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.012168791, 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 43812 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027737+0000", 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027737+0000", 2024-08-30T23:49:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027738+0000", 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3549999999999999e-06 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027795+0000", 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6474000000000002e-05 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027806+0000", 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1267000000000001e-05 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027813+0000", 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.533e-06 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.027862+0000", 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8776999999999997e-05 2024-08-30T23:49:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028003+0000", 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014167300000000001 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,10", 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028125+0000", 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012149799999999999 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.029487+0000", 2024-08-30T23:49:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001362631 2024-08-30T23:49:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:43813 7.6 7:61c3ef88:::benchmark_data_smithi173_129296_object43812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:36.028743+0000", 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.011131879000000001, 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.011214325000000001, 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 43813 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028743+0000", 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028743+0000", 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028744+0000", 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8299999999999995e-07 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028800+0000", 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.647e-05 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028813+0000", 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2262999999999999e-05 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028821+0000", 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4130000000000006e-06 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.028881+0000", 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0068000000000002e-05 2024-08-30T23:49:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.029021+0000", 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013950200000000001 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,10", 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.029167+0000", 2024-08-30T23:49:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000146806 2024-08-30T23:49:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.030580+0000", 2024-08-30T23:49:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001412735 2024-08-30T23:49:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:43820 7.3 7:c282ffb4:::benchmark_data_smithi173_129296_object43819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-08-30T23:49:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:36.037967+0000", 2024-08-30T23:49:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.001907962, 2024-08-30T23:49:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020288200000000002, 2024-08-30T23:49:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:49:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 43820 2024-08-30T23:49:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.037967+0000", 2024-08-30T23:49:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.037967+0000", 2024-08-30T23:49:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.037968+0000", 2024-08-30T23:49:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.42e-06 2024-08-30T23:49:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.038012+0000", 2024-08-30T23:49:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4079000000000003e-05 2024-08-30T23:49:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.038028+0000", 2024-08-30T23:49:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5973000000000001e-05 2024-08-30T23:49:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:36.062 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.038037+0000", 2024-08-30T23:49:36.062 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6109999999999994e-06 2024-08-30T23:49:36.062 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.062 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.062 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:36.063 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.038084+0000", 2024-08-30T23:49:36.063 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7107000000000001e-05 2024-08-30T23:49:36.063 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.063 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.063 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:36.064 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.038177+0000", 2024-08-30T23:49:36.064 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2990000000000002e-05 2024-08-30T23:49:36.064 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.065 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.065 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,4", 2024-08-30T23:49:36.065 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.038264+0000", 2024-08-30T23:49:36.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7274000000000005e-05 2024-08-30T23:49:36.066 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.066 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.066 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:36.066 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.039648+0000", 2024-08-30T23:49:36.066 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00138429 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:43788 7.d 7:bbb4df97:::benchmark_data_smithi173_129296_object43787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:36.007131+0000", 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.032743623999999999, 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032901822999999997, 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:36.067 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 43788 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007131+0000", 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.068 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007131+0000", 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007132+0000", 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.314e-06 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007174+0000", 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1165000000000003e-05 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.069 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007188+0000", 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4883e-05 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007196+0000", 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8120000000000008e-06 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:36.070 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007251+0000", 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4407000000000003e-05 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007382+0000", 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000131471 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.007515+0000", 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013305700000000001 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.071 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.009319+0000", 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018040899999999999 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.009671+0000", 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035163300000000001 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:43809 7.3 7:cd90245e:::benchmark_data_smithi173_129296_object43808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e740)", 2024-08-30T23:49:36.072 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:49:36.023847+0000", 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.016027454, 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016234546999999998, 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 43809 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.073 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023847+0000", 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023847+0000", 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023848+0000", 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.75e-07 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.074 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023888+0000", 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0636999999999997e-05 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023896+0000", 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.072e-06 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023903+0000", 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1179999999999997e-06 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.075 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.023959+0000", 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6427e-05 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.024080+0000", 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012104000000000001 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,4", 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.024203+0000", 2024-08-30T23:49:36.076 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012283899999999999 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.025678+0000", 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001474894 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:49:36.029777+0000", 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0040993829999999998 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.077 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:49:36.078 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:49:36.078 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 5 2024-08-30T23:49:36.078 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:49:36.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:36.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:35 smithi173 ceph-mon[20554]: pgmap v1650: 37 pgs: 37 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 995 op/s 2024-08-30T23:49:36.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:35 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:36.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:35 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:36.100 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:36.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:36.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:36.275 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:36.275 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:36.275 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:36.276 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:36.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:36.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:36.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:36.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:36.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 44276 44260 83.8191 78.9375 0.00167411 0.0119211 2024-08-30T23:49:36.579 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:36.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:36.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:36.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:36.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:36.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:36.860 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:36.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:36.987 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:36.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:37.018 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:36 smithi106 ceph-mon[16603]: osdmap e740: 12 total, 12 up, 12 in 2024-08-30T23:49:37.018 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:37.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:37.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:36 smithi136 ceph-mon[20207]: osdmap e740: 12 total, 12 up, 12 in 2024-08-30T23:49:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:37.061 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:37.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-30T23:49:37.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:36 smithi173 ceph-mon[20554]: osdmap e740: 12 total, 12 up, 12 in 2024-08-30T23:49:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:37.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:37.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:37.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:37.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:37.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:37.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:37.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:37.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 45491 45475 83.5871 75.9375 0.00797193 0.011958 2024-08-30T23:49:37.508 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:37.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:37.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:37.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:37.764 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:37 smithi106 ceph-mon[16603]: pgmap v1652: 37 pgs: 37 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:49:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:37 smithi106 ceph-mon[16603]: osdmap e741: 12 total, 12 up, 12 in 2024-08-30T23:49:37.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:37.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:37.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:37.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:49:37.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:38.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:37 smithi136 ceph-mon[20207]: pgmap v1652: 37 pgs: 37 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:49:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:37 smithi136 ceph-mon[20207]: osdmap e741: 12 total, 12 up, 12 in 2024-08-30T23:49:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:37 smithi173 ceph-mon[20554]: pgmap v1652: 37 pgs: 37 active+clean; 2.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-08-30T23:49:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:37 smithi173 ceph-mon[20554]: osdmap e741: 12 total, 12 up, 12 in 2024-08-30T23:49:38.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:38.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:38.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:38.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:38.381 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:38.381 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:38.386 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:38.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:38.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 46538 46522 83.0684 65.4375 0.00223041 0.0120266 2024-08-30T23:49:38.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:38.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:38.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:38.664 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:38.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:38.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:38.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:38.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:38.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:38.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:38.969 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:38.969 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:38.969 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:38 smithi106 ceph-mon[16603]: pgmap v1654: 37 pgs: 37 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-08-30T23:49:38.996 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:38.997 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:39.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:39.023 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:39.023 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:39.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:38 smithi136 ceph-mon[20207]: pgmap v1654: 37 pgs: 37 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-08-30T23:49:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:39.050 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:39.050 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:38 smithi173 ceph-mon[20554]: pgmap v1654: 37 pgs: 37 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.27k op/s 2024-08-30T23:49:39.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:39.075 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:39.101 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:39.101 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:39.129 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:39.129 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:39.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:39.154 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:39.155 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:39.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:39.190 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:39.190 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:39.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:39.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:39.245 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:39.245 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:39.269 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:39.269 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:39.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:39.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:39.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:39.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:39.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 47807 47791 82.9639 79.3125 0.0098046 0.0120476 2024-08-30T23:49:39.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:39.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:39.660 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:39.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:39.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:39.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1324576860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:39.912 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:39.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:39.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:39.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:39.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1324576860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:40.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1324576860' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:40.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:40.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:40.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:40.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:40.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:40.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:49:40.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:40.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:40.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 48902 48886 82.5711 68.4375 0.0163879 0.0121043 2024-08-30T23:49:40.557 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:40.558 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:40.563 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:40.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:40.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:40.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:40.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:40.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:40.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:40 smithi106 ceph-mon[16603]: pgmap v1655: 37 pgs: 37 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-08-30T23:49:40.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:40.969 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:40.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:41.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:40 smithi136 ceph-mon[20207]: pgmap v1655: 37 pgs: 37 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-08-30T23:49:41.071 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:41.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:40 smithi173 ceph-mon[20554]: pgmap v1655: 37 pgs: 37 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.21k op/s 2024-08-30T23:49:41.101 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:41.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:41.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:41.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:41.349 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:41.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:41.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:41.505 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 50071 50055 82.3207 73.0625 0.00622007 0.0121372 2024-08-30T23:49:41.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:41.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:41.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:41.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:41.820 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:41.820 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:41.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:41.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:41.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:41.951 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:41.978 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:41.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:42.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:41 smithi106 ceph-mon[16603]: osdmap e742: 12 total, 12 up, 12 in 2024-08-30T23:49:42.108 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:42.210 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:42.236 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:41 smithi136 ceph-mon[20207]: osdmap e742: 12 total, 12 up, 12 in 2024-08-30T23:49:42.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:42.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:42.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:41 smithi173 ceph-mon[20554]: osdmap e742: 12 total, 12 up, 12 in 2024-08-30T23:49:42.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:42.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:42.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:42.492 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:42.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 51320 51304 82.2114 78.0625 0.0073269 0.0121562 2024-08-30T23:49:42.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:42.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:42.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:49:42.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:42.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:42.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:42.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:42.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:42.990 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:42.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:43.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:43.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:43.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:43.247 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:42 smithi106 ceph-mon[16603]: pgmap v1657: 37 pgs: 37 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-30T23:49:43.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:43.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:42 smithi136 ceph-mon[20207]: pgmap v1657: 37 pgs: 37 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-30T23:49:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:43.290 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:43.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-30T23:49:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:42 smithi173 ceph-mon[20554]: pgmap v1657: 37 pgs: 37 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-08-30T23:49:43.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:43.373 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:43.380 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:43.381 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:43.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:43.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:49:43.506610+0000 min lat: 0.00104498 max lat: 0.307952 avg lat: 0.0122189 2024-08-30T23:49:43.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:49:43.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 52361 52345 81.7825 65.0625 0.00647389 0.0122189 2024-08-30T23:49:43.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:43.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:49:43.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:43.643 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:43.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:43.745 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:43.775 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:43.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:43.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:43.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:44.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:44.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:44.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:44.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:44.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:43 smithi106 ceph-mon[16603]: osdmap e743: 12 total, 12 up, 12 in 2024-08-30T23:49:44.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:44.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:44.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:44.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:43 smithi136 ceph-mon[20207]: osdmap e743: 12 total, 12 up, 12 in 2024-08-30T23:49:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:43 smithi173 ceph-mon[20554]: osdmap e743: 12 total, 12 up, 12 in 2024-08-30T23:49:44.340 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:44.365 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:44.365 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:44.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:44.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:44.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 53659 53643 81.7663 81.125 0.00418913 0.0122242 2024-08-30T23:49:44.522 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:44.549 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:44.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:44.679 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:44.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:44.809 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:44.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:44.943 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:44.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:44.959 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:49:45.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:45.077 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: pgmap v1659: 37 pgs: 37 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3785143710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1255019667,config=1460075751,health=179246142,kv=4115013320}) 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=818973998}) 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2796638562}) 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755093485}) 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2504650195}) 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274068420}) 2024-08-30T23:49:45.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4131371417}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1081195738}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180863273}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500865540}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=37989104}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=340299069}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987307511}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992639412}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2420058126}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2141590794}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2770106614}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235100703}) 2024-08-30T23:49:45.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033624411}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3365532724}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695124939}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2430788159}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588672541}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109765984}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053060704}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743766642}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2369216916}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1455594642}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242493822}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1295302971}) 2024-08-30T23:49:45.110 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967561083}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3270532533}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637435431}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419661944}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792165180}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2923800726}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1037114227}) 2024-08-30T23:49:45.111 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1064217484}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176374112}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085980282}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539428341}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=97150150}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408687053}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095436510}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596294740}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1102234824}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639169143}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566703634}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862265389}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830638287}) 2024-08-30T23:49:45.112 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231762878}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462129296}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791711653}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1232896745}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138839060}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763230786}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=192131738}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736999049}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3143318673}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3879711264}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=850178371}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411131509}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=51,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=15} crc {logm=2181462140,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=467001873,mon_config_key=2499250691}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2635907146}) 2024-08-30T23:49:45.113 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=132147542}) 2024-08-30T23:49:45.114 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=79,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=3} crc {mon_config_key=1626700482,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=3193883666,osdmap=401982819}) 2024-08-30T23:49:45.114 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4141489350}) 2024-08-30T23:49:45.114 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=592958772}) 2024-08-30T23:49:45.114 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:44 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=1189783572}) 2024-08-30T23:49:45.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:45.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:45.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:49:45.267 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: pgmap v1659: 37 pgs: 37 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-30T23:49:45.267 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3785143710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:45.267 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:45.267 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1255019667,config=1460075751,health=179246142,kv=4115013320}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=818973998}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2796638562}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755093485}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2504650195}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274068420}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4131371417}) 2024-08-30T23:49:45.268 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1081195738}) 2024-08-30T23:49:45.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180863273}) 2024-08-30T23:49:45.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500865540}) 2024-08-30T23:49:45.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=37989104}) 2024-08-30T23:49:45.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=340299069}) 2024-08-30T23:49:45.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987307511}) 2024-08-30T23:49:45.269 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992639412}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2420058126}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2141590794}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2770106614}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235100703}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033624411}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3365532724}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695124939}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2430788159}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588672541}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109765984}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053060704}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743766642}) 2024-08-30T23:49:45.270 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2369216916}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1455594642}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242493822}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1295302971}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967561083}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3270532533}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637435431}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419661944}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792165180}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2923800726}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1037114227}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1064217484}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176374112}) 2024-08-30T23:49:45.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085980282}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539428341}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=97150150}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408687053}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095436510}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596294740}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1102234824}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639169143}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566703634}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862265389}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830638287}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231762878}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462129296}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791711653}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1232896745}) 2024-08-30T23:49:45.272 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138839060}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763230786}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=192131738}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736999049}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3143318673}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3879711264}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=850178371}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411131509}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=51,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=15} crc {logm=2181462140,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=467001873,mon_config_key=2499250691}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2635907146}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=132147542}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=79,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=3} crc {mon_config_key=1626700482,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=3193883666,osdmap=401982819}) 2024-08-30T23:49:45.273 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4141489350}) 2024-08-30T23:49:45.274 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=592958772}) 2024-08-30T23:49:45.274 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:44 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=1189783572}) 2024-08-30T23:49:45.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: pgmap v1659: 37 pgs: 37 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-30T23:49:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3785143710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1255019667,config=1460075751,health=179246142,kv=4115013320}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=818973998}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2796638562}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755093485}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2504650195}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=274068420}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4131371417}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1081195738}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2180863273}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500865540}) 2024-08-30T23:49:45.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=37989104}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=340299069}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=987307511}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3992639412}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2420058126}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2141590794}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2770106614}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2235100703}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1033624411}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3365532724}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695124939}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2430788159}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3588672541}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3109765984}) 2024-08-30T23:49:45.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053060704}) 2024-08-30T23:49:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743766642}) 2024-08-30T23:49:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2369216916}) 2024-08-30T23:49:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1455594642}) 2024-08-30T23:49:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1242493822}) 2024-08-30T23:49:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1295302971}) 2024-08-30T23:49:45.327 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1967561083}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3270532533}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637435431}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1419661944}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=792165180}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2923800726}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1037114227}) 2024-08-30T23:49:45.328 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1064217484}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176374112}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085980282}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1539428341}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=97150150}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1408687053}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095436510}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596294740}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1102234824}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639169143}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566703634}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1862265389}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1830638287}) 2024-08-30T23:49:45.329 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231762878}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=462129296}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2791711653}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1232896745}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2138839060}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1763230786}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=192131738}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=736999049}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3143318673}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3879711264}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=850178371}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411131509}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=51,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=15} crc {logm=2181462140,mdsmap=425822562,mgr=1711937972,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=467001873,mon_config_key=2499250691}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2635907146}) 2024-08-30T23:49:45.330 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=132147542}) 2024-08-30T23:49:45.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=79,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=3} crc {mon_config_key=1626700482,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=3193883666,osdmap=401982819}) 2024-08-30T23:49:45.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4141489350}) 2024-08-30T23:49:45.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=592958772}) 2024-08-30T23:49:45.331 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:44 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=1189783572}) 2024-08-30T23:49:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:45.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:45.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:45.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:45.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 55047 55031 81.8848 86.75 0.00809359 0.0122055 2024-08-30T23:49:45.513 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:45.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:45.583 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:45.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:45.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:45.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:45.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:45.940 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:45.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:45.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:46.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:46.097 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:45 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:45 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:49:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:45 smithi106 ceph-mon[16603]: osdmap e744: 12 total, 12 up, 12 in 2024-08-30T23:49:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:45 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:49:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:45 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:49:46.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:46.214 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:45 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:46.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:45 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:49:46.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:45 smithi136 ceph-mon[20207]: osdmap e744: 12 total, 12 up, 12 in 2024-08-30T23:49:46.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:45 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:49:46.215 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:45 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:49:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:46.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:46.244 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:46.244 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:45 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:45 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:49:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:45 smithi173 ceph-mon[20554]: osdmap e744: 12 total, 12 up, 12 in 2024-08-30T23:49:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:45 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:49:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:45 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:49:46.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:46.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:46.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:46.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:46.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 56449 56433 82.0181 87.625 0.0376012 0.0121859 2024-08-30T23:49:46.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:46.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.616 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:46.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.746 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:46.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:46.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:46.857 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:46.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:46.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:46.915 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:46.916 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:46.917 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:46.975 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:47.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:47.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:47.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: pgmap v1661: 37 pgs: 37 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-30T23:49:47.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:49:47.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:49:47.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:47.019 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:47.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:47.020 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:47.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: pgmap v1661: 37 pgs: 37 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-30T23:49:47.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:49:47.088 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:49:47.089 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:47.089 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:47.089 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:47.089 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:47.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:47.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:49:47.131 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:47.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:47.256 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:47.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: pgmap v1661: 37 pgs: 37 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-30T23:49:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:49:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:49:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:47.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:47.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:47.407 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:47.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:47.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:47.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 57502 57486 81.6497 65.8125 0.00748548 0.0122334 2024-08-30T23:49:47.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:47.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:47.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:47.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:47.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:47.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:47.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:47.935 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:47.935 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:47.936 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:47.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-30T23:49:47.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:48.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:48.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:48.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:48.168 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:48.168 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:48.168 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:47 smithi106 ceph-mon[16603]: osdmap e745: 12 total, 12 up, 12 in 2024-08-30T23:49:48.197 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:48.198 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:48.226 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:48.226 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:48.247 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:47 smithi136 ceph-mon[20207]: osdmap e745: 12 total, 12 up, 12 in 2024-08-30T23:49:48.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:48.255 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:48.255 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:48.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:48.287 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:48.287 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:48.296 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:48.317 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:48.317 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:48.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:47 smithi173 ceph-mon[20554]: osdmap e745: 12 total, 12 up, 12 in 2024-08-30T23:49:48.349 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:48.349 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:48.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:48.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:48.386 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:48.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:49:48.411 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:48.411 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.412 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:48.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:48.446 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:48.447 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:48.479 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:48.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 58575 58559 81.3255 67.0625 0.00313492 0.0122906 2024-08-30T23:49:48.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:48.514 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:48.514 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:48.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:48.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:49:48.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.669 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:48.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:48.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:48.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:48.872 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:48.897 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:48.923 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:48.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:49.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:49.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:49.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:49.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:49.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:48 smithi136 ceph-mon[20207]: pgmap v1663: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-30T23:49:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/535806641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:49.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:49.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:48 smithi173 ceph-mon[20554]: pgmap v1663: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-30T23:49:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/535806641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:49.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:48 smithi106 ceph-mon[16603]: pgmap v1663: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-30T23:49:49.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/535806641' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:49.408 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(5.c e389 from shard 0)", 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 540.23168009699998, 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.840740928, 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.409 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.410 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:49:49.410 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175854+0000", 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175855+0000", 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8100000000000002e-07 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:49.175892+0000", 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7345000000000002e-05 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016334+0000", 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.84044191000000001 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016342+0000", 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0420000000000003e-06 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016459+0000", 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116371 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016469+0000", 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0713999999999999e-05 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:40:50.016595+0000", 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012576500000000001 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.414 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.11 542/536 [PushOp(5:884f9bdc:::benchmark_data_smithi173_87639_object32045:head, version: 407'1909, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:884f9bdc:::benchmark_data_smithi173_87639_object32045:head@407'1909, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:40.779709+0000", 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.62782468699999, 2024-08-30T23:49:49.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.046096248, 2024-08-30T23:49:49.416 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.416 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.416 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:49:49.416 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779709+0000", 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779709+0000", 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779711+0000", 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8509999999999999e-06 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779756+0000", 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5099999999999998e-05 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779818+0000", 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2032999999999994e-05 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:40.779841+0000", 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2836e-05 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.824706+0000", 2024-08-30T23:49:49.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.044864397 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.824719+0000", 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3073e-05 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.825806+0000", 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010869580000000001 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.420 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99099 5.1 5:87e96639:::benchmark_data_smithi173_87639_object18859:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038696+0000", 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36883766400001, 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63555977200000002, 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.421 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99099 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038696+0000", 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038696+0000", 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038698+0000", 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1000000000000001e-06 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038702+0000", 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4279999999999998e-06 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038713+0000", 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0597e-05 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038720+0000", 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2729999999999997e-06 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.671475+0000", 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63275495100000001 2024-08-30T23:49:49.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.674256+0000", 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027814229999999999 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99121 5.1 5:865db40b:::benchmark_data_smithi173_87639_object18881:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:49:49.425 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36879842500002, 2024-08-30T23:49:49.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65404301300000001, 2024-08-30T23:49:49.426 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.426 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.426 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99121 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038736+0000", 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.06e-07 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038744+0000", 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0180000000000001e-06 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038754+0000", 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3589999999999997e-06 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038760+0000", 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1e-06 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.689969+0000", 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65120950300000002 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.692779+0000", 2024-08-30T23:49:49.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002809527 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99122 5.1 5:832b5c08:::benchmark_data_smithi173_87639_object18882:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038768+0000", 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36876570300001, 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65762801900000001, 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.432 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99122 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038768+0000", 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038768+0000", 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038769+0000", 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3599999999999999e-07 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038771+0000", 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3549999999999999e-06 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038778+0000", 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.939e-06 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038784+0000", 2024-08-30T23:49:49.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3700000000000003e-06 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.693122+0000", 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65433889999999995 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.696396+0000", 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003274019 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99195 5.1 5:812e864c:::benchmark_data_smithi173_87639_object18955:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038790+0000", 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36874369100002, 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68118318899999997, 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.437 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99195 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038790+0000", 2024-08-30T23:49:49.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038790+0000", 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038791+0000", 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000001e-07 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038793+0000", 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.261e-06 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038799+0000", 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.976e-06 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038804+0000", 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8779999999999997e-06 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.716446+0000", 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67764244299999998 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.719974+0000", 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035272369999999999 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.441 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99201 5.1 5:82ea3608:::benchmark_data_smithi173_87639_object18961:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36872406700002, 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68427159100000001, 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99201 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038810+0000", 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9900000000000001e-07 2024-08-30T23:49:49.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038813+0000", 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1509999999999998e-06 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038820+0000", 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0430000000000002e-06 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038825+0000", 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9259999999999999e-06 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.720250+0000", 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68142502100000002 2024-08-30T23:49:49.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.723082+0000", 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028320509999999999 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.446 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99217 5.1 5:868d9725:::benchmark_data_smithi173_87639_object18977:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36870347399997, 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68806426499999995, 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99217 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038831+0000", 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038836+0000", 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1460000000000004e-06 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038844+0000", 2024-08-30T23:49:49.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8199999999999997e-06 2024-08-30T23:49:49.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038849+0000", 2024-08-30T23:49:49.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.955e-06 2024-08-30T23:49:49.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.723373+0000", 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.68452376699999995 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.726895+0000", 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035221760000000001 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99248 5.1 5:83333dac:::benchmark_data_smithi173_87639_object19008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038855+0000", 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36867874199999, 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71883598999999998, 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99248 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038855+0000", 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038855+0000", 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038856+0000", 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038858+0000", 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3190000000000002e-06 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038866+0000", 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3699999999999997e-06 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038870+0000", 2024-08-30T23:49:49.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9019999999999998e-06 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.754877+0000", 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.716006633 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.757691+0000", 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028143650000000001 2024-08-30T23:49:49.457 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99261 5.1 5:8451e403:::benchmark_data_smithi173_87639_object19021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36865607499999, 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.72356547599999999, 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.458 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99261 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038878+0000", 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038881+0000", 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2539999999999999e-06 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038888+0000", 2024-08-30T23:49:49.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1069999999999999e-06 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038893+0000", 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9819999999999999e-06 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.758644+0000", 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71975094500000003 2024-08-30T23:49:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.762444+0000", 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037997970000000002 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:99284 5.1 5:817284a4:::benchmark_data_smithi173_87639_object19044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e541)", 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:49:49.462 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 428.36863538199998, 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.72655738000000003, 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 99284 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.463 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038899+0000", 2024-08-30T23:49:49.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038901+0000", 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0899999999999999e-06 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038907+0000", 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5910000000000003e-06 2024-08-30T23:49:49.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.038911+0000", 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4340000000000002e-06 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.762677+0000", 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.72376565299999995 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:42:41.765456+0000", 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027792110000000002 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.1 562/540 [PushOp(5:85a0f8be:::benchmark_data_smithi173_87639_object42058:head, version: 409'2536, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:85a0f8be:::benchmark_data_smithi173_87639_object42058:head@409'2536, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:49:49.467 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:16.142061+0000", 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 393.26547315300002, 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.69249434399999998, 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142061+0000", 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142061+0000", 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142062+0000", 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6599999999999996e-07 2024-08-30T23:49:49.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.142089+0000", 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7192999999999999e-05 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.819497+0000", 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67740837499999995 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.819514+0000", 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6929e-05 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.824723+0000", 2024-08-30T23:49:49.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005208959 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.824928+0000", 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020489799999999999 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:16.834555+0000", 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096273239999999996 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.472 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(5.1 576/540 [PushOp(5:801bc841:::benchmark_data_smithi173_87639_object73320:head, version: 436'4483, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:801bc841:::benchmark_data_smithi173_87639_object73320:head@436'4483, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:36.896183+0000", 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 372.51135117400003, 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88981909400000003, 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.473 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.474 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:49:49.474 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896183+0000", 2024-08-30T23:49:49.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896183+0000", 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896184+0000", 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.052e-06 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:36.896215+0000", 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0482000000000001e-05 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783063+0000", 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.88684886399999996 2024-08-30T23:49:49.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783077+0000", 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3526000000000001e-05 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783301+0000", 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022379299999999999 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.783350+0000", 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9030000000000003e-05 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:37.786002+0000", 2024-08-30T23:49:49.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0026523470000000002 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.26173.0:144501 5.1 5:827f28b3:::benchmark_data_smithi173_87639_object64261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e590)", 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:43:57.229197+0000", 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 352.17833701400002, 2024-08-30T23:49:49.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64186927699999996, 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.26173", 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 144501 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229197+0000", 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229197+0000", 2024-08-30T23:49:49.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229198+0000", 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.235e-06 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229204+0000", 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229217+0000", 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3011e-05 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229223+0000", 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.404e-06 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229277+0000", 2024-08-30T23:49:49.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3495000000000002e-05 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229412+0000", 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013536900000000001 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.229538+0000", 2024-08-30T23:49:49.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125399 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.231599+0000", 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020610350000000001 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.231790+0000", 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000191387 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.870974+0000", 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63918373100000003 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.871023+0000", 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9081e-05 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:43:57.871066+0000", 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3579999999999999e-05 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.487 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(1.3 e535 from shard 8)", 2024-08-30T23:49:49.488 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:49:49.488 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 83.590336863000005, 2024-08-30T23:49:49.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74531784099999998, 2024-08-30T23:49:49.488 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.488 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.489 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:49:49.489 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:49:49.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817198+0000", 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0399999999999996e-07 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817203+0000", 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6729999999999997e-06 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562374+0000", 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74517085800000005 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562381+0000", 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.878e-06 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562485+0000", 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103906 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562493+0000", 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562515+0000", 2024-08-30T23:49:49.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2367000000000001e-05 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:199591 6.2 6:4b8b3ba8:::benchmark_data_smithi173_114000_object99793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.152613+0000", 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 63.254921011, 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48962377800000001, 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.493 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 199591 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152613+0000", 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152613+0000", 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152614+0000", 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000001e-07 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152618+0000", 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7440000000000001e-06 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152626+0000", 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3610000000000001e-06 2024-08-30T23:49:49.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152631+0000", 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6720000000000003e-06 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152660+0000", 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8901999999999999e-05 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152740+0000", 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0631999999999996e-05 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:49:49.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.152817+0000", 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6532000000000005e-05 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.156355+0000", 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035378810000000001 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.159092+0000", 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027373839999999998 2024-08-30T23:49:49.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642174+0000", 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48308173300000001 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642209+0000", 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5150000000000001e-05 2024-08-30T23:49:49.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642237+0000", 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7923e-05 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27034.0:199591 6.2 e720/619 ondisk, result = 0)", 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.158540+0000", 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 63.248994275000001, 2024-08-30T23:49:49.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.48370153100000002, 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158540+0000", 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158540+0000", 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158541+0000", 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1900000000000001e-07 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158546+0000", 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1739999999999999e-06 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642081+0000", 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.483535206 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642086+0000", 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2449999999999998e-06 2024-08-30T23:49:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642164+0000", 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7713999999999995e-05 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642170+0000", 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2770000000000002e-06 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.642241+0000", 2024-08-30T23:49:49.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0995999999999996e-05 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:34192 7.8 7:16fb6210:::benchmark_data_smithi173_129296_object34191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:27.833652+0000", 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.573882439999998, 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25566856799999998, 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:49.507 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 34192 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833652+0000", 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833652+0000", 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833652+0000", 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4900000000000005e-07 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833716+0000", 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3493999999999999e-05 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833723+0000", 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2300000000000002e-06 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833727+0000", 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1829999999999996e-06 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833765+0000", 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7138999999999998e-05 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833843+0000", 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8588e-05 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.833929+0000", 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5529000000000001e-05 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.835111+0000", 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011823039999999999 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.852289+0000", 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017177768999999999 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.089256+0000", 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23696736700000001 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.089295+0000", 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9038999999999997e-05 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.089320+0000", 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5177e-05 2024-08-30T23:49:49.514 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:34198 7.8 7:1ca020bc:::benchmark_data_smithi173_129296_object34197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:27.839788+0000", 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.567745805000001, 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26337683299999998, 2024-08-30T23:49:49.515 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.516 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.516 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:49.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:49:49.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:49.516 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 34198 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.839788+0000", 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.839788+0000", 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.839791+0000", 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.17e-06 2024-08-30T23:49:49.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.839836+0000", 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5136000000000003e-05 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.839864+0000", 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8166000000000001e-05 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.839877+0000", 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.358e-05 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.840013+0000", 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000136043 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.840139+0000", 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012548299999999999 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.840279+0000", 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013984100000000001 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.841128+0000", 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084923400000000003 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.089357+0000", 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.248229124 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.103099+0000", 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013741656999999999 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:49:49.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.103139+0000", 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0339000000000001e-05 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.103165+0000", 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6060000000000001e-05 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:34209 7.8 7:1774efa6:::benchmark_data_smithi173_129296_object34208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:27.850458+0000", 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.557075824000002, 2024-08-30T23:49:49.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25941584600000001, 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 34209 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:49.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850458+0000", 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850458+0000", 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850459+0000", 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2699999999999999e-07 2024-08-30T23:49:49.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850520+0000", 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1406000000000001e-05 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850530+0000", 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9739999999999997e-06 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:49.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850536+0000", 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2050000000000001e-06 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850582+0000", 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6054000000000003e-05 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850705+0000", 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123669 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.850838+0000", 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013305399999999999 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.851755+0000", 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00091701 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.103196+0000", 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.251440461 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.109759+0000", 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0065633920000000004 2024-08-30T23:49:49.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.109825+0000", 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6200999999999999e-05 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.109874+0000", 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8792999999999999e-05 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:49.531 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:49.531 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:49:49.531 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 59766 59750 81.1756 74.4375 0.00787938 0.0123112 2024-08-30T23:49:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:49.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:49.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.658 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:49.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:49.686 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:49.687 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:49.687 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:49.687 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:49.687 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:49.688 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:49.689 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:49.689 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:49.689 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:49.689 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:49.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:49.846 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:49.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:49.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:49.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:50.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:50.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:50.198 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:50.226 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:50.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:49 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:49:50.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:49 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-30T23:49:50.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:49 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:49:50.228 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:49 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:49:50.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:49:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:49 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:49:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:49 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-30T23:49:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:49 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:49:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:49 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:49:50.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:50.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:49 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:49:50.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:49 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-30T23:49:50.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:49 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:49:50.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:49 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:49:50.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:50.397 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:50.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-08-30T23:49:50.447 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:50.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:50.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:50.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:50.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 60884 60868 80.9351 69.875 0.00945784 0.0123495 2024-08-30T23:49:50.549 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:50.594 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:50.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:50.719 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:50.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:50.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:50.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:50.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:50.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:50.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:51.076 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:51.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:51.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:51.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: pgmap v1664: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-30T23:49:51.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:49:51.197 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: osdmap e746: 12 total, 12 up, 12 in 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:51.198 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:50 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:49:51.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:51.253 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:50 smithi136 ceph-mon[20207]: pgmap v1664: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: osdmap e746: 12 total, 12 up, 12 in 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:51.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: pgmap v1664: 37 pgs: 1 active+clean+scrubbing, 36 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: osdmap e746: 12 total, 12 up, 12 in 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:49:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:49:51.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:49:51.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:49:51.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:49:51.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:50 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:49:51.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:51.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:51.389 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:49:51.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:51.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 61902 61886 80.5743 63.625 0.0285963 0.0124042 2024-08-30T23:49:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.522 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:51.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:51.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:51.660 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:51.666 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:49:51.666 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:49:51.667 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:49:51.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:51.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:51.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:51.915 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:51.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:51.982 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:49:52.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:52.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:52.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.175 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:52.175 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:52.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:52.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:49:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:49:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1598482127' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:52.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:52.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:51 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:49:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:51 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:49:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1598482127' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:52.327 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:51 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:49:52.327 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:51 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:49:52.327 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1598482127' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:49:52.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:52.429 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:52.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 63214 63198 80.6033 82 0.00294743 0.0124005 2024-08-30T23:49:52.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:52.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:52.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:52.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:52.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:62371 7.c 7:31f0a417:::benchmark_data_smithi173_129296_object62370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:51.925421+0000", 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.029135371, 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029159313999999999, 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:52.698 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62371 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925421+0000", 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925421+0000", 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925423+0000", 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6559999999999999e-06 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925468+0000", 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4580999999999999e-05 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925481+0000", 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.305e-05 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925486+0000", 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4330000000000003e-06 2024-08-30T23:49:52.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925539+0000", 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2735999999999998e-05 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925636+0000", 2024-08-30T23:49:52.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7522000000000006e-05 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,11", 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.925730+0000", 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4032999999999999e-05 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.927453+0000", 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017225369999999999 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.948673+0000", 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021220455999999999 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.704 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:62395 7.c 7:36909ef0:::benchmark_data_smithi173_129296_object62394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:51.946201+0000", 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0083560649999999993, 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0084266319999999999, 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:49:52.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62395 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946201+0000", 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946201+0000", 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:49:52.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946206+0000", 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3090000000000003e-06 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:49:52.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946249+0000", 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3303999999999997e-05 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946270+0000", 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0653e-05 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:49:52.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946281+0000", 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0658e-05 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946342+0000", 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1403999999999997e-05 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946473+0000", 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013089199999999999 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,11", 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.946589+0000", 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011573700000000001 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:51.948254+0000", 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00166556 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-08-30T23:49:52.710 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:49:52.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:52.810 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:52.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:52.940 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:52.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:53.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:53.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.069 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:53.131 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:53.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:53.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: pgmap v1666: 37 pgs: 37 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1598482127' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:53.255 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:52 smithi106 ceph-mon[16603]: osdmap e747: 12 total, 12 up, 12 in 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: pgmap v1666: 37 pgs: 37 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1598482127' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:52 smithi136 ceph-mon[20207]: osdmap e747: 12 total, 12 up, 12 in 2024-08-30T23:49:53.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:53.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: pgmap v1666: 37 pgs: 37 active+clean; 3.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-30T23:49:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:49:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:49:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:49:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:49:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1598482127' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:49:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:52 smithi173 ceph-mon[20554]: osdmap e747: 12 total, 12 up, 12 in 2024-08-30T23:49:53.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:53.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:53.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 15 64331 64316 80.3886 69.875 0.00298946 0.0124332 2024-08-30T23:49:53.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.584 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:53.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:53.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:49:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:49:53.633 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:49:53.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:53.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:53.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:53.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:53.850 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:53.884 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:53.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:53.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:53.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:54.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:54.101 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:49:54.102 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:49:54.102 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:53 smithi106 ceph-mon[16603]: osdmap e748: 12 total, 12 up, 12 in 2024-08-30T23:49:54.102 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:53 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:49:54.102 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:53 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:49:54.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.121 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:54.153 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:49:54.154 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:49:54.183 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:49:54.184 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:49:54.209 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:49:54.209 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:49:54.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:54.236 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:49:54.236 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:49:54.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:54.264 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:49:54.265 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:49:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:49:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:49:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:53 smithi136 ceph-mon[20207]: osdmap e748: 12 total, 12 up, 12 in 2024-08-30T23:49:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:53 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:49:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:53 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:49:54.291 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:49:54.291 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:49:54.318 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:49:54.318 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:49:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:49:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:49:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:53 smithi173 ceph-mon[20554]: osdmap e748: 12 total, 12 up, 12 in 2024-08-30T23:49:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:53 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:49:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:53 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:49:54.345 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:49:54.345 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:49:54.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:54.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:49:54.373 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:49:54.379 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:54.408 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:49:54.408 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:49:54.432 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:54.432 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:54.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:49:54.437 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:49:54.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:49:54.469 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:49:54.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:54.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:54.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 65684 65668 80.4691 84.5 0.0164276 0.0124204 2024-08-30T23:49:54.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:54.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:54.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:54.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:54.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:54.895 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:54.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:54.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:55.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:54 smithi136 ceph-mon[20207]: pgmap v1669: 37 pgs: 37 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 948 op/s 2024-08-30T23:49:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1442492038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:55.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:49:55.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:49:55.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:55.122 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:54 smithi106 ceph-mon[16603]: pgmap v1669: 37 pgs: 37 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 948 op/s 2024-08-30T23:49:55.122 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1442492038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:55.147 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:55.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:55.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:55 smithi173 ceph-mon[20554]: pgmap v1669: 37 pgs: 37 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 948 op/s 2024-08-30T23:49:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1442492038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:49:55.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:55.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:55.411 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:55.432 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:55.499 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:55.506 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 66920 66904 80.4071 77.25 0.00809412 0.0124316 2024-08-30T23:49:55.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:55.623 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:55.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:55.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:55.772 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:55.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:55.898 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:55.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:55.991 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:55.991 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:56.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:56.026 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:56.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:56.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:56.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:49:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:56.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:49:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:56 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:49:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:56 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:49:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:56 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:49:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:56 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:49:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:56 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:49:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:56 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:49:56.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:56.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:56.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 68141 68125 80.3297 76.3125 0.00788849 0.012443 2024-08-30T23:49:56.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:56.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:56.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:56.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:56.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:56.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:56.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:56.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:49:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:56.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:56.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:57.017 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:49:57.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:57.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:57.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:57.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:57.216 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:57.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:57 smithi136 ceph-mon[20207]: pgmap v1670: 37 pgs: 37 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-30T23:49:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2861684000' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:57.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:57 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:57.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:57 smithi173 ceph-mon[20554]: pgmap v1670: 37 pgs: 37 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-30T23:49:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2861684000' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:57 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:57.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:49:57.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:57 smithi106 ceph-mon[16603]: pgmap v1670: 37 pgs: 37 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-30T23:49:57.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2861684000' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:57.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:57 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:49:57.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:57.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:57.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 15 69363 69348 80.2575 76.4375 0.0426783 0.0124542 2024-08-30T23:49:57.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:49:57.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:49:57.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:57.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:57.725 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:57.725 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:57.796 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:57.823 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:57.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:57.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:57.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:58.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:58.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:58.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:49:58.213 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:49:58.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:58 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:49:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:58 smithi136 ceph-mon[20207]: osdmap e749: 12 total, 12 up, 12 in 2024-08-30T23:49:58.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:49:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:58 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:49:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:58 smithi173 ceph-mon[20554]: osdmap e749: 12 total, 12 up, 12 in 2024-08-30T23:49:58.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:49:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:58 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:49:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:49:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:58 smithi106 ceph-mon[16603]: osdmap e749: 12 total, 12 up, 12 in 2024-08-30T23:49:58.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:49:58.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:49:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:58.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:49:58.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 70469 70453 80.0539 69.0625 0.00728371 0.0124851 2024-08-30T23:49:58.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:58.591 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:58.601 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:49:58.634 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:49:58.634 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:49:58.634 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-30T23:49:58.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:49:58.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:58.724 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:49:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:58.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:58.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:58.864 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:58.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:49:58.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:58.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:49:59.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:49:59.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:49:59.135 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:49:59 smithi136 ceph-mon[20207]: pgmap v1672: 37 pgs: 37 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-30T23:49:59.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:59.234 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:49:59.234 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:49:59.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:49:59 smithi173 ceph-mon[20554]: pgmap v1672: 37 pgs: 37 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-30T23:49:59.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:49:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:49:59 smithi106 ceph-mon[16603]: pgmap v1672: 37 pgs: 37 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-30T23:49:59.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:49:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:49:59.469 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:49:59.498 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:49:59.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 71554 71538 79.8352 67.8125 0.00799912 0.0125203 2024-08-30T23:49:59.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:49:59.631 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:49:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:49:59.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:49:59.759 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:49:59.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:49:59.892 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:49:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:49:59.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:49:59.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:49:59.994 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.027 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:00.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:00.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:00.152 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:00.172 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:50:00.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:00.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:00.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:00 smithi136 ceph-mon[20207]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-08-30T23:50:00.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:00 smithi136 ceph-mon[20207]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-30T23:50:00.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:00 smithi173 ceph-mon[20554]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-08-30T23:50:00.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:00 smithi173 ceph-mon[20554]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-30T23:50:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:00 smithi106 ceph-mon[16603]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-08-30T23:50:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:00 smithi106 ceph-mon[16603]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-30T23:50:00.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:00.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:00.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 72767 72751 79.7645 75.8125 0.0125186 0.0125321 2024-08-30T23:50:00.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:00.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:00.636 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:00.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:00.758 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:50:00.759 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:50:00.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.785 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:00.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:00.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:00.909 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:00.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:01.005 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:50:01.005 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:50:01.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:01.038 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:50:01.038 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:50:01.041 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:01.068 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:01.070 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:50:01.070 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:50:01.103 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:50:01.104 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:50:01.132 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:50:01.132 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:50:01.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:01.161 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:50:01.161 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:50:01.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:01.196 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:50:01.196 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:50:01.227 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:50:01.227 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:50:01.263 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:50:01.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:50:01.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:01 smithi106 ceph-mon[16603]: pgmap v1673: 37 pgs: 37 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-30T23:50:01.264 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3979894838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:01.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:01 smithi136 ceph-mon[20207]: pgmap v1673: 37 pgs: 37 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-30T23:50:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3979894838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:01.292 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:50:01.292 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:50:01.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:01.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:01.322 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:50:01.323 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:50:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:01 smithi173 ceph-mon[20554]: pgmap v1673: 37 pgs: 37 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-30T23:50:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3979894838' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:01.357 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:50:01.357 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:50:01.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:01.431 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:01.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:50:01.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 73772 73756 79.472 62.8125 0.0181729 0.0125777 2024-08-30T23:50:01.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:01.556 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:01.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:01.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:01.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:01.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:01.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:01.925 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:01.957 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:01.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:02.059 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:02.137 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:50:02.137 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:50:02.137 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 749, 2024-08-30T23:50:02.137 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:50:02.137 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:49:57.005720+0000", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:45:50.581901+0000", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:48:30.074918+0000", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5801984, 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "noscrub", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:50:02.138 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 29, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 7, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "luminous", 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.139 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:50:02.140 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 21, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 21, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 21, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 21, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 21, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:50:02.141 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "717", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "507", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:50:02.142 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:50:02.143 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:50:02.144 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.2799999713897705, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.9099999666213989, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.2000000476837158, 2024-08-30T23:50:02.145 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.940000057220459, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.2899999618530273, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.64216864109039307, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.76999998092651367, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.63999998569488525, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "error_message": "pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 7, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_5", 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:48:57.126565+0000", 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:50:02.146 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:50:02.147 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "728", 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:50:02.148 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:50:02.149 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:50:02.150 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:50:02.151 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 2.5, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 2.5, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.75, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.75, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.64216864109039307, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.76999998092651367, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.63999998569488525 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:50:02.152 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.2807464599609375, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 539, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.153 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.154 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.155 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.156 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.157 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:50:02.158 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:50:02.159 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.160 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.161 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.162 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-30T23:50:02.163 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.164 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 711, 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.165 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.166 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.167 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.168 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.169 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.170 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.171 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.172 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.173 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:50:02.174 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:50:02.175 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.176 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.177 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.178 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.179 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:50:02.180 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.181 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.426727294921875, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 566, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 593, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 649, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 706, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:50:02.182 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:50:02.183 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:50:02.184 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.185 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2739515404", 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2739515404", 2024-08-30T23:50:02.186 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2739515404", 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2739515404", 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7101593017578125, 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:50:02.187 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:50:02.191 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.191 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.192 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 0.7503204345703125, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.1631317138671875, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 653, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.194 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.196 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.197 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:50:02.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:50:02.199 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:50:02.204 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.205 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.2454376220703125, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.209 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.210 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.211 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:50:02.212 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.215 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:50:02.217 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 725, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:50:02.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.221 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:50:02.223 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:30.201561+0000", 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.224 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:21.864097+0000", 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:21.016490+0000", 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:34.788323+0000", 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.226 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:36.279200+0000", 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.227 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:42.914414+0000", 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:40.872069+0000", 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:46.213598+0000", 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.229 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:46.319751+0000", 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.230 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:14.405336+0000", 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:13.926436+0000", 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.231 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:49:52.330667+0000", 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [ 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.0", 2024-08-30T23:50:02.232 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: 7, 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: 11 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1", 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.233 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:50:02.234 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:50:02.235 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:50:02.236 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:50:02.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:02.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:02.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:02.382 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:02.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:02.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:50:02.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:02.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 74833 74817 79.2489 66.3125 0.0211457 0.0126132 2024-08-30T23:50:02.516 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:02.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.517 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:02.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:02.640 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:02.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:02.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:02.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:02.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:02.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:75372 7.f 7:f4a9c99f:::benchmark_data_smithi173_129296_object75371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:02.951394+0000", 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0054147850000000001, 2024-08-30T23:50:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005447403, 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75372 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951394+0000", 2024-08-30T23:50:02.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951394+0000", 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951395+0000", 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6840000000000001e-06 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:02.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951419+0000", 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3224999999999999e-05 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951442+0000", 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3053000000000001e-05 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951451+0000", 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9910000000000008e-06 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951558+0000", 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107396 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951769+0000", 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00021050800000000001 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,11", 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951904+0000", 2024-08-30T23:50:02.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013567399999999999 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.955439+0000", 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035347120000000002 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:75373 7.7 7:e1f4848d:::benchmark_data_smithi173_129296_object75372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:02.951574+0000", 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0052344269999999998, 2024-08-30T23:50:02.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053494750000000002, 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 75373 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951574+0000", 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:02.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951574+0000", 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951575+0000", 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1399999999999996e-07 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:02.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951597+0000", 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1963999999999998e-05 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951608+0000", 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0692e-05 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951628+0000", 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0630999999999999e-05 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951692+0000", 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3238999999999995e-05 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951824+0000", 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013214499999999999 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.951904+0000", 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0620999999999999e-05 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956901+0000", 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004996889 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27487.0:75373 7.7 e749/725 ondisk, result = 0)", 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:02.956749+0000", 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.9837999999999999e-05, 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000245135, 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:02.968 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956749+0000", 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956749+0000", 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956749+0000", 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4900000000000001e-07 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956752+0000", 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.61e-06 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956798+0000", 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6069999999999998e-05 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956817+0000", 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9086e-05 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956928+0000", 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111179 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:02.956934+0000", 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5899999999999998e-06 2024-08-30T23:50:02.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.972 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:02.972 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.972 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:02.972 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:50:02.972 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-08-30T23:50:02.972 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:03.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:03.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:03 smithi173 ceph-mon[20554]: pgmap v1674: 37 pgs: 37 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-30T23:50:03.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3835601243' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:50:03.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:03.163 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:03.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:03.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:03.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:03 smithi136 ceph-mon[20207]: pgmap v1674: 37 pgs: 37 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-30T23:50:03.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3835601243' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:50:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:03 smithi106 ceph-mon[16603]: pgmap v1674: 37 pgs: 37 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-30T23:50:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3835601243' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:50:03.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:03.402 INFO:tasks.thrashosds.thrasher:j is {'epoch': 749, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:49:57.005720+0000', 'last_up_change': '2024-08-30T23:45:50.581901+0000', 'last_in_change': '2024-08-30T23:48:30.074918+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '717', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '507', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2799999713897705, 'score_stable': 1.909999966621399, 'optimal_score': 1.2000000476837158, 'raw_score_acting': 3.940000057220459, 'raw_score_stable': 2.2899999618530273, 'primary_affinity_weighted': 0.6421686410903931, 'average_primary_affinity': 0.7699999809265137, 'average_primary_affinity_weighted': 0.6399999856948853, 'error_message': 'pool 1 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-08-30T23:48:57.126565+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '728', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.6421686410903931, 'average_primary_affinity': 0.7699999809265137, 'average_primary_affinity_weighted': 0.6399999856948853}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 0.2807464599609375, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 539, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 725, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 711, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 0.426727294921875, 'primary_affinity': 1, 'last_clean_begin': 566, 'last_clean_end': 593, 'up_from': 649, 'up_thru': 706, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2739515404}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2739515404}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2739515404}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2739515404}]}, 'public_addr': '172.21.15.136:6809/2739515404', 'cluster_addr': '172.21.15.136:6811/2739515404', 'heartbeat_back_addr': '172.21.15.136:6815/2739515404', 'heartbeat_front_addr': '172.21.15.136:6813/2739515404', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 0.7101593017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 1, 'in': 1, 'weight': 0.7503204345703125, 'primary_affinity': 0.1631317138671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 653, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2454376220703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 725, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:30.201561+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:21.864097+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:21.016490+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:34.788323+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:36.279200+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:42.914414+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:40.872069+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:46.213598+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:46.319751+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:14.405336+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:13.926436+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:49:52.330667+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 7, 11]}, {'pgid': '1.1', 'osds': [2, 0, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:50:03.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:50:03.413 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:03.481 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:03.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:50:03.508269+0000 min lat: 0.00104498 max lat: 0.307952 avg lat: 0.0126505 2024-08-30T23:50:03.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:50:03.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 75867 75851 79.0051 64.625 0.0512226 0.0126505 2024-08-30T23:50:03.515 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:03.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:03.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:03.676 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:03.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:03.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:03.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:03.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:03.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:03.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:50:04.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.064 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:04.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:04.113 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:04.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:04.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:04.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:04.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:04.392 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:50:04.404 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:04.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:04.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:04.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 76892 76876 78.76 64.0625 0.0428428 0.0126919 2024-08-30T23:50:04.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:04.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:04.689 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:04.711 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:04.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:04.835 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:04.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:04.969 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:04.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:04.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:05.033 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:05.033 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:50:05.033 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:50:05.033 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:50:05.033 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:05.043 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:50:05.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:05.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:05.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:05 smithi106 ceph-mon[16603]: pgmap v1675: 37 pgs: 37 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-30T23:50:05.127 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:05 smithi106 ceph-mon[16603]: from='client.18237 172.21.15.106:0/3465242481' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:05.190 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:50:05.191 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:50:05.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:05.231 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:05.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:05 smithi136 ceph-mon[20207]: pgmap v1675: 37 pgs: 37 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-30T23:50:05.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:05 smithi136 ceph-mon[20207]: from='client.18237 172.21.15.106:0/3465242481' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:05 smithi173 ceph-mon[20554]: pgmap v1675: 37 pgs: 37 active+clean; 4.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-30T23:50:05.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:05 smithi173 ceph-mon[20554]: from='client.18237 172.21.15.106:0/3465242481' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:05.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:05.361 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:05.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:05.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:05.507 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 78025 78009 78.6317 70.8125 0.00367527 0.0127125 2024-08-30T23:50:05.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:05.636 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:05.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:05.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:50:05.774 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:05.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:05.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:05.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:05.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:06.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:06.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:06 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:06.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:06 smithi173 ceph-mon[20554]: from='client.18237 172.21.15.106:0/3465242481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:06.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:06 smithi173 ceph-mon[20554]: osdmap e750: 12 total, 12 up, 12 in 2024-08-30T23:50:06.033 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:06 smithi173 ceph-mon[20554]: from='client.27625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:50:06.066 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:06.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:06.154 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:06.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:06.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:06 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:06.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:06 smithi136 ceph-mon[20207]: from='client.18237 172.21.15.106:0/3465242481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:06 smithi136 ceph-mon[20207]: osdmap e750: 12 total, 12 up, 12 in 2024-08-30T23:50:06.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:06 smithi136 ceph-mon[20207]: from='client.27625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:50:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:06 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:06 smithi106 ceph-mon[16603]: from='client.18237 172.21.15.106:0/3465242481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:06 smithi106 ceph-mon[16603]: osdmap e750: 12 total, 12 up, 12 in 2024-08-30T23:50:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:06 smithi106 ceph-mon[16603]: from='client.27625 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:50:06.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:06.408 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:06.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 78978 78962 78.3284 59.5625 0.0213028 0.0127612 2024-08-30T23:50:06.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:06.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:06.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 468/432 [PushOp(5:cf5427eb:::benchmark_data_smithi173_87639_object52812:head, version: 416'3330, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:cf5427eb:::benchmark_data_smithi173_87639_object52812:head@416'3330, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:24.454217+0000", 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 582.19900461700001, 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.80342548899999999, 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.656 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454217+0000", 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454217+0000", 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454218+0000", 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0500000000000003e-07 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:24.454326+0000", 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108422 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.255685+0000", 2024-08-30T23:50:06.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.80135822199999995 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.255715+0000", 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0593000000000003e-05 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.256047+0000", 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000332121 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.256081+0000", 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3705999999999999e-05 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:25.257643+0000", 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00156182 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.660 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.3 470/432 [PushOp(5:c32e7fa5:::benchmark_data_smithi173_87639_object55010:head, version: 417'3465, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c32e7fa5:::benchmark_data_smithi173_87639_object55010:head@417'3465, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:40:26.477253+0000", 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 580.17596897299995, 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93278049699999999, 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477253+0000", 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477253+0000", 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477254+0000", 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1960000000000001e-06 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:26.477375+0000", 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012103399999999999 2024-08-30T23:50:06.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408122+0000", 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93074724099999995 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408136+0000", 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3864e-05 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408249+0000", 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113028 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.408261+0000", 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1348e-05 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:40:27.410034+0000", 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0017727859999999999 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(5.3 e488 from shard 10)", 2024-08-30T23:50:06.664 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:09.918655+0000", 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 536.73456673700002, 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78852235699999995, 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:09.918655+0000", 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:09.918655+0000", 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:09.918656+0000", 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6599999999999995e-07 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:09.918672+0000", 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5998000000000002e-05 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:10.706952+0000", 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78827963999999995 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:10.706960+0000", 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7609999999999996e-06 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:10.707049+0000", 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8772999999999999e-05 2024-08-30T23:50:06.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:10.707059+0000", 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0122000000000001e-05 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:10.707178+0000", 2024-08-30T23:50:06.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118297 2024-08-30T23:50:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.669 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.12 511/491 [PushOp(5:4b951daa:::benchmark_data_smithi173_87639_object26201:head, version: 406'1644, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:4b951daa:::benchmark_data_smithi173_87639_object26201:head@406'1644, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:43.804895+0000", 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 502.84832672800002, 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74340975099999995, 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:43.804895+0000", 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:43.804895+0000", 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:43.804896+0000", 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-07 2024-08-30T23:50:06.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:43.804948+0000", 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2422000000000001e-05 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:44.547019+0000", 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.74207084099999998 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:44.547041+0000", 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1812000000000001e-05 2024-08-30T23:50:06.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:44.547153+0000", 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111679 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:44.547168+0000", 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5542000000000001e-05 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:44.548305+0000", 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001136857 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(5.15 e598,590 5:ab36a49d:::benchmark_data_smithi173_87639_object5842:head,446'5542)", 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:08.625253+0000", 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 358.02796870499998, 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.832386399, 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.674 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:08.625253+0000", 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:08.625253+0000", 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:08.625254+0000", 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0200000000000002e-07 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:08.625260+0000", 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1330000000000004e-06 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:09.457147+0000", 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83188728599999995 2024-08-30T23:50:06.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:09.457155+0000", 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7340000000000008e-06 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:09.457239+0000", 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4340999999999998e-05 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:09.457640+0000", 2024-08-30T23:50:06.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040030300000000002 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(5.15 e600,590 5:aa58dc10:::benchmark_data_smithi173_87639_object33094:head,557'6470)", 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:11.040636+0000", 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 355.61258595200002, 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76406434599999995, 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.678 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.040636+0000", 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.040636+0000", 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.040637+0000", 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2300000000000001e-07 2024-08-30T23:50:06.679 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.040641+0000", 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5410000000000002e-06 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.804187+0000", 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.76354638399999997 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.804202+0000", 2024-08-30T23:50:06.680 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4922999999999999e-05 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.804336+0000", 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000133895 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:11.804700+0000", 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036408000000000002 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.681 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93381 6.1 6:8c800d3f:::benchmark_data_smithi173_114000_object93380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906375+0000", 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.746847272, 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92827897100000001, 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.682 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93381 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906375+0000", 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906375+0000", 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.683 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906377+0000", 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.869e-06 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906429+0000", 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2918999999999999e-05 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906445+0000", 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5288e-05 2024-08-30T23:50:06.684 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906452+0000", 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0450000000000003e-06 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.909954+0000", 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035024349999999999 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.909975+0000", 2024-08-30T23:50:06.685 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0313e-05 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.834545+0000", 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92457082400000001 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.834654+0000", 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108278 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.686 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93434 6.1 6:8548ae37:::benchmark_data_smithi173_114000_object93433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906470+0000", 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.746751518, 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93097174400000005, 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93434 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.687 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906470+0000", 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906470+0000", 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906471+0000", 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1199999999999998e-07 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.688 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906514+0000", 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2752e-05 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906521+0000", 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0720000000000002e-06 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906524+0000", 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6650000000000001e-06 2024-08-30T23:50:06.689 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910095+0000", 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035705070000000001 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910104+0000", 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8009999999999993e-06 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837367+0000", 2024-08-30T23:50:06.690 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.927262896 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837442+0000", 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5638999999999999e-05 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93443 6.1 6:879a2a2b:::benchmark_data_smithi173_114000_object93442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.74669029500001, 2024-08-30T23:50:06.691 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96616541, 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93443 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.692 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.39e-07 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906568+0000", 2024-08-30T23:50:06.693 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6347999999999998e-05 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906575+0000", 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2110000000000003e-06 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906578+0000", 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3280000000000002e-06 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.694 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910143+0000", 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035652679999999999 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910151+0000", 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4250000000000001e-06 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837487+0000", 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92733659000000002 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.695 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837635+0000", 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014827699999999999 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837751+0000", 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011542199999999999 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.696 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842684+0000", 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049333650000000003 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842702+0000", 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7965000000000002e-05 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.869579+0000", 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026876494000000001 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.697 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.869625+0000", 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.651e-05 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.872697+0000", 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030718680000000002 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93451 6.1 6:89dbcc07:::benchmark_data_smithi173_114000_object93450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.698 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.74663810000001, 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96804708799999994, 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93451 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:50:06.699 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6600000000000003e-07 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.700 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906630+0000", 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5815000000000001e-05 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906636+0000", 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.694e-06 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906639+0000", 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0340000000000001e-06 2024-08-30T23:50:06.701 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910196+0000", 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035574980000000001 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910207+0000", 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0915e-05 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.702 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.840261+0000", 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93005400400000005 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.840396+0000", 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013501799999999999 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.840498+0000", 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010153300000000001 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.703 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847456+0000", 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069587240000000003 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847511+0000", 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4848000000000001e-05 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.872751+0000", 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025240058999999999 2024-08-30T23:50:06.704 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.872792+0000", 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0639000000000002e-05 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.874631+0000", 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001839041 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.705 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93453 6.1 6:8e4148d1:::benchmark_data_smithi173_114000_object93452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906644+0000", 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.74657764700001, 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97105458200000005, 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93453 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.706 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906644+0000", 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906644+0000", 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906645+0000", 2024-08-30T23:50:06.707 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2500000000000001e-07 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906675+0000", 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0706e-05 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906681+0000", 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4389999999999999e-06 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.708 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906684+0000", 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1190000000000001e-06 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910250+0000", 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035664389999999998 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910258+0000", 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6920000000000002e-06 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.709 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842735+0000", 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93247689499999997 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842833+0000", 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8061999999999997e-05 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.710 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:06.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842931+0000", 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7505999999999998e-05 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847499+0000", 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045683310000000001 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.712 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847519+0000", 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0072e-05 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.874984+0000", 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027465280000000002 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.875022+0000", 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7573000000000001e-05 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.713 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.877699+0000", 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026771429999999999 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93460 6.1 6:8557b2d5:::benchmark_data_smithi173_114000_object93459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.74653293599999, 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98645355199999996, 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.714 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93460 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.715 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0699999999999998e-07 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906719+0000", 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9921999999999999e-05 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.716 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906724+0000", 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9099999999999996e-06 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906727+0000", 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9639999999999999e-06 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910297+0000", 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035694099999999999 2024-08-30T23:50:06.717 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910307+0000", 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9699999999999994e-06 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.845188+0000", 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93488130300000005 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.718 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.845283+0000", 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5049000000000004e-05 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.845393+0000", 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011003299999999999 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853878+0000", 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084850429999999994 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.719 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853972+0000", 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4526999999999998e-05 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.889838+0000", 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035865478999999999 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.889888+0000", 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0467000000000003e-05 2024-08-30T23:50:06.720 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.893143+0000", 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003254168 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93467 6.1 6:81b403e8:::benchmark_data_smithi173_114000_object93466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.74649008599999, 2024-08-30T23:50:06.721 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98816438699999998, 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93467 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.722 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.84e-07 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.723 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906763+0000", 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0889000000000002e-05 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906768+0000", 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.989e-06 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906771+0000", 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.976e-06 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.724 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910341+0000", 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035697580000000001 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910345+0000", 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8039999999999999e-06 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847542+0000", 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93719730700000003 2024-08-30T23:50:06.725 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847633+0000", 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0718999999999997e-05 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847738+0000", 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010495100000000001 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.726 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853946+0000", 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0062085839999999996 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853982+0000", 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5568999999999998e-05 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.893189+0000", 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039206854999999999 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.727 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.893233+0000", 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4274999999999999e-05 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.894896+0000", 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016634270000000001 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.728 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93475 6.1 6:85aba04e:::benchmark_data_smithi173_114000_object93474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.74644610799999, 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99493798300000003, 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93475 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.729 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:50:06.730 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.27e-07 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906816+0000", 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9847e-05 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906828+0000", 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1671999999999999e-05 2024-08-30T23:50:06.731 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906833+0000", 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2290000000000002e-06 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910400+0000", 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035675080000000001 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.732 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910404+0000", 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7890000000000001e-06 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.850631+0000", 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94022669699999994 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.850804+0000", 2024-08-30T23:50:06.733 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017325500000000001 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.851007+0000", 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000203265 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856446+0000", 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054381899999999999 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.734 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856460+0000", 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4352000000000001e-05 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.900302+0000", 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.043842224999999999 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.900339+0000", 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6650999999999999e-05 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.735 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.901714+0000", 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001374976 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93483 6.1 6:89482277:::benchmark_data_smithi173_114000_object93482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.042217+0000", 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.61100471099999, 2024-08-30T23:50:06.736 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.861912593, 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93483 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042217+0000", 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.737 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042217+0000", 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042219+0000", 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.855e-06 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042243+0000", 2024-08-30T23:50:06.738 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3516e-05 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042260+0000", 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7037999999999998e-05 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042269+0000", 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6609999999999995e-06 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042338+0000", 2024-08-30T23:50:06.739 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8492999999999996e-05 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042346+0000", 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3440000000000001e-06 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.854003+0000", 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.81165673500000002 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.740 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.854114+0000", 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011126200000000001 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.854227+0000", 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113167 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861145+0000", 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006917978 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.741 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861224+0000", 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9060000000000005e-05 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.901772+0000", 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.040547444000000002 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.901823+0000", 2024-08-30T23:50:06.742 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1168000000000002e-05 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.904130+0000", 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023068720000000002 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93490 6.1 6:8ef0ade6:::benchmark_data_smithi173_114000_object93489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.058909+0000", 2024-08-30T23:50:06.743 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.59431268099999, 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85139444799999997, 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93490 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058909+0000", 2024-08-30T23:50:06.744 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058909+0000", 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058911+0000", 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8649999999999999e-06 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.745 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058933+0000", 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2123000000000001e-05 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058947+0000", 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4201e-05 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058957+0000", 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.746 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.059064+0000", 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107483 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.059073+0000", 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5420000000000005e-06 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856484+0000", 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79741141100000001 2024-08-30T23:50:06.747 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856594+0000", 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110189 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856728+0000", 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013400400000000001 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861206+0000", 2024-08-30T23:50:06.748 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044777849999999998 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861233+0000", 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7263999999999998e-05 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.907851+0000", 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.046617812000000002 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.749 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.907892+0000", 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0383999999999998e-05 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.910304+0000", 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024122369999999998 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93498 6.1 6:87f9e3d0:::benchmark_data_smithi173_114000_object93497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:50:06.750 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.078684+0000", 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.57453784, 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83696644099999995, 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93498 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.751 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078684+0000", 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078684+0000", 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078685+0000", 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1000000000000001e-06 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.752 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078705+0000", 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0077000000000001e-05 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078717+0000", 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1677e-05 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078723+0000", 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5049999999999998e-06 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.753 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078768+0000", 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5979000000000001e-05 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078776+0000", 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5870000000000004e-06 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.858569+0000", 2024-08-30T23:50:06.754 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77979302399999995 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.858718+0000", 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014851099999999999 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.858898+0000", 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000180581 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.755 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866139+0000", 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0072403650000000003 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866208+0000", 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9813000000000004e-05 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.913342+0000", 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.047133988000000002 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.756 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.913447+0000", 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104448 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.915651+0000", 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022037860000000001 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.757 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93539 6.1 6:8ab5d1ae:::benchmark_data_smithi173_114000_object93538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.262217+0000", 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 242.39100544199999, 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.655331102, 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.758 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93539 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262217+0000", 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262217+0000", 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.759 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262218+0000", 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.511e-06 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262241+0000", 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2792000000000001e-05 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262259+0000", 2024-08-30T23:50:06.760 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8172000000000002e-05 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262267+0000", 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6660000000000008e-06 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262325+0000", 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8696000000000002e-05 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.761 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262334+0000", 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2649999999999996e-06 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861251+0000", 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59891767799999995 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861352+0000", 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100416 2024-08-30T23:50:06.762 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861463+0000", 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011157799999999999 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866286+0000", 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048225619999999999 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866303+0000", 2024-08-30T23:50:06.763 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6894000000000001e-05 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.915703+0000", 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.049399799000000001 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.915740+0000", 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7845000000000001e-05 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.764 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.917548+0000", 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018072279999999999 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:199570 6.1 6:86daa506:::benchmark_data_smithi173_114000_object99772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 80.540931774000001, 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65846682899999998, 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:06.765 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199570 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:50:06.766 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112292+0000", 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.198e-06 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112301+0000", 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0219999999999993e-06 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.767 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112321+0000", 2024-08-30T23:50:06.768 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9842000000000001e-05 2024-08-30T23:50:06.768 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.768 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.768 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.768 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112331+0000", 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8949999999999993e-06 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112400+0000", 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8448999999999995e-05 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112494+0000", 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4380000000000001e-05 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.769 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112570+0000", 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5704999999999994e-05 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.117141+0000", 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045717809999999996 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.117454+0000", 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031256200000000001 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.770 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770689+0000", 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.653235393 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770728+0000", 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8445999999999997e-05 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770757+0000", 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9156e-05 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.771 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27034.0:199570 6.1 e720/715 ondisk, result = 0)", 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 80.534811926000003, 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65235226300000004, 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:06.772 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118411+0000", 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2900000000000004e-07 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.773 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118414+0000", 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8670000000000004e-06 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770637+0000", 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65222257100000003 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770642+0000", 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3920000000000002e-06 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.774 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770680+0000", 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7886999999999998e-05 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770686+0000", 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5670000000000002e-06 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770762+0000", 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6450000000000002e-05 2024-08-30T23:50:06.775 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.776 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.776 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.776 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:06.776 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:06.776 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:06.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:06.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:06.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:06.963 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:06.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:07.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:07.090 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:07.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:07.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:07.261 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1676,"stamp":"2024-08-30T23:50:05.044473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4886954528,"num_objects":74562,"num_object_clones":0,"num_object_copies":223686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74560,"num_write_kb":4771840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76492,"ondisk_log_size":76492,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":20475280,"kb_used_data":14656216,"kb_used_omap":109,"kb_used_meta":5818578,"kb_avail":1104466544,"statfs":{"total":1151940427776,"available":1130973741056,"internally_reserved":0,"allocated":15007965184,"data_stored":15002046640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":5958224536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":742785024,"num_objects":11334,"num_object_clones":0,"num_object_copies":34002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11334,"num_write_kb":725376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.344035"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":549,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019227+0000","last_change":"2024-08-30T23:48:45.566174+0000","last_active":"2024-08-30T23:49:57.019227+0000","last_peered":"2024-08-30T23:49:57.019227+0000","last_clean":"2024-08-30T23:49:57.019227+0000","last_became_active":"2024-08-30T23:48:30.880410+0000","last_became_peered":"2024-08-30T23:48:30.880410+0000","last_unstale":"2024-08-30T23:49:57.019227+0000","last_undegraded":"2024-08-30T23:49:57.019227+0000","last_fullsized":"2024-08-30T23:49:57.019227+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:45.566103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:45.566103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:07.638522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":499,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.015303+0000","last_change":"2024-08-30T23:49:46.108086+0000","last_active":"2024-08-30T23:49:57.015303+0000","last_peered":"2024-08-30T23:49:57.015303+0000","last_clean":"2024-08-30T23:49:57.015303+0000","last_became_active":"2024-08-30T23:42:43.951741+0000","last_became_peered":"2024-08-30T23:42:43.951741+0000","last_unstale":"2024-08-30T23:49:57.015303+0000","last_undegraded":"2024-08-30T23:49:57.015303+0000","last_fullsized":"2024-08-30T23:49:57.015303+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:46.108017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:46.108017+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:51:08.353873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":540,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.016561+0000","last_change":"2024-08-30T23:48:27.943034+0000","last_active":"2024-08-30T23:49:57.016561+0000","last_peered":"2024-08-30T23:49:57.016561+0000","last_clean":"2024-08-30T23:49:57.016561+0000","last_became_active":"2024-08-30T23:42:20.761577+0000","last_became_peered":"2024-08-30T23:42:20.761577+0000","last_unstale":"2024-08-30T23:49:57.016561+0000","last_undegraded":"2024-08-30T23:49:57.016561+0000","last_fullsized":"2024-08-30T23:49:57.016561+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:27.942972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:27.942972+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:57.113097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":97,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.020809+0000","last_change":"2024-08-30T23:49:48.578749+0000","last_active":"2024-08-30T23:49:57.020809+0000","last_peered":"2024-08-30T23:49:57.020809+0000","last_clean":"2024-08-30T23:49:57.020809+0000","last_became_active":"2024-08-30T23:48:30.881538+0000","last_became_peered":"2024-08-30T23:48:30.881538+0000","last_unstale":"2024-08-30T23:49:57.020809+0000","last_undegraded":"2024-08-30T23:49:57.020809+0000","last_fullsized":"2024-08-30T23:49:57.020809+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:48.578684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:48.578684+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:54.006830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.6","version":"749'4696","reported_seq":4755,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:04.063147+0000","last_change":"2024-08-30T23:48:59.165021+0000","last_active":"2024-08-30T23:50:04.063147+0000","last_peered":"2024-08-30T23:50:04.063147+0000","last_clean":"2024-08-30T23:50:04.063147+0000","last_became_active":"2024-08-30T23:48:59.164704+0000","last_became_peered":"2024-08-30T23:48:59.164704+0000","last_unstale":"2024-08-30T23:50:04.063147+0000","last_undegraded":"2024-08-30T23:50:04.063147+0000","last_fullsized":"2024-08-30T23:50:04.063147+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:18.468134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307757056,"num_objects":4696,"num_object_clones":0,"num_object_copies":14088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4696,"num_write_kb":300544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,10],"acting":[3,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":319,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.015784+0000","last_change":"2024-08-30T23:49:49.861149+0000","last_active":"2024-08-30T23:49:57.015784+0000","last_peered":"2024-08-30T23:49:57.015784+0000","last_clean":"2024-08-30T23:49:57.015784+0000","last_became_active":"2024-08-30T23:45:08.975961+0000","last_became_peered":"2024-08-30T23:45:08.975961+0000","last_unstale":"2024-08-30T23:49:57.015784+0000","last_undegraded":"2024-08-30T23:49:57.015784+0000","last_fullsized":"2024-08-30T23:49:57.015784+0000","mapping_epoch":631,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:49.861102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:49.861102+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:51:17.080144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"749'4778","reported_seq":4836,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:04.078154+0000","last_change":"2024-08-30T23:48:58.424377+0000","last_active":"2024-08-30T23:50:04.078154+0000","last_peered":"2024-08-30T23:50:04.078154+0000","last_clean":"2024-08-30T23:50:04.078154+0000","last_became_active":"2024-08-30T23:48:58.424101+0000","last_became_peered":"2024-08-30T23:48:58.424101+0000","last_unstale":"2024-08-30T23:50:04.078154+0000","last_undegraded":"2024-08-30T23:50:04.078154+0000","last_fullsized":"2024-08-30T23:50:04.078154+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313131008,"num_objects":4778,"num_object_clones":0,"num_object_copies":14334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":305792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":844,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019011+0000","last_change":"2024-08-30T23:49:51.861915+0000","last_active":"2024-08-30T23:49:57.019011+0000","last_peered":"2024-08-30T23:49:57.019011+0000","last_clean":"2024-08-30T23:49:57.019011+0000","last_became_active":"2024-08-30T23:41:38.805434+0000","last_became_peered":"2024-08-30T23:41:38.805434+0000","last_unstale":"2024-08-30T23:49:57.019011+0000","last_undegraded":"2024-08-30T23:49:57.019011+0000","last_fullsized":"2024-08-30T23:49:57.019011+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:51.861845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:51.861845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:51:08.618600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"749'4873","reported_seq":4932,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:04.073164+0000","last_change":"2024-08-30T23:48:58.424004+0000","last_active":"2024-08-30T23:50:04.073164+0000","last_peered":"2024-08-30T23:50:04.073164+0000","last_clean":"2024-08-30T23:50:04.073164+0000","last_became_active":"2024-08-30T23:48:58.423706+0000","last_became_peered":"2024-08-30T23:48:58.423706+0000","last_unstale":"2024-08-30T23:50:04.073164+0000","last_undegraded":"2024-08-30T23:50:04.073164+0000","last_fullsized":"2024-08-30T23:50:04.073164+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":311872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":103,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.020897+0000","last_change":"2024-08-30T23:48:41.523538+0000","last_active":"2024-08-30T23:49:57.020897+0000","last_peered":"2024-08-30T23:49:57.020897+0000","last_clean":"2024-08-30T23:49:57.020897+0000","last_became_active":"2024-08-30T23:48:30.881035+0000","last_became_peered":"2024-08-30T23:48:30.881035+0000","last_unstale":"2024-08-30T23:49:57.020897+0000","last_undegraded":"2024-08-30T23:49:57.020897+0000","last_fullsized":"2024-08-30T23:49:57.020897+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:41.523491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:41.523491+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:06.051875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"749'4504","reported_seq":4563,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:00.247439+0000","last_change":"2024-08-30T23:48:58.220192+0000","last_active":"2024-08-30T23:50:00.247439+0000","last_peered":"2024-08-30T23:50:00.247439+0000","last_clean":"2024-08-30T23:50:00.247439+0000","last_became_active":"2024-08-30T23:48:58.219378+0000","last_became_peered":"2024-08-30T23:48:58.219378+0000","last_unstale":"2024-08-30T23:50:00.247439+0000","last_undegraded":"2024-08-30T23:50:00.247439+0000","last_fullsized":"2024-08-30T23:50:00.247439+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:05.558102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295174144,"num_objects":4504,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4504,"num_write_kb":288256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,1],"acting":[9,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":106,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.021176+0000","last_change":"2024-08-30T23:48:46.718850+0000","last_active":"2024-08-30T23:49:57.021176+0000","last_peered":"2024-08-30T23:49:57.021176+0000","last_clean":"2024-08-30T23:49:57.021176+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:49:57.021176+0000","last_undegraded":"2024-08-30T23:49:57.021176+0000","last_fullsized":"2024-08-30T23:49:57.021176+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:46.718784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:46.718784+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:13.746641+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"749'4654","reported_seq":4713,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.554657+0000","last_change":"2024-08-30T23:48:58.222833+0000","last_active":"2024-08-30T23:50:01.554657+0000","last_peered":"2024-08-30T23:50:01.554657+0000","last_clean":"2024-08-30T23:50:01.554657+0000","last_became_active":"2024-08-30T23:48:58.222601+0000","last_became_peered":"2024-08-30T23:48:58.222601+0000","last_unstale":"2024-08-30T23:50:01.554657+0000","last_undegraded":"2024-08-30T23:50:01.554657+0000","last_fullsized":"2024-08-30T23:50:01.554657+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:19.583225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305004544,"num_objects":4654,"num_object_clones":0,"num_object_copies":13962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4654,"num_write_kb":297856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":737,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.015884+0000","last_change":"2024-08-30T23:48:47.941617+0000","last_active":"2024-08-30T23:49:57.015884+0000","last_peered":"2024-08-30T23:49:57.015884+0000","last_clean":"2024-08-30T23:49:57.015884+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:49:57.015884+0000","last_undegraded":"2024-08-30T23:49:57.015884+0000","last_fullsized":"2024-08-30T23:49:57.015884+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:47.941549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:47.941549+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:12.373169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"749'4616","reported_seq":4673,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.124945+0000","last_change":"2024-08-30T23:48:58.218271+0000","last_active":"2024-08-30T23:50:01.124945+0000","last_peered":"2024-08-30T23:50:01.124945+0000","last_clean":"2024-08-30T23:50:01.124945+0000","last_became_active":"2024-08-30T23:48:58.217654+0000","last_became_peered":"2024-08-30T23:48:58.217654+0000","last_unstale":"2024-08-30T23:50:01.124945+0000","last_undegraded":"2024-08-30T23:50:01.124945+0000","last_fullsized":"2024-08-30T23:50:01.124945+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:24.602157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302514176,"num_objects":4616,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4616,"num_write_kb":295424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,1],"acting":[8,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":97,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.022769+0000","last_change":"2024-08-30T23:48:45.498288+0000","last_active":"2024-08-30T23:49:57.022769+0000","last_peered":"2024-08-30T23:49:57.022769+0000","last_clean":"2024-08-30T23:49:57.022769+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:49:57.022769+0000","last_undegraded":"2024-08-30T23:49:57.022769+0000","last_fullsized":"2024-08-30T23:49:57.022769+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:45.498217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:45.498217+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:59.301394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.4","version":"749'4652","reported_seq":4711,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.526232+0000","last_change":"2024-08-30T23:48:58.655895+0000","last_active":"2024-08-30T23:50:01.526232+0000","last_peered":"2024-08-30T23:50:01.526232+0000","last_clean":"2024-08-30T23:50:01.526232+0000","last_became_active":"2024-08-30T23:48:58.655572+0000","last_became_peered":"2024-08-30T23:48:58.655572+0000","last_unstale":"2024-08-30T23:50:01.526232+0000","last_undegraded":"2024-08-30T23:50:01.526232+0000","last_fullsized":"2024-08-30T23:50:01.526232+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:25.642821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304873472,"num_objects":4652,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4652,"num_write_kb":297728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,9],"acting":[4,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1572,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019728+0000","last_change":"2024-08-30T23:49:53.578515+0000","last_active":"2024-08-30T23:49:57.019728+0000","last_peered":"2024-08-30T23:49:57.019728+0000","last_clean":"2024-08-30T23:49:57.019728+0000","last_became_active":"2024-08-30T23:42:38.911981+0000","last_became_peered":"2024-08-30T23:42:38.911981+0000","last_unstale":"2024-08-30T23:49:57.019728+0000","last_undegraded":"2024-08-30T23:49:57.019728+0000","last_fullsized":"2024-08-30T23:49:57.019728+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:53.578434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:53.578434+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:58.083473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,8],"acting":[4,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.5","version":"749'4648","reported_seq":4706,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:02.397489+0000","last_change":"2024-08-30T23:48:58.221798+0000","last_active":"2024-08-30T23:50:02.397489+0000","last_peered":"2024-08-30T23:50:02.397489+0000","last_clean":"2024-08-30T23:50:02.397489+0000","last_became_active":"2024-08-30T23:48:58.221081+0000","last_became_peered":"2024-08-30T23:48:58.221081+0000","last_unstale":"2024-08-30T23:50:02.397489+0000","last_undegraded":"2024-08-30T23:50:02.397489+0000","last_fullsized":"2024-08-30T23:50:02.397489+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:05.446051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304611328,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":297472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,1],"acting":[11,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":467,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019829+0000","last_change":"2024-08-30T23:49:55.528037+0000","last_active":"2024-08-30T23:49:57.019829+0000","last_peered":"2024-08-30T23:49:57.019829+0000","last_clean":"2024-08-30T23:49:57.019829+0000","last_became_active":"2024-08-30T23:42:34.864987+0000","last_became_peered":"2024-08-30T23:42:34.864987+0000","last_unstale":"2024-08-30T23:49:57.019829+0000","last_undegraded":"2024-08-30T23:49:57.019829+0000","last_fullsized":"2024-08-30T23:49:57.019829+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:55.527977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:55.527977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:51:14.353176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.a","version":"749'4632","reported_seq":4689,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:02.395382+0000","last_change":"2024-08-30T23:48:58.424632+0000","last_active":"2024-08-30T23:50:02.395382+0000","last_peered":"2024-08-30T23:50:02.395382+0000","last_clean":"2024-08-30T23:50:02.395382+0000","last_became_active":"2024-08-30T23:48:58.424212+0000","last_became_peered":"2024-08-30T23:48:58.424212+0000","last_unstale":"2024-08-30T23:50:02.395382+0000","last_undegraded":"2024-08-30T23:50:02.395382+0000","last_fullsized":"2024-08-30T23:50:02.395382+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:06.889636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4632,"num_write_kb":296448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":551,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019766+0000","last_change":"2024-08-30T23:49:09.741136+0000","last_active":"2024-08-30T23:49:57.019766+0000","last_peered":"2024-08-30T23:49:57.019766+0000","last_clean":"2024-08-30T23:49:57.019766+0000","last_became_active":"2024-08-30T23:43:56.833115+0000","last_became_peered":"2024-08-30T23:43:56.833115+0000","last_unstale":"2024-08-30T23:49:57.019766+0000","last_undegraded":"2024-08-30T23:49:57.019766+0000","last_fullsized":"2024-08-30T23:49:57.019766+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:09.741057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:09.741057+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:11.815256+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"749'4522","reported_seq":4581,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.112558+0000","last_change":"2024-08-30T23:48:58.219658+0000","last_active":"2024-08-30T23:50:01.112558+0000","last_peered":"2024-08-30T23:50:01.112558+0000","last_clean":"2024-08-30T23:50:01.112558+0000","last_became_active":"2024-08-30T23:48:58.219254+0000","last_became_peered":"2024-08-30T23:48:58.219254+0000","last_unstale":"2024-08-30T23:50:01.112558+0000","last_undegraded":"2024-08-30T23:50:01.112558+0000","last_fullsized":"2024-08-30T23:50:01.112558+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:23.300635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296353792,"num_objects":4522,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":289408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":537,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019875+0000","last_change":"2024-08-30T23:49:10.615399+0000","last_active":"2024-08-30T23:49:57.019875+0000","last_peered":"2024-08-30T23:49:57.019875+0000","last_clean":"2024-08-30T23:49:57.019875+0000","last_became_active":"2024-08-30T23:42:38.914746+0000","last_became_peered":"2024-08-30T23:42:38.914746+0000","last_unstale":"2024-08-30T23:49:57.019875+0000","last_undegraded":"2024-08-30T23:49:57.019875+0000","last_fullsized":"2024-08-30T23:49:57.019875+0000","mapping_epoch":539,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":540,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:10.615315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:10.615315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:32.623559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"749'4607","reported_seq":4666,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.559317+0000","last_change":"2024-08-30T23:48:59.164838+0000","last_active":"2024-08-30T23:50:01.559317+0000","last_peered":"2024-08-30T23:50:01.559317+0000","last_clean":"2024-08-30T23:50:01.559317+0000","last_became_active":"2024-08-30T23:48:59.164580+0000","last_became_peered":"2024-08-30T23:48:59.164580+0000","last_unstale":"2024-08-30T23:50:01.559317+0000","last_undegraded":"2024-08-30T23:50:01.559317+0000","last_fullsized":"2024-08-30T23:50:01.559317+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:07.146620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301924352,"num_objects":4607,"num_object_clones":0,"num_object_copies":13821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4607,"num_write_kb":294848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":467,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.020131+0000","last_change":"2024-08-30T23:48:41.757459+0000","last_active":"2024-08-30T23:49:57.020131+0000","last_peered":"2024-08-30T23:49:57.020131+0000","last_clean":"2024-08-30T23:49:57.020131+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:49:57.020131+0000","last_undegraded":"2024-08-30T23:49:57.020131+0000","last_fullsized":"2024-08-30T23:49:57.020131+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:41.757393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:41.757393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:00.979571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"749'4722","reported_seq":4781,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:02.165138+0000","last_change":"2024-08-30T23:48:58.221056+0000","last_active":"2024-08-30T23:50:02.165138+0000","last_peered":"2024-08-30T23:50:02.165138+0000","last_clean":"2024-08-30T23:50:02.165138+0000","last_became_active":"2024-08-30T23:48:58.220724+0000","last_became_peered":"2024-08-30T23:48:58.220724+0000","last_unstale":"2024-08-30T23:50:02.165138+0000","last_undegraded":"2024-08-30T23:50:02.165138+0000","last_fullsized":"2024-08-30T23:50:02.165138+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:04.558028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309460992,"num_objects":4722,"num_object_clones":0,"num_object_copies":14166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":302208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":739,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.015799+0000","last_change":"2024-08-30T23:48:46.906841+0000","last_active":"2024-08-30T23:49:57.015799+0000","last_peered":"2024-08-30T23:49:57.015799+0000","last_clean":"2024-08-30T23:49:57.015799+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:49:57.015799+0000","last_undegraded":"2024-08-30T23:49:57.015799+0000","last_fullsized":"2024-08-30T23:49:57.015799+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:46.906788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:46.906788+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:07.674002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"749'4683","reported_seq":4741,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.758241+0000","last_change":"2024-08-30T23:48:58.424683+0000","last_active":"2024-08-30T23:50:01.758241+0000","last_peered":"2024-08-30T23:50:01.758241+0000","last_clean":"2024-08-30T23:50:01.758241+0000","last_became_active":"2024-08-30T23:48:58.424286+0000","last_became_peered":"2024-08-30T23:48:58.424286+0000","last_unstale":"2024-08-30T23:50:01.758241+0000","last_undegraded":"2024-08-30T23:50:01.758241+0000","last_fullsized":"2024-08-30T23:50:01.758241+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:06.711370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":299712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":664,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.020608+0000","last_change":"2024-08-30T23:49:49.879872+0000","last_active":"2024-08-30T23:49:57.020608+0000","last_peered":"2024-08-30T23:49:57.020608+0000","last_clean":"2024-08-30T23:49:57.020608+0000","last_became_active":"2024-08-30T23:42:20.849632+0000","last_became_peered":"2024-08-30T23:42:20.849632+0000","last_unstale":"2024-08-30T23:49:57.020608+0000","last_undegraded":"2024-08-30T23:49:57.020608+0000","last_fullsized":"2024-08-30T23:49:57.020608+0000","mapping_epoch":529,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":530,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:49.879777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:49.879777+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:51:16.492112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"749'4628","reported_seq":4686,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:01.128775+0000","last_change":"2024-08-30T23:48:58.219444+0000","last_active":"2024-08-30T23:50:01.128775+0000","last_peered":"2024-08-30T23:50:01.128775+0000","last_clean":"2024-08-30T23:50:01.128775+0000","last_became_active":"2024-08-30T23:48:58.219114+0000","last_became_peered":"2024-08-30T23:48:58.219114+0000","last_unstale":"2024-08-30T23:50:01.128775+0000","last_undegraded":"2024-08-30T23:50:01.128775+0000","last_fullsized":"2024-08-30T23:50:01.128775+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:25.799671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303300608,"num_objects":4628,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":296192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,11],"acting":[8,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1558,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.020510+0000","last_change":"2024-08-30T23:49:45.295016+0000","last_active":"2024-08-30T23:49:57.020510+0000","last_peered":"2024-08-30T23:49:57.020510+0000","last_clean":"2024-08-30T23:49:57.020510+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:49:57.020510+0000","last_undegraded":"2024-08-30T23:49:57.020510+0000","last_fullsized":"2024-08-30T23:49:57.020510+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:45.294933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:45.294933+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:57.436476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"749'4579","reported_seq":4638,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:02.161318+0000","last_change":"2024-08-30T23:48:58.221080+0000","last_active":"2024-08-30T23:50:02.161318+0000","last_peered":"2024-08-30T23:50:02.161318+0000","last_clean":"2024-08-30T23:50:02.161318+0000","last_became_active":"2024-08-30T23:48:58.220797+0000","last_became_peered":"2024-08-30T23:48:58.220797+0000","last_unstale":"2024-08-30T23:50:02.161318+0000","last_undegraded":"2024-08-30T23:50:02.161318+0000","last_fullsized":"2024-08-30T23:50:02.161318+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300089344,"num_objects":4579,"num_object_clones":0,"num_object_copies":13737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4579,"num_write_kb":293056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":680,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019115+0000","last_change":"2024-08-30T23:49:50.688833+0000","last_active":"2024-08-30T23:49:57.019115+0000","last_peered":"2024-08-30T23:49:57.019115+0000","last_clean":"2024-08-30T23:49:57.019115+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:49:57.019115+0000","last_undegraded":"2024-08-30T23:49:57.019115+0000","last_fullsized":"2024-08-30T23:49:57.019115+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:50.688764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:50.688764+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:50:51.236298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"749'4766","reported_seq":4824,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:50:04.075915+0000","last_change":"2024-08-30T23:48:58.757075+0000","last_active":"2024-08-30T23:50:04.075915+0000","last_peered":"2024-08-30T23:50:04.075915+0000","last_clean":"2024-08-30T23:50:04.075915+0000","last_became_active":"2024-08-30T23:48:58.756762+0000","last_became_peered":"2024-08-30T23:48:58.756762+0000","last_unstale":"2024-08-30T23:50:04.075915+0000","last_undegraded":"2024-08-30T23:50:04.075915+0000","last_fullsized":"2024-08-30T23:50:04.075915+0000","mapping_epoch":725,"log_start":"0'0","ondisk_log_start":"0'0","created":725,"last_epoch_clean":726,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:48:57.189684+0000","last_clean_scrub_stamp":"2024-08-30T23:48:57.189684+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312344576,"num_objects":4766,"num_object_clones":0,"num_object_copies":14298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":305024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":453,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.019222+0000","last_change":"2024-08-30T23:48:41.578823+0000","last_active":"2024-08-30T23:49:57.019222+0000","last_peered":"2024-08-30T23:49:57.019222+0000","last_clean":"2024-08-30T23:49:57.019222+0000","last_became_active":"2024-08-30T23:42:43.957326+0000","last_became_peered":"2024-08-30T23:42:43.957326+0000","last_unstale":"2024-08-30T23:49:57.019222+0000","last_undegraded":"2024-08-30T23:49:57.019222+0000","last_fullsized":"2024-08-30T23:49:57.019222+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:41.578733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:41.578733+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:57.912595+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1092,"reported_epoch":749,"state":"active+clean","last_fresh":"2024-08-30T23:49:57.020462+0000","last_change":"2024-08-30T23:48:43.414875+0000","last_active":"2024-08-30T23:49:57.020462+0000","last_peered":"2024-08-30T23:49:57.020462+0000","last_clean":"2024-08-30T23:49:57.020462+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:49:57.020462+0000","last_undegraded":"2024-08-30T23:49:57.020462+0000","last_fullsized":"2024-08-30T23:49:57.020462+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:48:43.414802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:48:43.414802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:49:58.368584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4886364160,"num_objects":74560,"num_object_clones":0,"num_object_copies":223680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74560,"num_write_kb":4771840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14647427072,"data_stored":14647427072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74560,"ondisk_log_size":74560,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187762,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2330224,"kb_used_data":1775648,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91414928,"statfs":{"total":95995035648,"available":93608886272,"internally_reserved":0,"allocated":1818263552,"data_stored":1817771228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":567867844},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449403,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2643312,"kb_used_data":2192864,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91101840,"statfs":{"total":95995035648,"available":93288284160,"internally_reserved":0,"allocated":2245492736,"data_stored":2245000412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":461240768},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711036,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2318384,"kb_used_data":1723552,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91426768,"statfs":{"total":95995035648,"available":93621010432,"internally_reserved":0,"allocated":1764917248,"data_stored":1764424924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":609089984},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972674,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1309112,"kb_used_data":885032,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92436040,"statfs":{"total":95995035648,"available":94654504960,"internally_reserved":0,"allocated":906272768,"data_stored":905776892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1262448,"kb_used_data":591264,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92482704,"statfs":{"total":95995035648,"available":94702288896,"internally_reserved":0,"allocated":605454336,"data_stored":604962012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":687274442},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528653,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707952,"kb_used_data":293600,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93037200,"statfs":{"total":95995035648,"available":95270092800,"internally_reserved":0,"allocated":300646400,"data_stored":300154076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424278470},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775156,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137064,"kb_used_data":1312,"kb_used_omap":5,"kb_used_meta":135546,"kb_avail":93608088,"statfs":{"total":95995035648,"available":95854682112,"internally_reserved":0,"allocated":1343488,"data_stored":851164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":138799178},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953816,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1699568,"kb_used_data":1176928,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92045584,"statfs":{"total":95995035648,"available":94254678016,"internally_reserved":0,"allocated":1205174272,"data_stored":1204681948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150042,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3119288,"kb_used_data":2434216,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":90625864,"statfs":{"total":95995035648,"available":92800884736,"internally_reserved":0,"allocated":2492637184,"data_stored":2492141308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":701495751},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411680,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1979832,"kb_used_data":1525224,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":91765320,"statfs":{"total":95995035648,"available":93967687680,"internally_reserved":0,"allocated":1561829376,"data_stored":1561333500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830495,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2066800,"kb_used_data":1754272,"kb_used_omap":87,"kb_used_meta":312424,"kb_avail":91678352,"statfs":{"total":95995035648,"available":93878632448,"internally_reserved":0,"allocated":1796374528,"data_stored":1795882204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":319922335},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738730,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901296,"kb_used_data":302304,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92843856,"statfs":{"total":95995035648,"available":95072108544,"internally_reserved":0,"allocated":309559296,"data_stored":309066972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309723136,"data_stored":309723136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1798766592,"data_stored":1798766592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1566441472,"data_stored":1566441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2500984832,"data_stored":2500984832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1220608000,"data_stored":1220608000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":300154880,"data_stored":300154880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":607715328,"data_stored":607715328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":909574144,"data_stored":909574144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1784414208,"data_stored":1784414208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1821048832,"data_stored":1821048832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1827995648,"data_stored":1827995648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:50:07.264 INFO:tasks.thrashosds.thrasher:Setting 7.9 to [11, 7, 9, 1, 8, 0] 2024-08-30T23:50:07.264 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.9', '11', '7', '9', '1', '8', '0'] 2024-08-30T23:50:07.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 7.9 11 7 9 1 8 0 2024-08-30T23:50:07.286 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:07 smithi106 ceph-mon[16603]: pgmap v1677: 37 pgs: 37 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:50:07.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:07.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:07 smithi173 ceph-mon[20554]: pgmap v1677: 37 pgs: 37 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:50:07.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:07.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:07.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:07.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 79911 79895 78.0156 58.3125 0.00389185 0.0128124 2024-08-30T23:50:07.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:07 smithi136 ceph-mon[20207]: pgmap v1677: 37 pgs: 37 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:50:07.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:07.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:07.661 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:50:07.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:07.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:07.803 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:07.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:07.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:07.885 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:07.980 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:50:07.981 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:50:07.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:08.014 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:50:08.014 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:50:08.016 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:08.045 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:50:08.045 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:50:08.046 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:08.077 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:50:08.078 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:50:08.103 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:50:08.103 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:50:08.118 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:08.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:08.147 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:50:08.148 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:50:08.175 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:50:08.176 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:50:08.201 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:50:08.201 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:50:08.231 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2250737741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:08.234 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:50:08.234 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:50:08.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:08.268 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:50:08.268 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:50:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:08.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:08.301 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:50:08.301 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:50:08.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2250737741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:08.335 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:50:08.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:50:08.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:08.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.416 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:08.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 81021 81005 77.8825 69.375 0.00583147 0.0128335 2024-08-30T23:50:08.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:08.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2250737741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:08.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:08.632 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:50:08.632 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:50:08.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:08.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:08.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:08.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:08.913 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:08.934 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:08.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:09.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:09.056 INFO:teuthology.orchestra.run.smithi106.stderr:set 7.9 pg_upmap_items mapping to [11->7,9->1,8->0] 2024-08-30T23:50:09.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:09.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:09.186 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:09.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:09.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:09.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:09 smithi173 ceph-mon[20554]: pgmap v1678: 37 pgs: 37 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:50:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3002030004' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [11, 7, 9, 1, 8, 0]} : dispatch 2024-08-30T23:50:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:09 smithi106 ceph-mon[16603]: pgmap v1678: 37 pgs: 37 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:50:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3002030004' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [11, 7, 9, 1, 8, 0]} : dispatch 2024-08-30T23:50:09.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:50:09.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:09.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.471 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:09.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 81673 81657 77.3198 40.75 0.00329862 0.0128778 2024-08-30T23:50:09.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:09 smithi136 ceph-mon[20207]: pgmap v1678: 37 pgs: 37 active+clean; 4.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:50:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3002030004' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [11, 7, 9, 1, 8, 0]} : dispatch 2024-08-30T23:50:09.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:09.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:09.650 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:09.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:09.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:09.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:50:09.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:09.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:09.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:09.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:10.013 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:10.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:10.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.089 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:10.147 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:50:10.147 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:50:10.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:10.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:10.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:10.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3002030004' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [11, 7, 9, 1, 8, 0]}]': finished 2024-08-30T23:50:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:10 smithi173 ceph-mon[20554]: osdmap e751: 12 total, 12 up, 12 in 2024-08-30T23:50:10.335 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.336 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3002030004' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [11, 7, 9, 1, 8, 0]}]': finished 2024-08-30T23:50:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:10 smithi106 ceph-mon[16603]: osdmap e751: 12 total, 12 up, 12 in 2024-08-30T23:50:10.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:10.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:10.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 81745 81729 76.233 4.5 0.00357066 0.0129049 2024-08-30T23:50:10.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3002030004' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.9", "id": [11, 7, 9, 1, 8, 0]}]': finished 2024-08-30T23:50:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:10 smithi136 ceph-mon[20207]: osdmap e751: 12 total, 12 up, 12 in 2024-08-30T23:50:10.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:10.582 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:10.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:10.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:10.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:50:10.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:10.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:10.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:10.965 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:10.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:11.020 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:11.061 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:50:11.065 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:11 smithi173 ceph-mon[20554]: pgmap v1680: 37 pgs: 37 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:50:11.065 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:11 smithi173 ceph-mon[20554]: osdmap e752: 12 total, 12 up, 12 in 2024-08-30T23:50:11.065 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/230624144' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:50:11.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:11.093 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:11.136 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:50:11.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:11.216 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:11.317 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:11.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:11 smithi106 ceph-mon[16603]: pgmap v1680: 37 pgs: 37 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:50:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:11 smithi106 ceph-mon[16603]: osdmap e752: 12 total, 12 up, 12 in 2024-08-30T23:50:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/230624144' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:50:11.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:11.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:11.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 81787 81771 75.1505 2.625 0.00510132 0.0131128 2024-08-30T23:50:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:11 smithi136 ceph-mon[20207]: pgmap v1680: 37 pgs: 37 active+clean; 4.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s 2024-08-30T23:50:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:11 smithi136 ceph-mon[20207]: osdmap e752: 12 total, 12 up, 12 in 2024-08-30T23:50:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/230624144' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:50:11.558 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:50:11.558 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:50:11.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:11.598 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.599 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:11.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:11.723 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:11.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:11.848 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:11.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:11.982 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:11.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:12.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.109 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:12.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:12.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:12.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:12.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:12 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:12.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/230624144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:50:12.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:12 smithi106 ceph-mon[16603]: osdmap e753: 12 total, 12 up, 12 in 2024-08-30T23:50:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:12 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/230624144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:50:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:12 smithi173 ceph-mon[20554]: osdmap e753: 12 total, 12 up, 12 in 2024-08-30T23:50:12.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:12.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:12.430 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:12.466 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:12.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:12.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 82718 82702 74.9046 58.1875 0.0076091 0.0133431 2024-08-30T23:50:12.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:12 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:12.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/230624144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:50:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:12 smithi136 ceph-mon[20207]: osdmap e753: 12 total, 12 up, 12 in 2024-08-30T23:50:12.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:12.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:12.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:12.739 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:12.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:12.864 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:12.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:12.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:12.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:12.993 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:50:12.993 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:50:13.020 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:50:13.020 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:50:13.053 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:50:13.053 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:50:13.085 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:50:13.086 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:50:13.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:13 smithi106 ceph-mon[16603]: pgmap v1683: 37 pgs: 37 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:50:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3553987835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:13.115 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:50:13.115 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:50:13.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:13.149 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:50:13.149 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:50:13.177 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:50:13.177 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:50:13.209 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:50:13.209 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:50:13.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:13.238 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:50:13.239 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:50:13.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:13.270 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:50:13.270 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:50:13.296 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:50:13.296 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:50:13.318 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:50:13.318 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:50:13.342 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:13.363 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:13.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:13.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:13.508 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 83598 83582 74.6202 55 0.0106803 0.0133929 2024-08-30T23:50:13.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:13 smithi136 ceph-mon[20207]: pgmap v1683: 37 pgs: 37 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:50:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3553987835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:13 smithi173 ceph-mon[20554]: pgmap v1683: 37 pgs: 37 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-08-30T23:50:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3553987835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:13.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:13.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:13.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:13.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:13.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:13.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:13.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:13.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:13.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:50:14.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:14.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:14.156 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:14.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:14.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:14.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:50:14.378 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:14.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:14.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:14.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 84606 84590 74.4565 63 0.00368981 0.0134238 2024-08-30T23:50:14.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:14.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:14.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:14.677 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:14.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:14.773 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:50:14.773 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:50:14.773 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-08-30T23:50:14.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd out 11 2024-08-30T23:50:14.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:14.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:14.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:14.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:14.943 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:14.987 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:84972 7.9 7:9f2df097:::benchmark_data_smithi173_129296_object84971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e753)", 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:14.966564+0000", 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.019822675000000001, 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.019854622999999998, 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:50:14.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 84972 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.966564+0000", 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:14.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.966564+0000", 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.966565+0000", 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.497e-06 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.966594+0000", 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9275e-05 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.966611+0000", 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6257999999999999e-05 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.966618+0000", 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8180000000000004e-06 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.968510+0000", 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001891935 2024-08-30T23:50:14.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.968679+0000", 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000168837 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.968857+0000", 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017728199999999999 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:14.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.983020+0000", 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014163485 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27487.0:84978 7.1 e753/725 7:80c916f0:::benchmark_data_smithi173_129296_object84977:head v 753'5360, mlcod=753'5358)", 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:14.974132+0000", 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.012253979999999999, 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012359103999999999, 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:14.993 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.974132+0000", 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.974132+0000", 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.974133+0000", 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5400000000000001e-07 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.974169+0000", 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6341000000000003e-05 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.974209+0000", 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9196999999999998e-05 2024-08-30T23:50:14.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.974215+0000", 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8820000000000003e-06 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.982227+0000", 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0080116900000000001 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.982247+0000", 2024-08-30T23:50:14.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.033e-05 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27487.0:84987 7.6 e753/725 7:6b108d26:::benchmark_data_smithi173_129296_object84986:head v 753'5214, mlcod=753'5212)", 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:14.984175+0000", 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0022113710000000002, 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023649040000000001, 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:14.997 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984175+0000", 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984175+0000", 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984176+0000", 2024-08-30T23:50:14.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.201e-06 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984211+0000", 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5005000000000001e-05 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984257+0000", 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5711999999999999e-05 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:14.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984264+0000", 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2849999999999997e-06 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984377+0000", 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000112458 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984396+0000", 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9119000000000001e-05 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.000 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27487.0:84988 7.2 e753/725 7:4aa64be9:::benchmark_data_smithi173_129296_object84987:head v 753'5339, mlcod=753'5337)", 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:14.984216+0000", 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0021704979999999999, 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023706880000000001, 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984216+0000", 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984216+0000", 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984216+0000", 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2599999999999998e-07 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984308+0000", 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1828000000000002e-05 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984318+0000", 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0237e-05 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984323+0000", 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6330000000000004e-06 2024-08-30T23:50:15.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984439+0000", 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011628099999999999 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.984453+0000", 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.361e-05 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:84969 7.e 7:797024a7:::benchmark_data_smithi173_129296_object84968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e753)", 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:14.959764+0000", 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.026622789000000001, 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026878144, 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 84969 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.005 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959764+0000", 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959764+0000", 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959765+0000", 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.342e-06 2024-08-30T23:50:15.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959784+0000", 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9369999999999999e-05 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959896+0000", 2024-08-30T23:50:15.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111827 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959902+0000", 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.088e-06 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.959954+0000", 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1459000000000003e-05 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.960064+0000", 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011045899999999999 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.960154+0000", 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0228000000000004e-05 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.963304+0000", 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0031494549999999998 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:15.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:14.968224+0000", 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049206040000000003 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 5 2024-08-30T23:50:15.010 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:15.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:15.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:15.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:15.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:15.270 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:15.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:15 smithi173 ceph-mon[20554]: pgmap v1684: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s; 10276/245727 objects misplaced (4.182%); 53 KiB/s, 0 objects/s recovering 2024-08-30T23:50:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:15 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:50:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:15 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:50:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:15 smithi173 ceph-mon[20554]: 7.0 scrub starts 2024-08-30T23:50:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:15.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:15 smithi106 ceph-mon[16603]: pgmap v1684: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s; 10276/245727 objects misplaced (4.182%); 53 KiB/s, 0 objects/s recovering 2024-08-30T23:50:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:15 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:50:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:15 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:50:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:15 smithi106 ceph-mon[16603]: 7.0 scrub starts 2024-08-30T23:50:15.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:15.452 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:15.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 85378 85362 74.0924 48.25 0.00822423 0.0134827 2024-08-30T23:50:15.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:15 smithi136 ceph-mon[20207]: pgmap v1684: 37 pgs: 1 active+recovering+undersized+remapped, 36 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s; 10276/245727 objects misplaced (4.182%); 53 KiB/s, 0 objects/s recovering 2024-08-30T23:50:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:15 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:50:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:15 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:50:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:15 smithi136 ceph-mon[20207]: 7.0 scrub starts 2024-08-30T23:50:15.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:15.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:15.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:15.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:15.816 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:50:15.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:15.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:15.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:15.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:16.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:16.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:16.088 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:50:16.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:16.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:50:16.197 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:16.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:16.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:16 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:50:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:16 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:50:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3319668641' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:16 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:16.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:16 smithi173 ceph-mon[20554]: 7.2 scrub starts 2024-08-30T23:50:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:16 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:50:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:16 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:50:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3319668641' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:16 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:16 smithi106 ceph-mon[16603]: 7.2 scrub starts 2024-08-30T23:50:16.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:16 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:50:16.133+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e754 definitely_dead 0 2024-08-30T23:50:16.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:16.425 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:16.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 15 86205 86190 73.7863 51.75 0.0368988 0.0135465 2024-08-30T23:50:16.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:16 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:50:16.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:16 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:50:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3319668641' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:16 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:16 smithi136 ceph-mon[20207]: 7.2 scrub starts 2024-08-30T23:50:16.550 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:16.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:16.680 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:16.781 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:16.781 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:50:16.781 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:50:16.781 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:50:16.781 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:16.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:16.817 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:16.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:16.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:16.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:16.954 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:17.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:17.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:17.087 INFO:teuthology.orchestra.run.smithi106.stderr:marked out osd.11. 2024-08-30T23:50:17.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:50:17.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:50:17.105 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: pgmap v1685: 37 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 35 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 661 op/s; 10276/248442 objects misplaced (4.136%); 48 KiB/s, 0 objects/s recovering 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: osdmap e754: 12 total, 12 up, 12 in 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/802774369' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-30T23:50:17.106 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:17 smithi106 ceph-mon[16603]: 7.0 scrub ok 2024-08-30T23:50:17.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:17.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:17.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:17.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:17.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:17.446 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:17.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 86733 86717 73.2342 32.9375 0.00343173 0.0135806 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: pgmap v1685: 37 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 35 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 661 op/s; 10276/248442 objects misplaced (4.136%); 48 KiB/s, 0 objects/s recovering 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:50:17.516 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:17.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: osdmap e754: 12 total, 12 up, 12 in 2024-08-30T23:50:17.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/802774369' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:50:17.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-30T23:50:17.517 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:17 smithi173 ceph-mon[20554]: 7.0 scrub ok 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: pgmap v1685: 37 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 35 active+clean; 5.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 661 op/s; 10276/248442 objects misplaced (4.136%); 48 KiB/s, 0 objects/s recovering 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: osdmap e754: 12 total, 12 up, 12 in 2024-08-30T23:50:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/802774369' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:50:17.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-30T23:50:17.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:17 smithi136 ceph-mon[20207]: 7.0 scrub ok 2024-08-30T23:50:17.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:50:17.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:17.570 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:17.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:17.674 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:50:17.674 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:50:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:17.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:17.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:50:17.696 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:50:17.721 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:50:17.722 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:50:17.742 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:50:17.742 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:50:17.763 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:50:17.763 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:50:17.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:50:17.785 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:50:17.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:17.798 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.819 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:17.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:17.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:17.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:17.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:18.007 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:50:18.007 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:50:18.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:18.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:18.130 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:50:18.131 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:50:18.156 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:50:18.157 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:50:18.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:18.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:50:18.186 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:50:18.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:18.261 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:50:18.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:50:18.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/802774369' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-30T23:50:18.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: osdmap e755: 12 total, 12 up, 11 in 2024-08-30T23:50:18.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1720185844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:18.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: 7.2 scrub ok 2024-08-30T23:50:18.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:18 smithi106 ceph-mon[16603]: 7.3 scrub starts 2024-08-30T23:50:18.269 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:50:18.270 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:50:18.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:18.293 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:50:18.293 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:50:18.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/802774369' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: osdmap e755: 12 total, 12 up, 11 in 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1720185844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: 7.2 scrub ok 2024-08-30T23:50:18.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:18 smithi173 ceph-mon[20554]: 7.3 scrub starts 2024-08-30T23:50:18.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:18.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:18.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 86736 86720 72.2602 0.1875 0.615989 0.0135992 2024-08-30T23:50:18.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:50:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:50:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/802774369' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-30T23:50:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: osdmap e755: 12 total, 12 up, 11 in 2024-08-30T23:50:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1720185844' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: 7.2 scrub ok 2024-08-30T23:50:18.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:18 smithi136 ceph-mon[20207]: 7.3 scrub starts 2024-08-30T23:50:18.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:18.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:18.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:18.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:18.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:18.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:50:18.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:18.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:18.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:19.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:19.058 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:19.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:19.183 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:19.190 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:19.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:19.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 7.e scrub starts 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 7.8 scrub starts 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: pgmap v1688: 37 pgs: 2 peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 33 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 50 MiB/s wr, 799 op/s; 10040/255654 objects misplaced (3.927%); 1.2 MiB/s, 18 objects/s recovering 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: osdmap e756: 12 total, 12 up, 11 in 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 7.e scrub ok 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 7.3 scrub ok 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:50:19.324 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:19 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 7.e scrub starts 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 7.8 scrub starts 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: pgmap v1688: 37 pgs: 2 peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 33 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 50 MiB/s wr, 799 op/s; 10040/255654 objects misplaced (3.927%); 1.2 MiB/s, 18 objects/s recovering 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: osdmap e756: 12 total, 12 up, 11 in 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 7.e scrub ok 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 7.3 scrub ok 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:50:19.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:19 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:50:19.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.434 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:19.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:19.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 86816 86800 71.3752 5 0.0154537 0.0139589 2024-08-30T23:50:19.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 7.e scrub starts 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 7.8 scrub starts 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: pgmap v1688: 37 pgs: 2 peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 33 active+clean; 5.2 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 50 MiB/s wr, 799 op/s; 10040/255654 objects misplaced (3.927%); 1.2 MiB/s, 18 objects/s recovering 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: osdmap e756: 12 total, 12 up, 11 in 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 7.e scrub ok 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 7.3 scrub ok 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:50:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:19 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:50:19.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:19.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:19.687 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:19.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:19.815 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:19.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:19.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:19.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:19.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:20.043 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:20.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:20.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:20.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:20.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:20 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:50:20.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:20 smithi106 ceph-mon[16603]: 7.8 scrub ok 2024-08-30T23:50:20.281 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:20 smithi106 ceph-mon[16603]: osdmap e757: 12 total, 12 up, 11 in 2024-08-30T23:50:20.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:20.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:20.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:20.446 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:20.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:20.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 87087 87071 70.6682 16.9375 0.00442623 0.0139945 2024-08-30T23:50:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:20 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:50:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:20 smithi136 ceph-mon[20207]: 7.8 scrub ok 2024-08-30T23:50:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:20 smithi136 ceph-mon[20207]: osdmap e757: 12 total, 12 up, 11 in 2024-08-30T23:50:20.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:20.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:20.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:20 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:50:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:20 smithi173 ceph-mon[20554]: 7.8 scrub ok 2024-08-30T23:50:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:20 smithi173 ceph-mon[20554]: osdmap e757: 12 total, 12 up, 11 in 2024-08-30T23:50:20.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:20.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:20.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:20.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:20.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:20.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:20.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:21.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:21.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.072 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:21.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:50:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:50:21.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:21.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:21.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:21.247 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:50:21.247 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:50:21.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:21.318 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:21.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:21 smithi106 ceph-mon[16603]: pgmap v1691: 37 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 32 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 829 op/s; 10040/258402 objects misplaced (3.885%); 1.8 MiB/s, 29 objects/s recovering 2024-08-30T23:50:21.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3674733323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:50:21.421 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:21.443 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:21.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:21 smithi173 ceph-mon[20554]: pgmap v1691: 37 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 32 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 829 op/s; 10040/258402 objects misplaced (3.885%); 1.8 MiB/s, 29 objects/s recovering 2024-08-30T23:50:21.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3674733323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:50:21.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 87113 87097 69.783 1.625 0.00373862 0.0140007 2024-08-30T23:50:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:21 smithi136 ceph-mon[20207]: pgmap v1691: 37 pgs: 1 remapped+peering, 2 peering, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 32 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 52 MiB/s wr, 829 op/s; 10040/258402 objects misplaced (3.885%); 1.8 MiB/s, 29 objects/s recovering 2024-08-30T23:50:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3674733323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:50:21.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:21.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:21.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:21.690 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:21.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:21.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:21.813 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:21.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:21.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:21.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:21.956 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:22.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:22.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:22.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.187 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:22.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:22.231 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:22 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:22.231 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3674733323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:50:22.231 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:22 smithi106 ceph-mon[16603]: osdmap e758: 12 total, 12 up, 11 in 2024-08-30T23:50:22.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:22.311 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:22.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:22.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:22.503 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:50:22.503 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:50:22.509 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 87113 87097 68.8997 0 - 0.0140007 2024-08-30T23:50:22.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:50:22.532 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:50:22.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:22 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3674733323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:50:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:22 smithi136 ceph-mon[20207]: osdmap e758: 12 total, 12 up, 11 in 2024-08-30T23:50:22.556 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:50:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:50:22.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:22.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:22 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3674733323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:50:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:22 smithi173 ceph-mon[20554]: osdmap e758: 12 total, 12 up, 11 in 2024-08-30T23:50:22.578 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:50:22.578 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:50:22.600 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:50:22.601 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:50:22.623 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:50:22.623 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:50:22.649 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:50:22.649 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:50:22.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:22.672 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:50:22.673 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:50:22.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:22.700 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:50:22.700 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:50:22.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:22.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:22.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:22.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:22.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:23.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:23.063 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.064 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:23.165 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:23.188 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:23.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:23.296 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:50:23.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-08-30T23:50:23.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:23.415 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: pgmap v1693: 37 pgs: 1 remapped+peering, 2 peering, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 30 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 22 MiB/s wr, 344 op/s; 20692/259563 objects misplaced (7.972%); 2.3 MiB/s, 37 objects/s recovering 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: osdmap e759: 12 total, 12 up, 11 in 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/158715212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:23.437 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.438 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:23.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:23.440 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: pgmap v1693: 37 pgs: 1 remapped+peering, 2 peering, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 30 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 22 MiB/s wr, 344 op/s; 20692/259563 objects misplaced (7.972%); 2.3 MiB/s, 37 objects/s recovering 2024-08-30T23:50:23.440 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:23.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:23.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: osdmap e759: 12 total, 12 up, 11 in 2024-08-30T23:50:23.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/158715212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:23.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:23.441 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:23.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:50:23.510627+0000 min lat: 0.00104498 max lat: 2.61817 avg lat: 0.0140007 2024-08-30T23:50:23.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:50:23.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 87113 87097 68.0384 0 - 0.0140007 2024-08-30T23:50:23.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:23.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: pgmap v1693: 37 pgs: 1 remapped+peering, 2 peering, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 30 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 22 MiB/s wr, 344 op/s; 20692/259563 objects misplaced (7.972%); 2.3 MiB/s, 37 objects/s recovering 2024-08-30T23:50:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: osdmap e759: 12 total, 12 up, 11 in 2024-08-30T23:50:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/158715212' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:50:23.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:23.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:23.623 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:23.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:23.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:23.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:23.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:23.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:50:23.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:23.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:50:23.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:23.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:24.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:24.064 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:24.136 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:24 smithi173 ceph-mon[20554]: osdmap e760: 12 total, 12 up, 11 in 2024-08-30T23:50:24.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:24.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:24.205 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:24.230 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:24 smithi136 ceph-mon[20207]: osdmap e760: 12 total, 12 up, 11 in 2024-08-30T23:50:24.244 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:24.290 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:24.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:24 smithi106 ceph-mon[16603]: osdmap e760: 12 total, 12 up, 11 in 2024-08-30T23:50:24.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:24.439 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:24.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 87113 87097 67.1984 0 - 0.0140007 2024-08-30T23:50:24.541 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:24.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:24.666 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:24.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:24.717 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:50:24.717 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:50:24.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:24.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:24.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:24.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:24.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:25.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:25.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:25.142 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:50:25.143 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.0 to 1 (10000) 2024-08-30T23:50:25.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:25.191 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:25.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:25.315 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:25.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:25.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.440 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:25.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 87113 87097 66.3789 0 - 0.0140007 2024-08-30T23:50:25.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:25 smithi136 ceph-mon[20207]: pgmap v1696: 37 pgs: 1 remapped+peering, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 31 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 6.5 MiB/s wr, 103 op/s; 26158/259833 objects misplaced (10.067%); 5.7 MiB/s, 91 objects/s recovering 2024-08-30T23:50:25.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/548400440' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:50:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2912159325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:25.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:25.542 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:25 smithi173 ceph-mon[20554]: pgmap v1696: 37 pgs: 1 remapped+peering, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 31 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 6.5 MiB/s wr, 103 op/s; 26158/259833 objects misplaced (10.067%); 5.7 MiB/s, 91 objects/s recovering 2024-08-30T23:50:25.542 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/548400440' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:50:25.542 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2912159325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:25.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:25 smithi106 ceph-mon[16603]: pgmap v1696: 37 pgs: 1 remapped+peering, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 31 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 6.5 MiB/s wr, 103 op/s; 26158/259833 objects misplaced (10.067%); 5.7 MiB/s, 91 objects/s recovering 2024-08-30T23:50:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/548400440' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-30T23:50:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2912159325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:50:25.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:25.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:25.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:25.812 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:25.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:25.944 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:25.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:26.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:26.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:26.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:26.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:26.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:26.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:26.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:26.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:26.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 87113 87097 65.5792 0 - 0.0140007 2024-08-30T23:50:26.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/548400440' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:50:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2912159325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:26 smithi136 ceph-mon[20207]: osdmap e761: 12 total, 12 up, 11 in 2024-08-30T23:50:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:26 smithi136 ceph-mon[20207]: osdmap e762: 12 total, 12 up, 11 in 2024-08-30T23:50:26.547 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:26.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:26.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:26.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/548400440' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:50:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2912159325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:26 smithi173 ceph-mon[20554]: osdmap e761: 12 total, 12 up, 11 in 2024-08-30T23:50:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:26 smithi173 ceph-mon[20554]: osdmap e762: 12 total, 12 up, 11 in 2024-08-30T23:50:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/548400440' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-30T23:50:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2912159325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:50:26.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:26 smithi106 ceph-mon[16603]: osdmap e761: 12 total, 12 up, 11 in 2024-08-30T23:50:26.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:26 smithi106 ceph-mon[16603]: osdmap e762: 12 total, 12 up, 11 in 2024-08-30T23:50:26.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:26.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:26.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:26.821 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:26.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:26.950 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:26.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:26.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:27.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:27.074 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.075 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:27.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:27.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:27.300 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:27.324 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:27.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:27.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:27.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 87113 87097 64.7985 0 - 0.0140007 2024-08-30T23:50:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:27 smithi136 ceph-mon[20207]: pgmap v1698: 37 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 31 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 8.2 MiB/s wr, 131 op/s; 31360/260220 objects misplaced (12.051%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:50:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:27 smithi136 ceph-mon[20207]: osdmap e763: 12 total, 12 up, 11 in 2024-08-30T23:50:27.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:27 smithi173 ceph-mon[20554]: pgmap v1698: 37 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 31 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 8.2 MiB/s wr, 131 op/s; 31360/260220 objects misplaced (12.051%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:50:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:27 smithi173 ceph-mon[20554]: osdmap e763: 12 total, 12 up, 11 in 2024-08-30T23:50:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:27.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:27 smithi106 ceph-mon[16603]: pgmap v1698: 37 pgs: 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 31 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 8.2 MiB/s wr, 131 op/s; 31360/260220 objects misplaced (12.051%); 9.2 MiB/s, 147 objects/s recovering 2024-08-30T23:50:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:27 smithi106 ceph-mon[16603]: osdmap e763: 12 total, 12 up, 11 in 2024-08-30T23:50:27.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:27.704 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:27.806 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:27.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:27.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:27.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:27.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:28.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.078 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:28.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:28.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:28.207 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:28.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:28.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:28.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:28.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:28.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:28.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 87113 87097 64.0362 0 - 0.0140007 2024-08-30T23:50:28.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:28.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:28.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:28.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:50:28.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:28.704 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:28.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:28.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:28.832 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:50:28.832 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:50:28.860 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:50:28.861 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:50:28.886 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:50:28.886 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:50:28.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:28.931 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:28.954 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:28.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:29.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:29.081 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:29.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:29.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:29.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.207 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:29.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:29 smithi173 ceph-mon[20554]: pgmap v1701: 37 pgs: 2 remapped+peering, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 30 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 1.8 MiB/s wr, 28 op/s; 20876/260220 objects misplaced (8.022%); 12 MiB/s, 186 objects/s recovering 2024-08-30T23:50:29.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:29.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:29.460 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:29.502 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:29.502 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:50:29.502 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:50:29.502 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:50:29.502 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:29.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 87113 87097 63.2915 0 - 0.0140007 2024-08-30T23:50:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:29 smithi136 ceph-mon[20207]: pgmap v1701: 37 pgs: 2 remapped+peering, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 30 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 1.8 MiB/s wr, 28 op/s; 20876/260220 objects misplaced (8.022%); 12 MiB/s, 186 objects/s recovering 2024-08-30T23:50:29.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:29.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:29.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:29 smithi106 ceph-mon[16603]: pgmap v1701: 37 pgs: 2 remapped+peering, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 30 active+clean; 5.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 1.8 MiB/s wr, 28 op/s; 20876/260220 objects misplaced (8.022%); 12 MiB/s, 186 objects/s recovering 2024-08-30T23:50:29.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.709 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:29.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:29.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:29.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:29.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:29.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:29.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:30.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:30.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:30.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:30.187 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:50:30.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:30.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:30.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.331 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:30.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:30.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:30.491 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:50:30.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 87113 87097 62.5641 0 - 0.0140007 2024-08-30T23:50:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:30 smithi136 ceph-mon[20207]: osdmap e764: 12 total, 12 up, 11 in 2024-08-30T23:50:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:30.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:30.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:30 smithi173 ceph-mon[20554]: osdmap e764: 12 total, 12 up, 11 in 2024-08-30T23:50:30.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:30.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:30.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:30.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:30 smithi106 ceph-mon[16603]: osdmap e764: 12 total, 12 up, 11 in 2024-08-30T23:50:30.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:30.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:30.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:30.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:30.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:30.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:30.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:30.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:30.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:31.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:31.069 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:50:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:50:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:50:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:50:31.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 502/491 [PushOp(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head, version: 408'2535, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ed1b8264:::benchmark_data_smithi173_87639_object39574:head@408'2535, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 537.86852936800005, 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59550546699999996, 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199194+0000", 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199198+0000", 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5470000000000001e-06 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.199226+0000", 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8228000000000001e-05 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793558+0000", 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59433249099999996 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793576+0000", 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8108999999999999e-05 2024-08-30T23:50:31.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793661+0000", 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5122000000000003e-05 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.793671+0000", 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3889999999999994e-06 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:33.794699+0000", 2024-08-30T23:50:31.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001028581 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 509/491 [PushOp(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head, version: 417'3530, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ebdfd217:::benchmark_data_smithi173_87639_object55953:head@417'3530, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 530.72893608899994, 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42136218600000003, 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.074 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338787+0000", 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338788+0000", 2024-08-30T23:50:31.075 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.33e-07 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.338818+0000", 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9720999999999999e-05 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758562+0000", 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.419743904 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758588+0000", 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5839000000000001e-05 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758731+0000", 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014296099999999999 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.758750+0000", 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9196000000000001e-05 2024-08-30T23:50:31.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:40.760149+0000", 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001399632 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 510/491 [PushOp(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head, version: 418'3647, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ea2f98c7:::benchmark_data_smithi173_87639_object57634:head@418'3647, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 529.93897246699999, 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62881896699999995, 2024-08-30T23:50:31.078 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128751+0000", 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128752+0000", 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2219999999999999e-06 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.128784+0000", 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1445000000000001e-05 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756187+0000", 2024-08-30T23:50:31.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.627403029 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756208+0000", 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1382999999999998e-05 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756330+0000", 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121612 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.756346+0000", 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5962000000000001e-05 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:41.757570+0000", 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001224314 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 511/491 [PushOp(5:ef154109:::benchmark_data_smithi173_87639_object59696:head, version: 421'3773, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:ef154109:::benchmark_data_smithi173_87639_object59696:head@421'3773, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:31.082 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 528.97944705199995, 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.60072882599999999, 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088276+0000", 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088277+0000", 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4600000000000003e-07 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.088304+0000", 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6968999999999999e-05 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687871+0000", 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59956659400000001 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.687891+0000", 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0307999999999999e-05 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688018+0000", 2024-08-30T23:50:31.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012653499999999999 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.688034+0000", 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6728999999999999e-05 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:42.689005+0000", 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097084500000000002 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.086 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPush(5.17 512/491 [PushOp(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head, version: 430'4003, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:eeb72622:::benchmark_data_smithi173_87639_object63098:head@430'4003, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 524.803894378, 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.48614643400000002, 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:50:31.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263829+0000", 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263830+0000", 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8500000000000001e-07 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.263854+0000", 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3934000000000002e-05 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748876+0000", 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.485022907 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748889+0000", 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2785999999999999e-05 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748973+0000", 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.331e-05 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.748983+0000", 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210999999999999e-05 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:41:46.749975+0000", 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099270099999999991 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.090 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(5.d 542/530 [PushReplyOp(5:bce15829:::benchmark_data_smithi173_87639_object23219:head)])", 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 470.250912599, 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44671719199999999, 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816811+0000", 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816812+0000", 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1260000000000001e-06 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:40.816821+0000", 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5510000000000005e-06 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260989+0000", 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.44416764800000003 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.095 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.095 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.260997+0000", 2024-08-30T23:50:31.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5750000000000003e-06 2024-08-30T23:50:31.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.261062+0000", 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5520999999999998e-05 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:41.263528+0000", 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024657709999999998 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99435 5.8 5:154d2bda:::benchmark_data_smithi173_87639_object19195:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 468.70943149300001, 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39149939, 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99435 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358292+0000", 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358293+0000", 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5540000000000001e-06 2024-08-30T23:50:31.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358300+0000", 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4250000000000003e-06 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358312+0000", 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2333e-05 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.358321+0000", 2024-08-30T23:50:31.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4160000000000004e-06 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.383891+0000", 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025570521999999998 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.387506+0000", 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036149160000000001 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.705150+0000", 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31764353000000001 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.707627+0000", 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024771609999999999 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.712526+0000", 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048989070000000001 2024-08-30T23:50:31.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740121+0000", 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027595773000000001 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.740604+0000", 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000482196 2024-08-30T23:50:31.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.741372+0000", 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00076844900000000004 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.743276+0000", 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019037590000000001 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.103 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.749791+0000", 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0065154490000000004 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:99448 5.8 5:1453319e:::benchmark_data_smithi173_87639_object19208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e542)", 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 468.68765747499998, 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.415457309, 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.104 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 99448 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.105 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380066+0000", 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380068+0000", 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.637e-06 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380074+0000", 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3500000000000002e-06 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380092+0000", 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7768999999999999e-05 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.380101+0000", 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6180000000000001e-06 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.389216+0000", 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0091151319999999997 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.392658+0000", 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034414720000000001 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.727246+0000", 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.33458855300000001 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.729727+0000", 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024804380000000002 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.733843+0000", 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041159079999999997 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.750905+0000", 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.017061951999999998 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.752106+0000", 2024-08-30T23:50:31.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001200916 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.789800+0000", 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037693979000000002 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.790989+0000", 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001189563 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:42.795523+0000", 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045340220000000004 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102045 5.8 5:13843233:::benchmark_data_smithi173_87639_object21805:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 463.85646301899999, 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41801196699999998, 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.111 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102045 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211260+0000", 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211262+0000", 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7630000000000001e-06 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211268+0000", 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3489999999999997e-06 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211283+0000", 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4834000000000001e-05 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.211293+0000", 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0037999999999999e-05 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.235799+0000", 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024505741000000001 2024-08-30T23:50:31.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.239265+0000", 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034655129999999999 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.592550+0000", 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35328515199999999 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.594844+0000", 2024-08-30T23:50:31.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002294164 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.599415+0000", 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045711010000000002 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.620491+0000", 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021075930999999999 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.621208+0000", 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00071667200000000001 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.622191+0000", 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098297100000000002 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.623817+0000", 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016259740000000001 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629272+0000", 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054557640000000001 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102049 5.8 5:1445c9f0:::benchmark_data_smithi173_87639_object21809:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:50:31.118 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 463.83344703199998, 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40420456799999999, 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102049 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234276+0000", 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234278+0000", 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.012e-06 2024-08-30T23:50:31.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234286+0000", 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2799999999999998e-06 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234301+0000", 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5590000000000002e-05 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.234310+0000", 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0189999999999995e-06 2024-08-30T23:50:31.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.241629+0000", 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0073184249999999999 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.244833+0000", 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032044450000000002 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.610115+0000", 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.365282208 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.612362+0000", 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022467279999999999 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.616714+0000", 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043520410000000001 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.629561+0000", 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012847305999999999 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.630448+0000", 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00088642099999999998 2024-08-30T23:50:31.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.631622+0000", 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011737430000000001 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.633508+0000", 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018869030000000001 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.638481+0000", 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049724469999999996 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:102205 5.8 5:100735ae:::benchmark_data_smithi173_87639_object21965:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e544)", 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 463.39305292500001, 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41224830200000001, 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.126 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102205 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674670+0000", 2024-08-30T23:50:31.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674673+0000", 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.063e-06 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674679+0000", 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3380000000000001e-06 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.128 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674692+0000", 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3383e-05 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.674702+0000", 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4280000000000002e-06 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.691678+0000", 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016976577 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for rw locks", 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:47.694973+0000", 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032944049999999998 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.051542+0000", 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35656930599999997 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.054103+0000", 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0025610979999999999 2024-08-30T23:50:31.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.057950+0000", 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003847296 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.078442+0000", 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020491453 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.079182+0000", 2024-08-30T23:50:31.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073979099999999995 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.080424+0000", 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00124194 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.081991+0000", 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015673 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:42:48.086919+0000", 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049279240000000002 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:127378 5.8 5:10a9409e:::benchmark_data_smithi173_87639_object47138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 418.777957743, 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41103042400000001, 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.133 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127378 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289766+0000", 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289767+0000", 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.575e-06 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289774+0000", 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8510000000000001e-06 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289788+0000", 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4056999999999999e-05 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.289796+0000", 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5800000000000003e-06 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290170+0000", 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000373829 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290389+0000", 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000219559 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.290617+0000", 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00022764500000000001 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.293291+0000", 2024-08-30T23:50:31.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026742269999999999 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.293692+0000", 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040149700000000001 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700642+0000", 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.40694962699999998 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700700+0000", 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7865000000000002e-05 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:32.700796+0000", 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6112000000000004e-05 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.139 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.26173.0:144420 5.d 5:b9edf439:::benchmark_data_smithi173_87639_object64180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 394.94629081800002, 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.397110827, 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.26173", 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 144420 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121433+0000", 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121434+0000", 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.22e-06 2024-08-30T23:50:31.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121440+0000", 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0909999999999998e-06 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121453+0000", 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2656e-05 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121459+0000", 2024-08-30T23:50:31.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1489999999999999e-06 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121516+0000", 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7333000000000001e-05 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121593+0000", 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7107000000000006e-05 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,7", 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.121655+0000", 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2217000000000004e-05 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125135+0000", 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00347956 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125567+0000", 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00043209299999999998 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518466+0000", 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39289916899999999 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518501+0000", 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4971000000000001e-05 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518543+0000", 2024-08-30T23:50:31.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2261000000000001e-05 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.26173.0:144420 5.d e589/587 ondisk, result = 0)", 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 394.94257024699999, 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39339614299999998, 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125153+0000", 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125154+0000", 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2699999999999999e-07 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.125158+0000", 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9559999999999999e-06 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518369+0000", 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39321148299999997 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518374+0000", 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8439999999999997e-06 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518457+0000", 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2750999999999994e-05 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518462+0000", 2024-08-30T23:50:31.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6069999999999998e-06 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:43:56.518549+0000", 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6774999999999995e-05 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "rep_scrubmap(1.b e543 from shard 1)", 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.904544456, 2024-08-30T23:50:31.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91109331999999998, 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:50:31.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163179+0000", 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7400000000000003e-07 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:20.163183+0000", 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.19e-06 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074076+0000", 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.91089344999999999 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074089+0000", 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3113e-05 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074186+0000", 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7214999999999997e-05 2024-08-30T23:50:31.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074207+0000", 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0372000000000001e-05 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:44:21.074272+0000", 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5406000000000004e-05 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93436 6.d 6.1d3bdb7d (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e653)", 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:01.884570+0000", 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 269.18315303999998, 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99829890499999996, 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93436 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.155 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884570+0000", 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884570+0000", 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884572+0000", 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.406e-06 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884604+0000", 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1897000000000003e-05 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884614+0000", 2024-08-30T23:50:31.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0118999999999999e-05 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884623+0000", 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0680000000000003e-06 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884675+0000", 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1788e-05 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:01.884683+0000", 2024-08-30T23:50:31.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5010000000000004e-06 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.882832+0000", 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99814862000000004 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.882869+0000", 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7506000000000003e-05 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93458 6.d 6.b13a355d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:02.203482+0000", 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 268.86424118000002, 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67964718800000001, 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93458 2024-08-30T23:50:31.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203482+0000", 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203482+0000", 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.161 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203484+0000", 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0389999999999999e-06 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203598+0000", 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011348799999999999 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203613+0000", 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5554e-05 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.162 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203621+0000", 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4359999999999996e-06 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203691+0000", 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0599999999999995e-05 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.203704+0000", 2024-08-30T23:50:31.163 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2490000000000001e-05 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.883120+0000", 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.67941584099999996 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:02.883129+0000", 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7399999999999999e-06 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(6.d 684/654 [PushReplyOp(6:b9e0903d:::benchmark_data_smithi173_114000_object66702:head)])", 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:47:39.972811+0000", 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 171.09491221100001, 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39577864600000001, 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972811+0000", 2024-08-30T23:50:31.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972811+0000", 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972813+0000", 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.48e-06 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:39.972828+0000", 2024-08-30T23:50:31.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5063e-05 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.366857+0000", 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.39402893 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.366871+0000", 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.45e-05 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.167 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:31.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:31.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.367015+0000", 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014346399999999999 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:40.368590+0000", 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015752089999999999 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:157517 6.8 6:14fb8c3a:::benchmark_data_smithi173_114000_object57719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e686)", 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:47:42.803699+0000", 2024-08-30T23:50:31.169 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 168.26402432200001, 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28935165899999998, 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 157517 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803699+0000", 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803699+0000", 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803700+0000", 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3090000000000001e-06 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803706+0000", 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8209999999999998e-06 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803719+0000", 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2679e-05 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803726+0000", 2024-08-30T23:50:31.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8290000000000002e-06 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803776+0000", 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0138e-05 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803864+0000", 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8298000000000001e-05 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:50:31.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.803940+0000", 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5803999999999995e-05 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.807952+0000", 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040120879999999996 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:42.808487+0000", 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053511499999999996 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:43.092946+0000", 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28445894199999999 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:43.093010+0000", 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4331999999999993e-05 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:47:43.093051+0000", 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0303999999999999e-05 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.175 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:193121 6.8 6:15eca3e2:::benchmark_data_smithi173_114000_object93323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e715)", 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:37.131569+0000", 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 113.936154774, 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30851257300000001, 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:50:31.176 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 193121 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131569+0000", 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131569+0000", 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131570+0000", 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1200000000000001e-06 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131575+0000", 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1580000000000004e-06 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131585+0000", 2024-08-30T23:50:31.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0312999999999999e-05 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131593+0000", 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0879999999999995e-06 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131652+0000", 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9120999999999999e-05 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:50:31.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131792+0000", 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013964199999999999 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,7", 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.131931+0000", 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013943199999999999 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.136245+0000", 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043140009999999996 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.136820+0000", 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00057477899999999998 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.440008+0000", 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30318730999999999 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.440044+0000", 2024-08-30T23:50:31.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6702999999999997e-05 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:37.440081+0000", 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6906000000000002e-05 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:50:31.182 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:50:31.269 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:31.291 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:31.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:31.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:31.510 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 87113 87097 61.8531 0 - 0.0140007 2024-08-30T23:50:31.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:31.537 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:31 smithi173 ceph-mon[20554]: pgmap v1703: 37 pgs: 3 remapped+peering, 5 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4.9 MiB/s wr, 78 op/s; 25439/261300 objects misplaced (9.736%); 19 MiB/s, 301 objects/s recovering 2024-08-30T23:50:31.537 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:31 smithi173 ceph-mon[20554]: osdmap e765: 12 total, 12 up, 11 in 2024-08-30T23:50:31.537 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:31 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:31.537 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:31 smithi173 ceph-mon[20554]: osdmap e766: 12 total, 12 up, 11 in 2024-08-30T23:50:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:31 smithi136 ceph-mon[20207]: pgmap v1703: 37 pgs: 3 remapped+peering, 5 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4.9 MiB/s wr, 78 op/s; 25439/261300 objects misplaced (9.736%); 19 MiB/s, 301 objects/s recovering 2024-08-30T23:50:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:31 smithi136 ceph-mon[20207]: osdmap e765: 12 total, 12 up, 11 in 2024-08-30T23:50:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:31 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:31 smithi136 ceph-mon[20207]: osdmap e766: 12 total, 12 up, 11 in 2024-08-30T23:50:31.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:31.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:31 smithi106 ceph-mon[16603]: pgmap v1703: 37 pgs: 3 remapped+peering, 5 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4.9 MiB/s wr, 78 op/s; 25439/261300 objects misplaced (9.736%); 19 MiB/s, 301 objects/s recovering 2024-08-30T23:50:31.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:31 smithi106 ceph-mon[16603]: osdmap e765: 12 total, 12 up, 11 in 2024-08-30T23:50:31.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:31 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:31.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:31 smithi106 ceph-mon[16603]: osdmap e766: 12 total, 12 up, 11 in 2024-08-30T23:50:31.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:31.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:31.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.792 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:31.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:31.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:31.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:31.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:32.018 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:32.040 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:32.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:50:32.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:32.164 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:32.266 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:32.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:32.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:32.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:32.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 87113 87097 61.1581 0 - 0.0140007 2024-08-30T23:50:32.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:32.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:32.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:32.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:32.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:32.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:32.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:32.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:32.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:33.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:33.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:33.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:33.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:33.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:33.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:33.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:33.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:33.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 16 87113 87097 60.4786 0 - 0.0140007 2024-08-30T23:50:33.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:33.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:33.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:33 smithi136 ceph-mon[20207]: pgmap v1706: 37 pgs: 2 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4.7 MiB/s wr, 75 op/s; 30198/261300 objects misplaced (11.557%); 19 MiB/s, 305 objects/s recovering 2024-08-30T23:50:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:33 smithi136 ceph-mon[20207]: osdmap e767: 12 total, 12 up, 11 in 2024-08-30T23:50:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:33 smithi173 ceph-mon[20554]: pgmap v1706: 37 pgs: 2 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4.7 MiB/s wr, 75 op/s; 30198/261300 objects misplaced (11.557%); 19 MiB/s, 305 objects/s recovering 2024-08-30T23:50:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:33 smithi173 ceph-mon[20554]: osdmap e767: 12 total, 12 up, 11 in 2024-08-30T23:50:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:33 smithi106 ceph-mon[16603]: pgmap v1706: 37 pgs: 2 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 4.7 MiB/s wr, 75 op/s; 30198/261300 objects misplaced (11.557%); 19 MiB/s, 305 objects/s recovering 2024-08-30T23:50:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:33 smithi106 ceph-mon[16603]: osdmap e767: 12 total, 12 up, 11 in 2024-08-30T23:50:33.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:33.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:33.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:33.785 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:33.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:33.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:33.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:34.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:34.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:34.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:34.159 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:34.261 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:34.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:34.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:34.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:34.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:34.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 91 16 87113 87097 59.814 0 - 0.0140007 2024-08-30T23:50:34.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:34.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:34 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 5195/261300 objects degraded (1.988%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:50:34.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:34 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 5195/261300 objects degraded (1.988%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:50:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:34 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 5195/261300 objects degraded (1.988%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:50:34.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:34.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:34.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:34.784 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:34.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:34.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:34.909 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:35.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:35.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:35.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:35.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:35.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:35.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:35.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:35.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:35.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:35.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 92 16 87113 87097 59.1638 0 - 0.0140007 2024-08-30T23:50:35.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:35.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:35 smithi136 ceph-mon[20207]: pgmap v1708: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 29907/261300 objects misplaced (11.445%); 14 MiB/s, 218 objects/s recovering 2024-08-30T23:50:35.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:35 smithi173 ceph-mon[20554]: pgmap v1708: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 29907/261300 objects misplaced (11.445%); 14 MiB/s, 218 objects/s recovering 2024-08-30T23:50:35.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:35 smithi106 ceph-mon[16603]: pgmap v1708: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 29907/261300 objects misplaced (11.445%); 14 MiB/s, 218 objects/s recovering 2024-08-30T23:50:35.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:35.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:35.757 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:35.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:35.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:35.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:35.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:36.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:36.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.034 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:36.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:36.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:36.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:36.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:36.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:36.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:36.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:36.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 93 16 87113 87097 58.5276 0 - 0.0140007 2024-08-30T23:50:36.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:36.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:36.662 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:36.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:36.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:36.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:36.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:36.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:37.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:37.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:37.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.161 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:37.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:37.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:37.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:36 smithi136 ceph-mon[20207]: pgmap v1709: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 29393/261300 objects misplaced (11.249%); 16 MiB/s, 248 objects/s recovering 2024-08-30T23:50:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:36 smithi136 ceph-mon[20207]: osdmap e768: 12 total, 12 up, 11 in 2024-08-30T23:50:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:36 smithi173 ceph-mon[20554]: pgmap v1709: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 29393/261300 objects misplaced (11.249%); 16 MiB/s, 248 objects/s recovering 2024-08-30T23:50:37.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:36 smithi173 ceph-mon[20554]: osdmap e768: 12 total, 12 up, 11 in 2024-08-30T23:50:37.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:36 smithi106 ceph-mon[16603]: pgmap v1709: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 29393/261300 objects misplaced (11.249%); 16 MiB/s, 248 objects/s recovering 2024-08-30T23:50:37.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:36 smithi106 ceph-mon[16603]: osdmap e768: 12 total, 12 up, 11 in 2024-08-30T23:50:37.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:37.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:37.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 94 16 87113 87097 57.905 0 - 0.0140007 2024-08-30T23:50:37.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:37.524 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:50:37.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:37.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:37.662 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:37.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:37.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:37.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:37.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:37.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:38.012 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:38.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:38.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:38.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:38.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:38.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:38.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:38.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:38.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 95 16 87113 87097 57.2955 0 - 0.0140007 2024-08-30T23:50:38.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:38.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.540 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:38.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:38.768 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:38.790 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:38.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:38.916 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:38.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:39.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:39.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:39.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:39.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:39.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:39.289 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:38 smithi136 ceph-mon[20207]: pgmap v1711: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 28319/261300 objects misplaced (10.838%); 24 MiB/s, 377 objects/s recovering 2024-08-30T23:50:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:38 smithi173 ceph-mon[20554]: pgmap v1711: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 28319/261300 objects misplaced (10.838%); 24 MiB/s, 377 objects/s recovering 2024-08-30T23:50:39.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:38 smithi106 ceph-mon[16603]: pgmap v1711: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 5195/261300 objects degraded (1.988%); 28319/261300 objects misplaced (10.838%); 24 MiB/s, 377 objects/s recovering 2024-08-30T23:50:39.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:39.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:39.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 96 16 87113 87097 56.6986 0 - 0.0140007 2024-08-30T23:50:39.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:39.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:39.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:39.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:39.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:39.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:39.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:39.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:39.911 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:39.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:40.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:40.035 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:40.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:40.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:40.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:40.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:40.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:39 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10596/261300 objects degraded (4.055%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:40.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:39 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10596/261300 objects degraded (4.055%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:40.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:39 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10596/261300 objects degraded (4.055%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:40.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:40.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:40.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 97 16 87113 87097 56.1141 0 - 0.0140007 2024-08-30T23:50:40.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:40.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:40.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:40.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:40.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:40.782 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:40.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:40.907 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:40.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:41.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:41.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:41.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:41.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:41.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:41.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:40 smithi136 ceph-mon[20207]: pgmap v1712: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 27740/261300 objects misplaced (10.616%); 19 MiB/s, 307 objects/s recovering 2024-08-30T23:50:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:40 smithi136 ceph-mon[20207]: osdmap e769: 12 total, 12 up, 11 in 2024-08-30T23:50:41.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:40 smithi173 ceph-mon[20554]: pgmap v1712: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 27740/261300 objects misplaced (10.616%); 19 MiB/s, 307 objects/s recovering 2024-08-30T23:50:41.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:40 smithi173 ceph-mon[20554]: osdmap e769: 12 total, 12 up, 11 in 2024-08-30T23:50:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:40 smithi106 ceph-mon[16603]: pgmap v1712: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 27740/261300 objects misplaced (10.616%); 19 MiB/s, 307 objects/s recovering 2024-08-30T23:50:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:40 smithi106 ceph-mon[16603]: osdmap e769: 12 total, 12 up, 11 in 2024-08-30T23:50:41.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:41.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:41.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:41.509 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:41.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 98 16 87113 87097 55.5415 0 - 0.0140007 2024-08-30T23:50:41.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:41.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:41.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:41.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:41.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:41.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:41.904 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:41.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:42.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:42.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:42.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:42.153 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:42.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:42.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:42.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:42.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:42.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:42.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 99 16 87113 87097 54.9805 0 - 0.0140007 2024-08-30T23:50:42.527 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:42.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:42.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:42.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:42.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:42.877 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:42.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:42.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:43.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:43.026 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:43.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:43.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:43.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:43.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:43.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:42 smithi136 ceph-mon[20207]: pgmap v1714: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26895/261300 objects misplaced (10.293%); 24 MiB/s, 376 objects/s recovering 2024-08-30T23:50:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:42 smithi173 ceph-mon[20554]: pgmap v1714: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26895/261300 objects misplaced (10.293%); 24 MiB/s, 376 objects/s recovering 2024-08-30T23:50:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:42 smithi106 ceph-mon[16603]: pgmap v1714: 37 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26895/261300 objects misplaced (10.293%); 24 MiB/s, 376 objects/s recovering 2024-08-30T23:50:43.377 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:43.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.401 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:43.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:43.511 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:50:43.512498+0000 min lat: 0.00104498 max lat: 2.61817 avg lat: 0.0140007 2024-08-30T23:50:43.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:50:43.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 100 16 87113 87097 54.4307 0 - 0.0140007 2024-08-30T23:50:43.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:43.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:43.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:43.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:43.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:43.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:43.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:43.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:43.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.020 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:44.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:44.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:44.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:44.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:44.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:44.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:44.391 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:44.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:44.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 101 16 87113 87097 53.8917 0 - 0.0140007 2024-08-30T23:50:44.516 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:44.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:44.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:44.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:44.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:44.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:44.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:44.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:44.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:45.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:45.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:45.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:45.236 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:44 smithi106 ceph-mon[16603]: pgmap v1715: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26561/261300 objects misplaced (10.165%); 22 MiB/s, 353 objects/s recovering 2024-08-30T23:50:45.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:45.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:45.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-08-30T23:50:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:44 smithi136 ceph-mon[20207]: pgmap v1715: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26561/261300 objects misplaced (10.165%); 22 MiB/s, 353 objects/s recovering 2024-08-30T23:50:45.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:44 smithi173 ceph-mon[20554]: pgmap v1715: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26561/261300 objects misplaced (10.165%); 22 MiB/s, 353 objects/s recovering 2024-08-30T23:50:45.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:45.384 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:45.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:45.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:45.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 102 16 87113 87097 53.3634 0 - 0.0140007 2024-08-30T23:50:45.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:45.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:45.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:45.760 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:45.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:45.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:45.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:45.987 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:46.013 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:46.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:46.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:46.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:46.263 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:46.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:46.389 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:46.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:46.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 103 16 87113 87097 52.8453 0 - 0.0140007 2024-08-30T23:50:46.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:46.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:46.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:46.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:46.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:46.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:46.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:46.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:46.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:47.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:47.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:47.135 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:47.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:47.262 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:47.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:46 smithi136 ceph-mon[20207]: pgmap v1716: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26108/261300 objects misplaced (9.992%); 21 MiB/s, 335 objects/s recovering 2024-08-30T23:50:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:46 smithi136 ceph-mon[20207]: osdmap e770: 12 total, 12 up, 11 in 2024-08-30T23:50:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:46 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:46 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:46 smithi173 ceph-mon[20554]: pgmap v1716: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26108/261300 objects misplaced (9.992%); 21 MiB/s, 335 objects/s recovering 2024-08-30T23:50:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:46 smithi173 ceph-mon[20554]: osdmap e770: 12 total, 12 up, 11 in 2024-08-30T23:50:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:46 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:46 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:46 smithi106 ceph-mon[16603]: pgmap v1716: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 26108/261300 objects misplaced (9.992%); 21 MiB/s, 335 objects/s recovering 2024-08-30T23:50:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:46 smithi106 ceph-mon[16603]: osdmap e770: 12 total, 12 up, 11 in 2024-08-30T23:50:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:46 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:47.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:46 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:47.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:47.390 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:47.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:47.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 104 16 87113 87097 52.3372 0 - 0.0140007 2024-08-30T23:50:47.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:47.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:47.638 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:47.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:47.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:47.864 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:47.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:47.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:47.991 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:48.016 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:48.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:48.139 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:48.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:48.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:47 smithi136 ceph-mon[20207]: osdmap e771: 12 total, 12 up, 11 in 2024-08-30T23:50:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:47 smithi173 ceph-mon[20554]: osdmap e771: 12 total, 12 up, 11 in 2024-08-30T23:50:48.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:47 smithi106 ceph-mon[16603]: osdmap e771: 12 total, 12 up, 11 in 2024-08-30T23:50:48.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:48.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:48.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:48.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 105 16 87113 87097 51.8387 0 - 0.0140007 2024-08-30T23:50:48.512 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:48.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:48.639 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:48.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:48.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:48.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:48.889 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:48.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:48.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:49.014 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:49.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:49.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:49.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:49.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:48 smithi136 ceph-mon[20207]: pgmap v1719: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 24950/261300 objects misplaced (9.548%); 26 MiB/s, 410 objects/s recovering 2024-08-30T23:50:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:48 smithi173 ceph-mon[20554]: pgmap v1719: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 24950/261300 objects misplaced (9.548%); 26 MiB/s, 410 objects/s recovering 2024-08-30T23:50:49.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:48 smithi106 ceph-mon[16603]: pgmap v1719: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 24950/261300 objects misplaced (9.548%); 26 MiB/s, 410 objects/s recovering 2024-08-30T23:50:49.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.391 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:49.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:49.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:49.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 105.946 2024-08-30T23:50:49.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 87113 2024-08-30T23:50:49.512 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 51.3899 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 38.0916 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 144.312 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 822 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 609.465 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2309 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0194499 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.404538 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 32.3222 2024-08-30T23:50:49.513 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.00104498 2024-08-30T23:50:49.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:49.519 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:50:49.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:49.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:49.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:49.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:49.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:49.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:49.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:49.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:50.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:50.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:50.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:50.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:50.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:50.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:50.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:50.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:50.535 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:50.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:50.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:50.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:50.787 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:50.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:50.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:50.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:51.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:51.036 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:51.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:51.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:51.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:50 smithi136 ceph-mon[20207]: pgmap v1720: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 24327/261300 objects misplaced (9.310%); 20 MiB/s, 320 objects/s recovering 2024-08-30T23:50:51.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:50 smithi173 ceph-mon[20554]: pgmap v1720: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 24327/261300 objects misplaced (9.310%); 20 MiB/s, 320 objects/s recovering 2024-08-30T23:50:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:50 smithi106 ceph-mon[16603]: pgmap v1720: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 10596/261300 objects degraded (4.055%); 24327/261300 objects misplaced (9.310%); 20 MiB/s, 320 objects/s recovering 2024-08-30T23:50:51.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:51.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:51.544 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:51.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:51.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:51.704 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:51.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:51.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:51.833 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:51.870 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:51.870 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:50:51.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:51.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:51.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:52.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:50:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:51 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:51 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:51 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:51 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:52.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:52.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:52.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:50:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:51 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:51 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:51 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:51 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:51 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:50:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:52 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:52 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:52 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:52 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:52.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:52.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:52.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:52.527 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:52.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:52.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:52.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:52.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:52.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:52.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:52.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:53.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:53.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:53.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.196 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:53.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:53.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:53.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:53.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:53.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:53.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:53 smithi106 ceph-mon[16603]: pgmap v1721: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 50 op/s; 10596/260085 objects degraded (4.074%); 23484/260085 objects misplaced (9.029%); 24 MiB/s, 384 objects/s recovering 2024-08-30T23:50:53.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:53 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:53.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:53 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:53.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:53 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10596/260085 objects degraded (4.074%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:53 smithi136 ceph-mon[20207]: pgmap v1721: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 50 op/s; 10596/260085 objects degraded (4.074%); 23484/260085 objects misplaced (9.029%); 24 MiB/s, 384 objects/s recovering 2024-08-30T23:50:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:53 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:53 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:53 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10596/260085 objects degraded (4.074%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:53.550 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:53 smithi173 ceph-mon[20554]: pgmap v1721: 37 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 50 op/s; 10596/260085 objects degraded (4.074%); 23484/260085 objects misplaced (9.029%); 24 MiB/s, 384 objects/s recovering 2024-08-30T23:50:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:53 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:53 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:53 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10596/260085 objects degraded (4.074%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:53.585 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:53.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:53.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:53.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:53.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:53.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:50:53.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:53.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:53.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:54.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:54.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:54.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:54.221 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:54.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:54.368 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:54.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:54.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:54.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:54.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:54.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:54.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:54.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:54.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:54.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:54.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:54.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:55.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:55.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.136 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:55.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:55.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:55.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:55.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:55.395 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:55.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.539 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:55.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: pgmap v1722: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 10596/259254 objects degraded (4.087%); 23176/259254 objects misplaced (8.939%); 23 MiB/s, 366 objects/s recovering 2024-08-30T23:50:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: osdmap e772: 12 total, 12 up, 11 in 2024-08-30T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:50:55.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:55 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: pgmap v1722: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 10596/259254 objects degraded (4.087%); 23176/259254 objects misplaced (8.939%); 23 MiB/s, 366 objects/s recovering 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: osdmap e772: 12 total, 12 up, 11 in 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:50:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:55 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: pgmap v1722: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 10596/259254 objects degraded (4.087%); 23176/259254 objects misplaced (8.939%); 23 MiB/s, 366 objects/s recovering 2024-08-30T23:50:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: osdmap e772: 12 total, 12 up, 11 in 2024-08-30T23:50:55.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:55.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:55.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:50:55.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:50:55.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:55 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:50:55.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:55.678 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:55.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:55.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:55.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:55.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:55.930 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:55.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:56.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:50:56.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.056 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:50:56.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:56.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:56.282 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:56.306 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:50:56.408 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:56.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:56.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:56.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:56 smithi136 ceph-mon[20207]: osdmap e773: 12 total, 12 up, 11 in 2024-08-30T23:50:56.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:56 smithi173 ceph-mon[20554]: osdmap e773: 12 total, 12 up, 11 in 2024-08-30T23:50:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:56 smithi106 ceph-mon[16603]: osdmap e773: 12 total, 12 up, 11 in 2024-08-30T23:50:56.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:56.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:56.790 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:56.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:56.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:56.951 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:56.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:57.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:50:57.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:50:57.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:57.196 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:50:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:57.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:57.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:50:57.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:50:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:57 smithi106 ceph-mon[16603]: pgmap v1725: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 120 KiB/s wr, 233 op/s; 10594/255786 objects degraded (4.142%); 22855/255786 objects misplaced (8.935%); 16 MiB/s, 261 objects/s recovering 2024-08-30T23:50:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:50:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:50:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1803636601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:50:57.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:57.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.478 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:57 smithi136 ceph-mon[20207]: pgmap v1725: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 120 KiB/s wr, 233 op/s; 10594/255786 objects degraded (4.142%); 22855/255786 objects misplaced (8.935%); 16 MiB/s, 261 objects/s recovering 2024-08-30T23:50:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:50:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:50:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1803636601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:50:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:57 smithi173 ceph-mon[20554]: pgmap v1725: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 127 B/s rd, 120 KiB/s wr, 233 op/s; 10594/255786 objects degraded (4.142%); 22855/255786 objects misplaced (8.935%); 16 MiB/s, 261 objects/s recovering 2024-08-30T23:50:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:50:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:50:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1803636601' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:50:57.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:57.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:57.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:57.756 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:57.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:57.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:57.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:57.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:58.007 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:58.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:58.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:58.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:58.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:58.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:50:58.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:50:58.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.525 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:58.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:58 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1803636601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:50:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:58 smithi136 ceph-mon[20207]: osdmap e774: 12 total, 12 up, 11 in 2024-08-30T23:50:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:58 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1803636601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:50:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:58 smithi173 ceph-mon[20554]: osdmap e774: 12 total, 12 up, 11 in 2024-08-30T23:50:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:58 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:50:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1803636601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:50:58.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:58 smithi106 ceph-mon[16603]: osdmap e774: 12 total, 12 up, 11 in 2024-08-30T23:50:58.626 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:50:58.650 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:50:58.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:58.793 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:58.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:58.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:58.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:59.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:50:59.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:50:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:50:59.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:50:59.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:50:59.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:59.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:59.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:59.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:50:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:59.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:50:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:59 smithi136 ceph-mon[20207]: pgmap v1727: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 170 B/s rd, 160 KiB/s wr, 513 op/s; 10594/250938 objects degraded (4.222%); 21996/250938 objects misplaced (8.766%); 16 MiB/s, 248 objects/s recovering 2024-08-30T23:50:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:50:59 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10594/250938 objects degraded (4.222%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:59.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:50:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:59 smithi173 ceph-mon[20554]: pgmap v1727: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 170 B/s rd, 160 KiB/s wr, 513 op/s; 10594/250938 objects degraded (4.222%); 21996/250938 objects misplaced (8.766%); 16 MiB/s, 248 objects/s recovering 2024-08-30T23:50:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:50:59 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10594/250938 objects degraded (4.222%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:59.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:50:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:59 smithi106 ceph-mon[16603]: pgmap v1727: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 170 B/s rd, 160 KiB/s wr, 513 op/s; 10594/250938 objects degraded (4.222%); 21996/250938 objects misplaced (8.766%); 16 MiB/s, 248 objects/s recovering 2024-08-30T23:50:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:50:59 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10594/250938 objects degraded (4.222%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:50:59.698 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:50:59.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:50:59.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:50:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:50:59.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:50:59.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:50:59.993 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:50:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:50:59.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:00.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:00.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:00.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:00.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:00.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:00.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:00.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:00.505 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:00.607 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:00.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.631 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:00.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:00.768 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:00.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:00.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:00.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:00.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:01.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:01.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:01.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:01.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:01.278 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:01.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:01.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:01.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:01.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:01.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:01 smithi136 ceph-mon[20207]: pgmap v1728: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 170 B/s rd, 160 KiB/s wr, 698 op/s; 10594/246780 objects degraded (4.293%); 21723/246780 objects misplaced (8.803%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:51:01.540 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:01 smithi106 ceph-mon[16603]: pgmap v1728: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 170 B/s rd, 160 KiB/s wr, 698 op/s; 10594/246780 objects degraded (4.293%); 21723/246780 objects misplaced (8.803%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:51:01.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:01 smithi173 ceph-mon[20554]: pgmap v1728: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 5.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 170 B/s rd, 160 KiB/s wr, 698 op/s; 10594/246780 objects degraded (4.293%); 21723/246780 objects misplaced (8.803%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:51:01.632 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:01.668 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:01.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:01.805 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:01.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:01.942 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:01.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:02.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:02.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:02.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:02.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:02.230 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:02.230 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:94887 7.b 7:d7ceb5da:::benchmark_data_smithi173_129296_object7771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e775)", 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:02.234680+0000", 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.034819551999999997, 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034849685999999998, 2024-08-30T23:51:02.271 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 94887 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.234680+0000", 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:02.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.234680+0000", 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.234682+0000", 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0899999999999999e-06 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.234690+0000", 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3550000000000005e-06 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.234708+0000", 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7829000000000002e-05 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.234718+0000", 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0169999999999999e-05 2024-08-30T23:51:02.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.245139+0000", 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010420617 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.245517+0000", 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000378291 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,10", 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.245946+0000", 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042907999999999998 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27487.0:94887 7.b e775/756)", 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:02.248979+0000", 2024-08-30T23:51:02.276 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.020520432000000002, 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.020615943000000001, 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.248979+0000", 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:02.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.248979+0000", 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.248980+0000", 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.206e-06 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.248989+0000", 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4800000000000001e-06 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.249032+0000", 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3223000000000003e-05 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.249043+0000", 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0564000000000001e-05 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27487.0:94887 7.b e775/756)", 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:02.253962+0000", 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015537367999999999, 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015670555999999999, 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.253962+0000", 2024-08-30T23:51:02.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.253962+0000", 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.253964+0000", 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5099999999999999e-06 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.253975+0000", 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0873000000000001e-05 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.254047+0000", 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2193000000000005e-05 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:02.254056+0000", 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5289999999999997e-06 2024-08-30T23:51:02.282 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.283 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:02.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:02.283 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:51:02.283 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 3 2024-08-30T23:51:02.283 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:51:02.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:02.336 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:02.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:02.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:02 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:02 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:02.564 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:02 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:02 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:02.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:02 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:02 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:02.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:02.728 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:02.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:02.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:02.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:02.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:02.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:03.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.115 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:03.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:03.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.240 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:03.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:03.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:03.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:03.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:03.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:03 smithi136 ceph-mon[20207]: pgmap v1729: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 135 B/s rd, 127 KiB/s wr, 699 op/s; 10594/243042 objects degraded (4.359%); 21127/243042 objects misplaced (8.693%); 16 MiB/s, 252 objects/s recovering 2024-08-30T23:51:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:03 smithi136 ceph-mon[20207]: osdmap e775: 12 total, 12 up, 11 in 2024-08-30T23:51:03.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:03 smithi173 ceph-mon[20554]: pgmap v1729: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 135 B/s rd, 127 KiB/s wr, 699 op/s; 10594/243042 objects degraded (4.359%); 21127/243042 objects misplaced (8.693%); 16 MiB/s, 252 objects/s recovering 2024-08-30T23:51:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:03 smithi173 ceph-mon[20554]: osdmap e775: 12 total, 12 up, 11 in 2024-08-30T23:51:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:03 smithi106 ceph-mon[16603]: pgmap v1729: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 135 B/s rd, 127 KiB/s wr, 699 op/s; 10594/243042 objects degraded (4.359%); 21127/243042 objects misplaced (8.693%); 16 MiB/s, 252 objects/s recovering 2024-08-30T23:51:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:03 smithi106 ceph-mon[16603]: osdmap e775: 12 total, 12 up, 11 in 2024-08-30T23:51:03.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:03.645 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:03.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:03.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:03.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:03.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:03.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:04.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:51:04.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:04.028 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:04.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:04.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:04.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:04.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:04 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10594/241227 objects degraded (4.392%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:04.379 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:04.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:04.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:04.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:04 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10594/241227 objects degraded (4.392%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:04.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:04.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:04 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10594/241227 objects degraded (4.392%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:04.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:04.683 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:04.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:04.808 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:04.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:04.935 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:04.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:05.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:05.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:05.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:05.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:05.286 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:05.310 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:05.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:05.435 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:05.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:05 smithi136 ceph-mon[20207]: pgmap v1731: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 10594/241227 objects degraded (4.392%); 20872/241227 objects misplaced (8.652%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:51:05.561 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:05 smithi173 ceph-mon[20554]: pgmap v1731: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 10594/241227 objects degraded (4.392%); 20872/241227 objects misplaced (8.652%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:51:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:05 smithi106 ceph-mon[16603]: pgmap v1731: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 10594/241227 objects degraded (4.392%); 20872/241227 objects misplaced (8.652%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:51:05.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:05.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:05.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:05.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:05.927 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:05.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:05.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:06.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.093 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:06.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:06.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:06.236 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:06.338 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:06.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:06.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:06.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:06.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:06.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:06 smithi136 ceph-mon[20207]: osdmap e776: 12 total, 12 up, 11 in 2024-08-30T23:51:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:06 smithi173 ceph-mon[20554]: osdmap e776: 12 total, 12 up, 11 in 2024-08-30T23:51:06.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:06.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:06.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:06 smithi106 ceph-mon[16603]: osdmap e776: 12 total, 12 up, 11 in 2024-08-30T23:51:06.617 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:06.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:06.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:06.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:06.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:06.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:06.923 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:06.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:07.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:07.066 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:07.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:07.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:07.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:07.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:07.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:07.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:07.555 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:07.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:07.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:07.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:07 smithi136 ceph-mon[20207]: pgmap v1733: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 10594/237456 objects degraded (4.461%); 20563/237456 objects misplaced (8.660%); 11 MiB/s, 179 objects/s recovering 2024-08-30T23:51:07.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:07.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:07 smithi173 ceph-mon[20554]: pgmap v1733: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 10594/237456 objects degraded (4.461%); 20563/237456 objects misplaced (8.660%); 11 MiB/s, 179 objects/s recovering 2024-08-30T23:51:07.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:07 smithi106 ceph-mon[16603]: pgmap v1733: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 10594/237456 objects degraded (4.461%); 20563/237456 objects misplaced (8.660%); 11 MiB/s, 179 objects/s recovering 2024-08-30T23:51:07.862 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:07.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:51:07.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:07.985 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:07.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:08.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:08.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:08.219 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:08.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:08.250 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:51:08.250 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:51:08.264 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:08.291 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:51:08.291 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:51:08.334 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:51:08.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:51:08.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:08.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:51:08.378 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:51:08.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.407 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:08.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:08.410 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:51:08.411 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:51:08.446 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:51:08.446 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:51:08.484 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:51:08.485 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:51:08.495 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.1 to 1 (10000) 2024-08-30T23:51:08.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:08.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:08.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:08.687 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:08.721 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:51:08.721 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:51:08.721 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2316098952' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:51:08.722 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:08 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:51:08.722 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/518457314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:08.756 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:51:08.757 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:51:08.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2316098952' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:51:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:08 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:51:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/518457314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:08.793 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:51:08.793 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:51:08.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2316098952' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:51:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:08 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-30T23:51:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/518457314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:08.828 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:51:08.832 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:51:08.879 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:51:08.879 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:51:08.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:08.975 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:08.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:09.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:09.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:09.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:09.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:09.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:09.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:09.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:09.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:09.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:09.514 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:51:09.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:09.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:09 smithi106 ceph-mon[16603]: pgmap v1734: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 10594/232434 objects degraded (4.558%); 19654/232434 objects misplaced (8.456%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:51:09.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:09 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:51:09.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:09 smithi106 ceph-mon[16603]: osdmap e777: 12 total, 12 up, 11 in 2024-08-30T23:51:09.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3831836079' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:09.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:09 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:09.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:09 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:51:09.617 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:09.719 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.756 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:09.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:09 smithi136 ceph-mon[20207]: pgmap v1734: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 10594/232434 objects degraded (4.558%); 19654/232434 objects misplaced (8.456%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:51:09.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:09 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:51:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:09 smithi136 ceph-mon[20207]: osdmap e777: 12 total, 12 up, 11 in 2024-08-30T23:51:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3831836079' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:09 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:09 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:51:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:09 smithi173 ceph-mon[20554]: pgmap v1734: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 10594/232434 objects degraded (4.558%); 19654/232434 objects misplaced (8.456%); 16 MiB/s, 258 objects/s recovering 2024-08-30T23:51:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:09 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-30T23:51:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:09 smithi173 ceph-mon[20554]: osdmap e777: 12 total, 12 up, 11 in 2024-08-30T23:51:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3831836079' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:09 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:09 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:51:09.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:09.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:09.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:10.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:10.040 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:10.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:10.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:10.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:10.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:10.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:10.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:10.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:10.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:10.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:10 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:51:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:10 smithi106 ceph-mon[16603]: osdmap e778: 12 total, 12 up, 11 in 2024-08-30T23:51:10.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:10.668 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:10.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:10.782 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:10 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:51:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:10 smithi136 ceph-mon[20207]: osdmap e778: 12 total, 12 up, 11 in 2024-08-30T23:51:10.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:10.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:10 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:51:10.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:10 smithi173 ceph-mon[20554]: osdmap e778: 12 total, 12 up, 11 in 2024-08-30T23:51:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:10.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-08-30T23:51:10.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:10.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:10.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:11.052 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:11.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:11.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:11.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.200 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:11.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.353 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:11.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:11.398 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:51:11.398 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:51:11.412 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:11.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:11.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:11.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:11.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.624 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:11.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:11.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:11 smithi136 ceph-mon[20207]: pgmap v1737: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 10593/228420 objects degraded (4.638%); 19366/228420 objects misplaced (8.478%); 16 MiB/s, 251 objects/s recovering 2024-08-30T23:51:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10593/228420 objects degraded (4.638%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:11 smithi173 ceph-mon[20554]: pgmap v1737: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 10593/228420 objects degraded (4.638%); 19366/228420 objects misplaced (8.478%); 16 MiB/s, 251 objects/s recovering 2024-08-30T23:51:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10593/228420 objects degraded (4.638%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:11.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:11 smithi106 ceph-mon[16603]: pgmap v1737: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 10593/228420 objects degraded (4.638%); 19366/228420 objects misplaced (8.478%); 16 MiB/s, 251 objects/s recovering 2024-08-30T23:51:11.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10593/228420 objects degraded (4.638%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:11.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:11.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:11.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:12.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:12.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:12.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.127 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:12.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:12.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:12.353 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.388 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:12.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:12.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:51:12.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:12.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:12.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:12.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:12.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:12.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:12.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:12.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:12.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:51:12.898 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:12.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:13.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:13.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:13.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:13.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:13.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:13.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:13.375 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:13.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:13.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:13.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:13.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:13.661 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:13.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:13.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:13 smithi136 ceph-mon[20207]: pgmap v1738: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 10593/225084 objects degraded (4.706%); 18743/225084 objects misplaced (8.327%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:51:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:13 smithi173 ceph-mon[20554]: pgmap v1738: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 10593/225084 objects degraded (4.706%); 18743/225084 objects misplaced (8.327%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:51:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:13 smithi106 ceph-mon[16603]: pgmap v1738: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 661 op/s; 10593/225084 objects degraded (4.706%); 18743/225084 objects misplaced (8.327%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:51:13.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:13.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:13.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:14.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:14.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:14.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:14.174 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:14.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:14.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:14.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:14.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:14.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:14.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:14.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:14.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:14.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:14.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:14.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:14.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:14.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:15.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:15.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:15.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:15.225 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:15.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:15.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:15.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:15.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.481 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:15.582 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:15.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:51:15.625 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:15.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.763 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:15.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:15 smithi136 ceph-mon[20207]: pgmap v1739: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 10593/223413 objects degraded (4.741%); 18475/223413 objects misplaced (8.269%); 16 MiB/s, 260 objects/s recovering 2024-08-30T23:51:15.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:15 smithi106 ceph-mon[16603]: pgmap v1739: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 10593/223413 objects degraded (4.741%); 18475/223413 objects misplaced (8.269%); 16 MiB/s, 260 objects/s recovering 2024-08-30T23:51:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:15 smithi173 ceph-mon[20554]: pgmap v1739: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 584 op/s; 10593/223413 objects degraded (4.741%); 18475/223413 objects misplaced (8.269%); 16 MiB/s, 260 objects/s recovering 2024-08-30T23:51:15.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:15.886 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:15.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:15.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:16.032 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:16.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:16.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:16.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:16.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:16.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:16.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:16.432 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:16.433 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:16.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:16.466 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:16.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:16.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:16.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:16.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:16.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:16 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:16 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10593/223413 objects degraded (4.741%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:16.900 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:16.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:16.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:16 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:16.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:16 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:16.901 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10593/223413 objects degraded (4.741%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:16.901 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:16 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:16.902 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:16 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:16.902 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10593/223413 objects degraded (4.741%), 2 pgs degraded (PG_DEGRADED) 2024-08-30T23:51:17.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.036 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:17.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:17.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:17.160 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:17.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:17.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:17.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:17.423 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:17.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:17.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:17.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:17.552 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.2 to 1 (10000) 2024-08-30T23:51:17.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:17.674 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:17.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:17.812 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:17 smithi173 ceph-mon[20554]: pgmap v1740: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 10592/219477 objects degraded (4.826%); 18168/219477 objects misplaced (8.278%); 12 MiB/s, 185 objects/s recovering 2024-08-30T23:51:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:17 smithi173 ceph-mon[20554]: osdmap e779: 12 total, 12 up, 11 in 2024-08-30T23:51:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2634823328' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:51:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:17 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:51:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:17 smithi106 ceph-mon[16603]: pgmap v1740: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 10592/219477 objects degraded (4.826%); 18168/219477 objects misplaced (8.278%); 12 MiB/s, 185 objects/s recovering 2024-08-30T23:51:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:17 smithi106 ceph-mon[16603]: osdmap e779: 12 total, 12 up, 11 in 2024-08-30T23:51:17.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2634823328' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:51:17.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:17 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:51:17.914 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:17.942 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:17.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:17 smithi136 ceph-mon[20207]: pgmap v1740: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 539 op/s; 10592/219477 objects degraded (4.826%); 18168/219477 objects misplaced (8.278%); 12 MiB/s, 185 objects/s recovering 2024-08-30T23:51:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:17 smithi136 ceph-mon[20207]: osdmap e779: 12 total, 12 up, 11 in 2024-08-30T23:51:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2634823328' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:51:18.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:17 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-30T23:51:18.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:18.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:18.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:18.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:18.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:18.317 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:18.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:18.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:18.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:18.591 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:18.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:18.710 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:18 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:51:18.710 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:18 smithi106 ceph-mon[16603]: osdmap e780: 12 total, 12 up, 11 in 2024-08-30T23:51:18.710 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/965944483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:51:18.710 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:18 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:51:18.710 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/340829826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:18.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:18 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:51:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:18 smithi173 ceph-mon[20554]: osdmap e780: 12 total, 12 up, 11 in 2024-08-30T23:51:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/965944483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:51:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:18 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:51:18.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/340829826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:18.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.863 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:18.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:18.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:18.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:18.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:18 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-30T23:51:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:18 smithi136 ceph-mon[20207]: osdmap e780: 12 total, 12 up, 11 in 2024-08-30T23:51:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/965944483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:51:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:18 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:51:19.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/340829826' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:19.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.121 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:19.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:19.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:19.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:19.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:19.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:19.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:19.517 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:19.619 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:19.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:19.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:19.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:19 smithi106 ceph-mon[16603]: pgmap v1743: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 10589/214806 objects degraded (4.930%); 17135/214806 objects misplaced (7.977%); 17 MiB/s, 279 objects/s recovering 2024-08-30T23:51:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:19 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:19.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:19 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:51:19.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:19 smithi106 ceph-mon[16603]: osdmap e781: 12 total, 12 up, 11 in 2024-08-30T23:51:19.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:19.900 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:19.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:20.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:20.024 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:20.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:19 smithi136 ceph-mon[20207]: pgmap v1743: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 10589/214806 objects degraded (4.930%); 17135/214806 objects misplaced (7.977%); 17 MiB/s, 279 objects/s recovering 2024-08-30T23:51:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:19 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:19 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:51:20.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:19 smithi136 ceph-mon[20207]: osdmap e781: 12 total, 12 up, 11 in 2024-08-30T23:51:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:19 smithi173 ceph-mon[20554]: pgmap v1743: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 567 op/s; 10589/214806 objects degraded (4.930%); 17135/214806 objects misplaced (7.977%); 17 MiB/s, 279 objects/s recovering 2024-08-30T23:51:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:19 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:19 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:51:20.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:19 smithi173 ceph-mon[20554]: osdmap e781: 12 total, 12 up, 11 in 2024-08-30T23:51:20.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:20.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:20.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:20.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:20.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:20.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:20.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:20.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:20.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:20.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:20.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:20.790 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:20.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:20.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:20.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:21.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:21.043 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:21.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:21.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:21.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:21.295 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:21.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:21.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:21.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:21.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:21.586 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:51:21.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:21.690 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:21.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:21.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:21.836 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:21 smithi106 ceph-mon[16603]: pgmap v1745: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 671 op/s; 10587/211326 objects degraded (5.010%); 16816/211326 objects misplaced (7.957%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:51:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10587/211326 objects degraded (5.010%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:21.875 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:21 smithi173 ceph-mon[20554]: pgmap v1745: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 671 op/s; 10587/211326 objects degraded (5.010%); 16816/211326 objects misplaced (7.957%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:51:21.875 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10587/211326 objects degraded (5.010%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:21.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:21.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:21.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:21 smithi136 ceph-mon[20207]: pgmap v1745: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 671 op/s; 10587/211326 objects degraded (5.010%); 16816/211326 objects misplaced (7.957%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:51:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10587/211326 objects degraded (5.010%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:22.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:22.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:22.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:22.235 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:22.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:22.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:22.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:22.511 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:22.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:22.635 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:22.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:22.759 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:22.799 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:51:22.799 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:51:22.799 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:51:22.799 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:51:22.799 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86177 5.7 5.7fdfa7e7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76457369100001, 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97022776200000005, 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86177 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.800 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731724+0000", 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.801 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731726+0000", 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.736e-06 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731731+0000", 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1209999999999998e-06 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731739+0000", 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6569999999999997e-06 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.802 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.731746+0000", 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732139+0000", 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039219999999999999 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732147+0000", 2024-08-30T23:51:22.803 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8910000000000001e-06 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701940+0000", 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979272699999997 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701952+0000", 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1629e-05 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.804 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86218 5.7 5.665a2e07 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76427919000002, 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96998314699999999, 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.805 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86218 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.806 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732019+0000", 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.82e-07 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732023+0000", 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8060000000000001e-06 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732043+0000", 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9873000000000001e-05 2024-08-30T23:51:22.807 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.808 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.808 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.808 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732050+0000", 2024-08-30T23:51:22.808 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4950000000000002e-06 2024-08-30T23:51:22.808 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732187+0000", 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013689799999999999 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.810 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732194+0000", 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6039999999999998e-06 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.701991+0000", 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979669899999998 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702002+0000", 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.119e-05 2024-08-30T23:51:22.811 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86288 5.7 5.4dc78ea7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76423865100003, 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96997772299999996, 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.812 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86288 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732059+0000", 2024-08-30T23:51:22.813 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732060+0000", 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9799999999999999e-07 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.814 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732062+0000", 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2019999999999998e-06 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732068+0000", 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6710000000000002e-06 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732072+0000", 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9770000000000002e-06 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.815 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732233+0000", 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016027700000000001 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732240+0000", 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9990000000000002e-06 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702030+0000", 2024-08-30T23:51:22.816 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96979006599999995 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702037+0000", 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1330000000000001e-06 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86300 5.7 5.f6fa46a7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:51:22.817 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76422043299999, 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97000272399999998, 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86300 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:51:22.818 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732077+0000", 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732078+0000", 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4400000000000001e-07 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.819 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732079+0000", 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4839999999999999e-06 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732083+0000", 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2819999999999998e-06 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.820 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732094+0000", 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.075e-05 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732281+0000", 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000186517 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732286+0000", 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.553e-06 2024-08-30T23:51:22.821 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702067+0000", 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96978114199999998 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702080+0000", 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2751999999999999e-05 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.822 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86325 5.7 5.e2384ca7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76419671300005, 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97003113200000002, 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86325 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.823 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732101+0000", 2024-08-30T23:51:22.824 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732102+0000", 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5499999999999998e-07 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.825 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732106+0000", 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4460000000000003e-06 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732113+0000", 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7989999999999996e-06 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732119+0000", 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4080000000000003e-06 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.826 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732342+0000", 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00022250899999999999 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732347+0000", 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7969999999999997e-06 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702121+0000", 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96977379200000002 2024-08-30T23:51:22.827 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702132+0000", 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0926e-05 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86376 5.7 5.ad230c87 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76417205200005, 2024-08-30T23:51:22.828 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97006071599999999, 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86376 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:51:22.829 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732126+0000", 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4799999999999999e-07 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.830 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732129+0000", 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6529999999999998e-06 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732135+0000", 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2210000000000002e-06 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732139+0000", 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.211e-06 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.831 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732389+0000", 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00024981999999999999 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732396+0000", 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1189999999999999e-06 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702177+0000", 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96978100300000003 2024-08-30T23:51:22.832 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702186+0000", 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2410000000000001e-06 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.833 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86452 5.7 5.ed9357e7 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76414432399997, 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97006287000000002, 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.834 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86452 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732153+0000", 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.835 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732154+0000", 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4200000000000002e-07 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732162+0000", 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6659999999999997e-06 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.836 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732169+0000", 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6839999999999999e-06 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732173+0000", 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1049999999999997e-06 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732425+0000", 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000251508 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.837 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732428+0000", 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.202e-06 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702206+0000", 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96977835099999998 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702216+0000", 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0012e-05 2024-08-30T23:51:22.838 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86465 5.7 5.831aa467 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76411913699997, 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97009033300000003, 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.839 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86465 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.840 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732179+0000", 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6199999999999999e-07 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732180+0000", 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1510000000000001e-06 2024-08-30T23:51:22.841 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732183+0000", 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3289999999999998e-06 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732189+0000", 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1149999999999999e-06 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.842 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732442+0000", 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025276900000000002 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732445+0000", 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.638e-06 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702257+0000", 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96981182700000002 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.843 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.844 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702269+0000", 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2242000000000001e-05 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86565 5.7 5.628bf187 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76410480599998, 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97012332999999995, 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.845 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86565 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.846 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732193+0000", 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2499999999999999e-07 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732194+0000", 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.06e-06 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732198+0000", 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2650000000000001e-06 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732202+0000", 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4150000000000001e-06 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732476+0000", 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00027440400000000001 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732482+0000", 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1539999999999999e-06 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702307+0000", 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982474900000004 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702316+0000", 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0580000000000004e-06 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86603 5.7 5.32d9a587 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.764090076, 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97014088799999998, 2024-08-30T23:51:22.850 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86603 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732208+0000", 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4299999999999999e-07 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732210+0000", 2024-08-30T23:51:22.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.655e-06 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732215+0000", 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6910000000000002e-06 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732220+0000", 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6489999999999999e-06 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732509+0000", 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028865599999999998 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732511+0000", 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7049999999999999e-06 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702338+0000", 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.969826246 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702349+0000", 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0943e-05 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86625 5.7 5.9d66eb87 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76407176500004, 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97015716600000002, 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86625 2024-08-30T23:51:22.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732226+0000", 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5199999999999998e-07 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732228+0000", 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0389999999999999e-06 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732234+0000", 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3979999999999998e-06 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732239+0000", 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3029999999999999e-06 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732541+0000", 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000301582 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732548+0000", 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8709999999999999e-06 2024-08-30T23:51:22.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702376+0000", 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982850899999995 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702383+0000", 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1119999999999998e-06 2024-08-30T23:51:22.860 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86638 5.7 5.ef24e247 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76405278799996, 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97016687199999996, 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.861 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86638 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732245+0000", 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8000000000000001e-07 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732247+0000", 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8500000000000001e-06 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732253+0000", 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5600000000000001e-06 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732257+0000", 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4939999999999997e-06 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732573+0000", 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00031563199999999999 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732576+0000", 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6120000000000001e-06 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702405+0000", 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982966100000001 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702412+0000", 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6830000000000002e-06 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86682 5.7 5.ad2d3f47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76402430600001, 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97017973300000004, 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.866 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86682 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732273+0000", 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732274+0000", 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2000000000000001e-07 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732275+0000", 2024-08-30T23:51:22.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.257e-06 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732279+0000", 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7469999999999999e-06 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732282+0000", 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1089999999999998e-06 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732606+0000", 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032443800000000003 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732613+0000", 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3810000000000004e-06 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702433+0000", 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96982075199999995 2024-08-30T23:51:22.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702453+0000", 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9729e-05 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86686 5.7 5.c68ec547 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76401240200005, 2024-08-30T23:51:22.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97021215299999997, 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86686 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:51:22.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732285+0000", 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732286+0000", 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.28e-07 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732287+0000", 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.065e-06 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732290+0000", 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5499999999999999e-06 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732294+0000", 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2649999999999998e-06 2024-08-30T23:51:22.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732637+0000", 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034266199999999998 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732640+0000", 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5790000000000002e-06 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702487+0000", 2024-08-30T23:51:22.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96984751400000002 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702498+0000", 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.029e-05 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86696 5.7 5.36b4ca47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:51:22.876 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76399891699998, 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.970319865, 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86696 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:51:22.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732299+0000", 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.53e-07 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732301+0000", 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.618e-06 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732308+0000", 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6399999999999997e-06 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732314+0000", 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2700000000000004e-06 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732670+0000", 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035588900000000002 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732676+0000", 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1500000000000004e-06 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702608+0000", 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96993175200000004 2024-08-30T23:51:22.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702619+0000", 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1192999999999999e-05 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:86697 5.7 5.f642fc47 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e525)", 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 550.76397806, 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.97034427400000001, 2024-08-30T23:51:22.881 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 86697 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732320+0000", 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8500000000000002e-07 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732323+0000", 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4320000000000002e-06 2024-08-30T23:51:22.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732344+0000", 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1539000000000002e-05 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732349+0000", 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0289999999999996e-06 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732694+0000", 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034466299999999999 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:11.732696+0000", 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6139999999999999e-06 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702655+0000", 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96995884499999996 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:12.702664+0000", 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7669999999999992e-06 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.26173.0:95915 5.5 5:a7a5f5b9:::benchmark_data_smithi173_87639_object15675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e532)", 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 533.20538810200003, 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.567720742, 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.26173", 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2462033347", 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 95915 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290910+0000", 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290911+0000", 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6500000000000001e-06 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290917+0000", 2024-08-30T23:51:22.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3020000000000002e-06 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290930+0000", 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.305e-05 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.290939+0000", 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0669999999999998e-06 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.371235+0000", 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.080296505000000004 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for rw locks", 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.374504+0000", 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0032687430000000002 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858349+0000", 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48384462 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:42:29.858630+0000", 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00028180499999999999 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.26173.0:128523 5.15 e573/539 5:ab7a4302:::benchmark_data_smithi173_87639_object48283:head v 573'6948, mlcod=410'2803)", 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:43:34.346043+0000", 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 468.15025508700001, 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55079527699999997, 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.891 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346043+0000", 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346043+0000", 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346044+0000", 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6139999999999999e-06 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.346053+0000", 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8829999999999995e-06 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894635+0000", 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54858180999999995 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894641+0000", 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5949999999999998e-06 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894717+0000", 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6699000000000005e-05 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.894736+0000", 2024-08-30T23:51:22.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8321999999999998e-05 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.896812+0000", 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020768509999999998 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:43:34.896838+0000", 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5503e-05 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.895 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(5.15 e602,590 5:ae155e6a:::benchmark_data_smithi173_87639_object50280:head,576'7007)", 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 425.123153849, 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54375525700000005, 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373144+0000", 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8999999999999998e-07 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.373147+0000", 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3369999999999998e-06 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916687+0000", 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54354048600000004 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916702+0000", 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5315000000000002e-05 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916802+0000", 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9502999999999999e-05 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916815+0000", 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2757e-05 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:44:17.916899+0000", 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4568999999999996e-05 2024-08-30T23:51:22.899 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.b 763/756 [PushReplyOp(7:d4034f8a:::benchmark_data_smithi173_129296_object8907:head)])", 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 54.108316154000001, 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61886441400000003, 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9800000000000003e-07 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387988+0000", 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9800000000000003e-06 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001726+0000", 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61373751200000004 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001738+0000", 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1746000000000001e-05 2024-08-30T23:51:22.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001796+0000", 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8470000000000001e-05 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.006846+0000", 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050499079999999997 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:51:22.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:51:22.904 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:51:22.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:22.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:23.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:23.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:23.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:23.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:23.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:23.331 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:23.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.476 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:23.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:23.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:23.610 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:51:23.662 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:51:23.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:51:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:51:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:51:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:51:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:51:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:51:23.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:23.745 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:23.841 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:23 smithi136 ceph-mon[20207]: pgmap v1746: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 625 op/s; 10587/208218 objects degraded (5.085%); 16116/208218 objects misplaced (7.740%); 21 MiB/s, 342 objects/s recovering 2024-08-30T23:51:23.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:23.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:23 smithi106 ceph-mon[16603]: pgmap v1746: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 625 op/s; 10587/208218 objects degraded (5.085%); 16116/208218 objects misplaced (7.740%); 21 MiB/s, 342 objects/s recovering 2024-08-30T23:51:23.886 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:23.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:23.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:24.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:24.049 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:23 smithi173 ceph-mon[20554]: pgmap v1746: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 625 op/s; 10587/208218 objects degraded (5.085%); 16116/208218 objects misplaced (7.740%); 21 MiB/s, 342 objects/s recovering 2024-08-30T23:51:24.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:24.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:24.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:24.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:24.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:24.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:24.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:24.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:24.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:24.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:24.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:24.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:24.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:24.931 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:24.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:25.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:25.059 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:25.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:25.184 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:25.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:25.309 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:25.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:25.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:25.550 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:25.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-08-30T23:51:25.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:25.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:25.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:25.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:25.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:25 smithi106 ceph-mon[16603]: pgmap v1747: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 524 op/s; 10585/206622 objects degraded (5.123%); 15815/206622 objects misplaced (7.654%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:51:25.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:25.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:25.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:25.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:25 smithi136 ceph-mon[20207]: pgmap v1747: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 524 op/s; 10585/206622 objects degraded (5.123%); 15815/206622 objects misplaced (7.654%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:51:26.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:25 smithi173 ceph-mon[20554]: pgmap v1747: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 524 op/s; 10585/206622 objects degraded (5.123%); 15815/206622 objects misplaced (7.654%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:51:26.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:26.135 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:26.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:51:26.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:26.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:26.261 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:26.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:26.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:26.477 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:51:26.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:26.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:26.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:51:26.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:26.639 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.640 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:26.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:26.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:26.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:26.913 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:26.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:26.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:51:27.002 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:27.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:27.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10585/206622 objects degraded (5.123%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:27.056 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10585/206622 objects degraded (5.123%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:27.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10585/206622 objects degraded (5.123%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:27.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:27.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:27.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:27.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:27.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:27.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.442 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:27.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:27.572 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:27.658 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.3 to 1 (10000) 2024-08-30T23:51:27.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:27.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:27.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:27.824 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:27.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:27.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:27.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:27 smithi136 ceph-mon[20207]: pgmap v1748: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 10581/202632 objects degraded (5.222%); 15517/202632 objects misplaced (7.658%); 12 MiB/s, 198 objects/s recovering 2024-08-30T23:51:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3894853310' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:51:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:27 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:51:28.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:27 smithi173 ceph-mon[20554]: pgmap v1748: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 10581/202632 objects degraded (5.222%); 15517/202632 objects misplaced (7.658%); 12 MiB/s, 198 objects/s recovering 2024-08-30T23:51:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3894853310' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:51:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:27 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:51:28.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:27 smithi106 ceph-mon[16603]: pgmap v1748: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 10581/202632 objects degraded (5.222%); 15517/202632 objects misplaced (7.658%); 12 MiB/s, 198 objects/s recovering 2024-08-30T23:51:28.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3894853310' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:51:28.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:27 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-30T23:51:28.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:28.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:28.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:28.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:28.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:28.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:28.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:28.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:28.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:28.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:28.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.849 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:28.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:28 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:51:28.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:28 smithi106 ceph-mon[16603]: osdmap e782: 12 total, 12 up, 11 in 2024-08-30T23:51:28.940 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:28 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:51:28.941 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:28 smithi136 ceph-mon[20207]: osdmap e782: 12 total, 12 up, 11 in 2024-08-30T23:51:28.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:28.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:28.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:29.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:28 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-30T23:51:29.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:28 smithi173 ceph-mon[20554]: osdmap e782: 12 total, 12 up, 11 in 2024-08-30T23:51:29.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:29.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:29.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:29.233 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:29.297 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:51:29.297 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:51:29.298 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:51:29.298 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:51:29.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:29.369 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:29.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:29.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:29.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:29.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:29.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:29.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:29.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:29.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:29 smithi106 ceph-mon[16603]: pgmap v1750: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 10581/197907 objects degraded (5.346%); 14524/197907 objects misplaced (7.339%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:51:29.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:29.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:29.990 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:29 smithi136 ceph-mon[20207]: pgmap v1750: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 10581/197907 objects degraded (5.346%); 14524/197907 objects misplaced (7.339%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:51:29.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:29.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:30.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:29 smithi173 ceph-mon[20554]: pgmap v1750: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 10581/197907 objects degraded (5.346%); 14524/197907 objects misplaced (7.339%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:51:30.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:30.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:30.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:51:30.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:30.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:30.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:30.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:30.484 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:51:30.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:30.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:30.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:30.653 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:30.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:30.781 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:30.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:30.925 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:30.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:31.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:31.058 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:51:31.058 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:51:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.073 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:31.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:31.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:31.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:31.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:31.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:31.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:31.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:31.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:31.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:31.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:31.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:31.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-08-30T23:51:31.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:51:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:51:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:51:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:51:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:51:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:51:31.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:51:31.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:51:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:31 smithi136 ceph-mon[20207]: pgmap v1751: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 10578/194226 objects degraded (5.446%); 14183/194226 objects misplaced (7.302%); 16 MiB/s, 264 objects/s recovering 2024-08-30T23:51:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/937516663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10578/194226 objects degraded (5.446%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:31.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:31.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:31.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:31.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:31.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:31 smithi173 ceph-mon[20554]: pgmap v1751: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 10578/194226 objects degraded (5.446%); 14183/194226 objects misplaced (7.302%); 16 MiB/s, 264 objects/s recovering 2024-08-30T23:51:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/937516663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10578/194226 objects degraded (5.446%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:32.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:31 smithi106 ceph-mon[16603]: pgmap v1751: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 10578/194226 objects degraded (5.446%); 14183/194226 objects misplaced (7.302%); 16 MiB/s, 264 objects/s recovering 2024-08-30T23:51:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/937516663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:32.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10578/194226 objects degraded (5.446%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:32.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:32.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:32.236 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:32.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:32.360 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:32.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.504 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:32.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:32.521 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:32.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:32.644 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:32.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:51:32.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:32.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:32.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:32.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:32.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:32.996 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:51:33.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:33.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:33.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:33.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:33.189 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:33.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:33.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:33.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:33.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:33.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:33.529 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:51:33.529 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:51:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:33.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:33.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:33.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:33.692 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:51:33.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:33.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:33.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:33.908 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:33.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:34.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:34.032 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:33 smithi136 ceph-mon[20207]: pgmap v1752: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 10578/190980 objects degraded (5.539%); 13432/190980 objects misplaced (7.033%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:51:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2019494771' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:34.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:33 smithi173 ceph-mon[20554]: pgmap v1752: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 10578/190980 objects degraded (5.539%); 13432/190980 objects misplaced (7.033%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:51:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2019494771' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:34.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:33 smithi106 ceph-mon[16603]: pgmap v1752: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 10578/190980 objects degraded (5.539%); 13432/190980 objects misplaced (7.033%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:51:34.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2019494771' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:34.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:51:34.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:34.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:34.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:51:34.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:34.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:34.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:34.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:34.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:51:34.559 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:34.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.583 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:34.663 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:34.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:34.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:34 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:34.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:51:34.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:34 smithi106 ceph-mon[16603]: osdmap e783: 12 total, 12 up, 11 in 2024-08-30T23:51:34.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:34 smithi106 ceph-mon[16603]: pgmap v1754: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 10574/187497 objects degraded (5.640%); 12975/187497 objects misplaced (6.920%); 20 MiB/s, 318 objects/s recovering 2024-08-30T23:51:34.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3875183316' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:51:34.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:51:34.710 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.4 to 1 (10000) 2024-08-30T23:51:34.748 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:34.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:34.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.874 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:34.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:34.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:34.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:34 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:51:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:34 smithi136 ceph-mon[20207]: osdmap e783: 12 total, 12 up, 11 in 2024-08-30T23:51:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:34 smithi136 ceph-mon[20207]: pgmap v1754: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 10574/187497 objects degraded (5.640%); 12975/187497 objects misplaced (6.920%); 20 MiB/s, 318 objects/s recovering 2024-08-30T23:51:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3875183316' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:51:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:51:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:34 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:51:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:34 smithi173 ceph-mon[20554]: osdmap e783: 12 total, 12 up, 11 in 2024-08-30T23:51:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:34 smithi173 ceph-mon[20554]: pgmap v1754: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 10574/187497 objects degraded (5.640%); 12975/187497 objects misplaced (6.920%); 20 MiB/s, 318 objects/s recovering 2024-08-30T23:51:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3875183316' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:51:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-30T23:51:35.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:35.140 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:35.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:35.275 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:35.377 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:35.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:35.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:35.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:35.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:35.647 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:35.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-08-30T23:51:35.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:35.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:35.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:35.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:35.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:35.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:36.023 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:36.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:35 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:51:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:35 smithi136 ceph-mon[20207]: osdmap e784: 12 total, 12 up, 11 in 2024-08-30T23:51:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:35 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:51:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:35 smithi173 ceph-mon[20554]: osdmap e784: 12 total, 12 up, 11 in 2024-08-30T23:51:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:35 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-30T23:51:36.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:35 smithi106 ceph-mon[16603]: osdmap e784: 12 total, 12 up, 11 in 2024-08-30T23:51:36.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:36.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:36.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.293 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:36.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:36.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:36.419 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.420 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:36.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:51:36.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:36.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:36.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:36.726 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:36.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:36.852 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:36.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:36.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:36.970 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:51:36.970 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:51:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:37.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:37.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:36 smithi136 ceph-mon[20207]: osdmap e785: 12 total, 12 up, 11 in 2024-08-30T23:51:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:36 smithi136 ceph-mon[20207]: pgmap v1757: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 10574/185838 objects degraded (5.690%); 12800/185838 objects misplaced (6.888%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:51:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10574/187497 objects degraded (5.640%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:36 smithi173 ceph-mon[20554]: osdmap e785: 12 total, 12 up, 11 in 2024-08-30T23:51:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:36 smithi173 ceph-mon[20554]: pgmap v1757: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 10574/185838 objects degraded (5.690%); 12800/185838 objects misplaced (6.888%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:51:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10574/187497 objects degraded (5.640%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:36 smithi106 ceph-mon[16603]: osdmap e785: 12 total, 12 up, 11 in 2024-08-30T23:51:37.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:36 smithi106 ceph-mon[16603]: pgmap v1757: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 10574/185838 objects degraded (5.690%); 12800/185838 objects misplaced (6.888%); 14 MiB/s, 230 objects/s recovering 2024-08-30T23:51:37.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10574/187497 objects degraded (5.640%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:37.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:37.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:37.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:37.283 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:37.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.424 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:37.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:37.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:37.568 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:37.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:37.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.716 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:37.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:37.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:51:37.729 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:37.801 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:37.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:37.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:37.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:37.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:37.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:38.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:51:38.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:51:38.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:51:38.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:37 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:51:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:51:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:51:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:51:38.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:37 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:51:38.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:51:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:51:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:51:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:37 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:51:38.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:38.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:38.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:38.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:38.340 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:38.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:38.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:38.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:38.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:38.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:38.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:38.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:38.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:38.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:38.992 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:38.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:38.997 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:38 smithi106 ceph-mon[16603]: pgmap v1758: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 10569/181143 objects degraded (5.835%); 11813/181143 objects misplaced (6.521%); 17 MiB/s, 271 objects/s recovering 2024-08-30T23:51:39.002 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:38 smithi173 ceph-mon[20554]: pgmap v1758: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 10569/181143 objects degraded (5.835%); 11813/181143 objects misplaced (6.521%); 17 MiB/s, 271 objects/s recovering 2024-08-30T23:51:39.090 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:51:39.090 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:51:39.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:39.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:39.128 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:51:39.128 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:51:39.171 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:51:39.171 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:51:39.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:39.269 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:39.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:38 smithi136 ceph-mon[20207]: pgmap v1758: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 10569/181143 objects degraded (5.835%); 11813/181143 objects misplaced (6.521%); 17 MiB/s, 271 objects/s recovering 2024-08-30T23:51:39.371 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:39.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:39.413 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:51:39.413 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:51:39.455 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:51:39.455 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:51:39.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:39.498 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:51:39.498 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:51:39.536 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:51:39.537 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:51:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.538 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:39.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:39.571 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:51:39.572 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:51:39.604 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:51:39.605 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:51:39.632 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:51:39.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:51:39.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:39.661 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:39.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:51:39.665 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:51:39.696 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:51:39.697 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:51:39.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:39.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:39.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:39.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:39.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:40.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:40.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:40.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3403032703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3403032703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:40.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:40.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:40.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:40.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3403032703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:51:40.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:40.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:40.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:40.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:40.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:40.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:40.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:40.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:40.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:40.808 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:51:40.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:40.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:40.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:40.949 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:40 smithi106 ceph-mon[16603]: pgmap v1759: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 10567/177348 objects degraded (5.958%); 11481/177348 objects misplaced (6.474%); 15 MiB/s, 248 objects/s recovering 2024-08-30T23:51:40.950 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4187087949' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:51:40.950 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:40 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:51:41.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:41.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:40 smithi136 ceph-mon[20207]: pgmap v1759: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 10567/177348 objects degraded (5.958%); 11481/177348 objects misplaced (6.474%); 15 MiB/s, 248 objects/s recovering 2024-08-30T23:51:41.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4187087949' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:51:41.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:40 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:51:41.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:40 smithi173 ceph-mon[20554]: pgmap v1759: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.6 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 10567/177348 objects degraded (5.958%); 11481/177348 objects misplaced (6.474%); 15 MiB/s, 248 objects/s recovering 2024-08-30T23:51:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4187087949' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:51:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:40 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:51:41.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:41.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:41.317 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:41.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:41.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:41.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:41.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:41.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:41.728 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:41.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:41.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:41.949 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.5 to 1 (10000) 2024-08-30T23:51:41.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:41.977 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:41.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:41 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:41 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:51:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:41 smithi136 ceph-mon[20207]: osdmap e786: 12 total, 12 up, 11 in 2024-08-30T23:51:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10567/177348 objects degraded (5.958%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:41 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:41 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:51:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:41 smithi173 ceph-mon[20554]: osdmap e786: 12 total, 12 up, 11 in 2024-08-30T23:51:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10567/177348 objects degraded (5.958%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:42.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:42.102 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:41 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:51:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:41 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:51:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:41 smithi106 ceph-mon[16603]: osdmap e786: 12 total, 12 up, 11 in 2024-08-30T23:51:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10567/177348 objects degraded (5.958%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:42.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:42.225 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:42.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:42.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:42.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:42.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:42.479 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:42.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:42.596 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:42.699 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:42.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:42.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:51:42.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:42.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:42 smithi106 ceph-mon[16603]: pgmap v1761: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 10567/174072 objects degraded (6.070%); 10673/174072 objects misplaced (6.131%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:51:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3872842636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:51:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:42 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:51:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:42 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:51:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:42 smithi106 ceph-mon[16603]: osdmap e787: 12 total, 12 up, 11 in 2024-08-30T23:51:42.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:42.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:42.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:43.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:42 smithi173 ceph-mon[20554]: pgmap v1761: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 10567/174072 objects degraded (6.070%); 10673/174072 objects misplaced (6.131%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:51:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3872842636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:51:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:42 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:51:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:42 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:51:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:42 smithi173 ceph-mon[20554]: osdmap e787: 12 total, 12 up, 11 in 2024-08-30T23:51:43.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:43.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:43.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-08-30T23:51:43.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:42 smithi136 ceph-mon[20207]: pgmap v1761: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 29 active+clean; 3.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 557 op/s; 10567/174072 objects degraded (6.070%); 10673/174072 objects misplaced (6.131%); 19 MiB/s, 303 objects/s recovering 2024-08-30T23:51:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3872842636' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:51:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:42 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-30T23:51:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:42 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-30T23:51:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:42 smithi136 ceph-mon[20207]: osdmap e787: 12 total, 12 up, 11 in 2024-08-30T23:51:43.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:43.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:43.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:43.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:43.478 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:51:43.478 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:51:43.478 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:51:43.478 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:51:43.478 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:51:43.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:43.595 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:43.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:43.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:43.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:43.842 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:43.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:43.965 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:43.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:44.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:44.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:44.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:44.228 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:43 smithi136 ceph-mon[20207]: osdmap e788: 12 total, 12 up, 11 in 2024-08-30T23:51:44.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:51:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:43 smithi173 ceph-mon[20554]: osdmap e788: 12 total, 12 up, 11 in 2024-08-30T23:51:44.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:44.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:44.353 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:43 smithi106 ceph-mon[16603]: osdmap e788: 12 total, 12 up, 11 in 2024-08-30T23:51:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:44.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:44.479 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:44.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:44.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:44.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:44.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:44.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:44.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:44.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:45.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:45.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:45.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.176 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:45.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:45.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:44 smithi136 ceph-mon[20207]: pgmap v1764: 37 pgs: 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 581 op/s; 10566/170664 objects degraded (6.191%); 10060/170664 objects misplaced (5.895%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:51:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:44 smithi136 ceph-mon[20207]: osdmap e789: 12 total, 12 up, 11 in 2024-08-30T23:51:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:44 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:51:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:44 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:51:45.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:45.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:44 smithi173 ceph-mon[20554]: pgmap v1764: 37 pgs: 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 581 op/s; 10566/170664 objects degraded (6.191%); 10060/170664 objects misplaced (5.895%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:51:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:44 smithi173 ceph-mon[20554]: osdmap e789: 12 total, 12 up, 11 in 2024-08-30T23:51:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:44 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:51:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:44 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:51:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:44 smithi106 ceph-mon[16603]: pgmap v1764: 37 pgs: 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 581 op/s; 10566/170664 objects degraded (6.191%); 10060/170664 objects misplaced (5.895%); 18 MiB/s, 292 objects/s recovering 2024-08-30T23:51:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:44 smithi106 ceph-mon[16603]: osdmap e789: 12 total, 12 up, 11 in 2024-08-30T23:51:45.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:44 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:51:45.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:44 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:51:45.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:45.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:45.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:45.589 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:45.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:45.729 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:45.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:45.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:45.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:45.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:45.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:46.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:46.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:46.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:46.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:46.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:45 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10566/170664 objects degraded (6.191%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:45 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10566/170664 objects degraded (6.191%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:46.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:46.355 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:46.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:45 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10566/170664 objects degraded (6.191%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:46.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:46.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:46.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:46.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:46.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:46.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:46.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:46.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:46.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:46.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:47.024 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:47.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:47.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:47.163 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:47.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:47.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:47.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:47.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:47.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:46 smithi136 ceph-mon[20207]: pgmap v1766: 37 pgs: 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 10565/170007 objects degraded (6.214%); 9887/170007 objects misplaced (5.816%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:51:47.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:46 smithi173 ceph-mon[20554]: pgmap v1766: 37 pgs: 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 10565/170007 objects degraded (6.214%); 9887/170007 objects misplaced (5.816%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:51:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:46 smithi106 ceph-mon[16603]: pgmap v1766: 37 pgs: 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 28 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 496 op/s; 10565/170007 objects degraded (6.214%); 9887/170007 objects misplaced (5.816%); 20 MiB/s, 323 objects/s recovering 2024-08-30T23:51:47.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:47.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:47.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:47.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:47.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:47.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:47.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:47.951 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:47.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:48.054 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:48.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:48.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:48.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:48.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:48.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:48.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:48.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:48.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:48.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:48.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:48.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:48.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:48.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:48.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:48.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:48.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:49.000 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:49.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:49.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:49.227 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:49.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:48 smithi136 ceph-mon[20207]: pgmap v1767: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 10566/167283 objects degraded (6.316%); 11091/167283 objects misplaced (6.630%); 21 MiB/s, 339 objects/s recovering 2024-08-30T23:51:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:48 smithi173 ceph-mon[20554]: pgmap v1767: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 10566/167283 objects degraded (6.316%); 11091/167283 objects misplaced (6.630%); 21 MiB/s, 339 objects/s recovering 2024-08-30T23:51:49.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:48 smithi106 ceph-mon[16603]: pgmap v1767: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 10566/167283 objects degraded (6.316%); 11091/167283 objects misplaced (6.630%); 21 MiB/s, 339 objects/s recovering 2024-08-30T23:51:49.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:49.381 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:49.483 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:49.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:49.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:49.629 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:49.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:49.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:49.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:49.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:49.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:49.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:50.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:50.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:50.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:50.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:50.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:49 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:49 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:49 smithi136 ceph-mon[20207]: osdmap e790: 12 total, 12 up, 11 in 2024-08-30T23:51:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:49 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:49 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:50.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:49 smithi173 ceph-mon[20554]: osdmap e790: 12 total, 12 up, 11 in 2024-08-30T23:51:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:49 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:49 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:49 smithi106 ceph-mon[16603]: osdmap e790: 12 total, 12 up, 11 in 2024-08-30T23:51:50.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:50.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:50.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:50.517 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:50.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:50.641 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:50.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:50.767 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:50.869 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:50.904 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:50.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:51.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:51.046 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:51.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:51.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:51.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:50 smithi136 ceph-mon[20207]: pgmap v1769: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 413 op/s; 10620/164097 objects degraded (6.472%); 14013/164097 objects misplaced (8.539%); 20 MiB/s, 322 objects/s recovering 2024-08-30T23:51:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:50 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:50 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:50 smithi136 ceph-mon[20207]: osdmap e791: 12 total, 12 up, 11 in 2024-08-30T23:51:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:50 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10620/164097 objects degraded (6.472%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:51.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:50 smithi173 ceph-mon[20554]: pgmap v1769: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 413 op/s; 10620/164097 objects degraded (6.472%); 14013/164097 objects misplaced (8.539%); 20 MiB/s, 322 objects/s recovering 2024-08-30T23:51:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:50 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:50 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:50 smithi173 ceph-mon[20554]: osdmap e791: 12 total, 12 up, 11 in 2024-08-30T23:51:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:50 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10620/164097 objects degraded (6.472%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:50 smithi106 ceph-mon[16603]: pgmap v1769: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 413 op/s; 10620/164097 objects degraded (6.472%); 14013/164097 objects misplaced (8.539%); 20 MiB/s, 322 objects/s recovering 2024-08-30T23:51:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:50 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:50 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:50 smithi106 ceph-mon[16603]: osdmap e791: 12 total, 12 up, 11 in 2024-08-30T23:51:51.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:50 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10620/164097 objects degraded (6.472%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:51.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:51.428 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:51.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:51.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:51.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:51.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:51.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:51.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:51.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:51.963 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:51.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:52.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:52.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:52.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:52.211 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:52.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:52.337 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:52.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:52.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:52.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:52.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.589 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:52.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:52.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:52.818 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.843 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:52.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:52.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:52.967 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:52.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:53.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:53.110 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:53.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:53.236 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:53.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.238 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:52 smithi136 ceph-mon[20207]: pgmap v1771: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 400 op/s; 10620/161304 objects degraded (6.584%); 13297/161304 objects misplaced (8.243%); 21 MiB/s, 341 objects/s recovering 2024-08-30T23:51:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:52 smithi173 ceph-mon[20554]: pgmap v1771: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 400 op/s; 10620/161304 objects degraded (6.584%); 13297/161304 objects misplaced (8.243%); 21 MiB/s, 341 objects/s recovering 2024-08-30T23:51:53.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:52 smithi106 ceph-mon[16603]: pgmap v1771: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 400 op/s; 10620/161304 objects degraded (6.584%); 13297/161304 objects misplaced (8.243%); 21 MiB/s, 341 objects/s recovering 2024-08-30T23:51:53.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.362 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:53.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:53.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:53.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:53.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:53.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:53.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:53.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:53.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:53.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:53.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:54.007 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:54.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:54.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:54.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:54.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:54 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:51:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:54 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:51:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:54 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:51:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:54 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:51:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:54 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:51:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:54 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:51:54.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:54.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:54.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:54.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:54.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:54.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:54.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:54.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:54.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:54.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:54.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:54.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:55.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:55.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:55.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:55.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:55.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:55 smithi136 ceph-mon[20207]: pgmap v1772: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 10619/157503 objects degraded (6.742%); 12754/157503 objects misplaced (8.098%); 24 MiB/s, 378 objects/s recovering 2024-08-30T23:51:55.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:55 smithi136 ceph-mon[20207]: osdmap e792: 12 total, 12 up, 11 in 2024-08-30T23:51:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:55 smithi173 ceph-mon[20554]: pgmap v1772: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 10619/157503 objects degraded (6.742%); 12754/157503 objects misplaced (8.098%); 24 MiB/s, 378 objects/s recovering 2024-08-30T23:51:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:55 smithi173 ceph-mon[20554]: osdmap e792: 12 total, 12 up, 11 in 2024-08-30T23:51:55.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:55 smithi106 ceph-mon[16603]: pgmap v1772: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 10619/157503 objects degraded (6.742%); 12754/157503 objects misplaced (8.098%); 24 MiB/s, 378 objects/s recovering 2024-08-30T23:51:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:55 smithi106 ceph-mon[16603]: osdmap e792: 12 total, 12 up, 11 in 2024-08-30T23:51:55.398 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:55.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:55.524 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:55.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:55.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.651 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:55.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:55.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:55.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:55.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:56.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:56.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:56.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:56.150 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:56.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:51:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:56.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:51:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:56 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:56 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:56 smithi136 ceph-mon[20207]: osdmap e793: 12 total, 12 up, 11 in 2024-08-30T23:51:56.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10619/157503 objects degraded (6.742%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:56 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:56 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:56 smithi173 ceph-mon[20554]: osdmap e793: 12 total, 12 up, 11 in 2024-08-30T23:51:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10619/157503 objects degraded (6.742%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:56 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:56.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:56 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:56.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:56 smithi106 ceph-mon[16603]: osdmap e793: 12 total, 12 up, 11 in 2024-08-30T23:51:56.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10619/157503 objects degraded (6.742%), 4 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-30T23:51:56.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:56.411 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:56.513 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:56.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.536 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:56.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.659 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:56.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:56.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.794 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:56.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:56.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:56.919 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:56.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:57.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:57.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:57.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:57.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:57.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:57.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:57 smithi136 ceph-mon[20207]: pgmap v1775: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 10616/156528 objects degraded (6.782%); 12578/156528 objects misplaced (8.036%); 15 MiB/s, 239 objects/s recovering 2024-08-30T23:51:57.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:51:57.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:51:57.308 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:57 smithi173 ceph-mon[20554]: pgmap v1775: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 10616/156528 objects degraded (6.782%); 12578/156528 objects misplaced (8.036%); 15 MiB/s, 239 objects/s recovering 2024-08-30T23:51:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:51:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:51:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:57 smithi106 ceph-mon[16603]: pgmap v1775: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 3.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 420 op/s; 10616/156528 objects degraded (6.782%); 12578/156528 objects misplaced (8.036%); 15 MiB/s, 239 objects/s recovering 2024-08-30T23:51:57.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:51:57.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:51:57.412 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:57.435 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:57.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:57.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:57.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:57.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:57.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:57.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.813 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:57.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:57.937 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:57.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:58.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:58.080 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:51:58.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:51:58.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:51:58.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:58.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:58 smithi173 ceph-mon[20554]: osdmap e794: 12 total, 12 up, 11 in 2024-08-30T23:51:58.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:58 smithi106 ceph-mon[16603]: osdmap e794: 12 total, 12 up, 11 in 2024-08-30T23:51:58.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:51:58.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:58.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:58.470 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:51:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:58 smithi136 ceph-mon[20207]: osdmap e794: 12 total, 12 up, 11 in 2024-08-30T23:51:58.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:58.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:58.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:51:58.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:51:58.812 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:51:58.834 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:51:58.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:51:58.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:58.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:51:59.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:51:59.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:51:59.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:51:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.208 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:59.208 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:51:59.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:51:59 smithi173 ceph-mon[20554]: pgmap v1777: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 10612/152586 objects degraded (6.955%); 11511/152586 objects misplaced (7.544%); 19 MiB/s, 299 objects/s recovering 2024-08-30T23:51:59.331 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:51:59 smithi106 ceph-mon[16603]: pgmap v1777: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 10612/152586 objects degraded (6.955%); 11511/152586 objects misplaced (7.544%); 19 MiB/s, 299 objects/s recovering 2024-08-30T23:51:59.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:51:59.468 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:51:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:51:59 smithi136 ceph-mon[20207]: pgmap v1777: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.1 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 484 op/s; 10612/152586 objects degraded (6.955%); 11511/152586 objects misplaced (7.544%); 19 MiB/s, 299 objects/s recovering 2024-08-30T23:51:59.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:51:59.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:51:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:51:59.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:51:59.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:51:59.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:51:59.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:51:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:51:59.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:51:59.979 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:51:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:51:59.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:00.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:00.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.125 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:00.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:00.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:00.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:00.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:00.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:00.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:00.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:00.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:00.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:00.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:00.750 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:00.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:00.886 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:00.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:00.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:01.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:01.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:01.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:01.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:01.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:01.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:01 smithi173 ceph-mon[20554]: pgmap v1778: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 431 op/s; 10610/149739 objects degraded (7.086%); 10896/149739 objects misplaced (7.277%); 19 MiB/s, 311 objects/s recovering 2024-08-30T23:52:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 10610/149739 objects degraded (7.086%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:01 smithi106 ceph-mon[16603]: pgmap v1778: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 431 op/s; 10610/149739 objects degraded (7.086%); 10896/149739 objects misplaced (7.277%); 19 MiB/s, 311 objects/s recovering 2024-08-30T23:52:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 10610/149739 objects degraded (7.086%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:01.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:01.432 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:01.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:01.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:01 smithi136 ceph-mon[20207]: pgmap v1778: 37 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 431 op/s; 10610/149739 objects degraded (7.086%); 10896/149739 objects misplaced (7.277%); 19 MiB/s, 311 objects/s recovering 2024-08-30T23:52:01.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 10610/149739 objects degraded (7.086%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:01.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:01.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.702 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:01.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:01.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:01.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:01.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:01.963 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:01.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:02.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:02.104 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:02.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:02.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.233 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:02.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:02 smithi173 ceph-mon[20554]: 7.c scrub starts 2024-08-30T23:52:02.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:02.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:02 smithi106 ceph-mon[16603]: 7.c scrub starts 2024-08-30T23:52:02.358 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:02.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:02.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:02 smithi136 ceph-mon[20207]: 7.c scrub starts 2024-08-30T23:52:02.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.608 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:02.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:02.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:02.731 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:02.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:02.856 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:02.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:02.980 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:02.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:03.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:03.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:03.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:03.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:03.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:03 smithi173 ceph-mon[20554]: pgmap v1779: 37 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 468 op/s; 10610/146616 objects degraded (7.237%); 10103/146616 objects misplaced (6.891%); 21 MiB/s, 343 objects/s recovering 2024-08-30T23:52:03.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:03 smithi173 ceph-mon[20554]: 7.c scrub ok 2024-08-30T23:52:03.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:03 smithi173 ceph-mon[20554]: osdmap e795: 12 total, 12 up, 11 in 2024-08-30T23:52:03.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:03 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:52:03.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:03 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:52:03.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:03.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:03 smithi106 ceph-mon[16603]: pgmap v1779: 37 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 468 op/s; 10610/146616 objects degraded (7.237%); 10103/146616 objects misplaced (6.891%); 21 MiB/s, 343 objects/s recovering 2024-08-30T23:52:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:03 smithi106 ceph-mon[16603]: 7.c scrub ok 2024-08-30T23:52:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:03 smithi106 ceph-mon[16603]: osdmap e795: 12 total, 12 up, 11 in 2024-08-30T23:52:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:03 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:52:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:03 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:52:03.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:03.491 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:03 smithi136 ceph-mon[20207]: pgmap v1779: 37 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 3.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 468 op/s; 10610/146616 objects degraded (7.237%); 10103/146616 objects misplaced (6.891%); 21 MiB/s, 343 objects/s recovering 2024-08-30T23:52:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:03 smithi136 ceph-mon[20207]: 7.c scrub ok 2024-08-30T23:52:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:03 smithi136 ceph-mon[20207]: osdmap e795: 12 total, 12 up, 11 in 2024-08-30T23:52:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:03 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:52:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:03 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:52:03.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:03.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:03.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:03.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:03.779 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:03.807 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:03.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:03.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:03.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:03.989 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:03.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:04.091 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:04.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:04.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:04.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:04.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:04.293 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: osdmap e796: 12 total, 12 up, 11 in 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:04.294 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:04 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:04.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:04.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:04.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:04.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:04.504 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:52:04.504 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:52:04.533 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: osdmap e796: 12 total, 12 up, 11 in 2024-08-30T23:52:04.533 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.533 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.533 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:52:04.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:52:04.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:04.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:04 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:04.543 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:52:04.543 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: osdmap e796: 12 total, 12 up, 11 in 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:04 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:04.578 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:52:04.578 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:52:04.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:04.618 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:52:04.618 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:52:04.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:04.655 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:52:04.655 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:52:04.692 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:52:04.693 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:52:04.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:04.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:52:04.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:52:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:04.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:04.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:52:04.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:52:04.794 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:52:04.795 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:52:04.834 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:52:04.835 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:52:04.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:04.870 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:52:04.870 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:52:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:04.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:04.906 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:52:04.906 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:52:04.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:05.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:05.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:05.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:05.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:05.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:05.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:05.340 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-30T23:52:05.339+0000 7fa1ead9d640 0 --2- 172.21.15.106:0/3189022935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1f4109eb0 0x7fa1f4110f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-30T23:52:05.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:05 smithi106 ceph-mon[16603]: pgmap v1782: 37 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 7612/142716 objects degraded (5.334%); 12746/142716 objects misplaced (8.931%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:52:05.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:05 smithi106 ceph-mon[16603]: 7.4 scrub starts 2024-08-30T23:52:05.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/953984411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:52:05.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:05 smithi106 ceph-mon[16603]: osdmap e797: 12 total, 12 up, 11 in 2024-08-30T23:52:05.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:05.368 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:05.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:05.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.520 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:05.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:05 smithi136 ceph-mon[20207]: pgmap v1782: 37 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 7612/142716 objects degraded (5.334%); 12746/142716 objects misplaced (8.931%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:52:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:05 smithi136 ceph-mon[20207]: 7.4 scrub starts 2024-08-30T23:52:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/953984411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:52:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:05 smithi136 ceph-mon[20207]: osdmap e797: 12 total, 12 up, 11 in 2024-08-30T23:52:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:05 smithi173 ceph-mon[20554]: pgmap v1782: 37 pgs: 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 7612/142716 objects degraded (5.334%); 12746/142716 objects misplaced (8.931%); 17 MiB/s, 277 objects/s recovering 2024-08-30T23:52:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:05 smithi173 ceph-mon[20554]: 7.4 scrub starts 2024-08-30T23:52:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/953984411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:52:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:05 smithi173 ceph-mon[20554]: osdmap e797: 12 total, 12 up, 11 in 2024-08-30T23:52:05.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:05.657 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:05.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:05.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:05.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:05.926 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:05.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:52:06.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:06.064 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:06.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:06.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:06.241 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: 7.4 scrub ok 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: 7.1 scrub starts 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: 7.1 scrub ok 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4244215740' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7612/142716 objects degraded (5.334%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: osdmap e798: 12 total, 12 up, 11 in 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4198476203' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:52:06.242 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:06 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:52:06.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:06.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:06.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.465 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:06.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: 7.4 scrub ok 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: 7.1 scrub starts 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: 7.1 scrub ok 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4244215740' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7612/142716 objects degraded (5.334%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:52:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:52:06.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: osdmap e798: 12 total, 12 up, 11 in 2024-08-30T23:52:06.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4198476203' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:52:06.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:06 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:52:06.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: 7.4 scrub ok 2024-08-30T23:52:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: 7.1 scrub starts 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: 7.1 scrub ok 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4244215740' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7612/142716 objects degraded (5.334%), 4 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: osdmap e798: 12 total, 12 up, 11 in 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4198476203' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:52:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:06 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-30T23:52:06.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:06.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.721 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:06.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:06.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:06.861 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:06.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:06.956 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.6 to 1 (10000) 2024-08-30T23:52:06.963 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:07.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:07.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:07.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:07.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:07.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:07.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:07 smithi106 ceph-mon[16603]: pgmap v1784: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 7612/141882 objects degraded (5.365%); 12474/141882 objects misplaced (8.792%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:52:07.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:07 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:52:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:07 smithi106 ceph-mon[16603]: osdmap e799: 12 total, 12 up, 11 in 2024-08-30T23:52:07.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:07.384 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:07.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:07.512 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:07 smithi136 ceph-mon[20207]: pgmap v1784: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 7612/141882 objects degraded (5.365%); 12474/141882 objects misplaced (8.792%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:52:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:07 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:52:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:07 smithi136 ceph-mon[20207]: osdmap e799: 12 total, 12 up, 11 in 2024-08-30T23:52:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:07 smithi173 ceph-mon[20554]: pgmap v1784: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 29 active+clean; 2.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 7612/141882 objects degraded (5.365%); 12474/141882 objects misplaced (8.792%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:52:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:07 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-30T23:52:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:07 smithi173 ceph-mon[20554]: osdmap e799: 12 total, 12 up, 11 in 2024-08-30T23:52:07.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.636 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:07.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:07.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:07.763 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:07.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:07.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:07.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:07.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:08.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:08.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:08.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:08.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:52:08.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:08.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:08.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:08.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:08.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:08.508 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:08.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:08.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:08.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:08.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:08.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:08.879 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:08.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:08.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:09.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:09.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:09.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:09.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:09.250 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:08 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:52:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:08 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:52:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:08 smithi136 ceph-mon[20207]: pgmap v1787: 37 pgs: 2 unknown, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 7757/120951 objects degraded (6.413%); 11502/120951 objects misplaced (9.510%); 19 MiB/s, 307 objects/s recovering 2024-08-30T23:52:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:08 smithi136 ceph-mon[20207]: osdmap e800: 12 total, 12 up, 11 in 2024-08-30T23:52:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:08 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:52:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:08 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:52:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:08 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:52:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:08 smithi173 ceph-mon[20554]: pgmap v1787: 37 pgs: 2 unknown, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 7757/120951 objects degraded (6.413%); 11502/120951 objects misplaced (9.510%); 19 MiB/s, 307 objects/s recovering 2024-08-30T23:52:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:08 smithi173 ceph-mon[20554]: osdmap e800: 12 total, 12 up, 11 in 2024-08-30T23:52:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:08 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:52:09.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:08 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:52:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:08 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:52:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:08 smithi106 ceph-mon[16603]: pgmap v1787: 37 pgs: 2 unknown, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 7757/120951 objects degraded (6.413%); 11502/120951 objects misplaced (9.510%); 19 MiB/s, 307 objects/s recovering 2024-08-30T23:52:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:08 smithi106 ceph-mon[16603]: osdmap e800: 12 total, 12 up, 11 in 2024-08-30T23:52:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:08 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:52:09.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:09.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:09.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:09.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:09.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:09.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:09.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:09.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:09.871 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:09.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:09.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:10.012 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:09 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:52:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:09 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:52:10.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:09 smithi106 ceph-mon[16603]: osdmap e801: 12 total, 12 up, 11 in 2024-08-30T23:52:10.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:10.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:10.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.277 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:10.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:10.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:09 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:52:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:09 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:52:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:09 smithi136 ceph-mon[20207]: osdmap e801: 12 total, 12 up, 11 in 2024-08-30T23:52:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:09 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:52:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:09 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:52:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:09 smithi173 ceph-mon[20554]: osdmap e801: 12 total, 12 up, 11 in 2024-08-30T23:52:10.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:10.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:10.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:10.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:10.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:10.653 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:10.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:10.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:10.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:10.935 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:10.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:11.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:11.065 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:52:11.065 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:52:11.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:11.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:11.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:10 smithi136 ceph-mon[20207]: pgmap v1790: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 7757/127497 objects degraded (6.084%); 10576/127497 objects misplaced (8.295%); 27 MiB/s, 438 objects/s recovering 2024-08-30T23:52:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7757/127497 objects degraded (6.084%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:11.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:11.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:10 smithi173 ceph-mon[20554]: pgmap v1790: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 7757/127497 objects degraded (6.084%); 10576/127497 objects misplaced (8.295%); 27 MiB/s, 438 objects/s recovering 2024-08-30T23:52:11.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7757/127497 objects degraded (6.084%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:10 smithi106 ceph-mon[16603]: pgmap v1790: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 7757/127497 objects degraded (6.084%); 10576/127497 objects misplaced (8.295%); 27 MiB/s, 438 objects/s recovering 2024-08-30T23:52:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7757/127497 objects degraded (6.084%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:11.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:11.445 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:11.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:11.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:11.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:11.721 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:11.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:11.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:11.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:11.948 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:11.986 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:11.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:12.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.113 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:12.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:12.214 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:12.247 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:11 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:11 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:11 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:12.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:11 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:12.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:11 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:11 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:12.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:12.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:12.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:12.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:12.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:12.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:12.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:12.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:12.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:12.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:12.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:13.010 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:13.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:13.146 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:13.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:13.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:12 smithi136 ceph-mon[20207]: pgmap v1791: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 429 op/s; 7757/134397 objects degraded (5.772%); 12695/134397 objects misplaced (9.446%); 26 MiB/s, 422 objects/s recovering 2024-08-30T23:52:13.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:12 smithi136 ceph-mon[20207]: osdmap e802: 12 total, 12 up, 11 in 2024-08-30T23:52:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:12 smithi173 ceph-mon[20554]: pgmap v1791: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 429 op/s; 7757/134397 objects degraded (5.772%); 12695/134397 objects misplaced (9.446%); 26 MiB/s, 422 objects/s recovering 2024-08-30T23:52:13.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:12 smithi173 ceph-mon[20554]: osdmap e802: 12 total, 12 up, 11 in 2024-08-30T23:52:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:12 smithi106 ceph-mon[16603]: pgmap v1791: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 1 peering, 3 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 27 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 429 op/s; 7757/134397 objects degraded (5.772%); 12695/134397 objects misplaced (9.446%); 26 MiB/s, 422 objects/s recovering 2024-08-30T23:52:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:13 smithi106 ceph-mon[16603]: osdmap e802: 12 total, 12 up, 11 in 2024-08-30T23:52:13.388 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:13.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:13.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:13.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:13.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:13.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:13.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:13.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:13.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:13.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:13.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:14.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:14.047 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:14.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.175 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:14.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:14.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:14.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:14 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:52:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:14.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:14 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:52:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:14 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:52:14.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:14.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:14.538 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.582 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:14.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:14.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:14.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:14.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:14.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:14.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:14.959 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:14.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:15.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:15.095 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:15.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:15.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:15.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:15 smithi173 ceph-mon[20554]: pgmap v1793: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 28 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 170 B/s rd, 113 MiB/s wr, 3.03k op/s; 7770/131619 objects degraded (5.903%); 14680/131619 objects misplaced (11.153%); 25 MiB/s, 404 objects/s recovering 2024-08-30T23:52:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:15 smithi106 ceph-mon[16603]: pgmap v1793: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 28 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 170 B/s rd, 113 MiB/s wr, 3.03k op/s; 7770/131619 objects degraded (5.903%); 14680/131619 objects misplaced (11.153%); 25 MiB/s, 404 objects/s recovering 2024-08-30T23:52:15.365 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:15.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:15.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:15 smithi136 ceph-mon[20207]: pgmap v1793: 37 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 28 active+clean; 2.7 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 170 B/s rd, 113 MiB/s wr, 3.03k op/s; 7770/131619 objects degraded (5.903%); 14680/131619 objects misplaced (11.153%); 25 MiB/s, 404 objects/s recovering 2024-08-30T23:52:15.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:15.619 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:15.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.758 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:15.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:15.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:15.902 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:15.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:16.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:16.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:16.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:16.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:16.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:16.267 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:16.288 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:16 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:52:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:16 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:52:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7770/131619 objects degraded (5.903%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:16.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:16 smithi173 ceph-mon[20554]: osdmap e803: 12 total, 12 up, 11 in 2024-08-30T23:52:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:16 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:52:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:16 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:52:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7770/131619 objects degraded (5.903%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:16.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:16 smithi106 ceph-mon[16603]: osdmap e803: 12 total, 12 up, 11 in 2024-08-30T23:52:16.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:16.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:16.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:16.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:16 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:52:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:16 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:52:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7770/131619 objects degraded (5.903%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:16 smithi136 ceph-mon[20207]: osdmap e803: 12 total, 12 up, 11 in 2024-08-30T23:52:16.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:16.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:16.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:16.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:16.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:16.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:16.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:17.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:17.073 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:17.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:17.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:17.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:17 smithi173 ceph-mon[20554]: pgmap v1794: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 28 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 131 B/s rd, 87 MiB/s wr, 2.37k op/s; 7768/130776 objects degraded (5.940%); 14396/130776 objects misplaced (11.008%); 22 MiB/s, 348 objects/s recovering 2024-08-30T23:52:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:17 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:17 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:17 smithi173 ceph-mon[20554]: osdmap e804: 12 total, 12 up, 11 in 2024-08-30T23:52:17.336 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:17.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:17 smithi106 ceph-mon[16603]: pgmap v1794: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 28 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 131 B/s rd, 87 MiB/s wr, 2.37k op/s; 7768/130776 objects degraded (5.940%); 14396/130776 objects misplaced (11.008%); 22 MiB/s, 348 objects/s recovering 2024-08-30T23:52:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:17 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:17 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:17.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:17 smithi106 ceph-mon[16603]: osdmap e804: 12 total, 12 up, 11 in 2024-08-30T23:52:17.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:17.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:17 smithi136 ceph-mon[20207]: pgmap v1794: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 28 active+clean; 2.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 131 B/s rd, 87 MiB/s wr, 2.37k op/s; 7768/130776 objects degraded (5.940%); 14396/130776 objects misplaced (11.008%); 22 MiB/s, 348 objects/s recovering 2024-08-30T23:52:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:17 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:17 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:17 smithi136 ceph-mon[20207]: osdmap e804: 12 total, 12 up, 11 in 2024-08-30T23:52:17.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:17.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:17.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:17.730 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.731 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:17.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.870 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:17.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:17.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:17.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:17.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:18.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:18.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:18.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.261 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:18.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:18.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:18.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:18.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:18.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:18.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:18.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:18.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:18.807 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:18.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:18.932 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:18.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:19.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:19.058 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:19.160 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.207 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:19.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:19.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:19.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:19 smithi173 ceph-mon[20554]: pgmap v1797: 37 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 396 op/s; 7768/127266 objects degraded (6.104%); 13604/127266 objects misplaced (10.689%); 20 MiB/s, 313 objects/s recovering 2024-08-30T23:52:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:19 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:19 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:19.338 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:19 smithi106 ceph-mon[16603]: pgmap v1797: 37 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 396 op/s; 7768/127266 objects degraded (6.104%); 13604/127266 objects misplaced (10.689%); 20 MiB/s, 313 objects/s recovering 2024-08-30T23:52:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:19 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:19 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:19.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:19.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:19.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:19 smithi136 ceph-mon[20207]: pgmap v1797: 37 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 2.6 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 396 op/s; 7768/127266 objects degraded (6.104%); 13604/127266 objects misplaced (10.689%); 20 MiB/s, 313 objects/s recovering 2024-08-30T23:52:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:19.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.599 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:19.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:19.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:19.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:19.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:19.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:19.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:19.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:19.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:20.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:20.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:20.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:20.254 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:20 smithi173 ceph-mon[20554]: osdmap e805: 12 total, 12 up, 11 in 2024-08-30T23:52:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:20 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:52:20.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:20 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:52:20.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:20 smithi106 ceph-mon[16603]: osdmap e805: 12 total, 12 up, 11 in 2024-08-30T23:52:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:20 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:52:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:20 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:52:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.398 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:20.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:20.499 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:20.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:20.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:20 smithi136 ceph-mon[20207]: osdmap e805: 12 total, 12 up, 11 in 2024-08-30T23:52:20.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:20 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:52:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:20 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:52:20.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:20.646 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:20.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:20.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:20.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:20.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:20.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:20.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:21.021 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:21.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.148 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:21.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:21.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:21.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:21.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:21 smithi106 ceph-mon[16603]: pgmap v1799: 37 pgs: 1 peering, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 2.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 373 op/s; 5191/124899 objects degraded (4.156%); 15104/124899 objects misplaced (12.093%); 22 MiB/s, 359 objects/s recovering 2024-08-30T23:52:21.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5191/124899 objects degraded (4.156%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:21.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:21.426 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:21.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:21 smithi136 ceph-mon[20207]: pgmap v1799: 37 pgs: 1 peering, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 2.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 373 op/s; 5191/124899 objects degraded (4.156%); 15104/124899 objects misplaced (12.093%); 22 MiB/s, 359 objects/s recovering 2024-08-30T23:52:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5191/124899 objects degraded (4.156%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:21.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:21 smithi173 ceph-mon[20554]: pgmap v1799: 37 pgs: 1 peering, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 28 active+clean; 2.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 373 op/s; 5191/124899 objects degraded (4.156%); 15104/124899 objects misplaced (12.093%); 22 MiB/s, 359 objects/s recovering 2024-08-30T23:52:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5191/124899 objects degraded (4.156%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:21.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:21.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:21.789 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:21.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:21.916 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:21.940 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:21.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:22.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:22.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:52:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.092 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:22.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:22.193 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.230 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:22.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:22.331 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:22.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:22.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:22.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:22.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:22.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:22.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:22.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:22.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:22.848 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:22.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:52:22.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:22.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:23.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:23.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:23.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:23.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:23.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:23.341 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:23 smithi136 ceph-mon[20207]: pgmap v1800: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 5191/122355 objects degraded (4.243%); 14730/122355 objects misplaced (12.039%); 23 MiB/s, 373 objects/s recovering 2024-08-30T23:52:23.341 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:23 smithi136 ceph-mon[20207]: osdmap e806: 12 total, 12 up, 11 in 2024-08-30T23:52:23.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:23 smithi106 ceph-mon[16603]: pgmap v1800: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 5191/122355 objects degraded (4.243%); 14730/122355 objects misplaced (12.039%); 23 MiB/s, 373 objects/s recovering 2024-08-30T23:52:23.348 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:23 smithi106 ceph-mon[16603]: osdmap e806: 12 total, 12 up, 11 in 2024-08-30T23:52:23.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:23.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:23.485 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:23 smithi173 ceph-mon[20554]: pgmap v1800: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 29 active+clean; 2.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 5191/122355 objects degraded (4.243%); 14730/122355 objects misplaced (12.039%); 23 MiB/s, 373 objects/s recovering 2024-08-30T23:52:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:23 smithi173 ceph-mon[20554]: osdmap e806: 12 total, 12 up, 11 in 2024-08-30T23:52:23.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:23.611 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:23.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:23.745 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:23.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:23.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:23.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:23.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:24.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:24.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.155 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:24.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:24.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:24.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:24 smithi106 ceph-mon[16603]: osdmap e807: 12 total, 12 up, 11 in 2024-08-30T23:52:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:52:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:52:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:52:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:52:24.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:24.437 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:24.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:24.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:24 smithi136 ceph-mon[20207]: osdmap e807: 12 total, 12 up, 11 in 2024-08-30T23:52:24.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:52:24.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:52:24.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:52:24.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:52:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:24 smithi173 ceph-mon[20554]: osdmap e807: 12 total, 12 up, 11 in 2024-08-30T23:52:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:52:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:52:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:52:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:52:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.585 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:24.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:24.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.709 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:24.709 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:24.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:24.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.836 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:24.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:24.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:24.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:25.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:25.088 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:25.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.212 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:25.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:25.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:25.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:25.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:25.465 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:25 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:52:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:25 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:52:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:25 smithi136 ceph-mon[20207]: 7.8 scrub starts 2024-08-30T23:52:25.568 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:25.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:25 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:52:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:25 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:52:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:25 smithi173 ceph-mon[20554]: 7.8 scrub starts 2024-08-30T23:52:25.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:25.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:25 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:52:25.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:25 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:52:25.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:25 smithi106 ceph-mon[16603]: 7.8 scrub starts 2024-08-30T23:52:25.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:25.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:25.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:25.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:25.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:25.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:25.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:25.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:25.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:26.098 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:26.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:26.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:26.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:26.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:26.411 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:26.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:26 smithi136 ceph-mon[20207]: 7.8 scrub ok 2024-08-30T23:52:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5191/118710 objects degraded (4.373%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:26.546 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:26 smithi173 ceph-mon[20554]: 7.8 scrub ok 2024-08-30T23:52:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5191/118710 objects degraded (4.373%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:26 smithi106 ceph-mon[16603]: 7.8 scrub ok 2024-08-30T23:52:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5191/118710 objects degraded (4.373%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:26.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:26.683 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:26.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:26.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:26.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:26.948 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:26.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:27.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.097 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:27.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:27.199 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:27.223 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:27.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:27.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:27.461 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.484 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:27.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:27 smithi136 ceph-mon[20207]: pgmap v1804: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 30 active+clean; 2.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 364 op/s; 5191/118320 objects degraded (4.387%); 13496/118320 objects misplaced (11.406%); 19 MiB/s, 301 objects/s recovering 2024-08-30T23:52:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:27 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:52:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:27 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:52:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:27 smithi173 ceph-mon[20554]: pgmap v1804: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 30 active+clean; 2.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 364 op/s; 5191/118320 objects degraded (4.387%); 13496/118320 objects misplaced (11.406%); 19 MiB/s, 301 objects/s recovering 2024-08-30T23:52:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:27 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:52:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:27 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:52:27.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:27 smithi106 ceph-mon[16603]: pgmap v1804: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 30 active+clean; 2.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 364 op/s; 5191/118320 objects degraded (4.387%); 13496/118320 objects misplaced (11.406%); 19 MiB/s, 301 objects/s recovering 2024-08-30T23:52:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:27 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:52:27.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:27 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:52:27.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:27.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.615 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:27.716 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:27.751 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:27.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:27.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:27.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:27.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:28.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:28.102 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:28.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:28.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-08-30T23:52:28.201 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:28.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:28.301 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:28.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:28.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:28.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.438 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:28.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:28.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:28.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:28.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.688 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:28.731 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:28.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:28.860 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:28.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:28.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:28.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:29.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:29.125 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:29.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:29.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:29.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:29.400 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:29.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:29.538 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:29.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:29 smithi136 ceph-mon[20207]: pgmap v1805: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 30 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 5191/114618 objects degraded (4.529%); 12953/114618 objects misplaced (11.301%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:52:29.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:29 smithi173 ceph-mon[20554]: pgmap v1805: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 30 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 5191/114618 objects degraded (4.529%); 12953/114618 objects misplaced (11.301%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:52:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:29 smithi106 ceph-mon[16603]: pgmap v1805: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 30 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 428 op/s; 5191/114618 objects degraded (4.529%); 12953/114618 objects misplaced (11.301%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:52:29.640 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:29.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:29.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:29.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:29.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:29.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:29.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:29.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:30.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:30.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:30.142 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.164 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:30.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:30.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:30.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:30.405 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:30.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:30.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:30.566 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:30.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:30.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:30.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:30.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:30.832 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:30.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:30.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:30.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:31.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:31.102 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:31.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:31.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:31.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:31 smithi106 ceph-mon[16603]: pgmap v1806: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 5188/111252 objects degraded (4.663%); 12296/111252 objects misplaced (11.052%); 19 MiB/s, 304 objects/s recovering 2024-08-30T23:52:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5188/111252 objects degraded (4.663%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:31.375 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:31.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:31.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:31.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:31 smithi136 ceph-mon[20207]: pgmap v1806: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 5188/111252 objects degraded (4.663%); 12296/111252 objects misplaced (11.052%); 19 MiB/s, 304 objects/s recovering 2024-08-30T23:52:31.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5188/111252 objects degraded (4.663%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:31 smithi173 ceph-mon[20554]: pgmap v1806: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 5188/111252 objects degraded (4.663%); 12296/111252 objects misplaced (11.052%); 19 MiB/s, 304 objects/s recovering 2024-08-30T23:52:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5188/111252 objects degraded (4.663%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:31.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:31.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:31.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:31.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:31.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:31.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:31.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:31.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:31.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:32.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:32.136 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:32.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.275 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:32.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:32.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:32.404 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:32.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:32.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:32.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:32.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:32.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:32.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:32.910 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:32.954 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:32.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:33.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:33.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:33.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:33.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:33.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:33 smithi106 ceph-mon[16603]: pgmap v1807: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 5188/108294 objects degraded (4.791%); 11948/108294 objects misplaced (11.033%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:52:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:33 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:33 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:33.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:33 smithi106 ceph-mon[16603]: osdmap e808: 12 total, 12 up, 11 in 2024-08-30T23:52:33.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2293301432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:52:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.385 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:33.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:33.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:33.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:33 smithi136 ceph-mon[20207]: pgmap v1807: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 5188/108294 objects degraded (4.791%); 11948/108294 objects misplaced (11.033%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:52:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:33 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:33 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:33 smithi136 ceph-mon[20207]: osdmap e808: 12 total, 12 up, 11 in 2024-08-30T23:52:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2293301432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:52:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:33 smithi173 ceph-mon[20554]: pgmap v1807: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.2 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 5188/108294 objects degraded (4.791%); 11948/108294 objects misplaced (11.033%); 17 MiB/s, 268 objects/s recovering 2024-08-30T23:52:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:33 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:33 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:33 smithi173 ceph-mon[20554]: osdmap e808: 12 total, 12 up, 11 in 2024-08-30T23:52:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2293301432' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:52:33.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:33.649 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:33.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:52:33.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:52:33.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:52:33.726 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:52:33.727 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:52:33.728 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:52:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:52:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:52:33.729 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:52:33.729 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:52:33.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:33.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:33.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:33.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:33.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:34.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:34.048 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:34.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.176 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:34.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:34.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.299 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:34.300 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:34.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.424 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:34.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:34.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:34.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:34.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:34.696 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:34.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:34.841 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:34.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:34.982 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:34.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:35.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:35.118 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:35.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:35.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:35.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:35.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:35.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:35.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:35.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:35 smithi136 ceph-mon[20207]: pgmap v1809: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 5188/103725 objects degraded (5.002%); 11164/103725 objects misplaced (10.763%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:52:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:35 smithi173 ceph-mon[20554]: pgmap v1809: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 5188/103725 objects degraded (5.002%); 11164/103725 objects misplaced (10.763%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:52:35.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:35.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:35 smithi106 ceph-mon[16603]: pgmap v1809: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 5188/103725 objects degraded (5.002%); 11164/103725 objects misplaced (10.763%); 15 MiB/s, 247 objects/s recovering 2024-08-30T23:52:35.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:35.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:35.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:35.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:35.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:35.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:35.994 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:36.035 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:36.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:36.162 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:36.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.286 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:36.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:36.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.425 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:36.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:36.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:36.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5188/103725 objects degraded (5.002%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:36 smithi136 ceph-mon[20207]: osdmap e809: 12 total, 12 up, 11 in 2024-08-30T23:52:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:36.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:36.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5188/103725 objects degraded (5.002%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:36 smithi173 ceph-mon[20554]: osdmap e809: 12 total, 12 up, 11 in 2024-08-30T23:52:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5188/103725 objects degraded (5.002%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:36 smithi106 ceph-mon[16603]: osdmap e809: 12 total, 12 up, 11 in 2024-08-30T23:52:36.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:36.686 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:36.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:36.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:36.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:36.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:36.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:37.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:37.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:37.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:37.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:37.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:37.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:37.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:37.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:37.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:37 smithi136 ceph-mon[20207]: pgmap v1810: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 502 op/s; 5214/103242 objects degraded (5.050%); 11011/103242 objects misplaced (10.665%); 15 MiB/s, 246 objects/s recovering 2024-08-30T23:52:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:37 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:52:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:37 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:52:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:37 smithi136 ceph-mon[20207]: osdmap e810: 12 total, 12 up, 11 in 2024-08-30T23:52:37.558 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:37 smithi173 ceph-mon[20554]: pgmap v1810: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 502 op/s; 5214/103242 objects degraded (5.050%); 11011/103242 objects misplaced (10.665%); 15 MiB/s, 246 objects/s recovering 2024-08-30T23:52:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:37 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:52:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:37 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:52:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:37 smithi173 ceph-mon[20554]: osdmap e810: 12 total, 12 up, 11 in 2024-08-30T23:52:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:37 smithi106 ceph-mon[16603]: pgmap v1810: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.1 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 502 op/s; 5214/103242 objects degraded (5.050%); 11011/103242 objects misplaced (10.665%); 15 MiB/s, 246 objects/s recovering 2024-08-30T23:52:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:37 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:52:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:37 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:52:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:37 smithi106 ceph-mon[16603]: osdmap e810: 12 total, 12 up, 11 in 2024-08-30T23:52:37.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:37.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:37.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:37.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:37.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:37.933 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:37.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:38.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:38.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:38.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:38.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:38.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:38.343 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:38.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:38.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:38.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:38.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:38.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:38.717 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:38.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:38.866 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:38.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:38.995 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:38.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:39.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.119 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:39.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:39.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:39.243 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:38 smithi136 ceph-mon[20207]: pgmap v1813: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 532 op/s; 5214/98703 objects degraded (5.283%); 10472/98703 objects misplaced (10.610%); 15 MiB/s, 241 objects/s recovering 2024-08-30T23:52:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:38 smithi136 ceph-mon[20207]: osdmap e811: 12 total, 12 up, 11 in 2024-08-30T23:52:39.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:38 smithi173 ceph-mon[20554]: pgmap v1813: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 532 op/s; 5214/98703 objects degraded (5.283%); 10472/98703 objects misplaced (10.610%); 15 MiB/s, 241 objects/s recovering 2024-08-30T23:52:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:38 smithi173 ceph-mon[20554]: osdmap e811: 12 total, 12 up, 11 in 2024-08-30T23:52:39.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:39.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:38 smithi106 ceph-mon[16603]: pgmap v1813: 37 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 30 active+clean; 2.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 532 op/s; 5214/98703 objects degraded (5.283%); 10472/98703 objects misplaced (10.610%); 15 MiB/s, 241 objects/s recovering 2024-08-30T23:52:39.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:38 smithi106 ceph-mon[16603]: osdmap e811: 12 total, 12 up, 11 in 2024-08-30T23:52:39.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:39.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:39.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:39.659 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:39.761 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:39.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:39.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:39.930 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:39.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:40.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:40.075 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:40.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:40.204 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:40.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:40.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:40.431 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:40.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:40.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:40.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:40.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:40.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:40.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:40.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:40.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:40.971 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:41.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:41.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:41.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:41.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:41.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:41.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:40 smithi136 ceph-mon[20207]: pgmap v1815: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 458 op/s; 5238/95478 objects degraded (5.486%); 9793/95478 objects misplaced (10.257%); 14 MiB/s, 220 objects/s recovering 2024-08-30T23:52:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:40 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5238/95478 objects degraded (5.486%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:41.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:40 smithi173 ceph-mon[20554]: pgmap v1815: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 458 op/s; 5238/95478 objects degraded (5.486%); 9793/95478 objects misplaced (10.257%); 14 MiB/s, 220 objects/s recovering 2024-08-30T23:52:41.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:40 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5238/95478 objects degraded (5.486%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:40 smithi106 ceph-mon[16603]: pgmap v1815: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 458 op/s; 5238/95478 objects degraded (5.486%); 9793/95478 objects misplaced (10.257%); 14 MiB/s, 220 objects/s recovering 2024-08-30T23:52:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:40 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5238/95478 objects degraded (5.486%), 5 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:41.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:41.408 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:41.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:41.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:41.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:41.664 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:41.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:41.791 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:41.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:41.926 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:41.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:42.028 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:42.052 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:42.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:42.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:42.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:42.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:42.303 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:42.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:42.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:42.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:42.566 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:42.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:42.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:42.792 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:42.814 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:42.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:42.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:42.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:43.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:43.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.067 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:43.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:43.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:42 smithi136 ceph-mon[20207]: pgmap v1816: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 5238/92592 objects degraded (5.657%); 9390/92592 objects misplaced (10.141%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:52:43.294 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:43.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:42 smithi173 ceph-mon[20554]: pgmap v1816: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 5238/92592 objects degraded (5.657%); 9390/92592 objects misplaced (10.141%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:52:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:42 smithi106 ceph-mon[16603]: pgmap v1816: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 591 op/s; 5238/92592 objects degraded (5.657%); 9390/92592 objects misplaced (10.141%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:52:43.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:43.442 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:43.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:43.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:43.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:43.696 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:43.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:43.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:43.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:43.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:43.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:44.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:44.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.082 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:44.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:44.233 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:44.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:44.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:44.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:44.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:44.510 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:44.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:44.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:44.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:44.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:44.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:44.889 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:44.890 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:44.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:45.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:45.117 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:45.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:45.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:44 smithi106 ceph-mon[16603]: pgmap v1817: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 5238/87996 objects degraded (5.953%); 8835/87996 objects misplaced (10.040%); 17 MiB/s, 275 objects/s recovering 2024-08-30T23:52:45.239 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:44 smithi106 ceph-mon[16603]: osdmap e812: 12 total, 12 up, 11 in 2024-08-30T23:52:45.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:45.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:44 smithi136 ceph-mon[20207]: pgmap v1817: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 5238/87996 objects degraded (5.953%); 8835/87996 objects misplaced (10.040%); 17 MiB/s, 275 objects/s recovering 2024-08-30T23:52:45.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:44 smithi136 ceph-mon[20207]: osdmap e812: 12 total, 12 up, 11 in 2024-08-30T23:52:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:44 smithi173 ceph-mon[20554]: pgmap v1817: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 5238/87996 objects degraded (5.953%); 8835/87996 objects misplaced (10.040%); 17 MiB/s, 275 objects/s recovering 2024-08-30T23:52:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:44 smithi173 ceph-mon[20554]: osdmap e812: 12 total, 12 up, 11 in 2024-08-30T23:52:45.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:45.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:45.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:45.544 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:45.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:45.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:45.770 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:45.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:45.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:45.919 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:45.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:46.021 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:46.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:46.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:46.180 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:46.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:46.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:45 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5238/87996 objects degraded (5.953%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:46.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:45 smithi136 ceph-mon[20207]: osdmap e813: 12 total, 12 up, 11 in 2024-08-30T23:52:46.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:45 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5238/87996 objects degraded (5.953%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:46.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:45 smithi173 ceph-mon[20554]: osdmap e813: 12 total, 12 up, 11 in 2024-08-30T23:52:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:45 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5238/87996 objects degraded (5.953%), 5 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:46.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:45 smithi106 ceph-mon[16603]: osdmap e813: 12 total, 12 up, 11 in 2024-08-30T23:52:46.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:46.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:46.548 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:46.572 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:46.674 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:46.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:46.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.838 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:46.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:46.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:46.964 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:46.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:47.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:47.103 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:47.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:47.240 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:52:47.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:47.246 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:46 smithi106 ceph-mon[16603]: pgmap v1819: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 5280/87480 objects degraded (6.036%); 8654/87480 objects misplaced (9.893%); 14 MiB/s, 218 objects/s recovering 2024-08-30T23:52:47.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:46 smithi136 ceph-mon[20207]: pgmap v1819: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 5280/87480 objects degraded (6.036%); 8654/87480 objects misplaced (9.893%); 14 MiB/s, 218 objects/s recovering 2024-08-30T23:52:47.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:46 smithi173 ceph-mon[20554]: pgmap v1819: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.8 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 5280/87480 objects degraded (6.036%); 8654/87480 objects misplaced (9.893%); 14 MiB/s, 218 objects/s recovering 2024-08-30T23:52:47.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:47.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:47.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:47.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:47.593 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:47.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:47.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:47.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:47.862 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:47.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:47.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:47.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:48.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:48.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:48.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:48.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:48.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:48.368 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:48.406 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:48.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:48.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:48.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:48.657 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:48.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:48.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:48.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:48.931 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:48.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:49.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:49.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:49.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:49.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:49.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:48 smithi136 ceph-mon[20207]: pgmap v1821: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 5280/82977 objects degraded (6.363%); 7921/82977 objects misplaced (9.546%); 14 MiB/s, 228 objects/s recovering 2024-08-30T23:52:49.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:48 smithi106 ceph-mon[16603]: pgmap v1821: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 5280/82977 objects degraded (6.363%); 7921/82977 objects misplaced (9.546%); 14 MiB/s, 228 objects/s recovering 2024-08-30T23:52:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:48 smithi173 ceph-mon[20554]: pgmap v1821: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 520 op/s; 5280/82977 objects degraded (6.363%); 7921/82977 objects misplaced (9.546%); 14 MiB/s, 228 objects/s recovering 2024-08-30T23:52:49.329 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:49.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:49.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:49.557 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:49.650 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:49.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:49.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:49.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:49.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:49.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:49.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:50.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:50.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:50.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.170 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:50.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:50.231 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:52:50.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:50.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4168439440' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:52:50.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:50.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:50.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4168439440' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:52:50.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4168439440' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:52:50.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:50.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:50.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:50.574 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:50.675 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:50.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:50.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:50.847 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:50.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:50.975 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:50.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:51.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:51.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:51.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:51.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:51.266 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:50 smithi106 ceph-mon[16603]: pgmap v1822: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 5339/79059 objects degraded (6.753%); 7379/79059 objects misplaced (9.334%); 15 MiB/s, 238 objects/s recovering 2024-08-30T23:52:51.266 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:51 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:52:51.266 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4168439440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:52:51.267 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:51 smithi106 ceph-mon[16603]: osdmap e814: 12 total, 12 up, 11 in 2024-08-30T23:52:51.267 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5339/79059 objects degraded (6.753%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:51.267 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:51 smithi106 ceph-mon[16603]: osdmap e815: 12 total, 12 up, 11 in 2024-08-30T23:52:51.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:50 smithi136 ceph-mon[20207]: pgmap v1822: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 5339/79059 objects degraded (6.753%); 7379/79059 objects misplaced (9.334%); 15 MiB/s, 238 objects/s recovering 2024-08-30T23:52:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:50 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:52:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4168439440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:52:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:50 smithi136 ceph-mon[20207]: osdmap e814: 12 total, 12 up, 11 in 2024-08-30T23:52:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:50 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5339/79059 objects degraded (6.753%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:50 smithi136 ceph-mon[20207]: osdmap e815: 12 total, 12 up, 11 in 2024-08-30T23:52:51.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:50 smithi173 ceph-mon[20554]: pgmap v1822: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 5339/79059 objects degraded (6.753%); 7379/79059 objects misplaced (9.334%); 15 MiB/s, 238 objects/s recovering 2024-08-30T23:52:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:50 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:52:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4168439440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:52:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:50 smithi173 ceph-mon[20554]: osdmap e814: 12 total, 12 up, 11 in 2024-08-30T23:52:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:50 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5339/79059 objects degraded (6.753%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:50 smithi173 ceph-mon[20554]: osdmap e815: 12 total, 12 up, 11 in 2024-08-30T23:52:51.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:51.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:51.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:51.504 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:51.606 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:51.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:51.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:51.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:51.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:51.895 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:51.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:51.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:52.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:52.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:52.151 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:52.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:52.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:52.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:52.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:52.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:52.569 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:52.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:52.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:52.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:52.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:52.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:52.993 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:52.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:53.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:53.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:53.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.119 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:53.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:53.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:53 smithi136 ceph-mon[20207]: pgmap v1825: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 5339/75897 objects degraded (7.035%); 6951/75897 objects misplaced (9.158%); 17 MiB/s, 273 objects/s recovering 2024-08-30T23:52:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2958846512' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:52:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:53 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:52:53.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:52:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:53 smithi173 ceph-mon[20554]: pgmap v1825: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 5339/75897 objects degraded (7.035%); 6951/75897 objects misplaced (9.158%); 17 MiB/s, 273 objects/s recovering 2024-08-30T23:52:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2958846512' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:52:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:53 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:52:53.346 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:53.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:53 smithi106 ceph-mon[16603]: pgmap v1825: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 643 op/s; 5339/75897 objects degraded (7.035%); 6951/75897 objects misplaced (9.158%); 17 MiB/s, 273 objects/s recovering 2024-08-30T23:52:53.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2958846512' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:52:53.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:53 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-30T23:52:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.395 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:53.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:53.496 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:53.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:53.622 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:53.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:53.690 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.7 to 1 (10000) 2024-08-30T23:52:53.746 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:53.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:53.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:53.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:53.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:53.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:53.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:54.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:54.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:54.157 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:54.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:54.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:54 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:52:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:54 smithi136 ceph-mon[20207]: osdmap e816: 12 total, 12 up, 11 in 2024-08-30T23:52:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:54 smithi136 ceph-mon[20207]: osdmap e817: 12 total, 12 up, 11 in 2024-08-30T23:52:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.298 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:54.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:54 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:52:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:54 smithi173 ceph-mon[20554]: osdmap e816: 12 total, 12 up, 11 in 2024-08-30T23:52:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:54 smithi173 ceph-mon[20554]: osdmap e817: 12 total, 12 up, 11 in 2024-08-30T23:52:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:54 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-30T23:52:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:54 smithi106 ceph-mon[16603]: osdmap e816: 12 total, 12 up, 11 in 2024-08-30T23:52:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:54 smithi106 ceph-mon[16603]: osdmap e817: 12 total, 12 up, 11 in 2024-08-30T23:52:54.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:54.438 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:54.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:54.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:54.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:54.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:54.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:54.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:54.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:54.962 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:54.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:55.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:55.087 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:55.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:55.223 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:55.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:55 smithi136 ceph-mon[20207]: pgmap v1827: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 5343/73836 objects degraded (7.236%); 6577/73836 objects misplaced (8.908%); 13 MiB/s, 213 objects/s recovering 2024-08-30T23:52:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:55 smithi173 ceph-mon[20554]: pgmap v1827: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 5343/73836 objects degraded (7.236%); 6577/73836 objects misplaced (8.908%); 13 MiB/s, 213 objects/s recovering 2024-08-30T23:52:55.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:55 smithi106 ceph-mon[16603]: pgmap v1827: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.5 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 507 op/s; 5343/73836 objects degraded (7.236%); 6577/73836 objects misplaced (8.908%); 13 MiB/s, 213 objects/s recovering 2024-08-30T23:52:55.360 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:55.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:55.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:55.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:55.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:55.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:55.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:55.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:55.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:55.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:55.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:55.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:56.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:56.121 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:56.223 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:56.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:56.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5343/73836 objects degraded (7.236%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:56.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5343/73836 objects degraded (7.236%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:56.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:56.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5343/73836 objects degraded (7.236%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:52:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:56.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:56.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:56.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:56.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:56.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:56.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:56.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:56.770 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:56.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:56.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:56.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:56.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:57.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:57.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:52:57.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:52:57.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:57.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:57.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:57 smithi173 ceph-mon[20554]: pgmap v1829: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 5347/70887 objects degraded (7.543%); 6177/70887 objects misplaced (8.714%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:52:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:57.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:57 smithi106 ceph-mon[16603]: pgmap v1829: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 5347/70887 objects degraded (7.543%); 6177/70887 objects misplaced (8.714%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:52:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:57.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:57.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:57.411 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.412 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:57.513 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:52:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:57 smithi136 ceph-mon[20207]: pgmap v1829: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 5347/70887 objects degraded (7.543%); 6177/70887 objects misplaced (8.714%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:52:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:57.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:52:57.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:57.677 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:57.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:57.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:57.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:57.932 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:57.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:58.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:52:58.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:52:58.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:58.196 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:58.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:52:58.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:52:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:58 smithi173 ceph-mon[20554]: osdmap e818: 12 total, 12 up, 11 in 2024-08-30T23:52:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:58 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:52:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:58 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:52:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:58 smithi106 ceph-mon[16603]: osdmap e818: 12 total, 12 up, 11 in 2024-08-30T23:52:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:58 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:52:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:58 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:52:58.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:58.449 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:52:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:58 smithi136 ceph-mon[20207]: osdmap e818: 12 total, 12 up, 11 in 2024-08-30T23:52:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:58 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:52:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:58 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:52:58.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:58.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:58.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:58.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:58.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:52:58.825 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:52:58.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:52:58.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:58.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:58.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:52:59.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:59.096 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:59.198 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:52:59.221 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:52:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:52:59.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:52:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:59 smithi173 ceph-mon[20554]: pgmap v1831: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 5347/66498 objects degraded (8.041%); 5489/66498 objects misplaced (8.254%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:52:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:52:59 smithi173 ceph-mon[20554]: osdmap e819: 12 total, 12 up, 11 in 2024-08-30T23:52:59.347 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.348 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:52:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:59 smithi106 ceph-mon[16603]: pgmap v1831: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 5347/66498 objects degraded (8.041%); 5489/66498 objects misplaced (8.254%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:52:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:52:59 smithi106 ceph-mon[16603]: osdmap e819: 12 total, 12 up, 11 in 2024-08-30T23:52:59.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:59.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:59 smithi136 ceph-mon[20207]: pgmap v1831: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.4 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 5347/66498 objects degraded (8.041%); 5489/66498 objects misplaced (8.254%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:52:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:52:59 smithi136 ceph-mon[20207]: osdmap e819: 12 total, 12 up, 11 in 2024-08-30T23:52:59.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:52:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.602 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:59.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:52:59.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:52:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:59.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:52:59.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:52:59.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:52:59.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:52:59.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:52:59.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:52:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:52:59.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:00.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.116 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:00.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:00.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:00.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:00.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:00.379 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:00.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:00.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:00.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:00.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:00.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:00.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.759 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:00.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:00.897 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:00.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:00.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:01.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:01 smithi106 ceph-mon[16603]: pgmap v1833: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 5376/62901 objects degraded (8.547%); 4926/62901 objects misplaced (7.831%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:53:01.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5376/62901 objects degraded (8.547%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:01.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:01.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:01.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:01.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:01.282 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.283 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:01.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:01 smithi173 ceph-mon[20554]: pgmap v1833: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 5376/62901 objects degraded (8.547%); 4926/62901 objects misplaced (7.831%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:53:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5376/62901 objects degraded (8.547%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:01.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:01.407 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:01.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:01.530 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:01.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:01 smithi136 ceph-mon[20207]: pgmap v1833: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 607 op/s; 5376/62901 objects degraded (8.547%); 4926/62901 objects misplaced (7.831%); 17 MiB/s, 269 objects/s recovering 2024-08-30T23:53:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5376/62901 objects degraded (8.547%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:01.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:01.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:01.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:01.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:01.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:01.938 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:01.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:02.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:02.063 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:02.132 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:53:02.133 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:53:02.133 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:53:02.133 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:53:02.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:02.188 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:02.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:02.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:02.329 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:02.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:02.431 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:02.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.457 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:02.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:02.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:02.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:02.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:02.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:02.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:02.831 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:02.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:02.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:02.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:03.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:03.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:03.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:03.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:03.225 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:03.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:03 smithi173 ceph-mon[20554]: pgmap v1834: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.2 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 5376/59613 objects degraded (9.018%); 4450/59613 objects misplaced (7.465%); 15 MiB/s, 240 objects/s recovering 2024-08-30T23:53:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:03 smithi173 ceph-mon[20554]: osdmap e820: 12 total, 12 up, 11 in 2024-08-30T23:53:03.327 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:03 smithi106 ceph-mon[16603]: pgmap v1834: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.2 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 5376/59613 objects degraded (9.018%); 4450/59613 objects misplaced (7.465%); 15 MiB/s, 240 objects/s recovering 2024-08-30T23:53:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:03 smithi106 ceph-mon[16603]: osdmap e820: 12 total, 12 up, 11 in 2024-08-30T23:53:03.358 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:03.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:03.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:03 smithi136 ceph-mon[20207]: pgmap v1834: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.2 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 5376/59613 objects degraded (9.018%); 4450/59613 objects misplaced (7.465%); 15 MiB/s, 240 objects/s recovering 2024-08-30T23:53:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:03 smithi136 ceph-mon[20207]: osdmap e820: 12 total, 12 up, 11 in 2024-08-30T23:53:03.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:03.610 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:03.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:03.738 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:03.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:03.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:03.965 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:03.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:03.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:04.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:04.133 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:04.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:04.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:04.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:04.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:04.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:04.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:04 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:04 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:04 smithi136 ceph-mon[20207]: osdmap e821: 12 total, 12 up, 11 in 2024-08-30T23:53:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:04 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:04 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:04 smithi173 ceph-mon[20554]: osdmap e821: 12 total, 12 up, 11 in 2024-08-30T23:53:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:04 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:04 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:04 smithi106 ceph-mon[16603]: osdmap e821: 12 total, 12 up, 11 in 2024-08-30T23:53:04.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:04.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:04.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:04.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:04.881 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:04.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:04.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:05.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:05.046 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:05.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:05.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:05.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:05.299 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:05.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:05.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:05.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:05.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:05 smithi136 ceph-mon[20207]: pgmap v1837: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.2 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 492 op/s; 5396/57636 objects degraded (9.362%); 4012/57636 objects misplaced (6.961%); 15 MiB/s, 237 objects/s recovering 2024-08-30T23:53:05.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:05 smithi173 ceph-mon[20554]: pgmap v1837: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.2 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 492 op/s; 5396/57636 objects degraded (9.362%); 4012/57636 objects misplaced (6.961%); 15 MiB/s, 237 objects/s recovering 2024-08-30T23:53:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:05 smithi106 ceph-mon[16603]: pgmap v1837: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.2 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 492 op/s; 5396/57636 objects degraded (9.362%); 4012/57636 objects misplaced (6.961%); 15 MiB/s, 237 objects/s recovering 2024-08-30T23:53:05.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:05.676 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:05.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:05.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:05.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:05.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:05.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:06.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:06.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:06.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:06.198 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:06.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.337 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:06.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5396/57636 objects degraded (9.362%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:06.438 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:06.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:06.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5396/57636 objects degraded (9.362%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:06.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:06.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5396/57636 objects degraded (9.362%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.595 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:06.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:06.696 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:06.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.719 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:06.819 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:06.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:06.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:06.991 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:06.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:07.093 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:07.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:07.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:07.256 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:07.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:07.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:07.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:07.508 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:07.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:07 smithi136 ceph-mon[20207]: pgmap v1838: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 5445/54420 objects degraded (10.006%); 3592/54420 objects misplaced (6.601%); 15 MiB/s, 233 objects/s recovering 2024-08-30T23:53:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:07 smithi173 ceph-mon[20554]: pgmap v1838: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 5445/54420 objects degraded (10.006%); 3592/54420 objects misplaced (6.601%); 15 MiB/s, 233 objects/s recovering 2024-08-30T23:53:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:07 smithi106 ceph-mon[16603]: pgmap v1838: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 5445/54420 objects degraded (10.006%); 3592/54420 objects misplaced (6.601%); 15 MiB/s, 233 objects/s recovering 2024-08-30T23:53:07.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:07.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:07.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.774 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:07.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:07.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:07.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:07.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:08.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:08.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:08.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:08.171 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:08.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:08.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:08.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:08.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:08.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:08.555 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:08.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:08.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:08.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:08.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:08.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:08.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:08.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:08.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:09.039 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:09.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:09.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:09.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:09.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:09.326 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:09.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:09.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:09 smithi136 ceph-mon[20207]: pgmap v1839: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.0 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 5430/49410 objects degraded (10.990%); 2959/49410 objects misplaced (5.989%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:53:09.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:09.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:09 smithi173 ceph-mon[20554]: pgmap v1839: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.0 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 5430/49410 objects degraded (10.990%); 2959/49410 objects misplaced (5.989%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:53:09.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:09 smithi106 ceph-mon[16603]: pgmap v1839: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 1.0 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 561 op/s; 5430/49410 objects degraded (10.990%); 2959/49410 objects misplaced (5.989%); 15 MiB/s, 242 objects/s recovering 2024-08-30T23:53:09.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:09.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:09.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:09.835 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:09.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:09.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:09.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:10.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:10.088 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:10.190 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:10.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:10.313 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:10.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:10.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:10.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:10 smithi136 ceph-mon[20207]: osdmap e822: 12 total, 12 up, 11 in 2024-08-30T23:53:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:10 smithi173 ceph-mon[20554]: osdmap e822: 12 total, 12 up, 11 in 2024-08-30T23:53:10.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:10 smithi106 ceph-mon[16603]: osdmap e822: 12 total, 12 up, 11 in 2024-08-30T23:53:10.612 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:10.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:10.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:10.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:10.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:10.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:10.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:11.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:11.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:11.131 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:11.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:11.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:11.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:11.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:11.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:11.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:11.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:11 smithi136 ceph-mon[20207]: pgmap v1841: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 954 MiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 5466/45780 objects degraded (11.940%); 2451/45780 objects misplaced (5.354%); 13 MiB/s, 227 objects/s recovering 2024-08-30T23:53:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5466/45780 objects degraded (11.940%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:11 smithi136 ceph-mon[20207]: osdmap e823: 12 total, 12 up, 11 in 2024-08-30T23:53:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:11 smithi173 ceph-mon[20554]: pgmap v1841: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 954 MiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 5466/45780 objects degraded (11.940%); 2451/45780 objects misplaced (5.354%); 13 MiB/s, 227 objects/s recovering 2024-08-30T23:53:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5466/45780 objects degraded (11.940%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:11 smithi173 ceph-mon[20554]: osdmap e823: 12 total, 12 up, 11 in 2024-08-30T23:53:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:11 smithi106 ceph-mon[16603]: pgmap v1841: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 954 MiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 5466/45780 objects degraded (11.940%); 2451/45780 objects misplaced (5.354%); 13 MiB/s, 227 objects/s recovering 2024-08-30T23:53:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5466/45780 objects degraded (11.940%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:11.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:11 smithi106 ceph-mon[16603]: osdmap e823: 12 total, 12 up, 11 in 2024-08-30T23:53:11.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:11.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:11.707 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-08-30T23:53:11.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:11.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:11.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:11.926 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:11.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:12.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.065 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:12.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:12.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:12.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:12.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:12.314 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:12.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:53:12.416 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:12.502 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:12.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:12.572 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:12.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:12.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:12.741 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:12.763 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:12.865 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:12.887 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:12.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:12.989 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:13.046 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:13.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:13.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:13.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:53:13.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:13.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:13.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:13.422 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:13.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:13 smithi106 ceph-mon[16603]: pgmap v1843: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 889 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 5466/42636 objects degraded (12.820%); 2275/42636 objects misplaced (5.336%); 13 MiB/s, 264 objects/s recovering 2024-08-30T23:53:13.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:13.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:13.524 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:13.548 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:13.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:13.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:13.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:13.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:13 smithi136 ceph-mon[20207]: pgmap v1843: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 889 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 5466/42636 objects degraded (12.820%); 2275/42636 objects misplaced (5.336%); 13 MiB/s, 264 objects/s recovering 2024-08-30T23:53:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:13.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:13 smithi173 ceph-mon[20554]: pgmap v1843: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 889 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 624 op/s; 5466/42636 objects degraded (12.820%); 2275/42636 objects misplaced (5.336%); 13 MiB/s, 264 objects/s recovering 2024-08-30T23:53:13.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:13.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:13.833 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:13.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:13.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:13.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:14.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:14.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:14.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:14.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:14.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:14.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:14.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:14.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:14.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:14.587 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:14.636 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:53:14.638 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.8 to 1 (10000) 2024-08-30T23:53:14.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:14.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:14.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:53:14.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:53:14.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:14.741 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1411685012' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:14.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:14.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3786126988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:53:14.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:14 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1411685012' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3786126988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:53:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:14 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1411685012' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3786126988' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:53:14.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:14 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-30T23:53:14.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:14.874 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:14.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:14.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:14.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:15.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:15.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:15.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:15.254 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:15.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:15.380 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:15.482 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:15.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:15.546 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:53:15.546 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:53:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:53:15.547 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:53:15.547 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:53:15.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:15.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.630 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:15.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:15.770 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:15 smithi136 ceph-mon[20207]: pgmap v1844: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 857 MiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 5190/41097 objects degraded (12.629%); 2275/41097 objects misplaced (5.536%); 10 MiB/s, 252 objects/s recovering 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:15 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:15 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:15 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:15 smithi136 ceph-mon[20207]: osdmap e824: 12 total, 12 up, 11 in 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:15 smithi106 ceph-mon[16603]: pgmap v1844: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 857 MiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 5190/41097 objects degraded (12.629%); 2275/41097 objects misplaced (5.536%); 10 MiB/s, 252 objects/s recovering 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:15 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:15 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:15 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:53:15.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:15 smithi106 ceph-mon[16603]: osdmap e824: 12 total, 12 up, 11 in 2024-08-30T23:53:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:15 smithi173 ceph-mon[20554]: pgmap v1844: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 857 MiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 5190/41097 objects degraded (12.629%); 2275/41097 objects misplaced (5.536%); 10 MiB/s, 252 objects/s recovering 2024-08-30T23:53:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:15 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:15 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:53:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:15 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-30T23:53:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:15 smithi173 ceph-mon[20554]: osdmap e824: 12 total, 12 up, 11 in 2024-08-30T23:53:15.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:15.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:15.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:15.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:16.025 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:16.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:16.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:16.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:16.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:16.405 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:16.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:16.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:16.555 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5190/41097 objects degraded (12.629%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:16 smithi106 ceph-mon[16603]: osdmap e825: 12 total, 12 up, 11 in 2024-08-30T23:53:16.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:16.681 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:16.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5190/41097 objects degraded (12.629%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:16 smithi136 ceph-mon[20207]: osdmap e825: 12 total, 12 up, 11 in 2024-08-30T23:53:16.806 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:16.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-08-30T23:53:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5190/41097 objects degraded (12.629%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:16 smithi173 ceph-mon[20554]: osdmap e825: 12 total, 12 up, 11 in 2024-08-30T23:53:16.840 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:16.908 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:16.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:16.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:17.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:17.055 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:17.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:17.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:17.295 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:17.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.319 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:17.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:53:17.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:17.453 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:17.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.590 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:17.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:17.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:17.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:17 smithi136 ceph-mon[20207]: pgmap v1846: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 783 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 442 op/s; 5225/37566 objects degraded (13.909%); 1950/37566 objects misplaced (5.191%); 5.6 MiB/s, 229 objects/s recovering 2024-08-30T23:53:17.800 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:17 smithi106 ceph-mon[16603]: pgmap v1846: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 783 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 442 op/s; 5225/37566 objects degraded (13.909%); 1950/37566 objects misplaced (5.191%); 5.6 MiB/s, 229 objects/s recovering 2024-08-30T23:53:17.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:17.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:17 smithi173 ceph-mon[20554]: pgmap v1846: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 783 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 442 op/s; 5225/37566 objects degraded (13.909%); 1950/37566 objects misplaced (5.191%); 5.6 MiB/s, 229 objects/s recovering 2024-08-30T23:53:17.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:17.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:17.983 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:17.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:18.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:18.108 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:18.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:18.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:18.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:18.376 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:18.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:18.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:18.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:18.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:18.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:18.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:18.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:18.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:18.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:18.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:18.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:19.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:19.134 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:19.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:19.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:19.369 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:19.400 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:19.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:19.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:19.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:19.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:19.761 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:19 smithi136 ceph-mon[20207]: pgmap v1848: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 690 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 4972/33120 objects degraded (15.012%); 1752/33120 objects misplaced (5.290%); 6.1 MiB/s, 327 objects/s recovering 2024-08-30T23:53:19.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:19.812 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:19 smithi106 ceph-mon[16603]: pgmap v1848: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 690 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 4972/33120 objects degraded (15.012%); 1752/33120 objects misplaced (5.290%); 6.1 MiB/s, 327 objects/s recovering 2024-08-30T23:53:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:19 smithi173 ceph-mon[20554]: pgmap v1848: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 690 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 4972/33120 objects degraded (15.012%); 1752/33120 objects misplaced (5.290%); 6.1 MiB/s, 327 objects/s recovering 2024-08-30T23:53:19.850 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:53:19.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:19.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:19.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:20.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:20.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:20.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:20.183 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:20.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:20.322 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:20.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:20.454 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:20.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:20.607 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1563271139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:20.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:20.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1563271139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:20.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1563271139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:20.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:20.857 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:20.959 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:20.996 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:20.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:21.045 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:53:21.046 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:53:21.047 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:53:21.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:53:21.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:53:21.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:53:21.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:53:21.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:53:21.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:53:21.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:21.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:21.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.256 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:21.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:21.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:21.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:21.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:21.509 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:21.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:21.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:21.643 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:21.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:21.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:21 smithi136 ceph-mon[20207]: pgmap v1849: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 614 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 5025/29466 objects degraded (17.054%); 1383/29466 objects misplaced (4.694%); 6.3 MiB/s, 301 objects/s recovering 2024-08-30T23:53:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5025/29466 objects degraded (17.054%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:21 smithi173 ceph-mon[20554]: pgmap v1849: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 614 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 5025/29466 objects degraded (17.054%); 1383/29466 objects misplaced (4.694%); 6.3 MiB/s, 301 objects/s recovering 2024-08-30T23:53:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5025/29466 objects degraded (17.054%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:21 smithi106 ceph-mon[16603]: pgmap v1849: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 614 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 5025/29466 objects degraded (17.054%); 1383/29466 objects misplaced (4.694%); 6.3 MiB/s, 301 objects/s recovering 2024-08-30T23:53:21.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5025/29466 objects degraded (17.054%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:21.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:21.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:21.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:21.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:22.019 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:22.121 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:22.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:22.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:22.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:22.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.415 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:22.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:22.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:22.554 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:22.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:22.680 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:22.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.806 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:22.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:22.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:22.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:22.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:23.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:23.057 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:23.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:23.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:23.282 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:23.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:23.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:23.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:23.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.566 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:23.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:23.699 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:23 smithi136 ceph-mon[20207]: pgmap v1850: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 548 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 5025/26271 objects degraded (19.128%); 1182/26271 objects misplaced (4.499%); 7.8 MiB/s, 342 objects/s recovering 2024-08-30T23:53:23.801 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:23.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:23 smithi173 ceph-mon[20554]: pgmap v1850: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 548 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 5025/26271 objects degraded (19.128%); 1182/26271 objects misplaced (4.499%); 7.8 MiB/s, 342 objects/s recovering 2024-08-30T23:53:23.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:23.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:23 smithi106 ceph-mon[16603]: pgmap v1850: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 548 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 617 op/s; 5025/26271 objects degraded (19.128%); 1182/26271 objects misplaced (4.499%); 7.8 MiB/s, 342 objects/s recovering 2024-08-30T23:53:23.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:23.953 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:23.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:24.055 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:24.088 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:24.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:24.211 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:24.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:24.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:24.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:24.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:24.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:24.591 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:24.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:24.718 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:24.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:24.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:24.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:24.969 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:24.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:25.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:25.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:25.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:25.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:25.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:25.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:25.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:25.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:25.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:25.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:25.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:25.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:25 smithi136 ceph-mon[20207]: pgmap v1851: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 518 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 463 op/s; 4766/24846 objects degraded (19.182%); 1182/24846 objects misplaced (4.757%); 4.9 MiB/s, 263 objects/s recovering 2024-08-30T23:53:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:25 smithi136 ceph-mon[20207]: osdmap e826: 12 total, 12 up, 11 in 2024-08-30T23:53:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:25 smithi173 ceph-mon[20554]: pgmap v1851: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 518 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 463 op/s; 4766/24846 objects degraded (19.182%); 1182/24846 objects misplaced (4.757%); 4.9 MiB/s, 263 objects/s recovering 2024-08-30T23:53:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:25 smithi173 ceph-mon[20554]: osdmap e826: 12 total, 12 up, 11 in 2024-08-30T23:53:25.846 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:25 smithi106 ceph-mon[16603]: pgmap v1851: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 518 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 463 op/s; 4766/24846 objects degraded (19.182%); 1182/24846 objects misplaced (4.757%); 4.9 MiB/s, 263 objects/s recovering 2024-08-30T23:53:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:25.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:25 smithi106 ceph-mon[16603]: osdmap e826: 12 total, 12 up, 11 in 2024-08-30T23:53:25.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:25.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:25.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.024 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:26.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:26.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:26.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:26.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:26.303 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:26.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:26.451 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:26.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:26.578 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:26.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:26.703 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:26.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:26.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:26 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:26 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:26 smithi173 ceph-mon[20554]: osdmap e827: 12 total, 12 up, 11 in 2024-08-30T23:53:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4766/24846 objects degraded (19.182%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:26.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:26 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:26 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:26 smithi106 ceph-mon[16603]: osdmap e827: 12 total, 12 up, 11 in 2024-08-30T23:53:26.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4766/24846 objects degraded (19.182%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:26.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:26.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:26.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:26 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:26 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:26 smithi136 ceph-mon[20207]: osdmap e827: 12 total, 12 up, 11 in 2024-08-30T23:53:27.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4766/24846 objects degraded (19.182%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:27.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:27.094 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:27.196 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:27.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:27.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:27.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:27.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:27.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:27.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:27.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:27.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:27.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:27.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:27 smithi173 ceph-mon[20554]: pgmap v1854: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 438 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 4816/20991 objects degraded (22.943%); 822/20991 objects misplaced (3.916%); 6.4 MiB/s, 277 objects/s recovering 2024-08-30T23:53:27.847 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:27.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:27 smithi106 ceph-mon[16603]: pgmap v1854: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 438 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 4816/20991 objects degraded (22.943%); 822/20991 objects misplaced (3.916%); 6.4 MiB/s, 277 objects/s recovering 2024-08-30T23:53:27.891 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:27.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:27.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:28.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:28.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:27 smithi136 ceph-mon[20207]: pgmap v1854: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 438 MiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 4816/20991 objects degraded (22.943%); 822/20991 objects misplaced (3.916%); 6.4 MiB/s, 277 objects/s recovering 2024-08-30T23:53:28.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:28.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:28.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:28.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:28.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.420 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:28.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:28.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:28.541 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:28.543 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:28.624 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:28.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:28.669 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:28.676 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:28.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:28.793 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:28.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:28.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:28.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:28.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:29.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:29.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:29.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:29.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:29.307 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.352 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:29.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:29.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:29.523 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:29.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:29.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:29.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:29 smithi106 ceph-mon[16603]: pgmap v1855: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 337 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 4513/16137 objects degraded (27.967%); 740/16137 objects misplaced (4.586%); 4.6 MiB/s, 320 objects/s recovering 2024-08-30T23:53:29.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:29.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:29.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:29 smithi106 ceph-mon[16603]: osdmap e828: 12 total, 12 up, 11 in 2024-08-30T23:53:29.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:29.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.778 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:29.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:29 smithi173 ceph-mon[20554]: pgmap v1855: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 337 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 4513/16137 objects degraded (27.967%); 740/16137 objects misplaced (4.586%); 4.6 MiB/s, 320 objects/s recovering 2024-08-30T23:53:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:29 smithi173 ceph-mon[20554]: osdmap e828: 12 total, 12 up, 11 in 2024-08-30T23:53:29.864 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:53:29.864 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:53:29.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:29.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:29.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:30.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:30.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:29 smithi136 ceph-mon[20207]: pgmap v1855: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 337 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 555 op/s; 4513/16137 objects degraded (27.967%); 740/16137 objects misplaced (4.586%); 4.6 MiB/s, 320 objects/s recovering 2024-08-30T23:53:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:30.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:29 smithi136 ceph-mon[20207]: osdmap e828: 12 total, 12 up, 11 in 2024-08-30T23:53:30.127 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:30.154 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:30.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:30.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:30.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:30.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:30.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:30.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:30.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:53:30.607 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.9 to 1 (10000) 2024-08-30T23:53:30.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:30.649 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:30.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:30.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:30.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:30 smithi106 ceph-mon[16603]: osdmap e829: 12 total, 12 up, 11 in 2024-08-30T23:53:30.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2218022063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:53:30.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:30 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:53:30.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:30.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:30.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:30.972 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:53:31.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:31.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:30 smithi136 ceph-mon[20207]: osdmap e829: 12 total, 12 up, 11 in 2024-08-30T23:53:31.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2218022063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:53:31.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:30 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:53:31.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:31.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:30 smithi173 ceph-mon[20554]: osdmap e829: 12 total, 12 up, 11 in 2024-08-30T23:53:31.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2218022063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:53:31.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:30 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-30T23:53:31.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:31.182 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:31.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:31.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:31.407 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:31.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:31.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:31.558 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:31.660 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:31.685 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:31.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:31.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: pgmap v1858: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 256 MiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 4536/12270 objects degraded (36.968%); 372/12270 objects misplaced (3.032%); 5.1 MiB/s, 340 objects/s recovering 2024-08-30T23:53:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:53:31.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: osdmap e830: 12 total, 12 up, 11 in 2024-08-30T23:53:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2076070291' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:53:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4536/12270 objects degraded (36.968%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2076070291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:53:31.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:31 smithi106 ceph-mon[16603]: osdmap e831: 12 total, 12 up, 11 in 2024-08-30T23:53:31.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:31.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:31.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: pgmap v1858: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 256 MiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 4536/12270 objects degraded (36.968%); 372/12270 objects misplaced (3.032%); 5.1 MiB/s, 340 objects/s recovering 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: osdmap e830: 12 total, 12 up, 11 in 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2076070291' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4536/12270 objects degraded (36.968%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2076070291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:53:32.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:31 smithi136 ceph-mon[20207]: osdmap e831: 12 total, 12 up, 11 in 2024-08-30T23:53:32.041 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: pgmap v1858: 37 pgs: 1 active+recovering+remapped, 5 active+recovering+undersized+degraded+remapped, 31 active+clean; 256 MiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 554 op/s; 4536/12270 objects degraded (36.968%); 372/12270 objects misplaced (3.032%); 5.1 MiB/s, 340 objects/s recovering 2024-08-30T23:53:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-30T23:53:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: osdmap e830: 12 total, 12 up, 11 in 2024-08-30T23:53:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2076070291' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:53:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4536/12270 objects degraded (36.968%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:32.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2076070291' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:53:32.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:31 smithi173 ceph-mon[20554]: osdmap e831: 12 total, 12 up, 11 in 2024-08-30T23:53:32.082 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:32.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:32.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:32.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:32.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:32.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:32.350 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:32.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:32.476 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:32.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.617 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:32.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:32.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:32.744 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:32.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:32.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:32.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:32.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:32.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:32.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:33.096 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:33.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:33.221 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:33.247 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:33.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:33.371 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:33.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:33.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:33.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:53:33.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:33.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:33.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:33.766 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:33 smithi106 ceph-mon[16603]: pgmap v1861: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 198 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4531/9507 objects degraded (47.660%); 372/9507 objects misplaced (3.913%); 5.4 MiB/s, 341 objects/s recovering 2024-08-30T23:53:33.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:33.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:33.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:33.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.027 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:34.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:33 smithi136 ceph-mon[20207]: pgmap v1861: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 198 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4531/9507 objects degraded (47.660%); 372/9507 objects misplaced (3.913%); 5.4 MiB/s, 341 objects/s recovering 2024-08-30T23:53:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:33 smithi173 ceph-mon[20554]: pgmap v1861: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 198 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 551 op/s; 4531/9507 objects degraded (47.660%); 372/9507 objects misplaced (3.913%); 5.4 MiB/s, 341 objects/s recovering 2024-08-30T23:53:34.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:34.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:34.279 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:34.305 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:34.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:34.431 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:34.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:34.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:34.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-08-30T23:53:34.627 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:34.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:34.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:34 smithi106 ceph-mon[16603]: 7.4 scrub starts 2024-08-30T23:53:34.685 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:34 smithi106 ceph-mon[16603]: 7.4 scrub ok 2024-08-30T23:53:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.737 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:34.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:34.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:34.887 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:34.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:34.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:35.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:34 smithi136 ceph-mon[20207]: 7.4 scrub starts 2024-08-30T23:53:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:34 smithi136 ceph-mon[20207]: 7.4 scrub ok 2024-08-30T23:53:35.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:34 smithi173 ceph-mon[20554]: 7.4 scrub starts 2024-08-30T23:53:35.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:34 smithi173 ceph-mon[20554]: 7.4 scrub ok 2024-08-30T23:53:35.113 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:35.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:35.181 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:35.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.261 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:35.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:35.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:35.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.386 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:35.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:35.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:35.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:35.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:35.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:35.734 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:35.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:35.795 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 87113 objects 2024-08-30T23:53:35.796 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :166.276 2024-08-30T23:53:35.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:35 smithi106 ceph-mon[16603]: pgmap v1862: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 169 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 512 op/s; 4147/8112 objects degraded (51.122%); 372/8112 objects misplaced (4.586%); 4.1 MiB/s, 335 objects/s recovering 2024-08-30T23:53:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:35 smithi106 ceph-mon[16603]: 7.1 scrub starts 2024-08-30T23:53:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:35 smithi106 ceph-mon[16603]: 7.1 scrub ok 2024-08-30T23:53:35.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1737426623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:35.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:35.878 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:53:35.878 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:53:35.887 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:35.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:35.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:36.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:36.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:35 smithi136 ceph-mon[20207]: pgmap v1862: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 169 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 512 op/s; 4147/8112 objects degraded (51.122%); 372/8112 objects misplaced (4.586%); 4.1 MiB/s, 335 objects/s recovering 2024-08-30T23:53:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:35 smithi136 ceph-mon[20207]: 7.1 scrub starts 2024-08-30T23:53:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:35 smithi136 ceph-mon[20207]: 7.1 scrub ok 2024-08-30T23:53:36.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1737426623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:36.064 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:53:36.065 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:53:36.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:35 smithi173 ceph-mon[20554]: pgmap v1862: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 169 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 512 op/s; 4147/8112 objects degraded (51.122%); 372/8112 objects misplaced (4.586%); 4.1 MiB/s, 335 objects/s recovering 2024-08-30T23:53:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:35 smithi173 ceph-mon[20554]: 7.1 scrub starts 2024-08-30T23:53:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:35 smithi173 ceph-mon[20554]: 7.1 scrub ok 2024-08-30T23:53:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1737426623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:36.097 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:53:36.097 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:53:36.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:36.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:53:36.124 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:53:36.125 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:53:36.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:36.154 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:53:36.154 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:53:36.179 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:53:36.180 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:53:36.213 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:53:36.213 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:53:36.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:36.240 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:53:36.240 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:53:36.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:36.264 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:53:36.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:53:36.288 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:53:36.288 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:53:36.316 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:53:36.316 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:53:36.344 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:53:36.345 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:53:36.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:36.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:53:36.372 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:53:36.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:36.486 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:36.511 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:36.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:36.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:36.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:36.758 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:36.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:36.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:36.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:36.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:37.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:36 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:53:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:36 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:53:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:36 smithi136 ceph-mon[20207]: osdmap e832: 12 total, 12 up, 11 in 2024-08-30T23:53:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4147/8112 objects degraded (51.122%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:37.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:36 smithi136 ceph-mon[20207]: osdmap e833: 12 total, 12 up, 11 in 2024-08-30T23:53:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:36 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:53:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:36 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:53:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:36 smithi173 ceph-mon[20554]: osdmap e832: 12 total, 12 up, 11 in 2024-08-30T23:53:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4147/8112 objects degraded (51.122%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:36 smithi173 ceph-mon[20554]: osdmap e833: 12 total, 12 up, 11 in 2024-08-30T23:53:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:36 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:53:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:36 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:53:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:36 smithi106 ceph-mon[16603]: osdmap e832: 12 total, 12 up, 11 in 2024-08-30T23:53:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4147/8112 objects degraded (51.122%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:36 smithi106 ceph-mon[16603]: osdmap e833: 12 total, 12 up, 11 in 2024-08-30T23:53:37.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:37.129 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:37.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:37.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:37.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:37.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:37.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:37.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:37.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:37.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:37.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:37.746 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:37.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:37.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:37.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:37.992 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:37.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:38.005 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-08-30T23:53:38.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-08-30T23:53:38.028 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:37 smithi106 ceph-mon[16603]: pgmap v1864: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 86 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 454 op/s; 4072/4089 objects degraded (99.584%); 43/4089 objects misplaced (1.052%); 3.4 MiB/s, 306 objects/s recovering 2024-08-30T23:53:38.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:37 smithi136 ceph-mon[20207]: pgmap v1864: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 86 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 454 op/s; 4072/4089 objects degraded (99.584%); 43/4089 objects misplaced (1.052%); 3.4 MiB/s, 306 objects/s recovering 2024-08-30T23:53:38.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:37 smithi173 ceph-mon[20554]: pgmap v1864: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 86 MiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 454 op/s; 4072/4089 objects degraded (99.584%); 43/4089 objects misplaced (1.052%); 3.4 MiB/s, 306 objects/s recovering 2024-08-30T23:53:38.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:38.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:38.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:38.241 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:38.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:38.366 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:38.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:38.489 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:38.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:38.612 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:38.714 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:38.736 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:38.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:38.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:38.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:38.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:38.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:38.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:39.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:38 smithi136 ceph-mon[20207]: 7.8 scrub starts 2024-08-30T23:53:39.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:38 smithi136 ceph-mon[20207]: 7.8 scrub ok 2024-08-30T23:53:39.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:38 smithi173 ceph-mon[20554]: 7.8 scrub starts 2024-08-30T23:53:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:38 smithi173 ceph-mon[20554]: 7.8 scrub ok 2024-08-30T23:53:39.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:39.105 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:38 smithi106 ceph-mon[16603]: 7.8 scrub starts 2024-08-30T23:53:39.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:38 smithi106 ceph-mon[16603]: 7.8 scrub ok 2024-08-30T23:53:39.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:39.228 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:39.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:39.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:39.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:39.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:39.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:39.601 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:39.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:39.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:39.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:39.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:39.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:39.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:40.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:39 smithi136 ceph-mon[20207]: pgmap v1866: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 22 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 3515/1038 objects degraded (338.632%); 43/1038 objects misplaced (4.143%); 3.0 MiB/s, 411 objects/s recovering 2024-08-30T23:53:40.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:39 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:53:40.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:39 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:53:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:39 smithi173 ceph-mon[20554]: pgmap v1866: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 22 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 3515/1038 objects degraded (338.632%); 43/1038 objects misplaced (4.143%); 3.0 MiB/s, 411 objects/s recovering 2024-08-30T23:53:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:39 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:53:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:39 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:53:40.077 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:40.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:39 smithi106 ceph-mon[16603]: pgmap v1866: 37 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+degraded+remapped, 32 active+clean; 22 MiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 3515/1038 objects degraded (338.632%); 43/1038 objects misplaced (4.143%); 3.0 MiB/s, 411 objects/s recovering 2024-08-30T23:53:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:39 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:53:40.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:39 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:53:40.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:40.222 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:40.324 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:40.347 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:40.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:40.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:40.470 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:40.556 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:40.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:40.594 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.596 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:40.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:40.719 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:40.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:40.845 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:40.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:40.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:40.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:41.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:40 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:53:41.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:40 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-30T23:53:41.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:40 smithi136 ceph-mon[20207]: osdmap e834: 12 total, 12 up, 11 in 2024-08-30T23:53:41.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:41.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:40 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:53:41.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:40 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-30T23:53:41.076 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:40 smithi173 ceph-mon[20554]: osdmap e834: 12 total, 12 up, 11 in 2024-08-30T23:53:41.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:40 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:53:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:40 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-30T23:53:41.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:40 smithi106 ceph-mon[16603]: osdmap e834: 12 total, 12 up, 11 in 2024-08-30T23:53:41.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:41.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:41.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:41.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:41.442 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:41.464 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:41.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:41.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:41.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:41.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:41.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:41.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:41.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:41.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:41.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: pgmap v1868: 37 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 3013/6 objects degraded (50216.667%); 43/6 objects misplaced (716.667%); 3.4 MiB/s, 491 objects/s recovering 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: osdmap e835: 12 total, 12 up, 11 in 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3013/6 objects degraded (50216.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:53:42.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:41 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:53:42.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: pgmap v1868: 37 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 3013/6 objects degraded (50216.667%); 43/6 objects misplaced (716.667%); 3.4 MiB/s, 491 objects/s recovering 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: osdmap e835: 12 total, 12 up, 11 in 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3013/6 objects degraded (50216.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:53:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:53:42.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:41 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:53:42.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: pgmap v1868: 37 pgs: 1 active+recovering+remapped, 3 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 3013/6 objects degraded (50216.667%); 43/6 objects misplaced (716.667%); 3.4 MiB/s, 491 objects/s recovering 2024-08-30T23:53:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: osdmap e835: 12 total, 12 up, 11 in 2024-08-30T23:53:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3013/6 objects degraded (50216.667%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:53:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:53:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:53:42.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:41 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:53:42.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:42.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:42.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:42.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:42.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:42.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:42.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:42.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:42.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:42.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:42.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:42.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:42.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:42.947 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:42.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 7.0 scrub starts 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 7.0 scrub ok 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:53:43.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:42 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:53:43.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:43.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 7.0 scrub starts 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 7.0 scrub ok 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:53:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:42 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:53:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:53:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:53:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:53:43.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:53:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 7.0 scrub starts 2024-08-30T23:53:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 7.0 scrub ok 2024-08-30T23:53:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:53:43.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:42 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:53:43.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:43.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.196 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:43.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:43.318 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:43.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:43.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:43.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:43.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:43.668 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:43.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:43.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:43.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:43.815 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: pgmap v1870: 37 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 226 op/s; 2739/6 objects degraded (45650.000%); 43/6 objects misplaced (716.667%); 0 B/s, 397 objects/s recovering 2024-08-30T23:53:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:53:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:53:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:53:43.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:43 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:53:43.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:43.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:53:43.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:43.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: pgmap v1870: 37 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 226 op/s; 2739/6 objects degraded (45650.000%); 43/6 objects misplaced (716.667%); 0 B/s, 397 objects/s recovering 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:53:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:43 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:53:44.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:44.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: pgmap v1870: 37 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 226 op/s; 2739/6 objects degraded (45650.000%); 43/6 objects misplaced (716.667%); 0 B/s, 397 objects/s recovering 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:53:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:43 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:53:44.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:44.187 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:44.211 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:44.233 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:44.261 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:44.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:44.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:44.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:44.434 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:44.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:44.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:44.636 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.10 to 1 (10000) 2024-08-30T23:53:44.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:44.683 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:44.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:44.809 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:44.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:44.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:44.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:45.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:44 smithi106 ceph-mon[16603]: osdmap e836: 12 total, 12 up, 11 in 2024-08-30T23:53:45.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:44 smithi106 ceph-mon[16603]: 7.e scrub starts 2024-08-30T23:53:45.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:44 smithi106 ceph-mon[16603]: 7.e scrub ok 2024-08-30T23:53:45.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3106780875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:53:45.010 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:44 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:53:45.035 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:45.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:44 smithi136 ceph-mon[20207]: osdmap e836: 12 total, 12 up, 11 in 2024-08-30T23:53:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:44 smithi136 ceph-mon[20207]: 7.e scrub starts 2024-08-30T23:53:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:44 smithi136 ceph-mon[20207]: 7.e scrub ok 2024-08-30T23:53:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3106780875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:53:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:44 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:53:45.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:44 smithi173 ceph-mon[20554]: osdmap e836: 12 total, 12 up, 11 in 2024-08-30T23:53:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:44 smithi173 ceph-mon[20554]: 7.e scrub starts 2024-08-30T23:53:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:44 smithi173 ceph-mon[20554]: 7.e scrub ok 2024-08-30T23:53:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3106780875' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:53:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:44 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-30T23:53:45.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:45.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:45.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:53:45.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:45.309 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:45.411 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:45.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:45.536 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:45.559 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:45.571 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:53:45.572 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:53:45.572 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:53:45.572 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:53:45.572 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:53:45.638 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_5' removed 2024-08-30T23:53:45.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:45.703 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:45.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:45.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:45.928 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:45.949 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:45.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:45 smithi136 ceph-mon[20207]: pgmap v1872: 37 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 2739/6 objects degraded (45650.000%); 43/6 objects misplaced (716.667%); 0 B/s, 237 objects/s recovering 2024-08-30T23:53:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:45 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:53:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:45 smithi136 ceph-mon[20207]: osdmap e837: 12 total, 12 up, 11 in 2024-08-30T23:53:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3714303631' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:53:46.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.078 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:46.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:46.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:45 smithi173 ceph-mon[20554]: pgmap v1872: 37 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 2739/6 objects degraded (45650.000%); 43/6 objects misplaced (716.667%); 0 B/s, 237 objects/s recovering 2024-08-30T23:53:46.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:45 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:53:46.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:45 smithi173 ceph-mon[20554]: osdmap e837: 12 total, 12 up, 11 in 2024-08-30T23:53:46.079 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3714303631' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:53:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:45 smithi106 ceph-mon[16603]: pgmap v1872: 37 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 33 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 57 op/s; 2739/6 objects degraded (45650.000%); 43/6 objects misplaced (716.667%); 0 B/s, 237 objects/s recovering 2024-08-30T23:53:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:45 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-30T23:53:46.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:45 smithi106 ceph-mon[16603]: osdmap e837: 12 total, 12 up, 11 in 2024-08-30T23:53:46.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3714303631' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:53:46.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:46.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:46.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:46.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:46.375 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:53:46.376 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:53:46.377 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-08-30T23:53:46.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-08-30T23:53:46.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:46.456 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:46.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:46.580 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:46.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.704 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:46.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:46.708 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-30T23:53:46.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:53:46.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:46.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:46.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:46.932 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:46.954 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:46.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:47.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3714303631' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:53:47.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:46 smithi106 ceph-mon[16603]: osdmap e838: 12 total, 12 up, 11 in 2024-08-30T23:53:47.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2739/6 objects degraded (45650.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:47.036 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:46 smithi106 ceph-mon[16603]: osdmap e839: 12 total, 12 up, 11 in 2024-08-30T23:53:47.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3714303631' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:53:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:46 smithi136 ceph-mon[20207]: osdmap e838: 12 total, 12 up, 11 in 2024-08-30T23:53:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2739/6 objects degraded (45650.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:46 smithi136 ceph-mon[20207]: osdmap e839: 12 total, 12 up, 11 in 2024-08-30T23:53:47.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3714303631' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:53:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:46 smithi173 ceph-mon[20554]: osdmap e838: 12 total, 12 up, 11 in 2024-08-30T23:53:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2739/6 objects degraded (45650.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-30T23:53:47.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:46 smithi173 ceph-mon[20554]: osdmap e839: 12 total, 12 up, 11 in 2024-08-30T23:53:47.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:47.166 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:47.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:47.221 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:47.323 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:47.346 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:47.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:47.474 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:47.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.475 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:47.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:47.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:47.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:47.726 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:47.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:47.849 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:47.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:47.960 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:47 smithi106 ceph-mon[16603]: pgmap v1875: 21 pgs: 21 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:47.960 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:47 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2739/6 objects degraded (45650.000%), 2 pgs degraded, 2 pgs undersized) 2024-08-30T23:53:47.960 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:47 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:53:47.961 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:47 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:53:47.976 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:47.977 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:47 smithi136 ceph-mon[20207]: pgmap v1875: 21 pgs: 21 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:47 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2739/6 objects degraded (45650.000%), 2 pgs degraded, 2 pgs undersized) 2024-08-30T23:53:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:47 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:53:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:47 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:53:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:47 smithi173 ceph-mon[20554]: pgmap v1875: 21 pgs: 21 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:47 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2739/6 objects degraded (45650.000%), 2 pgs degraded, 2 pgs undersized) 2024-08-30T23:53:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:47 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:53:48.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:47 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:53:48.077 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:48.103 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:48.205 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:48.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:48.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:48.358 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:48.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:48.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:48.583 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:48.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:48.657 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_6' created 2024-08-30T23:53:48.657 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:53:48.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:48.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:53:48.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:48.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.863 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:48.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:48.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:48.990 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:48.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:48.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3634817071' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:48.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3155800446' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-08-30T23:53:48.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:48 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:53:48.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:48 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:53:48.994 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1257173628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:49.013 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:53:49.013 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:53:49.034 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:53:49.034 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:53:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3634817071' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3155800446' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-08-30T23:53:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:48 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:53:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:48 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:53:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1257173628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:49.054 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:53:49.055 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:53:49.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3634817071' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:53:49.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3155800446' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-08-30T23:53:49.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:48 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:53:49.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:48 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:53:49.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1257173628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:49.074 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:53:49.074 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:53:49.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:49.095 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:53:49.095 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:53:49.117 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:53:49.117 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:53:49.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:49.139 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:53:49.139 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:53:49.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:53:49.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:53:49.181 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:53:49.181 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:53:49.202 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:53:49.203 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:53:49.219 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:49.223 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:53:49.224 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:53:49.245 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:53:49.245 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:53:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.246 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:49.247 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:49.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:49.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:49.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-08-30T23:53:49.476 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:49.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:49.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:49.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:49.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.755 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:49.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:49.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:49.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:49.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:49.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:49.984 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:50.007 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: pgmap v1877: 21 pgs: 21 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3634817071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3155800446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: osdmap e840: 12 total, 12 up, 11 in 2024-08-30T23:53:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:53:50.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:49 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: pgmap v1877: 21 pgs: 21 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3634817071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3155800446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: osdmap e840: 12 total, 12 up, 11 in 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:53:50.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:49 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:53:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: pgmap v1877: 21 pgs: 21 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:53:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:53:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3634817071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:53:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3155800446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-08-30T23:53:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: osdmap e840: 12 total, 12 up, 11 in 2024-08-30T23:53:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:53:50.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:49 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:53:50.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:50.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:50.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:50.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:50.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:50.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:50.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.510 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:50.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:50.611 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:50.637 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:50.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.761 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:50.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:50.790 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:50.831 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:50.835 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:50.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:50.883 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:50.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:50.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:50 smithi106 ceph-mon[16603]: osdmap e841: 12 total, 12 up, 11 in 2024-08-30T23:53:50.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:50 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:53:50.972 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:50 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:53:50.985 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:51.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:50 smithi136 ceph-mon[20207]: osdmap e841: 12 total, 12 up, 11 in 2024-08-30T23:53:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:50 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:53:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:50 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:53:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:50 smithi173 ceph-mon[20554]: osdmap e841: 12 total, 12 up, 11 in 2024-08-30T23:53:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:50 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:53:51.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:50 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:53:51.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:51.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:51.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:51.261 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:51.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:51.385 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:51.477 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:53:51.477 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:53:51.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:51.513 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:51.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:51.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:51.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:51.744 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:51.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: pgmap v1880: 37 pgs: 1 creating+activating, 1 creating+peering, 11 unknown, 24 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:53:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: osdmap e842: 12 total, 12 up, 11 in 2024-08-30T23:53:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: osdmap e843: 12 total, 12 up, 11 in 2024-08-30T23:53:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:53:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:53:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:53:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:51 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:53:51.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:51.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:51.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: pgmap v1880: 37 pgs: 1 creating+activating, 1 creating+peering, 11 unknown, 24 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:53:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: osdmap e842: 12 total, 12 up, 11 in 2024-08-30T23:53:51.975 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: osdmap e843: 12 total, 12 up, 11 in 2024-08-30T23:53:51.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:53:51.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:53:51.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:53:51.976 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:51 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:53:51.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:52.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: pgmap v1880: 37 pgs: 1 creating+activating, 1 creating+peering, 11 unknown, 24 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: osdmap e842: 12 total, 12 up, 11 in 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: osdmap e843: 12 total, 12 up, 11 in 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:53:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:51 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:53:52.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:52.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:52.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:52.153 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1882,"stamp":"2024-08-30T23:53:50.977226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6607104,"kb_used_data":548608,"kb_used_omap":107,"kb_used_meta":6058132,"kb_avail":1024589568,"statfs":{"total":1055945392128,"available":1049179717632,"internally_reserved":0,"allocated":561774592,"data_stored":556254430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110380,"internal_metadata":6203527380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.999114"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":641,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805633+0000","last_change":"2024-08-30T23:53:42.112471+0000","last_active":"2024-08-30T23:53:46.805633+0000","last_peered":"2024-08-30T23:53:46.805633+0000","last_clean":"2024-08-30T23:53:46.805633+0000","last_became_active":"2024-08-30T23:50:18.102927+0000","last_became_peered":"2024-08-30T23:50:18.102927+0000","last_unstale":"2024-08-30T23:53:46.805633+0000","last_undegraded":"2024-08-30T23:53:46.805633+0000","last_fullsized":"2024-08-30T23:53:46.805633+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.112421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.564463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":126,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.669566+0000","last_change":"2024-08-30T23:53:38.104290+0000","last_active":"2024-08-30T23:53:49.669566+0000","last_peered":"2024-08-30T23:53:49.669566+0000","last_clean":"2024-08-30T23:53:49.669566+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:53:49.669566+0000","last_undegraded":"2024-08-30T23:53:49.669566+0000","last_fullsized":"2024-08-30T23:53:49.669566+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:38.104244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:38.104244+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:45.249002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":749,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.669193+0000","last_change":"2024-08-30T23:53:47.855440+0000","last_active":"2024-08-30T23:53:49.669193+0000","last_peered":"2024-08-30T23:53:49.669193+0000","last_clean":"2024-08-30T23:53:49.669193+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:53:49.669193+0000","last_undegraded":"2024-08-30T23:53:49.669193+0000","last_fullsized":"2024-08-30T23:53:49.669193+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:47.855365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:47.855365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.334601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":126,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.671006+0000","last_change":"2024-08-30T23:53:39.095429+0000","last_active":"2024-08-30T23:53:49.671006+0000","last_peered":"2024-08-30T23:53:49.671006+0000","last_clean":"2024-08-30T23:53:49.671006+0000","last_became_active":"2024-08-30T23:51:42.948334+0000","last_became_peered":"2024-08-30T23:51:42.948334+0000","last_unstale":"2024-08-30T23:53:49.671006+0000","last_undegraded":"2024-08-30T23:53:49.671006+0000","last_fullsized":"2024-08-30T23:53:49.671006+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:39.095359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:39.095359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.506854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":523,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805269+0000","last_change":"2024-08-30T23:52:54.012729+0000","last_active":"2024-08-30T23:53:46.805269+0000","last_peered":"2024-08-30T23:53:46.805269+0000","last_clean":"2024-08-30T23:53:46.805269+0000","last_became_active":"2024-08-30T23:52:54.012402+0000","last_became_peered":"2024-08-30T23:52:54.012402+0000","last_unstale":"2024-08-30T23:53:46.805269+0000","last_undegraded":"2024-08-30T23:53:46.805269+0000","last_fullsized":"2024-08-30T23:53:46.805269+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:02.344155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:02.344155+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.707123+0000","last_change":"2024-08-30T23:53:49.706496+0000","last_active":"2024-08-30T23:53:49.707123+0000","last_peered":"2024-08-30T23:53:49.707123+0000","last_clean":"2024-08-30T23:53:49.707123+0000","last_became_active":"2024-08-30T23:53:49.706120+0000","last_became_peered":"2024-08-30T23:53:49.706120+0000","last_unstale":"2024-08-30T23:53:49.707123+0000","last_undegraded":"2024-08-30T23:53:49.707123+0000","last_fullsized":"2024-08-30T23:53:49.707123+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:05.307017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":337,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.669591+0000","last_change":"2024-08-30T23:53:46.814589+0000","last_active":"2024-08-30T23:53:49.669591+0000","last_peered":"2024-08-30T23:53:49.669591+0000","last_clean":"2024-08-30T23:53:49.669591+0000","last_became_active":"2024-08-30T23:50:18.099827+0000","last_became_peered":"2024-08-30T23:50:18.099827+0000","last_unstale":"2024-08-30T23:53:49.669591+0000","last_undegraded":"2024-08-30T23:53:49.669591+0000","last_fullsized":"2024-08-30T23:53:49.669591+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:46.814527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:46.814527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.311584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.708066+0000","last_change":"2024-08-30T23:53:49.707214+0000","last_active":"2024-08-30T23:53:49.708066+0000","last_peered":"2024-08-30T23:53:49.708066+0000","last_clean":"2024-08-30T23:53:49.708066+0000","last_became_active":"2024-08-30T23:53:49.706776+0000","last_became_peered":"2024-08-30T23:53:49.706776+0000","last_unstale":"2024-08-30T23:53:49.708066+0000","last_undegraded":"2024-08-30T23:53:49.708066+0000","last_fullsized":"2024-08-30T23:53:49.708066+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.193520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":314,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805833+0000","last_change":"2024-08-30T23:52:07.961174+0000","last_active":"2024-08-30T23:53:46.805833+0000","last_peered":"2024-08-30T23:53:46.805833+0000","last_clean":"2024-08-30T23:53:46.805833+0000","last_became_active":"2024-08-30T23:52:07.960637+0000","last_became_peered":"2024-08-30T23:52:07.960637+0000","last_unstale":"2024-08-30T23:53:46.805833+0000","last_undegraded":"2024-08-30T23:53:46.805833+0000","last_fullsized":"2024-08-30T23:53:46.805833+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:50:18.534661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:50:18.534661+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":296,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805929+0000","last_change":"2024-08-30T23:53:43.091100+0000","last_active":"2024-08-30T23:53:46.805929+0000","last_peered":"2024-08-30T23:53:46.805929+0000","last_clean":"2024-08-30T23:53:46.805929+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:53:46.805929+0000","last_undegraded":"2024-08-30T23:53:46.805929+0000","last_fullsized":"2024-08-30T23:53:46.805929+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.091048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.091048+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:10.926425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":927,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805562+0000","last_change":"2024-08-30T23:51:53.368702+0000","last_active":"2024-08-30T23:53:46.805562+0000","last_peered":"2024-08-30T23:53:46.805562+0000","last_clean":"2024-08-30T23:53:46.805562+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:53:46.805562+0000","last_undegraded":"2024-08-30T23:53:46.805562+0000","last_fullsized":"2024-08-30T23:53:46.805562+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:51:53.368650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:51:53.368650+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.708339+0000","last_change":"2024-08-30T23:53:49.708022+0000","last_active":"2024-08-30T23:53:49.708339+0000","last_peered":"2024-08-30T23:53:49.708339+0000","last_clean":"2024-08-30T23:53:49.708339+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:53:49.708339+0000","last_undegraded":"2024-08-30T23:53:49.708339+0000","last_fullsized":"2024-08-30T23:53:49.708339+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:57.765552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":287,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805770+0000","last_change":"2024-08-30T23:53:41.104980+0000","last_active":"2024-08-30T23:53:46.805770+0000","last_peered":"2024-08-30T23:53:46.805770+0000","last_clean":"2024-08-30T23:53:46.805770+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:53:46.805770+0000","last_undegraded":"2024-08-30T23:53:46.805770+0000","last_fullsized":"2024-08-30T23:53:46.805770+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.104925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.104925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:09.944567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.672268+0000","last_change":"2024-08-30T23:53:49.671937+0000","last_active":"2024-08-30T23:53:49.672268+0000","last_peered":"2024-08-30T23:53:49.672268+0000","last_clean":"2024-08-30T23:53:49.672268+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:53:49.672268+0000","last_undegraded":"2024-08-30T23:53:49.672268+0000","last_fullsized":"2024-08-30T23:53:49.672268+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.199307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1769,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:45.976773+0000","last_change":"2024-08-30T23:52:54.017234+0000","last_active":"2024-08-30T23:53:45.976773+0000","last_peered":"2024-08-30T23:53:45.976773+0000","last_clean":"2024-08-30T23:53:45.976773+0000","last_became_active":"2024-08-30T23:52:54.016899+0000","last_became_peered":"2024-08-30T23:52:54.016899+0000","last_unstale":"2024-08-30T23:53:45.976773+0000","last_undegraded":"2024-08-30T23:53:45.976773+0000","last_fullsized":"2024-08-30T23:53:45.976773+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:19.242356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:19.242356+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:53:47.502500+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.709620+0000","last_change":"2024-08-30T23:53:49.709301+0000","last_active":"2024-08-30T23:53:49.709620+0000","last_peered":"2024-08-30T23:53:49.709620+0000","last_clean":"2024-08-30T23:53:49.709620+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:53:49.709620+0000","last_undegraded":"2024-08-30T23:53:49.709620+0000","last_fullsized":"2024-08-30T23:53:49.709620+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:13.339804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":98,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805421+0000","last_change":"2024-08-30T23:52:08.307369+0000","last_active":"2024-08-30T23:53:46.805421+0000","last_peered":"2024-08-30T23:53:46.805421+0000","last_clean":"2024-08-30T23:53:46.805421+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:53:46.805421+0000","last_undegraded":"2024-08-30T23:53:46.805421+0000","last_fullsized":"2024-08-30T23:53:46.805421+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:08.307325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:08.307325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":746,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-08-30T23:53:48.664611+0000","last_change":"2024-08-30T23:50:25.930351+0000","last_active":"2024-08-30T23:53:48.664611+0000","last_peered":"2024-08-30T23:53:48.664611+0000","last_clean":"2024-08-30T23:53:48.664611+0000","last_became_active":"2024-08-30T23:50:25.926833+0000","last_became_peered":"2024-08-30T23:50:25.926833+0000","last_unstale":"2024-08-30T23:53:48.664611+0000","last_undegraded":"2024-08-30T23:53:48.664611+0000","last_fullsized":"2024-08-30T23:53:48.664611+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:09.741057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:09.741057+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.707817+0000","last_change":"2024-08-30T23:53:49.707268+0000","last_active":"2024-08-30T23:53:49.707817+0000","last_peered":"2024-08-30T23:53:49.707817+0000","last_clean":"2024-08-30T23:53:49.707817+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:53:49.707817+0000","last_undegraded":"2024-08-30T23:53:49.707817+0000","last_fullsized":"2024-08-30T23:53:49.707817+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:15.119175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":741,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:45.976768+0000","last_change":"2024-08-30T23:53:41.049687+0000","last_active":"2024-08-30T23:53:45.976768+0000","last_peered":"2024-08-30T23:53:45.976768+0000","last_clean":"2024-08-30T23:53:45.976768+0000","last_became_active":"2024-08-30T23:52:54.016432+0000","last_became_peered":"2024-08-30T23:52:54.016432+0000","last_unstale":"2024-08-30T23:53:45.976768+0000","last_undegraded":"2024-08-30T23:53:45.976768+0000","last_fullsized":"2024-08-30T23:53:45.976768+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.049626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.049626+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.750692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":672,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:45.976792+0000","last_change":"2024-08-30T23:53:35.098110+0000","last_active":"2024-08-30T23:53:45.976792+0000","last_peered":"2024-08-30T23:53:45.976792+0000","last_clean":"2024-08-30T23:53:45.976792+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:53:45.976792+0000","last_undegraded":"2024-08-30T23:53:45.976792+0000","last_fullsized":"2024-08-30T23:53:45.976792+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:35.098031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:35.098031+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.053662+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":937,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805269+0000","last_change":"2024-08-30T23:52:07.301252+0000","last_active":"2024-08-30T23:53:46.805269+0000","last_peered":"2024-08-30T23:53:46.805269+0000","last_clean":"2024-08-30T23:53:46.805269+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:53:46.805269+0000","last_undegraded":"2024-08-30T23:53:46.805269+0000","last_fullsized":"2024-08-30T23:53:46.805269+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:07.301208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:07.301208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":138,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.670874+0000","last_change":"2024-08-30T23:53:43.164672+0000","last_active":"2024-08-30T23:53:49.670874+0000","last_peered":"2024-08-30T23:53:49.670874+0000","last_clean":"2024-08-30T23:53:49.670874+0000","last_became_active":"2024-08-30T23:51:42.952967+0000","last_became_peered":"2024-08-30T23:51:42.952967+0000","last_unstale":"2024-08-30T23:53:49.670874+0000","last_undegraded":"2024-08-30T23:53:49.670874+0000","last_fullsized":"2024-08-30T23:53:49.670874+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.164609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.164609+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.951863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":840,"state":"creating+peering","last_fresh":"2024-08-30T23:53:48.664258+0000","last_change":"2024-08-30T23:53:48.664258+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.664258+0000","last_undegraded":"2024-08-30T23:53:48.664258+0000","last_fullsized":"2024-08-30T23:53:48.664258+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1756,"reported_epoch":839,"state":"active+clean","last_fresh":"2024-08-30T23:53:46.805519+0000","last_change":"2024-08-30T23:53:42.303311+0000","last_active":"2024-08-30T23:53:46.805519+0000","last_peered":"2024-08-30T23:53:46.805519+0000","last_clean":"2024-08-30T23:53:46.805519+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:53:46.805519+0000","last_undegraded":"2024-08-30T23:53:46.805519+0000","last_fullsized":"2024-08-30T23:53:46.805519+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.303263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.303263+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:47.897994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":4,"reported_epoch":841,"state":"creating+activating","last_fresh":"2024-08-30T23:53:49.670639+0000","last_change":"2024-08-30T23:53:49.670207+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:49.670639+0000","last_undegraded":"2024-08-30T23:53:49.670639+0000","last_fullsized":"2024-08-30T23:53:49.670639+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":["5","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":864,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-08-30T23:53:48.662757+0000","last_change":"2024-08-30T23:49:50.688833+0000","last_active":"2024-08-30T23:53:48.662757+0000","last_peered":"2024-08-30T23:53:48.662757+0000","last_clean":"2024-08-30T23:53:48.662757+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:53:48.662757+0000","last_undegraded":"2024-08-30T23:53:48.662757+0000","last_fullsized":"2024-08-30T23:53:48.662757+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:50.688764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:50.688764+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:53:48.650807+0000","last_change":"2024-08-30T23:53:48.650807+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.650807+0000","last_undegraded":"2024-08-30T23:53:48.650807+0000","last_fullsized":"2024-08-30T23:53:48.650807+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":126,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.670389+0000","last_change":"2024-08-30T23:53:41.078765+0000","last_active":"2024-08-30T23:53:49.670389+0000","last_peered":"2024-08-30T23:53:49.670389+0000","last_clean":"2024-08-30T23:53:49.670389+0000","last_became_active":"2024-08-30T23:51:42.950294+0000","last_became_peered":"2024-08-30T23:51:42.950294+0000","last_unstale":"2024-08-30T23:53:49.670389+0000","last_undegraded":"2024-08-30T23:53:49.670389+0000","last_fullsized":"2024-08-30T23:53:49.670389+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.078720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.078720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:44.759291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1292,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-08-30T23:53:48.658610+0000","last_change":"2024-08-30T23:52:23.554154+0000","last_active":"2024-08-30T23:53:48.658610+0000","last_peered":"2024-08-30T23:53:48.658610+0000","last_clean":"2024-08-30T23:53:48.658610+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:53:48.658610+0000","last_undegraded":"2024-08-30T23:53:48.658610+0000","last_fullsized":"2024-08-30T23:53:48.658610+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:23.554079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:23.554079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:53:52.501508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":369367187807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449448,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699032,"kb_used_data":212476,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93046120,"statfs":{"total":95995035648,"available":95279226880,"internally_reserved":0,"allocated":217575424,"data_stored":217079918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":498203072},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711078,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":650348,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93094804,"statfs":{"total":95995035648,"available":95329079296,"internally_reserved":0,"allocated":1396736,"data_stored":892260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972720,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":474036,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271116,"statfs":{"total":95995035648,"available":95509622784,"internally_reserved":0,"allocated":1409024,"data_stored":900101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234356,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727516,"kb_used_data":20232,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93017636,"statfs":{"total":95995035648,"available":95250059264,"internally_reserved":0,"allocated":20717568,"data_stored":20217215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":724236746},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528698,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492372,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93252780,"statfs":{"total":95995035648,"available":95490846720,"internally_reserved":0,"allocated":1699840,"data_stored":1204334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775201,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195444,"kb_used_data":1564,"kb_used_omap":5,"kb_used_meta":193850,"kb_avail":93549708,"statfs":{"total":95995035648,"available":95794900992,"internally_reserved":0,"allocated":1601536,"data_stored":1093508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":198502474},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":756628,"kb_used_data":129852,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92988524,"statfs":{"total":95995035648,"available":95220248576,"internally_reserved":0,"allocated":132968448,"data_stored":132472942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150087,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964716,"kb_used_data":175508,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92780436,"statfs":{"total":95995035648,"available":95007166464,"internally_reserved":0,"allocated":179720192,"data_stored":179212215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":808122823},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411725,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542256,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93202896,"statfs":{"total":95995035648,"available":95439765504,"internally_reserved":0,"allocated":1601536,"data_stored":1093508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830541,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":354132,"kb_used_data":1340,"kb_used_omap":87,"kb_used_meta":352680,"kb_avail":93391020,"statfs":{"total":95995035648,"available":95632404480,"internally_reserved":0,"allocated":1372160,"data_stored":876654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":361144479},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738775,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750624,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92994528,"statfs":{"total":95995035648,"available":95226396672,"internally_reserved":0,"allocated":1712128,"data_stored":1211775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:53:52.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:53:52.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:53:52.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:52.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:52.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:52.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:52.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:52.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:52.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:52.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:52.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:53:52.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:52.777 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:53:52.777 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:53:52.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:52.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:52 smithi106 ceph-mon[16603]: from='client.18474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:53:52.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:52 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:53:52.814 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:52 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:53:52.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:52.923 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:52.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:53.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:52 smithi136 ceph-mon[20207]: from='client.18474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:53:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:52 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:53:53.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:52 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:53:53.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.052 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:53.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:52 smithi173 ceph-mon[20554]: from='client.18474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:53:53.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:52 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:53:53.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:52 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:53:53.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:53.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:53.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:53.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:53.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.436 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:53.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:53.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:53:53.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:53.553 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:53.568 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:53.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:53.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:53.680 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-08-30T23:53:53.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:53.746 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:53 smithi106 ceph-mon[16603]: pgmap v1883: 37 pgs: 1 creating+activating, 1 creating+peering, 5 unknown, 30 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:53.747 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/733702959' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:53:53.761 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:53:53.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:53 smithi136 ceph-mon[20207]: pgmap v1883: 37 pgs: 1 creating+activating, 1 creating+peering, 5 unknown, 30 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:53.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/733702959' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:53:53.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:53.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:53.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:53.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:53.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:54.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:54.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:53 smithi173 ceph-mon[20554]: pgmap v1883: 37 pgs: 1 creating+activating, 1 creating+peering, 5 unknown, 30 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:54.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/733702959' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:53:54.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:54.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:54.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:54.264 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:53:54.265 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:53:54.304 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:54.330 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:54.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:54.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:54.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:54.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:54.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:54.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:54.718 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:53:54.718 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:53:54.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:54.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:54.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:54.954 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/733702959' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:53:54.954 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:54 smithi136 ceph-mon[20207]: osdmap e844: 12 total, 12 up, 11 in 2024-08-30T23:53:54.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:53:54.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:54.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:54.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:55.061 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:55.069 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/733702959' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:53:55.070 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:54 smithi106 ceph-mon[16603]: osdmap e844: 12 total, 12 up, 11 in 2024-08-30T23:53:55.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/733702959' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:53:55.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:54 smithi173 ceph-mon[20554]: osdmap e844: 12 total, 12 up, 11 in 2024-08-30T23:53:55.085 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:55.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:55.198 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:53:55.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:55.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:55.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:55.338 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-08-30T23:53:55.341 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:53:55.341 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:53:55.403 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:53:55.403 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:53:55.403 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_143816 2024-08-30T23:53:55.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:53:55.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:53:55.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:55.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:55.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:55.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:55.602 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1885,"stamp":"2024-08-30T23:53:53.787468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6576500,"kb_used_data":518012,"kb_used_omap":107,"kb_used_meta":6058132,"kb_avail":1024620172,"statfs":{"total":1055945392128,"available":1049211056128,"internally_reserved":0,"allocated":530444288,"data_stored":524831731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110380,"internal_metadata":6203527380},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.124288"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":648,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041183+0000","last_change":"2024-08-30T23:53:42.112471+0000","last_active":"2024-08-30T23:53:51.041183+0000","last_peered":"2024-08-30T23:53:51.041183+0000","last_clean":"2024-08-30T23:53:51.041183+0000","last_became_active":"2024-08-30T23:50:18.102927+0000","last_became_peered":"2024-08-30T23:50:18.102927+0000","last_unstale":"2024-08-30T23:53:51.041183+0000","last_undegraded":"2024-08-30T23:53:51.041183+0000","last_fullsized":"2024-08-30T23:53:51.041183+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.112421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.564463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":126,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.669566+0000","last_change":"2024-08-30T23:53:38.104290+0000","last_active":"2024-08-30T23:53:49.669566+0000","last_peered":"2024-08-30T23:53:49.669566+0000","last_clean":"2024-08-30T23:53:49.669566+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:53:49.669566+0000","last_undegraded":"2024-08-30T23:53:49.669566+0000","last_fullsized":"2024-08-30T23:53:49.669566+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:38.104244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:38.104244+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:45.249002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":749,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.669193+0000","last_change":"2024-08-30T23:53:47.855440+0000","last_active":"2024-08-30T23:53:49.669193+0000","last_peered":"2024-08-30T23:53:49.669193+0000","last_clean":"2024-08-30T23:53:49.669193+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:53:49.669193+0000","last_undegraded":"2024-08-30T23:53:49.669193+0000","last_fullsized":"2024-08-30T23:53:49.669193+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:47.855365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:47.855365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.334601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":126,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.671006+0000","last_change":"2024-08-30T23:53:39.095429+0000","last_active":"2024-08-30T23:53:49.671006+0000","last_peered":"2024-08-30T23:53:49.671006+0000","last_clean":"2024-08-30T23:53:49.671006+0000","last_became_active":"2024-08-30T23:51:42.948334+0000","last_became_peered":"2024-08-30T23:51:42.948334+0000","last_unstale":"2024-08-30T23:53:49.671006+0000","last_undegraded":"2024-08-30T23:53:49.671006+0000","last_fullsized":"2024-08-30T23:53:49.671006+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:39.095359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:39.095359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.506854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041601+0000","last_change":"2024-08-30T23:53:49.706683+0000","last_active":"2024-08-30T23:53:51.041601+0000","last_peered":"2024-08-30T23:53:51.041601+0000","last_clean":"2024-08-30T23:53:51.041601+0000","last_became_active":"2024-08-30T23:53:49.706310+0000","last_became_peered":"2024-08-30T23:53:49.706310+0000","last_unstale":"2024-08-30T23:53:51.041601+0000","last_undegraded":"2024-08-30T23:53:51.041601+0000","last_fullsized":"2024-08-30T23:53:51.041601+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.917544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":530,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.042882+0000","last_change":"2024-08-30T23:52:54.012729+0000","last_active":"2024-08-30T23:53:51.042882+0000","last_peered":"2024-08-30T23:53:51.042882+0000","last_clean":"2024-08-30T23:53:51.042882+0000","last_became_active":"2024-08-30T23:52:54.012402+0000","last_became_peered":"2024-08-30T23:52:54.012402+0000","last_unstale":"2024-08-30T23:53:51.042882+0000","last_undegraded":"2024-08-30T23:53:51.042882+0000","last_fullsized":"2024-08-30T23:53:51.042882+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:02.344155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:02.344155+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.707123+0000","last_change":"2024-08-30T23:53:49.706496+0000","last_active":"2024-08-30T23:53:49.707123+0000","last_peered":"2024-08-30T23:53:49.707123+0000","last_clean":"2024-08-30T23:53:49.707123+0000","last_became_active":"2024-08-30T23:53:49.706120+0000","last_became_peered":"2024-08-30T23:53:49.706120+0000","last_unstale":"2024-08-30T23:53:49.707123+0000","last_undegraded":"2024-08-30T23:53:49.707123+0000","last_fullsized":"2024-08-30T23:53:49.707123+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:05.307017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":337,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.669591+0000","last_change":"2024-08-30T23:53:46.814589+0000","last_active":"2024-08-30T23:53:49.669591+0000","last_peered":"2024-08-30T23:53:49.669591+0000","last_clean":"2024-08-30T23:53:49.669591+0000","last_became_active":"2024-08-30T23:50:18.099827+0000","last_became_peered":"2024-08-30T23:50:18.099827+0000","last_unstale":"2024-08-30T23:53:49.669591+0000","last_undegraded":"2024-08-30T23:53:49.669591+0000","last_fullsized":"2024-08-30T23:53:49.669591+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:46.814527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:46.814527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.311584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.708066+0000","last_change":"2024-08-30T23:53:49.707214+0000","last_active":"2024-08-30T23:53:49.708066+0000","last_peered":"2024-08-30T23:53:49.708066+0000","last_clean":"2024-08-30T23:53:49.708066+0000","last_became_active":"2024-08-30T23:53:49.706776+0000","last_became_peered":"2024-08-30T23:53:49.706776+0000","last_unstale":"2024-08-30T23:53:49.708066+0000","last_undegraded":"2024-08-30T23:53:49.708066+0000","last_fullsized":"2024-08-30T23:53:49.708066+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.193520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":329,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.042769+0000","last_change":"2024-08-30T23:53:51.042769+0000","last_active":"2024-08-30T23:53:51.042769+0000","last_peered":"2024-08-30T23:53:51.042769+0000","last_clean":"2024-08-30T23:53:51.042769+0000","last_became_active":"2024-08-30T23:52:07.960637+0000","last_became_peered":"2024-08-30T23:52:07.960637+0000","last_unstale":"2024-08-30T23:53:51.042769+0000","last_undegraded":"2024-08-30T23:53:51.042769+0000","last_fullsized":"2024-08-30T23:53:51.042769+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.042710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.042710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.950115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":16,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.635080+0000","last_change":"2024-08-30T23:53:49.706382+0000","last_active":"2024-08-30T23:53:51.635080+0000","last_peered":"2024-08-30T23:53:51.635080+0000","last_clean":"2024-08-30T23:53:51.635080+0000","last_became_active":"2024-08-30T23:53:49.706069+0000","last_became_peered":"2024-08-30T23:53:49.706069+0000","last_unstale":"2024-08-30T23:53:51.635080+0000","last_undegraded":"2024-08-30T23:53:51.635080+0000","last_fullsized":"2024-08-30T23:53:51.635080+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:52.380827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":303,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041541+0000","last_change":"2024-08-30T23:53:43.091100+0000","last_active":"2024-08-30T23:53:51.041541+0000","last_peered":"2024-08-30T23:53:51.041541+0000","last_clean":"2024-08-30T23:53:51.041541+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:53:51.041541+0000","last_undegraded":"2024-08-30T23:53:51.041541+0000","last_fullsized":"2024-08-30T23:53:51.041541+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.091048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.091048+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:10.926425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":16,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:50.982831+0000","last_change":"2024-08-30T23:53:49.706502+0000","last_active":"2024-08-30T23:53:50.982831+0000","last_peered":"2024-08-30T23:53:50.982831+0000","last_clean":"2024-08-30T23:53:50.982831+0000","last_became_active":"2024-08-30T23:53:49.705978+0000","last_became_peered":"2024-08-30T23:53:49.705978+0000","last_unstale":"2024-08-30T23:53:50.982831+0000","last_undegraded":"2024-08-30T23:53:50.982831+0000","last_fullsized":"2024-08-30T23:53:50.982831+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.382598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":942,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.507308+0000","last_change":"2024-08-30T23:53:51.507308+0000","last_active":"2024-08-30T23:53:51.507308+0000","last_peered":"2024-08-30T23:53:51.507308+0000","last_clean":"2024-08-30T23:53:51.507308+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:53:51.507308+0000","last_undegraded":"2024-08-30T23:53:51.507308+0000","last_fullsized":"2024-08-30T23:53:51.507308+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.507248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.507248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.951121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.708339+0000","last_change":"2024-08-30T23:53:49.708022+0000","last_active":"2024-08-30T23:53:49.708339+0000","last_peered":"2024-08-30T23:53:49.708339+0000","last_clean":"2024-08-30T23:53:49.708339+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:53:49.708339+0000","last_undegraded":"2024-08-30T23:53:49.708339+0000","last_fullsized":"2024-08-30T23:53:49.708339+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:57.765552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":294,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041136+0000","last_change":"2024-08-30T23:53:41.104980+0000","last_active":"2024-08-30T23:53:51.041136+0000","last_peered":"2024-08-30T23:53:51.041136+0000","last_clean":"2024-08-30T23:53:51.041136+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:53:51.041136+0000","last_undegraded":"2024-08-30T23:53:51.041136+0000","last_fullsized":"2024-08-30T23:53:51.041136+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.104925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.104925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:09.944567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.672268+0000","last_change":"2024-08-30T23:53:49.671937+0000","last_active":"2024-08-30T23:53:49.672268+0000","last_peered":"2024-08-30T23:53:49.672268+0000","last_clean":"2024-08-30T23:53:49.672268+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:53:49.672268+0000","last_undegraded":"2024-08-30T23:53:49.672268+0000","last_fullsized":"2024-08-30T23:53:49.672268+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.199307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1785,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:50.999427+0000","last_change":"2024-08-30T23:53:50.999427+0000","last_active":"2024-08-30T23:53:50.999427+0000","last_peered":"2024-08-30T23:53:50.999427+0000","last_clean":"2024-08-30T23:53:50.999427+0000","last_became_active":"2024-08-30T23:52:54.016899+0000","last_became_peered":"2024-08-30T23:52:54.016899+0000","last_unstale":"2024-08-30T23:53:50.999427+0000","last_undegraded":"2024-08-30T23:53:50.999427+0000","last_fullsized":"2024-08-30T23:53:50.999427+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:50.999362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:50.999362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.278203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.709620+0000","last_change":"2024-08-30T23:53:49.709301+0000","last_active":"2024-08-30T23:53:49.709620+0000","last_peered":"2024-08-30T23:53:49.709620+0000","last_clean":"2024-08-30T23:53:49.709620+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:53:49.709620+0000","last_undegraded":"2024-08-30T23:53:49.709620+0000","last_fullsized":"2024-08-30T23:53:49.709620+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:13.339804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":105,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.043049+0000","last_change":"2024-08-30T23:52:08.307369+0000","last_active":"2024-08-30T23:53:51.043049+0000","last_peered":"2024-08-30T23:53:51.043049+0000","last_clean":"2024-08-30T23:53:51.043049+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:53:51.043049+0000","last_undegraded":"2024-08-30T23:53:51.043049+0000","last_fullsized":"2024-08-30T23:53:51.043049+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:08.307325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:08.307325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041369+0000","last_change":"2024-08-30T23:53:49.670752+0000","last_active":"2024-08-30T23:53:51.041369+0000","last_peered":"2024-08-30T23:53:51.041369+0000","last_clean":"2024-08-30T23:53:51.041369+0000","last_became_active":"2024-08-30T23:53:49.670369+0000","last_became_peered":"2024-08-30T23:53:49.670369+0000","last_unstale":"2024-08-30T23:53:51.041369+0000","last_undegraded":"2024-08-30T23:53:51.041369+0000","last_fullsized":"2024-08-30T23:53:51.041369+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:48.852476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":746,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-08-30T23:53:48.664611+0000","last_change":"2024-08-30T23:50:25.930351+0000","last_active":"2024-08-30T23:53:48.664611+0000","last_peered":"2024-08-30T23:53:48.664611+0000","last_clean":"2024-08-30T23:53:48.664611+0000","last_became_active":"2024-08-30T23:50:25.926833+0000","last_became_peered":"2024-08-30T23:50:25.926833+0000","last_unstale":"2024-08-30T23:53:48.664611+0000","last_undegraded":"2024-08-30T23:53:48.664611+0000","last_fullsized":"2024-08-30T23:53:48.664611+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:09.741057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:09.741057+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.707817+0000","last_change":"2024-08-30T23:53:49.707268+0000","last_active":"2024-08-30T23:53:49.707817+0000","last_peered":"2024-08-30T23:53:49.707817+0000","last_clean":"2024-08-30T23:53:49.707817+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:53:49.707817+0000","last_undegraded":"2024-08-30T23:53:49.707817+0000","last_fullsized":"2024-08-30T23:53:49.707817+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:15.119175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":749,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:50.982707+0000","last_change":"2024-08-30T23:53:41.049687+0000","last_active":"2024-08-30T23:53:50.982707+0000","last_peered":"2024-08-30T23:53:50.982707+0000","last_clean":"2024-08-30T23:53:50.982707+0000","last_became_active":"2024-08-30T23:52:54.016432+0000","last_became_peered":"2024-08-30T23:52:54.016432+0000","last_unstale":"2024-08-30T23:53:50.982707+0000","last_undegraded":"2024-08-30T23:53:50.982707+0000","last_fullsized":"2024-08-30T23:53:50.982707+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.049626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.049626+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.750692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":15,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041445+0000","last_change":"2024-08-30T23:53:49.706802+0000","last_active":"2024-08-30T23:53:51.041445+0000","last_peered":"2024-08-30T23:53:51.041445+0000","last_clean":"2024-08-30T23:53:51.041445+0000","last_became_active":"2024-08-30T23:53:49.706231+0000","last_became_peered":"2024-08-30T23:53:49.706231+0000","last_unstale":"2024-08-30T23:53:51.041445+0000","last_undegraded":"2024-08-30T23:53:51.041445+0000","last_fullsized":"2024-08-30T23:53:51.041445+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:58.038937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":681,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:50.982813+0000","last_change":"2024-08-30T23:53:35.098110+0000","last_active":"2024-08-30T23:53:50.982813+0000","last_peered":"2024-08-30T23:53:50.982813+0000","last_clean":"2024-08-30T23:53:50.982813+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:53:50.982813+0000","last_undegraded":"2024-08-30T23:53:50.982813+0000","last_fullsized":"2024-08-30T23:53:50.982813+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:35.098031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:35.098031+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.053662+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":15,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041543+0000","last_change":"2024-08-30T23:53:49.707500+0000","last_active":"2024-08-30T23:53:51.041543+0000","last_peered":"2024-08-30T23:53:51.041543+0000","last_clean":"2024-08-30T23:53:51.041543+0000","last_became_active":"2024-08-30T23:53:49.707122+0000","last_became_peered":"2024-08-30T23:53:49.707122+0000","last_unstale":"2024-08-30T23:53:51.041543+0000","last_undegraded":"2024-08-30T23:53:51.041543+0000","last_fullsized":"2024-08-30T23:53:51.041543+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.649839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":944,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.043253+0000","last_change":"2024-08-30T23:52:07.301252+0000","last_active":"2024-08-30T23:53:51.043253+0000","last_peered":"2024-08-30T23:53:51.043253+0000","last_clean":"2024-08-30T23:53:51.043253+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:53:51.043253+0000","last_undegraded":"2024-08-30T23:53:51.043253+0000","last_fullsized":"2024-08-30T23:53:51.043253+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:07.301208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:07.301208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":16,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.634996+0000","last_change":"2024-08-30T23:53:49.706361+0000","last_active":"2024-08-30T23:53:51.634996+0000","last_peered":"2024-08-30T23:53:51.634996+0000","last_clean":"2024-08-30T23:53:51.634996+0000","last_became_active":"2024-08-30T23:53:49.706070+0000","last_became_peered":"2024-08-30T23:53:49.706070+0000","last_unstale":"2024-08-30T23:53:51.634996+0000","last_undegraded":"2024-08-30T23:53:51.634996+0000","last_fullsized":"2024-08-30T23:53:51.634996+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.901446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":138,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.670874+0000","last_change":"2024-08-30T23:53:43.164672+0000","last_active":"2024-08-30T23:53:49.670874+0000","last_peered":"2024-08-30T23:53:49.670874+0000","last_clean":"2024-08-30T23:53:49.670874+0000","last_became_active":"2024-08-30T23:51:42.952967+0000","last_became_peered":"2024-08-30T23:51:42.952967+0000","last_unstale":"2024-08-30T23:53:49.670874+0000","last_undegraded":"2024-08-30T23:53:49.670874+0000","last_fullsized":"2024-08-30T23:53:49.670874+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.164609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.164609+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.951863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":840,"state":"creating+peering","last_fresh":"2024-08-30T23:53:48.664258+0000","last_change":"2024-08-30T23:53:48.664258+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:48.664258+0000","last_undegraded":"2024-08-30T23:53:48.664258+0000","last_fullsized":"2024-08-30T23:53:48.664258+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1763,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:50.979611+0000","last_change":"2024-08-30T23:53:42.303311+0000","last_active":"2024-08-30T23:53:50.979611+0000","last_peered":"2024-08-30T23:53:50.979611+0000","last_clean":"2024-08-30T23:53:50.979611+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:53:50.979611+0000","last_undegraded":"2024-08-30T23:53:50.979611+0000","last_fullsized":"2024-08-30T23:53:50.979611+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.303263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.303263+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:47.897994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":4,"reported_epoch":841,"state":"creating+activating","last_fresh":"2024-08-30T23:53:49.670639+0000","last_change":"2024-08-30T23:53:49.670207+0000","last_active":"2024-08-30T23:53:48.650807+0000","last_peered":"2024-08-30T23:53:48.650807+0000","last_clean":"2024-08-30T23:53:48.650807+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:53:49.670639+0000","last_undegraded":"2024-08-30T23:53:49.670639+0000","last_fullsized":"2024-08-30T23:53:49.670639+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":["5","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":864,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-08-30T23:53:48.662757+0000","last_change":"2024-08-30T23:49:50.688833+0000","last_active":"2024-08-30T23:53:48.662757+0000","last_peered":"2024-08-30T23:53:48.662757+0000","last_clean":"2024-08-30T23:53:48.662757+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:53:48.662757+0000","last_undegraded":"2024-08-30T23:53:48.662757+0000","last_fullsized":"2024-08-30T23:53:48.662757+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:49:50.688764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:49:50.688764+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":843,"state":"active+clean","last_fresh":"2024-08-30T23:53:51.041335+0000","last_change":"2024-08-30T23:53:49.706603+0000","last_active":"2024-08-30T23:53:51.041335+0000","last_peered":"2024-08-30T23:53:51.041335+0000","last_clean":"2024-08-30T23:53:51.041335+0000","last_became_active":"2024-08-30T23:53:49.706194+0000","last_became_peered":"2024-08-30T23:53:49.706194+0000","last_unstale":"2024-08-30T23:53:51.041335+0000","last_undegraded":"2024-08-30T23:53:51.041335+0000","last_fullsized":"2024-08-30T23:53:51.041335+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.544790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":126,"reported_epoch":841,"state":"active+clean","last_fresh":"2024-08-30T23:53:49.670389+0000","last_change":"2024-08-30T23:53:41.078765+0000","last_active":"2024-08-30T23:53:49.670389+0000","last_peered":"2024-08-30T23:53:49.670389+0000","last_clean":"2024-08-30T23:53:49.670389+0000","last_became_active":"2024-08-30T23:51:42.950294+0000","last_became_peered":"2024-08-30T23:51:42.950294+0000","last_unstale":"2024-08-30T23:53:49.670389+0000","last_undegraded":"2024-08-30T23:53:49.670389+0000","last_fullsized":"2024-08-30T23:53:49.670389+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.078720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.078720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:44.759291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1292,"reported_epoch":840,"state":"active+clean","last_fresh":"2024-08-30T23:53:48.658610+0000","last_change":"2024-08-30T23:52:23.554154+0000","last_active":"2024-08-30T23:53:48.658610+0000","last_peered":"2024-08-30T23:53:48.658610+0000","last_clean":"2024-08-30T23:53:48.658610+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:53:48.658610+0000","last_undegraded":"2024-08-30T23:53:48.658610+0000","last_fullsized":"2024-08-30T23:53:48.658610+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:23.554079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:23.554079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:53:52.501508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":369367187808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449449,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":689524,"kb_used_data":202976,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93055628,"statfs":{"total":95995035648,"available":95288963072,"internally_reserved":0,"allocated":207847424,"data_stored":207338501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":498203072},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711078,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":650348,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93094804,"statfs":{"total":95995035648,"available":95329079296,"internally_reserved":0,"allocated":1396736,"data_stored":892260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972720,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":474036,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271116,"statfs":{"total":95995035648,"available":95509622784,"internally_reserved":0,"allocated":1409024,"data_stored":900101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234357,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717968,"kb_used_data":10680,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93027184,"statfs":{"total":95995035648,"available":95259836416,"internally_reserved":0,"allocated":10936320,"data_stored":10418734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":724236746},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528699,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492432,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93252720,"statfs":{"total":95995035648,"available":95490785280,"internally_reserved":0,"allocated":1761280,"data_stored":1243694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775201,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195444,"kb_used_data":1564,"kb_used_omap":5,"kb_used_meta":193850,"kb_avail":93549708,"statfs":{"total":95995035648,"available":95794900992,"internally_reserved":0,"allocated":1601536,"data_stored":1093508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":198502474},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953862,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744976,"kb_used_data":118200,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93000176,"statfs":{"total":95995035648,"available":95232180224,"internally_reserved":0,"allocated":121036800,"data_stored":120519214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150087,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964716,"kb_used_data":175508,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92780436,"statfs":{"total":95995035648,"available":95007166464,"internally_reserved":0,"allocated":179720192,"data_stored":179212215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":808122823},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411725,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542256,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93202896,"statfs":{"total":95995035648,"available":95439765504,"internally_reserved":0,"allocated":1601536,"data_stored":1093508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830542,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":354176,"kb_used_data":1388,"kb_used_omap":87,"kb_used_meta":352680,"kb_avail":93390976,"statfs":{"total":95995035648,"available":95632359424,"internally_reserved":0,"allocated":1421312,"data_stored":908221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":361144479},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738775,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750624,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92994528,"statfs":{"total":95995035648,"available":95226396672,"internally_reserved":0,"allocated":1712128,"data_stored":1211775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:53:55.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-08-30T23:53:55.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:53:55.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:53:55.754 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:53:55.754 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:53:55.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:55.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:55 smithi106 ceph-mon[16603]: pgmap v1885: 37 pgs: 1 creating+activating, 1 creating+peering, 35 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:55 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:53:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:55 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:53:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:55 smithi106 ceph-mon[16603]: from='client.18486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:53:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:55 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:53:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:55 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:53:55.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:55.955 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:55.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:55.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:56.015 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:55 smithi136 ceph-mon[20207]: pgmap v1885: 37 pgs: 1 creating+activating, 1 creating+peering, 35 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:55 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:53:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:55 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:53:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:55 smithi136 ceph-mon[20207]: from='client.18486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:53:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:55 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:53:56.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:55 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:53:56.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:56.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:55 smithi173 ceph-mon[20554]: pgmap v1885: 37 pgs: 1 creating+activating, 1 creating+peering, 35 active+clean; 577 KiB data, 6.3 GiB used, 977 GiB / 983 GiB avail 2024-08-30T23:53:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:55 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:53:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:55 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:53:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:55 smithi173 ceph-mon[20554]: from='client.18486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:53:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:55 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:53:56.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:55 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:53:56.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:56.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:56.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:56.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:56.359 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:56.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1982 1966 122.865 122.875 0.0100118 0.0080589 2024-08-30T23:53:56.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:53:56.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:56.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:56.590 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:53:56.617 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:53:56.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:56.737 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:53:56.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.747 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:56.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:56.845 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:53:56.845 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:53:56.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:56.870 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:53:56.870 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:53:56.874 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:56.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:56.897 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:53:56.897 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:53:56.922 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:53:56.922 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:53:56.948 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:53:56.948 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:53:56.973 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:53:56.974 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:53:56.976 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:56.995 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:53:57.002 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:53:57.002 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:53:57.003 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:57.030 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:53:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:53:57.054 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:53:57.054 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:53:57.076 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:53:57.076 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:53:57.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:57.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:57.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:56 smithi136 ceph-mon[20207]: pgmap v1886: 37 pgs: 37 active+clean; 5.8 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 896 KiB/s wr, 13 op/s 2024-08-30T23:53:57.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:56 smithi136 ceph-mon[20207]: osdmap e845: 12 total, 12 up, 11 in 2024-08-30T23:53:57.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1555975242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:53:57.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:53:57.161 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4281875045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:57.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:57.261 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:57.276 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:53:57.276 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(1.3 e535 from shard 8)", 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 331.45720033600003, 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74531784099999998, 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:53:57.277 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817198+0000", 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0399999999999996e-07 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817203+0000", 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6729999999999997e-06 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562374+0000", 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74517085800000005 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562381+0000", 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.878e-06 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562485+0000", 2024-08-30T23:53:57.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103906 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562493+0000", 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562515+0000", 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2367000000000001e-05 2024-08-30T23:53:57.281 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 763/762 [PushReplyOp(7:23574efc:::benchmark_data_smithi173_129296_object75:head)])", 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 209.98607988500001, 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71604419900000005, 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:53:57.282 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0699999999999997e-07 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288323+0000", 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3490000000000002e-06 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001046+0000", 2024-08-30T23:53:57.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71272310000000005 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001053+0000", 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0230000000000004e-06 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001113+0000", 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0402000000000002e-05 2024-08-30T23:53:57.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.004362+0000", 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032488180000000001 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.286 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 766/762 [PushReplyOp(7:21ab2b56:::benchmark_data_smithi173_129296_object2772:head)])", 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 206.118658504, 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92479003900000001, 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155740+0000", 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0780000000000001e-06 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155745+0000", 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077308+0000", 2024-08-30T23:53:57.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92156320800000002 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077323+0000", 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4443e-05 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077429+0000", 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105683 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.080529+0000", 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031005659999999999 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 767/762 [PushReplyOp(7:26f21d66:::benchmark_data_smithi173_129296_object4816:head)])", 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:53:57.291 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 202.441213425, 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0014494869999999, 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833185+0000", 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833190+0000", 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0830000000000003e-06 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830783+0000", 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99759255400000002 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830792+0000", 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7779999999999996e-06 2024-08-30T23:53:57.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830874+0000", 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2557000000000003e-05 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.834634+0000", 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003759465 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.b 769/756 [PushOp(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head, version: 734'1560, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head@734'1560, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 192.721240905, 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60120657700000002, 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553158+0000", 2024-08-30T23:53:57.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3799999999999999e-06 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553211+0000", 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2676999999999997e-05 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153021+0000", 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59981003200000005 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153032+0000", 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1415e-05 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153139+0000", 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107075 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153149+0000", 2024-08-30T23:53:57.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9629999999999993e-06 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.154363+0000", 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214035 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:123065 7.1 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e792)", 2024-08-30T23:53:57.300 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 122.786883929, 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64777179500000004, 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 123065 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.301 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487515+0000", 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.477e-06 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487521+0000", 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487533+0000", 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.234e-05 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487541+0000", 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487594+0000", 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3134000000000003e-05 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487681+0000", 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7437999999999999e-05 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487753+0000", 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1632999999999997e-05 2024-08-30T23:53:57.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.490939+0000", 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031855550000000001 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.492073+0000", 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011345039999999999 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135209+0000", 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64313627299999998 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135251+0000", 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1378000000000003e-05 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135285+0000", 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4730999999999997e-05 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:142377 7.8 7:18815242:::benchmark_data_smithi173_129296_object55261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 79.878825401, 2024-08-30T23:53:57.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61694005799999996, 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 142377 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:53:57.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.311 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:53:57.311 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:53:57.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:56 smithi106 ceph-mon[16603]: pgmap v1886: 37 pgs: 37 active+clean; 5.8 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 896 KiB/s wr, 13 op/s 2024-08-30T23:53:57.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:56 smithi106 ceph-mon[16603]: osdmap e845: 12 total, 12 up, 11 in 2024-08-30T23:53:57.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1555975242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:53:57.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:53:57.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4281875045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395573+0000", 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-30T23:53:57.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395576+0000", 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395586+0000", 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4360000000000008e-06 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395591+0000", 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8910000000000003e-06 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395638+0000", 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7027000000000002e-05 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395723+0000", 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5181999999999999e-05 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395790+0000", 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7430999999999999e-05 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400118+0000", 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043280920000000004 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400495+0000", 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037659899999999998 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012439+0000", 2024-08-30T23:53:57.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61194435700000005 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012480+0000", 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1180000000000002e-05 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012512+0000", 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1869999999999998e-05 2024-08-30T23:53:57.318 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145475 7.4 7:2251fdb7:::benchmark_data_smithi173_129296_object58359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.101390097999996, 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61919969799999997, 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.319 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145475 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173010+0000", 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7190000000000001e-06 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173019+0000", 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6289999999999999e-06 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173041+0000", 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658000000000002e-05 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173051+0000", 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0139999999999999e-05 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173114+0000", 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3289999999999999e-05 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173214+0000", 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100282 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173291+0000", 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6983999999999993e-05 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.177105+0000", 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003813355 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.178585+0000", 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001480608 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792092+0000", 2024-08-30T23:53:57.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61350662700000003 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792154+0000", 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1884000000000006e-05 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792207+0000", 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3522e-05 2024-08-30T23:53:57.326 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145497 7.4 7:2208bbfd:::benchmark_data_smithi173_129296_object58381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.066004715999995, 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60666463800000003, 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.327 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145497 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208395+0000", 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.66e-06 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208403+0000", 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2309999999999996e-06 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208421+0000", 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7825999999999998e-05 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208428+0000", 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0539999999999997e-06 2024-08-30T23:53:57.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208486+0000", 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8517000000000002e-05 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208625+0000", 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138517 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208748+0000", 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012329600000000001 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.212469+0000", 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037205519999999998 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.213491+0000", 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010220489999999999 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.814962+0000", 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.601471588 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815023+0000", 2024-08-30T23:53:57.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0732000000000001e-05 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815058+0000", 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4616000000000002e-05 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145502 7.4 7:2d669461:::benchmark_data_smithi173_129296_object58386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.058521632999998, 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59929806799999996, 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145502 2024-08-30T23:53:57.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215878+0000", 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6220000000000001e-06 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215886+0000", 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9219999999999997e-06 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215902+0000", 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6677e-05 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215910+0000", 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8499999999999994e-06 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215967+0000", 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6898999999999999e-05 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216108+0000", 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014146699999999999 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216238+0000", 2024-08-30T23:53:57.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012992900000000001 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223310+0000", 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070715079999999998 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224153+0000", 2024-08-30T23:53:57.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084321899999999998 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815104+0000", 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59095095600000003 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815137+0000", 2024-08-30T23:53:57.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2759000000000003e-05 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815174+0000", 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7259999999999999e-05 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.342 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145504 7.4 7:2d76a73e:::benchmark_data_smithi173_129296_object58388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.054756964999996, 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59561452400000003, 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145504 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.343 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219642+0000", 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.782e-06 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219653+0000", 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0723999999999999e-05 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219670+0000", 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.681e-05 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219679+0000", 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219829+0000", 2024-08-30T23:53:57.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149863 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219920+0000", 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1633999999999997e-05 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.220015+0000", 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4344000000000003e-05 2024-08-30T23:53:57.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223383+0000", 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033687449999999998 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224195+0000", 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081209999999999995 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815208+0000", 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59101230400000004 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815235+0000", 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7310000000000001e-05 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815255+0000", 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0075e-05 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145510 7.4 7:2d7edcbd:::benchmark_data_smithi173_129296_object58394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:53:57.350 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.033902468999997, 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.57487306000000005, 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145510 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240497+0000", 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.381e-06 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240502+0000", 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.575e-06 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240514+0000", 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1996e-05 2024-08-30T23:53:57.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240522+0000", 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7079999999999997e-06 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240565+0000", 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3095e-05 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240644+0000", 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9509000000000001e-05 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240712+0000", 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7859000000000002e-05 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.244050+0000", 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003337727 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.245826+0000", 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017758229999999999 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815284+0000", 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56945848700000001 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815331+0000", 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6276999999999998e-05 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815368+0000", 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7623000000000002e-05 2024-08-30T23:53:57.357 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145529 7.4 7:2339dd6d:::benchmark_data_smithi173_129296_object58413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.979321583000001, 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53035033899999995, 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145529 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295077+0000", 2024-08-30T23:53:57.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.47e-07 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295085+0000", 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9750000000000007e-06 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295097+0000", 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2391999999999999e-05 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295103+0000", 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8529999999999997e-06 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295150+0000", 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6684000000000002e-05 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295235+0000", 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4892e-05 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295302+0000", 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6934999999999995e-05 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.298587+0000", 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032851030000000002 2024-08-30T23:53:57.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.299413+0000", 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00082625299999999999 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825357+0000", 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52594381499999998 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825395+0000", 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7898999999999997e-05 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825426+0000", 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1591000000000003e-05 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.365 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145538 7.4 7:2666a83b:::benchmark_data_smithi173_129296_object58422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.952307571999995, 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51489389200000002, 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.366 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145538 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:53:57.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322092+0000", 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.683e-06 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322100+0000", 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:53:57.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322119+0000", 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9368000000000002e-05 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322128+0000", 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1599999999999998e-06 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322189+0000", 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0976e-05 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322326+0000", 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013735099999999999 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322456+0000", 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012996399999999999 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326052+0000", 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035965490000000001 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328903+0000", 2024-08-30T23:53:57.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028504350000000001 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836909+0000", 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.508006556 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836949+0000", 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9864000000000003e-05 2024-08-30T23:53:57.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836984+0000", 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4674e-05 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145540 7.4 7:22c5e8c4:::benchmark_data_smithi173_129296_object58424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:53:57.373 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.948139679999997, 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52274398799999999, 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145540 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326259+0000", 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2440000000000001e-06 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326268+0000", 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3760000000000008e-06 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326280+0000", 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2665e-05 2024-08-30T23:53:57.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326288+0000", 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326325+0000", 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6508999999999999e-05 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326403+0000", 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7877e-05 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326473+0000", 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9820999999999994e-05 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328987+0000", 2024-08-30T23:53:57.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025140520000000001 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.330596+0000", 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001609367 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848889+0000", 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51829265199999996 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848951+0000", 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2806999999999997e-05 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849002+0000", 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0544000000000003e-05 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145558 7.4 7:2fe416e2:::benchmark_data_smithi173_129296_object58442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.868713763000002, 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44345267799999999, 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145558 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:53:57.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405686+0000", 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4059999999999999e-06 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405696+0000", 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0073e-05 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405726+0000", 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9686999999999999e-05 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405740+0000", 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4072e-05 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405842+0000", 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010199700000000001 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405928+0000", 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5784000000000004e-05 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.406015+0000", 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7096999999999996e-05 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.409500+0000", 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034854629999999998 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.411030+0000", 2024-08-30T23:53:57.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015296820000000001 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849046+0000", 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43801564300000001 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849097+0000", 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1424000000000001e-05 2024-08-30T23:53:57.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849137+0000", 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9350000000000001e-05 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145576 7.4 7:2d26fcfb:::benchmark_data_smithi173_129296_object58460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:53:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.716532908000005, 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29138068, 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145576 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557867+0000", 2024-08-30T23:53:57.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9910000000000001e-06 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557875+0000", 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557897+0000", 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2021000000000001e-05 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557906+0000", 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3360000000000001e-06 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557987+0000", 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1238999999999999e-05 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.558139+0000", 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000151368 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.558263+0000", 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123824 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.561781+0000", 2024-08-30T23:53:57.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035184819999999999 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.562154+0000", 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037294599999999998 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849181+0000", 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28702652299999998 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849221+0000", 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0670000000000002e-05 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849245+0000", 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4122e-05 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:164938 7.4 7:21341da6:::benchmark_data_smithi173_129296_object77822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e825)", 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:18.564906+0000", 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.709491870000001, 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21211349400000001, 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 164938 2024-08-30T23:53:57.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564906+0000", 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564906+0000", 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564907+0000", 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564911+0000", 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.313e-06 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564921+0000", 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036e-05 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564928+0000", 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9419999999999998e-06 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.564979+0000", 2024-08-30T23:53:57.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1354999999999999e-05 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.565101+0000", 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121558 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.565197+0000", 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5783000000000001e-05 2024-08-30T23:53:57.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.569306+0000", 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041089459999999996 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.577677+0000", 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083709560000000006 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.776935+0000", 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19925833600000001 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.776974+0000", 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8813999999999998e-05 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.777019+0000", 2024-08-30T23:53:57.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5485000000000002e-05 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:3221 8.e 8:7d12e9a6:::benchmark_data_smithi173_143816_object3220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e846)", 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:57.061771+0000", 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.212626128, 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019556773999999999, 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.403 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3221 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061771+0000", 2024-08-30T23:53:57.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061771+0000", 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061773+0000", 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.457e-06 2024-08-30T23:53:57.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061854+0000", 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1110000000000001e-05 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061868+0000", 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4183e-05 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061877+0000", 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6759999999999996e-06 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.061935+0000", 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7840999999999997e-05 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.062098+0000", 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016348900000000001 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.062242+0000", 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014381299999999999 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.062740+0000", 2024-08-30T23:53:57.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00049759299999999995 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.063148+0000", 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040792299999999998 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.081233+0000", 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018085740999999999 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.081288+0000", 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4959e-05 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.081328+0000", 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9988999999999999e-05 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:3598 8.e 8:70f4850d:::benchmark_data_smithi173_143816_object3597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e846)", 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:57.265078+0000", 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0093200209999999995, 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001388525, 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 3598 2024-08-30T23:53:57.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265078+0000", 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265078+0000", 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:53:57.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265079+0000", 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.491e-06 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265130+0000", 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0513000000000002e-05 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265141+0000", 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1781e-05 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265147+0000", 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.823e-06 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265199+0000", 2024-08-30T23:53:57.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1523999999999997e-05 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265340+0000", 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141648 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265439+0000", 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9098000000000006e-05 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.265812+0000", 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037215799999999998 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.266337+0000", 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052555500000000003 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.266413+0000", 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5692000000000006e-05 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.266442+0000", 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.889e-05 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:57.266466+0000", 2024-08-30T23:53:57.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4352e-05 2024-08-30T23:53:57.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.418 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:53:57.418 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:53:57.418 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:53:57.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:57.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:56 smithi173 ceph-mon[20554]: pgmap v1886: 37 pgs: 37 active+clean; 5.8 MiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 896 KiB/s wr, 13 op/s 2024-08-30T23:53:57.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:56 smithi173 ceph-mon[20554]: osdmap e845: 12 total, 12 up, 11 in 2024-08-30T23:53:57.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1555975242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:53:57.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:53:57.420 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4281875045' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:53:57.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:53:57.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 15 3815 3800 118.741 114.625 0.00203768 0.00840004 2024-08-30T23:53:57.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:53:57.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:57.456 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:53:57.456 INFO:teuthology.orchestra.run.smithi106.stdout:{"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":2052,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":846,"num_osds":12,"num_up_osds":12,"osd_up_since":1725061550,"num_in_osds":11,"osd_in_since":1725061817,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":37}],"num_pgs":37,"num_pools":2,"num_objects":86,"data_bytes":6095392,"bytes_used":6724743168,"bytes_avail":1049220648960,"bytes_total":1055945392128,"write_bytes_sec":917334,"read_op_per_sec":0,"write_op_per_sec":13},"fsmap":{"epoch":1,"btime":"2024-08-30T23:18:02:086399+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.106:8443/"}},"servicemap":{"epoch":48,"modified":"2024-08-30T23:53:07.769156+0000","services":{}},"progress_events":{}} 2024-08-30T23:53:57.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:57.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:57.671 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:57.695 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:57.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:57.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:57.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:53:57.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:57.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:53:58.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:58.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:58.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:58.172 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:53:58.193 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:53:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:57 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:57 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:53:58.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:57 smithi136 ceph-mon[20207]: osdmap e846: 12 total, 12 up, 11 in 2024-08-30T23:53:58.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:53:58.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:53:58.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2012546805' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:53:58.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:53:58.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:53:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:57 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:57 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:53:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:57 smithi173 ceph-mon[20554]: osdmap e846: 12 total, 12 up, 11 in 2024-08-30T23:53:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:53:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:53:58.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2012546805' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:53:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:57 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:53:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:57 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:53:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:57 smithi106 ceph-mon[16603]: osdmap e846: 12 total, 12 up, 11 in 2024-08-30T23:53:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:53:58.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:53:58.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2012546805' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-30T23:53:58.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5222 5206 108.447 87.875 0.00279244 0.00910789 2024-08-30T23:53:58.413 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-30T23:53:58.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:53:58.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:58.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:58.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.570 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:58.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:58.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:53:58.671 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:58.697 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:53:58.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:53:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.821 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:58.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:53:58.912 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:53:58.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:53:58.944 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:58.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:53:59.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:59.067 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:58 smithi106 ceph-mon[16603]: pgmap v1889: 37 pgs: 37 active+clean; 97 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 16 MiB/s wr, 256 op/s 2024-08-30T23:53:59.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:53:58 smithi106 ceph-mon[16603]: osdmap e847: 12 total, 12 up, 11 in 2024-08-30T23:53:59.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:53:59.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:53:59.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:58 smithi136 ceph-mon[20207]: pgmap v1889: 37 pgs: 37 active+clean; 97 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 16 MiB/s wr, 256 op/s 2024-08-30T23:53:59.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:53:58 smithi136 ceph-mon[20207]: osdmap e847: 12 total, 12 up, 11 in 2024-08-30T23:53:59.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:59.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:59.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:58 smithi173 ceph-mon[20554]: pgmap v1889: 37 pgs: 37 active+clean; 97 MiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 16 MiB/s wr, 256 op/s 2024-08-30T23:53:59.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:53:58 smithi173 ceph-mon[20554]: osdmap e847: 12 total, 12 up, 11 in 2024-08-30T23:53:59.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 6466 6450 100.771 77.75 0.00141083 0.00988131 2024-08-30T23:53:59.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:59.425 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:53:59.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:59.467 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:53:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:53:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:53:59.468 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:53:59.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:53:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:53:59.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:53:59.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:53:59.701 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:53:59.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:53:59.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:53:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.829 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:53:59.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:53:59.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:53:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:53:59.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:00.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:00.083 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:00.134 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:54:00.135 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:54:00.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:00.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:00.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:54:00.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:00.335 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:00.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 7862 7846 98.0652 87.25 0.0181446 0.0101143 2024-08-30T23:54:00.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:00.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:00.517 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:00.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:00.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:00.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:00.715 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:00.817 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:00.841 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:00.849 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:54:00.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:00.968 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:00.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:01.055 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1891,"stamp":"2024-08-30T23:53:59.788593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237240864,"num_objects":3613,"num_object_clones":0,"num_object_copies":10839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":231104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5543,"ondisk_log_size":5543,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":7268256,"kb_used_data":1154232,"kb_used_omap":107,"kb_used_meta":6113684,"kb_avail":1023928416,"statfs":{"total":1055945392128,"available":1048502697984,"internally_reserved":0,"allocated":1181933568,"data_stored":1176090442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110380,"internal_metadata":6260412628},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":231104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001124"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":652,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.982354+0000","last_change":"2024-08-30T23:53:42.112471+0000","last_active":"2024-08-30T23:53:55.982354+0000","last_peered":"2024-08-30T23:53:55.982354+0000","last_clean":"2024-08-30T23:53:55.982354+0000","last_became_active":"2024-08-30T23:50:18.102927+0000","last_became_peered":"2024-08-30T23:50:18.102927+0000","last_unstale":"2024-08-30T23:53:55.982354+0000","last_undegraded":"2024-08-30T23:53:55.982354+0000","last_fullsized":"2024-08-30T23:53:55.982354+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.112421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.564463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":138,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.999923+0000","last_change":"2024-08-30T23:53:38.104290+0000","last_active":"2024-08-30T23:53:57.999923+0000","last_peered":"2024-08-30T23:53:57.999923+0000","last_clean":"2024-08-30T23:53:57.999923+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:53:57.999923+0000","last_undegraded":"2024-08-30T23:53:57.999923+0000","last_fullsized":"2024-08-30T23:53:57.999923+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:38.104244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:38.104244+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:45.249002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":754,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-08-30T23:53:53.681743+0000","last_change":"2024-08-30T23:53:47.855440+0000","last_active":"2024-08-30T23:53:53.681743+0000","last_peered":"2024-08-30T23:53:53.681743+0000","last_clean":"2024-08-30T23:53:53.681743+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:53:53.681743+0000","last_undegraded":"2024-08-30T23:53:53.681743+0000","last_fullsized":"2024-08-30T23:53:53.681743+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:47.855365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:47.855365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.334601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":138,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.999968+0000","last_change":"2024-08-30T23:53:39.095429+0000","last_active":"2024-08-30T23:53:57.999968+0000","last_peered":"2024-08-30T23:53:57.999968+0000","last_clean":"2024-08-30T23:53:57.999968+0000","last_became_active":"2024-08-30T23:51:42.948334+0000","last_became_peered":"2024-08-30T23:51:42.948334+0000","last_unstale":"2024-08-30T23:53:57.999968+0000","last_undegraded":"2024-08-30T23:53:57.999968+0000","last_fullsized":"2024-08-30T23:53:57.999968+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:39.095359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:39.095359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.506854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"845'165","reported_seq":184,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.911912+0000","last_change":"2024-08-30T23:53:49.706683+0000","last_active":"2024-08-30T23:53:56.911912+0000","last_peered":"2024-08-30T23:53:56.911912+0000","last_clean":"2024-08-30T23:53:56.911912+0000","last_became_active":"2024-08-30T23:53:49.706310+0000","last_became_peered":"2024-08-30T23:53:49.706310+0000","last_unstale":"2024-08-30T23:53:56.911912+0000","last_undegraded":"2024-08-30T23:53:56.911912+0000","last_fullsized":"2024-08-30T23:53:56.911912+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.016839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":536,"reported_epoch":846,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.007425+0000","last_change":"2024-08-30T23:52:54.012729+0000","last_active":"2024-08-30T23:53:57.007425+0000","last_peered":"2024-08-30T23:53:57.007425+0000","last_clean":"2024-08-30T23:53:57.007425+0000","last_became_active":"2024-08-30T23:52:54.012402+0000","last_became_peered":"2024-08-30T23:52:54.012402+0000","last_unstale":"2024-08-30T23:53:57.007425+0000","last_undegraded":"2024-08-30T23:53:57.007425+0000","last_fullsized":"2024-08-30T23:53:57.007425+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:02.344155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:02.344155+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"847'466","reported_seq":490,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:59.674109+0000","last_change":"2024-08-30T23:53:49.706496+0000","last_active":"2024-08-30T23:53:59.674109+0000","last_peered":"2024-08-30T23:53:59.674109+0000","last_clean":"2024-08-30T23:53:59.674109+0000","last_became_active":"2024-08-30T23:53:49.706120+0000","last_became_peered":"2024-08-30T23:53:49.706120+0000","last_unstale":"2024-08-30T23:53:59.674109+0000","last_undegraded":"2024-08-30T23:53:59.674109+0000","last_fullsized":"2024-08-30T23:53:59.674109+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:13.019256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":342,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-08-30T23:53:53.681748+0000","last_change":"2024-08-30T23:53:46.814589+0000","last_active":"2024-08-30T23:53:53.681748+0000","last_peered":"2024-08-30T23:53:53.681748+0000","last_clean":"2024-08-30T23:53:53.681748+0000","last_became_active":"2024-08-30T23:50:18.099827+0000","last_became_peered":"2024-08-30T23:50:18.099827+0000","last_unstale":"2024-08-30T23:53:53.681748+0000","last_undegraded":"2024-08-30T23:53:53.681748+0000","last_fullsized":"2024-08-30T23:53:53.681748+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:46.814527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:46.814527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.311584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"844'27","reported_seq":44,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.600393+0000","last_change":"2024-08-30T23:53:49.707214+0000","last_active":"2024-08-30T23:53:55.600393+0000","last_peered":"2024-08-30T23:53:55.600393+0000","last_clean":"2024-08-30T23:53:55.600393+0000","last_became_active":"2024-08-30T23:53:49.706776+0000","last_became_peered":"2024-08-30T23:53:49.706776+0000","last_unstale":"2024-08-30T23:53:55.600393+0000","last_undegraded":"2024-08-30T23:53:55.600393+0000","last_fullsized":"2024-08-30T23:53:55.600393+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.556043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":333,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.982424+0000","last_change":"2024-08-30T23:53:51.042769+0000","last_active":"2024-08-30T23:53:55.982424+0000","last_peered":"2024-08-30T23:53:55.982424+0000","last_clean":"2024-08-30T23:53:55.982424+0000","last_became_active":"2024-08-30T23:52:07.960637+0000","last_became_peered":"2024-08-30T23:52:07.960637+0000","last_unstale":"2024-08-30T23:53:55.982424+0000","last_undegraded":"2024-08-30T23:53:55.982424+0000","last_fullsized":"2024-08-30T23:53:55.982424+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.042710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.042710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.950115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"845'173","reported_seq":194,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.718047+0000","last_change":"2024-08-30T23:53:49.706382+0000","last_active":"2024-08-30T23:53:56.718047+0000","last_peered":"2024-08-30T23:53:56.718047+0000","last_clean":"2024-08-30T23:53:56.718047+0000","last_became_active":"2024-08-30T23:53:49.706069+0000","last_became_peered":"2024-08-30T23:53:49.706069+0000","last_unstale":"2024-08-30T23:53:56.718047+0000","last_undegraded":"2024-08-30T23:53:56.718047+0000","last_fullsized":"2024-08-30T23:53:56.718047+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:07.084129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":307,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.982532+0000","last_change":"2024-08-30T23:53:43.091100+0000","last_active":"2024-08-30T23:53:55.982532+0000","last_peered":"2024-08-30T23:53:55.982532+0000","last_clean":"2024-08-30T23:53:55.982532+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:53:55.982532+0000","last_undegraded":"2024-08-30T23:53:55.982532+0000","last_fullsized":"2024-08-30T23:53:55.982532+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.091048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.091048+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:10.926425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"845'171","reported_seq":191,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.664322+0000","last_change":"2024-08-30T23:53:49.706502+0000","last_active":"2024-08-30T23:53:56.664322+0000","last_peered":"2024-08-30T23:53:56.664322+0000","last_clean":"2024-08-30T23:53:56.664322+0000","last_became_active":"2024-08-30T23:53:49.705978+0000","last_became_peered":"2024-08-30T23:53:49.705978+0000","last_unstale":"2024-08-30T23:53:56.664322+0000","last_undegraded":"2024-08-30T23:53:56.664322+0000","last_fullsized":"2024-08-30T23:53:56.664322+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.097093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":948,"reported_epoch":846,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.007669+0000","last_change":"2024-08-30T23:53:51.507308+0000","last_active":"2024-08-30T23:53:57.007669+0000","last_peered":"2024-08-30T23:53:57.007669+0000","last_clean":"2024-08-30T23:53:57.007669+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:53:57.007669+0000","last_undegraded":"2024-08-30T23:53:57.007669+0000","last_fullsized":"2024-08-30T23:53:57.007669+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.507248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.507248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.951121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"847'385","reported_seq":408,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:59.682811+0000","last_change":"2024-08-30T23:53:49.708022+0000","last_active":"2024-08-30T23:53:59.682811+0000","last_peered":"2024-08-30T23:53:59.682811+0000","last_clean":"2024-08-30T23:53:59.682811+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:53:59.682811+0000","last_undegraded":"2024-08-30T23:53:59.682811+0000","last_fullsized":"2024-08-30T23:53:59.682811+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.990219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":385,"num_write_kb":24640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":298,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.982470+0000","last_change":"2024-08-30T23:53:41.104980+0000","last_active":"2024-08-30T23:53:55.982470+0000","last_peered":"2024-08-30T23:53:55.982470+0000","last_clean":"2024-08-30T23:53:55.982470+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:53:55.982470+0000","last_undegraded":"2024-08-30T23:53:55.982470+0000","last_fullsized":"2024-08-30T23:53:55.982470+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.104925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.104925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:09.944567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"844'24","reported_seq":42,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.608751+0000","last_change":"2024-08-30T23:53:49.671937+0000","last_active":"2024-08-30T23:53:55.608751+0000","last_peered":"2024-08-30T23:53:55.608751+0000","last_clean":"2024-08-30T23:53:55.608751+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:53:55.608751+0000","last_undegraded":"2024-08-30T23:53:55.608751+0000","last_fullsized":"2024-08-30T23:53:55.608751+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:14.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1572864,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1789,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.981548+0000","last_change":"2024-08-30T23:53:50.999427+0000","last_active":"2024-08-30T23:53:55.981548+0000","last_peered":"2024-08-30T23:53:55.981548+0000","last_clean":"2024-08-30T23:53:55.981548+0000","last_became_active":"2024-08-30T23:52:54.016899+0000","last_became_peered":"2024-08-30T23:52:54.016899+0000","last_unstale":"2024-08-30T23:53:55.981548+0000","last_undegraded":"2024-08-30T23:53:55.981548+0000","last_fullsized":"2024-08-30T23:53:55.981548+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:50.999362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:50.999362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.278203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"844'33","reported_seq":50,"reported_epoch":844,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.606797+0000","last_change":"2024-08-30T23:53:49.709301+0000","last_active":"2024-08-30T23:53:55.606797+0000","last_peered":"2024-08-30T23:53:55.606797+0000","last_clean":"2024-08-30T23:53:55.606797+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:53:55.606797+0000","last_undegraded":"2024-08-30T23:53:55.606797+0000","last_fullsized":"2024-08-30T23:53:55.606797+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.536121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":111,"reported_epoch":846,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.007536+0000","last_change":"2024-08-30T23:52:08.307369+0000","last_active":"2024-08-30T23:53:57.007536+0000","last_peered":"2024-08-30T23:53:57.007536+0000","last_clean":"2024-08-30T23:53:57.007536+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:53:57.007536+0000","last_undegraded":"2024-08-30T23:53:57.007536+0000","last_fullsized":"2024-08-30T23:53:57.007536+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:08.307325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:08.307325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"845'200","reported_seq":219,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.922972+0000","last_change":"2024-08-30T23:53:49.670752+0000","last_active":"2024-08-30T23:53:56.922972+0000","last_peered":"2024-08-30T23:53:56.922972+0000","last_clean":"2024-08-30T23:53:56.922972+0000","last_became_active":"2024-08-30T23:53:49.670369+0000","last_became_peered":"2024-08-30T23:53:49.670369+0000","last_unstale":"2024-08-30T23:53:56.922972+0000","last_undegraded":"2024-08-30T23:53:56.922972+0000","last_fullsized":"2024-08-30T23:53:56.922972+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.808262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":200,"num_write_kb":12800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":768,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.001286+0000","last_change":"2024-08-30T23:53:49.354762+0000","last_active":"2024-08-30T23:53:58.001286+0000","last_peered":"2024-08-30T23:53:58.001286+0000","last_clean":"2024-08-30T23:53:58.001286+0000","last_became_active":"2024-08-30T23:50:25.926833+0000","last_became_peered":"2024-08-30T23:50:25.926833+0000","last_unstale":"2024-08-30T23:53:58.001286+0000","last_undegraded":"2024-08-30T23:53:58.001286+0000","last_fullsized":"2024-08-30T23:53:58.001286+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.354704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.354704+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.152954+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"847'405","reported_seq":430,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:59.652579+0000","last_change":"2024-08-30T23:53:49.707268+0000","last_active":"2024-08-30T23:53:59.652579+0000","last_peered":"2024-08-30T23:53:59.652579+0000","last_clean":"2024-08-30T23:53:59.652579+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:53:59.652579+0000","last_undegraded":"2024-08-30T23:53:59.652579+0000","last_fullsized":"2024-08-30T23:53:59.652579+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.387982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":405,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":754,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.981517+0000","last_change":"2024-08-30T23:53:41.049687+0000","last_active":"2024-08-30T23:53:55.981517+0000","last_peered":"2024-08-30T23:53:55.981517+0000","last_clean":"2024-08-30T23:53:55.981517+0000","last_became_active":"2024-08-30T23:52:54.016432+0000","last_became_peered":"2024-08-30T23:52:54.016432+0000","last_unstale":"2024-08-30T23:53:55.981517+0000","last_undegraded":"2024-08-30T23:53:55.981517+0000","last_fullsized":"2024-08-30T23:53:55.981517+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.049626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.049626+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.750692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"845'194","reported_seq":213,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.926243+0000","last_change":"2024-08-30T23:53:49.706802+0000","last_active":"2024-08-30T23:53:56.926243+0000","last_peered":"2024-08-30T23:53:56.926243+0000","last_clean":"2024-08-30T23:53:56.926243+0000","last_became_active":"2024-08-30T23:53:49.706231+0000","last_became_peered":"2024-08-30T23:53:49.706231+0000","last_unstale":"2024-08-30T23:53:56.926243+0000","last_undegraded":"2024-08-30T23:53:56.926243+0000","last_fullsized":"2024-08-30T23:53:56.926243+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:13.051742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":194,"num_write_kb":12416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":685,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:55.981485+0000","last_change":"2024-08-30T23:53:35.098110+0000","last_active":"2024-08-30T23:53:55.981485+0000","last_peered":"2024-08-30T23:53:55.981485+0000","last_clean":"2024-08-30T23:53:55.981485+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:53:55.981485+0000","last_undegraded":"2024-08-30T23:53:55.981485+0000","last_fullsized":"2024-08-30T23:53:55.981485+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:35.098031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:35.098031+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.053662+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"845'209","reported_seq":229,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.913093+0000","last_change":"2024-08-30T23:53:49.707500+0000","last_active":"2024-08-30T23:53:56.913093+0000","last_peered":"2024-08-30T23:53:56.913093+0000","last_clean":"2024-08-30T23:53:56.913093+0000","last_became_active":"2024-08-30T23:53:49.707122+0000","last_became_peered":"2024-08-30T23:53:49.707122+0000","last_unstale":"2024-08-30T23:53:56.913093+0000","last_undegraded":"2024-08-30T23:53:56.913093+0000","last_fullsized":"2024-08-30T23:53:56.913093+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:57.343508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13697024,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":209,"num_write_kb":13376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":950,"reported_epoch":846,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.007576+0000","last_change":"2024-08-30T23:52:07.301252+0000","last_active":"2024-08-30T23:53:57.007576+0000","last_peered":"2024-08-30T23:53:57.007576+0000","last_clean":"2024-08-30T23:53:57.007576+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:53:57.007576+0000","last_undegraded":"2024-08-30T23:53:57.007576+0000","last_fullsized":"2024-08-30T23:53:57.007576+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:07.301208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:07.301208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"845'152","reported_seq":173,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.701816+0000","last_change":"2024-08-30T23:53:49.706361+0000","last_active":"2024-08-30T23:53:56.701816+0000","last_peered":"2024-08-30T23:53:56.701816+0000","last_clean":"2024-08-30T23:53:56.701816+0000","last_became_active":"2024-08-30T23:53:49.706070+0000","last_became_peered":"2024-08-30T23:53:49.706070+0000","last_unstale":"2024-08-30T23:53:56.701816+0000","last_undegraded":"2024-08-30T23:53:56.701816+0000","last_fullsized":"2024-08-30T23:53:56.701816+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:02.364402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":152,"num_write_kb":9728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":151,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.999873+0000","last_change":"2024-08-30T23:53:43.164672+0000","last_active":"2024-08-30T23:53:57.999873+0000","last_peered":"2024-08-30T23:53:57.999873+0000","last_clean":"2024-08-30T23:53:57.999873+0000","last_became_active":"2024-08-30T23:51:42.952967+0000","last_became_peered":"2024-08-30T23:51:42.952967+0000","last_unstale":"2024-08-30T23:53:57.999873+0000","last_undegraded":"2024-08-30T23:53:57.999873+0000","last_fullsized":"2024-08-30T23:53:57.999873+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.164609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.164609+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.951863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"847'361","reported_seq":386,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.765041+0000","last_change":"2024-08-30T23:53:49.706815+0000","last_active":"2024-08-30T23:53:58.765041+0000","last_peered":"2024-08-30T23:53:58.765041+0000","last_clean":"2024-08-30T23:53:58.765041+0000","last_became_active":"2024-08-30T23:53:49.706543+0000","last_became_peered":"2024-08-30T23:53:49.706543+0000","last_unstale":"2024-08-30T23:53:58.765041+0000","last_undegraded":"2024-08-30T23:53:58.765041+0000","last_fullsized":"2024-08-30T23:53:58.765041+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.291074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":361,"num_write_kb":23104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1769,"reported_epoch":846,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.994360+0000","last_change":"2024-08-30T23:53:42.303311+0000","last_active":"2024-08-30T23:53:56.994360+0000","last_peered":"2024-08-30T23:53:56.994360+0000","last_clean":"2024-08-30T23:53:56.994360+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:53:56.994360+0000","last_undegraded":"2024-08-30T23:53:56.994360+0000","last_fullsized":"2024-08-30T23:53:56.994360+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.303263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.303263+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:47.897994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"847'454","reported_seq":478,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:59.641798+0000","last_change":"2024-08-30T23:53:49.879494+0000","last_active":"2024-08-30T23:53:59.641798+0000","last_peered":"2024-08-30T23:53:59.641798+0000","last_clean":"2024-08-30T23:53:59.641798+0000","last_became_active":"2024-08-30T23:53:49.879153+0000","last_became_peered":"2024-08-30T23:53:49.879153+0000","last_unstale":"2024-08-30T23:53:59.641798+0000","last_undegraded":"2024-08-30T23:53:59.641798+0000","last_fullsized":"2024-08-30T23:53:59.641798+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:11.036607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753344,"num_objects":454,"num_object_clones":0,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":29056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":885,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.997958+0000","last_change":"2024-08-30T23:53:49.897094+0000","last_active":"2024-08-30T23:53:57.997958+0000","last_peered":"2024-08-30T23:53:57.997958+0000","last_clean":"2024-08-30T23:53:57.997958+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:53:57.997958+0000","last_undegraded":"2024-08-30T23:53:57.997958+0000","last_fullsized":"2024-08-30T23:53:57.997958+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.897038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.897038+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.963756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"845'192","reported_seq":211,"reported_epoch":845,"state":"active+clean","last_fresh":"2024-08-30T23:53:56.925527+0000","last_change":"2024-08-30T23:53:49.706603+0000","last_active":"2024-08-30T23:53:56.925527+0000","last_peered":"2024-08-30T23:53:56.925527+0000","last_clean":"2024-08-30T23:53:56.925527+0000","last_became_active":"2024-08-30T23:53:49.706194+0000","last_became_peered":"2024-08-30T23:53:49.706194+0000","last_unstale":"2024-08-30T23:53:56.925527+0000","last_undegraded":"2024-08-30T23:53:56.925527+0000","last_fullsized":"2024-08-30T23:53:56.925527+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:04.835283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":139,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:57.999975+0000","last_change":"2024-08-30T23:53:41.078765+0000","last_active":"2024-08-30T23:53:57.999975+0000","last_peered":"2024-08-30T23:53:57.999975+0000","last_clean":"2024-08-30T23:53:57.999975+0000","last_became_active":"2024-08-30T23:51:42.950294+0000","last_became_peered":"2024-08-30T23:51:42.950294+0000","last_unstale":"2024-08-30T23:53:57.999975+0000","last_undegraded":"2024-08-30T23:53:57.999975+0000","last_fullsized":"2024-08-30T23:53:57.999975+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.078720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.078720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:44.759291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1313,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.003914+0000","last_change":"2024-08-30T23:53:54.855614+0000","last_active":"2024-08-30T23:53:58.003914+0000","last_peered":"2024-08-30T23:53:58.003914+0000","last_clean":"2024-08-30T23:53:58.003914+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:53:58.003914+0000","last_undegraded":"2024-08-30T23:53:58.003914+0000","last_fullsized":"2024-08-30T23:53:58.003914+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:54.855553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:54.855553+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:02.123392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":231104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":892076032,"data_stored":892076032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3611,"ondisk_log_size":3611,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":369367187809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449450,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708708,"kb_used_data":222160,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93036444,"statfs":{"total":95995035648,"available":95269318656,"internally_reserved":0,"allocated":227491840,"data_stored":226965630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":498203072},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711080,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783928,"kb_used_data":139112,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92961224,"statfs":{"total":95995035648,"available":95192293376,"internally_reserved":0,"allocated":142450688,"data_stored":141915564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972721,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":474072,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93271080,"statfs":{"total":95995035648,"available":95509585920,"internally_reserved":0,"allocated":1445888,"data_stored":924160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234358,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":726964,"kb_used_data":32156,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93018188,"statfs":{"total":95995035648,"available":95250624512,"internally_reserved":0,"allocated":32927744,"data_stored":32397052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":711457226},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528700,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501220,"kb_used_data":10512,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93243932,"statfs":{"total":95995035648,"available":95481786368,"internally_reserved":0,"allocated":10764288,"data_stored":10238078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775203,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329412,"kb_used_data":143856,"kb_used_omap":5,"kb_used_meta":185530,"kb_avail":93415740,"statfs":{"total":95995035648,"available":95657717760,"internally_reserved":0,"allocated":147308544,"data_stored":146769868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":189982794},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778136,"kb_used_data":129156,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92967016,"statfs":{"total":95995035648,"available":95198224384,"internally_reserved":0,"allocated":132255744,"data_stored":131734016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150089,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1010952,"kb_used_data":217588,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92734200,"statfs":{"total":95995035648,"available":94959820800,"internally_reserved":0,"allocated":222810112,"data_stored":222267364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":812382663},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411727,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758920,"kb_used_data":168240,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92986232,"statfs":{"total":95995035648,"available":95217901568,"internally_reserved":0,"allocated":172277760,"data_stored":171739084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830543,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":383912,"kb_used_data":31120,"kb_used_omap":87,"kb_used_meta":352680,"kb_avail":93361240,"statfs":{"total":95995035648,"available":95601909760,"internally_reserved":0,"allocated":31866880,"data_stored":31340670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":361144479},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738777,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":812032,"kb_used_data":58920,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92933120,"statfs":{"total":95995035648,"available":95163514880,"internally_reserved":0,"allocated":60334080,"data_stored":59798956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":70189056,"data_stored":70189056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67043328,"data_stored":67043328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176685056,"data_stored":176685056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68812800,"data_stored":68812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":55508992,"data_stored":55508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":168361984,"data_stored":168361984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15138816,"data_stored":15138816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38338560,"data_stored":38338560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5505024,"data_stored":5505024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":151388160,"data_stored":151388160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":75104256,"data_stored":75104256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:54:01.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:01.078 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:54:01.078 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:54:01.078 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:54:01.078 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:54:01.078 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:54:01.097 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:01.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:01.213 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:01.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:01 smithi136 ceph-mon[20207]: pgmap v1891: 37 pgs: 37 active+clean; 226 MiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 38 MiB/s wr, 601 op/s 2024-08-30T23:54:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:01 smithi136 ceph-mon[20207]: from='client.27743 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:01 smithi173 ceph-mon[20554]: pgmap v1891: 37 pgs: 37 active+clean; 226 MiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 38 MiB/s wr, 601 op/s 2024-08-30T23:54:01.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:01 smithi173 ceph-mon[20554]: from='client.27743 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:01.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:01.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:01 smithi106 ceph-mon[16603]: pgmap v1891: 37 pgs: 37 active+clean; 226 MiB data, 6.9 GiB used, 976 GiB / 983 GiB avail; 38 MiB/s wr, 601 op/s 2024-08-30T23:54:01.341 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:01 smithi106 ceph-mon[16603]: from='client.27743 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:01.353 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:01.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 9750 9734 101.386 118 0.00278538 0.00984314 2024-08-30T23:54:01.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:54:01.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:01.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:01.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.602 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:01.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:01.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:01.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:01.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:01.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:54:01.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:01.935 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:01.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:01.979 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:01.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:02.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:02.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:02.126 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:02.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.232 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:02.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:02.250 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:02 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:02.251 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:02 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:02.251 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2022917763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:02 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:02 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:02.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2022917763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:02.334 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:02.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:02 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:02.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:02 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:02.346 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2022917763' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:02.365 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:02.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 11136 11120 99.2763 86.625 0.0142032 0.010062 2024-08-30T23:54:02.470 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:02.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:02.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:02.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(5.19 e599,595 5:99c57c29:::benchmark_data_smithi173_87639_object64709:head,592'7134)", 2024-08-30T23:54:02.645 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 592.29043084299997, 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41795327700000001, 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:54:02.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353223+0000", 2024-08-30T23:54:02.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3300000000000002e-07 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.353228+0000", 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1930000000000003e-06 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771046+0000", 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.41781854200000001 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771056+0000", 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771111+0000", 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5040999999999997e-05 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771118+0000", 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:54:02.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.771176+0000", 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8e-05 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:155747 6.3 6:c7cf5f70:::benchmark_data_smithi173_114000_object55949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e684)", 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:39.838529+0000", 2024-08-30T23:54:02.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 382.80512487700003, 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.33831727, 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155747 2024-08-30T23:54:02.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838529+0000", 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838529+0000", 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838531+0000", 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.962e-06 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838539+0000", 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-06 2024-08-30T23:54:02.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838558+0000", 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9082000000000001e-05 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838569+0000", 2024-08-30T23:54:02.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1069e-05 2024-08-30T23:54:02.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838628+0000", 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8347999999999998e-05 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838722+0000", 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4955000000000001e-05 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.838799+0000", 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6344e-05 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.842747+0000", 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039481250000000002 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.842865+0000", 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117719 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.176770+0000", 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.333905754 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.176811+0000", 2024-08-30T23:54:02.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0238999999999999e-05 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.176846+0000", 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5333000000000002e-05 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:185798 6.a 6:57949d66:::benchmark_data_smithi173_114000_object86000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:25.388902+0000", 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 337.25475155200002, 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35288384699999997, 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 185798 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.388902+0000", 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.388902+0000", 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.388904+0000", 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5680000000000001e-06 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.388911+0000", 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8870000000000003e-06 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.388923+0000", 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2823000000000001e-05 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.388933+0000", 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5869999999999999e-06 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.389031+0000", 2024-08-30T23:54:02.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7919999999999998e-05 2024-08-30T23:54:02.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.389122+0000", 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1419e-05 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.389205+0000", 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2366000000000004e-05 2024-08-30T23:54:02.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.392769+0000", 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035640099999999998 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.667134+0000", 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27436494299999997 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.741702+0000", 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074568214999999993 2024-08-30T23:54:02.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.741738+0000", 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6384999999999998e-05 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.741786+0000", 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7723999999999999e-05 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.667 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:199583 6.3 6:cfed5daf:::benchmark_data_smithi173_114000_object99785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.140717+0000", 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 316.50293621499998, 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71268692700000003, 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:02.668 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 199583 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140717+0000", 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140717+0000", 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140719+0000", 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4759999999999999e-06 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140728+0000", 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140743+0000", 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5667000000000001e-05 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140750+0000", 2024-08-30T23:54:02.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.742e-06 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140802+0000", 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1521999999999999e-05 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140895+0000", 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3666999999999996e-05 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140982+0000", 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6316000000000005e-05 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.144913+0000", 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003931197 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146322+0000", 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001408786 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853242+0000", 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70692048699999999 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853334+0000", 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2075000000000002e-05 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853404+0000", 2024-08-30T23:54:02.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0159000000000004e-05 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27034.0:199583 6.3 e720/657 ondisk, result = 0)", 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:54:02.675 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 316.49755804500001, 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70731964599999997, 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:54:02.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4000000000000001e-07 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146100+0000", 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1679999999999997e-06 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853064+0000", 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70696384899999998 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853077+0000", 2024-08-30T23:54:02.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2799999999999999e-05 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853219+0000", 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141734 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853233+0000", 2024-08-30T23:54:02.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4042e-05 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853415+0000", 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000182313 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.f 760/756 [PushOp(7:f4cf17c7:::benchmark_data_smithi173_129296_object4282:head, version: 730'263, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f4cf17c7:::benchmark_data_smithi173_129296_object4282:head@730'263, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:24.161126+0000", 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 218.48252791900001, 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53930470500000005, 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161126+0000", 2024-08-30T23:54:02.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161126+0000", 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161127+0000", 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161173+0000", 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5976000000000002e-05 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699030+0000", 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53785763600000003 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699044+0000", 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3576e-05 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699155+0000", 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011168299999999999 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699165+0000", 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5820000000000008e-06 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.700430+0000", 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00126541 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 760/756 [PushReplyOp(7:3fbe9260:::benchmark_data_smithi173_129296_object4371:head)])", 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:24.540715+0000", 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 218.10293891200001, 2024-08-30T23:54:02.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52877982700000004, 2024-08-30T23:54:02.686 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.686 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.686 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:02.686 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540715+0000", 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540715+0000", 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540716+0000", 2024-08-30T23:54:02.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540720+0000", 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5770000000000004e-06 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.065515+0000", 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.524794439 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.065533+0000", 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8516e-05 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.065702+0000", 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016925999999999999 2024-08-30T23:54:02.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.069495+0000", 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037920779999999999 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 766/756 [PushReplyOp(7:323ebbfd:::benchmark_data_smithi173_129296_object11913:head)])", 2024-08-30T23:54:02.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:31.914679+0000", 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 210.728974421, 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63426543899999999, 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914679+0000", 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914679+0000", 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914680+0000", 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181e-06 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914687+0000", 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8299999999999998e-06 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.543684+0000", 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.62899660499999999 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.543700+0000", 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5726000000000002e-05 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.543829+0000", 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012904400000000001 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.548945+0000", 2024-08-30T23:54:02.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051160529999999997 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 769/756 [PushReplyOp(7:3b2647a8:::benchmark_data_smithi173_129296_object24066:head)])", 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:43.867979+0000", 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 198.77567444799999, 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.84462474499999995, 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867979+0000", 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867979+0000", 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867980+0000", 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17e-06 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867986+0000", 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.709251+0000", 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.841265076 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.709278+0000", 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6812999999999999e-05 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.709368+0000", 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0655999999999994e-05 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.712604+0000", 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032354409999999999 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 769/756 [PushReplyOp(7:3f75d97c:::benchmark_data_smithi173_129296_object24809:head)])", 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 197.56280008600001, 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66064861900000005, 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.700 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.701 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:02.701 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:54:02.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:54:02.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9899999999999999e-07 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080861+0000", 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6080000000000001e-06 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.738045+0000", 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65718375100000004 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.738061+0000", 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6336000000000001e-05 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.738224+0000", 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016336000000000001 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.741502+0000", 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032776649999999999 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 787/756 [PushReplyOp(7:38fe4451:::benchmark_data_smithi173_129296_object76141:head)])", 2024-08-30T23:54:02.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.375087081, 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30744231999999999, 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5700000000000002e-07 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268572+0000", 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8579999999999999e-06 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.575023+0000", 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.306450746 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.575031+0000", 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.575086+0000", 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5577e-05 2024-08-30T23:54:02.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.576009+0000", 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092242300000000001 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 788/756 [PushReplyOp(7:3655d6ab:::benchmark_data_smithi173_129296_object76872:head)])", 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:43.176815+0000", 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 139.466838196, 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.254290707, 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176815+0000", 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176815+0000", 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176816+0000", 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9400000000000004e-07 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176824+0000", 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6620000000000005e-06 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.430155+0000", 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.253331419 2024-08-30T23:54:02.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.430163+0000", 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4080000000000001e-06 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.430257+0000", 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4209999999999994e-05 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.431106+0000", 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084921400000000005 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:126882 7.c 7:35900c7d:::benchmark_data_smithi173_129296_object39766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:02.132879+0000", 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 120.51077461600001, 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56331322800000005, 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.718 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 126882 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132879+0000", 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132879+0000", 2024-08-30T23:54:02.720 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132881+0000", 2024-08-30T23:54:02.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1299999999999999e-06 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132890+0000", 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.636e-06 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132912+0000", 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1744999999999999e-05 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132921+0000", 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8449999999999998e-06 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.133008+0000", 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6881999999999999e-05 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.133146+0000", 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013792899999999999 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.133269+0000", 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012271100000000001 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.136955+0000", 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003686506 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.139302+0000", 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023464520000000002 2024-08-30T23:54:02.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.696119+0000", 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.55681687499999999 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.696158+0000", 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9539999999999998e-05 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.696192+0000", 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3976999999999997e-05 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.7 e833,800 7:ead71d3e:::benchmark_data_smithi173_129296_object19426:head,781'6655)", 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.698786806000001, 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.558258531, 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6199999999999998e-07 2024-08-30T23:54:02.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944871+0000", 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7969999999999998e-06 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.502987+0000", 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.55811628899999999 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503016+0000", 2024-08-30T23:54:02.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8617999999999999e-05 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503073+0000", 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6978999999999998e-05 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503083+0000", 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0237999999999999e-05 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503125+0000", 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2147999999999997e-05 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e833,800 7:bc8b5569:::benchmark_data_smithi173_129296_object11595:head,777'6152)", 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:39.231853+0000", 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.411800313000001, 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45119504599999999, 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231853+0000", 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231853+0000", 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231855+0000", 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.663e-06 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231865+0000", 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0277e-05 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682696+0000", 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45083097300000002 2024-08-30T23:54:02.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682714+0000", 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.751e-05 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682880+0000", 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016595600000000001 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682909+0000", 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9620000000000001e-05 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.683048+0000", 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139047 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.7 e833,800 7:eeb0af75:::benchmark_data_smithi173_129296_object21006:head,781'6745)", 2024-08-30T23:54:02.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:39.488031+0000", 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.155622481999998, 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25498426000000002, 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488031+0000", 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488031+0000", 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488033+0000", 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5600000000000001e-06 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488043+0000", 2024-08-30T23:54:02.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.057e-05 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742898+0000", 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25485495600000002 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742903+0000", 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742971+0000", 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7314e-05 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742977+0000", 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6150000000000002e-06 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.743015+0000", 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8027e-05 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e834,800 7:b3496454:::benchmark_data_smithi173_129296_object13177:head,778'6249)", 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:40.366733+0000", 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.276920229000002, 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26684242000000002, 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.739 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366733+0000", 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366733+0000", 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366734+0000", 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366738+0000", 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1910000000000002e-06 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633427+0000", 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26668847600000001 2024-08-30T23:54:02.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633434+0000", 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2429999999999999e-06 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633516+0000", 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1892999999999997e-05 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633524+0000", 2024-08-30T23:54:02.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7319999999999998e-06 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633576+0000", 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2286e-05 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.7 e834,800 7:e204aafa:::benchmark_data_smithi173_129296_object23666:head,782'6902)", 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:40.447246+0000", 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.196407283999999, 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25044494499999997, 2024-08-30T23:54:02.743 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447246+0000", 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447246+0000", 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447247+0000", 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0200000000000002e-07 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447251+0000", 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9689999999999996e-06 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697523+0000", 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.250271937 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697545+0000", 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2374999999999998e-05 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697635+0000", 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9909999999999998e-05 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697643+0000", 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1969999999999996e-06 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697691+0000", 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7954999999999997e-05 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e835,800 7:b9e4363b:::benchmark_data_smithi173_129296_object16399:head,779'6452)", 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:42.354445+0000", 2024-08-30T23:54:02.747 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 20.289208274, 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.946729178, 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354445+0000", 2024-08-30T23:54:02.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354445+0000", 2024-08-30T23:54:02.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354446+0000", 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6199999999999998e-07 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354449+0000", 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2940000000000001e-06 2024-08-30T23:54:02.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301025+0000", 2024-08-30T23:54:02.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94657636899999997 2024-08-30T23:54:02.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301031+0000", 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4519999999999996e-06 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:02.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301123+0000", 2024-08-30T23:54:02.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1745999999999999e-05 2024-08-30T23:54:02.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:02.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301129+0000", 2024-08-30T23:54:02.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.657e-06 2024-08-30T23:54:02.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301175+0000", 2024-08-30T23:54:02.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5198000000000003e-05 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e837,800 7:b580be49:::benchmark_data_smithi173_129296_object20352:head,781'6705)", 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:54:02.756 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.006365897999999, 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36897256299999998, 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6499999999999999e-07 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637295+0000", 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.353e-06 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:46.006128+0000", 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36883366099999998 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:46.006147+0000", 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8706999999999999e-05 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:02.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:46.006260+0000", 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113377 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:02.760 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:54:02.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:02.761 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:02.761 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:02.762 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:02.762 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:02.777 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:02.777 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:02.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:02.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:02.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:02.817 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:02.818 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:02.839 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:02.839 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:02.860 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:02.860 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:02.881 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:02.881 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:02.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:02.902 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:02.902 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:02.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:02.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:02.925 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:02.925 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:02.949 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:02.950 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:03.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:03.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:03.057 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:54:03.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:03.163 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:03.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:03.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:03 smithi136 ceph-mon[20207]: pgmap v1892: 37 pgs: 37 active+clean; 399 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-30T23:54:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:03 smithi136 ceph-mon[20207]: osdmap e848: 12 total, 12 up, 11 in 2024-08-30T23:54:03.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4274534823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:03 smithi173 ceph-mon[20554]: pgmap v1892: 37 pgs: 37 active+clean; 399 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-30T23:54:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:03 smithi173 ceph-mon[20554]: osdmap e848: 12 total, 12 up, 11 in 2024-08-30T23:54:03.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4274534823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:03 smithi106 ceph-mon[16603]: pgmap v1892: 37 pgs: 37 active+clean; 399 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-08-30T23:54:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:03 smithi106 ceph-mon[16603]: osdmap e848: 12 total, 12 up, 11 in 2024-08-30T23:54:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4274534823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:03.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:03.404 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 12206 12190 95.2256 66.875 0.0043035 0.0104897 2024-08-30T23:54:03.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:03.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.538 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:03.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:03.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:03.669 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:03.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:03.800 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:03.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:03.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:03.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:04.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:04.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:04.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:54:04.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:04.182 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:04.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:54:04.284 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:04.288 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:04 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4274534823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:04.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:04 smithi136 ceph-mon[20207]: osdmap e849: 12 total, 12 up, 11 in 2024-08-30T23:54:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:04.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:04.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:04 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4274534823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:04.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:04 smithi173 ceph-mon[20554]: osdmap e849: 12 total, 12 up, 11 in 2024-08-30T23:54:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:04 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4274534823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:04 smithi106 ceph-mon[16603]: osdmap e849: 12 total, 12 up, 11 in 2024-08-30T23:54:04.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 14277 14261 99.0255 129.438 0.00592907 0.0100824 2024-08-30T23:54:04.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:04.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:04.544 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:04.547 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:04.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:04.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:04.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:04.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:04.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:04.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:04.911 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:04.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:04.950 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:04.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:05.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:05.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:05.101 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:05.101 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:05.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:05 smithi106 ceph-mon[16603]: pgmap v1895: 37 pgs: 37 active+clean; 545 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-30T23:54:05.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:05.209 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:05.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:05 smithi136 ceph-mon[20207]: pgmap v1895: 37 pgs: 37 active+clean; 545 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-30T23:54:05.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:05.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:05 smithi173 ceph-mon[20554]: pgmap v1895: 37 pgs: 37 active+clean; 545 MiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-30T23:54:05.333 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.334 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:05.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 15623 15607 97.5347 84.125 0.00708345 0.0102075 2024-08-30T23:54:05.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:05.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:05.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:05.580 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:05.682 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:05.708 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:05.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:54:05.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:05.837 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:05.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:05.966 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:05.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:06.068 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:06.093 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:06.096 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:06.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:06.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:06.277 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:54:06.278 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:54:06.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:06.339 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:06.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 16752 16736 95.0821 70.5625 0.0562681 0.0104497 2024-08-30T23:54:06.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:06.463 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:06.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:54:06.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:06.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:06.609 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:06.609 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:06.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:06.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:06.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:06.844 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:06.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:54:06.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:06.974 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:06.975 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:07.076 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:07.100 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:07.103 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1897,"stamp":"2024-08-30T23:54:06.065031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":823656992,"num_objects":12561,"num_object_clones":0,"num_object_copies":37683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12559,"num_write_kb":803776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14491,"ondisk_log_size":14491,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":8827124,"kb_used_data":2640932,"kb_used_omap":107,"kb_used_meta":6185876,"kb_avail":1022369548,"statfs":{"total":1055945392128,"available":1046906417152,"internally_reserved":0,"allocated":2704314368,"data_stored":2698359842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110380,"internal_metadata":6334337236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,2,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":586416128,"num_objects":8948,"num_object_clones":0,"num_object_copies":26844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":572672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.276438"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":656,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.000496+0000","last_change":"2024-08-30T23:53:42.112471+0000","last_active":"2024-08-30T23:53:58.000496+0000","last_peered":"2024-08-30T23:53:58.000496+0000","last_clean":"2024-08-30T23:53:58.000496+0000","last_became_active":"2024-08-30T23:50:18.102927+0000","last_became_peered":"2024-08-30T23:50:18.102927+0000","last_unstale":"2024-08-30T23:53:58.000496+0000","last_undegraded":"2024-08-30T23:53:58.000496+0000","last_fullsized":"2024-08-30T23:53:58.000496+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.112421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.564463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":143,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.061033+0000","last_change":"2024-08-30T23:53:38.104290+0000","last_active":"2024-08-30T23:54:03.061033+0000","last_peered":"2024-08-30T23:54:03.061033+0000","last_clean":"2024-08-30T23:54:03.061033+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:54:03.061033+0000","last_undegraded":"2024-08-30T23:54:03.061033+0000","last_fullsized":"2024-08-30T23:54:03.061033+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:38.104244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:38.104244+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:45.249002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":764,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.071289+0000","last_change":"2024-08-30T23:53:47.855440+0000","last_active":"2024-08-30T23:54:03.071289+0000","last_peered":"2024-08-30T23:54:03.071289+0000","last_clean":"2024-08-30T23:54:03.071289+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:54:03.071289+0000","last_undegraded":"2024-08-30T23:54:03.071289+0000","last_fullsized":"2024-08-30T23:54:03.071289+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:47.855365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:47.855365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.334601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":143,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.061056+0000","last_change":"2024-08-30T23:53:39.095429+0000","last_active":"2024-08-30T23:54:03.061056+0000","last_peered":"2024-08-30T23:54:03.061056+0000","last_clean":"2024-08-30T23:54:03.061056+0000","last_became_active":"2024-08-30T23:51:42.948334+0000","last_became_peered":"2024-08-30T23:51:42.948334+0000","last_unstale":"2024-08-30T23:54:03.061056+0000","last_undegraded":"2024-08-30T23:54:03.061056+0000","last_fullsized":"2024-08-30T23:54:03.061056+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:39.095359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:39.095359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.506854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"847'613","reported_seq":635,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.924054+0000","last_change":"2024-08-30T23:53:49.706683+0000","last_active":"2024-08-30T23:54:01.924054+0000","last_peered":"2024-08-30T23:54:01.924054+0000","last_clean":"2024-08-30T23:54:01.924054+0000","last_became_active":"2024-08-30T23:53:49.706310+0000","last_became_peered":"2024-08-30T23:53:49.706310+0000","last_unstale":"2024-08-30T23:54:01.924054+0000","last_undegraded":"2024-08-30T23:54:01.924054+0000","last_fullsized":"2024-08-30T23:54:01.924054+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.016839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":613,"num_write_kb":39232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":540,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-08-30T23:54:02.023178+0000","last_change":"2024-08-30T23:52:54.012729+0000","last_active":"2024-08-30T23:54:02.023178+0000","last_peered":"2024-08-30T23:54:02.023178+0000","last_clean":"2024-08-30T23:54:02.023178+0000","last_became_active":"2024-08-30T23:52:54.012402+0000","last_became_peered":"2024-08-30T23:52:54.012402+0000","last_unstale":"2024-08-30T23:54:02.023178+0000","last_undegraded":"2024-08-30T23:54:02.023178+0000","last_fullsized":"2024-08-30T23:54:02.023178+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:02.344155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:02.344155+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:04.501461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"849'919","reported_seq":948,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:04.709886+0000","last_change":"2024-08-30T23:53:49.706496+0000","last_active":"2024-08-30T23:54:04.709886+0000","last_peered":"2024-08-30T23:54:04.709886+0000","last_clean":"2024-08-30T23:54:04.709886+0000","last_became_active":"2024-08-30T23:53:49.706120+0000","last_became_peered":"2024-08-30T23:53:49.706120+0000","last_unstale":"2024-08-30T23:54:04.709886+0000","last_undegraded":"2024-08-30T23:54:04.709886+0000","last_fullsized":"2024-08-30T23:54:04.709886+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:13.019256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60227584,"num_objects":919,"num_object_clones":0,"num_object_copies":2757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":919,"num_write_kb":58816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":352,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.071414+0000","last_change":"2024-08-30T23:53:46.814589+0000","last_active":"2024-08-30T23:54:03.071414+0000","last_peered":"2024-08-30T23:54:03.071414+0000","last_clean":"2024-08-30T23:54:03.071414+0000","last_became_active":"2024-08-30T23:50:18.099827+0000","last_became_peered":"2024-08-30T23:50:18.099827+0000","last_unstale":"2024-08-30T23:54:03.071414+0000","last_undegraded":"2024-08-30T23:54:03.071414+0000","last_fullsized":"2024-08-30T23:54:03.071414+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":756,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:46.814527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:46.814527+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.311584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.c","version":"849'1002","reported_seq":1029,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:05.609342+0000","last_change":"2024-08-30T23:53:49.707214+0000","last_active":"2024-08-30T23:54:05.609342+0000","last_peered":"2024-08-30T23:54:05.609342+0000","last_clean":"2024-08-30T23:54:05.609342+0000","last_became_active":"2024-08-30T23:53:49.706776+0000","last_became_peered":"2024-08-30T23:53:49.706776+0000","last_unstale":"2024-08-30T23:54:05.609342+0000","last_undegraded":"2024-08-30T23:54:05.609342+0000","last_fullsized":"2024-08-30T23:54:05.609342+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.556043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65667072,"num_objects":1002,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1002,"num_write_kb":64128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":337,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.000642+0000","last_change":"2024-08-30T23:53:51.042769+0000","last_active":"2024-08-30T23:53:58.000642+0000","last_peered":"2024-08-30T23:53:58.000642+0000","last_clean":"2024-08-30T23:53:58.000642+0000","last_became_active":"2024-08-30T23:52:07.960637+0000","last_became_peered":"2024-08-30T23:52:07.960637+0000","last_unstale":"2024-08-30T23:53:58.000642+0000","last_undegraded":"2024-08-30T23:53:58.000642+0000","last_fullsized":"2024-08-30T23:53:58.000642+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.042710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.042710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.950115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"847'637","reported_seq":662,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.731567+0000","last_change":"2024-08-30T23:53:49.706382+0000","last_active":"2024-08-30T23:54:01.731567+0000","last_peered":"2024-08-30T23:54:01.731567+0000","last_clean":"2024-08-30T23:54:01.731567+0000","last_became_active":"2024-08-30T23:53:49.706069+0000","last_became_peered":"2024-08-30T23:53:49.706069+0000","last_unstale":"2024-08-30T23:54:01.731567+0000","last_undegraded":"2024-08-30T23:54:01.731567+0000","last_fullsized":"2024-08-30T23:54:01.731567+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:07.084129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41746432,"num_objects":637,"num_object_clones":0,"num_object_copies":1911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":637,"num_write_kb":40768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":311,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.000701+0000","last_change":"2024-08-30T23:53:43.091100+0000","last_active":"2024-08-30T23:53:58.000701+0000","last_peered":"2024-08-30T23:53:58.000701+0000","last_clean":"2024-08-30T23:53:58.000701+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:53:58.000701+0000","last_undegraded":"2024-08-30T23:53:58.000701+0000","last_fullsized":"2024-08-30T23:53:58.000701+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.091048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.091048+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:10.926425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"847'630","reported_seq":654,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.672730+0000","last_change":"2024-08-30T23:53:49.706502+0000","last_active":"2024-08-30T23:54:01.672730+0000","last_peered":"2024-08-30T23:54:01.672730+0000","last_clean":"2024-08-30T23:54:01.672730+0000","last_became_active":"2024-08-30T23:53:49.705978+0000","last_became_peered":"2024-08-30T23:53:49.705978+0000","last_unstale":"2024-08-30T23:54:01.672730+0000","last_undegraded":"2024-08-30T23:54:01.672730+0000","last_fullsized":"2024-08-30T23:54:01.672730+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.097093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41287680,"num_objects":630,"num_object_clones":0,"num_object_copies":1890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":40320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":952,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-08-30T23:54:02.023353+0000","last_change":"2024-08-30T23:53:51.507308+0000","last_active":"2024-08-30T23:54:02.023353+0000","last_peered":"2024-08-30T23:54:02.023353+0000","last_clean":"2024-08-30T23:54:02.023353+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:54:02.023353+0000","last_undegraded":"2024-08-30T23:54:02.023353+0000","last_fullsized":"2024-08-30T23:54:02.023353+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.507248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.507248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.951121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"849'838","reported_seq":865,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:04.727426+0000","last_change":"2024-08-30T23:53:49.708022+0000","last_active":"2024-08-30T23:54:04.727426+0000","last_peered":"2024-08-30T23:54:04.727426+0000","last_clean":"2024-08-30T23:54:04.727426+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:54:04.727426+0000","last_undegraded":"2024-08-30T23:54:04.727426+0000","last_fullsized":"2024-08-30T23:54:04.727426+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.990219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54919168,"num_objects":838,"num_object_clones":0,"num_object_copies":2514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":53632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":302,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.000476+0000","last_change":"2024-08-30T23:53:41.104980+0000","last_active":"2024-08-30T23:53:58.000476+0000","last_peered":"2024-08-30T23:53:58.000476+0000","last_clean":"2024-08-30T23:53:58.000476+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:53:58.000476+0000","last_undegraded":"2024-08-30T23:53:58.000476+0000","last_fullsized":"2024-08-30T23:53:58.000476+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.104925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.104925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:09.944567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"849'1018","reported_seq":1046,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:05.558020+0000","last_change":"2024-08-30T23:53:49.671937+0000","last_active":"2024-08-30T23:54:05.558020+0000","last_peered":"2024-08-30T23:54:05.558020+0000","last_clean":"2024-08-30T23:54:05.558020+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:54:05.558020+0000","last_undegraded":"2024-08-30T23:54:05.558020+0000","last_fullsized":"2024-08-30T23:54:05.558020+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":1018,"log_dups_size":0,"ondisk_log_size":1018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:14.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66715648,"num_objects":1018,"num_object_clones":0,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1018,"num_write_kb":65152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1793,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.002500+0000","last_change":"2024-08-30T23:53:50.999427+0000","last_active":"2024-08-30T23:53:58.002500+0000","last_peered":"2024-08-30T23:53:58.002500+0000","last_clean":"2024-08-30T23:53:58.002500+0000","last_became_active":"2024-08-30T23:52:54.016899+0000","last_became_peered":"2024-08-30T23:52:54.016899+0000","last_unstale":"2024-08-30T23:53:58.002500+0000","last_undegraded":"2024-08-30T23:53:58.002500+0000","last_fullsized":"2024-08-30T23:53:58.002500+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:50.999362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:50.999362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.278203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"849'964","reported_seq":992,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:05.607925+0000","last_change":"2024-08-30T23:53:49.709301+0000","last_active":"2024-08-30T23:54:05.607925+0000","last_peered":"2024-08-30T23:54:05.607925+0000","last_clean":"2024-08-30T23:54:05.607925+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:54:05.607925+0000","last_undegraded":"2024-08-30T23:54:05.607925+0000","last_fullsized":"2024-08-30T23:54:05.607925+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":964,"log_dups_size":0,"ondisk_log_size":964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.536121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176704,"num_objects":964,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":964,"num_write_kb":61696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":115,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-08-30T23:54:02.023309+0000","last_change":"2024-08-30T23:52:08.307369+0000","last_active":"2024-08-30T23:54:02.023309+0000","last_peered":"2024-08-30T23:54:02.023309+0000","last_clean":"2024-08-30T23:54:02.023309+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:54:02.023309+0000","last_undegraded":"2024-08-30T23:54:02.023309+0000","last_fullsized":"2024-08-30T23:54:02.023309+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:08.307325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:08.307325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:05.541063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"847'686","reported_seq":708,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.922848+0000","last_change":"2024-08-30T23:53:49.670752+0000","last_active":"2024-08-30T23:54:01.922848+0000","last_peered":"2024-08-30T23:54:01.922848+0000","last_clean":"2024-08-30T23:54:01.922848+0000","last_became_active":"2024-08-30T23:53:49.670369+0000","last_became_peered":"2024-08-30T23:53:49.670369+0000","last_unstale":"2024-08-30T23:54:01.922848+0000","last_undegraded":"2024-08-30T23:54:01.922848+0000","last_fullsized":"2024-08-30T23:54:01.922848+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.808262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44957696,"num_objects":686,"num_object_clones":0,"num_object_copies":2058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":772,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.066875+0000","last_change":"2024-08-30T23:53:49.354762+0000","last_active":"2024-08-30T23:54:03.066875+0000","last_peered":"2024-08-30T23:54:03.066875+0000","last_clean":"2024-08-30T23:54:03.066875+0000","last_became_active":"2024-08-30T23:50:25.926833+0000","last_became_peered":"2024-08-30T23:50:25.926833+0000","last_unstale":"2024-08-30T23:54:03.066875+0000","last_undegraded":"2024-08-30T23:54:03.066875+0000","last_fullsized":"2024-08-30T23:54:03.066875+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.354704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.354704+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.152954+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"849'894","reported_seq":922,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:04.706603+0000","last_change":"2024-08-30T23:53:49.707268+0000","last_active":"2024-08-30T23:54:04.706603+0000","last_peered":"2024-08-30T23:54:04.706603+0000","last_clean":"2024-08-30T23:54:04.706603+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:54:04.706603+0000","last_undegraded":"2024-08-30T23:54:04.706603+0000","last_fullsized":"2024-08-30T23:54:04.706603+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.387982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58589184,"num_objects":894,"num_object_clones":0,"num_object_copies":2682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":894,"num_write_kb":57216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":758,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.002627+0000","last_change":"2024-08-30T23:53:41.049687+0000","last_active":"2024-08-30T23:53:58.002627+0000","last_peered":"2024-08-30T23:53:58.002627+0000","last_clean":"2024-08-30T23:53:58.002627+0000","last_became_active":"2024-08-30T23:52:54.016432+0000","last_became_peered":"2024-08-30T23:52:54.016432+0000","last_unstale":"2024-08-30T23:53:58.002627+0000","last_undegraded":"2024-08-30T23:53:58.002627+0000","last_fullsized":"2024-08-30T23:53:58.002627+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.049626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.049626+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.750692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"847'666","reported_seq":689,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.904605+0000","last_change":"2024-08-30T23:53:49.706802+0000","last_active":"2024-08-30T23:54:01.904605+0000","last_peered":"2024-08-30T23:54:01.904605+0000","last_clean":"2024-08-30T23:54:01.904605+0000","last_became_active":"2024-08-30T23:53:49.706231+0000","last_became_peered":"2024-08-30T23:53:49.706231+0000","last_unstale":"2024-08-30T23:54:01.904605+0000","last_undegraded":"2024-08-30T23:54:01.904605+0000","last_fullsized":"2024-08-30T23:54:01.904605+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":666,"log_dups_size":0,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:13.051742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43646976,"num_objects":666,"num_object_clones":0,"num_object_copies":1998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":42624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":689,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:53:58.002701+0000","last_change":"2024-08-30T23:53:35.098110+0000","last_active":"2024-08-30T23:53:58.002701+0000","last_peered":"2024-08-30T23:53:58.002701+0000","last_clean":"2024-08-30T23:53:58.002701+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:53:58.002701+0000","last_undegraded":"2024-08-30T23:53:58.002701+0000","last_fullsized":"2024-08-30T23:53:58.002701+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:35.098031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:35.098031+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.053662+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"847'686","reported_seq":709,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.920885+0000","last_change":"2024-08-30T23:53:49.707500+0000","last_active":"2024-08-30T23:54:01.920885+0000","last_peered":"2024-08-30T23:54:01.920885+0000","last_clean":"2024-08-30T23:54:01.920885+0000","last_became_active":"2024-08-30T23:53:49.707122+0000","last_became_peered":"2024-08-30T23:53:49.707122+0000","last_unstale":"2024-08-30T23:54:01.920885+0000","last_undegraded":"2024-08-30T23:54:01.920885+0000","last_fullsized":"2024-08-30T23:54:01.920885+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:57.343508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44957696,"num_objects":686,"num_object_clones":0,"num_object_copies":2058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":954,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-08-30T23:54:02.023276+0000","last_change":"2024-08-30T23:52:07.301252+0000","last_active":"2024-08-30T23:54:02.023276+0000","last_peered":"2024-08-30T23:54:02.023276+0000","last_clean":"2024-08-30T23:54:02.023276+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:54:02.023276+0000","last_undegraded":"2024-08-30T23:54:02.023276+0000","last_fullsized":"2024-08-30T23:54:02.023276+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:52:07.301208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:52:07.301208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:06.579801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"847'582","reported_seq":607,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.739209+0000","last_change":"2024-08-30T23:53:49.706361+0000","last_active":"2024-08-30T23:54:01.739209+0000","last_peered":"2024-08-30T23:54:01.739209+0000","last_clean":"2024-08-30T23:54:01.739209+0000","last_became_active":"2024-08-30T23:53:49.706070+0000","last_became_peered":"2024-08-30T23:53:49.706070+0000","last_unstale":"2024-08-30T23:54:01.739209+0000","last_undegraded":"2024-08-30T23:54:01.739209+0000","last_fullsized":"2024-08-30T23:54:01.739209+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:02.364402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141952,"num_objects":582,"num_object_clones":0,"num_object_copies":1746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":582,"num_write_kb":37248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":155,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.060912+0000","last_change":"2024-08-30T23:53:43.164672+0000","last_active":"2024-08-30T23:54:03.060912+0000","last_peered":"2024-08-30T23:54:03.060912+0000","last_clean":"2024-08-30T23:54:03.060912+0000","last_became_active":"2024-08-30T23:51:42.952967+0000","last_became_peered":"2024-08-30T23:51:42.952967+0000","last_unstale":"2024-08-30T23:54:03.060912+0000","last_undegraded":"2024-08-30T23:54:03.060912+0000","last_fullsized":"2024-08-30T23:54:03.060912+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.164609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.164609+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.951863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"849'871","reported_seq":899,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.830341+0000","last_change":"2024-08-30T23:53:49.706815+0000","last_active":"2024-08-30T23:54:03.830341+0000","last_peered":"2024-08-30T23:54:03.830341+0000","last_clean":"2024-08-30T23:54:03.830341+0000","last_became_active":"2024-08-30T23:53:49.706543+0000","last_became_peered":"2024-08-30T23:53:49.706543+0000","last_unstale":"2024-08-30T23:54:03.830341+0000","last_undegraded":"2024-08-30T23:54:03.830341+0000","last_fullsized":"2024-08-30T23:54:03.830341+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":871,"log_dups_size":0,"ondisk_log_size":871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.291074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57081856,"num_objects":871,"num_object_clones":0,"num_object_copies":2613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":871,"num_write_kb":55744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1773,"reported_epoch":848,"state":"active+clean","last_fresh":"2024-08-30T23:54:02.011925+0000","last_change":"2024-08-30T23:53:42.303311+0000","last_active":"2024-08-30T23:54:02.011925+0000","last_peered":"2024-08-30T23:54:02.011925+0000","last_clean":"2024-08-30T23:54:02.011925+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:54:02.011925+0000","last_undegraded":"2024-08-30T23:54:02.011925+0000","last_fullsized":"2024-08-30T23:54:02.011925+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.303263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.303263+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:47.897994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"849'910","reported_seq":939,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:04.723675+0000","last_change":"2024-08-30T23:53:49.879494+0000","last_active":"2024-08-30T23:54:04.723675+0000","last_peered":"2024-08-30T23:54:04.723675+0000","last_clean":"2024-08-30T23:54:04.723675+0000","last_became_active":"2024-08-30T23:53:49.879153+0000","last_became_peered":"2024-08-30T23:53:49.879153+0000","last_unstale":"2024-08-30T23:54:04.723675+0000","last_undegraded":"2024-08-30T23:54:04.723675+0000","last_fullsized":"2024-08-30T23:54:04.723675+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":910,"log_dups_size":0,"ondisk_log_size":910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:11.036607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":58240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":889,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.062947+0000","last_change":"2024-08-30T23:53:49.897094+0000","last_active":"2024-08-30T23:54:03.062947+0000","last_peered":"2024-08-30T23:54:03.062947+0000","last_clean":"2024-08-30T23:54:03.062947+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:54:03.062947+0000","last_undegraded":"2024-08-30T23:54:03.062947+0000","last_fullsized":"2024-08-30T23:54:03.062947+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.897038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.897038+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.963756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"847'643","reported_seq":667,"reported_epoch":847,"state":"active+clean","last_fresh":"2024-08-30T23:54:01.884341+0000","last_change":"2024-08-30T23:53:49.706603+0000","last_active":"2024-08-30T23:54:01.884341+0000","last_peered":"2024-08-30T23:54:01.884341+0000","last_clean":"2024-08-30T23:54:01.884341+0000","last_became_active":"2024-08-30T23:53:49.706194+0000","last_became_peered":"2024-08-30T23:53:49.706194+0000","last_unstale":"2024-08-30T23:54:01.884341+0000","last_undegraded":"2024-08-30T23:54:01.884341+0000","last_fullsized":"2024-08-30T23:54:01.884341+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:04.835283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42139648,"num_objects":643,"num_object_clones":0,"num_object_copies":1929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":643,"num_write_kb":41152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":143,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.061161+0000","last_change":"2024-08-30T23:53:41.078765+0000","last_active":"2024-08-30T23:54:03.061161+0000","last_peered":"2024-08-30T23:54:03.061161+0000","last_clean":"2024-08-30T23:54:03.061161+0000","last_became_active":"2024-08-30T23:51:42.950294+0000","last_became_peered":"2024-08-30T23:51:42.950294+0000","last_unstale":"2024-08-30T23:54:03.061161+0000","last_undegraded":"2024-08-30T23:54:03.061161+0000","last_fullsized":"2024-08-30T23:54:03.061161+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.078720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.078720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:44.759291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1317,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-08-30T23:54:03.063151+0000","last_change":"2024-08-30T23:53:54.855614+0000","last_active":"2024-08-30T23:54:03.063151+0000","last_peered":"2024-08-30T23:54:03.063151+0000","last_clean":"2024-08-30T23:54:03.063151+0000","last_became_active":"2024-08-30T23:46:12.976986+0000","last_became_peered":"2024-08-30T23:46:12.976986+0000","last_unstale":"2024-08-30T23:54:03.063151+0000","last_undegraded":"2024-08-30T23:54:03.063151+0000","last_fullsized":"2024-08-30T23:54:03.063151+0000","mapping_epoch":657,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:54.855553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:54.855553+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:02.123392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":823066624,"num_objects":12559,"num_object_clones":0,"num_object_copies":37677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12559,"num_write_kb":803776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2463236096,"data_stored":2463236096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12559,"ondisk_log_size":12559,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":369367187810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449451,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898616,"kb_used_data":398184,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92846536,"statfs":{"total":95995035648,"available":95074852864,"internally_reserved":0,"allocated":407740416,"data_stored":407205292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":512424384},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711081,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969240,"kb_used_data":316096,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92775912,"statfs":{"total":95995035648,"available":95002533888,"internally_reserved":0,"allocated":323682304,"data_stored":323138266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":668793280},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972723,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648336,"kb_used_data":179840,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93096816,"statfs":{"total":95995035648,"available":95331139584,"internally_reserved":0,"allocated":184156160,"data_stored":183612122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234359,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790728,"kb_used_data":91764,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92954424,"statfs":{"total":95995035648,"available":95185330176,"internally_reserved":0,"allocated":93966336,"data_stored":93426780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528701,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521784,"kb_used_data":43560,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93223368,"statfs":{"total":95995035648,"available":95460728832,"internally_reserved":0,"allocated":44605440,"data_stored":44070316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":489683398},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775204,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":518112,"kb_used_data":320072,"kb_used_omap":5,"kb_used_meta":198010,"kb_avail":93227040,"statfs":{"total":95995035648,"available":95464488960,"internally_reserved":0,"allocated":327753728,"data_stored":327206138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":202762314},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914168,"kb_used_data":273512,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92830984,"statfs":{"total":95995035648,"available":95058927616,"internally_reserved":0,"allocated":280076288,"data_stored":279541164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150090,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1089764,"kb_used_data":292236,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92655388,"statfs":{"total":95995035648,"available":94879117312,"internally_reserved":0,"allocated":299249664,"data_stored":298698002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":816642503},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411728,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962652,"kb_used_data":376136,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92782500,"statfs":{"total":95995035648,"available":95009280000,"internally_reserved":0,"allocated":385163264,"data_stored":384615674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830544,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614648,"kb_used_data":211880,"kb_used_omap":87,"kb_used_meta":402664,"kb_avail":93130504,"statfs":{"total":95995035648,"available":95365636096,"internally_reserved":0,"allocated":216965120,"data_stored":216429996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":412328095},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738778,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899076,"kb_used_data":137652,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92846076,"statfs":{"total":95995035648,"available":95074381824,"internally_reserved":0,"allocated":140955648,"data_stored":140416092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158203904,"data_stored":158203904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249757696,"data_stored":249757696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397082624,"data_stored":397082624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":160563200,"data_stored":160563200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":206766080,"data_stored":206766080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353304576,"data_stored":353304576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46071808,"data_stored":46071808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":97517568,"data_stored":97517568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":195559424,"data_stored":195559424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":341835776,"data_stored":341835776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":256573440,"data_stored":256573440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:54:07.105 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-30T23:54:07.106 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:54:07.106 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-08-30T23:54:07.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd in 11 2024-08-30T23:54:07.136 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:07 smithi106 ceph-mon[16603]: pgmap v1896: 37 pgs: 37 active+clean; 786 MiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-30T23:54:07.136 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:07.136 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:07.136 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:07 smithi106 ceph-mon[16603]: osdmap e850: 12 total, 12 up, 11 in 2024-08-30T23:54:07.136 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:07 smithi106 ceph-mon[16603]: from='client.18537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:07.193 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:07.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:07.227 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:07.232 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:07 smithi136 ceph-mon[20207]: pgmap v1896: 37 pgs: 37 active+clean; 786 MiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-30T23:54:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:07 smithi136 ceph-mon[20207]: osdmap e850: 12 total, 12 up, 11 in 2024-08-30T23:54:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:07 smithi136 ceph-mon[20207]: from='client.18537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:07.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:07 smithi173 ceph-mon[20554]: pgmap v1896: 37 pgs: 37 active+clean; 786 MiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-30T23:54:07.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:07 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:07 smithi173 ceph-mon[20554]: osdmap e850: 12 total, 12 up, 11 in 2024-08-30T23:54:07.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:07 smithi173 ceph-mon[20554]: from='client.18537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:07.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:07.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:07.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 17883 17867 93.0488 70.6875 0.00237405 0.0107395 2024-08-30T23:54:07.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:07.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:07.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:07.606 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:07.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.731 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:07.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:07.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:07.864 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:07.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:07.995 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:07.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:07.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:08.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:08.123 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:08.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:08.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:08.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:08.376 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:08.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 19254 19238 92.482 85.6875 0.00795297 0.0108041 2024-08-30T23:54:08.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:08.503 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:08.605 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:08.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.634 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:08.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:08.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:08.786 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:08.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:08.889 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:08.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:08.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:09.016 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:09.077 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:54:09.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:09.200 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:09.200 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:09.200 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: pgmap v1898: 37 pgs: 37 active+clean; 1003 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-30T23:54:09.201 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: osdmap e851: 12 total, 12 up, 11 in 2024-08-30T23:54:09.201 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/41387719' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:54:09.201 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:09.201 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:09.201 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3532764751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:09.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:54:09.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:09.277 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:09.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: pgmap v1898: 37 pgs: 37 active+clean; 1003 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: osdmap e851: 12 total, 12 up, 11 in 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/41387719' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:09.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3532764751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:09.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:09.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:09.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 15 20772 20757 92.6568 94.9375 0.0764339 0.010777 2024-08-30T23:54:09.490 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:09.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:09.509 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:09.509 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:09.530 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:09.533 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:09.534 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: pgmap v1898: 37 pgs: 37 active+clean; 1003 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: osdmap e851: 12 total, 12 up, 11 in 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/41387719' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3532764751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:09.558 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:09.559 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:09.580 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:09.580 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:09.604 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:09.605 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:09.629 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:09.629 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:09.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:09.652 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:09.652 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:09.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:09.678 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:09.679 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:09.704 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:09.704 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:09.730 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:09.730 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:09.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:09.759 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:09.784 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:09.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:09.785 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:09 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:54:09.546+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e852 definitely_dead 0 2024-08-30T23:54:09.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:09.906 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:09.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:10.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:10.039 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:10.046 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:54:10.046 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:54:10.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:54:10.046 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDeleteReply(5.19 e599,595 5:9f3d16fd:::benchmark_data_smithi173_87639_object64699:head,592'7133)", 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 599.69035997599997, 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.420813033, 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:54:10.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354526+0000", 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354527+0000", 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3099999999999997e-07 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.354530+0000", 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8600000000000003e-06 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775055+0000", 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.42052456799999999 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775063+0000", 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8010000000000004e-06 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775123+0000", 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9885e-05 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:44:10.775339+0000", 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000216188 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:93583 6.8 6:14dca3a3:::benchmark_data_smithi173_114000_object93582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 485.54179568299998, 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34108081800000001, 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93583 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.051 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503091+0000", 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503137+0000", 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6109000000000002e-05 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503151+0000", 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3179999999999999e-05 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503156+0000", 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5639999999999996e-06 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503212+0000", 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5946999999999998e-05 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503322+0000", 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011007500000000001 2024-08-30T23:54:10.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,11", 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503414+0000", 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2143000000000002e-05 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.505854+0000", 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002439887 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.506967+0000", 2024-08-30T23:54:10.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001112341 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844102+0000", 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.33713554299999998 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844139+0000", 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6683000000000001e-05 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844171+0000", 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2095999999999999e-05 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:155745 6.5 6:a23d3df7:::benchmark_data_smithi173_114000_object55947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e684)", 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:39.831928+0000", 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 390.21295813199998, 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67798276800000001, 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.057 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155745 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831928+0000", 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831928+0000", 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831929+0000", 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7829999999999999e-06 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831935+0000", 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2630000000000003e-06 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831949+0000", 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4025999999999999e-05 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831956+0000", 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1829999999999997e-06 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.832017+0000", 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1079000000000003e-05 2024-08-30T23:54:10.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.832135+0000", 2024-08-30T23:54:10.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118416 2024-08-30T23:54:10.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.832248+0000", 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011268300000000001 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.836115+0000", 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038669939999999999 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.837197+0000", 2024-08-30T23:54:10.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001082132 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.509846+0000", 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67264882100000001 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.509880+0000", 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3590999999999997e-05 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.509910+0000", 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0796999999999997e-05 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:199592 6.5 6:a0027b35:::benchmark_data_smithi173_114000_object99794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.158645+0000", 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 323.88624049600003, 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.39176461600000001, 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.064 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 199592 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158645+0000", 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158645+0000", 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158647+0000", 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1160000000000002e-06 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158655+0000", 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6979999999999998e-06 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158673+0000", 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7689e-05 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158680+0000", 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6799999999999993e-06 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158732+0000", 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1285000000000001e-05 2024-08-30T23:54:10.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158820+0000", 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8617999999999998e-05 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158897+0000", 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6822000000000004e-05 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.162811+0000", 2024-08-30T23:54:10.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00391414 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.163203+0000", 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039150099999999999 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.550332+0000", 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38712935599999998 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.550375+0000", 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2718e-05 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.550410+0000", 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4993000000000002e-05 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:34198 7.8 e736/725 7:1ca020bc:::benchmark_data_smithi173_129296_object34197:head v 736'2123, mlcod=736'2119)", 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:27.842045+0000", 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 282.202840837, 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26091951200000002, 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842045+0000", 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842045+0000", 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842046+0000", 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842064+0000", 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8097e-05 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101631+0000", 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25956776599999998 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101639+0000", 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.464e-06 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101695+0000", 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6348000000000003e-05 2024-08-30T23:54:10.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101712+0000", 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6501999999999999e-05 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.102931+0000", 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012193320000000001 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.102964+0000", 2024-08-30T23:54:10.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3353999999999999e-05 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:87841 7.6 e771/762 7:6125c906:::benchmark_data_smithi173_129296_object725:head v 771'5373, mlcod=734'1200)", 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:50.635013+0000", 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 199.40987250000001, 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26843487199999999, 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635013+0000", 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635013+0000", 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635014+0000", 2024-08-30T23:54:10.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2699999999999999e-07 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635020+0000", 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900819+0000", 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26579951600000001 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900828+0000", 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000003e-06 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900899+0000", 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1598000000000001e-05 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900921+0000", 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2001000000000002e-05 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.903412+0000", 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024904189999999998 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.903448+0000", 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6421000000000002e-05 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.079 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:93823 7.8 e774/725 7:1693af46:::benchmark_data_smithi173_129296_object6707:head v 774'5852, mlcod=774'5848)", 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:00.487850+0000", 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 189.55703624500001, 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21028736100000001, 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487850+0000", 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487850+0000", 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487851+0000", 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0529999999999999e-06 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487859+0000", 2024-08-30T23:54:10.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4239999999999993e-06 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695495+0000", 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20763561 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695502+0000", 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2590000000000003e-06 2024-08-30T23:54:10.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695559+0000", 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7188000000000002e-05 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695583+0000", 2024-08-30T23:54:10.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3428e-05 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.698085+0000", 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025024489999999999 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.698137+0000", 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1950000000000002e-05 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:122999 7.8 e792/725 7:1caaedd7:::benchmark_data_smithi173_129296_object35883:head v 792'7654, mlcod=792'7651)", 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:54.161832+0000", 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.88305332799999, 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10797648, 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.161832+0000", 2024-08-30T23:54:10.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.161832+0000", 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.161834+0000", 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6929999999999999e-06 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.161844+0000", 2024-08-30T23:54:10.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6220000000000004e-06 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.265955+0000", 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10411155800000001 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.265963+0000", 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6939999999999995e-06 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.266044+0000", 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1148000000000003e-05 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.266070+0000", 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6081999999999999e-05 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.269759+0000", 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036883329999999998 2024-08-30T23:54:10.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.269809+0000", 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0349999999999997e-05 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:144892 7.6 e811/810 7:645a80f3:::benchmark_data_smithi173_129296_object57776:head v 811'8848, mlcod=811'8845)", 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:42.156476+0000", 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 87.888409327000005, 2024-08-30T23:54:10.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14011880700000001, 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156476+0000", 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156476+0000", 2024-08-30T23:54:10.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156477+0000", 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156482+0000", 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1279999999999999e-06 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293035+0000", 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.136552056 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293042+0000", 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9419999999999995e-06 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293104+0000", 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1273999999999996e-05 2024-08-30T23:54:10.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293129+0000", 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5641000000000001e-05 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.296564+0000", 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034348130000000001 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.296595+0000", 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1102000000000001e-05 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:147040 7.8 e813/725 7:155da638:::benchmark_data_smithi173_129296_object59924:head v 813'9140, mlcod=813'9136)", 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:46.130984+0000", 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 83.913901623000001, 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.121825357, 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.094 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130984+0000", 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130984+0000", 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130985+0000", 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6499999999999998e-07 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130997+0000", 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1625e-05 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248812+0000", 2024-08-30T23:54:10.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.117814857 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248817+0000", 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1259999999999997e-06 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248869+0000", 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2024000000000002e-05 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248893+0000", 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3992999999999999e-05 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.252767+0000", 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038744970000000002 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.252810+0000", 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2370000000000003e-05 2024-08-30T23:54:10.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:149982 7.8 e815/725 7:1c54880d:::benchmark_data_smithi173_129296_object62866:head v 815'9327, mlcod=815'9323)", 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:51.302587+0000", 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 78.742299109000001, 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20097135399999999, 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302587+0000", 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302587+0000", 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302588+0000", 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6300000000000004e-07 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302594+0000", 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5309999999999998e-06 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499896+0000", 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19730151900000001 2024-08-30T23:54:10.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499903+0000", 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3000000000000004e-06 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499959+0000", 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5631000000000003e-05 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499979+0000", 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0647000000000001e-05 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.503520+0000", 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003540891 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.503558+0000", 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7972000000000002e-05 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:153765 7.6 e819/810 7:66a9e493:::benchmark_data_smithi173_129296_object66649:head v 819'9422, mlcod=819'9419)", 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:58.539690+0000", 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.505196049999995, 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20419347600000001, 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539690+0000", 2024-08-30T23:54:10.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539690+0000", 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539691+0000", 2024-08-30T23:54:10.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2999999999999999e-07 2024-08-30T23:54:10.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539697+0000", 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3040000000000001e-06 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739495+0000", 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19979808499999999 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739501+0000", 2024-08-30T23:54:10.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.212e-06 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739559+0000", 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7454999999999998e-05 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739582+0000", 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3349e-05 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.743844+0000", 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042621229999999996 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.743883+0000", 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9118000000000001e-05 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:160742 7.8 e823/725 7:1aa16899:::benchmark_data_smithi173_129296_object73626:head v 823'10015, mlcod=822'10011)", 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:10.973818+0000", 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.071067339999999, 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12619274899999999, 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973818+0000", 2024-08-30T23:54:10.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973818+0000", 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973820+0000", 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4500000000000001e-06 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973828+0000", 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.371e-06 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.094925+0000", 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12109639 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.094931+0000", 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2469999999999997e-06 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.095009+0000", 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7690999999999999e-05 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.095031+0000", 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2415000000000001e-05 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.099954+0000", 2024-08-30T23:54:10.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049231769999999999 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.100011+0000", 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7008e-05 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:164977 7.8 e825/725 7:17fdc8b0:::benchmark_data_smithi173_129296_object77861:head v 825'10271, mlcod=825'10268)", 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:18.643415+0000", 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.401470502999999, 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.125165576, 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643415+0000", 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643415+0000", 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643417+0000", 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.548e-06 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643426+0000", 2024-08-30T23:54:10.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3759999999999997e-06 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763421+0000", 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.119994487 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763426+0000", 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2950000000000001e-06 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763755+0000", 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032881400000000003 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763777+0000", 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2407000000000001e-05 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.768549+0000", 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047720870000000004 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.768581+0000", 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1562e-05 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:165040 7.8 e825/725 7:1003126f:::benchmark_data_smithi173_129296_object77924:head v 825'10276, mlcod=825'10273)", 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:18.845533+0000", 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.199353049999999, 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.110421014, 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.845533+0000", 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.845533+0000", 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.845534+0000", 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3039999999999999e-06 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.845544+0000", 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-06 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.952066+0000", 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.10652173099999999 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.952073+0000", 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7479999999999996e-06 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.952124+0000", 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1288000000000001e-05 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.952150+0000", 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6151000000000001e-05 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.955921+0000", 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003771052 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.955954+0000", 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.277e-05 2024-08-30T23:54:10.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:165746 7.8 e825/725 7:11156502:::benchmark_data_smithi173_129296_object78630:head v 825'10317, mlcod=825'10314)", 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:20.061291+0000", 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 49.983594734, 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17562274, 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061291+0000", 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061291+0000", 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061292+0000", 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2860000000000001e-06 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061301+0000", 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7730000000000005e-06 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232871+0000", 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.171569736 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232877+0000", 2024-08-30T23:54:10.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2620000000000004e-06 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232939+0000", 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1764e-05 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232959+0000", 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0119000000000002e-05 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.236876+0000", 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039168390000000001 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.236914+0000", 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7960999999999998e-05 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:5745 8.6 e847/840 8:699b143f:::benchmark_data_smithi173_143816_object5744:head v 847'375, mlcod=847'371)", 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:58.798499+0000", 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 11.24638652, 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.124204167, 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798499+0000", 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798499+0000", 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798500+0000", 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0200000000000003e-07 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798566+0000", 2024-08-30T23:54:10.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6397999999999997e-05 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922116+0000", 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.123550164 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922125+0000", 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2740000000000007e-06 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922178+0000", 2024-08-30T23:54:10.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3798999999999998e-05 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922203+0000", 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4834999999999999e-05 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922671+0000", 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00046786899999999999 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922703+0000", 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2425999999999998e-05 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21049 8.6 e852/840 8:6d913e32:::benchmark_data_smithi173_143816_object21048:head v 852'1372, mlcod=852'1369)", 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:09.673321+0000", 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.37156473299999998, 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.02844435, 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.133 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.673321+0000", 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.673321+0000", 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.673322+0000", 2024-08-30T23:54:10.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.673366+0000", 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3961999999999997e-05 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.700893+0000", 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027527758999999999 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.700899+0000", 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0760000000000002e-06 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.700946+0000", 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7803000000000003e-05 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.700959+0000", 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3145e-05 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.701722+0000", 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00076251699999999999 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.701765+0000", 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3424999999999998e-05 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.137 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21092 8.6 e852/840 8:6ec24d3c:::benchmark_data_smithi173_143816_object21091:head v 852'1374, mlcod=852'1371)", 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:09.703613+0000", 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.34127244899999998, 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035460931000000001, 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.703613+0000", 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.703613+0000", 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.703614+0000", 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6300000000000005e-07 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.703665+0000", 2024-08-30T23:54:10.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1415000000000002e-05 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.738170+0000", 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034504140000000003 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.738177+0000", 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7680000000000008e-06 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.738222+0000", 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4558000000000003e-05 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.738238+0000", 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.59e-05 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.739043+0000", 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080533799999999995 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.739074+0000", 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1149000000000003e-05 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21309 8.6 e852/840 8:62797eb2:::benchmark_data_smithi173_143816_object21308:head v 852'1386, mlcod=852'1383)", 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:09.848741+0000", 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.19614441899999999, 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029997961, 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:10.142 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.848741+0000", 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.848741+0000", 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.848742+0000", 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3200000000000004e-07 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.848909+0000", 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016638699999999999 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.877760+0000", 2024-08-30T23:54:10.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028851547000000002 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.877765+0000", 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0470000000000001e-06 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.877815+0000", 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9728999999999998e-05 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:10.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.877837+0000", 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2436999999999999e-05 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.878698+0000", 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086030199999999996 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:09.878739+0000", 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1680000000000001e-05 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:10.147 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:10.147 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:54:10.147 INFO:teuthology.orchestra.run.smithi106.stderr:marked in osd.11. 2024-08-30T23:54:10.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:10.169 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:10.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:10 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/41387719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:54:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:10 smithi136 ceph-mon[20207]: osdmap e852: 12 total, 12 up, 11 in 2024-08-30T23:54:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3119124926' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-08-30T23:54:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:10 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:54:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:10 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:54:10.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:10 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/41387719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:54:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:10 smithi173 ceph-mon[20554]: osdmap e852: 12 total, 12 up, 11 in 2024-08-30T23:54:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3119124926' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-08-30T23:54:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:10 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:54:10.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:10 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:54:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:10 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/41387719' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:54:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:10 smithi106 ceph-mon[16603]: osdmap e852: 12 total, 12 up, 11 in 2024-08-30T23:54:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3119124926' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-08-30T23:54:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:10 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:54:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:10 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:54:10.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:10.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 21791 21775 90.7208 63.625 0.153125 0.0109061 2024-08-30T23:54:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:10.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:10.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:10.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:10.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:10.663 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:10.756 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-08-30T23:54:10.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:10.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.787 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:10.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:54:10.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:10.910 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:10.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:11.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:11.034 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:11.109 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:11.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:11.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:11.229 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:11 smithi173 ceph-mon[20554]: pgmap v1901: 37 pgs: 37 active+clean; 1.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-08-30T23:54:11.229 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3119124926' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-30T23:54:11.230 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:11 smithi173 ceph-mon[20554]: osdmap e853: 12 total, 12 up, 12 in 2024-08-30T23:54:11.230 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:11 smithi173 ceph-mon[20554]: osdmap e854: 12 total, 12 up, 12 in 2024-08-30T23:54:11.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:11.281 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:11 smithi106 ceph-mon[16603]: pgmap v1901: 37 pgs: 37 active+clean; 1.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-08-30T23:54:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3119124926' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-30T23:54:11.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:11 smithi106 ceph-mon[16603]: osdmap e853: 12 total, 12 up, 12 in 2024-08-30T23:54:11.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:11 smithi106 ceph-mon[16603]: osdmap e854: 12 total, 12 up, 12 in 2024-08-30T23:54:11.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:11.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 21815 21799 85.1446 1.5 0.00175384 0.0110255 2024-08-30T23:54:11.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:11.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:11.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:11 smithi136 ceph-mon[20207]: pgmap v1901: 37 pgs: 37 active+clean; 1.1 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-08-30T23:54:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3119124926' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-30T23:54:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:11 smithi136 ceph-mon[20207]: osdmap e853: 12 total, 12 up, 12 in 2024-08-30T23:54:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:11 smithi136 ceph-mon[20207]: osdmap e854: 12 total, 12 up, 12 in 2024-08-30T23:54:11.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:11.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:11.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:11.775 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:11.775 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:54:11.775 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:54:11.775 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:11.779 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:11.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:11.907 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:11.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:12.009 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:12.031 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:12.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:12.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:54:12.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:12.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:12.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:12 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:54:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:12 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:54:12.357 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:12 smithi173 ceph-mon[20554]: osdmap e855: 12 total, 12 up, 12 in 2024-08-30T23:54:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:12 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:54:12.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:12 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:54:12.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:12 smithi106 ceph-mon[16603]: osdmap e855: 12 total, 12 up, 12 in 2024-08-30T23:54:12.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:12.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 22422 22406 82.3675 37.9375 0.00417932 0.0121316 2024-08-30T23:54:12.419 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:12.510 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:12.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:12.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:12 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:54:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:12 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:54:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:12 smithi136 ceph-mon[20207]: osdmap e855: 12 total, 12 up, 12 in 2024-08-30T23:54:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:12.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:12.679 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:12.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:12.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:12.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:54:12.904 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:12.945 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:12.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:13.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:13.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:13.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:13.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:13.211 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:13.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:13.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:13 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:54:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:13 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:54:13.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:13 smithi106 ceph-mon[16603]: pgmap v1904: 37 pgs: 3 remapped+peering, 34 active+clean; 1.2 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:54:13.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:13 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:54:13.376 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:13 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:54:13.377 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:13 smithi173 ceph-mon[20554]: pgmap v1904: 37 pgs: 3 remapped+peering, 34 active+clean; 1.2 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:54:13.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 23554 23538 81.7218 70.75 0.0145151 0.0122284 2024-08-30T23:54:13.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:13.467 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:13.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:54:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:13 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:54:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:13 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:54:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:13 smithi136 ceph-mon[20207]: pgmap v1904: 37 pgs: 3 remapped+peering, 34 active+clean; 1.2 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-08-30T23:54:13.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:13.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:13.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:13.721 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:13.823 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:13.829 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:13.830 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:54:13.830 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:54:13.830 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:54:13.830 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:13.859 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:13.886 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:13.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:13.986 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:13.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:14.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:14.098 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:54:14.114 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:14.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:14.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.245 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:14.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:54:14.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:54:14.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:54:14.275 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:54:14.276 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2619891855' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:14.345 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:54:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:54:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:54:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:54:14.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2619891855' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:14.371 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:14.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 24998 24982 82.1702 90.25 0.0137037 0.0121589 2024-08-30T23:54:14.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:54:14.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:54:14.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:54:14.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:54:14.435 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2619891855' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:14.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:14.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:14.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:14.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:14.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:14.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:14.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:54:14.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:14.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:14.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:15.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:15.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:15.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:15.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:15.193 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:15.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:15 smithi136 ceph-mon[20207]: pgmap v1906: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 33 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 1/64896 objects degraded (0.002%); 1344/64896 objects misplaced (2.071%); 950 KiB/s, 14 objects/s recovering 2024-08-30T23:54:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:15 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 1/64896 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:54:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:15 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:54:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:15 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:54:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2619891855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:15.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:15 smithi136 ceph-mon[20207]: osdmap e856: 12 total, 12 up, 12 in 2024-08-30T23:54:15.295 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:15 smithi106 ceph-mon[16603]: pgmap v1906: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 33 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 1/64896 objects degraded (0.002%); 1344/64896 objects misplaced (2.071%); 950 KiB/s, 14 objects/s recovering 2024-08-30T23:54:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:15 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 1/64896 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:54:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:15 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:54:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:15 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:54:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2619891855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:15 smithi106 ceph-mon[16603]: osdmap e856: 12 total, 12 up, 12 in 2024-08-30T23:54:15.362 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:15 smithi173 ceph-mon[20554]: pgmap v1906: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 33 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 1/64896 objects degraded (0.002%); 1344/64896 objects misplaced (2.071%); 950 KiB/s, 14 objects/s recovering 2024-08-30T23:54:15.362 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:15 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 1/64896 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:54:15.362 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:15 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:54:15.362 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:15 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:54:15.362 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2619891855' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:15.362 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:15 smithi173 ceph-mon[20554]: osdmap e856: 12 total, 12 up, 12 in 2024-08-30T23:54:15.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:15.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:54:15.406416+0000 min lat: 0.0010542 max lat: 2.02823 avg lat: 0.0121871 2024-08-30T23:54:15.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:54:15.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 26255 26239 81.9895 78.5625 0.00371065 0.0121871 2024-08-30T23:54:15.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:15.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:15.554 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:15.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:15.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:15.682 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:15.756 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:54:15.757 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:54:15.757 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-08-30T23:54:15.757 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:54:15.757 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:54:15.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-08-30T23:54:15.783 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:15.811 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:15.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:15.890 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93381 6.1 6:8c800d3f:::benchmark_data_smithi173_114000_object93380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906375+0000", 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98110096900001, 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92827897100000001, 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.891 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93381 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906375+0000", 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906375+0000", 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906377+0000", 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.869e-06 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906429+0000", 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2918999999999999e-05 2024-08-30T23:54:15.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906445+0000", 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5288e-05 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906452+0000", 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0450000000000003e-06 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.909954+0000", 2024-08-30T23:54:15.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035024349999999999 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.909975+0000", 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0313e-05 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.834545+0000", 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92457082400000001 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.834654+0000", 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108278 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93434 6.1 6:8548ae37:::benchmark_data_smithi173_114000_object93433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906470+0000", 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98100521499998, 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93097174400000005, 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.896 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93434 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906470+0000", 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906470+0000", 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906471+0000", 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1199999999999998e-07 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906514+0000", 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2752e-05 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906521+0000", 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0720000000000002e-06 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906524+0000", 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6650000000000001e-06 2024-08-30T23:54:15.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910095+0000", 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035705070000000001 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910104+0000", 2024-08-30T23:54:15.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8009999999999993e-06 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837367+0000", 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.927262896 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837442+0000", 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5638999999999999e-05 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93443 6.1 6:879a2a2b:::benchmark_data_smithi173_114000_object93442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98094399199999, 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96616541, 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93443 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.902 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906532+0000", 2024-08-30T23:54:15.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.39e-07 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906568+0000", 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6347999999999998e-05 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906575+0000", 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2110000000000003e-06 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906578+0000", 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3280000000000002e-06 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910143+0000", 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035652679999999999 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910151+0000", 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4250000000000001e-06 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837487+0000", 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.92733659000000002 2024-08-30T23:54:15.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837635+0000", 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014827699999999999 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.837751+0000", 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011542199999999999 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842684+0000", 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049333650000000003 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842702+0000", 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7965000000000002e-05 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.869579+0000", 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.026876494000000001 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.869625+0000", 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.651e-05 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.872697+0000", 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0030718680000000002 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.909 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93451 6.1 6:89dbcc07:::benchmark_data_smithi173_114000_object93450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98089179700003, 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96804708799999994, 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93451 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906584+0000", 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6600000000000003e-07 2024-08-30T23:54:15.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906630+0000", 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5815000000000001e-05 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906636+0000", 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.694e-06 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906639+0000", 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0340000000000001e-06 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910196+0000", 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035574980000000001 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910207+0000", 2024-08-30T23:54:15.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0915e-05 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.840261+0000", 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93005400400000005 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.840396+0000", 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013501799999999999 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.840498+0000", 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010153300000000001 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847456+0000", 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069587240000000003 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847511+0000", 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4848000000000001e-05 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.872751+0000", 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.025240058999999999 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.872792+0000", 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0639000000000002e-05 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.874631+0000", 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001839041 2024-08-30T23:54:15.916 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93453 6.1 6:8e4148d1:::benchmark_data_smithi173_114000_object93452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906644+0000", 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98083134400002, 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97105458200000005, 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.917 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93453 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906644+0000", 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906644+0000", 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906645+0000", 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2500000000000001e-07 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906675+0000", 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0706e-05 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906681+0000", 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4389999999999999e-06 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906684+0000", 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1190000000000001e-06 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910250+0000", 2024-08-30T23:54:15.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035664389999999998 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910258+0000", 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6920000000000002e-06 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842735+0000", 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93247689499999997 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842833+0000", 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8061999999999997e-05 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.842931+0000", 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7505999999999998e-05 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847499+0000", 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045683310000000001 2024-08-30T23:54:15.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847519+0000", 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0072e-05 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.874984+0000", 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027465280000000002 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.875022+0000", 2024-08-30T23:54:15.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7573000000000001e-05 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.877699+0000", 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026771429999999999 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93460 6.1 6:8557b2d5:::benchmark_data_smithi173_114000_object93459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:54:15.924 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98078663299998, 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98645355199999996, 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93460 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:54:15.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906689+0000", 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0699999999999998e-07 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906719+0000", 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9921999999999999e-05 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906724+0000", 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9099999999999996e-06 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906727+0000", 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9639999999999999e-06 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910297+0000", 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035694099999999999 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910307+0000", 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9699999999999994e-06 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.845188+0000", 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93488130300000005 2024-08-30T23:54:15.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.845283+0000", 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5049000000000004e-05 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.845393+0000", 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011003299999999999 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853878+0000", 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084850429999999994 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853972+0000", 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4526999999999998e-05 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.889838+0000", 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035865478999999999 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.889888+0000", 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0467000000000003e-05 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.893143+0000", 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003254168 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93467 6.1 6:81b403e8:::benchmark_data_smithi173_114000_object93466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.931 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98074378299998, 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.98816438699999998, 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93467 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906732+0000", 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.84e-07 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906763+0000", 2024-08-30T23:54:15.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0889000000000002e-05 2024-08-30T23:54:15.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.934 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:15.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906768+0000", 2024-08-30T23:54:15.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.989e-06 2024-08-30T23:54:15.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906771+0000", 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.976e-06 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910341+0000", 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035697580000000001 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910345+0000", 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8039999999999999e-06 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847542+0000", 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93719730700000003 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847633+0000", 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0718999999999997e-05 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.847738+0000", 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010495100000000001 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853946+0000", 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0062085839999999996 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.853982+0000", 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5568999999999998e-05 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.893189+0000", 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039206854999999999 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.893233+0000", 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4274999999999999e-05 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.894896+0000", 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016634270000000001 2024-08-30T23:54:15.939 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93475 6.1 6:85aba04e:::benchmark_data_smithi173_114000_object93474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e654)", 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.98069980499997, 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.99493798300000003, 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.940 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93475 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906776+0000", 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.27e-07 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906816+0000", 2024-08-30T23:54:15.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9847e-05 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906828+0000", 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1671999999999999e-05 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.906833+0000", 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2290000000000002e-06 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910400+0000", 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035675080000000001 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:03.910404+0000", 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7890000000000001e-06 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.850631+0000", 2024-08-30T23:54:15.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94022669699999994 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.850804+0000", 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017325500000000001 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.851007+0000", 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000203265 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856446+0000", 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054381899999999999 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856460+0000", 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4352000000000001e-05 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.900302+0000", 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.043842224999999999 2024-08-30T23:54:15.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.900339+0000", 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6650999999999999e-05 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.901714+0000", 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001374976 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93483 6.1 6:89482277:::benchmark_data_smithi173_114000_object93482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.042217+0000", 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.84525840800001, 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.861912593, 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93483 2024-08-30T23:54:15.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042217+0000", 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042217+0000", 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042219+0000", 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.855e-06 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042243+0000", 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3516e-05 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042260+0000", 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7037999999999998e-05 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042269+0000", 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6609999999999995e-06 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042338+0000", 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8492999999999996e-05 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.042346+0000", 2024-08-30T23:54:15.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3440000000000001e-06 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.854003+0000", 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.81165673500000002 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.854114+0000", 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011126200000000001 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.854227+0000", 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113167 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861145+0000", 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.006917978 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861224+0000", 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9060000000000005e-05 2024-08-30T23:54:15.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.901772+0000", 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.040547444000000002 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.901823+0000", 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1168000000000002e-05 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.954 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.904130+0000", 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023068720000000002 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93490 6.1 6:8ef0ade6:::benchmark_data_smithi173_114000_object93489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.058909+0000", 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.828566378, 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.85139444799999997, 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.955 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93490 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058909+0000", 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058909+0000", 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058911+0000", 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8649999999999999e-06 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058933+0000", 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2123000000000001e-05 2024-08-30T23:54:15.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058947+0000", 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4201e-05 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.058957+0000", 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1479999999999995e-06 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.059064+0000", 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107483 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.059073+0000", 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5420000000000005e-06 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856484+0000", 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79741141100000001 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856594+0000", 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110189 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.856728+0000", 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013400400000000001 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861206+0000", 2024-08-30T23:54:15.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044777849999999998 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861233+0000", 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7263999999999998e-05 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.907851+0000", 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.046617812000000002 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.907892+0000", 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0383999999999998e-05 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.910304+0000", 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024122369999999998 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93498 6.1 6:87f9e3d0:::benchmark_data_smithi173_114000_object93497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:54:15.962 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.078684+0000", 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.80879153699999, 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.83696644099999995, 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93498 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078684+0000", 2024-08-30T23:54:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078684+0000", 2024-08-30T23:54:15.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078685+0000", 2024-08-30T23:54:15.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1000000000000001e-06 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078705+0000", 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0077000000000001e-05 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078717+0000", 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1677e-05 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078723+0000", 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5049999999999998e-06 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078768+0000", 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5979000000000001e-05 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.078776+0000", 2024-08-30T23:54:15.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5870000000000004e-06 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.858569+0000", 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.77979302399999995 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.858718+0000", 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014851099999999999 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.858898+0000", 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000180581 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866139+0000", 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0072403650000000003 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866208+0000", 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9813000000000004e-05 2024-08-30T23:54:15.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.913342+0000", 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.047133988000000002 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.913447+0000", 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104448 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.915651+0000", 2024-08-30T23:54:15.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022037860000000001 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:93539 6.1 6:8ab5d1ae:::benchmark_data_smithi173_114000_object93538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:46:04.262217+0000", 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 491.62525913899998, 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.655331102, 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 93539 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262217+0000", 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262217+0000", 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262218+0000", 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.511e-06 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262241+0000", 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2792000000000001e-05 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262259+0000", 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8172000000000002e-05 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262267+0000", 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6660000000000008e-06 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262325+0000", 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8696000000000002e-05 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.262334+0000", 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2649999999999996e-06 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861251+0000", 2024-08-30T23:54:15.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.59891767799999995 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861352+0000", 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000100416 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 7,9", 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.861463+0000", 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011157799999999999 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866286+0000", 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0048225619999999999 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.866303+0000", 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6894000000000001e-05 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.915703+0000", 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.049399799000000001 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.915740+0000", 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7845000000000001e-05 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:46:04.917548+0000", 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018072279999999999 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.978 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:199570 6.1 6:86daa506:::benchmark_data_smithi173_114000_object99772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 329.77518547099999, 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65846682899999998, 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199570 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.979 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112292+0000", 2024-08-30T23:54:15.980 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.198e-06 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112301+0000", 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0219999999999993e-06 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112321+0000", 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9842000000000001e-05 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112331+0000", 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8949999999999993e-06 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112400+0000", 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8448999999999995e-05 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112494+0000", 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4380000000000001e-05 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112570+0000", 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5704999999999994e-05 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.117141+0000", 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045717809999999996 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.117454+0000", 2024-08-30T23:54:15.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031256200000000001 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770689+0000", 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.653235393 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770728+0000", 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8445999999999997e-05 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770757+0000", 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9156e-05 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27034.0:199570 6.1 e720/715 ondisk, result = 0)", 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 329.76906562300002, 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65235226300000004, 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.985 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118411+0000", 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2900000000000004e-07 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118414+0000", 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8670000000000004e-06 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770637+0000", 2024-08-30T23:54:15.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65222257100000003 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770642+0000", 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3920000000000002e-06 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770680+0000", 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7886999999999998e-05 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770686+0000", 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5670000000000002e-06 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770762+0000", 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6450000000000002e-05 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 765/761 [PushReplyOp(7:f2a3ccc8:::benchmark_data_smithi173_129296_object3685:head)])", 2024-08-30T23:54:15.989 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 225.40041967799999, 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63433747900000004, 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487057+0000", 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0219999999999999e-06 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487063+0000", 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0469999999999999e-06 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115718+0000", 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62865490300000004 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115728+0000", 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9080000000000007e-06 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115812+0000", 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3713000000000003e-05 2024-08-30T23:54:15.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.121393+0000", 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055818860000000003 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 769/761 [PushReplyOp(7:f0c70446:::benchmark_data_smithi173_129296_object15083:head)])", 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 212.46652478499999, 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97997442000000001, 2024-08-30T23:54:15.993 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420952+0000", 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.062e-06 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420958+0000", 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395034+0000", 2024-08-30T23:54:15.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97407640699999998 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395059+0000", 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4338999999999999e-05 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395138+0000", 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9306000000000003e-05 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.400925+0000", 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057873259999999998 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:106574 7.7 7:e13bf739:::benchmark_data_smithi173_129296_object19458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 173.414663539, 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29613937600000001, 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:15.997 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 106574 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472813+0000", 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3600000000000003e-07 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472817+0000", 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2119999999999997e-06 2024-08-30T23:54:15.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472827+0000", 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.91e-06 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472833+0000", 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5310000000000001e-06 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476588+0000", 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037553450000000002 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476692+0000", 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103875 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476803+0000", 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011145999999999999 2024-08-30T23:54:16.001 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.480260+0000", 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034566480000000001 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.486997+0000", 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067370900000000003 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768836+0000", 2024-08-30T23:54:16.002 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28183939600000002 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768874+0000", 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7537000000000003e-05 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768952+0000", 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7636e-05 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 792/761 [PushReplyOp(7:ff5d9ae4:::benchmark_data_smithi173_129296_object65310:head)])", 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 141.53688145300001, 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79968256000000004, 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:54:16.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350595+0000", 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9299999999999998e-07 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350601+0000", 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0379999999999996e-06 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148049+0000", 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79744800299999996 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148074+0000", 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5244000000000001e-05 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148181+0000", 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106742 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.150277+0000", 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020959400000000001 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.007 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:21766 8.c 8:382de8f9:::benchmark_data_smithi173_143816_object21765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:10.080087+0000", 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 5.8073889320000003, 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24394918099999999, 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21766 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:16.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080087+0000", 2024-08-30T23:54:16.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:16.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080087+0000", 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080088+0000", 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4499999999999995e-07 2024-08-30T23:54:16.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080126+0000", 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8760000000000002e-05 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080137+0000", 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0701e-05 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:16.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080142+0000", 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1479999999999997e-06 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080194+0000", 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1773e-05 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080311+0000", 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011705900000000001 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.012 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080452+0000", 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014123200000000001 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.081415+0000", 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096264400000000002 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.081665+0000", 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025000000000000001 2024-08-30T23:54:16.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.323952+0000", 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.24228713800000001 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.323996+0000", 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4317999999999998e-05 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:16.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.324036+0000", 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9462999999999998e-05 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:22800 8.b 8:df24115a:::benchmark_data_smithi173_143816_object22799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:12.688045+0000", 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.199430269, 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.141727301, 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:16.015 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 22800 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688045+0000", 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:16.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688045+0000", 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688047+0000", 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.871e-06 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688070+0000", 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2289999999999998e-05 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688087+0000", 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7336000000000001e-05 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688094+0000", 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6789999999999999e-06 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688148+0000", 2024-08-30T23:54:16.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4218999999999997e-05 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688242+0000", 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4470999999999997e-05 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.688333+0000", 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0777000000000002e-05 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.689295+0000", 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00096176499999999995 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.689602+0000", 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030679200000000001 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.829688+0000", 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.140086658 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.829727+0000", 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9137999999999997e-05 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:12.829773+0000", 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5305000000000001e-05 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:16.021 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:16.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:16.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:16.147 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:16.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:16.273 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:16.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:16.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:16.405 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 27600 27584 82.0879 84.0625 0.0773187 0.0121679 2024-08-30T23:54:16.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:16.531 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:16.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:16.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:16.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:16.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:16.783 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:16.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:16.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:16.908 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:16.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:16.916 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:16.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:54:17.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:17.032 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:17.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:17.155 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:17.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:17.281 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.282 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:17.286 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:16 smithi106 ceph-mon[16603]: pgmap v1908: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 32 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 1/70626 objects degraded (0.001%); 2503/70626 objects misplaced (3.544%); 3.0 MiB/s, 48 objects/s recovering 2024-08-30T23:54:17.286 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:16 smithi106 ceph-mon[16603]: osdmap e857: 12 total, 12 up, 12 in 2024-08-30T23:54:17.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:16 smithi136 ceph-mon[20207]: pgmap v1908: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 32 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 1/70626 objects degraded (0.001%); 2503/70626 objects misplaced (3.544%); 3.0 MiB/s, 48 objects/s recovering 2024-08-30T23:54:17.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:16 smithi136 ceph-mon[20207]: osdmap e857: 12 total, 12 up, 12 in 2024-08-30T23:54:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:16 smithi173 ceph-mon[20554]: pgmap v1908: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 remapped+peering, 32 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 1/70626 objects degraded (0.001%); 2503/70626 objects misplaced (3.544%); 3.0 MiB/s, 48 objects/s recovering 2024-08-30T23:54:17.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:16 smithi173 ceph-mon[20554]: osdmap e857: 12 total, 12 up, 12 in 2024-08-30T23:54:17.383 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:17.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 28254 28238 80.2142 40.875 0.0299347 0.0124527 2024-08-30T23:54:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:17.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:17.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:17.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:17.461 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:17.461 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:17.495 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:17.495 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:17.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:17.523 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:17.523 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:17.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:17.552 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:17.552 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:17.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:17.582 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:17.611 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:17.611 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:17.637 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:17.638 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:17.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:17.654 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:17.654 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "" 2024-08-30T23:54:17.655 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:17.665 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:17.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:17.665 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:17.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:17.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:17.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:17.695 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:17.725 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:17.726 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:17.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:17.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:17.779 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:17.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:17.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:17.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:17.933 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:17.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:18.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:18.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:18.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:18.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2619207697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:54:18.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:17 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.290 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.319 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:18.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2619207697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.320 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:54:18.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:54:18.321 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:17 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2619207697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:18.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:54:18.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:54:18.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:17 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:54:18.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 29531 29515 80.1965 79.8125 0.00701701 0.0124604 2024-08-30T23:54:18.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:18.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:18.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:18.575 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:18.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:18.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:18.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:18.831 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:18.877 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:18.877 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:18.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:18.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:18.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:19.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:19.079 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:18 smithi106 ceph-mon[16603]: pgmap v1910: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 2779/80961 objects degraded (3.433%); 2887/80961 objects misplaced (3.566%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:54:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:18 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:54:19.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:18 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:54:19.181 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:19.209 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:19.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:18 smithi136 ceph-mon[20207]: pgmap v1910: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 2779/80961 objects degraded (3.433%); 2887/80961 objects misplaced (3.566%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:54:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:18 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:54:19.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:18 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:54:19.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:18 smithi173 ceph-mon[20554]: pgmap v1910: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s; 2779/80961 objects degraded (3.433%); 2887/80961 objects misplaced (3.566%); 13 MiB/s, 206 objects/s recovering 2024-08-30T23:54:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:18 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:54:19.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:18 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:54:19.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:19.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 30735 30719 79.9901 75.25 0.00396244 0.0124937 2024-08-30T23:54:19.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:19.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:19.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:19.606 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:19.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:54:19.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:19.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:19.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:19.865 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:19.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:19.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:19.992 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:20.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:20.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:54:20.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:20.139 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:20.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:20.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:20.285 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:20.285 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:20.285 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:20.285 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:20.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:20.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:20.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:19 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:19 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:19 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:20.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:19 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:20.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:20.396 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:20.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 15 32269 32254 80.6277 95.9375 0.00507717 0.0123976 2024-08-30T23:54:20.414 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:20.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.524 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:20.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:20.575 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:20.576 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:20.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.656 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:20.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:20.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:20.779 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:20.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:54:20.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:20.901 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:20.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:21.003 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:21.026 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:21 smithi106 ceph-mon[16603]: pgmap v1911: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 2779/85122 objects degraded (3.265%); 2356/85122 objects misplaced (2.768%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:54:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2779/85122 objects degraded (3.265%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:54:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:21 smithi106 ceph-mon[16603]: osdmap e858: 12 total, 12 up, 12 in 2024-08-30T23:54:21.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:21 smithi106 ceph-mon[16603]: osdmap e859: 12 total, 12 up, 12 in 2024-08-30T23:54:21.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:21.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:21.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:21.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:21.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:21.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:21.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:21 smithi136 ceph-mon[20207]: pgmap v1911: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 2779/85122 objects degraded (3.265%); 2356/85122 objects misplaced (2.768%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:54:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2779/85122 objects degraded (3.265%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:54:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:21 smithi136 ceph-mon[20207]: osdmap e858: 12 total, 12 up, 12 in 2024-08-30T23:54:21.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:21 smithi136 ceph-mon[20207]: osdmap e859: 12 total, 12 up, 12 in 2024-08-30T23:54:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:21 smithi173 ceph-mon[20554]: pgmap v1911: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 2779/85122 objects degraded (3.265%); 2356/85122 objects misplaced (2.768%); 14 MiB/s, 226 objects/s recovering 2024-08-30T23:54:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2779/85122 objects degraded (3.265%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:54:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:21 smithi173 ceph-mon[20554]: osdmap e858: 12 total, 12 up, 12 in 2024-08-30T23:54:21.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:21 smithi173 ceph-mon[20554]: osdmap e859: 12 total, 12 up, 12 in 2024-08-30T23:54:21.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:21.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 33165 33149 79.6779 55.9375 0.0041847 0.0125437 2024-08-30T23:54:21.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:21.513 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:21.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:21.638 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:21.661 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:21.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:21.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:54:21.785 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:21.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:21.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:22.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:22.061 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:22.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:22.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:22.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:22.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:22.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:22.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:22.406 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 34273 34257 79.2914 69.25 0.0034098 0.0126034 2024-08-30T23:54:22.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:22.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:22.538 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:22.553 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:22.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:22.662 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:22.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:22.788 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:22.890 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:22.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:22.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:23.013 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:54:23.013 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:23.014 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:23.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:23.049 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:23.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:23.086 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:23.086 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:23.086 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:23 smithi106 ceph-mon[16603]: pgmap v1914: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 2779/93867 objects degraded (2.961%); 1727/93867 objects misplaced (1.840%); 22 MiB/s, 353 objects/s recovering 2024-08-30T23:54:23.086 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/794719252' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:54:23.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:23.113 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:23.140 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:23.140 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:23.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:23.168 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:23.168 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:23.178 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:23.196 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:23.196 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:23.225 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:23.225 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:23.252 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:23.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:23.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:23.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:23.284 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:23 smithi136 ceph-mon[20207]: pgmap v1914: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 2779/93867 objects degraded (2.961%); 1727/93867 objects misplaced (1.840%); 22 MiB/s, 353 objects/s recovering 2024-08-30T23:54:23.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/794719252' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:54:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.305 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:23.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:23.319 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:23.319 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:23 smithi173 ceph-mon[20554]: pgmap v1914: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s; 2779/93867 objects degraded (2.961%); 1727/93867 objects misplaced (1.840%); 22 MiB/s, 353 objects/s recovering 2024-08-30T23:54:23.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/794719252' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:54:23.350 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:23.351 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:23.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:54:23.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:54:23.367 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-08-30T23:54:23.367 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:54:23.367 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:54:23.368 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-08-30T23:54:23.380 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:23.380 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:23.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:23.407 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 35328 35312 78.8142 65.9375 0.00867423 0.0126819 2024-08-30T23:54:23.411 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:23.412 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:23.437 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:23.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:23.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:23.663 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:23.665 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:23.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:23.721 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:54:23.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:23.825 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:23.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:23.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:23.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:24 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:24.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:24.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:24.105 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:24 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/794719252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:54:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:24 smithi106 ceph-mon[16603]: osdmap e860: 12 total, 12 up, 12 in 2024-08-30T23:54:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1738419061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:24.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:24.204 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:24.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:24.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:24 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/794719252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:54:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:24 smithi173 ceph-mon[20554]: osdmap e860: 12 total, 12 up, 12 in 2024-08-30T23:54:24.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1738419061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:24.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:24.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 36287 36271 78.1294 59.9375 0.00771244 0.0127107 2024-08-30T23:54:24.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:24.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/794719252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:54:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:24 smithi136 ceph-mon[20207]: osdmap e860: 12 total, 12 up, 12 in 2024-08-30T23:54:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1738419061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:24.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:24.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:24.612 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:54:24.612 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "" 2024-08-30T23:54:24.612 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:54:24.643 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:24.643 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:54:24.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.643 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36337 8.2 8:45e7b720:::benchmark_data_smithi173_143816_object36336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:24.643 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.595138+0000", 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.046903704999999997, 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.046933915, 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:24.644 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36337 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.595138+0000", 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.595138+0000", 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.595139+0000", 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.64e-07 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.595177+0000", 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8201000000000002e-05 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.595188+0000", 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0210000000000001e-05 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.595192+0000", 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7779999999999998e-06 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.596612+0000", 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014197579999999999 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.596725+0000", 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011309 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.596858+0000", 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013242399999999999 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.597700+0000", 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00084216300000000005 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.598012+0000", 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031228499999999998 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36362 8.2 8:48a99260:::benchmark_data_smithi173_143816_object36361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.629912+0000", 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.012129773, 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012215126, 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36362 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.629912+0000", 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.629912+0000", 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.629914+0000", 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.99e-06 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.629943+0000", 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9119999999999999e-05 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.629974+0000", 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0326000000000001e-05 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.629983+0000", 2024-08-30T23:54:24.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5859999999999994e-06 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.630061+0000", 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8207000000000005e-05 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.630218+0000", 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000156766 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:54:24.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.630362+0000", 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014355800000000001 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.632323+0000", 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019611910000000001 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.634024+0000", 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001700642 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36364 8.5 8:a1ed3b38:::benchmark_data_smithi173_143816_object36363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.631886+0000", 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.010156109, 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010289652999999999, 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.655 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36364 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631886+0000", 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.656 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631886+0000", 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631887+0000", 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.615e-06 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631913+0000", 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6010999999999999e-05 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631927+0000", 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3993e-05 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631935+0000", 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.631996+0000", 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0726000000000001e-05 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.632140+0000", 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014479200000000001 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.632318+0000", 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000177381 2024-08-30T23:54:24.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.633620+0000", 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001302409 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.634246+0000", 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062594699999999996 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.660 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36369 8.2 8:4b794a39:::benchmark_data_smithi173_143816_object36368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.640232+0000", 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0018095450000000001, 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019990860000000002, 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:24.661 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36369 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640232+0000", 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640232+0000", 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.662 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640234+0000", 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9700000000000002e-06 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640283+0000", 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8618000000000001e-05 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640301+0000", 2024-08-30T23:54:24.663 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8216e-05 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640310+0000", 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8910000000000001e-06 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640374+0000", 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3689000000000006e-05 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:24.664 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640524+0000", 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015062499999999999 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640673+0000", 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000148818 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642072+0000", 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001398577 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36370 8.9 8:9dbf7e79:::benchmark_data_smithi173_143816_object36369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.640923+0000", 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.001118862, 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001346957, 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36370 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640923+0000", 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640923+0000", 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.667 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640924+0000", 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.102e-06 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640985+0000", 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0847999999999997e-05 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.640999+0000", 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3762e-05 2024-08-30T23:54:24.668 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.641006+0000", 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4250000000000001e-06 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.641070+0000", 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4010000000000005e-05 2024-08-30T23:54:24.669 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.641215+0000", 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014517499999999999 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.641375+0000", 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015998799999999999 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.670 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36337 8.2 e861/840 ondisk, result = 0)", 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.597805+0000", 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.044236428000000001, 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.044509813000000002, 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.597805+0000", 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.671 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.597805+0000", 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.597806+0000", 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8299999999999997e-07 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.672 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.597810+0000", 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3619999999999999e-06 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642210+0000", 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.044399507999999997 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642215+0000", 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0119999999999996e-06 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.673 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642254+0000", 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9507e-05 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642259+0000", 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7500000000000003e-06 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.674 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36364 8.5 e861/840)", 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.642306+0000", 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4294967295.9997363, 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0189000000000003e-05, 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "queued for pg", 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642306+0000", 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.675 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642306+0000", 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642306+0000", 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8999999999999996e-07 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642310+0000", 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2189999999999998e-06 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.676 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642347+0000", 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.697e-05 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.642354+0000", 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9310000000000002e-06 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:54:24.677 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 7 2024-08-30T23:54:24.678 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:24.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:24.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:24.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:24.825 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:24.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:24.948 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:24.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:25.050 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:25.076 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:25.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:25.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:25 smithi136 ceph-mon[20207]: pgmap v1916: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 2779/98178 objects degraded (2.831%); 1105/98178 objects misplaced (1.126%); 19 MiB/s, 296 objects/s recovering 2024-08-30T23:54:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:25 smithi136 ceph-mon[20207]: osdmap e861: 12 total, 12 up, 12 in 2024-08-30T23:54:25.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:25 smithi173 ceph-mon[20554]: pgmap v1916: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 2779/98178 objects degraded (2.831%); 1105/98178 objects misplaced (1.126%); 19 MiB/s, 296 objects/s recovering 2024-08-30T23:54:25.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:25 smithi173 ceph-mon[20554]: osdmap e861: 12 total, 12 up, 12 in 2024-08-30T23:54:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:25.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:25 smithi106 ceph-mon[16603]: pgmap v1916: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 956 op/s; 2779/98178 objects degraded (2.831%); 1105/98178 objects misplaced (1.126%); 19 MiB/s, 296 objects/s recovering 2024-08-30T23:54:25.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:25 smithi106 ceph-mon[16603]: osdmap e861: 12 total, 12 up, 12 in 2024-08-30T23:54:25.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 36538 36522 76.0488 15.6875 0.0378688 0.013117 2024-08-30T23:54:25.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:25.447 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:25.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:54:25.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:25.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:25.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:25.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:25.735 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:25.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:25.823 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:25.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:54:25.925 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:25.948 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:25.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:26.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:26.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:26.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:26 smithi173 ceph-mon[20554]: osdmap e862: 12 total, 12 up, 12 in 2024-08-30T23:54:26.166 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2779/98178 objects degraded (2.831%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:54:26.167 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:26 smithi173 ceph-mon[20554]: osdmap e863: 12 total, 12 up, 12 in 2024-08-30T23:54:26.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:26.189 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:26.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.210 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:26.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:26.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:26 smithi106 ceph-mon[16603]: osdmap e862: 12 total, 12 up, 12 in 2024-08-30T23:54:26.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2779/98178 objects degraded (2.831%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:54:26.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:26 smithi106 ceph-mon[16603]: osdmap e863: 12 total, 12 up, 12 in 2024-08-30T23:54:26.324 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:54:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:54:26.324 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:54:26.324 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:26.342 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:26.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 37176 37160 74.8823 39.875 0.0216599 0.013347 2024-08-30T23:54:26.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:26.472 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.473 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:26 smithi136 ceph-mon[20207]: osdmap e862: 12 total, 12 up, 12 in 2024-08-30T23:54:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2779/98178 objects degraded (2.831%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:54:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:26 smithi136 ceph-mon[20207]: osdmap e863: 12 total, 12 up, 12 in 2024-08-30T23:54:26.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:26.599 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:26.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:54:26.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:26.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:26.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:26.846 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:26.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:26.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:26.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:27.070 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:54:27.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:27.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:27.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:27.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:27.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:27 smithi173 ceph-mon[20554]: pgmap v1919: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 2779/104535 objects degraded (2.658%); 708/104535 objects misplaced (0.677%); 21 MiB/s, 342 objects/s recovering 2024-08-30T23:54:27.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:27.349 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:27.351 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:27 smithi106 ceph-mon[16603]: pgmap v1919: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 2779/104535 objects degraded (2.658%); 708/104535 objects misplaced (0.677%); 21 MiB/s, 342 objects/s recovering 2024-08-30T23:54:27.351 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:27.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 37808 37792 73.777 39.5 0.0108897 0.0134927 2024-08-30T23:54:27.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:27.472 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:27 smithi136 ceph-mon[20207]: pgmap v1919: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 32 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 2779/104535 objects degraded (2.658%); 708/104535 objects misplaced (0.677%); 21 MiB/s, 342 objects/s recovering 2024-08-30T23:54:27.574 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:27.595 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:27.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:27.719 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:27.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:27.843 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:27.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:27.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:27.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:28.051 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:28.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:28.090 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:28.112 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:54:28.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:28.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:28.274 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:54:28.274 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:54:28.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86717 7.e 7:7cc26902:::benchmark_data_smithi173_129296_object86716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:17.100563+0000", 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.78143495399999, 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45683305899999999, 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:54:28.275 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86717 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100563+0000", 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100563+0000", 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100564+0000", 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100582+0000", 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8272999999999999e-05 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100591+0000", 2024-08-30T23:54:28.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7930000000000003e-06 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100595+0000", 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6970000000000001e-06 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100638+0000", 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2515999999999997e-05 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100737+0000", 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9041000000000003e-05 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100828+0000", 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1121999999999999e-05 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103141+0000", 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023128519999999998 2024-08-30T23:54:28.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103779+0000", 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063816000000000003 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557286+0000", 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45350701999999998 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557348+0000", 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2018000000000002e-05 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557396+0000", 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7963000000000001e-05 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27487.0:86717 7.e e755/725 ondisk, result = 0)", 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:17.103595+0000", 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 250.77840275700001, 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45380901800000001, 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103595+0000", 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103595+0000", 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103596+0000", 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2099999999999996e-07 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103600+0000", 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3919999999999996e-06 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557219+0000", 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45361822299999999 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557226+0000", 2024-08-30T23:54:28.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6750000000000002e-06 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557268+0000", 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1674000000000001e-05 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557279+0000", 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0969e-05 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557404+0000", 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012536399999999999 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87834 7.e 7:7ff39eeb:::benchmark_data_smithi173_129296_object718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:50.625937+0000", 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 217.256060644, 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36790855099999997, 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.286 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87834 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625937+0000", 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625937+0000", 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625938+0000", 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5700000000000002e-07 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625942+0000", 2024-08-30T23:54:28.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.393e-06 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625951+0000", 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8600000000000005e-06 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625957+0000", 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627053+0000", 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001096334 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627171+0000", 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011757899999999999 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627278+0000", 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010718299999999999 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.629558+0000", 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022799309999999998 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.630384+0000", 2024-08-30T23:54:28.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082581199999999999 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993774+0000", 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.363390306 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993807+0000", 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3179999999999997e-05 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993846+0000", 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8665999999999999e-05 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.293 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27487.0:87834 7.e e771/725 ondisk, result = 0)", 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:50.634204+0000", 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 217.24779368099999, 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35964531900000002, 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634204+0000", 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634204+0000", 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634205+0000", 2024-08-30T23:54:28.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634208+0000", 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9459999999999998e-06 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993718+0000", 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.359510361 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993722+0000", 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0999999999999997e-06 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993764+0000", 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1885999999999998e-05 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993769+0000", 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2140000000000004e-06 2024-08-30T23:54:28.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993850+0000", 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0378999999999997e-05 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27487.0:123065 7.1 e792/725 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head v 792'7770, mlcod=792'7767)", 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:54.487833+0000", 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 153.39416497299999, 2024-08-30T23:54:28.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64708236699999999, 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487833+0000", 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487833+0000", 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487834+0000", 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9999999999999996e-07 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487840+0000", 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2280000000000003e-06 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130467+0000", 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64262676900000004 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130489+0000", 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2246e-05 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130670+0000", 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018123 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130713+0000", 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2786e-05 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.134863+0000", 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041495330000000004 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.134915+0000", 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2775000000000002e-05 2024-08-30T23:54:28.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:127005 7.e 7:7c88bf46:::benchmark_data_smithi173_129296_object39889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:52:02.842372+0000", 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 145.03962558000001, 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.363980886, 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127005 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842372+0000", 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842372+0000", 2024-08-30T23:54:28.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842374+0000", 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5570000000000001e-06 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842381+0000", 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3799999999999996e-06 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842395+0000", 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842405+0000", 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5689999999999994e-06 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842460+0000", 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5458000000000003e-05 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842547+0000", 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842621+0000", 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3875000000000007e-05 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.846407+0000", 2024-08-30T23:54:28.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037859090000000001 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.846902+0000", 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000495124 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206262+0000", 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35936010899999998 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206314+0000", 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1866000000000001e-05 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206353+0000", 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9736e-05 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:21765 8.9 8:9868b323:::benchmark_data_smithi173_143816_object21764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:10.078955+0000", 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.803042933, 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30126246000000001, 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21765 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.310 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078955+0000", 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078955+0000", 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078956+0000", 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.049e-06 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078983+0000", 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6998000000000001e-05 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078994+0000", 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1265e-05 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079000+0000", 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3399999999999997e-06 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079044+0000", 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4563e-05 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079162+0000", 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117231 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079249+0000", 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7340000000000001e-05 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080081+0000", 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083194900000000001 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080390+0000", 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030953600000000001 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380119+0000", 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29972854700000001 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380176+0000", 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6737999999999998e-05 2024-08-30T23:54:28.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380218+0000", 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1903999999999997e-05 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:21765 8.9 e852/840 ondisk, result = 0)", 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:10.084863+0000", 2024-08-30T23:54:28.316 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.797134559, 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29536081199999997, 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.084863+0000", 2024-08-30T23:54:28.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.084863+0000", 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.084864+0000", 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3799999999999997e-07 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.084869+0000", 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2480000000000004e-06 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.379994+0000", 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29512453 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.379999+0000", 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9239999999999998e-06 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380104+0000", 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000105413 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380113+0000", 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8959999999999992e-06 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380224+0000", 2024-08-30T23:54:28.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011116300000000001 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:21811 8.5 8:a2778f2f:::benchmark_data_smithi173_143816_object21810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 16.929411503000001, 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34422484399999997, 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.321 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21811 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5899999999999995e-07 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952609+0000", 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1517000000000001e-05 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952621+0000", 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2041999999999999e-05 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952628+0000", 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8589999999999999e-06 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952666+0000", 2024-08-30T23:54:28.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8696999999999999e-05 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952746+0000", 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9958999999999998e-05 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952828+0000", 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2070000000000005e-05 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.953606+0000", 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077772200000000001 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.954157+0000", 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055049299999999999 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:11.296625+0000", 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34246814399999997 2024-08-30T23:54:28.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:11.296725+0000", 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010038400000000001 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:11.296811+0000", 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6198000000000004e-05 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36149 8.6 e860/854)", 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.055209+0000", 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.8267894920000001, 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41090775600000001, 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055209+0000", 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055209+0000", 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055210+0000", 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0380000000000001e-06 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055243+0000", 2024-08-30T23:54:28.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3021000000000001e-05 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.465880+0000", 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.410637788 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.465894+0000", 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3969e-05 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.466085+0000", 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019035300000000001 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.466116+0000", 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1587000000000001e-05 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.331 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36256 8.2 8:4043fdb0:::benchmark_data_smithi173_143816_object36255:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.264309+0000", 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.6176887799999999, 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27882582299999997, 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36256 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264309+0000", 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264309+0000", 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264310+0000", 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0100000000000004e-07 2024-08-30T23:54:28.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264349+0000", 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8699999999999999e-05 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264359+0000", 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9690000000000006e-06 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264364+0000", 2024-08-30T23:54:28.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8999999999999997e-06 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264405+0000", 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0926000000000001e-05 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264515+0000", 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110708 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:54:28.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.264654+0000", 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013863100000000001 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.265629+0000", 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097529699999999997 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.265713+0000", 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3609999999999994e-05 2024-08-30T23:54:28.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543061+0000", 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27734780999999997 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543098+0000", 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7237000000000003e-05 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543135+0000", 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7234000000000003e-05 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36256 8.2 e861/840 ondisk, result = 0)", 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.265923+0000", 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.6160749939999999, 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27721905000000002, 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:28.338 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.265923+0000", 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.265923+0000", 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.265924+0000", 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9699999999999996e-07 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.265927+0000", 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4460000000000001e-06 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543008+0000", 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27708116599999999 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543013+0000", 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7480000000000001e-06 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543052+0000", 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9134000000000002e-05 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543057+0000", 2024-08-30T23:54:28.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1739999999999999e-06 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.543142+0000", 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4884999999999998e-05 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36263 8.9 8:9ee2fa42:::benchmark_data_smithi173_143816_object36262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.278949+0000", 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.6030495419999999, 2024-08-30T23:54:28.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27773698899999999, 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36263 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.278949+0000", 2024-08-30T23:54:28.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.278949+0000", 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.278950+0000", 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1370000000000001e-06 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.278976+0000", 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6271e-05 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.278988+0000", 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1674999999999999e-05 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.278993+0000", 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3349999999999997e-06 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.345 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.279042+0000", 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8751000000000001e-05 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.279148+0000", 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010614499999999999 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.279242+0000", 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3714000000000004e-05 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.280431+0000", 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001189655 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.280631+0000", 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019949099999999999 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556617+0000", 2024-08-30T23:54:28.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27598609200000002 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556654+0000", 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6816000000000001e-05 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556685+0000", 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1906999999999998e-05 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36263 8.9 e861/840 ondisk, result = 0)", 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.280838+0000", 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 3.6011599859999999, 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.275852767, 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.280838+0000", 2024-08-30T23:54:28.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.280838+0000", 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.280839+0000", 2024-08-30T23:54:28.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3900000000000004e-07 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.280842+0000", 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.455e-06 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556567+0000", 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.27572452600000003 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556571+0000", 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.42e-06 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556607+0000", 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5475000000000002e-05 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556613+0000", 2024-08-30T23:54:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3709999999999996e-06 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.556691+0000", 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7880999999999995e-05 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36527 8.5 8:a4f61870:::benchmark_data_smithi173_143816_object36526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.366820+0000", 2024-08-30T23:54:28.354 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.5151783170000002, 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42580382900000002, 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36527 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366820+0000", 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366820+0000", 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366821+0000", 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366840+0000", 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9429e-05 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366849+0000", 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366854+0000", 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4880000000000001e-06 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366888+0000", 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4350000000000001e-05 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366994+0000", 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010563700000000001 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.367078+0000", 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3735999999999999e-05 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.367886+0000", 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00080812600000000005 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368184+0000", 2024-08-30T23:54:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029837400000000001 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792529+0000", 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42434479800000002 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792588+0000", 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8922000000000003e-05 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792624+0000", 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5751999999999999e-05 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36527 8.5 e862/840 ondisk, result = 0)", 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.368528+0000", 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.5134697570000002, 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.424101687, 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368528+0000", 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368528+0000", 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368529+0000", 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9100000000000004e-07 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368532+0000", 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8169999999999999e-06 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792461+0000", 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.423929155 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792467+0000", 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8019999999999997e-06 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792512+0000", 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5624999999999998e-05 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792521+0000", 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1540000000000008e-06 2024-08-30T23:54:28.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792630+0000", 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108643 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36528 8.7 e862/854 8:ed4e2712:::benchmark_data_smithi173_143816_object36527:head v 862'2374, mlcod=845'96)", 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.368738+0000", 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.5132596830000002, 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45802140800000002, 2024-08-30T23:54:28.365 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368738+0000", 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368738+0000", 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368739+0000", 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5300000000000004e-07 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368831+0000", 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2077999999999995e-05 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825237+0000", 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45640643600000003 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825246+0000", 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0749999999999998e-06 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825305+0000", 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9893999999999999e-05 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825332+0000", 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6239000000000001e-05 2024-08-30T23:54:28.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.826715+0000", 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013836230000000001 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.826760+0000", 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4509999999999999e-05 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36535 8.9 8:9fb542f2:::benchmark_data_smithi173_143816_object36534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.395482+0000", 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.4865159019999998, 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41703795100000002, 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36535 2024-08-30T23:54:28.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395482+0000", 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395482+0000", 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395484+0000", 2024-08-30T23:54:28.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.562e-06 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395507+0000", 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3461000000000001e-05 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395523+0000", 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6069000000000001e-05 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395531+0000", 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0039999999999999e-06 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395580+0000", 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8621000000000001e-05 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395686+0000", 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106156 2024-08-30T23:54:28.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395786+0000", 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9801000000000002e-05 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.396991+0000", 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012049350000000001 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.397402+0000", 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041134199999999998 2024-08-30T23:54:28.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812443+0000", 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41504124100000001 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812477+0000", 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4093e-05 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812520+0000", 2024-08-30T23:54:28.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2666000000000001e-05 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36543 8.7 e862/854 8:ea468d17:::benchmark_data_smithi173_143816_object36542:head v 862'2375, mlcod=845'96)", 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.550435+0000", 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 2.3315632860000002, 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28063033500000001, 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.376 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.550435+0000", 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.550435+0000", 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.550436+0000", 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1999999999999998e-07 2024-08-30T23:54:28.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.550503+0000", 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7707999999999996e-05 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.829743+0000", 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.279239295 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.829749+0000", 2024-08-30T23:54:28.378 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6699999999999997e-06 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.829805+0000", 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6221000000000003e-05 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.829835+0000", 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9144000000000001e-05 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.379 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.831032+0000", 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00119709 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.831065+0000", 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3386999999999997e-05 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37809 8.2 8:45f707ad:::benchmark_data_smithi173_143816_object37808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:27.427735+0000", 2024-08-30T23:54:28.380 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.45426270800000002, 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37169286800000001, 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37809 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427735+0000", 2024-08-30T23:54:28.381 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427735+0000", 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427737+0000", 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1850000000000001e-06 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.382 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427757+0000", 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0934999999999999e-05 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427771+0000", 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3672999999999999e-05 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427779+0000", 2024-08-30T23:54:28.383 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7999999999999999e-06 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427829+0000", 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9795e-05 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427918+0000", 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9624999999999997e-05 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:54:28.384 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.428017+0000", 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8312000000000003e-05 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.429169+0000", 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011526660000000001 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.456373+0000", 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027204145999999998 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.385 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799342+0000", 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34296861200000001 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799393+0000", 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1048999999999999e-05 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799428+0000", 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5070000000000001e-05 2024-08-30T23:54:28.386 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.387 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.387 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.387 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:54:28.387 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:54:28.387 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:54:28.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:28.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:28 smithi106 ceph-mon[16603]: osdmap e864: 12 total, 12 up, 12 in 2024-08-30T23:54:28.388 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1623434449' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:28.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:28.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 37856 37840 71.633 3 0.00233248 0.0135391 2024-08-30T23:54:28.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:28.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:28.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:28 smithi136 ceph-mon[20207]: osdmap e864: 12 total, 12 up, 12 in 2024-08-30T23:54:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1623434449' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:28.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:28 smithi173 ceph-mon[20554]: osdmap e864: 12 total, 12 up, 12 in 2024-08-30T23:54:28.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1623434449' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:28.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:28.658 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:28.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:28.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:28.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:28.905 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:28.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:29.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:29.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:29.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:29.154 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:29.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:29.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:29.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:29.415 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:29.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 38506 38490 70.7213 40.625 0.00316018 0.0141329 2024-08-30T23:54:29.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:29 smithi136 ceph-mon[20207]: pgmap v1922: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 2205/111360 objects misplaced (1.980%); 26 MiB/s, 419 objects/s recovering 2024-08-30T23:54:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:29 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2779/104535 objects degraded (2.658%), 3 pgs degraded) 2024-08-30T23:54:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:29 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1623434449' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:29 smithi136 ceph-mon[20207]: osdmap e865: 12 total, 12 up, 12 in 2024-08-30T23:54:29.548 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:29.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:29.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:29 smithi106 ceph-mon[16603]: pgmap v1922: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 2205/111360 objects misplaced (1.980%); 26 MiB/s, 419 objects/s recovering 2024-08-30T23:54:29.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:29 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2779/104535 objects degraded (2.658%), 3 pgs degraded) 2024-08-30T23:54:29.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:29 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:29.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1623434449' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:29.570 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:29 smithi106 ceph-mon[16603]: osdmap e865: 12 total, 12 up, 12 in 2024-08-30T23:54:29.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:29 smithi173 ceph-mon[20554]: pgmap v1922: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s; 2205/111360 objects misplaced (1.980%); 26 MiB/s, 419 objects/s recovering 2024-08-30T23:54:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:29 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2779/104535 objects degraded (2.658%), 3 pgs degraded) 2024-08-30T23:54:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:29 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1623434449' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:29 smithi173 ceph-mon[20554]: osdmap e865: 12 total, 12 up, 12 in 2024-08-30T23:54:29.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:29.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:29.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:29.809 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:29.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:29.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:29.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:30.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:30.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:30.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:30.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:30.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:30.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:30.419 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 39757 39741 70.9343 78.1875 0.00184814 0.014087 2024-08-30T23:54:30.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:30.447 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:30.460 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:54:30.461 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:54:30.461 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-30T23:54:30.461 INFO:tasks.thrashosds.thrasher:Setting osd 9 primary_affinity to 0.701449 2024-08-30T23:54:30.461 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 9 0.701449462334963 2024-08-30T23:54:30.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:30.582 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:30.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:30.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:30.667 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:30.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:30.702 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:30.703 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:30.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:30.739 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:30.739 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:30.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:30.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:30.801 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:30.801 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:30.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:30.834 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:30.834 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:30.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:30.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:30.846 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:30.864 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:30.865 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:30.896 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:30.896 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:30.925 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:30.926 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:30.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:30.955 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:30.955 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:30.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:30.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:30.981 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:30.982 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:31.011 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:31.011 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:31.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:31.090 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:31.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:54:31.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:31.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:31.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:31.346 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:31 smithi106 ceph-mon[16603]: pgmap v1924: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 729 op/s; 2205/112767 objects misplaced (1.955%); 17 MiB/s, 270 objects/s recovering 2024-08-30T23:54:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1746510460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:31.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 41088 41072 71.2741 83.1875 0.0125451 0.0140239 2024-08-30T23:54:31.448 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:31.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:31.496 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:31.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:31 smithi136 ceph-mon[20207]: pgmap v1924: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 729 op/s; 2205/112767 objects misplaced (1.955%); 17 MiB/s, 270 objects/s recovering 2024-08-30T23:54:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1746510460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:31.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:31 smithi173 ceph-mon[20554]: pgmap v1924: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 729 op/s; 2205/112767 objects misplaced (1.955%); 17 MiB/s, 270 objects/s recovering 2024-08-30T23:54:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1746510460' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:31.577 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:31.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:31.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:31.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:31.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:31.865 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:31.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:31.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:32.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:32.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:54:32.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:32.112 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:32.113 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:32.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:32.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:32.259 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:32.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:32.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:32.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 42435 42419 71.6228 84.1875 0.00419184 0.0139562 2024-08-30T23:54:32.445 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:32.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:32.523 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:32.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:32.649 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:32.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:32.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:32.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:54:32.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:32.925 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:32.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:33.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:33.051 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:33.124 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.9 primary-affinity to 0.701449 (8459702) 2024-08-30T23:54:33.152 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:33.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:33.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:33.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:33.302 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.303 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:33 smithi106 ceph-mon[16603]: pgmap v1925: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 720 op/s; 1066/117498 objects misplaced (0.907%); 25 MiB/s, 403 objects/s recovering 2024-08-30T23:54:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1099154487' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963} : dispatch 2024-08-30T23:54:33.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:33 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963} : dispatch 2024-08-30T23:54:33.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:33.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 16 43282 43266 71.1312 52.9375 0.136853 0.014017 2024-08-30T23:54:33.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:33 smithi173 ceph-mon[20554]: pgmap v1925: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 720 op/s; 1066/117498 objects misplaced (0.907%); 25 MiB/s, 403 objects/s recovering 2024-08-30T23:54:33.421 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1099154487' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963} : dispatch 2024-08-30T23:54:33.422 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:33 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963} : dispatch 2024-08-30T23:54:33.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:33.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:33.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:33 smithi136 ceph-mon[20207]: pgmap v1925: 37 pgs: 1 creating+peering, 2 active+recovering+undersized+remapped, 34 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 720 op/s; 1066/117498 objects misplaced (0.907%); 25 MiB/s, 403 objects/s recovering 2024-08-30T23:54:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1099154487' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963} : dispatch 2024-08-30T23:54:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:33 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963} : dispatch 2024-08-30T23:54:33.553 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:33.655 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:33.669 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:33.669 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:33.677 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:33.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:33.801 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:33.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:33.924 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:54:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:33.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:33.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:34.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:34.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:34.122 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:54:34.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:34.173 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:34.275 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:34.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:34 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963}]': finished 2024-08-30T23:54:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:34 smithi173 ceph-mon[20554]: osdmap e866: 12 total, 12 up, 12 in 2024-08-30T23:54:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3504391365' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:54:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:34 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3504391365' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:54:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:34 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963}]': finished 2024-08-30T23:54:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:34 smithi106 ceph-mon[16603]: osdmap e866: 12 total, 12 up, 12 in 2024-08-30T23:54:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3504391365' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:54:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:34 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:34.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3504391365' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:54:34.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:54:34.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:34.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 15 43924 43909 70.3379 40.1875 0.00737664 0.0142112 2024-08-30T23:54:34.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:34.529 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:34.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:34 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 0.701449462334963}]': finished 2024-08-30T23:54:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:34 smithi136 ceph-mon[20207]: osdmap e866: 12 total, 12 up, 12 in 2024-08-30T23:54:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3504391365' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:54:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:34 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3504391365' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:54:34.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:34.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:34.684 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:34.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:34.793 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:54:34.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:34.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:34.938 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:34.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:34.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:35.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:35.074 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:35.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:35.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:35.232 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:35 smithi173 ceph-mon[20554]: pgmap v1927: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 649 op/s; 1066/122661 objects misplaced (0.869%); 17 MiB/s, 276 objects/s recovering 2024-08-30T23:54:35.232 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:35 smithi173 ceph-mon[20554]: osdmap e867: 12 total, 12 up, 12 in 2024-08-30T23:54:35.232 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:35 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-30T23:54:35.232 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:35 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-30T23:54:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:35 smithi136 ceph-mon[20207]: pgmap v1927: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 649 op/s; 1066/122661 objects misplaced (0.869%); 17 MiB/s, 276 objects/s recovering 2024-08-30T23:54:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:35 smithi136 ceph-mon[20207]: osdmap e867: 12 total, 12 up, 12 in 2024-08-30T23:54:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:35 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-30T23:54:35.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:35 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-30T23:54:35.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:35.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:35.339 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:35 smithi106 ceph-mon[16603]: pgmap v1927: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 649 op/s; 1066/122661 objects misplaced (0.869%); 17 MiB/s, 276 objects/s recovering 2024-08-30T23:54:35.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:35 smithi106 ceph-mon[16603]: osdmap e867: 12 total, 12 up, 12 in 2024-08-30T23:54:35.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:35 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-30T23:54:35.340 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:35 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-30T23:54:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:35.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:54:35.420832+0000 min lat: 0.0010542 max lat: 2.02823 avg lat: 0.0140537 2024-08-30T23:54:35.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:54:35.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 45409 45393 70.8978 92.75 0.0476758 0.0140537 2024-08-30T23:54:35.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:35.450 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:35.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:35.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:35.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:35.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:35.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:35.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:35.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:35.942 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:35.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:35.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:36.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:36.070 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:36.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:36.193 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:36.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:36 smithi173 ceph-mon[20554]: osdmap e868: 12 total, 12 up, 12 in 2024-08-30T23:54:36.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2966272137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:36.214 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:36 smithi173 ceph-mon[20554]: osdmap e869: 12 total, 12 up, 12 in 2024-08-30T23:54:36.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:54:36.286 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:36.286 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:36.295 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:36.308 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:36.309 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:36.317 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:36.331 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:36.331 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:36.351 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:36.351 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:36.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:36.372 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:36.393 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:36.393 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:36.393 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:36 smithi106 ceph-mon[16603]: osdmap e868: 12 total, 12 up, 12 in 2024-08-30T23:54:36.394 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2966272137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:36.394 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:36 smithi106 ceph-mon[16603]: osdmap e869: 12 total, 12 up, 12 in 2024-08-30T23:54:36.414 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:36.414 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:36.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:36.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 45691 45675 69.5988 17.625 0.0205026 0.0142547 2024-08-30T23:54:36.441 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:36.491 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:36.534 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:36.534 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:36 smithi136 ceph-mon[20207]: osdmap e868: 12 total, 12 up, 12 in 2024-08-30T23:54:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2966272137' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:36 smithi136 ceph-mon[20207]: osdmap e869: 12 total, 12 up, 12 in 2024-08-30T23:54:36.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:36.556 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:36.565 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:36.579 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:36.579 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:36.600 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:36.600 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:36.623 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:36.624 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:36.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:36.689 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:36.792 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.822 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:36.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:36.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:54:36.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:36.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:36.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:37.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:37.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:37.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:37.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:37.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:37.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:37 smithi106 ceph-mon[16603]: pgmap v1930: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 55 MiB/s wr, 878 op/s; 1066/128574 objects misplaced (0.829%); 12 MiB/s, 189 objects/s recovering 2024-08-30T23:54:37.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:37 smithi106 ceph-mon[16603]: osdmap e870: 12 total, 12 up, 12 in 2024-08-30T23:54:37.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:37.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 45736 45720 68.0091 2.8125 0.00264217 0.0142761 2024-08-30T23:54:37.440 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:37 smithi136 ceph-mon[20207]: pgmap v1930: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 55 MiB/s wr, 878 op/s; 1066/128574 objects misplaced (0.829%); 12 MiB/s, 189 objects/s recovering 2024-08-30T23:54:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:37 smithi136 ceph-mon[20207]: osdmap e870: 12 total, 12 up, 12 in 2024-08-30T23:54:37.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:37.564 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:37 smithi173 ceph-mon[20554]: pgmap v1930: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 55 MiB/s wr, 878 op/s; 1066/128574 objects misplaced (0.829%); 12 MiB/s, 189 objects/s recovering 2024-08-30T23:54:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:37 smithi173 ceph-mon[20554]: osdmap e870: 12 total, 12 up, 12 in 2024-08-30T23:54:37.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:37.697 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:37.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:37.824 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:37.856 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:46102 8.1 8:8569b0ea:::benchmark_data_smithi173_143816_object46101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:37.839686+0000", 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.015812818999999999, 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.015846387999999999, 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:54:37.857 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 46102 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839686+0000", 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839686+0000", 2024-08-30T23:54:37.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839687+0000", 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0589999999999999e-06 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839718+0000", 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1285000000000003e-05 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839729+0000", 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0254e-05 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839735+0000", 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3450000000000002e-06 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839778+0000", 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2459999999999997e-05 2024-08-30T23:54:37.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.839913+0000", 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013511900000000001 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.840066+0000", 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015362700000000001 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.841282+0000", 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001215341 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.841900+0000", 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00061827699999999998 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:46123 8.7 e870/869 8:e8def338:::benchmark_data_smithi173_143816_object46122:head v 870'2974, mlcod=870'2972)", 2024-08-30T23:54:37.862 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:37.854624+0000", 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00087462200000000003, 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00098701400000000003, 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854624+0000", 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:54:37.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854624+0000", 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854625+0000", 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1099999999999995e-07 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854684+0000", 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8477000000000003e-05 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854703+0000", 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9228999999999998e-05 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854711+0000", 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1319999999999994e-06 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854786+0000", 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4935e-05 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.854807+0000", 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1070999999999999e-05 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:46125 8.f e870/868 8:fdc60027:::benchmark_data_smithi173_143816_object46124:head v 870'2782, mlcod=870'2780)", 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:37.855067+0000", 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.00043204099999999998, 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00059430500000000005, 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:54:37.866 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855067+0000", 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855067+0000", 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855067+0000", 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0900000000000002e-07 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855125+0000", 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7135000000000001e-05 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855137+0000", 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2401e-05 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855145+0000", 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5399999999999998e-06 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855196+0000", 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1180000000000001e-05 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.855218+0000", 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1869999999999999e-05 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.869 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:54:37.870 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.870 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:54:37.870 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:54:37.870 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 3 2024-08-30T23:54:37.870 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:54:37.870 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:37.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:37.947 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:37.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:38.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:38.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:38.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:38.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:38.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:38.329 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:38.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 46786 46770 67.9536 65.625 0.0297474 0.0147076 2024-08-30T23:54:38.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:38.454 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:38.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:38 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 1/135804 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:54:38.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:38 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 1/135804 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:54:38.583 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:38 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 1/135804 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:54:38.584 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.586 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:38.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:54:38.686 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:38.711 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:38.813 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:38.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:54:38.828 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:54:38.828 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_6 2024-08-30T23:54:38.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:54:38.845 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:38.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:38.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:38.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:39.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:39.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:39.190 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:39.194 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:39.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:39.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.227 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:39.328 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:39.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:39.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 47929 47913 68.0326 71.4375 0.0193745 0.0146907 2024-08-30T23:54:39.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:39.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:39 smithi106 ceph-mon[16603]: pgmap v1933: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 68 MiB/s wr, 1.10k op/s; 1/135804 objects degraded (0.001%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:54:39.475 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:39.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:39 smithi136 ceph-mon[20207]: pgmap v1933: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 68 MiB/s wr, 1.10k op/s; 1/135804 objects degraded (0.001%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:54:39.571 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:54:39.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:39 smithi173 ceph-mon[20554]: pgmap v1933: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 68 MiB/s wr, 1.10k op/s; 1/135804 objects degraded (0.001%); 17 MiB/s, 266 objects/s recovering 2024-08-30T23:54:39.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:39.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:39.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:39.733 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:39.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:39.864 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:39.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:39.986 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:39.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:40.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:40.109 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:40.210 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:54:40.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:40.240 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:40.277 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:54:40.278 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:54:40.318 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:54:40.318 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:54:40.318 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:54:40.318 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:54:40.318 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:54:40.319 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2436590503' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:40.319 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:40 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:40.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:40.370 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:40.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 49111 49095 68.1623 73.875 0.00760745 0.0146624 2024-08-30T23:54:40.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:40.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2436590503' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:40.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:40 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2436590503' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:40.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:40 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:40.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:40.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:40.733 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:40.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:40.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:40.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:40.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:40.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:41.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:41.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:41.151 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:41.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:41.284 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:41.386 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:41.411 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:41.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 50736 50720 68.888 101.562 0.0532637 0.0145079 2024-08-30T23:54:41.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:41.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: pgmap v1934: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 886 op/s; 1/138825 objects degraded (0.001%); 12 MiB/s, 187 objects/s recovering 2024-08-30T23:54:41.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:41.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:41.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:54:41.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:41.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: osdmap e871: 12 total, 12 up, 12 in 2024-08-30T23:54:41.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:41 smithi106 ceph-mon[16603]: from='client.18621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:41.536 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:41.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: pgmap v1934: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 886 op/s; 1/138825 objects degraded (0.001%); 12 MiB/s, 187 objects/s recovering 2024-08-30T23:54:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:54:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: osdmap e871: 12 total, 12 up, 12 in 2024-08-30T23:54:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:41 smithi136 ceph-mon[20207]: from='client.18621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:41.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: pgmap v1934: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 886 op/s; 1/138825 objects degraded (0.001%); 12 MiB/s, 187 objects/s recovering 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: osdmap e871: 12 total, 12 up, 12 in 2024-08-30T23:54:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:41 smithi173 ceph-mon[20554]: from='client.18621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:41.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:41.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:41.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:41.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:41.893 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:41.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:41.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:42.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:42.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:42.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:42.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:42.186 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:42.224 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1935,"stamp":"2024-08-30T23:54:40.207683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3033137696,"num_objects":46275,"num_object_clones":0,"num_object_copies":138825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46273,"num_write_kb":2961472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6908,"num_bytes_recovered":453246976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48205,"ondisk_log_size":48205,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16174332,"kb_used_data":9290568,"kb_used_omap":109,"kb_used_meta":6883410,"kb_avail":1108767492,"statfs":{"total":1151940427776,"available":1135377911808,"internally_reserved":0,"allocated":9513541632,"data_stored":9505938128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":7048612504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":269072926,"num_objects":4106,"num_object_clones":0,"num_object_copies":12318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":-1066,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":4102,"num_write_kb":262719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1066,"num_bytes_recovered":69813292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.065823"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":44,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996725+0000","last_change":"2024-08-30T23:54:10.991736+0000","last_active":"2024-08-30T23:54:36.996725+0000","last_peered":"2024-08-30T23:54:36.996725+0000","last_clean":"2024-08-30T23:54:36.996725+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:54:36.996725+0000","last_undegraded":"2024-08-30T23:54:36.996725+0000","last_fullsized":"2024-08-30T23:54:36.996725+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.112421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.388213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":185,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.265120+0000","last_change":"2024-08-30T23:53:38.104290+0000","last_active":"2024-08-30T23:54:37.265120+0000","last_peered":"2024-08-30T23:54:37.265120+0000","last_clean":"2024-08-30T23:54:37.265120+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:54:37.265120+0000","last_undegraded":"2024-08-30T23:54:37.265120+0000","last_fullsized":"2024-08-30T23:54:37.265120+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:38.104244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:38.104244+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:45.249002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":802,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.146597+0000","last_change":"2024-08-30T23:53:47.855440+0000","last_active":"2024-08-30T23:54:35.146597+0000","last_peered":"2024-08-30T23:54:35.146597+0000","last_clean":"2024-08-30T23:54:35.146597+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:54:35.146597+0000","last_undegraded":"2024-08-30T23:54:35.146597+0000","last_fullsized":"2024-08-30T23:54:35.146597+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:47.855365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:47.855365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.334601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":185,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.266013+0000","last_change":"2024-08-30T23:53:39.095429+0000","last_active":"2024-08-30T23:54:37.266013+0000","last_peered":"2024-08-30T23:54:37.266013+0000","last_clean":"2024-08-30T23:54:37.266013+0000","last_became_active":"2024-08-30T23:51:42.948334+0000","last_became_peered":"2024-08-30T23:51:42.948334+0000","last_unstale":"2024-08-30T23:54:37.266013+0000","last_undegraded":"2024-08-30T23:54:37.266013+0000","last_fullsized":"2024-08-30T23:54:37.266013+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:39.095359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:39.095359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.506854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"869'2824","reported_seq":2892,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.733193+0000","last_change":"2024-08-30T23:53:49.706683+0000","last_active":"2024-08-30T23:54:36.733193+0000","last_peered":"2024-08-30T23:54:36.733193+0000","last_clean":"2024-08-30T23:54:36.733193+0000","last_became_active":"2024-08-30T23:53:49.706310+0000","last_became_peered":"2024-08-30T23:53:49.706310+0000","last_unstale":"2024-08-30T23:54:36.733193+0000","last_undegraded":"2024-08-30T23:54:36.733193+0000","last_fullsized":"2024-08-30T23:54:36.733193+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.016839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185073664,"num_objects":2824,"num_object_clones":0,"num_object_copies":8472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2824,"num_write_kb":180736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":599,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.136639+0000","last_change":"2024-08-30T23:54:11.556855+0000","last_active":"2024-08-30T23:54:37.136639+0000","last_peered":"2024-08-30T23:54:37.136639+0000","last_clean":"2024-08-30T23:54:37.136639+0000","last_became_active":"2024-08-30T23:54:10.990434+0000","last_became_peered":"2024-08-30T23:54:10.990434+0000","last_unstale":"2024-08-30T23:54:37.136639+0000","last_undegraded":"2024-08-30T23:54:37.136639+0000","last_fullsized":"2024-08-30T23:54:37.136639+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:11.556806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:11.556806+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:27.495366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"869'2932","reported_seq":5748,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996770+0000","last_change":"2024-08-30T23:54:29.061695+0000","last_active":"2024-08-30T23:54:36.996770+0000","last_peered":"2024-08-30T23:54:36.996770+0000","last_clean":"2024-08-30T23:54:36.996770+0000","last_became_active":"2024-08-30T23:54:29.059713+0000","last_became_peered":"2024-08-30T23:54:29.059713+0000","last_unstale":"2024-08-30T23:54:36.996770+0000","last_undegraded":"2024-08-30T23:54:36.996770+0000","last_fullsized":"2024-08-30T23:54:36.996770+0000","mapping_epoch":864,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":865,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2932,"log_dups_size":0,"ondisk_log_size":2932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:15.589492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192151552,"num_objects":2932,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":187648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":89522176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":52,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996732+0000","last_change":"2024-08-30T23:54:11.516693+0000","last_active":"2024-08-30T23:54:36.996732+0000","last_peered":"2024-08-30T23:54:36.996732+0000","last_clean":"2024-08-30T23:54:36.996732+0000","last_became_active":"2024-08-30T23:54:10.989215+0000","last_became_peered":"2024-08-30T23:54:10.989215+0000","last_unstale":"2024-08-30T23:54:36.996732+0000","last_undegraded":"2024-08-30T23:54:36.996732+0000","last_fullsized":"2024-08-30T23:54:36.996732+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:11.516643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:11.516643+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:26.816354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"870'2975","reported_seq":3052,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:39.327753+0000","last_change":"2024-08-30T23:54:34.130863+0000","last_active":"2024-08-30T23:54:39.327753+0000","last_peered":"2024-08-30T23:54:39.327753+0000","last_clean":"2024-08-30T23:54:39.327753+0000","last_became_active":"2024-08-30T23:54:34.127322+0000","last_became_peered":"2024-08-30T23:54:34.127322+0000","last_unstale":"2024-08-30T23:54:39.327753+0000","last_undegraded":"2024-08-30T23:54:39.327753+0000","last_fullsized":"2024-08-30T23:54:39.327753+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2975,"log_dups_size":0,"ondisk_log_size":2975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.332392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194969600,"num_objects":2975,"num_object_clones":0,"num_object_copies":8925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":190400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":381,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.994411+0000","last_change":"2024-08-30T23:53:51.042769+0000","last_active":"2024-08-30T23:54:35.994411+0000","last_peered":"2024-08-30T23:54:35.994411+0000","last_clean":"2024-08-30T23:54:35.994411+0000","last_became_active":"2024-08-30T23:52:07.960637+0000","last_became_peered":"2024-08-30T23:52:07.960637+0000","last_unstale":"2024-08-30T23:54:35.994411+0000","last_undegraded":"2024-08-30T23:54:35.994411+0000","last_fullsized":"2024-08-30T23:54:35.994411+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.042710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.042710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.950115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"870'2760","reported_seq":5583,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.279463+0000","last_change":"2024-08-30T23:54:35.999956+0000","last_active":"2024-08-30T23:54:37.279463+0000","last_peered":"2024-08-30T23:54:37.279463+0000","last_clean":"2024-08-30T23:54:37.279463+0000","last_became_active":"2024-08-30T23:54:35.995990+0000","last_became_peered":"2024-08-30T23:54:35.995990+0000","last_unstale":"2024-08-30T23:54:37.279463+0000","last_undegraded":"2024-08-30T23:54:37.279463+0000","last_fullsized":"2024-08-30T23:54:37.279463+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2760,"log_dups_size":0,"ondisk_log_size":2760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:05.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180879360,"num_objects":2760,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2760,"num_write_kb":176640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1369,"num_bytes_recovered":89718784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":355,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.994529+0000","last_change":"2024-08-30T23:53:43.091100+0000","last_active":"2024-08-30T23:54:35.994529+0000","last_peered":"2024-08-30T23:54:35.994529+0000","last_clean":"2024-08-30T23:54:35.994529+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:54:35.994529+0000","last_undegraded":"2024-08-30T23:54:35.994529+0000","last_fullsized":"2024-08-30T23:54:35.994529+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.091048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.091048+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:10.926425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"869'2885","reported_seq":2954,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.327506+0000","last_change":"2024-08-30T23:53:49.706502+0000","last_active":"2024-08-30T23:54:36.327506+0000","last_peered":"2024-08-30T23:54:36.327506+0000","last_clean":"2024-08-30T23:54:36.327506+0000","last_became_active":"2024-08-30T23:53:49.705978+0000","last_became_peered":"2024-08-30T23:53:49.705978+0000","last_unstale":"2024-08-30T23:54:36.327506+0000","last_undegraded":"2024-08-30T23:54:36.327506+0000","last_fullsized":"2024-08-30T23:54:36.327506+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.097093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189071360,"num_objects":2885,"num_object_clones":0,"num_object_copies":8655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":184640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":997,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.136744+0000","last_change":"2024-08-30T23:53:51.507308+0000","last_active":"2024-08-30T23:54:37.136744+0000","last_peered":"2024-08-30T23:54:37.136744+0000","last_clean":"2024-08-30T23:54:37.136744+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:54:37.136744+0000","last_undegraded":"2024-08-30T23:54:37.136744+0000","last_fullsized":"2024-08-30T23:54:37.136744+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.507248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.507248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.951121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"870'2965","reported_seq":3036,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:39.697103+0000","last_change":"2024-08-30T23:53:49.708022+0000","last_active":"2024-08-30T23:54:39.697103+0000","last_peered":"2024-08-30T23:54:39.697103+0000","last_clean":"2024-08-30T23:54:39.697103+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:54:39.697103+0000","last_undegraded":"2024-08-30T23:54:39.697103+0000","last_fullsized":"2024-08-30T23:54:39.697103+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.990219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194314240,"num_objects":2965,"num_object_clones":0,"num_object_copies":8895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":189760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":346,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.994496+0000","last_change":"2024-08-30T23:53:41.104980+0000","last_active":"2024-08-30T23:54:35.994496+0000","last_peered":"2024-08-30T23:54:35.994496+0000","last_clean":"2024-08-30T23:54:35.994496+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:54:35.994496+0000","last_undegraded":"2024-08-30T23:54:35.994496+0000","last_fullsized":"2024-08-30T23:54:35.994496+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.104925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.104925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:09.944567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"868'2884","reported_seq":2950,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.621908+0000","last_change":"2024-08-30T23:53:49.671937+0000","last_active":"2024-08-30T23:54:35.621908+0000","last_peered":"2024-08-30T23:54:35.621908+0000","last_clean":"2024-08-30T23:54:35.621908+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:54:35.621908+0000","last_undegraded":"2024-08-30T23:54:35.621908+0000","last_fullsized":"2024-08-30T23:54:35.621908+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2884,"log_dups_size":0,"ondisk_log_size":2884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:14.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189005824,"num_objects":2884,"num_object_clones":0,"num_object_copies":8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2884,"num_write_kb":184576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1837,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.992331+0000","last_change":"2024-08-30T23:53:50.999427+0000","last_active":"2024-08-30T23:54:35.992331+0000","last_peered":"2024-08-30T23:54:35.992331+0000","last_clean":"2024-08-30T23:54:35.992331+0000","last_became_active":"2024-08-30T23:52:54.016899+0000","last_became_peered":"2024-08-30T23:52:54.016899+0000","last_unstale":"2024-08-30T23:54:35.992331+0000","last_undegraded":"2024-08-30T23:54:35.992331+0000","last_fullsized":"2024-08-30T23:54:35.992331+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:50.999362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:50.999362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.278203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"868'2827","reported_seq":2893,"reported_epoch":868,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.487452+0000","last_change":"2024-08-30T23:53:49.709301+0000","last_active":"2024-08-30T23:54:35.487452+0000","last_peered":"2024-08-30T23:54:35.487452+0000","last_clean":"2024-08-30T23:54:35.487452+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:54:35.487452+0000","last_undegraded":"2024-08-30T23:54:35.487452+0000","last_fullsized":"2024-08-30T23:54:35.487452+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2827,"log_dups_size":0,"ondisk_log_size":2827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.536121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185270272,"num_objects":2827,"num_object_clones":0,"num_object_copies":8481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":180928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":168,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.136646+0000","last_change":"2024-08-30T23:54:18.583660+0000","last_active":"2024-08-30T23:54:37.136646+0000","last_peered":"2024-08-30T23:54:37.136646+0000","last_clean":"2024-08-30T23:54:37.136646+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:54:37.136646+0000","last_undegraded":"2024-08-30T23:54:37.136646+0000","last_fullsized":"2024-08-30T23:54:37.136646+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:18.583610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:18.583610+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:22.571284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"869'2935","reported_seq":3003,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.721792+0000","last_change":"2024-08-30T23:53:49.670752+0000","last_active":"2024-08-30T23:54:36.721792+0000","last_peered":"2024-08-30T23:54:36.721792+0000","last_clean":"2024-08-30T23:54:36.721792+0000","last_became_active":"2024-08-30T23:53:49.670369+0000","last_became_peered":"2024-08-30T23:53:49.670369+0000","last_unstale":"2024-08-30T23:54:36.721792+0000","last_undegraded":"2024-08-30T23:54:36.721792+0000","last_fullsized":"2024-08-30T23:54:36.721792+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2935,"log_dups_size":0,"ondisk_log_size":2935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.808262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192348160,"num_objects":2935,"num_object_clones":0,"num_object_copies":8805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":187840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":814,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996041+0000","last_change":"2024-08-30T23:53:49.354762+0000","last_active":"2024-08-30T23:54:36.996041+0000","last_peered":"2024-08-30T23:54:36.996041+0000","last_clean":"2024-08-30T23:54:36.996041+0000","last_became_active":"2024-08-30T23:50:25.926833+0000","last_became_peered":"2024-08-30T23:50:25.926833+0000","last_unstale":"2024-08-30T23:54:36.996041+0000","last_undegraded":"2024-08-30T23:54:36.996041+0000","last_fullsized":"2024-08-30T23:54:36.996041+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.354704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.354704+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.152954+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"870'2934","reported_seq":3005,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:39.704654+0000","last_change":"2024-08-30T23:53:49.707268+0000","last_active":"2024-08-30T23:54:39.704654+0000","last_peered":"2024-08-30T23:54:39.704654+0000","last_clean":"2024-08-30T23:54:39.704654+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:54:39.704654+0000","last_undegraded":"2024-08-30T23:54:39.704654+0000","last_fullsized":"2024-08-30T23:54:39.704654+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2934,"log_dups_size":0,"ondisk_log_size":2934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.387982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192282624,"num_objects":2934,"num_object_clones":0,"num_object_copies":8802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2934,"num_write_kb":187776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":802,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.992389+0000","last_change":"2024-08-30T23:53:41.049687+0000","last_active":"2024-08-30T23:54:35.992389+0000","last_peered":"2024-08-30T23:54:35.992389+0000","last_clean":"2024-08-30T23:54:35.992389+0000","last_became_active":"2024-08-30T23:52:54.016432+0000","last_became_peered":"2024-08-30T23:52:54.016432+0000","last_unstale":"2024-08-30T23:54:35.992389+0000","last_undegraded":"2024-08-30T23:54:35.992389+0000","last_fullsized":"2024-08-30T23:54:35.992389+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.049626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.049626+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.750692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"868'2942","reported_seq":5836,"reported_epoch":870,"state":"creating+activating+degraded","last_fresh":"2024-08-30T23:54:37.000027+0000","last_change":"2024-08-30T23:54:37.000027+0000","last_active":"2024-08-30T23:54:36.001066+0000","last_peered":"2024-08-30T23:54:35.485157+0000","last_clean":"2024-08-30T23:54:10.042640+0000","last_became_active":"2024-08-30T23:54:12.083699+0000","last_became_peered":"2024-08-30T23:54:12.083699+0000","last_unstale":"2024-08-30T23:54:37.000027+0000","last_undegraded":"2024-08-30T23:54:36.999017+0000","last_fullsized":"2024-08-30T23:54:37.000027+0000","mapping_epoch":869,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":188288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1409,"num_bytes_recovered":92340224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":["11","1","2"],"object_location_counts":[{"shards":"1,10,11","objects":2941},{"shards":"1,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":733,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:35.992410+0000","last_change":"2024-08-30T23:53:35.098110+0000","last_active":"2024-08-30T23:54:35.992410+0000","last_peered":"2024-08-30T23:54:35.992410+0000","last_clean":"2024-08-30T23:54:35.992410+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:54:35.992410+0000","last_undegraded":"2024-08-30T23:54:35.992410+0000","last_fullsized":"2024-08-30T23:54:35.992410+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:35.098031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:35.098031+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.053662+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"869'2915","reported_seq":5831,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996796+0000","last_change":"2024-08-30T23:54:25.133351+0000","last_active":"2024-08-30T23:54:36.996796+0000","last_peered":"2024-08-30T23:54:36.996796+0000","last_clean":"2024-08-30T23:54:36.996796+0000","last_became_active":"2024-08-30T23:54:25.106180+0000","last_became_peered":"2024-08-30T23:54:25.106180+0000","last_unstale":"2024-08-30T23:54:36.996796+0000","last_undegraded":"2024-08-30T23:54:36.996796+0000","last_fullsized":"2024-08-30T23:54:36.996796+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2915,"log_dups_size":0,"ondisk_log_size":2915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.077859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191037440,"num_objects":2915,"num_object_clones":0,"num_object_copies":8745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2915,"num_write_kb":186560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":92798976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1006,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.136735+0000","last_change":"2024-08-30T23:54:09.628811+0000","last_active":"2024-08-30T23:54:37.136735+0000","last_peered":"2024-08-30T23:54:37.136735+0000","last_clean":"2024-08-30T23:54:37.136735+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:54:37.136735+0000","last_undegraded":"2024-08-30T23:54:37.136735+0000","last_fullsized":"2024-08-30T23:54:37.136735+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:09.628739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:09.628739+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.854773+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"869'2759","reported_seq":2828,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.565901+0000","last_change":"2024-08-30T23:53:49.706361+0000","last_active":"2024-08-30T23:54:36.565901+0000","last_peered":"2024-08-30T23:54:36.565901+0000","last_clean":"2024-08-30T23:54:36.565901+0000","last_became_active":"2024-08-30T23:53:49.706070+0000","last_became_peered":"2024-08-30T23:53:49.706070+0000","last_unstale":"2024-08-30T23:54:36.565901+0000","last_undegraded":"2024-08-30T23:54:36.565901+0000","last_fullsized":"2024-08-30T23:54:36.565901+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2759,"log_dups_size":0,"ondisk_log_size":2759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:02.364402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180813824,"num_objects":2759,"num_object_clones":0,"num_object_copies":8277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":176576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":197,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.265983+0000","last_change":"2024-08-30T23:53:43.164672+0000","last_active":"2024-08-30T23:54:37.265983+0000","last_peered":"2024-08-30T23:54:37.265983+0000","last_clean":"2024-08-30T23:54:37.265983+0000","last_became_active":"2024-08-30T23:51:42.952967+0000","last_became_peered":"2024-08-30T23:51:42.952967+0000","last_unstale":"2024-08-30T23:54:37.265983+0000","last_undegraded":"2024-08-30T23:54:37.265983+0000","last_fullsized":"2024-08-30T23:54:37.265983+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.164609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.164609+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.951863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"870'3072","reported_seq":3143,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:38.839643+0000","last_change":"2024-08-30T23:53:49.706815+0000","last_active":"2024-08-30T23:54:38.839643+0000","last_peered":"2024-08-30T23:54:38.839643+0000","last_clean":"2024-08-30T23:54:38.839643+0000","last_became_active":"2024-08-30T23:53:49.706543+0000","last_became_peered":"2024-08-30T23:53:49.706543+0000","last_unstale":"2024-08-30T23:54:38.839643+0000","last_undegraded":"2024-08-30T23:54:38.839643+0000","last_fullsized":"2024-08-30T23:54:38.839643+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.291074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":3072,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3072,"num_write_kb":196608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1818,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.237851+0000","last_change":"2024-08-30T23:53:42.303311+0000","last_active":"2024-08-30T23:54:37.237851+0000","last_peered":"2024-08-30T23:54:37.237851+0000","last_clean":"2024-08-30T23:54:37.237851+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:54:37.237851+0000","last_undegraded":"2024-08-30T23:54:37.237851+0000","last_fullsized":"2024-08-30T23:54:37.237851+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.303263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.303263+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:47.897994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"869'2836","reported_seq":5613,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996712+0000","last_change":"2024-08-30T23:54:25.999482+0000","last_active":"2024-08-30T23:54:36.996712+0000","last_peered":"2024-08-30T23:54:36.996712+0000","last_clean":"2024-08-30T23:54:36.996712+0000","last_became_active":"2024-08-30T23:54:25.996662+0000","last_became_peered":"2024-08-30T23:54:25.996662+0000","last_unstale":"2024-08-30T23:54:36.996712+0000","last_undegraded":"2024-08-30T23:54:36.996712+0000","last_fullsized":"2024-08-30T23:54:36.996712+0000","mapping_epoch":862,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":863,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2836,"log_dups_size":0,"ondisk_log_size":2836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:11.653706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185860096,"num_objects":2836,"num_object_clones":0,"num_object_copies":8508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2836,"num_write_kb":181504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":88211456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":932,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.265427+0000","last_change":"2024-08-30T23:53:49.897094+0000","last_active":"2024-08-30T23:54:37.265427+0000","last_peered":"2024-08-30T23:54:37.265427+0000","last_clean":"2024-08-30T23:54:37.265427+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:54:37.265427+0000","last_undegraded":"2024-08-30T23:54:37.265427+0000","last_fullsized":"2024-08-30T23:54:37.265427+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.897038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.897038+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.963756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"869'2828","reported_seq":2896,"reported_epoch":869,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.724023+0000","last_change":"2024-08-30T23:53:49.706603+0000","last_active":"2024-08-30T23:54:36.724023+0000","last_peered":"2024-08-30T23:54:36.724023+0000","last_clean":"2024-08-30T23:54:36.724023+0000","last_became_active":"2024-08-30T23:53:49.706194+0000","last_became_peered":"2024-08-30T23:53:49.706194+0000","last_unstale":"2024-08-30T23:54:36.724023+0000","last_undegraded":"2024-08-30T23:54:36.724023+0000","last_fullsized":"2024-08-30T23:54:36.724023+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:04.835283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2828,"num_write_kb":180992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":44,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:36.996865+0000","last_change":"2024-08-30T23:54:10.988040+0000","last_active":"2024-08-30T23:54:36.996865+0000","last_peered":"2024-08-30T23:54:36.996865+0000","last_clean":"2024-08-30T23:54:36.996865+0000","last_became_active":"2024-08-30T23:54:10.987627+0000","last_became_peered":"2024-08-30T23:54:10.987627+0000","last_unstale":"2024-08-30T23:54:36.996865+0000","last_undegraded":"2024-08-30T23:54:36.996865+0000","last_fullsized":"2024-08-30T23:54:36.996865+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.078720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.078720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:55.813319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":905,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-08-30T23:54:37.265528+0000","last_change":"2024-08-30T23:54:34.892672+0000","last_active":"2024-08-30T23:54:37.265528+0000","last_peered":"2024-08-30T23:54:37.265528+0000","last_clean":"2024-08-30T23:54:37.265528+0000","last_became_active":"2024-08-30T23:54:34.127104+0000","last_became_peered":"2024-08-30T23:54:34.127104+0000","last_unstale":"2024-08-30T23:54:37.265528+0000","last_undegraded":"2024-08-30T23:54:37.265528+0000","last_fullsized":"2024-08-30T23:54:37.265528+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:50.478726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3032547328,"num_objects":46273,"num_object_clones":0,"num_object_copies":138819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46273,"num_write_kb":2961472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6906,"num_bytes_recovered":452591616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9416212480,"data_stored":9416212480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46273,"ondisk_log_size":46273,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1526932,"kb_used_data":922364,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92218220,"statfs":{"total":95995035648,"available":94431457280,"internally_reserved":0,"allocated":944500736,"data_stored":943865025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":619051460},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449458,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1722440,"kb_used_data":1217840,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92022712,"statfs":{"total":95995035648,"available":94231257088,"internally_reserved":0,"allocated":1247068160,"data_stored":1246436923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":516684224},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711088,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801356,"kb_used_data":1130172,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91943796,"statfs":{"total":95995035648,"available":94150447104,"internally_reserved":0,"allocated":1157296128,"data_stored":1156660417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":687274432},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972729,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1009896,"kb_used_data":537240,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92735256,"statfs":{"total":95995035648,"available":94960902144,"internally_reserved":0,"allocated":550133760,"data_stored":549511392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234366,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1074896,"kb_used_data":375932,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92670256,"statfs":{"total":95995035648,"available":94894342144,"internally_reserved":0,"allocated":384954368,"data_stored":384318657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484408,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93260744,"statfs":{"total":95995035648,"available":95499001856,"internally_reserved":0,"allocated":2068480,"data_stored":1441688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775211,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1225880,"kb_used_data":1013956,"kb_used_omap":5,"kb_used_meta":211898,"kb_avail":92519272,"statfs":{"total":95995035648,"available":94739734528,"internally_reserved":0,"allocated":1038290944,"data_stored":1037651681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":216983626},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504772,"kb_used_data":855792,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92240380,"statfs":{"total":95995035648,"available":94454149120,"internally_reserved":0,"allocated":876331008,"data_stored":875699771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150097,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1437336,"kb_used_data":643972,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92307816,"statfs":{"total":95995035648,"available":94523203584,"internally_reserved":0,"allocated":659427328,"data_stored":658788065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":812382663},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411735,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1545624,"kb_used_data":928580,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92199528,"statfs":{"total":95995035648,"available":94412316672,"internally_reserved":0,"allocated":950865920,"data_stored":950226657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830551,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521668,"kb_used_data":1096688,"kb_used_omap":87,"kb_used_meta":424872,"kb_avail":92223484,"statfs":{"total":95995035648,"available":94436847616,"internally_reserved":0,"allocated":1123008512,"data_stored":1122377275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":435069087},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319124,"kb_used_data":566012,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":92426028,"statfs":{"total":95995035648,"available":94644252672,"internally_reserved":0,"allocated":579596288,"data_stored":578960577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":577699840,"data_stored":577699840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1122304000,"data_stored":1122304000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":969932800,"data_stored":969932800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":584187904,"data_stored":584187904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":843776000,"data_stored":843776000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1050673152,"data_stored":1050673152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":88211456,"data_stored":88211456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":384368640,"data_stored":384368640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":558170112,"data_stored":558170112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1166802944,"data_stored":1166802944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1127350272,"data_stored":1127350272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":942735360,"data_stored":942735360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:54:42.271 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:42.303 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:42.405 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:42.420 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 51934 51918 69.0152 74.875 0.00364048 0.0144828 2024-08-30T23:54:42.433 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:42.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:42.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:42.562 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:42.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:42.692 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:42.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:42.826 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:42.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:42.952 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:42.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:43.053 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:43.074 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:43.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:43.199 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:43.301 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:43.367 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:43.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 53171 53155 69.1879 77.3125 0.00382445 0.0144461 2024-08-30T23:54:43.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:43.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:43.530 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:43 smithi106 ceph-mon[16603]: pgmap v1936: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 760 op/s; 1/142278 objects degraded (0.001%); 11 MiB/s, 177 objects/s recovering 2024-08-30T23:54:43.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:43 smithi136 ceph-mon[20207]: pgmap v1936: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 760 op/s; 1/142278 objects degraded (0.001%); 11 MiB/s, 177 objects/s recovering 2024-08-30T23:54:43.556 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:43.556 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:43 smithi173 ceph-mon[20554]: pgmap v1936: 37 pgs: 1 creating+activating+degraded, 36 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 760 op/s; 1/142278 objects degraded (0.001%); 11 MiB/s, 177 objects/s recovering 2024-08-30T23:54:43.582 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:43.587 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:43.588 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:43.591 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:43.614 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:43.620 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:43.620 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:43.648 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:43.648 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:43.675 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:43.675 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:43.702 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:43.703 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:43.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:43.729 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:43.729 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:43.737 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:43.751 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:43.751 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:43.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:43.772 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:43.793 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:43.793 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:43.815 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:43.816 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:43.836 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:43.837 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:43.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:43.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:43.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:43.988 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:43.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:44.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:44.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:44.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:44.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:44.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:44.377 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:44.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 54310 54294 69.2285 71.1875 0.0509651 0.01442 2024-08-30T23:54:44.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:44.499 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1554628506' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:44.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1554628506' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:44.601 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:44.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1554628506' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:44.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:44.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:44.752 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:44.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:44.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:44.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:44.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:45.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:45.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:45.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:45.216 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:45.216 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:45.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:45.268 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:45.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:54:45.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:45.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:45.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 55323 55307 69.1101 63.3125 0.0180341 0.014464 2024-08-30T23:54:45.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: pgmap v1937: 37 pgs: 37 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 967 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/142278 objects degraded (0.001%), 1 pg degraded) 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3052591852,config=1460075751,health=3897446751,kv=2911001951}) 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2005314096,logm=2717917047}) 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688465814}) 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235536630}) 2024-08-30T23:54:45.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1764246426}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461621072}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1972208643}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3959856026}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1295809570}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1695877581}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605301699}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1061255844}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517242838}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1059494407}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615365130}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550405820}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3763428052}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420697972}) 2024-08-30T23:54:45.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4085578028}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1037454075}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3722921414}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1928145526}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693216856}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996939600}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998602552}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432771468}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2305158680}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807844962}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568623071}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17943830}) 2024-08-30T23:54:45.524 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3025477220}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2938923901}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715350732}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432525713}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1141079483}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386686879}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213581602}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4289115609}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277766805}) 2024-08-30T23:54:45.525 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360756247}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845493739}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3537283863}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1148833774}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2547103941}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4117328675}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067904757}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690201700}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757396318}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2387560431}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=388175533}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=427529314}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1589272667}) 2024-08-30T23:54:45.526 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834335646}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384666519}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464493013}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705682879}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487279484}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422675122}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2894585315}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982522273}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3832898399}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1857248198}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1678089271}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026702772}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603353491}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339660095}) 2024-08-30T23:54:45.527 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940725547}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284084243}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1343528554}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568417557}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=1020964975,mdsmap=425822562,mgr=2854436405}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mgr=10,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=77} crc {mgr=4173678721,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=965343331,mon_config_key=104925080}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2264810196}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1178396209}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=17,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=65} crc {mon_config_key=1402564663,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=1617681472,osdmap=4127284031}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2633842513}) 2024-08-30T23:54:45.528 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=2573850329}) 2024-08-30T23:54:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:45.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: pgmap v1937: 37 pgs: 37 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 967 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-30T23:54:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/142278 objects degraded (0.001%), 1 pg degraded) 2024-08-30T23:54:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:54:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3052591852,config=1460075751,health=3897446751,kv=2911001951}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2005314096,logm=2717917047}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688465814}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235536630}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1764246426}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461621072}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1972208643}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3959856026}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1295809570}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1695877581}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605301699}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1061255844}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517242838}) 2024-08-30T23:54:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1059494407}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615365130}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550405820}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3763428052}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420697972}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4085578028}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1037454075}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3722921414}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1928145526}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693216856}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996939600}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998602552}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432771468}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2305158680}) 2024-08-30T23:54:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807844962}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568623071}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17943830}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3025477220}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2938923901}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715350732}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432525713}) 2024-08-30T23:54:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1141079483}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386686879}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213581602}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4289115609}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277766805}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360756247}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845493739}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3537283863}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1148833774}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2547103941}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4117328675}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067904757}) 2024-08-30T23:54:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690201700}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757396318}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2387560431}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=388175533}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=427529314}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1589272667}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834335646}) 2024-08-30T23:54:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384666519}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464493013}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705682879}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487279484}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422675122}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2894585315}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982522273}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3832898399}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1857248198}) 2024-08-30T23:54:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1678089271}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026702772}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603353491}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339660095}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940725547}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284084243}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1343528554}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568417557}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=1020964975,mdsmap=425822562,mgr=2854436405}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mgr=10,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=77} crc {mgr=4173678721,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=965343331,mon_config_key=104925080}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2264810196}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1178396209}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=17,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=65} crc {mon_config_key=1402564663,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=1617681472,osdmap=4127284031}) 2024-08-30T23:54:45.549 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2633842513}) 2024-08-30T23:54:45.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=2573850329}) 2024-08-30T23:54:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: pgmap v1937: 37 pgs: 37 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 967 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-30T23:54:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/142278 objects degraded (0.001%), 1 pg degraded) 2024-08-30T23:54:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:54:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=3052591852,config=1460075751,health=3897446751,kv=2911001951}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {kv=9,logm=91} crc {kv=2005314096,logm=2717917047}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688465814}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235536630}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1764246426}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461621072}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1972208643}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3959856026}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1295809570}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1695877581}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605301699}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1061255844}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=517242838}) 2024-08-30T23:54:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1059494407}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1615365130}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550405820}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3763428052}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=420697972}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4085578028}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1037454075}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3722921414}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1928145526}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693216856}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1996939600}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2998602552}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432771468}) 2024-08-30T23:54:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2305158680}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3807844962}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3568623071}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=17943830}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3025477220}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2938923901}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3715350732}) 2024-08-30T23:54:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2432525713}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1141079483}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386686879}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2213581602}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4289115609}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277766805}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=360756247}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=845493739}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3537283863}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1148833774}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2547103941}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4117328675}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2067904757}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690201700}) 2024-08-30T23:54:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3757396318}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2387560431}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=388175533}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=427529314}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1589272667}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=834335646}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2384666519}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464493013}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705682879}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487279484}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422675122}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2894585315}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2982522273}) 2024-08-30T23:54:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3832898399}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1857248198}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1678089271}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1026702772}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603353491}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339660095}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940725547}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4284084243}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1343528554}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1568417557}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=1020964975,mdsmap=425822562,mgr=2854436405}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mgr=10,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=77} crc {mgr=4173678721,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=965343331,mon_config_key=104925080}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2264810196}) 2024-08-30T23:54:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1178396209}) 2024-08-30T23:54:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=17,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=65} crc {mon_config_key=1402564663,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=1617681472,osdmap=4127284031}) 2024-08-30T23:54:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2633842513}) 2024-08-30T23:54:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=2573850329}) 2024-08-30T23:54:45.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:45.655 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:45.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:45.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:45.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:45.911 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:45.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:46.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:46.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:54:46.042 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:46.069 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:46.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:46.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:46.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:46.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:46.338 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:46.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:46.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 56943 56927 69.7399 101.25 0.0081451 0.0143326 2024-08-30T23:54:46.427 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:46.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:46.550 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:46.651 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:46.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:46.708 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:54:46.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:46.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:46.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:46.939 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:46.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:47.034 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:47.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:47.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:47.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:47.190 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:47.220 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:54:47.227 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:54:47.227 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:54:47.228 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-30T23:54:47.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:54:47.249 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:47.250 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:47.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:47.316 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:47.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:47.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 57841 57825 69.478 56.125 0.036466 0.0143569 2024-08-30T23:54:47.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:47 smithi136 ceph-mon[20207]: pgmap v1938: 37 pgs: 37 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 978 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-08-30T23:54:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/199467242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:54:47.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:47.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:47.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:47 smithi173 ceph-mon[20554]: pgmap v1938: 37 pgs: 37 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 978 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-08-30T23:54:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/199467242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:54:47.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:47 smithi106 ceph-mon[16603]: pgmap v1938: 37 pgs: 37 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 978 op/s; 7.6 MiB/s, 121 objects/s recovering 2024-08-30T23:54:47.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/199467242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:54:47.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:47.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.698 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:47.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:47.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:47.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:47.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:47.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:48.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:48.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:54:48.084 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:48.186 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:48.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:48.311 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:48.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:48.368 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:48.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 59044 59028 69.5856 75.1875 0.0226709 0.0143637 2024-08-30T23:54:48.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.468 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:48.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:48.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:48 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:48.491 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/199467242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:54:48.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:48 smithi136 ceph-mon[20207]: osdmap e872: 12 total, 12 up, 12 in 2024-08-30T23:54:48.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:48 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:48.492 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:48 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:48.505 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:48.505 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:48.505 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:48 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:48.505 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/199467242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:54:48.505 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:48 smithi106 ceph-mon[16603]: osdmap e872: 12 total, 12 up, 12 in 2024-08-30T23:54:48.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:48 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:48.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:48 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:48.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:48.532 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:48.563 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:48.564 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:48.569 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:48.573 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:48 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/199467242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:54:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:48 smithi173 ceph-mon[20554]: osdmap e872: 12 total, 12 up, 12 in 2024-08-30T23:54:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:48 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:48 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:48.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:48.595 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:48.596 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:48.621 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:48.621 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:48.649 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:48.649 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:48.674 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:48.674 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:48.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:48.699 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:48.699 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:48.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:48.728 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:48.729 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:48.757 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:48.757 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:48.788 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:48.788 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:48.818 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:48.819 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:48.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:48.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:48.947 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:54:48.947 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:54:48.947 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:54:48.947 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:54:48.947 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:54:48.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:48.976 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:48.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:49.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:49.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:49.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:49.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:49.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:49.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:49.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 15 60386 60371 69.8512 83.9375 0.00792357 0.0143101 2024-08-30T23:54:49.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:49.496 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:49.534 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:54:49.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:49 smithi106 ceph-mon[16603]: pgmap v1940: 37 pgs: 37 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-08-30T23:54:49.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:49 smithi106 ceph-mon[16603]: osdmap e873: 12 total, 12 up, 12 in 2024-08-30T23:54:49.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3803171715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:49.536 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:54:49.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:49 smithi136 ceph-mon[20207]: pgmap v1940: 37 pgs: 37 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-08-30T23:54:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:49 smithi136 ceph-mon[20207]: osdmap e873: 12 total, 12 up, 12 in 2024-08-30T23:54:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3803171715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:49 smithi173 ceph-mon[20554]: pgmap v1940: 37 pgs: 37 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-08-30T23:54:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:49 smithi173 ceph-mon[20554]: osdmap e873: 12 total, 12 up, 12 in 2024-08-30T23:54:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3803171715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:49.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:49.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:49.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:49.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:49.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:54:49.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:49.880 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:49.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:49.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:50.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:50.058 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:54:50.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:50.138 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:50.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:50.263 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:50.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:54:50.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:50.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:50.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 61555 61539 69.9083 73 0.0196365 0.0142691 2024-08-30T23:54:50.448 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:50 smithi136 ceph-mon[20207]: from='client.18645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:50.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:50.514 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:50.560 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:54:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:54:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:54:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:54:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.7 653/650 [PushOp(6:e8735af4:::benchmark_data_smithi173_114000_object5598:head, version: 624'324, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e8735af4:::benchmark_data_smithi173_114000_object5598:head@624'324, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:02.524306+0000", 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 528.03380474400001, 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35381532799999998, 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524306+0000", 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524306+0000", 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524308+0000", 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.524353+0000", 2024-08-30T23:54:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5572999999999999e-05 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875433+0000", 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35107914800000001 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875455+0000", 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2251000000000001e-05 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875559+0000", 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103646 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.875575+0000", 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6943e-05 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:02.878122+0000", 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025463370000000001 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 655/654 [PushOp(6:f180e721:::benchmark_data_smithi173_114000_object484:head, version: 621'29, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f180e721:::benchmark_data_smithi173_114000_object484:head@621'29, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:04.555659+0000", 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 526.00245223000002, 2024-08-30T23:54:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29762476500000001, 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555659+0000", 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555659+0000", 2024-08-30T23:54:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555661+0000", 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6670000000000001e-06 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.555719+0000", 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8227999999999999e-05 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.849316+0000", 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29359667900000003 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.849333+0000", 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7226000000000001e-05 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.852058+0000", 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027255059999999999 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.852067+0000", 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4780000000000008e-06 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.853284+0000", 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001216981 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 684/654 [PushOp(6:f01ac110:::benchmark_data_smithi173_114000_object56049:head, version: 638'3433, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f01ac110:::benchmark_data_smithi173_114000_object56049:head@638'3433, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:39.648816+0000", 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 430.90929507200002, 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70308431000000005, 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648816+0000", 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648816+0000", 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648818+0000", 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5999999999999999e-06 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648891+0000", 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3188999999999993e-05 2024-08-30T23:54:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348689+0000", 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.69979756900000001 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348717+0000", 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8413000000000001e-05 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348915+0000", 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019764199999999999 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348938+0000", 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3536999999999998e-05 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.351900+0000", 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029623599999999998 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 691/654 [PushOp(6:f5b42d7a:::benchmark_data_smithi173_114000_object68947:head, version: 642'4266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f5b42d7a:::benchmark_data_smithi173_114000_object68947:head@642'4266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:57.450979+0000", 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 413.107132374, 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.50071664100000002, 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.450979+0000", 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.450979+0000", 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.450980+0000", 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0900000000000001e-07 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.451033+0000", 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3090999999999997e-05 2024-08-30T23:54:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.899123+0000", 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44809082900000002 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.899140+0000", 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6158e-05 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.947031+0000", 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047891488000000003 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.947045+0000", 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3534e-05 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.951696+0000", 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046508319999999997 2024-08-30T23:54:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:193162 6.4 e715/684 6:2b4eb615:::benchmark_data_smithi173_114000_object93364:head v 715'12123, mlcod=715'12119)", 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:37.213384+0000", 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 373.34472712500002, 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24900366600000001, 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213384+0000", 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213384+0000", 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213385+0000", 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4799999999999997e-07 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213392+0000", 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0249999999999997e-06 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459261+0000", 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24586888600000001 2024-08-30T23:54:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459269+0000", 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7559999999999999e-06 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459325+0000", 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5841000000000003e-05 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459341+0000", 2024-08-30T23:54:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.462353+0000", 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030117640000000001 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.462388+0000", 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4659000000000001e-05 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.0 792/788 [PushOp(7:0a4adabc:::benchmark_data_smithi173_129296_object36673:head, version: 736'2291, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:0a4adabc:::benchmark_data_smithi173_129296_object36673:head@736'2291, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:54.441081+0000", 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 176.11703073499999, 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.221514667, 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.441081+0000", 2024-08-30T23:54:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.441081+0000", 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.441081+0000", 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7899999999999997e-07 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.441112+0000", 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0488e-05 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.660792+0000", 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21967988999999999 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.660808+0000", 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6044000000000001e-05 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.660912+0000", 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104499 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.660921+0000", 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4919999999999993e-06 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.662595+0000", 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016744749999999999 2024-08-30T23:54:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.0 795/788 [PushOp(7:0859067a:::benchmark_data_smithi173_129296_object41869:head, version: 739'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:0859067a:::benchmark_data_smithi173_129296_object41869:head@739'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 167.74393577399999, 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.2782153, 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814176+0000", 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7199999999999998e-07 2024-08-30T23:54:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814204+0000", 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7526e-05 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090173+0000", 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27596965699999998 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090195+0000", 2024-08-30T23:54:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2075000000000001e-05 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090304+0000", 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010874699999999999 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090321+0000", 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7149000000000001e-05 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.092391+0000", 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002069474 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:6025 8.9 e847/840 8:9d1f5af9:::benchmark_data_smithi173_143816_object6024:head v 847'358, mlcod=847'355)", 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:59.018704+0000", 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.539407189000002, 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24216917299999999, 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018704+0000", 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018704+0000", 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018705+0000", 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5799999999999999e-07 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018783+0000", 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8517999999999996e-05 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260248+0000", 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.241465287 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260254+0000", 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.203e-06 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260307+0000", 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3344999999999998e-05 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260326+0000", 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8572000000000001e-05 2024-08-30T23:54:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260837+0000", 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000511926 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260873+0000", 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5763999999999998e-05 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21763 8.4 8:2f5d79f6:::benchmark_data_smithi173_143816_object21762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.480175209000002, 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30212884200000001, 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21763 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077938+0000", 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0480000000000001e-06 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077983+0000", 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5318999999999998e-05 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078009+0000", 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5554999999999998e-05 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078016+0000", 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4769999999999997e-06 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078078+0000", 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1748999999999994e-05 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078236+0000", 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000157434 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078373+0000", 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013750000000000001 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079802+0000", 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001428653 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079818+0000", 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6597999999999999e-05 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379984+0000", 2024-08-30T23:54:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30016611799999998 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380032+0000", 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7268000000000002e-05 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380065+0000", 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3123000000000001e-05 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21763 8.4 e852/840 ondisk, result = 0)", 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.474614932999998, 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.296575003, 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083497+0000", 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1500000000000004e-07 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083505+0000", 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379828+0000", 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29632308899999998 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379844+0000", 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5665999999999999e-05 2024-08-30T23:54:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379970+0000", 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012600999999999999 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379979+0000", 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9770000000000006e-06 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380071+0000", 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2387999999999997e-05 2024-08-30T23:54:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21778 8.8 8:10e57910:::benchmark_data_smithi173_143816_object21777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.330788231, 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28839689299999999, 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21778 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227333+0000", 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5489999999999995e-06 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227385+0000", 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2265999999999997e-05 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227427+0000", 2024-08-30T23:54:50.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2255000000000001e-05 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227452+0000", 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4443999999999998e-05 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227520+0000", 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8916000000000002e-05 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227528+0000", 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-30T23:54:50.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228639+0000", 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011110810000000001 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228736+0000", 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7001000000000002e-05 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228817+0000", 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1038000000000006e-05 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.229951+0000", 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001133319 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230356+0000", 2024-08-30T23:54:50.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040563399999999999 2024-08-30T23:54:50.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:50 smithi106 ceph-mon[16603]: from='client.18645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:50.613 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:50 smithi173 ceph-mon[20554]: from='client.18645 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:54:50.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515551+0000", 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28519461400000001 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515669+0000", 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117731 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515720+0000", 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1390000000000001e-05 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.614 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21778 8.8 e853/840 ondisk, result = 0)", 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.327738637000003, 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28535590100000002, 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230376+0000", 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2480000000000001e-06 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515346+0000", 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28496944699999999 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515361+0000", 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5573000000000002e-05 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515505+0000", 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515533+0000", 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7696e-05 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515728+0000", 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019533300000000001 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21786 8.9 e853/840 8:9a7f97a1:::benchmark_data_smithi173_143816_object21785:head v 853'1317, mlcod=852'1314)", 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.178375350000003, 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22536509800000001, 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379790+0000", 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3526000000000002e-05 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604264+0000", 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.224474275 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604267+0000", 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.779e-06 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604309+0000", 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2253999999999999e-05 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604325+0000", 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5483999999999999e-05 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.605072+0000", 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074713500000000001 2024-08-30T23:54:50.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.605101+0000", 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9158000000000001e-05 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21807 8.4 8:2833fcc7:::benchmark_data_smithi173_143816_object21806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 39.955264262999997, 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34692570299999997, 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.623 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21807 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602848+0000", 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009e-06 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602878+0000", 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0278999999999999e-05 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602891+0000", 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2371000000000001e-05 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602897+0000", 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602941+0000", 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4617000000000003e-05 2024-08-30T23:54:50.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603031+0000", 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0060000000000002e-05 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603115+0000", 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3387000000000006e-05 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603890+0000", 2024-08-30T23:54:50.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077577699999999996 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604297+0000", 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040658900000000001 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949697+0000", 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34539998399999999 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949739+0000", 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1758000000000002e-05 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949773+0000", 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3970999999999997e-05 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21807 8.4 e853/840 ondisk, result = 0)", 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:54:50.629 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 39.953880869999999, 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34559199699999998, 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604231+0000", 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7599999999999999e-07 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604232+0000", 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.105e-06 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949585+0000", 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34535339199999998 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949593+0000", 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.644e-06 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949684+0000", 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1680000000000003e-05 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949693+0000", 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.354e-06 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949822+0000", 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129646 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:22746 8.4 8:28254eaf:::benchmark_data_smithi173_143816_object22745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:12.618594+0000", 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.939517686000002, 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21147284299999999, 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.634 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22746 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618594+0000", 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618594+0000", 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618595+0000", 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.145e-06 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618653+0000", 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8257999999999997e-05 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618669+0000", 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5829999999999999e-05 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618675+0000", 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5270000000000004e-06 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618721+0000", 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5652000000000003e-05 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618805+0000", 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4043999999999997e-05 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.618903+0000", 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7726999999999995e-05 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.619771+0000", 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00086872399999999997 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.625478+0000", 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057062850000000002 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.829987+0000", 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20450881900000001 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.830038+0000", 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1165999999999998e-05 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.830066+0000", 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8666e-05 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:22751 8.4 8:2bf995d9:::benchmark_data_smithi173_143816_object22750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:12.621227+0000", 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.936884698, 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20893784000000001, 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.640 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 22751 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621227+0000", 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621227+0000", 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621228+0000", 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0130000000000001e-06 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621285+0000", 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7655999999999998e-05 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621295+0000", 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6220000000000004e-06 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621302+0000", 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9310000000000002e-06 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621358+0000", 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6019e-05 2024-08-30T23:54:50.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621481+0000", 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012337599999999999 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.621618+0000", 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013662400000000001 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.622270+0000", 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00065189800000000004 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.628932+0000", 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066622030000000002 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.830097+0000", 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20116551499999999 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.830135+0000", 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7951000000000003e-05 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.830164+0000", 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9031999999999999e-05 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:27884 8.8 8:12ca5787:::benchmark_data_smithi173_143816_object27883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:54:50.646 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:16.662015+0000", 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.896096651000001, 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21797259099999999, 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27884 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662015+0000", 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662015+0000", 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662016+0000", 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7150000000000001e-06 2024-08-30T23:54:50.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662059+0000", 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3142999999999997e-05 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662073+0000", 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3383999999999999e-05 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662081+0000", 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5229999999999995e-06 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662132+0000", 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0639000000000001e-05 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662250+0000", 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118465 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.662335+0000", 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4259999999999996e-05 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.663397+0000", 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001062255 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.733214+0000", 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.069817329999999997 2024-08-30T23:54:50.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.879925+0000", 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14671063200000001 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.879959+0000", 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4495000000000001e-05 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.879987+0000", 2024-08-30T23:54:50.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.775e-05 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:37810 8.8 8:1435bca5:::benchmark_data_smithi173_143816_object37809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:27.430041+0000", 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.128069923999998, 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22259082999999999, 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:54:50.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 37810 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430041+0000", 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430041+0000", 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430044+0000", 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8399999999999999e-06 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430093+0000", 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8578999999999997e-05 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430122+0000", 2024-08-30T23:54:50.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9166000000000002e-05 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430135+0000", 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3162999999999999e-05 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430302+0000", 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016710999999999999 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430449+0000", 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146382 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430534+0000", 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5075999999999997e-05 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.431643+0000", 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011095969999999999 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.432150+0000", 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050662799999999996 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652558+0000", 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.220408417 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652599+0000", 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0475999999999997e-05 2024-08-30T23:54:50.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652632+0000", 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3395999999999997e-05 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:37810 8.8 e864/840 ondisk, result = 0)", 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:27.432047+0000", 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.126064168999999, 2024-08-30T23:54:50.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.220590601, 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.432047+0000", 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.432047+0000", 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:54:50.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.432048+0000", 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.13e-07 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.432050+0000", 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.898e-06 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652493+0000", 2024-08-30T23:54:50.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22044219200000001 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652497+0000", 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.459e-06 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652548+0000", 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1280999999999999e-05 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:54:50.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652554+0000", 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8209999999999998e-06 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652638+0000", 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3436999999999994e-05 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:54:50.663 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:54:50.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:50.668 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":1941,"stamp":"2024-08-30T23:54:48.233171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3670934048,"num_objects":56007,"num_object_clones":0,"num_object_copies":168021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56005,"num_write_kb":3584320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6909,"num_bytes_recovered":453312512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57937,"ondisk_log_size":57937,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":17809788,"kb_used_data":10871928,"kb_used_omap":109,"kb_used_meta":6937554,"kb_avail":1107132036,"statfs":{"total":1151940427776,"available":1133703204864,"internally_reserved":0,"allocated":11132854272,"data_stored":11125161900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":7104055960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":637796352,"num_objects":9732,"num_object_clones":0,"num_object_copies":29196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9732,"num_write_kb":622848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.025487"},"pg_stats":[{"pgid":"1.13","version":"23'92","reported_seq":46,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.216550+0000","last_change":"2024-08-30T23:54:10.991736+0000","last_active":"2024-08-30T23:54:40.216550+0000","last_peered":"2024-08-30T23:54:40.216550+0000","last_clean":"2024-08-30T23:54:40.216550+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:54:40.216550+0000","last_undegraded":"2024-08-30T23:54:40.216550+0000","last_fullsized":"2024-08-30T23:54:40.216550+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.112421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.388213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":187,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.210232+0000","last_change":"2024-08-30T23:53:38.104290+0000","last_active":"2024-08-30T23:54:40.210232+0000","last_peered":"2024-08-30T23:54:40.210232+0000","last_clean":"2024-08-30T23:54:40.210232+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:54:40.210232+0000","last_undegraded":"2024-08-30T23:54:40.210232+0000","last_fullsized":"2024-08-30T23:54:40.210232+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:38.104244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:38.104244+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:45.249002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":808,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.213287+0000","last_change":"2024-08-30T23:53:47.855440+0000","last_active":"2024-08-30T23:54:40.213287+0000","last_peered":"2024-08-30T23:54:40.213287+0000","last_clean":"2024-08-30T23:54:40.213287+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:54:40.213287+0000","last_undegraded":"2024-08-30T23:54:40.213287+0000","last_fullsized":"2024-08-30T23:54:40.213287+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:47.855365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:47.855365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:54.334601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":187,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.210349+0000","last_change":"2024-08-30T23:53:39.095429+0000","last_active":"2024-08-30T23:54:40.210349+0000","last_peered":"2024-08-30T23:54:40.210349+0000","last_clean":"2024-08-30T23:54:40.210349+0000","last_became_active":"2024-08-30T23:51:42.948334+0000","last_became_peered":"2024-08-30T23:51:42.948334+0000","last_unstale":"2024-08-30T23:54:40.210349+0000","last_undegraded":"2024-08-30T23:54:40.210349+0000","last_fullsized":"2024-08-30T23:54:40.210349+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:39.095359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:39.095359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.506854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"871'3568","reported_seq":3640,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.917101+0000","last_change":"2024-08-30T23:53:49.706683+0000","last_active":"2024-08-30T23:54:46.917101+0000","last_peered":"2024-08-30T23:54:46.917101+0000","last_clean":"2024-08-30T23:54:46.917101+0000","last_became_active":"2024-08-30T23:53:49.706310+0000","last_became_peered":"2024-08-30T23:53:49.706310+0000","last_unstale":"2024-08-30T23:54:46.917101+0000","last_undegraded":"2024-08-30T23:54:46.917101+0000","last_fullsized":"2024-08-30T23:54:46.917101+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.016839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233832448,"num_objects":3568,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":228352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":603,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-08-30T23:54:47.224278+0000","last_change":"2024-08-30T23:54:11.556855+0000","last_active":"2024-08-30T23:54:47.224278+0000","last_peered":"2024-08-30T23:54:47.224278+0000","last_clean":"2024-08-30T23:54:47.224278+0000","last_became_active":"2024-08-30T23:54:10.990434+0000","last_became_peered":"2024-08-30T23:54:10.990434+0000","last_unstale":"2024-08-30T23:54:47.224278+0000","last_undegraded":"2024-08-30T23:54:47.224278+0000","last_fullsized":"2024-08-30T23:54:47.224278+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:11.556806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:11.556806+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:27.495366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"871'3595","reported_seq":6412,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:45.993977+0000","last_change":"2024-08-30T23:54:29.061695+0000","last_active":"2024-08-30T23:54:45.993977+0000","last_peered":"2024-08-30T23:54:45.993977+0000","last_clean":"2024-08-30T23:54:45.993977+0000","last_became_active":"2024-08-30T23:54:29.059713+0000","last_became_peered":"2024-08-30T23:54:29.059713+0000","last_unstale":"2024-08-30T23:54:45.993977+0000","last_undegraded":"2024-08-30T23:54:45.993977+0000","last_fullsized":"2024-08-30T23:54:45.993977+0000","mapping_epoch":864,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":865,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3595,"log_dups_size":0,"ondisk_log_size":3595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:15.589492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235601920,"num_objects":3595,"num_object_clones":0,"num_object_copies":10785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3595,"num_write_kb":230080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":89522176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":54,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.216470+0000","last_change":"2024-08-30T23:54:11.516693+0000","last_active":"2024-08-30T23:54:40.216470+0000","last_peered":"2024-08-30T23:54:40.216470+0000","last_clean":"2024-08-30T23:54:40.216470+0000","last_became_active":"2024-08-30T23:54:10.989215+0000","last_became_peered":"2024-08-30T23:54:10.989215+0000","last_unstale":"2024-08-30T23:54:40.216470+0000","last_undegraded":"2024-08-30T23:54:40.216470+0000","last_fullsized":"2024-08-30T23:54:40.216470+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:11.516643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:11.516643+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:26.816354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"871'3359","reported_seq":3438,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:44.300197+0000","last_change":"2024-08-30T23:54:34.130863+0000","last_active":"2024-08-30T23:54:44.300197+0000","last_peered":"2024-08-30T23:54:44.300197+0000","last_clean":"2024-08-30T23:54:44.300197+0000","last_became_active":"2024-08-30T23:54:34.127322+0000","last_became_peered":"2024-08-30T23:54:34.127322+0000","last_unstale":"2024-08-30T23:54:44.300197+0000","last_undegraded":"2024-08-30T23:54:44.300197+0000","last_fullsized":"2024-08-30T23:54:44.300197+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3359,"log_dups_size":0,"ondisk_log_size":3359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.332392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220135424,"num_objects":3359,"num_object_clones":0,"num_object_copies":10077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3359,"num_write_kb":214976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":386,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.214124+0000","last_change":"2024-08-30T23:53:51.042769+0000","last_active":"2024-08-30T23:54:40.214124+0000","last_peered":"2024-08-30T23:54:40.214124+0000","last_clean":"2024-08-30T23:54:40.214124+0000","last_became_active":"2024-08-30T23:52:07.960637+0000","last_became_peered":"2024-08-30T23:52:07.960637+0000","last_unstale":"2024-08-30T23:54:40.214124+0000","last_undegraded":"2024-08-30T23:54:40.214124+0000","last_fullsized":"2024-08-30T23:54:40.214124+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.042710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.042710+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.950115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"871'3411","reported_seq":6236,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.030469+0000","last_change":"2024-08-30T23:54:35.999956+0000","last_active":"2024-08-30T23:54:46.030469+0000","last_peered":"2024-08-30T23:54:46.030469+0000","last_clean":"2024-08-30T23:54:46.030469+0000","last_became_active":"2024-08-30T23:54:35.995990+0000","last_became_peered":"2024-08-30T23:54:35.995990+0000","last_unstale":"2024-08-30T23:54:46.030469+0000","last_undegraded":"2024-08-30T23:54:46.030469+0000","last_fullsized":"2024-08-30T23:54:46.030469+0000","mapping_epoch":868,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:05.532380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223543296,"num_objects":3411,"num_object_clones":0,"num_object_copies":10233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":218304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1369,"num_bytes_recovered":89718784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":359,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.214140+0000","last_change":"2024-08-30T23:53:43.091100+0000","last_active":"2024-08-30T23:54:40.214140+0000","last_peered":"2024-08-30T23:54:40.214140+0000","last_clean":"2024-08-30T23:54:40.214140+0000","last_became_active":"2024-08-30T23:48:30.883878+0000","last_became_peered":"2024-08-30T23:48:30.883878+0000","last_unstale":"2024-08-30T23:54:40.214140+0000","last_undegraded":"2024-08-30T23:54:40.214140+0000","last_fullsized":"2024-08-30T23:54:40.214140+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.091048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.091048+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:10.926425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"871'3550","reported_seq":3623,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.678069+0000","last_change":"2024-08-30T23:53:49.706502+0000","last_active":"2024-08-30T23:54:46.678069+0000","last_peered":"2024-08-30T23:54:46.678069+0000","last_clean":"2024-08-30T23:54:46.678069+0000","last_became_active":"2024-08-30T23:53:49.705978+0000","last_became_peered":"2024-08-30T23:53:49.705978+0000","last_unstale":"2024-08-30T23:54:46.678069+0000","last_undegraded":"2024-08-30T23:54:46.678069+0000","last_fullsized":"2024-08-30T23:54:46.678069+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3550,"log_dups_size":0,"ondisk_log_size":3550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.097093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232652800,"num_objects":3550,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3550,"num_write_kb":227200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1001,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-08-30T23:54:47.224461+0000","last_change":"2024-08-30T23:53:51.507308+0000","last_active":"2024-08-30T23:54:47.224461+0000","last_peered":"2024-08-30T23:54:47.224461+0000","last_clean":"2024-08-30T23:54:47.224461+0000","last_became_active":"2024-08-30T23:48:30.880433+0000","last_became_peered":"2024-08-30T23:48:30.880433+0000","last_unstale":"2024-08-30T23:54:47.224461+0000","last_undegraded":"2024-08-30T23:54:47.224461+0000","last_fullsized":"2024-08-30T23:54:47.224461+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:51.507248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:51.507248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.951121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"871'3381","reported_seq":3453,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:44.721346+0000","last_change":"2024-08-30T23:53:49.708022+0000","last_active":"2024-08-30T23:54:44.721346+0000","last_peered":"2024-08-30T23:54:44.721346+0000","last_clean":"2024-08-30T23:54:44.721346+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:54:44.721346+0000","last_undegraded":"2024-08-30T23:54:44.721346+0000","last_fullsized":"2024-08-30T23:54:44.721346+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3381,"log_dups_size":0,"ondisk_log_size":3381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.990219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221577216,"num_objects":3381,"num_object_clones":0,"num_object_copies":10143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3381,"num_write_kb":216384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":351,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.214547+0000","last_change":"2024-08-30T23:53:41.104980+0000","last_active":"2024-08-30T23:54:40.214547+0000","last_peered":"2024-08-30T23:54:40.214547+0000","last_clean":"2024-08-30T23:54:40.214547+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:54:40.214547+0000","last_undegraded":"2024-08-30T23:54:40.214547+0000","last_fullsized":"2024-08-30T23:54:40.214547+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.104925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.104925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:09.944567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"871'3529","reported_seq":3601,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:45.629580+0000","last_change":"2024-08-30T23:53:49.671937+0000","last_active":"2024-08-30T23:54:45.629580+0000","last_peered":"2024-08-30T23:54:45.629580+0000","last_clean":"2024-08-30T23:54:45.629580+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:54:45.629580+0000","last_undegraded":"2024-08-30T23:54:45.629580+0000","last_fullsized":"2024-08-30T23:54:45.629580+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:14.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231276544,"num_objects":3529,"num_object_clones":0,"num_object_copies":10587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3529,"num_write_kb":225856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1841,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.208810+0000","last_change":"2024-08-30T23:53:50.999427+0000","last_active":"2024-08-30T23:54:40.208810+0000","last_peered":"2024-08-30T23:54:40.208810+0000","last_clean":"2024-08-30T23:54:40.208810+0000","last_became_active":"2024-08-30T23:52:54.016899+0000","last_became_peered":"2024-08-30T23:52:54.016899+0000","last_unstale":"2024-08-30T23:54:40.208810+0000","last_undegraded":"2024-08-30T23:54:40.208810+0000","last_fullsized":"2024-08-30T23:54:40.208810+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:50.999362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:50.999362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.278203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"871'3468","reported_seq":3537,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:45.625740+0000","last_change":"2024-08-30T23:53:49.709301+0000","last_active":"2024-08-30T23:54:45.625740+0000","last_peered":"2024-08-30T23:54:45.625740+0000","last_clean":"2024-08-30T23:54:45.625740+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:54:45.625740+0000","last_undegraded":"2024-08-30T23:54:45.625740+0000","last_fullsized":"2024-08-30T23:54:45.625740+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3468,"log_dups_size":0,"ondisk_log_size":3468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:17.536121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227278848,"num_objects":3468,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":221952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":172,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-08-30T23:54:47.224388+0000","last_change":"2024-08-30T23:54:18.583660+0000","last_active":"2024-08-30T23:54:47.224388+0000","last_peered":"2024-08-30T23:54:47.224388+0000","last_clean":"2024-08-30T23:54:47.224388+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:54:47.224388+0000","last_undegraded":"2024-08-30T23:54:47.224388+0000","last_fullsized":"2024-08-30T23:54:47.224388+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:18.583610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:18.583610+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:22.571284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"871'3612","reported_seq":3683,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.928358+0000","last_change":"2024-08-30T23:53:49.670752+0000","last_active":"2024-08-30T23:54:46.928358+0000","last_peered":"2024-08-30T23:54:46.928358+0000","last_clean":"2024-08-30T23:54:46.928358+0000","last_became_active":"2024-08-30T23:53:49.670369+0000","last_became_peered":"2024-08-30T23:53:49.670369+0000","last_unstale":"2024-08-30T23:54:46.928358+0000","last_undegraded":"2024-08-30T23:54:46.928358+0000","last_fullsized":"2024-08-30T23:54:46.928358+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:00.808262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236716032,"num_objects":3612,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3612,"num_write_kb":231168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":816,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.212204+0000","last_change":"2024-08-30T23:53:49.354762+0000","last_active":"2024-08-30T23:54:40.212204+0000","last_peered":"2024-08-30T23:54:40.212204+0000","last_clean":"2024-08-30T23:54:40.212204+0000","last_became_active":"2024-08-30T23:50:25.926833+0000","last_became_peered":"2024-08-30T23:50:25.926833+0000","last_unstale":"2024-08-30T23:54:40.212204+0000","last_undegraded":"2024-08-30T23:54:40.212204+0000","last_fullsized":"2024-08-30T23:54:40.212204+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.354704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.354704+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.152954+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"871'3330","reported_seq":3403,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:44.713109+0000","last_change":"2024-08-30T23:53:49.707268+0000","last_active":"2024-08-30T23:54:44.713109+0000","last_peered":"2024-08-30T23:54:44.713109+0000","last_clean":"2024-08-30T23:54:44.713109+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:54:44.713109+0000","last_undegraded":"2024-08-30T23:54:44.713109+0000","last_fullsized":"2024-08-30T23:54:44.713109+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3330,"log_dups_size":0,"ondisk_log_size":3330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:01.387982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218234880,"num_objects":3330,"num_object_clones":0,"num_object_copies":9990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":213120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":806,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.208901+0000","last_change":"2024-08-30T23:53:41.049687+0000","last_active":"2024-08-30T23:54:40.208901+0000","last_peered":"2024-08-30T23:54:40.208901+0000","last_clean":"2024-08-30T23:54:40.208901+0000","last_became_active":"2024-08-30T23:52:54.016432+0000","last_became_peered":"2024-08-30T23:52:54.016432+0000","last_unstale":"2024-08-30T23:54:40.208901+0000","last_undegraded":"2024-08-30T23:54:40.208901+0000","last_fullsized":"2024-08-30T23:54:40.208901+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.049626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.049626+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.750692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"871'3650","reported_seq":6557,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.004669+0000","last_change":"2024-08-30T23:54:37.664630+0000","last_active":"2024-08-30T23:54:46.004669+0000","last_peered":"2024-08-30T23:54:46.004669+0000","last_clean":"2024-08-30T23:54:46.004669+0000","last_became_active":"2024-08-30T23:54:37.621282+0000","last_became_peered":"2024-08-30T23:54:37.621282+0000","last_unstale":"2024-08-30T23:54:46.004669+0000","last_undegraded":"2024-08-30T23:54:46.004669+0000","last_fullsized":"2024-08-30T23:54:46.004669+0000","mapping_epoch":869,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":870,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:11.002037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239206400,"num_objects":3650,"num_object_clones":0,"num_object_copies":10950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3650,"num_write_kb":233600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":92405760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":737,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.208939+0000","last_change":"2024-08-30T23:53:35.098110+0000","last_active":"2024-08-30T23:54:40.208939+0000","last_peered":"2024-08-30T23:54:40.208939+0000","last_clean":"2024-08-30T23:54:40.208939+0000","last_became_active":"2024-08-30T23:42:34.862929+0000","last_became_peered":"2024-08-30T23:42:34.862929+0000","last_unstale":"2024-08-30T23:54:40.208939+0000","last_undegraded":"2024-08-30T23:54:40.208939+0000","last_fullsized":"2024-08-30T23:54:40.208939+0000","mapping_epoch":535,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":536,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:35.098031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:35.098031+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:03.053662+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"871'3607","reported_seq":6524,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.032284+0000","last_change":"2024-08-30T23:54:25.133351+0000","last_active":"2024-08-30T23:54:46.032284+0000","last_peered":"2024-08-30T23:54:46.032284+0000","last_clean":"2024-08-30T23:54:46.032284+0000","last_became_active":"2024-08-30T23:54:25.106180+0000","last_became_peered":"2024-08-30T23:54:25.106180+0000","last_unstale":"2024-08-30T23:54:46.032284+0000","last_undegraded":"2024-08-30T23:54:46.032284+0000","last_fullsized":"2024-08-30T23:54:46.032284+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:06.077859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236388352,"num_objects":3607,"num_object_clones":0,"num_object_copies":10821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3607,"num_write_kb":230848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":92798976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1010,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-08-30T23:54:47.224449+0000","last_change":"2024-08-30T23:54:09.628811+0000","last_active":"2024-08-30T23:54:47.224449+0000","last_peered":"2024-08-30T23:54:47.224449+0000","last_clean":"2024-08-30T23:54:47.224449+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:54:47.224449+0000","last_undegraded":"2024-08-30T23:54:47.224449+0000","last_fullsized":"2024-08-30T23:54:47.224449+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:09.628739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:09.628739+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:16.854773+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"871'3495","reported_seq":3567,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.747978+0000","last_change":"2024-08-30T23:53:49.706361+0000","last_active":"2024-08-30T23:54:46.747978+0000","last_peered":"2024-08-30T23:54:46.747978+0000","last_clean":"2024-08-30T23:54:46.747978+0000","last_became_active":"2024-08-30T23:53:49.706070+0000","last_became_peered":"2024-08-30T23:53:49.706070+0000","last_unstale":"2024-08-30T23:54:46.747978+0000","last_undegraded":"2024-08-30T23:54:46.747978+0000","last_fullsized":"2024-08-30T23:54:46.747978+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:02.364402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229048320,"num_objects":3495,"num_object_clones":0,"num_object_copies":10485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":223680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":199,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.210285+0000","last_change":"2024-08-30T23:53:43.164672+0000","last_active":"2024-08-30T23:54:40.210285+0000","last_peered":"2024-08-30T23:54:40.210285+0000","last_clean":"2024-08-30T23:54:40.210285+0000","last_became_active":"2024-08-30T23:51:42.952967+0000","last_became_peered":"2024-08-30T23:51:42.952967+0000","last_unstale":"2024-08-30T23:54:40.210285+0000","last_undegraded":"2024-08-30T23:54:40.210285+0000","last_fullsized":"2024-08-30T23:54:40.210285+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:43.164609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:43.164609+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:56.951863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"871'3441","reported_seq":3513,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:43.711603+0000","last_change":"2024-08-30T23:53:49.706815+0000","last_active":"2024-08-30T23:54:43.711603+0000","last_peered":"2024-08-30T23:54:43.711603+0000","last_clean":"2024-08-30T23:54:43.711603+0000","last_became_active":"2024-08-30T23:53:49.706543+0000","last_became_peered":"2024-08-30T23:53:49.706543+0000","last_unstale":"2024-08-30T23:54:43.711603+0000","last_undegraded":"2024-08-30T23:54:43.711603+0000","last_fullsized":"2024-08-30T23:54:43.711603+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3441,"log_dups_size":0,"ondisk_log_size":3441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:49.291074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225509376,"num_objects":3441,"num_object_clones":0,"num_object_copies":10323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3441,"num_write_kb":220224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1822,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-08-30T23:54:47.225701+0000","last_change":"2024-08-30T23:53:42.303311+0000","last_active":"2024-08-30T23:54:47.225701+0000","last_peered":"2024-08-30T23:54:47.225701+0000","last_clean":"2024-08-30T23:54:47.225701+0000","last_became_active":"2024-08-30T23:39:16.716314+0000","last_became_peered":"2024-08-30T23:39:16.716314+0000","last_unstale":"2024-08-30T23:54:47.225701+0000","last_undegraded":"2024-08-30T23:54:47.225701+0000","last_fullsized":"2024-08-30T23:54:47.225701+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:42.303263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:42.303263+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:47.897994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"871'3451","reported_seq":6230,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.029290+0000","last_change":"2024-08-30T23:54:25.999482+0000","last_active":"2024-08-30T23:54:46.029290+0000","last_peered":"2024-08-30T23:54:46.029290+0000","last_clean":"2024-08-30T23:54:46.029290+0000","last_became_active":"2024-08-30T23:54:25.996662+0000","last_became_peered":"2024-08-30T23:54:25.996662+0000","last_unstale":"2024-08-30T23:54:46.029290+0000","last_undegraded":"2024-08-30T23:54:46.029290+0000","last_fullsized":"2024-08-30T23:54:46.029290+0000","mapping_epoch":862,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":863,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:11.653706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226164736,"num_objects":3451,"num_object_clones":0,"num_object_copies":10353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3451,"num_write_kb":220864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":88211456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":934,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.212552+0000","last_change":"2024-08-30T23:53:49.897094+0000","last_active":"2024-08-30T23:54:40.212552+0000","last_peered":"2024-08-30T23:54:40.212552+0000","last_clean":"2024-08-30T23:54:40.212552+0000","last_became_active":"2024-08-30T23:39:16.716002+0000","last_became_peered":"2024-08-30T23:39:16.716002+0000","last_unstale":"2024-08-30T23:54:40.212552+0000","last_undegraded":"2024-08-30T23:54:40.212552+0000","last_fullsized":"2024-08-30T23:54:40.212552+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:49.897038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:49.897038+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:51.963756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"871'3558","reported_seq":3630,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:46.939022+0000","last_change":"2024-08-30T23:53:49.706603+0000","last_active":"2024-08-30T23:54:46.939022+0000","last_peered":"2024-08-30T23:54:46.939022+0000","last_clean":"2024-08-30T23:54:46.939022+0000","last_became_active":"2024-08-30T23:53:49.706194+0000","last_became_peered":"2024-08-30T23:53:49.706194+0000","last_unstale":"2024-08-30T23:54:46.939022+0000","last_undegraded":"2024-08-30T23:54:46.939022+0000","last_fullsized":"2024-08-30T23:54:46.939022+0000","mapping_epoch":840,"log_start":"0'0","ondisk_log_start":"0'0","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:53:48.650807+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:04.835283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3558,"num_write_kb":227712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":46,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.216577+0000","last_change":"2024-08-30T23:54:10.988040+0000","last_active":"2024-08-30T23:54:40.216577+0000","last_peered":"2024-08-30T23:54:40.216577+0000","last_clean":"2024-08-30T23:54:40.216577+0000","last_became_active":"2024-08-30T23:54:10.987627+0000","last_became_peered":"2024-08-30T23:54:10.987627+0000","last_unstale":"2024-08-30T23:54:40.216577+0000","last_undegraded":"2024-08-30T23:54:40.216577+0000","last_fullsized":"2024-08-30T23:54:40.216577+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:53:41.078720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:53:41.078720+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:54:55.813319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":907,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-08-30T23:54:40.212683+0000","last_change":"2024-08-30T23:54:34.892672+0000","last_active":"2024-08-30T23:54:40.212683+0000","last_peered":"2024-08-30T23:54:40.212683+0000","last_clean":"2024-08-30T23:54:40.212683+0000","last_became_active":"2024-08-30T23:54:34.127104+0000","last_became_peered":"2024-08-30T23:54:34.127104+0000","last_unstale":"2024-08-30T23:54:40.212683+0000","last_undegraded":"2024-08-30T23:54:40.212683+0000","last_fullsized":"2024-08-30T23:54:40.212683+0000","mapping_epoch":866,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:55:50.478726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,7],"acting":[9,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3670343680,"num_objects":56005,"num_object_clones":0,"num_object_copies":168015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56005,"num_write_kb":3584320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6907,"num_bytes_recovered":452657152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11151671296,"data_stored":11151671296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56005,"ondisk_log_size":56005,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725656,"kb_used_data":1108616,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92019496,"statfs":{"total":95995035648,"available":94227963904,"internally_reserved":0,"allocated":1135222784,"data_stored":1134582641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":631830980},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449460,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2009048,"kb_used_data":1482248,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91736104,"statfs":{"total":95995035648,"available":93937770496,"internally_reserved":0,"allocated":1517821952,"data_stored":1517181809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":539425216},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711089,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1948440,"kb_used_data":1281416,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91796712,"statfs":{"total":95995035648,"available":93999833088,"internally_reserved":0,"allocated":1312169984,"data_stored":1311529841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":683014592},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972731,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1120408,"kb_used_data":656072,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92624744,"statfs":{"total":95995035648,"available":94847737856,"internally_reserved":0,"allocated":671817728,"data_stored":671177585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234368,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183256,"kb_used_data":475976,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92561896,"statfs":{"total":95995035648,"available":94783381504,"internally_reserved":0,"allocated":487399424,"data_stored":486759281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":724236746},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484444,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93260708,"statfs":{"total":95995035648,"available":95498964992,"internally_reserved":0,"allocated":2105344,"data_stored":1465201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775212,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1349472,"kb_used_data":1127824,"kb_used_omap":5,"kb_used_meta":221626,"kb_avail":92395680,"statfs":{"total":95995035648,"available":94613176320,"internally_reserved":0,"allocated":1154891776,"data_stored":1154248081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":226945098},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953873,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636056,"kb_used_data":991240,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92109096,"statfs":{"total":95995035648,"available":94319714304,"internally_reserved":0,"allocated":1015029760,"data_stored":1014389617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150098,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506656,"kb_used_data":709136,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92238496,"statfs":{"total":95995035648,"available":94452219904,"internally_reserved":0,"allocated":726155264,"data_stored":725511569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":816642503},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411736,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1660704,"kb_used_data":1051984,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92084448,"statfs":{"total":95995035648,"available":94294474752,"internally_reserved":0,"allocated":1077231616,"data_stored":1076587921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830553,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797144,"kb_used_data":1354120,"kb_used_omap":87,"kb_used_meta":442920,"kb_avail":91948008,"statfs":{"total":95995035648,"available":94154760192,"internally_reserved":0,"allocated":1386618880,"data_stored":1385978737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":453550239},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1388504,"kb_used_data":631240,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92356648,"statfs":{"total":95995035648,"available":94573207552,"internally_reserved":0,"allocated":646389760,"data_stored":645749617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":654114816,"data_stored":654114816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1401683968,"data_stored":1401683968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1097072640,"data_stored":1097072640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":660471808,"data_stored":660471808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1003028480,"data_stored":1003028480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1166737408,"data_stored":1166737408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":88211456,"data_stored":88211456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":488374272,"data_stored":488374272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":684261376,"data_stored":684261376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1324810240,"data_stored":1324810240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1421803520,"data_stored":1421803520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1161101312,"data_stored":1161101312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:54:50.670 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-30T23:54:50.670 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-08-30T23:54:50.711 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:50.734 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:50.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:50.857 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:50.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:50.983 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:50.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:51.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:51.110 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:51.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:51.235 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:51.337 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:51.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:51.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 16 62486 62470 69.699 58.1875 0.0362971 0.0143395 2024-08-30T23:54:51.462 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:51.483 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:51.530 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:51 smithi136 ceph-mon[20207]: pgmap v1942: 37 pgs: 37 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-08-30T23:54:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:51 smithi173 ceph-mon[20554]: pgmap v1942: 37 pgs: 37 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-08-30T23:54:51.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:51 smithi106 ceph-mon[16603]: pgmap v1942: 37 pgs: 37 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s; 8.0 KiB/s, 0 objects/s recovering 2024-08-30T23:54:51.607 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:51.710 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:51.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:51.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:51.842 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:54:51.866 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:51.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:51.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:51.997 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:51.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:52.081 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:54:52.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:52.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:52.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:52.249 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:52.351 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:52.374 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:52.422 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 15 63688 63673 69.7951 75.1875 0.0132601 0.0143217 2024-08-30T23:54:52.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:52.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:52.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:52.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:52.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:52.803 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:52.905 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:52.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:52.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:53.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:53.063 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:53.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:53.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:53.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:53.316 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:53.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:53.421 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 64777 64761 69.7641 68 0.0218618 0.0143275 2024-08-30T23:54:53.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:53.527 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:53.530 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:53.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:53 smithi136 ceph-mon[20207]: pgmap v1943: 37 pgs: 37 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:54:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:54:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:54:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:53 smithi136 ceph-mon[20207]: osdmap e874: 12 total, 12 up, 12 in 2024-08-30T23:54:53.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:53.560 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:53.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:53 smithi173 ceph-mon[20554]: pgmap v1943: 37 pgs: 37 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:54:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:54:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:54:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:53 smithi173 ceph-mon[20554]: osdmap e874: 12 total, 12 up, 12 in 2024-08-30T23:54:53.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:53 smithi106 ceph-mon[16603]: pgmap v1943: 37 pgs: 37 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-08-30T23:54:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:54:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:54:53.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:53 smithi106 ceph-mon[16603]: osdmap e874: 12 total, 12 up, 12 in 2024-08-30T23:54:53.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:53.715 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:53.817 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.841 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:53.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:53.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:53.965 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:53.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:54.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:54.093 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:54.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:54.220 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:54.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:54.256 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:54:54.322 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.353 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:54.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:54.422 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 65843 65827 69.7108 66.625 0.00609606 0.0143385 2024-08-30T23:54:54.439 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3474612958' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:54.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:54.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:54.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3474612958' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:54.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3474612958' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:54:54.588 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:54.610 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:54.712 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:54.737 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:54.815 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:54:54.815 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:54:54.815 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:54:54.816 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:54:54.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:54.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:54.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:54.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:54.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:55.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:55.124 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:55.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:55.251 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:55.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:54:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.385 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:55.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:54:55.422 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:54:55.422652+0000 min lat: 0.0010542 max lat: 2.02823 avg lat: 0.0143449 2024-08-30T23:54:55.422 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:54:55.422 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 66931 66915 69.6822 68 0.00650941 0.0143449 2024-08-30T23:54:55.487 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:55.516 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:55 smithi136 ceph-mon[20207]: pgmap v1945: 37 pgs: 37 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 894 op/s 2024-08-30T23:54:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:55 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3474612958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:55 smithi136 ceph-mon[20207]: osdmap e875: 12 total, 12 up, 12 in 2024-08-30T23:54:55.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:55 smithi106 ceph-mon[16603]: pgmap v1945: 37 pgs: 37 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 894 op/s 2024-08-30T23:54:55.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:55 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:55.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3474612958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:55.562 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:55 smithi106 ceph-mon[16603]: osdmap e875: 12 total, 12 up, 12 in 2024-08-30T23:54:55.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:55 smithi173 ceph-mon[20554]: pgmap v1945: 37 pgs: 37 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 894 op/s 2024-08-30T23:54:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:55 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:54:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3474612958' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:54:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:55 smithi173 ceph-mon[20554]: osdmap e875: 12 total, 12 up, 12 in 2024-08-30T23:54:55.588 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:54:55.588 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:54:55.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:55.621 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:54:55.621 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:54:55.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:55.656 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:54:55.657 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:54:55.683 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:54:55.683 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:54:55.710 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:54:55.710 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:54:55.735 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:54:55.735 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:54:55.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:55.767 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:54:55.767 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:54:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:55.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:55.790 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:54:55.790 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:54:55.816 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:54:55.816 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:54:55.846 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:54:55.847 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:54:55.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:55.882 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:54:55.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:54:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:55.908 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:55.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:55.914 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:54:55.914 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:54:56.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:56.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:56.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:56.181 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:56.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:56.312 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:56.414 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:56.423 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 68264 68248 69.904 83.3125 0.0114448 0.0143003 2024-08-30T23:54:56.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/453489751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:56.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:56.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/453489751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/453489751' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:54:56.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:56.696 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:56.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:56.830 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:56.932 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:56.962 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:56.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:57.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:57.097 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:57.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.229 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:57.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:57.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:57.361 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:57.423 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 15 69468 69453 69.9912 75.3125 0.0108414 0.0142821 2024-08-30T23:54:57.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:54:57.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:54:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:57 smithi136 ceph-mon[20207]: pgmap v1947: 37 pgs: 37 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-30T23:54:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:57 smithi173 ceph-mon[20554]: pgmap v1947: 37 pgs: 37 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-30T23:54:57.597 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:57.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:57 smithi106 ceph-mon[16603]: pgmap v1947: 37 pgs: 37 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-08-30T23:54:57.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:57.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:57.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:57.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:54:57.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:57.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:57.886 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:54:57.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:54:58.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:54:58.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:58.115 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:54:58.142 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:58.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:54:58.264 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:54:58.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:58.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:58.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:58.423 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 70475 70459 69.8781 62.875 0.0030656 0.0143013 2024-08-30T23:54:58.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:58.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:54:58.525 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:58.627 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:58.658 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:54:58.658 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:54:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.659 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:58.659 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:58.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:54:58.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:54:58.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:54:58.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:58.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:54:59.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:54:59.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:54:59.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:54:59.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:59.167 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:59.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:59.273 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pg_num to 31 2024-08-30T23:54:59.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:59 smithi106 ceph-mon[16603]: pgmap v1948: 37 pgs: 37 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-30T23:54:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:54:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3818337572' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-30T23:54:59.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:54:59.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.419 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:54:59.423 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 71502 71486 69.7892 64.1875 0.0189824 0.0143216 2024-08-30T23:54:59.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:54:59.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:54:59.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:59 smithi136 ceph-mon[20207]: pgmap v1948: 37 pgs: 37 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-30T23:54:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:54:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3818337572' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-30T23:54:59.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:54:59.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:59 smithi173 ceph-mon[20554]: pgmap v1948: 37 pgs: 37 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-30T23:54:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:54:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3818337572' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-30T23:54:59.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:54:59.678 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:54:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:54:59.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:54:59.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:54:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:54:59.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:54:59.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:54:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:54:59.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:00.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:00.064 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:00.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:00.105 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:00.159 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:00.165 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:00.194 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:00.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:00.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3818337572' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-30T23:55:00.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:00 smithi106 ceph-mon[16603]: osdmap e876: 12 total, 12 up, 12 in 2024-08-30T23:55:00.317 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-08-30T23:55:00.320 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:00.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:00.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 72769 72753 69.9336 79.1875 0.0289599 0.0142928 2024-08-30T23:55:00.444 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3818337572' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-30T23:55:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:00 smithi136 ceph-mon[20207]: osdmap e876: 12 total, 12 up, 12 in 2024-08-30T23:55:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-08-30T23:55:00.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:00.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3818337572' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-30T23:55:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:00 smithi173 ceph-mon[20554]: osdmap e876: 12 total, 12 up, 12 in 2024-08-30T23:55:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-08-30T23:55:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:00.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:00.704 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:00.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:00.806 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:00.835 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:00.837 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:00.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:00.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:00.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:01.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:01.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:01.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:01.210 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:01.280 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:55:01.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:01.338 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:01.423 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 16 73708 73692 69.7632 58.6875 0.0061356 0.0143192 2024-08-30T23:55:01.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:01.462 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:01.564 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:01 smithi173 ceph-mon[20554]: pgmap v1950: 37 pgs: 37 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s 2024-08-30T23:55:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:01 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-30T23:55:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:01 smithi173 ceph-mon[20554]: osdmap e877: 12 total, 12 up, 12 in 2024-08-30T23:55:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2895520217' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:01.587 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:01 smithi106 ceph-mon[16603]: pgmap v1950: 37 pgs: 37 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s 2024-08-30T23:55:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:01 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-30T23:55:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:01 smithi106 ceph-mon[16603]: osdmap e877: 12 total, 12 up, 12 in 2024-08-30T23:55:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2895520217' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:01.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:01.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:01.773 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:01.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:01 smithi136 ceph-mon[20207]: pgmap v1950: 37 pgs: 37 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 927 op/s 2024-08-30T23:55:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:01 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-30T23:55:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:01 smithi136 ceph-mon[20207]: osdmap e877: 12 total, 12 up, 12 in 2024-08-30T23:55:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2895520217' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:01.795 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:01.796 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:01.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:01.819 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:01.820 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:01.836 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:01.842 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:01.842 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:01.869 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:01.869 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:01.896 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:01.897 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:01.926 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:01.926 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:01.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:01.952 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:01.952 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:01.963 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:01.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:01.979 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:01.980 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:02.005 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:02.005 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:02.029 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:02.029 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:02.056 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:02.056 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:02.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.096 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:02.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:02.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:02.229 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:02.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:02.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:02.423 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 74746 74730 69.6902 64.875 0.016062 0.0143421 2024-08-30T23:55:02.463 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:02.488 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2895520217' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: osdmap e878: 12 total, 12 up, 12 in 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:55:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:55:02.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:55:02.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:55:02.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3660799367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:02.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-08-30T23:55:02.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:02 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:55:02.590 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2895520217' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: osdmap e878: 12 total, 12 up, 12 in 2024-08-30T23:55:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:55:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3660799367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-08-30T23:55:02.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:02 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:55:02.613 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:02.716 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:02.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2895520217' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: osdmap e878: 12 total, 12 up, 12 in 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-30T23:55:02.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3660799367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-08-30T23:55:02.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:02 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.9 760/755 [PushOp(7:92ccacc7:::benchmark_data_smithi173_129296_object5107:head, version: 730'310, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:92ccacc7:::benchmark_data_smithi173_129296_object5107:head@730'310, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:55:02.846 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 278.78189190199998, 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93410173799999996, 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.847 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061775+0000", 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4950000000000001e-06 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061824+0000", 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9017000000000002e-05 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993785+0000", 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93196087299999997 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993806+0000", 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1152e-05 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993912+0000", 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106251 2024-08-30T23:55:02.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993928+0000", 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6387000000000001e-05 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.995875+0000", 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019465629999999999 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 765/762 [PushOp(7:6c4bd07a:::benchmark_data_smithi173_129296_object3500:head, version: 730'216, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6c4bd07a:::benchmark_data_smithi173_129296_object3500:head@730'216, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 272.37009877700001, 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62968416199999999, 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473567+0000", 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6799999999999999e-07 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473595+0000", 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7571000000000001e-05 2024-08-30T23:55:02.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101453+0000", 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62785843200000002 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101470+0000", 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7164e-05 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101579+0000", 2024-08-30T23:55:02.853 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109252 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101595+0000", 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449000000000001e-05 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.103250+0000", 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001655526 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.854 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 767/762 [PushOp(7:2ff57e78:::benchmark_data_smithi173_129296_object4738:head, version: 730'300, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2ff57e78:::benchmark_data_smithi173_129296_object4738:head@730'300, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 269.09903795700001, 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.95161098399999999, 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.855 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744628+0000", 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744662+0000", 2024-08-30T23:55:02.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3967000000000002e-05 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694267+0000", 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94960559300000003 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694286+0000", 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.844e-05 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694373+0000", 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7074e-05 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694383+0000", 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7389999999999994e-06 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.696238+0000", 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018554509999999999 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.858 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 767/762 [PushOp(7:603ba3a2:::benchmark_data_smithi173_129296_object7838:head, version: 730'501, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:603ba3a2:::benchmark_data_smithi173_129296_object7838:head@730'501, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 267.23968997600002, 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.99532398200000005, 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603976+0000", 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.071e-06 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.604007+0000", 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0408000000000001e-05 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597608+0000", 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.99360170000000003 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597625+0000", 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7184999999999999e-05 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597718+0000", 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2985999999999994e-05 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597729+0000", 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0227999999999999e-05 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.599299+0000", 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015704040000000001 2024-08-30T23:55:02.862 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 768/762 [PushOp(7:6615b48e:::benchmark_data_smithi173_129296_object8839:head, version: 730'559, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6615b48e:::benchmark_data_smithi173_129296_object8839:head@730'559, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 265.38212996200002, 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63695423500000004, 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461536+0000", 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1519999999999999e-06 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461566+0000", 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9569000000000001e-05 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097105+0000", 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63553892199999995 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097118+0000", 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2814e-05 2024-08-30T23:55:02.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097203+0000", 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5037999999999995e-05 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097213+0000", 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9930000000000007e-06 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.098489+0000", 2024-08-30T23:55:02.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001276747 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 768/762 [PushOp(7:2ce0b962:::benchmark_data_smithi173_129296_object7093:head, version: 730'443, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2ce0b962:::benchmark_data_smithi173_129296_object7093:head@730'443, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 264.79832380900001, 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.819240623, 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045342+0000", 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-07 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045405+0000", 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2865000000000002e-05 2024-08-30T23:55:02.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.861929+0000", 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81652422599999996 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.861952+0000", 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2768999999999998e-05 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.862068+0000", 2024-08-30T23:55:02.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115807 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.862081+0000", 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3247e-05 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.864582+0000", 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002501021 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.872 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 769/762 [PushOp(7:6b3af24c:::benchmark_data_smithi173_129296_object11749:head, version: 730'750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6b3af24c:::benchmark_data_smithi173_129296_object11749:head@730'750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 261.63613168500001, 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72094049199999999, 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207535+0000", 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6759999999999999e-06 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207575+0000", 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0224e-05 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927098+0000", 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71952308300000001 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927109+0000", 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0771000000000001e-05 2024-08-30T23:55:02.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927189+0000", 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0348999999999999e-05 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927199+0000", 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3209999999999994e-06 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.928474+0000", 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012750680000000001 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 769/762 [PushOp(7:668cd238:::benchmark_data_smithi173_129296_object13196:head, version: 730'849, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:668cd238:::benchmark_data_smithi173_129296_object13196:head@730'849, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 259.17247204500001, 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96579458500000004, 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.877 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671194+0000", 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.297e-06 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671225+0000", 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0312e-05 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.634913+0000", 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96368826900000004 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.634929+0000", 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6371999999999998e-05 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.635041+0000", 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111471 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.635052+0000", 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1195e-05 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.636988+0000", 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019356690000000001 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.881 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.f 771/761 [PushOp(7:fed971b1:::benchmark_data_smithi173_129296_object18024:head, version: 733'1120, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:fed971b1:::benchmark_data_smithi173_129296_object18024:head@733'1120, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 255.26620854199999, 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58954947000000002, 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577458+0000", 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577571+0000", 2024-08-30T23:55:02.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112908 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164827+0000", 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58725667999999998 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164846+0000", 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8417e-05 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164937+0000", 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0884000000000006e-05 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164946+0000", 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7380000000000006e-06 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.167006+0000", 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020595629999999999 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.885 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 771/762 [PushOp(7:241f6387:::benchmark_data_smithi173_129296_object16791:head, version: 731'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:241f6387:::benchmark_data_smithi173_129296_object16791:head@731'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 255.21752454700001, 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44453464500000001, 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626142+0000", 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3740000000000001e-06 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626194+0000", 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1793999999999999e-05 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068657+0000", 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44246287299999998 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068671+0000", 2024-08-30T23:55:02.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4783999999999999e-05 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068746+0000", 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4684999999999994e-05 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068756+0000", 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5750000000000007e-06 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.070675+0000", 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019195600000000001 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.f 787/761 [PushOp(7:f52af3f4:::benchmark_data_smithi173_129296_object52699:head, version: 743'3327, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f52af3f4:::benchmark_data_smithi173_129296_object52699:head@743'3327, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 200.54928744599999, 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.66126428199999998, 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.890 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294379+0000", 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294427+0000", 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8597999999999998e-05 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952418+0000", 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.65799002699999998 2024-08-30T23:55:02.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952437+0000", 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9136000000000001e-05 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952574+0000", 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013768 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952590+0000", 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5860000000000001e-05 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.955642+0000", 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003051701 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:142397 7.d 7:bed45d06:::benchmark_data_smithi173_129296_object55281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-30T23:55:02.894 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 145.216746097, 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48225094800000001, 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 142397 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:55:02.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626921+0000", 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4219999999999999e-06 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626930+0000", 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5599999999999994e-06 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626945+0000", 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5495999999999999e-05 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626955+0000", 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6190000000000006e-06 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.628796+0000", 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018411910000000001 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.628976+0000", 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000179777 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.629127+0000", 2024-08-30T23:55:02.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000150848 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.633350+0000", 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042234500000000001 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.635612+0000", 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022614369999999998 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107166+0000", 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.471554168 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107232+0000", 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5927999999999996e-05 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.109170+0000", 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001938052 2024-08-30T23:55:02.900 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27487.0:142397 7.d e810/800 ondisk, result = 0)", 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 145.208869247, 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.474397232, 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5099999999999999e-07 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634801+0000", 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0550000000000001e-06 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.102650+0000", 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.467849444 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.102656+0000", 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:55:02.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107133+0000", 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044769750000000002 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107145+0000", 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2656999999999999e-05 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.109193+0000", 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020475850000000002 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.9 e833,762 7:9b279d9f:::benchmark_data_smithi173_129296_object38265:head,794'7954)", 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 85.656745486999995, 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.880300788, 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.905 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.61e-07 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186925+0000", 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4730000000000002e-06 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067100+0000", 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.88017544199999997 2024-08-30T23:55:02.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067106+0000", 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9030000000000001e-06 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067157+0000", 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0556000000000002e-05 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067162+0000", 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.587e-06 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067220+0000", 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8066000000000003e-05 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e835,800 7:bbbda60d:::benchmark_data_smithi173_129296_object15076:head,778'6374)", 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 81.596851266000002, 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74636665599999996, 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246816+0000", 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.716e-06 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246828+0000", 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.27e-05 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992675+0000", 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74584704999999996 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992682+0000", 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.241e-06 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992757+0000", 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5322999999999997e-05 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.993180+0000", 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042362600000000003 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e836,800 7:b1ca300e:::benchmark_data_smithi173_129296_object19401:head,781'6635)", 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.640173758000003, 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45044139700000002, 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:55:02.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203492+0000", 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7000000000000004e-07 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203497+0000", 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8380000000000001e-06 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653230+0000", 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44973348699999999 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653238+0000", 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1329999999999999e-06 2024-08-30T23:55:02.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653353+0000", 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114485 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653933+0000", 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057978399999999996 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.5 e837,761 7:ae197e78:::benchmark_data_smithi173_129296_object20842:head,781'6740)", 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.180575563000005, 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45953577400000001, 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:55:02.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663090+0000", 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2699999999999998e-07 2024-08-30T23:55:02.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663096+0000", 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4179999999999996e-06 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122458+0000", 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.459362202 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122466+0000", 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7600000000000002e-06 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122530+0000", 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3917000000000005e-05 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122541+0000", 2024-08-30T23:55:02.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0824e-05 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122625+0000", 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4726000000000002e-05 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.5 e837,761 7:a846cf89:::benchmark_data_smithi173_129296_object20850:head,781'6741)", 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:55:02.922 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.174181451999999, 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46070137100000003, 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669485+0000", 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669491+0000", 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7590000000000003e-06 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130069+0000", 2024-08-30T23:55:02.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.460578184 2024-08-30T23:55:02.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130075+0000", 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130126+0000", 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1564000000000003e-05 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130133+0000", 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6730000000000003e-06 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130185+0000", 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1891000000000002e-05 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e837,800 7:b35fe8f8:::benchmark_data_smithi173_129296_object19689:head,781'6653)", 2024-08-30T23:55:02.927 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.087159068000005, 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42114565399999998, 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756507+0000", 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8899999999999999e-07 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756510+0000", 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.839e-06 2024-08-30T23:55:02.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177074+0000", 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42056344499999998 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177083+0000", 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.397e-06 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177141+0000", 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7311000000000001e-05 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177652+0000", 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00051107300000000002 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e837,800 7:b5944d16:::benchmark_data_smithi173_129296_object19693:head,781'6654)", 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:55:02.931 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.082566099000005, 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42161228299999998, 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:55:02.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.39e-07 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761102+0000", 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8049999999999998e-06 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182311+0000", 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42120909499999998 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182317+0000", 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0440000000000001e-06 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182359+0000", 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1857000000000003e-05 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182711+0000", 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035204300000000002 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.938 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.939 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.939 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:02.939 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:02.939 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:55:02.948 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:02.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:03.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:03.073 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:03.175 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.199 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:03.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:03.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:03.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 76041 76025 69.8554 80.9375 0.00872216 0.0143093 2024-08-30T23:55:03.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:03.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: pgmap v1953: 47 pgs: 1 peering, 8 unknown, 38 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-30T23:55:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-30T23:55:03.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: osdmap e879: 12 total, 12 up, 12 in 2024-08-30T23:55:03.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:55:03.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-30T23:55:03.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:03 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-30T23:55:03.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: pgmap v1953: 47 pgs: 1 peering, 8 unknown, 38 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-30T23:55:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:55:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: osdmap e879: 12 total, 12 up, 12 in 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-30T23:55:03.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:03 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-30T23:55:03.680 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:03.702 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: pgmap v1953: 47 pgs: 1 peering, 8 unknown, 38 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-30T23:55:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-30T23:55:03.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-30T23:55:03.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: osdmap e879: 12 total, 12 up, 12 in 2024-08-30T23:55:03.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:55:03.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-30T23:55:03.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:03 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-30T23:55:03.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:03.831 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:03.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:03.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:03.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:04.035 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:55:04.057 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:04.104 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:04.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:04.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:04.320 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:04 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:55:04.320 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:04 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:55:04.320 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:04 smithi106 ceph-mon[16603]: 8.1 scrub starts 2024-08-30T23:55:04.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:04 smithi106 ceph-mon[16603]: osdmap e880: 12 total, 12 up, 12 in 2024-08-30T23:55:04.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:04 smithi106 ceph-mon[16603]: 1.1a scrub starts 2024-08-30T23:55:04.321 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:04 smithi106 ceph-mon[16603]: 1.1a scrub ok 2024-08-30T23:55:04.334 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:04.361 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:04.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:55:04.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 77009 76993 69.7198 60.5 0.0152995 0.0143371 2024-08-30T23:55:04.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:04.491 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:04.492 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:04.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:04 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:55:04.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:04 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:55:04.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:04 smithi173 ceph-mon[20554]: 8.1 scrub starts 2024-08-30T23:55:04.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:04 smithi173 ceph-mon[20554]: osdmap e880: 12 total, 12 up, 12 in 2024-08-30T23:55:04.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:04 smithi173 ceph-mon[20554]: 1.1a scrub starts 2024-08-30T23:55:04.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:04 smithi173 ceph-mon[20554]: 1.1a scrub ok 2024-08-30T23:55:04.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:04.620 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:04.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:04.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:04.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:04 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:55:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:04 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:55:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:04 smithi136 ceph-mon[20207]: 8.1 scrub starts 2024-08-30T23:55:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:04 smithi136 ceph-mon[20207]: osdmap e880: 12 total, 12 up, 12 in 2024-08-30T23:55:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:04 smithi136 ceph-mon[20207]: 1.1a scrub starts 2024-08-30T23:55:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:04 smithi136 ceph-mon[20207]: 1.1a scrub ok 2024-08-30T23:55:04.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:04.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:04.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:04.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:04.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:05.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:77629 8.f e880/868 8:fb9d9445:::benchmark_data_smithi173_143816_object77628:head v 880'4746, mlcod=880'4743)", 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:55:05.004254+0000", 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.010138365, 2024-08-30T23:55:05.018 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010165482, 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.004254+0000", 2024-08-30T23:55:05.019 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.020 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:05.020 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.004254+0000", 2024-08-30T23:55:05.020 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.004255+0000", 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5600000000000005e-07 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.004289+0000", 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3961000000000002e-05 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009495+0000", 2024-08-30T23:55:05.021 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052064169999999996 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009503+0000", 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7470000000000005e-06 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009625+0000", 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000121754 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.022 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009644+0000", 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9055e-05 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:77633 8.f e880/868 8:f12d4ac8:::benchmark_data_smithi173_143816_object77632:head v 880'4747, mlcod=880'4744)", 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:55:05.009405+0000", 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0049877059999999997, 2024-08-30T23:55:05.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050689569999999998, 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009405+0000", 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009405+0000", 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009405+0000", 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7499999999999999e-07 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009464+0000", 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9061999999999998e-05 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009570+0000", 2024-08-30T23:55:05.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105034 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009575+0000", 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8510000000000004e-06 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.012185+0000", 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0026096079999999998 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.012205+0000", 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9598000000000001e-05 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.027 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:77636 8.d e880/864 8:bb37f9f3:::benchmark_data_smithi173_143816_object77635:head v 880'4970, mlcod=880'4967)", 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:55:05.009468+0000", 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0049237589999999998, 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.005054022, 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:05.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009468+0000", 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009468+0000", 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009469+0000", 2024-08-30T23:55:05.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1699999999999999e-07 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.009558+0000", 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8832999999999995e-05 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013068+0000", 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0035099710000000002 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013078+0000", 2024-08-30T23:55:05.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0248000000000001e-05 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013155+0000", 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7438e-05 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013180+0000", 2024-08-30T23:55:05.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4899999999999999e-05 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:77638 8.d e880/864)", 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:55:05.012912+0000", 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.001480397, 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00165391, 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:05.032 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.012912+0000", 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.012912+0000", 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:05.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.012913+0000", 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1850000000000001e-06 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013006+0000", 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3256000000000003e-05 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013112+0000", 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106069 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:05.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.013117+0000", 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9609999999999996e-06 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 4 2024-08-30T23:55:05.036 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:55:05.087 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:55:05.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:55:05.088 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-30T23:55:05.088 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-08-30T23:55:05.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:05.157 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:05.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:05.265 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:05.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:05.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: pgmap v1956: 47 pgs: 2 peering, 6 unknown, 39 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 66 MiB/s wr, 1.05k op/s; 2.9 KiB/s, 0 objects/s recovering 2024-08-30T23:55:05.394 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:05.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:05.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 78050 78034 69.6532 65.0625 0.062358 0.0143434 2024-08-30T23:55:05.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: pgmap v1956: 47 pgs: 2 peering, 6 unknown, 39 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 66 MiB/s wr, 1.05k op/s; 2.9 KiB/s, 0 objects/s recovering 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.19 scrub starts 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.19 scrub ok 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.18 scrub starts 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.18 scrub ok 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.1c scrub starts 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.1c scrub ok 2024-08-30T23:55:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.15 scrub starts 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.15 scrub ok 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.16 scrub starts 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.1e scrub starts 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 1.1e scrub ok 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: 8.1 scrub ok 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:05.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.542 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:05.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: pgmap v1956: 47 pgs: 2 peering, 6 unknown, 39 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 66 MiB/s wr, 1.05k op/s; 2.9 KiB/s, 0 objects/s recovering 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.19 scrub starts 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.19 scrub ok 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.18 scrub starts 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.18 scrub ok 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.1c scrub starts 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.1c scrub ok 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.15 scrub starts 2024-08-30T23:55:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.15 scrub ok 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.16 scrub starts 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.1e scrub starts 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 1.1e scrub ok 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: 8.1 scrub ok 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.19 scrub starts 2024-08-30T23:55:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.19 scrub ok 2024-08-30T23:55:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.18 scrub starts 2024-08-30T23:55:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.18 scrub ok 2024-08-30T23:55:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.1c scrub starts 2024-08-30T23:55:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.1c scrub ok 2024-08-30T23:55:05.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.15 scrub starts 2024-08-30T23:55:05.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.15 scrub ok 2024-08-30T23:55:05.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.16 scrub starts 2024-08-30T23:55:05.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-08-30T23:55:05.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.1e scrub starts 2024-08-30T23:55:05.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 1.1e scrub ok 2024-08-30T23:55:05.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: 8.1 scrub ok 2024-08-30T23:55:05.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:05.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:05.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:05.663 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:05.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:05.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:05.889 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:05.911 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:05.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:06.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:06.037 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:06.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:06.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.175 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:06.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:06.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:06.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:06.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 78881 78865 69.4037 51.9375 0.00633146 0.0144023 2024-08-30T23:55:06.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:06.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:55:06.528 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:06.554 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:55:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:55:06.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.1b scrub starts 2024-08-30T23:55:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.1b scrub ok 2024-08-30T23:55:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.16 scrub ok 2024-08-30T23:55:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: osdmap e881: 12 total, 12 up, 12 in 2024-08-30T23:55:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.1d scrub starts 2024-08-30T23:55:06.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:06 smithi106 ceph-mon[16603]: 1.1d scrub ok 2024-08-30T23:55:06.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:06.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.683 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:06.784 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:06.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.1b scrub starts 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.1b scrub ok 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.16 scrub ok 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: osdmap e881: 12 total, 12 up, 12 in 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.1d scrub starts 2024-08-30T23:55:06.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:06 smithi136 ceph-mon[20207]: 1.1d scrub ok 2024-08-30T23:55:06.808 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.1b scrub starts 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.1b scrub ok 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.16 scrub ok 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: osdmap e881: 12 total, 12 up, 12 in 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.1d scrub starts 2024-08-30T23:55:06.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:06 smithi173 ceph-mon[20554]: 1.1d scrub ok 2024-08-30T23:55:06.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:55:06.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:06.931 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:06.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:07.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:07.061 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:07.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:07.186 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:07.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:07.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:07.332 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: 8.0 scrub starts 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: pgmap v1958: 47 pgs: 2 peering, 2 unknown, 43 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 452 B/s rd, 52 MiB/s wr, 835 op/s 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: 1.17 scrub starts 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: 1.17 scrub ok 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: osdmap e882: 12 total, 12 up, 12 in 2024-08-30T23:55:07.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:07 smithi106 ceph-mon[16603]: 8.0 scrub ok 2024-08-30T23:55:07.387 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:07.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:07.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 79856 79840 69.2861 60.9375 0.0217961 0.0144279 2024-08-30T23:55:07.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:07.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:07.566 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:07.669 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:07.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: 8.0 scrub starts 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: pgmap v1958: 47 pgs: 2 peering, 2 unknown, 43 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 452 B/s rd, 52 MiB/s wr, 835 op/s 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: 1.17 scrub starts 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: 1.17 scrub ok 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: osdmap e882: 12 total, 12 up, 12 in 2024-08-30T23:55:07.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:07 smithi136 ceph-mon[20207]: 8.0 scrub ok 2024-08-30T23:55:07.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.819 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:07.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: 8.0 scrub starts 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: pgmap v1958: 47 pgs: 2 peering, 2 unknown, 43 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 452 B/s rd, 52 MiB/s wr, 835 op/s 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: 1.17 scrub starts 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: 1.17 scrub ok 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: osdmap e882: 12 total, 12 up, 12 in 2024-08-30T23:55:07.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:07 smithi173 ceph-mon[20554]: 8.0 scrub ok 2024-08-30T23:55:07.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:07.945 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:07.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:08.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:08.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:08.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:08.196 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:08.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:08.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:08.377 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:55:08.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:08.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 80883 80867 69.2162 64.1875 0.0136229 0.014442 2024-08-30T23:55:08.453 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: osdmap e883: 12 total, 12 up, 12 in 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2984697155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:08.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:08 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:08.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:08.579 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:08.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:08.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: osdmap e883: 12 total, 12 up, 12 in 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2984697155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:08.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:08 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:08.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: osdmap e883: 12 total, 12 up, 12 in 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2984697155' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:08.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:08 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:08.834 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:08.836 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:08.836 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:08.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:08.962 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:08.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:09.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:09.089 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:09.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:09.213 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:09.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:09.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:09.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:09.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 82086 82070 69.2968 75.1875 0.0115624 0.0144256 2024-08-30T23:55:09.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:09.466 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:09.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:09.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: pgmap v1961: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-08-30T23:55:09.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: 8.4 scrub starts 2024-08-30T23:55:09.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: 8.9 scrub starts 2024-08-30T23:55:09.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:55:09.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:09.595 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:09.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: osdmap e884: 12 total, 12 up, 12 in 2024-08-30T23:55:09.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/680979637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.600 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:09.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:09.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:09.733 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:09.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:09.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:09.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:09.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:09.770 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:09.771 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:09.772 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: pgmap v1961: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-08-30T23:55:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: 8.4 scrub starts 2024-08-30T23:55:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: 8.9 scrub starts 2024-08-30T23:55:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:55:09.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:09.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:09.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: osdmap e884: 12 total, 12 up, 12 in 2024-08-30T23:55:09.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/680979637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: pgmap v1961: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: 8.4 scrub starts 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: 8.9 scrub starts 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: osdmap e884: 12 total, 12 up, 12 in 2024-08-30T23:55:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/680979637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:09.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:09.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.863 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:09.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:09.990 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:09.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:10.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:10.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:10.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:10.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:10.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:10.364 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:10.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 83145 83129 69.2552 66.1875 0.0198263 0.0144333 2024-08-30T23:55:10.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:10.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:10.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:10.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:10.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:10.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:10 smithi136 ceph-mon[20207]: 8.9 scrub ok 2024-08-30T23:55:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:10 smithi136 ceph-mon[20207]: 8.4 scrub ok 2024-08-30T23:55:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:10 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:55:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:10 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:55:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:10 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:10.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:10 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:10 smithi173 ceph-mon[20554]: 8.9 scrub ok 2024-08-30T23:55:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:10 smithi173 ceph-mon[20554]: 8.4 scrub ok 2024-08-30T23:55:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:10 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:55:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:10 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:55:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:10 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:10 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:10.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:10 smithi106 ceph-mon[16603]: 8.9 scrub ok 2024-08-30T23:55:10.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:10 smithi106 ceph-mon[16603]: 8.4 scrub ok 2024-08-30T23:55:10.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:10 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:55:10.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:10 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:55:10.847 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:10 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:10.847 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:10 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:10.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:10.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:55:10.895 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:10.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:10.938 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:10.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:11.028 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:11.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:11.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:11.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:11.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:11.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:11.410 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:11.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 84089 84073 69.1203 59 0.0118658 0.0144613 2024-08-30T23:55:11.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:11.533 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:11.551 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:11.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:11.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:55:11.664 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:11.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:11 smithi106 ceph-mon[16603]: pgmap v1963: 47 pgs: 47 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:55:11.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:11 smithi106 ceph-mon[16603]: 8.8 scrub starts 2024-08-30T23:55:11.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:11 smithi106 ceph-mon[16603]: 8.5 scrub starts 2024-08-30T23:55:11.668 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:11 smithi106 ceph-mon[16603]: osdmap e885: 12 total, 12 up, 12 in 2024-08-30T23:55:11.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:11 smithi136 ceph-mon[20207]: pgmap v1963: 47 pgs: 47 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:55:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:11 smithi136 ceph-mon[20207]: 8.8 scrub starts 2024-08-30T23:55:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:11 smithi136 ceph-mon[20207]: 8.5 scrub starts 2024-08-30T23:55:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:11 smithi136 ceph-mon[20207]: osdmap e885: 12 total, 12 up, 12 in 2024-08-30T23:55:11.792 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:11 smithi173 ceph-mon[20554]: pgmap v1963: 47 pgs: 47 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-08-30T23:55:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:11 smithi173 ceph-mon[20554]: 8.8 scrub starts 2024-08-30T23:55:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:11 smithi173 ceph-mon[20554]: 8.5 scrub starts 2024-08-30T23:55:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:11 smithi173 ceph-mon[20554]: osdmap e885: 12 total, 12 up, 12 in 2024-08-30T23:55:11.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:11.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:11.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:12.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:12.044 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.045 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:12.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:12.185 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:12.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:12.315 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:12.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-08-30T23:55:12.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:55:12.417 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:12.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 85019 85003 68.9774 58.125 0.02399 0.0144925 2024-08-30T23:55:12.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:12.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:12.566 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:12.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:12 smithi106 ceph-mon[16603]: 8.f scrub starts 2024-08-30T23:55:12.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:12 smithi106 ceph-mon[16603]: 8.8 scrub ok 2024-08-30T23:55:12.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:12 smithi106 ceph-mon[16603]: 8.5 scrub ok 2024-08-30T23:55:12.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:12 smithi106 ceph-mon[16603]: 8.6 scrub starts 2024-08-30T23:55:12.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:12.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:12 smithi136 ceph-mon[20207]: 8.f scrub starts 2024-08-30T23:55:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:12 smithi136 ceph-mon[20207]: 8.8 scrub ok 2024-08-30T23:55:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:12 smithi136 ceph-mon[20207]: 8.5 scrub ok 2024-08-30T23:55:12.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:12 smithi136 ceph-mon[20207]: 8.6 scrub starts 2024-08-30T23:55:12.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:12.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:12 smithi173 ceph-mon[20554]: 8.f scrub starts 2024-08-30T23:55:12.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:12 smithi173 ceph-mon[20554]: 8.8 scrub ok 2024-08-30T23:55:12.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:12 smithi173 ceph-mon[20554]: 8.5 scrub ok 2024-08-30T23:55:12.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:12 smithi173 ceph-mon[20554]: 8.6 scrub starts 2024-08-30T23:55:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:12.864 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:12.864 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:12.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:12.945 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:12.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:13.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:13.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:13.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:13.206 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:13.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:13.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:13.424 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 15 85931 85916 68.8246 57.0625 0.00426882 0.0145249 2024-08-30T23:55:13.431 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:13.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:13.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:13.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:13.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:13.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:13 smithi136 ceph-mon[20207]: pgmap v1965: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:55:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:13 smithi136 ceph-mon[20207]: 8.f scrub ok 2024-08-30T23:55:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:13 smithi136 ceph-mon[20207]: 8.7 scrub starts 2024-08-30T23:55:13.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:13.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:13 smithi173 ceph-mon[20554]: pgmap v1965: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:55:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:13 smithi173 ceph-mon[20554]: 8.f scrub ok 2024-08-30T23:55:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:13 smithi173 ceph-mon[20554]: 8.7 scrub starts 2024-08-30T23:55:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:13 smithi106 ceph-mon[16603]: pgmap v1965: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-30T23:55:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:13 smithi106 ceph-mon[16603]: 8.f scrub ok 2024-08-30T23:55:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:13 smithi106 ceph-mon[16603]: 8.7 scrub starts 2024-08-30T23:55:13.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:13.963 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:13.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:14.065 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:14.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:14.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:14.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:14.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:14.342 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:14.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 86846 86830 68.6765 57.125 0.0185569 0.0145551 2024-08-30T23:55:14.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:14.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:14.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:14.598 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:14.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:14.722 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:14.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:14 smithi136 ceph-mon[20207]: 8.6 scrub ok 2024-08-30T23:55:14.823 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:14 smithi106 ceph-mon[16603]: 8.6 scrub ok 2024-08-30T23:55:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:14 smithi173 ceph-mon[20554]: 8.6 scrub ok 2024-08-30T23:55:14.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:14.852 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:14.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:14.982 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:14.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:15.084 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:15.119 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:15.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:15.249 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:15.351 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.378 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:15.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:15.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:55:15.425553+0000 min lat: 0.0010542 max lat: 2.02823 avg lat: 0.0145539 2024-08-30T23:55:15.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:55:15.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 87953 87937 68.6828 69.1875 0.024919 0.0145539 2024-08-30T23:55:15.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:55:15.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:15.507 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:15.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:15.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:15.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:15 smithi106 ceph-mon[16603]: pgmap v1966: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s 2024-08-30T23:55:15.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:15 smithi106 ceph-mon[16603]: 8.7 scrub ok 2024-08-30T23:55:15.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:15 smithi106 ceph-mon[16603]: 8.b scrub starts 2024-08-30T23:55:15.659 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:15 smithi106 ceph-mon[16603]: 8.3 scrub starts 2024-08-30T23:55:15.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:15.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:15 smithi136 ceph-mon[20207]: pgmap v1966: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s 2024-08-30T23:55:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:15 smithi136 ceph-mon[20207]: 8.7 scrub ok 2024-08-30T23:55:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:15 smithi136 ceph-mon[20207]: 8.b scrub starts 2024-08-30T23:55:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:15 smithi136 ceph-mon[20207]: 8.3 scrub starts 2024-08-30T23:55:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:15 smithi173 ceph-mon[20554]: pgmap v1966: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s 2024-08-30T23:55:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:15 smithi173 ceph-mon[20554]: 8.7 scrub ok 2024-08-30T23:55:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:15 smithi173 ceph-mon[20554]: 8.b scrub starts 2024-08-30T23:55:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:15 smithi173 ceph-mon[20554]: 8.3 scrub starts 2024-08-30T23:55:15.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:15.885 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:15.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:15.923 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:15.924 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:15.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:16.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:16.118 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:16.145 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:16.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.274 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:16.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:16.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:16.392 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:55:16.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:16.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 88920 88904 68.581 60.4375 0.0452349 0.0145749 2024-08-30T23:55:16.478 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:16 smithi106 ceph-mon[16603]: 8.b scrub ok 2024-08-30T23:55:16.478 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/504597975' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:55:16.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:16.527 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:16.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:16.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:16.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:16.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:16 smithi136 ceph-mon[20207]: 8.b scrub ok 2024-08-30T23:55:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/504597975' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:55:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:16 smithi173 ceph-mon[20554]: 8.b scrub ok 2024-08-30T23:55:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/504597975' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:55:16.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:16.902 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:16.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:17.004 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:17.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:17.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:17.161 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:17.263 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:17.297 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:17.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:17.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 89945 89929 68.5256 64.0625 0.0170381 0.0145879 2024-08-30T23:55:17.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:17.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:17.565 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: pgmap v1967: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 877 op/s 2024-08-30T23:55:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: 8.3 scrub ok 2024-08-30T23:55:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: 8.d scrub starts 2024-08-30T23:55:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:17.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/504597975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:55:17.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: osdmap e886: 12 total, 12 up, 12 in 2024-08-30T23:55:17.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:17.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:17 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:17.667 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:17.695 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: pgmap v1967: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 877 op/s 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: 8.3 scrub ok 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: 8.d scrub starts 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/504597975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: osdmap e886: 12 total, 12 up, 12 in 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:17 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:17.798 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:17.822 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: pgmap v1967: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 877 op/s 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: 8.3 scrub ok 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: 8.d scrub starts 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/504597975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: osdmap e886: 12 total, 12 up, 12 in 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:17 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:17.841 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:17.894 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:17.894 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:17.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:17.925 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:17.925 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:17.947 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:17.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:17.949 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:17.949 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:17.977 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:17.977 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:18.008 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:18.008 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:18.042 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:18.042 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:18.050 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:18.073 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:18.073 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:18.076 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:18.100 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:18.100 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:18.129 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:18.129 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:18.160 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:18.160 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:18.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:18.189 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:18.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:18.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:18.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:18.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:18.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:18.333 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:18.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 90969 90953 68.471 64 0.0142881 0.0145977 2024-08-30T23:55:18.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:18.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-08-30T23:55:18.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:18.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:18.583 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:18.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:18 smithi136 ceph-mon[20207]: osdmap e887: 12 total, 12 up, 12 in 2024-08-30T23:55:18.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/556808353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:18.672 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:18 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:55:18.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:18.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:18.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:18 smithi173 ceph-mon[20554]: osdmap e887: 12 total, 12 up, 12 in 2024-08-30T23:55:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/556808353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:18 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:55:18.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:18 smithi106 ceph-mon[16603]: osdmap e887: 12 total, 12 up, 12 in 2024-08-30T23:55:18.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/556808353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:18.837 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:18 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:55:18.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:18.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:18.973 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:18.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:19.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:19.100 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:19.203 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:19.234 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:19.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:19.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:55:19.373 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:19.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 92011 91995 68.4311 65.125 0.0157349 0.014608 2024-08-30T23:55:19.471 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:19 smithi106 ceph-mon[16603]: pgmap v1970: 47 pgs: 47 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:55:19.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:19.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:19 smithi173 ceph-mon[20554]: pgmap v1970: 47 pgs: 47 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:55:19.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:19.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:19.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:19.757 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:55:19.758 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:55:19.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.758 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:92291 8.0 8:0c063278:::benchmark_data_smithi173_143816_object92290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e887)", 2024-08-30T23:55:19.758 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:55:19.717328+0000", 2024-08-30T23:55:19.758 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.039343775999999997, 2024-08-30T23:55:19.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.039384058999999999, 2024-08-30T23:55:19.759 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:55:19.759 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:55:19.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:55:19.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:55:19.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:55:19.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:55:19.760 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 92291 2024-08-30T23:55:19.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.761 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:55:19.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:55:19.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.717328+0000", 2024-08-30T23:55:19.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:19.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:55:19.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.717328+0000", 2024-08-30T23:55:19.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:55:19.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:55:19.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.717329+0000", 2024-08-30T23:55:19.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1500000000000003e-07 2024-08-30T23:55:19.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:55:19.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.717386+0000", 2024-08-30T23:55:19.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7457999999999998e-05 2024-08-30T23:55:19.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:55:19.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.717400+0000", 2024-08-30T23:55:19.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4055e-05 2024-08-30T23:55:19.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:55:19.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.717408+0000", 2024-08-30T23:55:19.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9549999999999992e-06 2024-08-30T23:55:19.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.722126+0000", 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047177859999999999 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.722262+0000", 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013652400000000001 2024-08-30T23:55:19.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.722397+0000", 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013416200000000001 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:55:19.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.725822+0000", 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0034251139999999999 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:19.725847+0000", 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.5726000000000001e-05 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:55:19.769 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:19.770 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:55:19.770 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:55:19.770 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 1 2024-08-30T23:55:19.770 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:55:19.771 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.772 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:19.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:19 smithi136 ceph-mon[20207]: pgmap v1970: 47 pgs: 47 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-30T23:55:19.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:19.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:19.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:19.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:20.020 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:20.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:20.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:20.150 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:20.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:20.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:20.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:20.408 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:20.425 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 93006 92990 68.3576 62.1875 0.00469297 0.0146233 2024-08-30T23:55:20.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:20.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:20.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.666 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:20.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:20.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:20.694 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:20 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:20.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:20.708 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:20 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:20.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:55:20.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:20 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.803 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:20.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:20.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:20.936 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:20.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:21.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:21.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:21.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:21.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:55:21.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:21.323 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:21.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:21.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 94049 94033 68.3206 65.1875 0.0140255 0.0146319 2024-08-30T23:55:21.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:21.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:21.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:21.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:21 smithi106 ceph-mon[16603]: pgmap v1971: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 883 op/s 2024-08-30T23:55:21.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:21.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:21.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:21 smithi136 ceph-mon[20207]: pgmap v1971: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 883 op/s 2024-08-30T23:55:21.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:21 smithi173 ceph-mon[20554]: pgmap v1971: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 883 op/s 2024-08-30T23:55:21.849 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:21.851 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:21.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:21.853 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:21.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:21.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:21.986 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:22.087 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:22.111 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:22.213 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:22.237 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:22.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:22.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:22.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 95145 95129 68.3226 68.5 0.0174655 0.0146309 2024-08-30T23:55:22.451 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-08-30T23:55:22.454 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:55:22.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:22.490 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:22.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:22.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:22.691 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3407619475' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:22.691 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:22 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:22.726 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:22.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3407619475' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:22 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3407619475' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:22 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:22.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:22.875 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:22.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:22.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:23.002 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:23.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:23.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:23.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:23.271 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.272 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:23.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:23.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:23.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 96153 96137 68.2621 63 0.0103127 0.0146442 2024-08-30T23:55:23.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:23.541 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:23.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:23.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:23.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:23 smithi106 ceph-mon[16603]: pgmap v1972: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:55:23.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:23 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:23.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:23 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:23.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:23 smithi106 ceph-mon[16603]: osdmap e888: 12 total, 12 up, 12 in 2024-08-30T23:55:23.769 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:23.769 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:23.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:23 smithi136 ceph-mon[20207]: pgmap v1972: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:55:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:23 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:23 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:23 smithi136 ceph-mon[20207]: osdmap e888: 12 total, 12 up, 12 in 2024-08-30T23:55:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:23.801 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:23 smithi173 ceph-mon[20554]: pgmap v1972: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-08-30T23:55:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:23 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:23 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:23 smithi173 ceph-mon[20554]: osdmap e888: 12 total, 12 up, 12 in 2024-08-30T23:55:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:23.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:23.919 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:23.919 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:23.931 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:23.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:23.956 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:23.956 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:23.984 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:23.984 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:24.013 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:24.014 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:24.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:24.038 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:24.038 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:24.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:24.070 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:24.070 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:24.100 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:24.100 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:24.127 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:24.128 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:24.156 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:24.156 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:24.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:24.185 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:24.185 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:24.200 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.201 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:24.207 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:24.220 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:24.220 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:24.253 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:24.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:24.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:24.325 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:24.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 97239 97223 68.2577 67.875 0.00509743 0.0146453 2024-08-30T23:55:24.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.466 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:24.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:24.503 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:24.504 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:24.568 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:24.601 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: osdmap e889: 12 total, 12 up, 12 in 2024-08-30T23:55:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1383207899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:55:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:55:24.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:24.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: osdmap e889: 12 total, 12 up, 12 in 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1383207899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:55:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: osdmap e889: 12 total, 12 up, 12 in 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1383207899' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:55:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-30T23:55:24.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.870 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:24.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:24.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:24.995 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:24.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:25.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:25.122 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:25.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:25.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:25.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:25.373 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:25.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 11 98311 98300 68.2471 67.3125 0.0309746 0.014644 2024-08-30T23:55:25.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:25.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.497 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:25.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:25.620 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:25.669 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:55:25.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:25.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:25.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:25 smithi136 ceph-mon[20207]: pgmap v1975: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 791 op/s 2024-08-30T23:55:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:25 smithi136 ceph-mon[20207]: osdmap e890: 12 total, 12 up, 12 in 2024-08-30T23:55:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:25 smithi173 ceph-mon[20554]: pgmap v1975: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 791 op/s 2024-08-30T23:55:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:25 smithi173 ceph-mon[20554]: osdmap e890: 12 total, 12 up, 12 in 2024-08-30T23:55:25.844 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:25 smithi106 ceph-mon[16603]: pgmap v1975: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 791 op/s 2024-08-30T23:55:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:25.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:25 smithi106 ceph-mon[16603]: osdmap e890: 12 total, 12 up, 12 in 2024-08-30T23:55:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:25.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:25.992 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:25.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:25.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:26.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:26.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:26.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:26.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:55:26.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:26.341 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:26.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:26.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 90.6935 2024-08-30T23:55:26.426 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 98311 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 67.7495 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 22.5461 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 129.438 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 1.5 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1083 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 360.738 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2071 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 24 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0146777 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0438023 2024-08-30T23:55:26.427 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.02823 2024-08-30T23:55:26.428 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.0010542 2024-08-30T23:55:26.432 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:55:26.465 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:26.493 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:26.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:26.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:26.720 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:26.745 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:26.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:26.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:26.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:26.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:27.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:27.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:27.127 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:27.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:27.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:27.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:27 smithi136 ceph-mon[20207]: pgmap v1977: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s 2024-08-30T23:55:27.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:27 smithi136 ceph-mon[20207]: osdmap e891: 12 total, 12 up, 12 in 2024-08-30T23:55:27.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:27 smithi173 ceph-mon[20554]: pgmap v1977: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s 2024-08-30T23:55:27.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:27 smithi173 ceph-mon[20554]: osdmap e891: 12 total, 12 up, 12 in 2024-08-30T23:55:27.353 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:27 smithi106 ceph-mon[16603]: pgmap v1977: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s 2024-08-30T23:55:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:27 smithi106 ceph-mon[16603]: osdmap e891: 12 total, 12 up, 12 in 2024-08-30T23:55:27.376 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:27.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-08-30T23:55:27.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:27.541 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:27.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:27.644 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:27.682 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:27.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:27.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:27.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:27.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:27.973 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:27.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:28.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:28.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:28.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:28.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:28.327 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:28.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:28.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:28.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:28.534 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:28.534 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:28.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.619 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:28.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:28.721 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:28.746 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:28.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:28.870 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:28.874 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:28.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:28.997 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:28.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:29.009 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:55:29.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:29.120 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:29.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:29.251 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:29.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:29 smithi136 ceph-mon[20207]: pgmap v1979: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 891 op/s 2024-08-30T23:55:29.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1506655700' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:29.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:29 smithi173 ceph-mon[20554]: pgmap v1979: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 891 op/s 2024-08-30T23:55:29.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1506655700' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:29.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:29 smithi106 ceph-mon[16603]: pgmap v1979: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 891 op/s 2024-08-30T23:55:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1506655700' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:29.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:29.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:29.498 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:29.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:29.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:29.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:55:29.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:29.746 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:29.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:29.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:55:29.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:29.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:29.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:29.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:30.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:30 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:30 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:30 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1506655700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:30.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:30 smithi106 ceph-mon[16603]: osdmap e892: 12 total, 12 up, 12 in 2024-08-30T23:55:30.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:30.121 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:30.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.239 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:30.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:30.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:30 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:30.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:30 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:30.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:30 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:30.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1506655700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:30.243 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:30 smithi136 ceph-mon[20207]: osdmap e892: 12 total, 12 up, 12 in 2024-08-30T23:55:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:30 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:30 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:30 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1506655700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:30.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:30 smithi173 ceph-mon[20554]: osdmap e892: 12 total, 12 up, 12 in 2024-08-30T23:55:30.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:30.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:30.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.486 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:30.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:30.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:30.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:30.623 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:30.623 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:30.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:30.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:30.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:30.858 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:30.956 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:30.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:30.981 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:30.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:31.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:31.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:31.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:31.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:31.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:31 smithi136 ceph-mon[20207]: 8.e scrub starts 2024-08-30T23:55:31.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:31 smithi136 ceph-mon[20207]: pgmap v1981: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 955 op/s 2024-08-30T23:55:31.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:31 smithi136 ceph-mon[20207]: 8.c scrub starts 2024-08-30T23:55:31.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:31 smithi136 ceph-mon[20207]: osdmap e893: 12 total, 12 up, 12 in 2024-08-30T23:55:31.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:31 smithi106 ceph-mon[16603]: 8.e scrub starts 2024-08-30T23:55:31.306 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:31 smithi106 ceph-mon[16603]: pgmap v1981: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 955 op/s 2024-08-30T23:55:31.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:31 smithi106 ceph-mon[16603]: 8.c scrub starts 2024-08-30T23:55:31.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:31 smithi106 ceph-mon[16603]: osdmap e893: 12 total, 12 up, 12 in 2024-08-30T23:55:31.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:31 smithi173 ceph-mon[20554]: 8.e scrub starts 2024-08-30T23:55:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:31 smithi173 ceph-mon[20554]: pgmap v1981: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 955 op/s 2024-08-30T23:55:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:31 smithi173 ceph-mon[20554]: 8.c scrub starts 2024-08-30T23:55:31.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:31 smithi173 ceph-mon[20554]: osdmap e893: 12 total, 12 up, 12 in 2024-08-30T23:55:31.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:31.350 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:31.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:55:31.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:31.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:31.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.601 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:31.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:31.671 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:31.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:31.705 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-08-30T23:55:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:31.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:31.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:31.847 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:31.949 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:31.970 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:31.971 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:32.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:32.094 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:32.138 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:32 smithi106 ceph-mon[16603]: 8.e scrub ok 2024-08-30T23:55:32.139 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:32 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-30T23:55:32.139 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:32 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-30T23:55:32.139 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:32 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:55:32.139 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:32 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:55:32.150 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:32.150 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:32.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:32.220 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:32.266 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:32.267 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:32.275 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:32.308 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:32.308 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:32.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:32.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:32 smithi173 ceph-mon[20554]: 8.e scrub ok 2024-08-30T23:55:32.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:32 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-30T23:55:32.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:32 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-30T23:55:32.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:32 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:55:32.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:32 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:55:32.342 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:32.342 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:32.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:32.370 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:32.370 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:32.397 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:32.397 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:32.425 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:32.425 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:32.451 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:32.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:32.458 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:32.479 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:32.479 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:32.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:32.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:55:32.507 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:32.508 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:32.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:32 smithi136 ceph-mon[20207]: 8.e scrub ok 2024-08-30T23:55:32.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:32 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-30T23:55:32.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:32 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-30T23:55:32.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:32 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:55:32.519 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:32 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:55:32.534 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:32.534 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:32.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:32.565 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:32.587 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:32.599 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:32.600 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:32.616 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:32.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:32.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:32.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:55:32.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:32.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:32.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:32.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:33.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:33.074 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:33.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:33.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:33.138 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:33.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:33 smithi106 ceph-mon[16603]: 8.c scrub ok 2024-08-30T23:55:33.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:33 smithi106 ceph-mon[16603]: pgmap v1983: 47 pgs: 47 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 30 MiB/s wr, 891 op/s 2024-08-30T23:55:33.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/917750709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:33.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:33.260 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:33.261 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:33 smithi136 ceph-mon[20207]: 8.c scrub ok 2024-08-30T23:55:33.261 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:33 smithi136 ceph-mon[20207]: pgmap v1983: 47 pgs: 47 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 30 MiB/s wr, 891 op/s 2024-08-30T23:55:33.261 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/917750709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:33 smithi173 ceph-mon[20554]: 8.c scrub ok 2024-08-30T23:55:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:33 smithi173 ceph-mon[20554]: pgmap v1983: 47 pgs: 47 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 30 MiB/s wr, 891 op/s 2024-08-30T23:55:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/917750709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:33.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:33.390 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:33.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:33.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:33.621 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:33.643 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(1.3 e535 from shard 8)", 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 427.86489118899999, 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74531784099999998, 2024-08-30T23:55:33.685 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817197+0000", 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817198+0000", 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0399999999999996e-07 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:25.817203+0000", 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6729999999999997e-06 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562374+0000", 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.74517085800000005 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562381+0000", 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.878e-06 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562485+0000", 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103906 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562493+0000", 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:26.562515+0000", 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2367000000000001e-05 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 763/762 [PushReplyOp(7:23574efc:::benchmark_data_smithi173_129296_object75:head)])", 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 306.393770738, 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71604419900000005, 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0699999999999997e-07 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288323+0000", 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3490000000000002e-06 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001046+0000", 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71272310000000005 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001053+0000", 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0230000000000004e-06 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001113+0000", 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0402000000000002e-05 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.004362+0000", 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032488180000000001 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 766/762 [PushReplyOp(7:21ab2b56:::benchmark_data_smithi173_129296_object2772:head)])", 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 302.52634935700002, 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92479003900000001, 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155740+0000", 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0780000000000001e-06 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155745+0000", 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077308+0000", 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92156320800000002 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077323+0000", 2024-08-30T23:55:33.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4443e-05 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077429+0000", 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105683 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.080529+0000", 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031005659999999999 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 767/762 [PushReplyOp(7:26f21d66:::benchmark_data_smithi173_129296_object4816:head)])", 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 298.84890427800002, 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0014494869999999, 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:55:33.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833185+0000", 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833190+0000", 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0830000000000003e-06 2024-08-30T23:55:33.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830783+0000", 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99759255400000002 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830792+0000", 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7779999999999996e-06 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830874+0000", 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2557000000000003e-05 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.834634+0000", 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003759465 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.701 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.b 769/756 [PushOp(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head, version: 734'1560, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head@734'1560, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 289.12893175800002, 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60120657700000002, 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553158+0000", 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3799999999999999e-06 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553211+0000", 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2676999999999997e-05 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153021+0000", 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59981003200000005 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153032+0000", 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1415e-05 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153139+0000", 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107075 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153149+0000", 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9629999999999993e-06 2024-08-30T23:55:33.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.154363+0000", 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214035 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:123065 7.1 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e792)", 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:55:33.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 219.19457478199999, 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64777179500000004, 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 123065 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487515+0000", 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.477e-06 2024-08-30T23:55:33.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487521+0000", 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487533+0000", 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.234e-05 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487541+0000", 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487594+0000", 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3134000000000003e-05 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487681+0000", 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7437999999999999e-05 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487753+0000", 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1632999999999997e-05 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.490939+0000", 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031855550000000001 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.492073+0000", 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011345039999999999 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135209+0000", 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64313627299999998 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135251+0000", 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1378000000000003e-05 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135285+0000", 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4730999999999997e-05 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:142377 7.8 7:18815242:::benchmark_data_smithi173_129296_object55261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 176.28651625399999, 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61694005799999996, 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 142377 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:55:33.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395573+0000", 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395576+0000", 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395586+0000", 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4360000000000008e-06 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395591+0000", 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8910000000000003e-06 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395638+0000", 2024-08-30T23:55:33.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7027000000000002e-05 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395723+0000", 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5181999999999999e-05 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395790+0000", 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7430999999999999e-05 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400118+0000", 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043280920000000004 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400495+0000", 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037659899999999998 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012439+0000", 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61194435700000005 2024-08-30T23:55:33.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012480+0000", 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1180000000000002e-05 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012512+0000", 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1869999999999998e-05 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.720 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145475 7.4 7:2251fdb7:::benchmark_data_smithi173_129296_object58359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.50908095099999, 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61919969799999997, 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.721 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145475 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173010+0000", 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7190000000000001e-06 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173019+0000", 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6289999999999999e-06 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173041+0000", 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658000000000002e-05 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173051+0000", 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0139999999999999e-05 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173114+0000", 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3289999999999999e-05 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173214+0000", 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100282 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173291+0000", 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6983999999999993e-05 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.177105+0000", 2024-08-30T23:55:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003813355 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.178585+0000", 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001480608 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792092+0000", 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61350662700000003 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792154+0000", 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1884000000000006e-05 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792207+0000", 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3522e-05 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145497 7.4 7:2208bbfd:::benchmark_data_smithi173_129296_object58381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.727 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.473695569, 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60666463800000003, 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145497 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208395+0000", 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.66e-06 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208403+0000", 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2309999999999996e-06 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208421+0000", 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7825999999999998e-05 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208428+0000", 2024-08-30T23:55:33.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0539999999999997e-06 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208486+0000", 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8517000000000002e-05 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208625+0000", 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138517 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208748+0000", 2024-08-30T23:55:33.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012329600000000001 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.212469+0000", 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037205519999999998 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.213491+0000", 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010220489999999999 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.814962+0000", 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.601471588 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815023+0000", 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0732000000000001e-05 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815058+0000", 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4616000000000002e-05 2024-08-30T23:55:33.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145502 7.4 7:2d669461:::benchmark_data_smithi173_129296_object58386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.46621248599999, 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59929806799999996, 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145502 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215878+0000", 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6220000000000001e-06 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215886+0000", 2024-08-30T23:55:33.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9219999999999997e-06 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215902+0000", 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6677e-05 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215910+0000", 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8499999999999994e-06 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215967+0000", 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6898999999999999e-05 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216108+0000", 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014146699999999999 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216238+0000", 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012992900000000001 2024-08-30T23:55:33.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223310+0000", 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070715079999999998 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224153+0000", 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084321899999999998 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815104+0000", 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59095095600000003 2024-08-30T23:55:33.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815137+0000", 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2759000000000003e-05 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815174+0000", 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7259999999999999e-05 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145504 7.4 7:2d76a73e:::benchmark_data_smithi173_129296_object58388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.46244781799999, 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59561452400000003, 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145504 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219642+0000", 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.782e-06 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219653+0000", 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0723999999999999e-05 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219670+0000", 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.681e-05 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219679+0000", 2024-08-30T23:55:33.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219829+0000", 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149863 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219920+0000", 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1633999999999997e-05 2024-08-30T23:55:33.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.220015+0000", 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4344000000000003e-05 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223383+0000", 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033687449999999998 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224195+0000", 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081209999999999995 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815208+0000", 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59101230400000004 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815235+0000", 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7310000000000001e-05 2024-08-30T23:55:33.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815255+0000", 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0075e-05 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145510 7.4 7:2d7edcbd:::benchmark_data_smithi173_129296_object58394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.44159332199999, 2024-08-30T23:55:33.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.57487306000000005, 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145510 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:55:33.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240497+0000", 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.381e-06 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240502+0000", 2024-08-30T23:55:33.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.575e-06 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240514+0000", 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1996e-05 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240522+0000", 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7079999999999997e-06 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240565+0000", 2024-08-30T23:55:33.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3095e-05 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240644+0000", 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9509000000000001e-05 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240712+0000", 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7859000000000002e-05 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.244050+0000", 2024-08-30T23:55:33.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003337727 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.245826+0000", 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017758229999999999 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815284+0000", 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56945848700000001 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815331+0000", 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6276999999999998e-05 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815368+0000", 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7623000000000002e-05 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145529 7.4 7:2339dd6d:::benchmark_data_smithi173_129296_object58413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.38701243599999, 2024-08-30T23:55:33.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53035033899999995, 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145529 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295077+0000", 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.47e-07 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295085+0000", 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9750000000000007e-06 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295097+0000", 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2391999999999999e-05 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295103+0000", 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8529999999999997e-06 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295150+0000", 2024-08-30T23:55:33.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6684000000000002e-05 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295235+0000", 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4892e-05 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295302+0000", 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6934999999999995e-05 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.298587+0000", 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032851030000000002 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.299413+0000", 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00082625299999999999 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825357+0000", 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52594381499999998 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825395+0000", 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7898999999999997e-05 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825426+0000", 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1591000000000003e-05 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145538 7.4 7:2666a83b:::benchmark_data_smithi173_129296_object58422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.35999842499999, 2024-08-30T23:55:33.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51489389200000002, 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145538 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322092+0000", 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.683e-06 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322100+0000", 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322119+0000", 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9368000000000002e-05 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322128+0000", 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1599999999999998e-06 2024-08-30T23:55:33.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322189+0000", 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0976e-05 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322326+0000", 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013735099999999999 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322456+0000", 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012996399999999999 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326052+0000", 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035965490000000001 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328903+0000", 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028504350000000001 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836909+0000", 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.508006556 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836949+0000", 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9864000000000003e-05 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836984+0000", 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4674e-05 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145540 7.4 7:22c5e8c4:::benchmark_data_smithi173_129296_object58424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.35583053299999, 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52274398799999999, 2024-08-30T23:55:33.766 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145540 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326259+0000", 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2440000000000001e-06 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326268+0000", 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3760000000000008e-06 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326280+0000", 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2665e-05 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326288+0000", 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326325+0000", 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6508999999999999e-05 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326403+0000", 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7877e-05 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326473+0000", 2024-08-30T23:55:33.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9820999999999994e-05 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328987+0000", 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025140520000000001 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.330596+0000", 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001609367 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848889+0000", 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51829265199999996 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848951+0000", 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2806999999999997e-05 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849002+0000", 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0544000000000003e-05 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.772 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:33.774 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145558 7.4 7:2fe416e2:::benchmark_data_smithi173_129296_object58442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.774 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.27640461600001, 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44345267799999999, 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145558 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405686+0000", 2024-08-30T23:55:33.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4059999999999999e-06 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405696+0000", 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0073e-05 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405726+0000", 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9686999999999999e-05 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405740+0000", 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4072e-05 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405842+0000", 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010199700000000001 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405928+0000", 2024-08-30T23:55:33.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5784000000000004e-05 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.406015+0000", 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7096999999999996e-05 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.409500+0000", 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034854629999999998 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.411030+0000", 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015296820000000001 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849046+0000", 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43801564300000001 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849097+0000", 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1424000000000001e-05 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849137+0000", 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9350000000000001e-05 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145576 7.4 7:2d26fcfb:::benchmark_data_smithi173_129296_object58460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 170.124223761, 2024-08-30T23:55:33.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29138068, 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145576 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557867+0000", 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9910000000000001e-06 2024-08-30T23:55:33.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557875+0000", 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557897+0000", 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2021000000000001e-05 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557906+0000", 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3360000000000001e-06 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557987+0000", 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1238999999999999e-05 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.558139+0000", 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000151368 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.558263+0000", 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123824 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.561781+0000", 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035184819999999999 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.562154+0000", 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037294599999999998 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849181+0000", 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28702652299999998 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849221+0000", 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0670000000000002e-05 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849245+0000", 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4122e-05 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21811 8.5 e853/840 8:a2778f2f:::benchmark_data_smithi173_143816_object21810:head v 853'1377, mlcod=853'1375)", 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.953091+0000", 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 82.728997296000003, 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34313867100000001, 2024-08-30T23:55:33.788 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953091+0000", 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953091+0000", 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953093+0000", 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.81e-06 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953143+0000", 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0099000000000003e-05 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295091+0000", 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34194817500000002 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295107+0000", 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6229999999999999e-05 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295194+0000", 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6767999999999993e-05 2024-08-30T23:55:33.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295221+0000", 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6829e-05 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.296190+0000", 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096933700000000002 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.296230+0000", 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9422999999999998e-05 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:27857 8.e 8:73643902:::benchmark_data_smithi173_143816_object27856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:16.629598+0000", 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.052490258999995, 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25793482000000001, 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.793 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 27857 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629598+0000", 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629598+0000", 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629600+0000", 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4649999999999999e-06 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629642+0000", 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2215999999999997e-05 2024-08-30T23:55:33.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629654+0000", 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2048000000000001e-05 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629659+0000", 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2989999999999996e-06 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629705+0000", 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5391e-05 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629794+0000", 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9042000000000006e-05 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.629878+0000", 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4351000000000006e-05 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.631014+0000", 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001135987 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.631353+0000", 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033861200000000002 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.887440+0000", 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25608689800000001 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.887495+0000", 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5571e-05 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:16.887533+0000", 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7939999999999999e-05 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:36537 8.e 8:78cdc0b0:::benchmark_data_smithi173_143816_object36536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:25.400191+0000", 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.281897043000001, 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.395384505, 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36537 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400191+0000", 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400191+0000", 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400193+0000", 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0690000000000001e-06 2024-08-30T23:55:33.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400228+0000", 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5423000000000003e-05 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400241+0000", 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2695e-05 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:33.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400248+0000", 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999993e-06 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400300+0000", 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1440000000000002e-05 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400391+0000", 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1564999999999995e-05 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400477+0000", 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5532999999999996e-05 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.401514+0000", 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037428 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.402322+0000", 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080706599999999995 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795505+0000", 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.393183318 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795541+0000", 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6442000000000001e-05 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:33.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795576+0000", 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4666000000000003e-05 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:33.806 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:55:33.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:33.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:33.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:33.940 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:33.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:34.041 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:34.075 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:34.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:34.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:34.200 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:34.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:34 smithi173 ceph-mon[20554]: 8.2 scrub starts 2024-08-30T23:55:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:34 smithi173 ceph-mon[20554]: 8.d scrub starts 2024-08-30T23:55:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:34 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:55:34.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:34 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:55:34.332 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:34 smithi106 ceph-mon[16603]: 8.2 scrub starts 2024-08-30T23:55:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:34 smithi106 ceph-mon[16603]: 8.d scrub starts 2024-08-30T23:55:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:34 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:55:34.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:34 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:55:34.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:34 smithi136 ceph-mon[20207]: 8.2 scrub starts 2024-08-30T23:55:34.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:34 smithi136 ceph-mon[20207]: 8.d scrub starts 2024-08-30T23:55:34.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:34 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:55:34.434 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:34 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:55:34.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:34.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:34.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:34.583 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:34.686 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:34.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:55:34.718 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:34.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-08-30T23:55:34.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:34.844 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:34.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:34.977 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:34.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:34.984 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:35.065 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:55:35.082 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:35.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:35.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:35.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:35.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:55:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:55:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: pgmap v1984: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s rd, 23 MiB/s wr, 736 op/s 2024-08-30T23:55:35.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: 8.2 scrub ok 2024-08-30T23:55:35.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: osdmap e894: 12 total, 12 up, 12 in 2024-08-30T23:55:35.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: 8.a scrub starts 2024-08-30T23:55:35.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3078335132' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:35.344 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: pgmap v1984: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s rd, 23 MiB/s wr, 736 op/s 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: 8.2 scrub ok 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: osdmap e894: 12 total, 12 up, 12 in 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: 8.a scrub starts 2024-08-30T23:55:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3078335132' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: pgmap v1984: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s rd, 23 MiB/s wr, 736 op/s 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: 8.2 scrub ok 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: osdmap e894: 12 total, 12 up, 12 in 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: 8.a scrub starts 2024-08-30T23:55:35.370 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3078335132' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:35.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:35.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:35.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:35.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:55:35.605 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:35.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:35.636 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:55:35.636 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:55:35.636 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:55:35.636 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:55:35.731 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:35.757 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:35.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:35.890 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:35.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:35.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:36.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:36.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:36.151 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:36.206 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:36 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:36.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:36 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:55:36.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3078335132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:36.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:36 smithi106 ceph-mon[16603]: osdmap e895: 12 total, 12 up, 12 in 2024-08-30T23:55:36.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:36 smithi106 ceph-mon[16603]: 8.d scrub ok 2024-08-30T23:55:36.253 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:36.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.284 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:36.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:36.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:36 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:36 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:55:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3078335132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:36 smithi136 ceph-mon[20207]: osdmap e895: 12 total, 12 up, 12 in 2024-08-30T23:55:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:36 smithi136 ceph-mon[20207]: 8.d scrub ok 2024-08-30T23:55:36.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:55:36.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:36.417 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:36.462 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:36.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:36.546 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:36.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:36 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:36 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:55:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3078335132' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:36 smithi173 ceph-mon[20554]: osdmap e895: 12 total, 12 up, 12 in 2024-08-30T23:55:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:36 smithi173 ceph-mon[20554]: 8.d scrub ok 2024-08-30T23:55:36.646 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:36.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:36.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:36.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:36.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:36.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:36.925 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:36.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:37.027 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.057 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:37.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:37.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:37.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:37.222 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:37.222 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:37.254 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:37.255 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:37.284 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:37.284 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:37.292 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:37.311 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:37.311 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:37.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:37 smithi106 ceph-mon[16603]: 8.a scrub ok 2024-08-30T23:55:37.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:37 smithi106 ceph-mon[16603]: pgmap v1987: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 170 B/s wr, 503 op/s 2024-08-30T23:55:37.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:37 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:55:37.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:37 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:55:37.315 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:37.339 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:37.339 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:37.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:37.379 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:37.407 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:37.408 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:37.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:37.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:37.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:37.451 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:37.470 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:37.470 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:37.499 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:37.499 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:37.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:37.532 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:37 smithi136 ceph-mon[20207]: 8.a scrub ok 2024-08-30T23:55:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:37 smithi136 ceph-mon[20207]: pgmap v1987: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 170 B/s wr, 503 op/s 2024-08-30T23:55:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:37 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:55:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:37 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:55:37.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:37.563 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:37.563 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:37 smithi173 ceph-mon[20554]: 8.a scrub ok 2024-08-30T23:55:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:37 smithi173 ceph-mon[20554]: pgmap v1987: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 170 B/s wr, 503 op/s 2024-08-30T23:55:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:37 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:55:37.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:37 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:55:37.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:37.691 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:37.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:37.742 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-08-30T23:55:37.799 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:37.799 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:37.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:37.857 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:37.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:55:37.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:37.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:37.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:38.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:38.115 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:38.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:38.239 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:38.341 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:38.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4163187318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:38.365 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:38.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:38.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:38.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4163187318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:38.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:55:38.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4163187318' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:38.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:38.621 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:38.657 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:38.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:38.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:38.831 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:38.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:38.877 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:38.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:38.944 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:38.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:39.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:39.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:39.133 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:39.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:39.255 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:39 smithi136 ceph-mon[20207]: pgmap v1988: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 150 B/s rd, 150 B/s wr, 797 op/s 2024-08-30T23:55:39.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:39.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:39 smithi106 ceph-mon[16603]: pgmap v1988: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 150 B/s rd, 150 B/s wr, 797 op/s 2024-08-30T23:55:39.360 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:39.363 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:39.363 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:39.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:39.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:39.507 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:39.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:39 smithi173 ceph-mon[20554]: pgmap v1988: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 150 B/s rd, 150 B/s wr, 797 op/s 2024-08-30T23:55:39.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:39.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:39.736 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.759 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:39.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:39.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:39.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:39.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:39.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:40.009 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:40.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-30T23:55:40.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:40.135 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:40.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:40.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:40.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:40.373 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:40.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:40.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:40.514 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:40.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:55:40.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:40.643 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:40.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:40.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:40.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:40.898 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:40.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:40.948 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:40.949 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:40.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:41.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:41.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:41.108 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:55:41.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:41.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:41.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:41.279 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:41 smithi106 ceph-mon[16603]: pgmap v1989: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-08-30T23:55:41.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2674292235' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:55:41.381 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.409 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:41.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:41.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:41.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-08-30T23:55:41.528 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:41 smithi136 ceph-mon[20207]: pgmap v1989: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-08-30T23:55:41.528 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2674292235' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:55:41.539 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:41 smithi173 ceph-mon[20554]: pgmap v1989: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s 2024-08-30T23:55:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2674292235' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:55:41.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:41.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:41.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:41.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:41.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:41.916 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:41.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:41.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-30T23:55:42.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:42.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:42 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2674292235' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:55:42.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:42 smithi106 ceph-mon[16603]: osdmap e896: 12 total, 12 up, 12 in 2024-08-30T23:55:42.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:42.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:42.249 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:42.260 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:42.269 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:42.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:42.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:42 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2674292235' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:55:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:42 smithi136 ceph-mon[20207]: osdmap e896: 12 total, 12 up, 12 in 2024-08-30T23:55:42.294 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:42.396 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:42.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:42.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.545 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:42.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:42 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2674292235' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:55:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:42 smithi173 ceph-mon[20554]: osdmap e896: 12 total, 12 up, 12 in 2024-08-30T23:55:42.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:42.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:42.778 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.808 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:42.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:42.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:42.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:42.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:43.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:43.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:43.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:43.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:43.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:43.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:43.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:43.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:43 smithi106 ceph-mon[16603]: pgmap v1991: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-08-30T23:55:43.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:43.453 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:43 smithi136 ceph-mon[20207]: pgmap v1991: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-08-30T23:55:43.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:43.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:43 smithi173 ceph-mon[20554]: pgmap v1991: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-08-30T23:55:43.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:43.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:43.727 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:43.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-08-30T23:55:43.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:43.859 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:43.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:43.987 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:43.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:44.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:44.101 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:44.111 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:44.206 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:55:44.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:44.241 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:44.260 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:44.260 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:44.260 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4021642456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:44.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:44.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4021642456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:44.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:44.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:44.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:44.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4021642456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:44.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:44.648 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:44.700 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:44.750 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:44.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:44.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:44.898 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:44.899 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:44.911 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:44.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:44.934 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:44.935 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:44.963 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:44.963 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:44.991 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:44.991 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:45.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:45.016 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:45.016 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:45.038 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:45.048 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:45.048 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:45.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-30T23:55:45.075 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:45.075 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:45.103 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:45.104 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:45.133 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:45.133 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:45.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:45.166 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:45.167 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:45.167 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:45 smithi106 ceph-mon[16603]: pgmap v1992: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 951 op/s 2024-08-30T23:55:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:45.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:45.196 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:45.197 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:45.231 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:45.231 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:45.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:45.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:45 smithi136 ceph-mon[20207]: pgmap v1992: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 951 op/s 2024-08-30T23:55:45.298 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:45.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-08-30T23:55:45.393 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:55:45.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:45.426 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:45.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:45.553 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:45.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:45 smithi173 ceph-mon[20554]: pgmap v1992: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 951 op/s 2024-08-30T23:55:45.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:45.684 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:45.786 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:45.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:45.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:45.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:45.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:45.955 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:93583 6.8 6:14dca3a3:::benchmark_data_smithi173_114000_object93582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e655)", 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 581.44773926799996, 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34108081800000001, 2024-08-30T23:55:45.956 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 93583 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503090+0000", 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503091+0000", 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503137+0000", 2024-08-30T23:55:45.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6109000000000002e-05 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503151+0000", 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3179999999999999e-05 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503156+0000", 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5639999999999996e-06 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503212+0000", 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5946999999999998e-05 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503322+0000", 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011007500000000001 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,11", 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.503414+0000", 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2143000000000002e-05 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.505854+0000", 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002439887 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.506967+0000", 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001112341 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844102+0000", 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.33713554299999998 2024-08-30T23:55:45.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844139+0000", 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6683000000000001e-05 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:46:04.844171+0000", 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2095999999999999e-05 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:155745 6.5 6:a23d3df7:::benchmark_data_smithi173_114000_object55947:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e684)", 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:39.831928+0000", 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 486.11890171700003, 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67798276800000001, 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 155745 2024-08-30T23:55:45.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831928+0000", 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831928+0000", 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831929+0000", 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7829999999999999e-06 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831935+0000", 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2630000000000003e-06 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831949+0000", 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4025999999999999e-05 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.831956+0000", 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1829999999999997e-06 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.832017+0000", 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1079000000000003e-05 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.832135+0000", 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118416 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.832248+0000", 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011268300000000001 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.836115+0000", 2024-08-30T23:55:45.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038669939999999999 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.837197+0000", 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001082132 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.509846+0000", 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.67264882100000001 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.509880+0000", 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3590999999999997e-05 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.509910+0000", 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0796999999999997e-05 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:199592 6.5 6:a0027b35:::benchmark_data_smithi173_114000_object99794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.158645+0000", 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 419.79218408100002, 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.39176461600000001, 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:55:45.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 199592 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158645+0000", 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158645+0000", 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158647+0000", 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1160000000000002e-06 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158655+0000", 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6979999999999998e-06 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158673+0000", 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7689e-05 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158680+0000", 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6799999999999993e-06 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158732+0000", 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1285000000000001e-05 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158820+0000", 2024-08-30T23:55:45.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8617999999999998e-05 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.158897+0000", 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6822000000000004e-05 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.162811+0000", 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00391414 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.163203+0000", 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00039150099999999999 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.550332+0000", 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.38712935599999998 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.550375+0000", 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2718e-05 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.550410+0000", 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4993000000000002e-05 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:34198 7.8 e736/725 7:1ca020bc:::benchmark_data_smithi173_129296_object34197:head v 736'2123, mlcod=736'2119)", 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:49:27.842045+0000", 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 378.10878442199999, 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26091951200000002, 2024-08-30T23:55:45.976 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842045+0000", 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842045+0000", 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842046+0000", 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:27.842064+0000", 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8097e-05 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101631+0000", 2024-08-30T23:55:45.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25956776599999998 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101639+0000", 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.464e-06 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101695+0000", 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6348000000000003e-05 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.101712+0000", 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6501999999999999e-05 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.102931+0000", 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012193320000000001 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:49:28.102964+0000", 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3353999999999999e-05 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.980 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:87841 7.6 e771/762 7:6125c906:::benchmark_data_smithi173_129296_object725:head v 771'5373, mlcod=734'1200)", 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:50.635013+0000", 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 295.31581608499999, 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26843487199999999, 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635013+0000", 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635013+0000", 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635014+0000", 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2699999999999999e-07 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.635020+0000", 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900819+0000", 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26579951600000001 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900828+0000", 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6400000000000003e-06 2024-08-30T23:55:45.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900899+0000", 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1598000000000001e-05 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.900921+0000", 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2001000000000002e-05 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.903412+0000", 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024904189999999998 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:50.903448+0000", 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6421000000000002e-05 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:93823 7.8 e774/725 7:1693af46:::benchmark_data_smithi173_129296_object6707:head v 774'5852, mlcod=774'5848)", 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:00.487850+0000", 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 285.46297982999999, 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21028736100000001, 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487850+0000", 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487850+0000", 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487851+0000", 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0529999999999999e-06 2024-08-30T23:55:45.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.487859+0000", 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4239999999999993e-06 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695495+0000", 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20763561 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695502+0000", 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2590000000000003e-06 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695559+0000", 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7188000000000002e-05 2024-08-30T23:55:45.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.695583+0000", 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3428e-05 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.698085+0000", 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025024489999999999 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:00.698137+0000", 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1950000000000002e-05 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:144892 7.6 e811/810 7:645a80f3:::benchmark_data_smithi173_129296_object57776:head v 811'8848, mlcod=811'8845)", 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:42.156476+0000", 2024-08-30T23:55:45.991 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 183.79435291199999, 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14011880700000001, 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156476+0000", 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156476+0000", 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156477+0000", 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.156482+0000", 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1279999999999999e-06 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293035+0000", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.136552056 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293042+0000", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9419999999999995e-06 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293104+0000", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1273999999999996e-05 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.293129+0000", 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5641000000000001e-05 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.296564+0000", 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034348130000000001 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:42.296595+0000", 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1102000000000001e-05 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:45.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:147040 7.8 e813/725 7:155da638:::benchmark_data_smithi173_129296_object59924:head v 813'9140, mlcod=813'9136)", 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:46.130984+0000", 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 179.819845208, 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.121825357, 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130984+0000", 2024-08-30T23:55:45.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130984+0000", 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130985+0000", 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6499999999999998e-07 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.130997+0000", 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1625e-05 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248812+0000", 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.117814857 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248817+0000", 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1259999999999997e-06 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248869+0000", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2024000000000002e-05 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.248893+0000", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3992999999999999e-05 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.252767+0000", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038744970000000002 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:45.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:46.252810+0000", 2024-08-30T23:55:46.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2370000000000003e-05 2024-08-30T23:55:46.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.000 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:149982 7.8 e815/725 7:1c54880d:::benchmark_data_smithi173_129296_object62866:head v 815'9327, mlcod=815'9323)", 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:51.302587+0000", 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 174.648242694, 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20097135399999999, 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.001 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302587+0000", 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302587+0000", 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302588+0000", 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6300000000000004e-07 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.302594+0000", 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5309999999999998e-06 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499896+0000", 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19730151900000001 2024-08-30T23:55:46.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499903+0000", 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3000000000000004e-06 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499959+0000", 2024-08-30T23:55:46.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5631000000000003e-05 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.499979+0000", 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0647000000000001e-05 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.503520+0000", 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003540891 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:51.503558+0000", 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7972000000000002e-05 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:153765 7.6 e819/810 7:66a9e493:::benchmark_data_smithi173_129296_object66649:head v 819'9422, mlcod=819'9419)", 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:58.539690+0000", 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 167.41113963500001, 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20419347600000001, 2024-08-30T23:55:46.006 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539690+0000", 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539690+0000", 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539691+0000", 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2999999999999999e-07 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.539697+0000", 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3040000000000001e-06 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739495+0000", 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19979808499999999 2024-08-30T23:55:46.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739501+0000", 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.212e-06 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739559+0000", 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7454999999999998e-05 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.739582+0000", 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3349e-05 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.743844+0000", 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0042621229999999996 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:58.743883+0000", 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9118000000000001e-05 2024-08-30T23:55:46.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:160742 7.8 e823/725 7:1aa16899:::benchmark_data_smithi173_129296_object73626:head v 823'10015, mlcod=822'10011)", 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:10.973818+0000", 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.977010925, 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12619274899999999, 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973818+0000", 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973818+0000", 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973820+0000", 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4500000000000001e-06 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:10.973828+0000", 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.371e-06 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.094925+0000", 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12109639 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.094931+0000", 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2469999999999997e-06 2024-08-30T23:55:46.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.095009+0000", 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7690999999999999e-05 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.095031+0000", 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2415000000000001e-05 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.099954+0000", 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049231769999999999 2024-08-30T23:55:46.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:11.100011+0000", 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7008e-05 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:164977 7.8 e825/725 7:17fdc8b0:::benchmark_data_smithi173_129296_object77861:head v 825'10271, mlcod=825'10268)", 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:18.643415+0000", 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 147.307414088, 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.125165576, 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643415+0000", 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643415+0000", 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643417+0000", 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.548e-06 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.643426+0000", 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3759999999999997e-06 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763421+0000", 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.119994487 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763426+0000", 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2950000000000001e-06 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763755+0000", 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032881400000000003 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.763777+0000", 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2407000000000001e-05 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.768549+0000", 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047720870000000004 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:18.768581+0000", 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1562e-05 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27487.0:165746 7.8 e825/725 7:11156502:::benchmark_data_smithi173_129296_object78630:head v 825'10317, mlcod=825'10314)", 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:20.061291+0000", 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 145.889538319, 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.17562274, 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061291+0000", 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061291+0000", 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061292+0000", 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2860000000000001e-06 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.061301+0000", 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7730000000000005e-06 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232871+0000", 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.171569736 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232877+0000", 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2620000000000004e-06 2024-08-30T23:55:46.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232939+0000", 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1764e-05 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.232959+0000", 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0119000000000002e-05 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.236876+0000", 2024-08-30T23:55:46.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039168390000000001 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:20.236914+0000", 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7960999999999998e-05 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:5745 8.6 e847/840 8:699b143f:::benchmark_data_smithi173_143816_object5744:head v 847'375, mlcod=847'371)", 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:58.798499+0000", 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.152330105, 2024-08-30T23:55:46.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.124204167, 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798499+0000", 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798499+0000", 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798500+0000", 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0200000000000003e-07 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.798566+0000", 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6397999999999997e-05 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922116+0000", 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.123550164 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922125+0000", 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2740000000000007e-06 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922178+0000", 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3798999999999998e-05 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922203+0000", 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4834999999999999e-05 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922671+0000", 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00046786899999999999 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:58.922703+0000", 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2425999999999998e-05 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(8.6 855/854 [PushReplyOp(8:69e20f90:::benchmark_data_smithi173_143816_object924:head)])", 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:12.646653+0000", 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.304176607000002, 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19664833000000001, 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.646653+0000", 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.646653+0000", 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.646653+0000", 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000003e-07 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.646657+0000", 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7280000000000001e-06 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.841123+0000", 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.19446561400000001 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.841130+0000", 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4850000000000003e-06 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.842124+0000", 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00099364399999999991 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:12.843301+0000", 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001177347 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.032 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(8.6 857/854 [PushReplyOp(8:6ec8f9fb:::benchmark_data_smithi173_143816_object8356:head)])", 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:17.061433+0000", 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.889396063000007, 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21921734000000001, 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.061433+0000", 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.061433+0000", 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.061433+0000", 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.11e-07 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.061436+0000", 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7690000000000001e-06 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.279764+0000", 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21832753099999999 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.279771+0000", 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7549999999999994e-06 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.279813+0000", 2024-08-30T23:55:46.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1687000000000003e-05 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:17.280651+0000", 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083748700000000004 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:61547 8.d e873/864 8:bcc3db7b:::benchmark_data_smithi173_143816_object61546:head v 873'3929, mlcod=873'3925)", 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.308972+0000", 2024-08-30T23:55:46.036 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.641857125999998, 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.20240103200000001, 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.308972+0000", 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.308972+0000", 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.308973+0000", 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8500000000000002e-07 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.309061+0000", 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7874999999999995e-05 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509308+0000", 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.200247286 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509316+0000", 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0870000000000007e-06 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509376+0000", 2024-08-30T23:55:46.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0092e-05 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509399+0000", 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2546999999999999e-05 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.511323+0000", 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0019248410000000001 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.511373+0000", 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9818999999999999e-05 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:78179 8.6 e881/861 8:6b40fcbc:::benchmark_data_smithi173_143816_object78178:head v 881'4963, mlcod=881'4961)", 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:55:05.588900+0000", 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.361929322000002, 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13093722299999999, 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.041 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.588900+0000", 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.588900+0000", 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.588901+0000", 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0070000000000001e-06 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.588927+0000", 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5967999999999999e-05 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.717629+0000", 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.12870168200000001 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.717635+0000", 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1129999999999997e-06 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.717682+0000", 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6894999999999997e-05 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.717700+0000", 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8481000000000001e-05 2024-08-30T23:55:46.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.719792+0000", 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020920869999999999 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.719837+0000", 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4990000000000001e-05 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:78180 8.d e881/864 8:b51a952d:::benchmark_data_smithi173_143816_object78179:head v 881'5003, mlcod=881'5000)", 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:55:05.592103+0000", 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.358726091000001, 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13873122500000001, 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.592103+0000", 2024-08-30T23:55:46.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.592103+0000", 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.592104+0000", 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000001e-07 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.592170+0000", 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6569000000000006e-05 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.728516+0000", 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13634563799999999 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.728523+0000", 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8079999999999999e-06 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.728584+0000", 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1054000000000002e-05 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.728604+0000", 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0445000000000002e-05 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.730805+0000", 2024-08-30T23:55:46.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002200446 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.730835+0000", 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9649e-05 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:78241 8.d e881/864 8:b494f79b:::benchmark_data_smithi173_143816_object78240:head v 881'5006, mlcod=881'5003)", 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:55:05.812773+0000", 2024-08-30T23:55:46.050 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.138056397, 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13710278400000001, 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.812773+0000", 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.812773+0000", 2024-08-30T23:55:46.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.812775+0000", 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.703e-06 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.812833+0000", 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7986999999999999e-05 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:55:46.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.947563+0000", 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.134730402 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.947571+0000", 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4909999999999999e-06 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.947628+0000", 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7080000000000002e-05 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.947652+0000", 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3971000000000001e-05 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.949846+0000", 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021945129999999999 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.949876+0000", 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9637000000000001e-05 2024-08-30T23:55:46.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:55:46.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:55:46.055 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:55:46.055 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:46.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:46.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:46.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:46.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:46.335 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:46.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:46.463 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.464 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:46.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:46.588 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:46.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:46.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:46.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:46.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:46.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:46.963 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:46.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:47.065 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:47.100 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:47.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:55:47.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:47.240 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:47.245 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:47 smithi136 ceph-mon[20207]: pgmap v1993: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-08-30T23:55:47.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:47 smithi136 ceph-mon[20207]: osdmap e897: 12 total, 12 up, 12 in 2024-08-30T23:55:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:47 smithi173 ceph-mon[20554]: pgmap v1993: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-08-30T23:55:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:47 smithi173 ceph-mon[20554]: osdmap e897: 12 total, 12 up, 12 in 2024-08-30T23:55:47.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:47 smithi106 ceph-mon[16603]: pgmap v1993: 47 pgs: 47 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 974 op/s 2024-08-30T23:55:47.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:47 smithi106 ceph-mon[16603]: osdmap e897: 12 total, 12 up, 12 in 2024-08-30T23:55:47.366 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:47.468 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:47.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:47.582 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:55:47.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:47.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:47.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:47.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:47.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:47.884 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:47.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:47.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:48.014 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:48.021 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:55:48.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:48.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:48.197 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:55:48.197 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:117520 8.c e898/866 8:3f0d5d8b:::benchmark_data_smithi173_143816_object19206:head v 898'7311, mlcod=898'7308)", 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:48.192143+0000", 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0041535920000000002, 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041800049999999997, 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192143+0000", 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192143+0000", 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192143+0000", 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4199999999999995e-07 2024-08-30T23:55:48.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192148+0000", 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9060000000000001e-06 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192172+0000", 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3830000000000001e-05 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192179+0000", 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4849999999999997e-06 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192229+0000", 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0089000000000001e-05 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.192248+0000", 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9394000000000001e-05 2024-08-30T23:55:48.201 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:117525 8.a 8:53f5d70f:::benchmark_data_smithi173_143816_object19211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e898)", 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:48.194830+0000", 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0014657450000000001, 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001548322, 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:48.202 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 117525 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194830+0000", 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194830+0000", 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194832+0000", 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2440000000000001e-06 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:48.204 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194837+0000", 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.797e-06 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194849+0000", 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2277e-05 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194856+0000", 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2289999999999997e-06 2024-08-30T23:55:48.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.194901+0000", 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5154000000000002e-05 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.195006+0000", 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000104329 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:48.195095+0000", 2024-08-30T23:55:48.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9646000000000002e-05 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 2 2024-08-30T23:55:48.207 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:55:48.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.278 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:48.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2478365925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:48.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:55:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2478365925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:48.345 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2478365925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:48.379 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:48.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:48.504 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:48.529 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:48.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:48.658 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:48.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:48.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:48.831 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:48.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:48.910 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:48.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:49.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:49.039 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:49.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:49.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:49.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:49.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:49 smithi136 ceph-mon[20207]: pgmap v1995: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-08-30T23:55:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:49 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2478365925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:49 smithi136 ceph-mon[20207]: osdmap e898: 12 total, 12 up, 12 in 2024-08-30T23:55:49.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:49.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:49 smithi106 ceph-mon[16603]: pgmap v1995: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-08-30T23:55:49.307 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:49 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:49.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2478365925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:49.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:49 smithi106 ceph-mon[16603]: osdmap e898: 12 total, 12 up, 12 in 2024-08-30T23:55:49.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-08-30T23:55:49.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:49 smithi173 ceph-mon[20554]: pgmap v1995: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s 2024-08-30T23:55:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:49 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2478365925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:49 smithi173 ceph-mon[20554]: osdmap e898: 12 total, 12 up, 12 in 2024-08-30T23:55:49.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:49.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:49.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:49.554 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:49.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:49.691 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:49.793 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:49.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:49.860 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:49.860 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:49.892 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:49.892 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:49.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:49.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:49.923 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:49.953 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:49.953 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:49.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:49.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:49.980 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:49.980 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:50.008 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:50.012 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:50.039 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:50.040 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:50.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:50.069 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:50.069 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:50.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:50.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:50.108 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:50.108 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:50.141 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:50.141 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:50.142 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3891287708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:50.172 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:50.172 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:50.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:50.205 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:50.205 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.217 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:50.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:50.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3891287708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:50.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:50.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3891287708' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:50.345 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:50.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:50.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:50.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:50.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:50.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:50.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:50.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:55:50.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:50.859 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:50.870 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-30T23:55:50.869+0000 7f86d103c640 0 --2- 172.21.15.106:0/3436339918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86cc102660 0x7f86cc074760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-30T23:55:50.961 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:50.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:50.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:51.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:51.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:55:51.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:51.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.241 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:51.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:51 smithi136 ceph-mon[20207]: pgmap v1997: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 919 op/s 2024-08-30T23:55:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:51 smithi136 ceph-mon[20207]: osdmap e899: 12 total, 12 up, 12 in 2024-08-30T23:55:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:51 smithi173 ceph-mon[20554]: pgmap v1997: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 919 op/s 2024-08-30T23:55:51.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:51 smithi173 ceph-mon[20554]: osdmap e899: 12 total, 12 up, 12 in 2024-08-30T23:55:51.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:51 smithi106 ceph-mon[16603]: pgmap v1997: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 919 op/s 2024-08-30T23:55:51.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:51 smithi106 ceph-mon[16603]: osdmap e899: 12 total, 12 up, 12 in 2024-08-30T23:55:51.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.380 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:51.429 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:55:51.481 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:51.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:51.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:51.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:51.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:51.753 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:51.817 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:51.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:51.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:51.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:51.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:52.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:52.055 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:52.055 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:52.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.147 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:52.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:52.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:52.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:52.383 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:52.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:52.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:52.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:52.643 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:52.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:52.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:52.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:52.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:52.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:52.920 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:52.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:53.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:53.034 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:55:53.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:53.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:55:53.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:53.181 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:53.284 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:53.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:53 smithi136 ceph-mon[20207]: pgmap v1999: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-08-30T23:55:53.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2741368061' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:53.314 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:53.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:53 smithi173 ceph-mon[20554]: pgmap v1999: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-08-30T23:55:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2741368061' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:53 smithi106 ceph-mon[16603]: pgmap v1999: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s 2024-08-30T23:55:53.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2741368061' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:55:53.417 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:53.448 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.449 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:53.451 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:55:53.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:53.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:53.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:53.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:53.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:53.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:53.860 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:53.862 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:55:53.935 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:53.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:53.971 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:54.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:54.105 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.106 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:54.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:54.231 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:54.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:54 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:54.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2741368061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:54.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:54 smithi106 ceph-mon[16603]: osdmap e900: 12 total, 12 up, 12 in 2024-08-30T23:55:54.273 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:54 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:55:54.274 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:54 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:55:54.286 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:54 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:54.287 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2741368061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:54.287 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:54 smithi136 ceph-mon[20207]: osdmap e900: 12 total, 12 up, 12 in 2024-08-30T23:55:54.287 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:54 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:55:54.287 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:54 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:55:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:54 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:55:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2741368061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:55:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:54 smithi173 ceph-mon[20554]: osdmap e900: 12 total, 12 up, 12 in 2024-08-30T23:55:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:54 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:55:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:54 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:55:54.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:54.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:54.406 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:54.406 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:54.434 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:54.434 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:54.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:54.465 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:54.465 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:54.490 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:54.491 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:54.494 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:54.502 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:54.502 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:54.514 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:54.515 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:54.542 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:54.542 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:54.569 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:54.569 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:54.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:54.598 INFO:teuthology.orchestra.run.smithi106.stdout:osd.7: {} 2024-08-30T23:55:54.598 INFO:teuthology.orchestra.run.smithi106.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-30T23:55:54.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:54.626 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:54.626 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:54.661 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:54.661 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:54.689 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:54.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:54.723 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:54.724 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:54.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:54.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:54.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:54.884 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:54.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:54.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:55.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:55.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:55.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:55.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:55:55.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:55.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.281 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:55.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:55 smithi173 ceph-mon[20554]: pgmap v2001: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s 2024-08-30T23:55:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/839664662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:55 smithi106 ceph-mon[16603]: pgmap v2001: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s 2024-08-30T23:55:55.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/839664662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:55.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:55.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:55.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:55 smithi136 ceph-mon[20207]: pgmap v2001: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s 2024-08-30T23:55:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/839664662' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:55.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:55.547 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:55:55.642 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7 2024-08-30T23:55:55.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.7' 2024-08-30T23:55:55.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:55:55.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:55.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:55.892 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:55.926 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:55.927 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:56.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:56.051 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:56.148 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:56.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:56.163 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:55:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:55:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:124584 8.4 e901/840 8:2a00c657:::benchmark_data_smithi173_143816_object26270:head v 901'7717, mlcod=901'7715)", 2024-08-30T23:55:56.163 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:56.139578+0000", 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.012872912, 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012902952000000001, 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.139578+0000", 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.164 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.139578+0000", 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.139579+0000", 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2900000000000002e-07 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.139586+0000", 2024-08-30T23:55:56.165 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6050000000000003e-06 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.146547+0000", 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069618659999999997 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.146554+0000", 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8299999999999998e-06 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:56.166 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.146690+0000", 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013543100000000001 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.146742+0000", 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1866000000000001e-05 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.167 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:124592 8.1 e901/840 8:8daf2487:::benchmark_data_smithi173_143816_object26278:head v 901'7727, mlcod=901'7725)", 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:56.148110+0000", 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0043413410000000003, 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044329720000000003, 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.148110+0000", 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.168 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.148110+0000", 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.148110+0000", 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1500000000000004e-07 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.148117+0000", 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5860000000000001e-06 2024-08-30T23:55:56.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.148148+0000", 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1401e-05 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.148156+0000", 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2860000000000002e-06 2024-08-30T23:55:56.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.150092+0000", 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0019359570000000001 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.150112+0000", 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0537999999999999e-05 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.171 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:124594 8.c 8:3617ee20:::benchmark_data_smithi173_143816_object26280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:56.151451+0000", 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0010005649999999999, 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001152148, 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:55:56.172 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124594 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151451+0000", 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151451+0000", 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151452+0000", 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.387e-06 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151456+0000", 2024-08-30T23:55:56.174 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.715e-06 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151471+0000", 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5483000000000001e-05 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151478+0000", 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:56.175 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151880+0000", 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00040201499999999998 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.152006+0000", 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012558 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.152107+0000", 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101517 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.176 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:124591 8.a e901/840 8:5e4539d6:::benchmark_data_smithi173_143816_object26277:head v 901'7737, mlcod=901'7735)", 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:56.151433+0000", 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.001018288, 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001321565, 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.177 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151433+0000", 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151433+0000", 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.178 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151434+0000", 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7000000000000003e-07 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151442+0000", 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8320000000000006e-06 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151792+0000", 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00035053500000000002 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.179 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151799+0000", 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.669e-06 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151868+0000", 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8949999999999995e-05 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:56.180 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.151886+0000", 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8728000000000001e-05 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:124577 8.c 8:341e5b25:::benchmark_data_smithi173_143816_object26263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e901)", 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:55:56.132486+0000", 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.019964587999999998, 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.020320734999999999, 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:55:56.181 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 124577 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.132486+0000", 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:55:56.182 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.132486+0000", 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.132488+0000", 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.017e-06 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.132491+0000", 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.428e-06 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.132503+0000", 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2261e-05 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.132510+0000", 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0879999999999997e-06 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.138374+0000", 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058633660000000001 2024-08-30T23:55:56.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.138511+0000", 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013782700000000001 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,8", 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.138627+0000", 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000115631 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.142811+0000", 2024-08-30T23:55:56.185 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041842270000000004 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:55:56.144159+0000", 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013478609999999999 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 5 2024-08-30T23:55:56.186 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:55:56.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:56.298 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:55:56.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:56 smithi173 ceph-mon[20554]: osdmap e901: 12 total, 12 up, 12 in 2024-08-30T23:55:56.327 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:56.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:55:56.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:55:56.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:55:56.331 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:55:56.332 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:55:56.332 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:55:56.332 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:56 smithi106 ceph-mon[16603]: osdmap e901: 12 total, 12 up, 12 in 2024-08-30T23:55:56.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:56.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:55:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:56 smithi136 ceph-mon[20207]: osdmap e901: 12 total, 12 up, 12 in 2024-08-30T23:55:56.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:56.580 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:56.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:56.706 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:56.802 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:55:56.803 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:55:56.803 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 1, 5] 2024-08-30T23:55:56.803 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-30T23:55:56.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7 2024-08-30T23:55:56.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:55:56.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:56.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:55:56.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.000 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:57.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:57.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:55:57.127 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:55:57.153 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:55:57.157 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:56 smithi136 systemd[1]: Stopping Ceph osd.7 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-30T23:55:57.159 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:57 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[42416]: 2024-08-30T23:55:57.156+0000 7f7eea2d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-30T23:55:57.159 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:57 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[42416]: 2024-08-30T23:55:57.156+0000 7f7eea2d9640 -1 osd.7 902 *** Got signal Terminated *** 2024-08-30T23:55:57.159 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:57 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[42416]: 2024-08-30T23:55:57.156+0000 7f7eea2d9640 -1 osd.7 902 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-30T23:55:57.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:57.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:57.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:57.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:57.406 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:55:57.436 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1046204736' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:57.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:57.509 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1046204736' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1046204736' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:55:57.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:57.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:57.742 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:57.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:55:57.886 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:57.913 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:57.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:58.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:58.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:58.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:58.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:58.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:55:58.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:55:58.297 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:55:58.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:58.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:58.479 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:55:58.523 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-30T23:55:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1046204736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: osdmap e902: 12 total, 12 up, 12 in 2024-08-30T23:55:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: osd.7 marked itself down and dead 2024-08-30T23:55:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:55:58.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:58 smithi136 ceph-mon[20207]: osdmap e903: 12 total, 11 up, 12 in 2024-08-30T23:55:58.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1046204736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: osdmap e902: 12 total, 12 up, 12 in 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: osd.7 marked itself down and dead 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:55:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:58 smithi173 ceph-mon[20554]: osdmap e903: 12 total, 11 up, 12 in 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1046204736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: osdmap e902: 12 total, 12 up, 12 in 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: osd.7 marked itself down and dead 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-30T23:55:58.600 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:58 smithi106 ceph-mon[16603]: osdmap e903: 12 total, 11 up, 12 in 2024-08-30T23:55:58.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:58.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:58.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:55:58.790 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:55:58.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:58.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:58.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:58.945 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:55:58.945 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:55:58.966 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:55:58.967 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:55:58.987 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:55:58.987 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:55:59.007 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:55:59.008 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:55:59.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:55:59.029 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:55:59.029 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:55:59.036 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:55:59.050 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:55:59.051 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:55:59.071 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:55:59.072 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:55:59.072 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:55:59.092 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:55:59.093 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:55:59.108 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:58 smithi136 podman[154949]: 2024-08-30 23:55:58.811597815 +0000 UTC m=+1.790969804 container died e2ee6c3378a8e623e8d648d97753a59b69064e0982b90dba2dad0a28767f784a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-30T23:55:59.113 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:55:59.113 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:55:59.134 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:55:59.135 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:55:59.135 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:55:59.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:55:59.156 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:55:59.157 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:55:59.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:55:59.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:55:59.286 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:55:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:55:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:59 smithi106 ceph-mon[16603]: pgmap v2005: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s 2024-08-30T23:55:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:55:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1787095807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:59.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:55:59.413 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:55:59.424 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:59 smithi173 ceph-mon[20554]: pgmap v2005: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s 2024-08-30T23:55:59.424 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:55:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1787095807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:59.433 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:59 smithi136 podman[154949]: 2024-08-30 23:55:59.206451244 +0000 UTC m=+2.185823234 container cleanup e2ee6c3378a8e623e8d648d97753a59b69064e0982b90dba2dad0a28767f784a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-30T23:55:59.433 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:59 smithi136 bash[154949]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7 2024-08-30T23:55:59.433 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:59 smithi136 ceph-mon[20207]: pgmap v2005: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s 2024-08-30T23:55:59.433 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:55:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1787095807' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:55:59.517 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:59.543 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:59.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:55:59.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:55:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:55:59.719 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:55:59.720 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:55:59.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:55:59.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:59 smithi136 podman[155033]: 2024-08-30 23:55:59.433459474 +0000 UTC m=+0.616309811 container remove e2ee6c3378a8e623e8d648d97753a59b69064e0982b90dba2dad0a28767f784a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-30T23:55:59.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:55:59.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:55:59.937 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:55:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:55:59.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:00.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:00.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:00.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:00.045 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:59 smithi136 podman[155168]: 2024-08-30 23:55:59.695311536 +0000 UTC m=+0.017529472 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-30T23:56:00.045 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:55:59 smithi136 podman[155168]: 2024-08-30 23:55:59.801038121 +0000 UTC m=+0.123256061 container create d7bc585edf573817a877b401e3bca83bf17152dc4833f2f94d924da1d8f73ca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-08-30T23:56:00.064 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:00.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:00.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:00.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:00.322 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:00.340 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:00 smithi136 ceph-mon[20207]: osdmap e904: 12 total, 11 up, 12 in 2024-08-30T23:56:00.340 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:00 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.340 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:00 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.340 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:56:00.340 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:56:00.340 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:00 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.341 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:56:00 smithi136 podman[155168]: 2024-08-30 23:56:00.045571402 +0000 UTC m=+0.367789333 container init d7bc585edf573817a877b401e3bca83bf17152dc4833f2f94d924da1d8f73ca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2) 2024-08-30T23:56:00.341 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:56:00 smithi136 podman[155168]: 2024-08-30 23:56:00.0499146 +0000 UTC m=+0.372132532 container start d7bc585edf573817a877b401e3bca83bf17152dc4833f2f94d924da1d8f73ca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-30T23:56:00.341 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:56:00 smithi136 podman[155168]: 2024-08-30 23:56:00.162318425 +0000 UTC m=+0.484536355 container attach d7bc585edf573817a877b401e3bca83bf17152dc4833f2f94d924da1d8f73ca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=) 2024-08-30T23:56:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:00 smithi106 ceph-mon[16603]: osdmap e904: 12 total, 11 up, 12 in 2024-08-30T23:56:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:00 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:00 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:56:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:56:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:00 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.379 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:56:00.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:56:00.424 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:00.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:00 smithi173 ceph-mon[20554]: osdmap e904: 12 total, 11 up, 12 in 2024-08-30T23:56:00.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:00 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.432 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:00 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:56:00.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:56:00.433 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:00 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:56:00.469 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.470 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:00.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:00.607 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:00.698 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:00.699 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 30 23:56:00 smithi136 podman[155168]: 2024-08-30 23:56:00.340867033 +0000 UTC m=+0.663084966 container died d7bc585edf573817a877b401e3bca83bf17152dc4833f2f94d924da1d8f73ca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-30T23:56:00.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:00.754 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:00.805 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:56:00.856 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:00.882 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:00.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:00.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:01.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:01.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7.service' 2024-08-30T23:56:01.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-30T23:56:01.104 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-30T23:56:01.104 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-08-30T23:56:01.104 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi136.front.sepia.ceph.com 2024-08-30T23:56:01.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.7 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-08-30T23:56:01.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:01.148 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:01.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:01.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:01 smithi173 ceph-mon[20554]: pgmap v2008: 47 pgs: 4 active+undersized, 1 active+undersized+degraded, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 886 op/s; 1/214161 objects degraded (0.000%); 21 KiB/s, 0 objects/s recovering 2024-08-30T23:56:01.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:01.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:01.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:01 smithi173 ceph-mon[20554]: Health check failed: Degraded data redundancy: 1/214161 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:56:01.279 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:01 smithi173 ceph-mon[20554]: osdmap e905: 12 total, 11 up, 12 in 2024-08-30T23:56:01.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:01 smithi136 ceph-mon[20207]: pgmap v2008: 47 pgs: 4 active+undersized, 1 active+undersized+degraded, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 886 op/s; 1/214161 objects degraded (0.000%); 21 KiB/s, 0 objects/s recovering 2024-08-30T23:56:01.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:01 smithi136 ceph-mon[20207]: Health check failed: Degraded data redundancy: 1/214161 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:56:01.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:01 smithi136 ceph-mon[20207]: osdmap e905: 12 total, 11 up, 12 in 2024-08-30T23:56:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:01 smithi106 ceph-mon[16603]: pgmap v2008: 47 pgs: 4 active+undersized, 1 active+undersized+degraded, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 886 op/s; 1/214161 objects degraded (0.000%); 21 KiB/s, 0 objects/s recovering 2024-08-30T23:56:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:01 smithi106 ceph-mon[16603]: Health check failed: Degraded data redundancy: 1/214161 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-08-30T23:56:01.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:01 smithi106 ceph-mon[16603]: osdmap e905: 12 total, 11 up, 12 in 2024-08-30T23:56:01.366 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.7/config 2024-08-30T23:56:01.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:01.417 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:01.426 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:01.426 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:01.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:01.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:01.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.677 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:01.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:01.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:01.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:01.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:01.929 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:01.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:02.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:56:02.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:02.073 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:02.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:02.207 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:02.302 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4055465118' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:02.302 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:02 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:02.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:02.317 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:56:02.318 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4055465118' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:02.318 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:02 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:02.327 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:02.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:02.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:02.475 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:02.515 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4055465118' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:02.515 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:02 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:02.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:02.598 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:56:02.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:02.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:02.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:02.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:02.865 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:02.967 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:02.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:02.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:03.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:03.124 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:03.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:03.258 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:03 smithi106 ceph-mon[16603]: pgmap v2010: 47 pgs: 10 active+undersized, 3 active+undersized+degraded, 34 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 662 op/s; 8648/207654 objects degraded (4.165%); 18 KiB/s, 0 objects/s recovering 2024-08-30T23:56:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:03 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:03 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:03 smithi106 ceph-mon[16603]: osdmap e906: 12 total, 11 up, 12 in 2024-08-30T23:56:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:03.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:03.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:03.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:03 smithi173 ceph-mon[20554]: pgmap v2010: 47 pgs: 10 active+undersized, 3 active+undersized+degraded, 34 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 662 op/s; 8648/207654 objects degraded (4.165%); 18 KiB/s, 0 objects/s recovering 2024-08-30T23:56:03.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:03 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:03.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:03 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:03.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:03 smithi173 ceph-mon[20554]: osdmap e906: 12 total, 11 up, 12 in 2024-08-30T23:56:03.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:03.441 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:03.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:56:03.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:03.512 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:03 smithi136 ceph-mon[20207]: pgmap v2010: 47 pgs: 10 active+undersized, 3 active+undersized+degraded, 34 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 662 op/s; 8648/207654 objects degraded (4.165%); 18 KiB/s, 0 objects/s recovering 2024-08-30T23:56:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:03 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:03 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:03 smithi136 ceph-mon[20207]: osdmap e906: 12 total, 11 up, 12 in 2024-08-30T23:56:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:03.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:03.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:03.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:03.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:03.787 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:56:03.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:03.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:03.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:03.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:04.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:04.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:04.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:04.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:04.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:04.337 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:04.338 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:04.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:04.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:04.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:04.554 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:04 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:04 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:04 smithi173 ceph-mon[20554]: osdmap e907: 12 total, 11 up, 12 in 2024-08-30T23:56:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:04 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:04 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:04 smithi106 ceph-mon[16603]: osdmap e907: 12 total, 11 up, 12 in 2024-08-30T23:56:04.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:04.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:04 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:04 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:04 smithi136 ceph-mon[20207]: osdmap e907: 12 total, 11 up, 12 in 2024-08-30T23:56:04.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:04.818 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:04.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:04.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:04.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:05.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:05.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:56:05.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:05.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:56:05.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:05.229 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:05.320 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:05.320 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:05.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:05.349 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:05.349 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:05.370 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:05.376 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:05.377 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:05.409 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:05.409 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:05.435 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:05.435 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:05.468 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:05.468 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:05.469 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:56:05.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:05.495 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:05.496 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:05.496 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:05.501 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:05.507 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:05.527 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:05.528 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:05.564 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:05.565 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:05 smithi173 ceph-mon[20554]: pgmap v2013: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 775 op/s; 8648/205365 objects degraded (4.211%); 18 KiB/s, 0 objects/s recovering 2024-08-30T23:56:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/309546200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:05.595 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:05.595 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:05.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:05 smithi106 ceph-mon[16603]: pgmap v2013: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 775 op/s; 8648/205365 objects degraded (4.211%); 18 KiB/s, 0 objects/s recovering 2024-08-30T23:56:05.596 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/309546200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:05.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:05.630 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:05.630 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:05.640 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:05.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:05.765 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:05.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:05 smithi136 ceph-mon[20207]: pgmap v2013: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 775 op/s; 8648/205365 objects degraded (4.211%); 18 KiB/s, 0 objects/s recovering 2024-08-30T23:56:05.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/309546200' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:05.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:05.892 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:05.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:05.994 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:06.021 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:06.045 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:56:06.045 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:56:06.045 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:56:06.045 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:56:06.045 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:56:06.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:06.127 INFO:teuthology.orchestra.run.smithi136.stdout:8.d 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.10 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:8.6 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.15 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.d 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.18 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.1a 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.9 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.1e 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.e 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.0 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.2 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.14 2024-08-30T23:56:06.128 INFO:teuthology.orchestra.run.smithi136.stdout:1.4 2024-08-30T23:56:06.146 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:06.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:06.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8648/205365 objects degraded (4.211%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:06.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:06.396 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:06.451 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:06.468 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8648/205365 objects degraded (4.211%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:06.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:06.522 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:06.624 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:06.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:06.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:06.781 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8648/205365 objects degraded (4.211%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:06.787 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:06.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:56:06.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:06.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:06.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:07.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:07.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:07.162 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:07.185 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:07.206 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:56:07.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.7 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 1.1a --file /var/log/ceph/exp.1.1a.7 2024-08-30T23:56:07.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:07.327 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:56:07.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:07.431 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:07.454 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:07.477 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.7/config 2024-08-30T23:56:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:07 smithi136 ceph-mon[20207]: pgmap v2014: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 8648/200148 objects degraded (4.321%) 2024-08-30T23:56:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3536704155' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:07.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:07.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:07 smithi106 ceph-mon[16603]: pgmap v2014: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 8648/200148 objects degraded (4.321%) 2024-08-30T23:56:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3536704155' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:07.641 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:07 smithi173 ceph-mon[20554]: pgmap v2014: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 778 op/s; 8648/200148 objects degraded (4.321%) 2024-08-30T23:56:07.641 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3536704155' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:07.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:07.714 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:07.817 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:07.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:07.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:07.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:07.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:08.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:08.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:08.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37699 8.d 8:b57c4cbd:::benchmark_data_smithi173_143816_object37698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664969393000007, 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93994113199999996, 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.191 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37699 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.192 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121523+0000", 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3480000000000001e-06 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121550+0000", 2024-08-30T23:56:08.193 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7135e-05 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121561+0000", 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.081e-05 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121569+0000", 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6990000000000003e-06 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.194 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3838999999999996e-05 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121654+0000", 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1416999999999999e-05 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061408+0000", 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93975412199999997 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.195 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061463+0000", 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4762000000000002e-05 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37724 8.d 8:bb898ecc:::benchmark_data_smithi173_143816_object37723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664900357999997, 2024-08-30T23:56:08.196 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94419919900000004, 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37724 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121616+0000", 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4661999999999999e-05 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121625+0000", 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6009999999999995e-06 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121631+0000", 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500000000000001e-06 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.199 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121691+0000", 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9901000000000001e-05 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121695+0000", 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6849999999999999e-06 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061884+0000", 2024-08-30T23:56:08.200 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94018891100000002 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061957+0000", 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2855999999999995e-05 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.062054+0000", 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7165999999999998e-05 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.201 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065642+0000", 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035878989999999999 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065656+0000", 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4899000000000001e-05 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065728+0000", 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1124999999999994e-05 2024-08-30T23:56:08.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065764+0000", 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6154e-05 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065790+0000", 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6273000000000001e-05 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37735 8.d 8:bd387e6f:::benchmark_data_smithi173_143816_object37734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664847774999998, 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94428094500000004, 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.204 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37735 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.205 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121644+0000", 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9500000000000002e-07 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121680+0000", 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6387000000000002e-05 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121688+0000", 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5839999999999997e-06 2024-08-30T23:56:08.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121694+0000", 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3119999999999999e-06 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121745+0000", 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0837999999999997e-05 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121749+0000", 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6150000000000001e-06 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063145+0000", 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94139648399999998 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063233+0000", 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199e-05 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.208 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063310+0000", 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6536e-05 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065662+0000", 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023518760000000001 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065685+0000", 2024-08-30T23:56:08.209 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2708999999999999e-05 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065841+0000", 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015618500000000001 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065890+0000", 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.888e-05 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.210 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065924+0000", 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4745e-05 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37745 8.d 8:b0c83413:::benchmark_data_smithi173_143816_object37744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664769930000006, 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94438678600000003, 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.211 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37745 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:56:08.212 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121722+0000", 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6700000000000003e-07 2024-08-30T23:56:08.213 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121746+0000", 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4036000000000002e-05 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121752+0000", 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2570000000000004e-06 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121756+0000", 2024-08-30T23:56:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9620000000000004e-06 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121787+0000", 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0587999999999999e-05 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121790+0000", 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3519999999999998e-06 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.215 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063974+0000", 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94218355200000004 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064046+0000", 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1983999999999994e-05 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064119+0000", 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3266e-05 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.216 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065799+0000", 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001680186 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065979+0000", 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179552 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066033+0000", 2024-08-30T23:56:08.217 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4085999999999997e-05 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066077+0000", 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4382000000000003e-05 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066108+0000", 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1016000000000003e-05 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37783 8.d 8:bd0ef9a3:::benchmark_data_smithi173_143816_object37782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664726868000002, 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94672526099999998, 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.219 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37783 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:56:08.220 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121765+0000", 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4799999999999999e-07 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.221 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121789+0000", 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4697000000000001e-05 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121795+0000", 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1429999999999997e-06 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.222 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121798+0000", 2024-08-30T23:56:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.472e-06 2024-08-30T23:56:08.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121834+0000", 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5528000000000003e-05 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121842+0000", 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3159999999999997e-06 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.224 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064918+0000", 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.943076154 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064992+0000", 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3893999999999994e-05 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065076+0000", 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3681000000000001e-05 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.225 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065991+0000", 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091558099999999995 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068378+0000", 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023872390000000002 2024-08-30T23:56:08.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068428+0000", 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9329999999999997e-05 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068464+0000", 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5917999999999997e-05 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068490+0000", 2024-08-30T23:56:08.227 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5959999999999999e-05 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37794 8.d 8:b33c5833:::benchmark_data_smithi173_143816_object37793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664685210000002, 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94686707599999997, 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.228 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37794 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5900000000000003e-07 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121850+0000", 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3831000000000001e-05 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121856+0000", 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2670000000000003e-06 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121861+0000", 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8799999999999999e-06 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121892+0000", 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1179000000000001e-05 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121896+0000", 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3780000000000001e-06 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066144+0000", 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94424862499999995 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066245+0000", 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010058800000000001 2024-08-30T23:56:08.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066373+0000", 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012802199999999999 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068538+0000", 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021646870000000002 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068591+0000", 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3541000000000001e-05 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068604+0000", 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2607e-05 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068648+0000", 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3781e-05 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068673+0000", 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5330999999999999e-05 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37798 8.d 8:b32c1d3f:::benchmark_data_smithi173_143816_object37797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664621890999996, 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96212161299999999, 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.235 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37798 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:56:08.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121870+0000", 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0800000000000001e-07 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121910+0000", 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0589000000000001e-05 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121916+0000", 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.981e-06 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121920+0000", 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5719999999999999e-06 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121956+0000", 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6510000000000001e-05 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121970+0000", 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3285000000000001e-05 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067392+0000", 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94542252100000002 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067488+0000", 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5859000000000005e-05 2024-08-30T23:56:08.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067570+0000", 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068681+0000", 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011110720000000001 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074538+0000", 2024-08-30T23:56:08.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058571530000000004 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083860+0000", 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0093212599999999996 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083936+0000", 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6007999999999995e-05 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083991+0000", 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5384999999999999e-05 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37806 8.d 8:b567e58a:::benchmark_data_smithi173_143816_object37805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664562695000001, 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96731055799999999, 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.242 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37806 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121929+0000", 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.22e-07 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122024+0000", 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5063999999999996e-05 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122031+0000", 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2060000000000001e-06 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122035+0000", 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7960000000000002e-06 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122067+0000", 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2122999999999997e-05 2024-08-30T23:56:08.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122072+0000", 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5140000000000003e-06 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068710+0000", 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94663852100000001 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068855+0000", 2024-08-30T23:56:08.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014504599999999999 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068990+0000", 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135177 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074497+0000", 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055070910000000004 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074576+0000", 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109000000000004e-05 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089138+0000", 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014561274000000001 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089193+0000", 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4864000000000002e-05 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089239+0000", 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6451e-05 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37807 8.d 8:bd227052:::benchmark_data_smithi173_143816_object37806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664448050000004, 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96736628899999999, 2024-08-30T23:56:08.249 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37807 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122079+0000", 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5790000000000001e-05 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122085+0000", 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9170000000000003e-06 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122088+0000", 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.084e-06 2024-08-30T23:56:08.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122122+0000", 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3738999999999997e-05 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2359999999999998e-06 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.253 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.072882+0000", 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95075315999999999 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073020+0000", 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013803900000000001 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073142+0000", 2024-08-30T23:56:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122043 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077477+0000", 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043347180000000004 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077486+0000", 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0759999999999992e-06 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089319+0000", 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011832666 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089368+0000", 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9446000000000001e-05 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089409+0000", 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1060999999999999e-05 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37812 8.d 8:bad38160:::benchmark_data_smithi173_143816_object37811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664397417000004, 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.967454445, 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.257 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37812 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8599999999999999e-07 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122123+0000", 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8960000000000001e-05 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5520000000000004e-06 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122132+0000", 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5379999999999998e-06 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122161+0000", 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9351000000000001e-05 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122166+0000", 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3440000000000003e-06 2024-08-30T23:56:08.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074606+0000", 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95243996799999997 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074746+0000", 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014037899999999999 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074867+0000", 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120969 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077539+0000", 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026718330000000002 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078510+0000", 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097061800000000002 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089459+0000", 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010949181000000001 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089509+0000", 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0547999999999998e-05 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089548+0000", 2024-08-30T23:56:08.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8918000000000002e-05 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37816 8.d 8:bceb9fa7:::benchmark_data_smithi173_143816_object37815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664353384999998, 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96754795299999996, 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.265 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37816 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6e-07 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122168+0000", 2024-08-30T23:56:08.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9663e-05 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122172+0000", 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-06 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122176+0000", 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2400000000000001e-06 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122205+0000", 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9184999999999999e-05 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122212+0000", 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1849999999999998e-06 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076169+0000", 2024-08-30T23:56:08.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95395610799999997 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076291+0000", 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122633 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076448+0000", 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015706699999999999 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079517+0000", 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003068777 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079533+0000", 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5656000000000001e-05 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089594+0000", 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010061151000000001 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089648+0000", 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4110999999999997e-05 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089686+0000", 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7762000000000002e-05 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37819 8.d 8:b27be7f0:::benchmark_data_smithi173_143816_object37818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664309587999995, 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96865621300000004, 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.273 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37819 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.72e-07 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122214+0000", 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1964000000000001e-05 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122218+0000", 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1649999999999999e-06 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122222+0000", 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7179999999999998e-06 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122251+0000", 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8810999999999999e-05 2024-08-30T23:56:08.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122254+0000", 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8410000000000002e-06 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077587+0000", 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95533246100000002 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077667+0000", 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0438000000000005e-05 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077751+0000", 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4217999999999999e-05 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079574+0000", 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018224580000000001 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080426+0000", 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085253 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090743+0000", 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010316967 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090798+0000", 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4866e-05 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090838+0000", 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9504e-05 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37827 8.d 8:bc586ec9:::benchmark_data_smithi173_143816_object37826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.280 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664262786999998, 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96986781300000002, 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37827 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122229+0000", 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8799999999999998e-07 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122263+0000", 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3946999999999999e-05 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122268+0000", 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122271+0000", 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1879999999999998e-06 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122300+0000", 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8824e-05 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122303+0000", 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4249999999999998e-06 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078551+0000", 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95624797399999995 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078639+0000", 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078716+0000", 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6747000000000002e-05 2024-08-30T23:56:08.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081587+0000", 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028711209999999999 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081602+0000", 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4864e-05 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.091996+0000", 2024-08-30T23:56:08.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010393823 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092053+0000", 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7574000000000001e-05 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092096+0000", 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2870000000000001e-05 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.287 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37830 8.d 8:b56bd01c:::benchmark_data_smithi173_143816_object37829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664213497999995, 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97002787499999998, 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.288 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37830 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8500000000000002e-07 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122306+0000", 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8231000000000001e-05 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122311+0000", 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:56:08.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122314+0000", 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-06 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122348+0000", 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3624999999999997e-05 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122359+0000", 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0685e-05 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079614+0000", 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95725496799999998 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079729+0000", 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114969 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079807+0000", 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8337000000000006e-05 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083762+0000", 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039546900000000003 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083812+0000", 2024-08-30T23:56:08.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0781000000000001e-05 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092177+0000", 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083641789999999994 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092265+0000", 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8883999999999998e-05 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092306+0000", 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0092999999999997e-05 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37833 8.d 8:bd5be583:::benchmark_data_smithi173_143816_object37832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664169689999994, 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97017067000000001, 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.295 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37833 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122322+0000", 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122349+0000", 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7472999999999999e-05 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122356+0000", 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7889999999999997e-06 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122360+0000", 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8330000000000004e-06 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122389+0000", 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.879e-05 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122393+0000", 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1980000000000003e-06 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080473+0000", 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95808044999999997 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080581+0000", 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010725700000000001 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080712+0000", 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013142199999999999 2024-08-30T23:56:08.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084054+0000", 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003342434 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084069+0000", 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4487e-05 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092412+0000", 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083428129999999993 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092458+0000", 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6440999999999998e-05 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092492+0000", 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4019000000000001e-05 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.302 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37856 8.d 8:b1ef2347:::benchmark_data_smithi173_143816_object37855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 99.664125163999998, 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97027297800000001, 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37856 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122397+0000", 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994000000000002e-05 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122401+0000", 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1439999999999996e-06 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122406+0000", 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:56:08.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122438+0000", 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2008000000000003e-05 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122445+0000", 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081647+0000", 2024-08-30T23:56:08.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95920211399999999 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081737+0000", 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0278999999999994e-05 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081881+0000", 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144036 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084081+0000", 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021993759999999998 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084140+0000", 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9562999999999999e-05 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092544+0000", 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084042449999999994 2024-08-30T23:56:08.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092598+0000", 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3677999999999997e-05 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092639+0000", 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1013999999999997e-05 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45699 8.7 8:eae4016d:::benchmark_data_smithi173_143816_object45698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 90.785382474000002, 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64772118199999995, 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45699 2024-08-30T23:56:08.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9100000000000003e-07 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001164+0000", 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4604000000000001e-05 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001173+0000", 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9269999999999994e-06 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001177+0000", 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4950000000000002e-06 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001209+0000", 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1758999999999999e-05 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001213+0000", 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9600000000000002e-06 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642148+0000", 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64093464099999997 2024-08-30T23:56:08.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642302+0000", 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015428999999999999 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642473+0000", 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000170843 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646115+0000", 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036422680000000002 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648167+0000", 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002051911 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648704+0000", 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053669899999999997 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648748+0000", 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4098999999999999e-05 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648830+0000", 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1995000000000004e-05 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45725 8.7 8:e95f37b8:::benchmark_data_smithi173_143816_object45724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 90.785304307999994, 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64797183599999997, 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45725 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.319 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001188+0000", 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9999999999999997e-07 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001240+0000", 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120999999999998e-05 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001248+0000", 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6430000000000001e-06 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001253+0000", 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4989999999999996e-06 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001286+0000", 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3328999999999999e-05 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001290+0000", 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0090000000000001e-06 2024-08-30T23:56:08.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.643899+0000", 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64260842600000001 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644052+0000", 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015375600000000001 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:56:08.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644192+0000", 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013943899999999999 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646441+0000", 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022493819999999999 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648867+0000", 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024262860000000002 2024-08-30T23:56:08.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649035+0000", 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167052 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649081+0000", 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6409999999999998e-05 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649159+0000", 2024-08-30T23:56:08.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7783999999999999e-05 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:132893 8.3 8:c1a30103:::benchmark_data_smithi173_143816_object34579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e907)", 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:06.063468+0000", 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.723023593, 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112661863, 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.326 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 132893 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063468+0000", 2024-08-30T23:56:08.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063468+0000", 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063469+0000", 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.235e-06 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063474+0000", 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9479999999999999e-06 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063483+0000", 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5189999999999998e-06 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063488+0000", 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7589999999999997e-06 2024-08-30T23:56:08.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.063992+0000", 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00050387299999999995 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.064119+0000", 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012682700000000001 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,5", 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.064241+0000", 2024-08-30T23:56:08.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121766 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.067901+0000", 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036606020000000002 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.084474+0000", 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016572967000000001 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.176036+0000", 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.091562213000000003 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.176089+0000", 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2738999999999998e-05 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.176129+0000", 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0414999999999998e-05 2024-08-30T23:56:08.332 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:132893 8.3 e907/862 ondisk, result = 0)", 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:06.067391+0000", 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.719100367, 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10874751100000001, 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:56:08.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.067391+0000", 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.067391+0000", 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.067392+0000", 2024-08-30T23:56:08.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1100000000000004e-07 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.067396+0000", 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4680000000000003e-06 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.175833+0000", 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10843641900000001 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:56:08.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.175839+0000", 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8360000000000003e-06 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.176023+0000", 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000183215 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.176030+0000", 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7319999999999998e-06 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:56:08.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:06.176138+0000", 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010793000000000001 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:56:08.337 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:56:08.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:08.361 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:08.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:08.452 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:56:08.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:08.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:08.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:08 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3536704155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:08 smithi173 ceph-mon[20554]: osdmap e908: 12 total, 11 up, 12 in 2024-08-30T23:56:08.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:08 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3536704155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:08 smithi106 ceph-mon[16603]: osdmap e908: 12 total, 11 up, 12 in 2024-08-30T23:56:08.618 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:08.619 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:08.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:08 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:08.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3536704155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:08.689 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:08 smithi136 ceph-mon[20207]: osdmap e908: 12 total, 11 up, 12 in 2024-08-30T23:56:08.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:08.731 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:56:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:08.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:08.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:08.758 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:08.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:08.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:08.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:08.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:09.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:09.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:09.127 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:09.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:09.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:56:09.256 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.257 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:09.358 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:09.385 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:09.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:09.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-30T23:56:09.476+0000 7fae6a59c640 0 --2- 172.21.15.106:0/408194573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae6c109110 0x7fae6c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-30T23:56:09.478 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:09.478 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:09.478 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:09 smithi106 ceph-mon[16603]: pgmap v2016: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 8055/191772 objects degraded (4.200%) 2024-08-30T23:56:09.478 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:09 smithi106 ceph-mon[16603]: osdmap e909: 12 total, 11 up, 12 in 2024-08-30T23:56:09.487 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:09.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:09.520 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:09.520 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:09.551 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:09.552 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:09 smithi173 ceph-mon[20554]: pgmap v2016: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 8055/191772 objects degraded (4.200%) 2024-08-30T23:56:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:09 smithi173 ceph-mon[20554]: osdmap e909: 12 total, 11 up, 12 in 2024-08-30T23:56:09.586 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:09.587 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:09.587 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:09.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:09.619 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:09.620 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:09.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:09.649 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:09.649 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:09.676 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:09.677 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:09.705 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:09.706 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:09.706 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:09.739 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:09.739 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:09.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:09.768 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:09.768 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:09.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:09 smithi136 ceph-mon[20207]: pgmap v2016: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 882 op/s; 8055/191772 objects degraded (4.200%) 2024-08-30T23:56:09.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:09 smithi136 ceph-mon[20207]: osdmap e909: 12 total, 11 up, 12 in 2024-08-30T23:56:09.802 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:09.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:09.831 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:09.831 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:09.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:09.900 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:09.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:10.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:10.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:56:10.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:10.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:10.153 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:10.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:10.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:10.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:10.412 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:10.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:10.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:10.537 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/408194573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:10.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:10.650 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:10.650 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:10.686 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:10.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:10.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/408194573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:10.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/408194573' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:10.826 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:10.918 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:10.928 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:10.955 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:10.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:11.057 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:11.090 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:11.192 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:11.218 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:11.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.351 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:11.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:11.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:11.477 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:11.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:56:11.576 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:11 smithi106 ceph-mon[16603]: pgmap v2018: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s; 8055/188355 objects degraded (4.276%) 2024-08-30T23:56:11.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8055/188355 objects degraded (4.276%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:11.579 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:11.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:11.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:11.736 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:11 smithi136 ceph-mon[20207]: pgmap v2018: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s; 8055/188355 objects degraded (4.276%) 2024-08-30T23:56:11.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8055/188355 objects degraded (4.276%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:11 smithi173 ceph-mon[20554]: pgmap v2018: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 871 op/s; 8055/188355 objects degraded (4.276%) 2024-08-30T23:56:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8055/188355 objects degraded (4.276%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:11.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:11.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:11.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:11.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:11.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:12.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.130 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:12.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:12.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:12.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:12.258 INFO:teuthology.orchestra.run.smithi136.stderr:Exporting 1.1a info 1.1a( v 23'92 (0'0,23'92] local-lis/les=879/880 n=0 ec=877/20 lis/c=879/879 les/c/f=880/880/0 sis=879) 2024-08-30T23:56:12.258 INFO:teuthology.orchestra.run.smithi136.stderr:Export successful 2024-08-30T23:56:12.357 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:56:12.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.391 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:12.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:12.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:12.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:12.524 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2357991684' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:12 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2357991684' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:12.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:12 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:12.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:12.657 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:12.709 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:12.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.784 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:12.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:12.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2357991684' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:12.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:12 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:12.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:12.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:12.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:13.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.040 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:13.040 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:13.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.166 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:13.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:13.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:13.294 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:13.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:13.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:13.427 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:13 smithi106 ceph-mon[16603]: pgmap v2019: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 7489/180702 objects degraded (4.144%) 2024-08-30T23:56:13.427 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:13 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:13.427 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:13 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:56:13.427 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:13 smithi106 ceph-mon[16603]: osdmap e910: 12 total, 11 up, 12 in 2024-08-30T23:56:13.428 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.7 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 1.1a 2024-08-30T23:56:13.522 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:13.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.553 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:13.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:13.686 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:13.691 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.7/config 2024-08-30T23:56:13.747 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:13.747 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:13.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:13 smithi136 ceph-mon[20207]: pgmap v2019: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 7489/180702 objects degraded (4.144%) 2024-08-30T23:56:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:13 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:13 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:56:13.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:13 smithi136 ceph-mon[20207]: osdmap e910: 12 total, 11 up, 12 in 2024-08-30T23:56:13.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:13 smithi173 ceph-mon[20554]: pgmap v2019: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.03k op/s; 7489/180702 objects degraded (4.144%) 2024-08-30T23:56:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:13 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:13 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:56:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:13 smithi173 ceph-mon[20554]: osdmap e910: 12 total, 11 up, 12 in 2024-08-30T23:56:13.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:13.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:13.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:14.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:14.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:14.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:14.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:14.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:14.321 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:14.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:56:14.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:14.482 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:14.495 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:56:14.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:14.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:14 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:56:14.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:14 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:56:14.615 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:14.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.740 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:14.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:14.763 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:14.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:14 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:56:14.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:14 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:56:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:14 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:56:14.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:14 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:56:14.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:14.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:14.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:14.959 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:56:14.976 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:15.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:15.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:15.134 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:15.214 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:56:15.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:15.299 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:15.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:15.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.431 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:15.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:15.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:15.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:15 smithi136 ceph-mon[20207]: pgmap v2021: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 7489/178236 objects degraded (4.202%) 2024-08-30T23:56:15.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:15 smithi136 ceph-mon[20207]: 8.5 scrub starts 2024-08-30T23:56:15.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:15 smithi106 ceph-mon[16603]: pgmap v2021: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 7489/178236 objects degraded (4.202%) 2024-08-30T23:56:15.546 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:15 smithi106 ceph-mon[16603]: 8.5 scrub starts 2024-08-30T23:56:15.568 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:15.580 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:15.580 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:15.670 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:15.797 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:15.821 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:15.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:15 smithi173 ceph-mon[20554]: pgmap v2021: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 695 op/s; 7489/178236 objects degraded (4.202%) 2024-08-30T23:56:15.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:15 smithi173 ceph-mon[20554]: 8.5 scrub starts 2024-08-30T23:56:15.922 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:15.950 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:15.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:16.052 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:16.083 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:16.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:16.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:16.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:56:16.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:16.345 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:16.346 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:16.400 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:16.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:16.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:16.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:16.575 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:16.599 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:16 smithi106 ceph-mon[16603]: 8.5 scrub ok 2024-08-30T23:56:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:16 smithi106 ceph-mon[16603]: 8.1 scrub starts 2024-08-30T23:56:16.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7489/178236 objects degraded (4.202%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:16.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:16 smithi106 ceph-mon[16603]: osdmap e911: 12 total, 11 up, 12 in 2024-08-30T23:56:16.613 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:56:16.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:16.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:16.738 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:16 smithi136 ceph-mon[20207]: 8.5 scrub ok 2024-08-30T23:56:16.738 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:16 smithi136 ceph-mon[20207]: 8.1 scrub starts 2024-08-30T23:56:16.738 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7489/178236 objects degraded (4.202%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:16.738 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:16 smithi136 ceph-mon[20207]: osdmap e911: 12 total, 11 up, 12 in 2024-08-30T23:56:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:16 smithi173 ceph-mon[20554]: 8.5 scrub ok 2024-08-30T23:56:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:16 smithi173 ceph-mon[20554]: 8.1 scrub starts 2024-08-30T23:56:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7489/178236 objects degraded (4.202%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:16 smithi173 ceph-mon[20554]: osdmap e911: 12 total, 11 up, 12 in 2024-08-30T23:56:16.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:16.855 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:16.957 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:16.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:16.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:17.083 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:17.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:17.196 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:17.196 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:17.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:17.248 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:17.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:17.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:17.466 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:17.466 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:17.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: pgmap v2022: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 7489/172977 objects degraded (4.329%) 2024-08-30T23:56:17.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 1.16 scrub starts 2024-08-30T23:56:17.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 1.16 scrub ok 2024-08-30T23:56:17.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:56:17.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:56:17.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 1.1c scrub starts 2024-08-30T23:56:17.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 1.1c scrub ok 2024-08-30T23:56:17.467 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:17 smithi106 ceph-mon[16603]: 8.1 scrub ok 2024-08-30T23:56:17.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:17.496 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:17.497 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:17.511 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:17.528 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:17.528 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:17.555 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:17.555 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:17.584 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:17.584 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:17.607 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:17.607 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:17.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:17.636 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:17.636 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:17.637 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:17.638 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:17.666 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:17.666 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:17.692 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:17.692 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:17.720 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:17.720 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:17.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:17.749 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:17.749 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:17.767 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: pgmap v2022: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 7489/172977 objects degraded (4.329%) 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 1.16 scrub starts 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 1.16 scrub ok 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 1.1c scrub starts 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 1.1c scrub ok 2024-08-30T23:56:17.768 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:17 smithi136 ceph-mon[20207]: 8.1 scrub ok 2024-08-30T23:56:17.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: pgmap v2022: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 782 op/s; 7489/172977 objects degraded (4.329%) 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 1.16 scrub starts 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 1.16 scrub ok 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 1.1c scrub starts 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 1.1c scrub ok 2024-08-30T23:56:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:17 smithi173 ceph-mon[20554]: 8.1 scrub ok 2024-08-30T23:56:17.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:17.901 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:17.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:17.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:56:18.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:18.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:18.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:18.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:18.235 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:56:18.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:18.287 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: 8.4 scrub starts 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3849897747' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: 1.1b scrub starts 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: 1.1b scrub ok 2024-08-30T23:56:18.379 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2229359692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:18.379 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:56:18.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.414 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:18.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:18.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:18.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:18.574 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:18.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:18.674 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:18.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: 8.4 scrub starts 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3849897747' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: 1.1b scrub starts 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: 1.1b scrub ok 2024-08-30T23:56:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2229359692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:18.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:18.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: 8.4 scrub starts 2024-08-30T23:56:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:56:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:56:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3849897747' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: 1.1b scrub starts 2024-08-30T23:56:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: 1.1b scrub ok 2024-08-30T23:56:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2229359692' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:18.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:18.932 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:18.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:19.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:19.066 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:19.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:19.192 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:19.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:19.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:19.338 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:56:19.338 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:56:19.338 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:56:19.338 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:56:19.339 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:56:19.423 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:19.450 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.451 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:19 smithi136 ceph-mon[20207]: 8.4 scrub ok 2024-08-30T23:56:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:19 smithi136 ceph-mon[20207]: pgmap v2024: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s; 6883/164463 objects degraded (4.185%) 2024-08-30T23:56:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:19 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:56:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2229359692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:19 smithi136 ceph-mon[20207]: osdmap e912: 12 total, 11 up, 12 in 2024-08-30T23:56:19.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:19 smithi106 ceph-mon[16603]: 8.4 scrub ok 2024-08-30T23:56:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:19 smithi106 ceph-mon[16603]: pgmap v2024: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s; 6883/164463 objects degraded (4.185%) 2024-08-30T23:56:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:19 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:56:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2229359692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:19.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:19 smithi106 ceph-mon[16603]: osdmap e912: 12 total, 11 up, 12 in 2024-08-30T23:56:19.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:19.712 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:19.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:19 smithi173 ceph-mon[20554]: 8.4 scrub ok 2024-08-30T23:56:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:19 smithi173 ceph-mon[20554]: pgmap v2024: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 995 op/s; 6883/164463 objects degraded (4.185%) 2024-08-30T23:56:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:19 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:56:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2229359692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:19 smithi173 ceph-mon[20554]: osdmap e912: 12 total, 11 up, 12 in 2024-08-30T23:56:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.840 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:19.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:19.942 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:19.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:19.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:20.016 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:56:20.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:20.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:20.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.242 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:20.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:20.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:20.361 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:56:20.380 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:20.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:20.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:20.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:56:20.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:20.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:20.677 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:20.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:20.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:20 smithi136 ceph-mon[20207]: osdmap e913: 12 total, 11 up, 12 in 2024-08-30T23:56:20.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:20 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:56:20.678 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:20 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:56:20.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:20.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:20 smithi173 ceph-mon[20554]: osdmap e913: 12 total, 11 up, 12 in 2024-08-30T23:56:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:20 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:56:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:20 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:56:20.846 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:20.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:20.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:20.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:20 smithi106 ceph-mon[16603]: osdmap e913: 12 total, 11 up, 12 in 2024-08-30T23:56:20.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:20 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:56:20.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:20 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:56:20.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:20.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:20.956 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:56:20.956 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:56:20.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 684/654 [PushOp(6:f01ac110:::benchmark_data_smithi173_114000_object56049:head, version: 638'3433, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f01ac110:::benchmark_data_smithi173_114000_object56049:head@638'3433, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:39.648816+0000", 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 521.30610060100003, 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70308431000000005, 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648816+0000", 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648816+0000", 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648818+0000", 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5999999999999999e-06 2024-08-30T23:56:20.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:39.648891+0000", 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3188999999999993e-05 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348689+0000", 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.69979756900000001 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348717+0000", 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8413000000000001e-05 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348915+0000", 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019764199999999999 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.348938+0000", 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3536999999999998e-05 2024-08-30T23:56:20.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:40.351900+0000", 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029623599999999998 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(6.f 691/654 [PushOp(6:f5b42d7a:::benchmark_data_smithi173_114000_object68947:head, version: 642'4266, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f5b42d7a:::benchmark_data_smithi173_114000_object68947:head@642'4266, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:47:57.450979+0000", 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 503.50393790300001, 2024-08-30T23:56:20.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.50071664100000002, 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.450979+0000", 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.450979+0000", 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.450980+0000", 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0900000000000001e-07 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.451033+0000", 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3090999999999997e-05 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.899123+0000", 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44809082900000002 2024-08-30T23:56:20.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.899140+0000", 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6158e-05 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.947031+0000", 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047891488000000003 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.947045+0000", 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3534e-05 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:47:57.951696+0000", 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046508319999999997 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27034.0:193162 6.4 e715/684 6:2b4eb615:::benchmark_data_smithi173_114000_object93364:head v 715'12123, mlcod=715'12119)", 2024-08-30T23:56:20.965 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:37.213384+0000", 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 463.74153265400003, 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24900366600000001, 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213384+0000", 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213384+0000", 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213385+0000", 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4799999999999997e-07 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.213392+0000", 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0249999999999997e-06 2024-08-30T23:56:20.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459261+0000", 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24586888600000001 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459269+0000", 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7559999999999999e-06 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459325+0000", 2024-08-30T23:56:20.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5841000000000003e-05 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.459341+0000", 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.462353+0000", 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030117640000000001 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:37.462388+0000", 2024-08-30T23:56:20.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4659000000000001e-05 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.0 795/788 [PushOp(7:0859067a:::benchmark_data_smithi173_129296_object41869:head, version: 739'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:0859067a:::benchmark_data_smithi173_129296_object41869:head@739'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 258.14074130300003, 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.2782153, 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:20.971 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814176+0000", 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7199999999999998e-07 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814204+0000", 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7526e-05 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090173+0000", 2024-08-30T23:56:20.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27596965699999998 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090195+0000", 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2075000000000001e-05 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090304+0000", 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010874699999999999 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090321+0000", 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7149000000000001e-05 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.092391+0000", 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002069474 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.975 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:6025 8.9 e847/840 8:9d1f5af9:::benchmark_data_smithi173_143816_object6024:head v 847'358, mlcod=847'355)", 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:59.018704+0000", 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 141.93621271800001, 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24216917299999999, 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018704+0000", 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018704+0000", 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018705+0000", 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5799999999999999e-07 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.018783+0000", 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8517999999999996e-05 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260248+0000", 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.241465287 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260254+0000", 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.203e-06 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260307+0000", 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3344999999999998e-05 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260326+0000", 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8572000000000001e-05 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260837+0000", 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000511926 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:59.260873+0000", 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5763999999999998e-05 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21763 8.4 8:2f5d79f6:::benchmark_data_smithi173_143816_object21762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.87698073799999, 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30212884200000001, 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.980 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21763 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077938+0000", 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0480000000000001e-06 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077983+0000", 2024-08-30T23:56:20.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5318999999999998e-05 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078009+0000", 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5554999999999998e-05 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078016+0000", 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4769999999999997e-06 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078078+0000", 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1748999999999994e-05 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078236+0000", 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000157434 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078373+0000", 2024-08-30T23:56:20.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013750000000000001 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079802+0000", 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001428653 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079818+0000", 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6597999999999999e-05 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379984+0000", 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30016611799999998 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380032+0000", 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7268000000000002e-05 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380065+0000", 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3123000000000001e-05 2024-08-30T23:56:20.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21763 8.4 e852/840 ondisk, result = 0)", 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.871420462, 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.296575003, 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083497+0000", 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1500000000000004e-07 2024-08-30T23:56:20.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083505+0000", 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379828+0000", 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29632308899999998 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379844+0000", 2024-08-30T23:56:20.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5665999999999999e-05 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379970+0000", 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012600999999999999 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379979+0000", 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9770000000000006e-06 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380071+0000", 2024-08-30T23:56:20.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2387999999999997e-05 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21778 8.8 8:10e57910:::benchmark_data_smithi173_143816_object21777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.72759375999999, 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28839689299999999, 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:20.991 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21778 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:56:20.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227333+0000", 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5489999999999995e-06 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227385+0000", 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2265999999999997e-05 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227427+0000", 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2255000000000001e-05 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227452+0000", 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4443999999999998e-05 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227520+0000", 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8916000000000002e-05 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227528+0000", 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228639+0000", 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011110810000000001 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228736+0000", 2024-08-30T23:56:20.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7001000000000002e-05 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228817+0000", 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1038000000000006e-05 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.229951+0000", 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001133319 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230356+0000", 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040563399999999999 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515551+0000", 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28519461400000001 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515669+0000", 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117731 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515720+0000", 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1390000000000001e-05 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21778 8.8 e853/840 ondisk, result = 0)", 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.72454416599999, 2024-08-30T23:56:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28535590100000002, 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:20.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230376+0000", 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2480000000000001e-06 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515346+0000", 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28496944699999999 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515361+0000", 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5573000000000002e-05 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515505+0000", 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515533+0000", 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7696e-05 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515728+0000", 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019533300000000001 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.002 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21786 8.9 e853/840 8:9a7f97a1:::benchmark_data_smithi173_143816_object21785:head v 853'1317, mlcod=852'1314)", 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.57518087899999, 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22536509800000001, 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379736+0000", 2024-08-30T23:56:21.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8699999999999995e-07 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379790+0000", 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3526000000000002e-05 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604264+0000", 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.224474275 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604267+0000", 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.779e-06 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604309+0000", 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2253999999999999e-05 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604325+0000", 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5483999999999999e-05 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.605072+0000", 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00074713500000000001 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.605101+0000", 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9158000000000001e-05 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.007 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21807 8.4 8:2833fcc7:::benchmark_data_smithi173_143816_object21806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.35206979200001, 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34692570299999997, 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21807 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602848+0000", 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009e-06 2024-08-30T23:56:21.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602878+0000", 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0278999999999999e-05 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602891+0000", 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2371000000000001e-05 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602897+0000", 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602941+0000", 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4617000000000003e-05 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603031+0000", 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0060000000000002e-05 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:56:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603115+0000", 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3387000000000006e-05 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603890+0000", 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077577699999999996 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604297+0000", 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040658900000000001 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949697+0000", 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34539998399999999 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949739+0000", 2024-08-30T23:56:21.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1758000000000002e-05 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949773+0000", 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3970999999999997e-05 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21807 8.4 e853/840 ondisk, result = 0)", 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 130.35068639900001, 2024-08-30T23:56:21.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34559199699999998, 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:56:21.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604231+0000", 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7599999999999999e-07 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604232+0000", 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.105e-06 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949585+0000", 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34535339199999998 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949593+0000", 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.644e-06 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949684+0000", 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1680000000000003e-05 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949693+0000", 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.354e-06 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949822+0000", 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129646 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.019 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:37810 8.8 8:1435bca5:::benchmark_data_smithi173_143816_object37809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:27.430041+0000", 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.52487545300001, 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22259082999999999, 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 37810 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430041+0000", 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430041+0000", 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430044+0000", 2024-08-30T23:56:21.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8399999999999999e-06 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430093+0000", 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8578999999999997e-05 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430122+0000", 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9166000000000002e-05 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430135+0000", 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3162999999999999e-05 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430302+0000", 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016710999999999999 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430449+0000", 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146382 2024-08-30T23:56:21.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.430534+0000", 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5075999999999997e-05 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.431643+0000", 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011095969999999999 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.432150+0000", 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050662799999999996 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652558+0000", 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.220408417 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652599+0000", 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0475999999999997e-05 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:27.652632+0000", 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3395999999999997e-05 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:61503 8.4 8:29953d46:::benchmark_data_smithi173_143816_object61502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 90.693629258000001, 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24835605799999999, 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.026 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61503 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9299999999999997e-07 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261343+0000", 2024-08-30T23:56:21.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5149999999999999e-05 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261352+0000", 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261356+0000", 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1420000000000003e-06 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261393+0000", 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899e-05 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261516+0000", 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123299 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261655+0000", 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139195 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:21.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263257+0000", 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016018880000000001 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263968+0000", 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071132499999999996 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509535+0000", 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24556666499999999 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509591+0000", 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6273000000000001e-05 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509644+0000", 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2148999999999998e-05 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.032 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:61503 8.4 e873/840 ondisk, result = 0)", 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.263970+0000", 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 90.690946417000006, 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24568077299999999, 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263970+0000", 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263970+0000", 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263971+0000", 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4700000000000002e-07 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263973+0000", 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.176e-06 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.508497+0000", 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24452453900000001 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.508502+0000", 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0189999999999997e-06 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509521+0000", 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001018915 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509528+0000", 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9430000000000003e-06 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509651+0000", 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000122834 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:61518 8.8 8:1736367f:::benchmark_data_smithi173_143816_object61517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-08-30T23:56:21.037 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.279013+0000", 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 90.675903950999995, 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23154129300000001, 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61518 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279013+0000", 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279013+0000", 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279014+0000", 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.124e-06 2024-08-30T23:56:21.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279060+0000", 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5943999999999999e-05 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279072+0000", 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1691e-05 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279079+0000", 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9949999999999999e-06 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279128+0000", 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9171000000000001e-05 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279224+0000", 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6108999999999997e-05 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.279314+0000", 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0507999999999995e-05 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.280887+0000", 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001573056 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.281834+0000", 2024-08-30T23:56:21.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094661300000000001 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510469+0000", 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22863545900000001 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510519+0000", 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9978000000000002e-05 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510554+0000", 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4644999999999998e-05 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:61518 8.8 e873/840 ondisk, result = 0)", 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.281793+0000", 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 90.673123951999997, 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22876733899999999, 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.044 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.281793+0000", 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.281793+0000", 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.281793+0000", 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7399999999999999e-07 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.281796+0000", 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.356e-06 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510411+0000", 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.22861595000000001 2024-08-30T23:56:21.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510416+0000", 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8010000000000003e-06 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510459+0000", 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2370000000000003e-05 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510465+0000", 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2339999999999999e-06 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.510560+0000", 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5254000000000007e-05 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:78084 8.4 8:26202172:::benchmark_data_smithi173_143816_object78083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-08-30T23:56:21.048 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:55:05.463500+0000", 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 75.491417038999998, 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.23480951799999999, 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 78084 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463500+0000", 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463500+0000", 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463501+0000", 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3769999999999999e-06 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463530+0000", 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8453e-05 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463541+0000", 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1408e-05 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463546+0000", 2024-08-30T23:56:21.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0889999999999999e-06 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463598+0000", 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2138000000000001e-05 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463730+0000", 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013202900000000001 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:56:21.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.463871+0000", 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014079899999999999 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.466109+0000", 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0022382880000000002 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.468587+0000", 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024775750000000001 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.698240+0000", 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.229652733 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.698279+0000", 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9019e-05 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:55:05.698309+0000", 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.061e-05 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:132914 8.4 8:2bd81915:::benchmark_data_smithi173_143816_object34600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e907)", 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:56:06.098327+0000", 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.856589315000001, 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.127872505, 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 132914 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.055 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098327+0000", 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098327+0000", 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098329+0000", 2024-08-30T23:56:21.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6619999999999999e-06 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098335+0000", 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0190000000000003e-06 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098353+0000", 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7601999999999999e-05 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098361+0000", 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5609999999999999e-06 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098419+0000", 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7501999999999999e-05 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098497+0000", 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8203999999999999e-05 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.098560+0000", 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3405999999999996e-05 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.101581+0000", 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0030201469999999999 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.102265+0000", 2024-08-30T23:56:21.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00068399499999999996 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.226111+0000", 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.123846 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.226149+0000", 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8433999999999998e-05 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:06.226200+0000", 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0973000000000001e-05 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:145331 8.3 e913/862 8:c82c0f38:::benchmark_data_smithi173_143816_object47017:head v 913'8990, mlcod=913'8987)", 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:56:19.574480+0000", 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.380436293, 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.149947042, 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:56:21.061 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.574480+0000", 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.574480+0000", 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.574481+0000", 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6599999999999996e-07 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.574487+0000", 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2530000000000001e-06 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.720368+0000", 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.14588035499999999 2024-08-30T23:56:21.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.720375+0000", 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8669999999999996e-06 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.720416+0000", 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1078999999999998e-05 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.720430+0000", 2024-08-30T23:56:21.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4352000000000001e-05 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.724373+0000", 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039429979999999996 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:56:19.724428+0000", 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4472000000000003e-05 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:56:21.066 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:56:21.066 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:56:21.092 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:21.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:21.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:56:21.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:21.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:21.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:21.453 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:21.463 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:21.463 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:21.480 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.481 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:21.498 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:21.498 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:21.499 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:21 smithi106 ceph-mon[16603]: pgmap v2027: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 6883/160884 objects degraded (4.278%) 2024-08-30T23:56:21.499 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:21 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:56:21.499 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:21 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:56:21.499 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6883/160884 objects degraded (4.278%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:21.528 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:21.528 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:21.556 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:21.556 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:21.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:21.586 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:21.586 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:21.595 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:21.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.610 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:21.614 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:21.614 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:21.646 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:21.646 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:21.646 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:21.672 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:21.672 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:21.702 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:21.702 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:21.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:21.724 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:21 smithi173 ceph-mon[20554]: pgmap v2027: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 6883/160884 objects degraded (4.278%) 2024-08-30T23:56:21.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:21 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:56:21.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:21 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:56:21.725 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6883/160884 objects degraded (4.278%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:21.729 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:21.730 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:21.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:21.759 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:21.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:21 smithi136 ceph-mon[20207]: pgmap v2027: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 6883/160884 objects degraded (4.278%) 2024-08-30T23:56:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:21 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:56:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:21 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:56:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6883/160884 objects degraded (4.278%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:21.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:21.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: marking collection for removal 2024-08-30T23:56:21.915 INFO:teuthology.orchestra.run.smithi136.stdout:setting '_remove' omap key 2024-08-30T23:56:21.936 INFO:teuthology.orchestra.run.smithi136.stdout:finish_remove_pgs 1.1a_head removing 1.1a 2024-08-30T23:56:21.952 INFO:teuthology.orchestra.run.smithi136.stdout:Remove successful 2024-08-30T23:56:21.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:21.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:56:21.994 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:21.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:22.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:22.119 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:22.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:22.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:22.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:22.369 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:22.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:22.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:22.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:22.691 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:22 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:56:22.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:22 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:56:22.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/853046240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:22.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:22.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:22.732 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:22.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:22.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:22 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:56:22.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:22 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:56:22.780 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/853046240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:22.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:22 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:56:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:22 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:56:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/853046240' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:22.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:22.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:23.008 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:23.038 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:23.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:23.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:23.175 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:23.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell -n osd.7 --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.1a.7 2024-08-30T23:56:23.277 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:23.307 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:23.409 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:23.417 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:56:23.430 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/osd.7/config 2024-08-30T23:56:23.437 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:23.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:23 smithi136 ceph-mon[20207]: pgmap v2028: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 6282/153000 objects degraded (4.106%) 2024-08-30T23:56:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1215784259' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:23 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:23.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:23.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:23 smithi106 ceph-mon[16603]: pgmap v2028: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 6282/153000 objects degraded (4.106%) 2024-08-30T23:56:23.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1215784259' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:23.581 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:23 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:23.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:23.690 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:23.791 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:23.791 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:23.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:23.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:23 smithi173 ceph-mon[20554]: pgmap v2028: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.11k op/s; 6282/153000 objects degraded (4.106%) 2024-08-30T23:56:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1215784259' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:23 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:23.919 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:23.953 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:23.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:24.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:24.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:24.193 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:24.221 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.222 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:24.323 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.346 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:24.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:24.447 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:24.470 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:24.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:24 smithi106 ceph-mon[16603]: 8.0 scrub starts 2024-08-30T23:56:24.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:24 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:24.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:24 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:24.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:24 smithi106 ceph-mon[16603]: osdmap e914: 12 total, 11 up, 12 in 2024-08-30T23:56:24.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:56:24.573 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:24.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:24.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:24.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:56:24.735 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:24 smithi136 ceph-mon[20207]: 8.0 scrub starts 2024-08-30T23:56:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:24 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:24 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:24.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:24 smithi136 ceph-mon[20207]: osdmap e914: 12 total, 11 up, 12 in 2024-08-30T23:56:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:24 smithi173 ceph-mon[20554]: 8.0 scrub starts 2024-08-30T23:56:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:24 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:24 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:24 smithi173 ceph-mon[20554]: osdmap e914: 12 total, 11 up, 12 in 2024-08-30T23:56:24.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:24.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:24.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:24.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:25.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:25.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:25.114 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:25.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:25.129 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:25.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:25.250 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:25.352 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:25.384 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:25.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.519 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:25.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:25.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:25.654 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:25.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:25.699 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:25 smithi106 ceph-mon[16603]: pgmap v2030: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 6282/150534 objects degraded (4.173%) 2024-08-30T23:56:25.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:25.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:25 smithi136 ceph-mon[20207]: pgmap v2030: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 6282/150534 objects degraded (4.173%) 2024-08-30T23:56:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:25 smithi173 ceph-mon[20554]: pgmap v2030: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 773 op/s; 6282/150534 objects degraded (4.173%) 2024-08-30T23:56:25.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:25.921 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:25.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:26.003 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:26.003 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:26.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:26.053 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:26.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.183 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:26.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:26.210 INFO:teuthology.orchestra.run.smithi136.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-30T23:56:26.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:26.312 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.313 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:26.414 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:26.445 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:26.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:26.572 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:26.674 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:26.697 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6282/150534 objects degraded (4.173%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:26 smithi136 ceph-mon[20207]: osdmap e915: 12 total, 11 up, 12 in 2024-08-30T23:56:26.742 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1396653871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:26.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6282/150534 objects degraded (4.173%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:26.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:26 smithi106 ceph-mon[16603]: osdmap e915: 12 total, 11 up, 12 in 2024-08-30T23:56:26.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1396653871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:26.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6282/150534 objects degraded (4.173%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:26 smithi173 ceph-mon[20554]: osdmap e915: 12 total, 11 up, 12 in 2024-08-30T23:56:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1396653871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:26.828 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:26.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:26.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:26.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:26.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:27.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:27.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:56:27.081 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:27.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:27.210 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:27.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:56:27.313 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:27.343 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.344 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:27.419 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:27.419 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:27.420 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:27.421 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:27.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:27.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:27.493 INFO:teuthology.orchestra.run.smithi136.stdout:get_pg_num_history pg_num_history pg_num_history(e902 pg_nums {1={507=21,877=31},7={725=16},8={840=16}} deleted_pools 838,7) 2024-08-30T23:56:27.494 INFO:teuthology.orchestra.run.smithi136.stdout:Importing pgid 1.1a 2024-08-30T23:56:27.494 INFO:teuthology.orchestra.run.smithi136.stdout:write_pg epoch 880 info 1.1a( v 23'92 (0'0,23'92] local-lis/les=879/880 n=0 ec=877/20 lis/c=879/879 les/c/f=880/880/0 sis=879) 2024-08-30T23:56:27.494 INFO:teuthology.orchestra.run.smithi136.stdout:Import successful 2024-08-30T23:56:27.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:27.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:27.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:27.724 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:27.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:27 smithi106 ceph-mon[16603]: pgmap v2031: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 6282/145596 objects degraded (4.315%) 2024-08-30T23:56:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:27 smithi136 ceph-mon[20207]: pgmap v2031: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 6282/145596 objects degraded (4.315%) 2024-08-30T23:56:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:27 smithi173 ceph-mon[20554]: pgmap v2031: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 6282/145596 objects degraded (4.315%) 2024-08-30T23:56:27.826 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.853 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:27.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:27.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:27.978 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:27.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:28.079 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:28.101 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:28.104 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:28.167 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:28.180 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:28.207 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:28.232 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:28.334 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:28.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:28.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:28.518 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:28.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:28 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:28.563 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:28 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:28.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:28.648 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:28.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:28.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:28 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:28.697 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:28 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:28.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/exp.1.1a.7 2024-08-30T23:56:28.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:28.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:28 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:28 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:28.887 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:28.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:28.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:29.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:29.048 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:29.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.174 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:29.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.9 760/755 [PushOp(7:92ccacc7:::benchmark_data_smithi173_129296_object5107:head, version: 730'310, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:92ccacc7:::benchmark_data_smithi173_129296_object5107:head@730'310, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.220 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 365.15586634700003, 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93410173799999996, 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061775+0000", 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4950000000000001e-06 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061824+0000", 2024-08-30T23:56:29.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9017000000000002e-05 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993785+0000", 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93196087299999997 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993806+0000", 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1152e-05 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993912+0000", 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106251 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993928+0000", 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6387000000000001e-05 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.995875+0000", 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019465629999999999 2024-08-30T23:56:29.224 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 765/762 [PushOp(7:6c4bd07a:::benchmark_data_smithi173_129296_object3500:head, version: 730'216, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6c4bd07a:::benchmark_data_smithi173_129296_object3500:head@730'216, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 358.744073222, 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62968416199999999, 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473567+0000", 2024-08-30T23:56:29.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6799999999999999e-07 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473595+0000", 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7571000000000001e-05 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101453+0000", 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62785843200000002 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101470+0000", 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7164e-05 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101579+0000", 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109252 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101595+0000", 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449000000000001e-05 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.103250+0000", 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001655526 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 767/762 [PushOp(7:2ff57e78:::benchmark_data_smithi173_129296_object4738:head, version: 730'300, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2ff57e78:::benchmark_data_smithi173_129296_object4738:head@730'300, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 355.47301240199999, 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.95161098399999999, 2024-08-30T23:56:29.229 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744628+0000", 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744662+0000", 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3967000000000002e-05 2024-08-30T23:56:29.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694267+0000", 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94960559300000003 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694286+0000", 2024-08-30T23:56:29.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.844e-05 2024-08-30T23:56:29.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694373+0000", 2024-08-30T23:56:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7074e-05 2024-08-30T23:56:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694383+0000", 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7389999999999994e-06 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.696238+0000", 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018554509999999999 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 767/762 [PushOp(7:603ba3a2:::benchmark_data_smithi173_129296_object7838:head, version: 730'501, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:603ba3a2:::benchmark_data_smithi173_129296_object7838:head@730'501, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 353.61366442100001, 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.99532398200000005, 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603976+0000", 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.071e-06 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.604007+0000", 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0408000000000001e-05 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597608+0000", 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.99360170000000003 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597625+0000", 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7184999999999999e-05 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597718+0000", 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2985999999999994e-05 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597729+0000", 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0227999999999999e-05 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.599299+0000", 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015704040000000001 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 768/762 [PushOp(7:6615b48e:::benchmark_data_smithi173_129296_object8839:head, version: 730'559, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6615b48e:::benchmark_data_smithi173_129296_object8839:head@730'559, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 351.75610440700001, 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63695423500000004, 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461536+0000", 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1519999999999999e-06 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461566+0000", 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9569000000000001e-05 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097105+0000", 2024-08-30T23:56:29.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63553892199999995 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097118+0000", 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2814e-05 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097203+0000", 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5037999999999995e-05 2024-08-30T23:56:29.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097213+0000", 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9930000000000007e-06 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.098489+0000", 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001276747 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 768/762 [PushOp(7:2ce0b962:::benchmark_data_smithi173_129296_object7093:head, version: 730'443, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2ce0b962:::benchmark_data_smithi173_129296_object7093:head@730'443, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 351.172298254, 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.819240623, 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.246 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:56:29.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045342+0000", 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-07 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045405+0000", 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2865000000000002e-05 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.861929+0000", 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81652422599999996 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.861952+0000", 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2768999999999998e-05 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.862068+0000", 2024-08-30T23:56:29.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115807 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.862081+0000", 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3247e-05 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.864582+0000", 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002501021 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 769/762 [PushOp(7:6b3af24c:::benchmark_data_smithi173_129296_object11749:head, version: 730'750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6b3af24c:::benchmark_data_smithi173_129296_object11749:head@730'750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 348.01010613, 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72094049199999999, 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:56:29.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207535+0000", 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6759999999999999e-06 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207575+0000", 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0224e-05 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927098+0000", 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71952308300000001 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927109+0000", 2024-08-30T23:56:29.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0771000000000001e-05 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927189+0000", 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0348999999999999e-05 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927199+0000", 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3209999999999994e-06 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.928474+0000", 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012750680000000001 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 769/762 [PushOp(7:668cd238:::benchmark_data_smithi173_129296_object13196:head, version: 730'849, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:668cd238:::benchmark_data_smithi173_129296_object13196:head@730'849, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 345.54644648999999, 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96579458500000004, 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.255 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671194+0000", 2024-08-30T23:56:29.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.297e-06 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671225+0000", 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0312e-05 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.634913+0000", 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96368826900000004 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.634929+0000", 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6371999999999998e-05 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.635041+0000", 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111471 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.635052+0000", 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1195e-05 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.636988+0000", 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019356690000000001 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.f 771/761 [PushOp(7:fed971b1:::benchmark_data_smithi173_129296_object18024:head, version: 733'1120, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:fed971b1:::benchmark_data_smithi173_129296_object18024:head@733'1120, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 341.640182987, 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58954947000000002, 2024-08-30T23:56:29.259 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577458+0000", 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577571+0000", 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112908 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164827+0000", 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58725667999999998 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164846+0000", 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8417e-05 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164937+0000", 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0884000000000006e-05 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164946+0000", 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7380000000000006e-06 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.167006+0000", 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020595629999999999 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 771/762 [PushOp(7:241f6387:::benchmark_data_smithi173_129296_object16791:head, version: 731'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:241f6387:::benchmark_data_smithi173_129296_object16791:head@731'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.263 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 341.59149899200003, 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44453464500000001, 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626142+0000", 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3740000000000001e-06 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626194+0000", 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1793999999999999e-05 2024-08-30T23:56:29.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068657+0000", 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44246287299999998 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068671+0000", 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4783999999999999e-05 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068746+0000", 2024-08-30T23:56:29.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4684999999999994e-05 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068756+0000", 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5750000000000007e-06 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.070675+0000", 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019195600000000001 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.f 787/761 [PushOp(7:f52af3f4:::benchmark_data_smithi173_129296_object52699:head, version: 743'3327, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f52af3f4:::benchmark_data_smithi173_129296_object52699:head@743'3327, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 286.92326189099998, 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.66126428199999998, 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:56:29.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294379+0000", 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294427+0000", 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8597999999999998e-05 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952418+0000", 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.65799002699999998 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952437+0000", 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9136000000000001e-05 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952574+0000", 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013768 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952590+0000", 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5860000000000001e-05 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.955642+0000", 2024-08-30T23:56:29.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003051701 2024-08-30T23:56:29.272 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.272 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.272 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:142397 7.d 7:bed45d06:::benchmark_data_smithi173_129296_object55281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 231.59072054200001, 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48225094800000001, 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 142397 2024-08-30T23:56:29.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626921+0000", 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4219999999999999e-06 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626930+0000", 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5599999999999994e-06 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626945+0000", 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5495999999999999e-05 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626955+0000", 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6190000000000006e-06 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.628796+0000", 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018411910000000001 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.628976+0000", 2024-08-30T23:56:29.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000179777 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.629127+0000", 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000150848 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.633350+0000", 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042234500000000001 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.635612+0000", 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022614369999999998 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107166+0000", 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.471554168 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107232+0000", 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5927999999999996e-05 2024-08-30T23:56:29.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.109170+0000", 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001938052 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27487.0:142397 7.d e810/800 ondisk, result = 0)", 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 231.58284369200001, 2024-08-30T23:56:29.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.474397232, 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:56:29.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5099999999999999e-07 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634801+0000", 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0550000000000001e-06 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.102650+0000", 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.467849444 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.102656+0000", 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107133+0000", 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044769750000000002 2024-08-30T23:56:29.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107145+0000", 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2656999999999999e-05 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.109193+0000", 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020475850000000002 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.9 e833,762 7:9b279d9f:::benchmark_data_smithi173_129296_object38265:head,794'7954)", 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 172.03071993200001, 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.880300788, 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.61e-07 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186925+0000", 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4730000000000002e-06 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067100+0000", 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.88017544199999997 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067106+0000", 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9030000000000001e-06 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067157+0000", 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0556000000000002e-05 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067162+0000", 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.587e-06 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067220+0000", 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8066000000000003e-05 2024-08-30T23:56:29.287 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e835,800 7:bbbda60d:::benchmark_data_smithi173_129296_object15076:head,778'6374)", 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 167.970825711, 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74636665599999996, 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246816+0000", 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.716e-06 2024-08-30T23:56:29.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246828+0000", 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.27e-05 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992675+0000", 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74584704999999996 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992682+0000", 2024-08-30T23:56:29.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.241e-06 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992757+0000", 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5322999999999997e-05 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.993180+0000", 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042362600000000003 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.291 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e836,800 7:b1ca300e:::benchmark_data_smithi173_129296_object19401:head,781'6635)", 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 165.01414820299999, 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45044139700000002, 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:29.292 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203492+0000", 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7000000000000004e-07 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203497+0000", 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8380000000000001e-06 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653230+0000", 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44973348699999999 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653238+0000", 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1329999999999999e-06 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653353+0000", 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114485 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653933+0000", 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057978399999999996 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.295 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.5 e837,761 7:ae197e78:::benchmark_data_smithi173_129296_object20842:head,781'6740)", 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.55455000800001, 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45953577400000001, 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663090+0000", 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2699999999999998e-07 2024-08-30T23:56:29.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663096+0000", 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4179999999999996e-06 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122458+0000", 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.459362202 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122466+0000", 2024-08-30T23:56:29.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7600000000000002e-06 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122530+0000", 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3917000000000005e-05 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122541+0000", 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0824e-05 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122625+0000", 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4726000000000002e-05 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.5 e837,761 7:a846cf89:::benchmark_data_smithi173_129296_object20850:head,781'6741)", 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.54815589699999, 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46070137100000003, 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.300 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669485+0000", 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669491+0000", 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7590000000000003e-06 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130069+0000", 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.460578184 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130075+0000", 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130126+0000", 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1564000000000003e-05 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130133+0000", 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6730000000000003e-06 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130185+0000", 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1891000000000002e-05 2024-08-30T23:56:29.305 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e837,800 7:b35fe8f8:::benchmark_data_smithi173_129296_object19689:head,781'6653)", 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.46113351299999, 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42114565399999998, 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756507+0000", 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8899999999999999e-07 2024-08-30T23:56:29.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756510+0000", 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.839e-06 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177074+0000", 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42056344499999998 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177083+0000", 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.397e-06 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177141+0000", 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7311000000000001e-05 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177652+0000", 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00051107300000000002 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.309 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e837,800 7:b5944d16:::benchmark_data_smithi173_129296_object19693:head,781'6654)", 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 164.45654054400001, 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42161228299999998, 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:29.310 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.39e-07 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761102+0000", 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8049999999999998e-06 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182311+0000", 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42120909499999998 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182317+0000", 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0440000000000001e-06 2024-08-30T23:56:29.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182359+0000", 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1857000000000003e-05 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182711+0000", 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035204300000000002 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:29.313 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:29.314 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:56:29.314 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:29.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:29.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:29.470 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:29.476 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:56:29.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.604 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:29.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:29.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:29.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:29 smithi136 ceph-mon[20207]: pgmap v2033: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 5728/137745 objects degraded (4.158%) 2024-08-30T23:56:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:29 smithi136 ceph-mon[20207]: osdmap e916: 12 total, 11 up, 12 in 2024-08-30T23:56:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2428872485' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:29 smithi173 ceph-mon[20554]: pgmap v2033: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 5728/137745 objects degraded (4.158%) 2024-08-30T23:56:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:29 smithi173 ceph-mon[20554]: osdmap e916: 12 total, 11 up, 12 in 2024-08-30T23:56:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2428872485' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:29.829 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:29.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:29.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:29 smithi106 ceph-mon[16603]: pgmap v2033: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 963 op/s; 5728/137745 objects degraded (4.158%) 2024-08-30T23:56:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:29 smithi106 ceph-mon[16603]: osdmap e916: 12 total, 11 up, 12 in 2024-08-30T23:56:29.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2428872485' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:29.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:29.991 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:29.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:30.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:30.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:56:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:30.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:30.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:56:30.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:30.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:30.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:30.372 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:30.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:30.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:30.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:30.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:30.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:30.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:30 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2428872485' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:30 smithi136 ceph-mon[20207]: osdmap e917: 12 total, 11 up, 12 in 2024-08-30T23:56:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:30 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2428872485' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:30 smithi173 ceph-mon[20554]: osdmap e917: 12 total, 11 up, 12 in 2024-08-30T23:56:30.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:30 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2428872485' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:30.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:30 smithi106 ceph-mon[16603]: osdmap e917: 12 total, 11 up, 12 in 2024-08-30T23:56:30.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:30.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:30.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:31.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:31.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:31.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:31.153 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:31.195 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:31.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:31.274 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:31.376 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:31.404 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:31.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.528 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:31.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:31 smithi106 ceph-mon[16603]: pgmap v2036: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 5728/134589 objects degraded (4.256%) 2024-08-30T23:56:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5728/134589 objects degraded (4.256%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:31.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:31.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:31.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:31 smithi136 ceph-mon[20207]: pgmap v2036: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 5728/134589 objects degraded (4.256%) 2024-08-30T23:56:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5728/134589 objects degraded (4.256%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:31.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:31.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:31 smithi173 ceph-mon[20554]: pgmap v2036: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 885 op/s; 5728/134589 objects degraded (4.256%) 2024-08-30T23:56:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5728/134589 objects degraded (4.256%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:31.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:31.920 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:31.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:32.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:32.047 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:32.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.177 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:32.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:32.278 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.307 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:32.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:32.409 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.441 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:32.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:32.542 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:32.565 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:32.602 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:32.602 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:32.630 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:32.631 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:32.660 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:32.660 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:32.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:32.689 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:32.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:32.695 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:32.711 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:32.711 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:32.741 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:32.742 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:32.742 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2558735873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:32.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:32.773 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:32.773 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:32.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2558735873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:32.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:32.804 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:32.804 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:32.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2558735873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:32.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:32.834 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:32.834 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:32.864 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:32.864 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:32.889 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:32.889 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:32.926 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:32.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:32.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:33.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:33.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:33.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:56:33.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:33.250 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:33.352 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:33.380 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:33.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:33.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:33.485 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:33 smithi173 ceph-mon[20554]: pgmap v2037: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5193/127425 objects degraded (4.075%) 2024-08-30T23:56:33.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:33.608 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:33.633 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:33.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:33.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:56:33.758 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:56:33.758 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-30T23:56:33.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:56:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.760 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:33.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:33.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:33 smithi106 ceph-mon[16603]: pgmap v2037: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5193/127425 objects degraded (4.075%) 2024-08-30T23:56:33.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:33 smithi136 ceph-mon[20207]: pgmap v2037: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 5193/127425 objects degraded (4.075%) 2024-08-30T23:56:33.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:33.887 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:33.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:33.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:34.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:34.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:34.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:34.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:34.291 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:34.393 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:34.420 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:34.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:34.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:34.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:34.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:34.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:34.778 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:34.800 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:34.866 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:34.866 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:34.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:34.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:34.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:35.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:35.056 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:35.151 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:35.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:35.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:35.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:35.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:35.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:35.440 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:35.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:35.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:35 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:35.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:35 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:35.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:35 smithi106 ceph-mon[16603]: pgmap v2038: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 5193/125049 objects degraded (4.153%) 2024-08-30T23:56:35.549 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:35 smithi106 ceph-mon[16603]: osdmap e918: 12 total, 11 up, 12 in 2024-08-30T23:56:35.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.567 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:35.669 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:35.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:35.697 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:35.739 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:35 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:35 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:35 smithi136 ceph-mon[20207]: pgmap v2038: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 5193/125049 objects degraded (4.153%) 2024-08-30T23:56:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:35 smithi136 ceph-mon[20207]: osdmap e918: 12 total, 11 up, 12 in 2024-08-30T23:56:35.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:35 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:35 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:35 smithi173 ceph-mon[20554]: pgmap v2038: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s; 5193/125049 objects degraded (4.153%) 2024-08-30T23:56:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:35 smithi173 ceph-mon[20554]: osdmap e918: 12 total, 11 up, 12 in 2024-08-30T23:56:35.827 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:35.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:35.957 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 918, 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:56:34.485309+0000", 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:55:58.153661+0000", 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:54:10.077277+0000", 2024-08-30T23:56:35.958 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5806080, 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "nodeep-scrub", 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "noscrub", 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 29, 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:56:35.959 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 8, 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "luminous", 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:56:35.960 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 31, 2024-08-30T23:56:35.961 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 31, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 31, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 31, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 31, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "879", 2024-08-30T23:56:35.962 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "877", 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:56:35.963 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:56:35.964 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:56:35.965 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.6100000143051147, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.6100000143051147, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:56:35.966 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 1.7699999809265137, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 1.7699999809265137, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.68016940355300903, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.81999999284744263, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.68000000715255737 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 8, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_6", 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:53:47.812719+0000", 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:56:35.967 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:56:35.968 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "844", 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:56:35.969 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:56:35.970 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:56:35.971 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.130000114440918, 2024-08-30T23:56:35.972 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 3.130000114440918, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.130000114440918, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3.130000114440918, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.68016940355300903, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.81999999284744263, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.68000000715255737 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:56:35.973 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 800, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.974 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.975 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:35.976 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.978 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.979 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:56:35.980 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.981 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:35.982 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:35.983 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.984 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:56:35.986 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 863, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.987 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.988 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:35.989 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.990 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.991 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:35.992 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 877, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.993 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:56:35.994 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.995 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:35.996 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.997 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:56:35.998 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:35.999 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:56:36.001 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.002 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.003 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.004 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 566, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 593, 2024-08-30T23:56:36.005 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 649, 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:56:36.006 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:56:36.007 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.008 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.009 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2739515404", 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2739515404", 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2739515404", 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2739515404", 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.010 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.011 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.012 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.013 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.014 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.015 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.015 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.015 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.015 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:36.015 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:36.016 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.016 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:56:36.016 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:56:36.017 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.1631317138671875, 2024-08-30T23:56:36.018 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 903, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.019 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.020 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.021 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.022 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.023 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:56:36.024 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:56:36.025 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.026 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.027 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:56:36.028 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:56:36.029 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.701446533203125, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:36.030 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.031 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:56:36.033 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.034 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:56:36.035 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:56:36.036 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:56:36.040 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:56:36.041 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:56:36.042 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.043 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:56:36.047 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.048 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:19.117919+0000", 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:00.081269+0000", 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.049 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:55:33.583297+0000", 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.050 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:02.657335+0000", 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:28.304555+0000", 2024-08-30T23:56:36.051 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:02.452267+0000", 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:56:36.052 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:27.519160+0000", 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.053 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:55:09.475333+0000", 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 902 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.054 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:33.620287+0000", 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:55:28.176035+0000", 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.055 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:55:24.018773+0000", 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:56:36.056 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:07.517807+0000", 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [ 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.0", 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-30T23:56:36.057 INFO:teuthology.orchestra.run.smithi106.stdout: 7, 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: 11 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1", 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:56:36.058 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:56:36.059 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:56:36.060 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:36.061 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:56:36.116 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:36.150 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:36.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:36.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:36.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:36.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:36.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1074462442' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:56:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:36 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:36 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5193/125049 objects degraded (4.153%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:36 smithi106 ceph-mon[16603]: osdmap e919: 12 total, 11 up, 12 in 2024-08-30T23:56:36.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:36.639 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:56:36.660 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:36.762 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1074462442' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:56:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:36 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:36 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5193/125049 objects degraded (4.153%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:36 smithi136 ceph-mon[20207]: osdmap e919: 12 total, 11 up, 12 in 2024-08-30T23:56:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.798 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:36.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1074462442' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:56:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:36 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:36 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5193/125049 objects degraded (4.153%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:36 smithi173 ceph-mon[20554]: osdmap e919: 12 total, 11 up, 12 in 2024-08-30T23:56:36.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:36.928 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:36.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:37.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:37.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:37.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:37.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:37.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:37.321 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:37.423 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:37.454 INFO:tasks.thrashosds.thrasher:j is {'epoch': 918, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:56:34.485309+0000', 'last_up_change': '2024-08-30T23:55:58.153661+0000', 'last_in_change': '2024-08-30T23:54:10.077277+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '879', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '877', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6100000143051147, 'score_stable': 1.6100000143051147, 'optimal_score': 1.100000023841858, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.680169403553009, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.6800000071525574}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-08-30T23:53:47.812719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '844', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.130000114440918, 'score_stable': 3.130000114440918, 'optimal_score': 1, 'raw_score_acting': 3.130000114440918, 'raw_score_stable': 3.130000114440918, 'primary_affinity_weighted': 0.680169403553009, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.6800000071525574}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 800, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 879, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 863, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 877, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 566, 'last_clean_end': 593, 'up_from': 649, 'up_thru': 903, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2739515404}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2739515404}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2739515404}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2739515404}]}, 'public_addr': '172.21.15.136:6809/2739515404', 'cluster_addr': '172.21.15.136:6811/2739515404', 'heartbeat_back_addr': '172.21.15.136:6815/2739515404', 'heartbeat_front_addr': '172.21.15.136:6813/2739515404', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.1631317138671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 879, 'down_at': 903, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 879, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.701446533203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 879, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:19.117919+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:00.081269+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:55:33.583297+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:02.657335+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:28.304555+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:02.452267+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:27.519160+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:55:09.475333+0000', 'dead_epoch': 902}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:33.620287+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:55:28.176035+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:55:24.018773+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:07.517807+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 7, 11]}, {'pgid': '1.1', 'osds': [2, 0, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:56:37.454 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-08-30T23:56:37.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:37.505 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:56:37.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.592 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:37.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:37.693 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:37.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:37 smithi136 ceph-mon[20207]: pgmap v2040: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 5193/120402 objects degraded (4.313%) 2024-08-30T23:56:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1928726960' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:37.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:37 smithi173 ceph-mon[20554]: pgmap v2040: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 5193/120402 objects degraded (4.313%) 2024-08-30T23:56:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1928726960' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:37.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:37.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:37.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:37 smithi106 ceph-mon[16603]: pgmap v2040: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s; 5193/120402 objects degraded (4.313%) 2024-08-30T23:56:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1928726960' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:56:37.899 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:56:37.953 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:38.010 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:38.112 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:38.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:38.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:38.268 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:38.370 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:38.391 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:38.493 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:38.515 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:38.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:38.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.641 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:38.656 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:38.705 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:38.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.768 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:38.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:38.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:38 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:38.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:38 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:38.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:38 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:38.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1928726960' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:56:38.783 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:38 smithi106 ceph-mon[16603]: osdmap e920: 12 total, 11 up, 12 in 2024-08-30T23:56:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:38 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:38 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:38 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1928726960' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:56:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:38 smithi136 ceph-mon[20207]: osdmap e920: 12 total, 11 up, 12 in 2024-08-30T23:56:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:38 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:38 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:38 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1928726960' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:56:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:38 smithi173 ceph-mon[20554]: osdmap e920: 12 total, 11 up, 12 in 2024-08-30T23:56:38.870 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:38.898 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:38.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:39.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:39.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:39.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:39.152 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:39.236 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:39.236 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:39.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:39.271 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:39.271 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:39.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:39.302 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:39.302 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:39.329 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:39.329 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:39.360 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:39.360 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:39.389 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:39.389 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:39.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:39.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:39.418 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:39.418 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:39.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:39.448 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:39.449 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:39.481 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:39.481 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:39.508 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:39.509 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:39.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:39.536 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:39.537 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:39.544 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:39.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:39 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:56:39.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:39 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:56:39.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:39 smithi106 ceph-mon[16603]: pgmap v2043: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 4649/112437 objects degraded (4.135%) 2024-08-30T23:56:39.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:39 smithi106 ceph-mon[16603]: osdmap e921: 12 total, 11 up, 12 in 2024-08-30T23:56:39.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2187600495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:39.646 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.683 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:39.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:39.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:39 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:56:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:39 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:56:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:39 smithi136 ceph-mon[20207]: pgmap v2043: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 4649/112437 objects degraded (4.135%) 2024-08-30T23:56:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:39 smithi136 ceph-mon[20207]: osdmap e921: 12 total, 11 up, 12 in 2024-08-30T23:56:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2187600495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:39.814 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:39 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:56:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:39 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:56:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:39 smithi173 ceph-mon[20554]: pgmap v2043: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s; 4649/112437 objects degraded (4.135%) 2024-08-30T23:56:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:39 smithi173 ceph-mon[20554]: osdmap e921: 12 total, 11 up, 12 in 2024-08-30T23:56:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2187600495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:39.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:39.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:39.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:56:40.026 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:56:40.026 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:56:40.026 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:56:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:56:40.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:40.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:40.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:40.242 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:40.343 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:40.374 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:40.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.509 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:40.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:40.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:56:40.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:40.672 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:40.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:40 smithi136 ceph-mon[20207]: 8.9 scrub starts 2024-08-30T23:56:40.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:40 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:56:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:40 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:56:40.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:40.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:40 smithi106 ceph-mon[16603]: 8.9 scrub starts 2024-08-30T23:56:40.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:40 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:56:40.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:40 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:56:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:40.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:40 smithi173 ceph-mon[20554]: 8.9 scrub starts 2024-08-30T23:56:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:40 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:56:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:40 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:56:40.854 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:56:40.904 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:40.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:40.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:41.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.081 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:41.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.214 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:41.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:41.345 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:41.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:41.466 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:41.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:41.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:41.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:41.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:41.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:41.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:41 smithi106 ceph-mon[16603]: 8.9 scrub ok 2024-08-30T23:56:41.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:41 smithi106 ceph-mon[16603]: pgmap v2045: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 4649/109440 objects degraded (4.248%) 2024-08-30T23:56:41.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:41 smithi106 ceph-mon[16603]: 1.17 scrub starts 2024-08-30T23:56:41.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:41 smithi106 ceph-mon[16603]: 1.17 scrub ok 2024-08-30T23:56:41.719 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4649/109440 objects degraded (4.248%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.745 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:41 smithi136 ceph-mon[20207]: 8.9 scrub ok 2024-08-30T23:56:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:41 smithi136 ceph-mon[20207]: pgmap v2045: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 4649/109440 objects degraded (4.248%) 2024-08-30T23:56:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:41 smithi136 ceph-mon[20207]: 1.17 scrub starts 2024-08-30T23:56:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:41 smithi136 ceph-mon[20207]: 1.17 scrub ok 2024-08-30T23:56:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4649/109440 objects degraded (4.248%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:41 smithi173 ceph-mon[20554]: 8.9 scrub ok 2024-08-30T23:56:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:41 smithi173 ceph-mon[20554]: pgmap v2045: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 874 op/s; 4649/109440 objects degraded (4.248%) 2024-08-30T23:56:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:41 smithi173 ceph-mon[20554]: 1.17 scrub starts 2024-08-30T23:56:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:41 smithi173 ceph-mon[20554]: 1.17 scrub ok 2024-08-30T23:56:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4649/109440 objects degraded (4.248%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:41.846 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:41.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:41.876 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:41.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:42.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:42.110 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:42.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:42.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.262 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:42.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:42.269 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:42.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:42.386 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:42.455 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:56:42.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:56:42.455 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:56:42.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:56:42.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.511 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:42.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:42.613 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:42.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.641 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:42.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:42.764 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:42.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:42.859 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:56:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:56:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:56:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:56:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(6.d 691/654 [PushOp(6:bc75eac3:::benchmark_data_smithi173_114000_object85166:head, version: 651'5133, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:bc75eac3:::benchmark_data_smithi173_114000_object85166:head@651'5133, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.860 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:56:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 525.38026048500001, 2024-08-30T23:56:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.43877167099999997, 2024-08-30T23:56:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.861 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:56:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:56:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.473960+0000", 2024-08-30T23:56:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8999999999999996e-07 2024-08-30T23:56:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.474010+0000", 2024-08-30T23:56:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9799000000000002e-05 2024-08-30T23:56:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909183+0000", 2024-08-30T23:56:42.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.435172475 2024-08-30T23:56:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909195+0000", 2024-08-30T23:56:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2727e-05 2024-08-30T23:56:42.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909335+0000", 2024-08-30T23:56:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014007399999999999 2024-08-30T23:56:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.909352+0000", 2024-08-30T23:56:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6487999999999999e-05 2024-08-30T23:56:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:47:57.912731+0000", 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033795180000000002 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 763/756 [PushReplyOp(7:59a9bd86:::benchmark_data_smithi173_129296_object9245:head)])", 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:28.337999+0000", 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.51622073099998, 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35323532299999999, 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.870 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.337999+0000", 2024-08-30T23:56:42.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.337999+0000", 2024-08-30T23:56:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.338001+0000", 2024-08-30T23:56:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:56:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.338007+0000", 2024-08-30T23:56:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2140000000000001e-06 2024-08-30T23:56:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.686245+0000", 2024-08-30T23:56:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34823851900000002 2024-08-30T23:56:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.686252+0000", 2024-08-30T23:56:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0690000000000004e-06 2024-08-30T23:56:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.686323+0000", 2024-08-30T23:56:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0389999999999995e-05 2024-08-30T23:56:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.691235+0000", 2024-08-30T23:56:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049119649999999999 2024-08-30T23:56:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 764/756 [PushOp(7:3f8ad8ed:::benchmark_data_smithi173_129296_object10539:head, version: 730'677, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3f8ad8ed:::benchmark_data_smithi173_129296_object10539:head@730'677, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:29.900241+0000", 2024-08-30T23:56:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 372.95397912700003, 2024-08-30T23:56:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69259163499999998, 2024-08-30T23:56:42.879 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900241+0000", 2024-08-30T23:56:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900241+0000", 2024-08-30T23:56:42.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900242+0000", 2024-08-30T23:56:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.06e-07 2024-08-30T23:56:42.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900310+0000", 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8467999999999996e-05 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591335+0000", 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69102508299999998 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591348+0000", 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2687000000000001e-05 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.884 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591489+0000", 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014089800000000001 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.885 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591503+0000", 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3661e-05 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.592833+0000", 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013301319999999999 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.886 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 767/756 [PushOp(7:3c0798f8:::benchmark_data_smithi173_129296_object12781:head, version: 730'820, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3c0798f8:::benchmark_data_smithi173_129296_object12781:head@730'820, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:33.363684+0000", 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 369.49053576199998, 2024-08-30T23:56:42.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.463505677, 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363684+0000", 2024-08-30T23:56:42.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363684+0000", 2024-08-30T23:56:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363691+0000", 2024-08-30T23:56:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5100000000000004e-06 2024-08-30T23:56:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363732+0000", 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0989999999999999e-05 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825755+0000", 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46202335 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825774+0000", 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8624999999999999e-05 2024-08-30T23:56:42.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825887+0000", 2024-08-30T23:56:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113225 2024-08-30T23:56:42.893 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825898+0000", 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1107000000000001e-05 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.827190+0000", 2024-08-30T23:56:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012918700000000001 2024-08-30T23:56:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 767/756 [PushOp(7:3b8bff84:::benchmark_data_smithi173_129296_object14349:head, version: 731'902, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3b8bff84:::benchmark_data_smithi173_129296_object14349:head@731'902, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:34.792891+0000", 2024-08-30T23:56:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 368.06132913099998, 2024-08-30T23:56:42.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75540186399999998, 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792891+0000", 2024-08-30T23:56:42.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792891+0000", 2024-08-30T23:56:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792893+0000", 2024-08-30T23:56:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4959999999999999e-06 2024-08-30T23:56:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792941+0000", 2024-08-30T23:56:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8745999999999997e-05 2024-08-30T23:56:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.545895+0000", 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75295403400000005 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.545914+0000", 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8332e-05 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.546016+0000", 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010255399999999999 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.546031+0000", 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4878000000000001e-05 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.548293+0000", 2024-08-30T23:56:42.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002261824 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 768/756 [PushOp(7:335c6484:::benchmark_data_smithi173_129296_object20286:head, version: 734'1249, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:335c6484:::benchmark_data_smithi173_129296_object20286:head@734'1249, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:40.509141+0000", 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 362.34507932299999, 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42131471799999998, 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.903 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509141+0000", 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509141+0000", 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509142+0000", 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509210+0000", 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7805999999999994e-05 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928275+0000", 2024-08-30T23:56:42.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.41906584200000002 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928303+0000", 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7571000000000001e-05 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928418+0000", 2024-08-30T23:56:42.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114824 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928433+0000", 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4754e-05 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.930456+0000", 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020230030000000002 2024-08-30T23:56:42.907 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 769/756 [PushReplyOp(7:5d5301f9:::benchmark_data_smithi173_129296_object21928:head)])", 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:41.792253+0000", 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 361.06196746400002, 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63873635799999995, 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.908 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792253+0000", 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792253+0000", 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792254+0000", 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.046e-06 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792260+0000", 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7849999999999997e-06 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.426542+0000", 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63428287000000005 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.426551+0000", 2024-08-30T23:56:42.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9460000000000004e-06 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.426617+0000", 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5765000000000005e-05 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.430989+0000", 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043719459999999998 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.912 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 770/756 [PushOp(7:34377c92:::benchmark_data_smithi173_129296_object26126:head, version: 734'1608, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:34377c92:::benchmark_data_smithi173_129296_object26126:head@734'1608, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:46.509140+0000", 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 356.34508050900001, 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75758949099999995, 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509140+0000", 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509140+0000", 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509141+0000", 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4750000000000001e-06 2024-08-30T23:56:42.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509183+0000", 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1538000000000001e-05 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265266+0000", 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75608357800000003 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265280+0000", 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3713000000000001e-05 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265360+0000", 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9806000000000001e-05 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265369+0000", 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4630000000000007e-06 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.266729+0000", 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013599180000000001 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 770/756 [PushReplyOp(7:5610cbb5:::benchmark_data_smithi173_129296_object27066:head)])", 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 356.30100564100002, 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80890829099999995, 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.917 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4e-07 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553220+0000", 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3499999999999999e-06 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.357935+0000", 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80471498500000005 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.357944+0000", 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5680000000000005e-06 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.358006+0000", 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1835999999999995e-05 2024-08-30T23:56:42.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.362123+0000", 2024-08-30T23:56:42.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041167119999999998 2024-08-30T23:56:42.921 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.921 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.922 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:42.925 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 771/756 [PushOp(7:3420d111:::benchmark_data_smithi173_129296_object27164:head, version: 735'1670, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3420d111:::benchmark_data_smithi173_129296_object27164:head@735'1670, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.925 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.927136+0000", 2024-08-30T23:56:42.925 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 354.92708397400003, 2024-08-30T23:56:42.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61515216699999997, 2024-08-30T23:56:42.925 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.926 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.926 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.926 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927136+0000", 2024-08-30T23:56:42.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927136+0000", 2024-08-30T23:56:42.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927138+0000", 2024-08-30T23:56:42.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.322e-06 2024-08-30T23:56:42.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927187+0000", 2024-08-30T23:56:42.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9691999999999998e-05 2024-08-30T23:56:42.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.540898+0000", 2024-08-30T23:56:42.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.613710423 2024-08-30T23:56:42.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.540917+0000", 2024-08-30T23:56:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9219999999999999e-05 2024-08-30T23:56:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.541014+0000", 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7189999999999996e-05 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.541027+0000", 2024-08-30T23:56:42.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3355e-05 2024-08-30T23:56:42.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.542288+0000", 2024-08-30T23:56:42.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001260965 2024-08-30T23:56:42.933 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.934 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.934 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.934 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 771/756 [PushReplyOp(7:5694c7fb:::benchmark_data_smithi173_129296_object28842:head)])", 2024-08-30T23:56:42.934 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:48.976044+0000", 2024-08-30T23:56:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 353.87817607300002, 2024-08-30T23:56:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55948937200000004, 2024-08-30T23:56:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:42.935 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976044+0000", 2024-08-30T23:56:42.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976044+0000", 2024-08-30T23:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976045+0000", 2024-08-30T23:56:42.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2200000000000003e-07 2024-08-30T23:56:42.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976050+0000", 2024-08-30T23:56:42.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7960000000000003e-06 2024-08-30T23:56:42.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.520874+0000", 2024-08-30T23:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54482465000000002 2024-08-30T23:56:42.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.520882+0000", 2024-08-30T23:56:42.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.469e-06 2024-08-30T23:56:42.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.526943+0000", 2024-08-30T23:56:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060610630000000002 2024-08-30T23:56:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.535534+0000", 2024-08-30T23:56:42.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085906720000000006 2024-08-30T23:56:42.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.943 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.944 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 788/756 [PushOp(7:3133a158:::benchmark_data_smithi173_129296_object77006:head, version: 749'4738, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3133a158:::benchmark_data_smithi173_129296_object77006:head@749'4738, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.944 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.467818+0000", 2024-08-30T23:56:42.944 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 299.38640210900002, 2024-08-30T23:56:42.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.41297604199999999, 2024-08-30T23:56:42.944 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.945 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.945 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.945 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467818+0000", 2024-08-30T23:56:42.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467818+0000", 2024-08-30T23:56:42.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467820+0000", 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.747e-06 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467883+0000", 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2741000000000001e-05 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878536+0000", 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.410653885 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878552+0000", 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5569e-05 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878644+0000", 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2165999999999998e-05 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878660+0000", 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.596e-05 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.880794+0000", 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021339739999999999 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.951 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 788/756 [PushOp(7:3db12067:::benchmark_data_smithi173_129296_object77016:head, version: 749'4739, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3db12067:::benchmark_data_smithi173_129296_object77016:head@749'4739, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.474899+0000", 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 299.37932098499999, 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.41134347300000002, 2024-08-30T23:56:42.952 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474899+0000", 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474899+0000", 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474901+0000", 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9870000000000002e-06 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474952+0000", 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1276000000000002e-05 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883728+0000", 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.40877536199999998 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883747+0000", 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9368000000000002e-05 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883844+0000", 2024-08-30T23:56:42.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6424999999999999e-05 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883857+0000", 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3427e-05 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.886243+0000", 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023856279999999999 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 788/756 [PushReplyOp(7:59265772:::benchmark_data_smithi173_129296_object57027:head)])", 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.476365+0000", 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 299.37785491400001, 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.510494157, 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476365+0000", 2024-08-30T23:56:42.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476365+0000", 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476367+0000", 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2699999999999999e-06 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476375+0000", 2024-08-30T23:56:42.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2579999999999995e-06 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.983888+0000", 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50751275399999995 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.983897+0000", 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6409999999999997e-06 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.983992+0000", 2024-08-30T23:56:42.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4927999999999996e-05 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.986859+0000", 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002867306 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:126882 7.c e795/792 7:35900c7d:::benchmark_data_smithi173_129296_object39766:head v 795'7834, mlcod=794'7831)", 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:02.133357+0000", 2024-08-30T23:56:42.960 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 280.72086335300003, 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56243918699999995, 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133357+0000", 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133357+0000", 2024-08-30T23:56:42.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133358+0000", 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.353e-06 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133367+0000", 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2790000000000004e-06 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693400+0000", 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56003286100000005 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693406+0000", 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0619999999999997e-06 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693461+0000", 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4506999999999998e-05 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693479+0000", 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7861999999999999e-05 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.695751+0000", 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022726489999999998 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.695796+0000", 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4613999999999997e-05 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.964 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:142377 7.8 e810/725 7:18815242:::benchmark_data_smithi173_129296_object55261:head v 810'8843, mlcod=810'8841)", 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.395840+0000", 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 245.458379858, 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61643008899999996, 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395840+0000", 2024-08-30T23:56:42.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395840+0000", 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395842+0000", 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3999999999999999e-06 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395850+0000", 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9079999999999995e-06 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007839+0000", 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61198960800000002 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007848+0000", 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6619999999999994e-06 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007916+0000", 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7676999999999996e-05 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007937+0000", 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1053999999999999e-05 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:42.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.012201+0000", 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042639740000000002 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.012270+0000", 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9806000000000002e-05 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:6054 8.e e847/840 8:7a0156cd:::benchmark_data_smithi173_143816_object6053:head v 847'378, mlcod=847'375)", 2024-08-30T23:56:42.969 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:59.025739+0000", 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 163.82848106500001, 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23799822800000001, 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025739+0000", 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025739+0000", 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025740+0000", 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8899999999999999e-07 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025910+0000", 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017013099999999999 2024-08-30T23:56:42.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262718+0000", 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23680768099999999 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262724+0000", 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5150000000000003e-06 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262826+0000", 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101853 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262845+0000", 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9255000000000001e-05 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.263700+0000", 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00085512499999999998 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.263737+0000", 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7079000000000002e-05 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:36537 8.e e862/840 8:78cdc0b0:::benchmark_data_smithi173_143816_object36536:head v 862'2321, mlcod=862'2318)", 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.400646+0000", 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 137.45357415800001, 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39470844900000002, 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.974 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400646+0000", 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400646+0000", 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400647+0000", 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7699999999999992e-07 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400688+0000", 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1428999999999999e-05 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793696+0000", 2024-08-30T23:56:42.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39300789699999999 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793705+0000", 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.132e-06 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793778+0000", 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2804000000000003e-05 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793805+0000", 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6543999999999999e-05 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.795319+0000", 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001514154 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.795354+0000", 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5512000000000002e-05 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:37753 8.a e864/840 8:57fc5b26:::benchmark_data_smithi173_143816_object37752:head v 864'2348, mlcod=864'2345)", 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.183034+0000", 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 135.671186173, 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25250152199999998, 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183034+0000", 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183034+0000", 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183035+0000", 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3099999999999996e-07 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183092+0000", 2024-08-30T23:56:42.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6889999999999999e-05 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434193+0000", 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25110125 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434202+0000", 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7280000000000001e-06 2024-08-30T23:56:42.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434259+0000", 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7534999999999997e-05 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434276+0000", 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6269999999999998e-05 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:42.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.435485+0000", 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012090029999999999 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.435536+0000", 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1014999999999999e-05 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:154526 8.1 e917/840 8:86f964dc:::benchmark_data_smithi173_143816_object56212:head v 917'9557, mlcod=917'9553)", 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:56:30.382511+0000", 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 12.471709506, 2024-08-30T23:56:42.983 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.21790885099999999, 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.382511+0000", 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.382511+0000", 2024-08-30T23:56:42.984 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.382512+0000", 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1900000000000001e-07 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.382518+0000", 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9399999999999999e-06 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:42.985 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.596672+0000", 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.214154552 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.596679+0000", 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2609999999999996e-06 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.596736+0000", 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6320999999999998e-05 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:42.986 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.596758+0000", 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2639000000000001e-05 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.600366+0000", 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0036074480000000001 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:30.600420+0000", 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3770999999999997e-05 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.987 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:42.988 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:42.988 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:56:43.010 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:43.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:43.139 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:43.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:43.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:43.372 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:43.405 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:43.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:56:43.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:43.539 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:56:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.549 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:43.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:43.650 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:43.673 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:43.775 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:43.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:43 smithi136 ceph-mon[20207]: pgmap v2046: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s; 4147/103389 objects degraded (4.011%) 2024-08-30T23:56:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1532936310' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:43 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:43.802 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:43 smithi173 ceph-mon[20554]: pgmap v2046: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s; 4147/103389 objects degraded (4.011%) 2024-08-30T23:56:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1532936310' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:43 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:43 smithi106 ceph-mon[16603]: pgmap v2046: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 944 op/s; 4147/103389 objects degraded (4.011%) 2024-08-30T23:56:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1532936310' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:43.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:43 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:43.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:43.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:43.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:43.949 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:56:44.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:44.050 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:44.152 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:44.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:44.285 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:44.314 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:44.392 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:44.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:44.446 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:44.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:44.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:44.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:44.699 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:44 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-30T23:56:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:44 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-30T23:56:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:44 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:56:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:44 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:44 smithi136 ceph-mon[20207]: osdmap e922: 12 total, 11 up, 12 in 2024-08-30T23:56:44.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:44 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-30T23:56:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:44 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-30T23:56:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:44 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:56:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:44 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:44 smithi173 ceph-mon[20554]: osdmap e922: 12 total, 11 up, 12 in 2024-08-30T23:56:44.827 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:44.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:44 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-30T23:56:44.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:44 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-30T23:56:44.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:44 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:56:44.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:44 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:44.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:44 smithi106 ceph-mon[16603]: osdmap e922: 12 total, 11 up, 12 in 2024-08-30T23:56:44.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:44.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:44.959 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:45.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:45.082 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:45.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.215 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:45.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:45.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:45.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:45.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:45.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:45.505 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:45.558 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:45.575 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:56:45.575 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:56:45.575 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:56:45.576 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:56:45.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:45.597 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:45.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:45 smithi106 ceph-mon[16603]: pgmap v2048: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 4147/100257 objects degraded (4.136%) 2024-08-30T23:56:45.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.732 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:45.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:45 smithi136 ceph-mon[20207]: pgmap v2048: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 4147/100257 objects degraded (4.136%) 2024-08-30T23:56:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:45 smithi173 ceph-mon[20554]: pgmap v2048: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 642 op/s; 4147/100257 objects degraded (4.136%) 2024-08-30T23:56:45.833 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:45.872 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:45.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:45.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:45.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:46.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:46.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:46.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:46.244 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:46.283 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:46.284 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:46.312 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:46.312 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:46.344 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:46.344 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:46.346 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:46.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:46.369 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:46.369 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:46.401 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:46.401 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:46.434 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:46.434 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:46.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:46.461 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:46.462 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:46.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:46.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:46.495 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:46.496 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:46.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:46.624 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:46.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:46.734 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:46.735 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 1.19 scrub starts 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 1.19 scrub ok 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 8.0 scrub starts 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4147/100257 objects degraded (4.136%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:46.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: osdmap e923: 12 total, 11 up, 12 in 2024-08-30T23:56:46.737 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1665980052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:46.737 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 1.1d scrub starts 2024-08-30T23:56:46.737 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:46 smithi106 ceph-mon[16603]: 1.1d scrub ok 2024-08-30T23:56:46.752 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:46.775 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:46.776 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:46.806 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:46.807 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 1.19 scrub starts 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 1.19 scrub ok 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 8.0 scrub starts 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4147/100257 objects degraded (4.136%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: osdmap e923: 12 total, 11 up, 12 in 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1665980052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 1.1d scrub starts 2024-08-30T23:56:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:46 smithi173 ceph-mon[20554]: 1.1d scrub ok 2024-08-30T23:56:46.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:46.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:46.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:46.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:47.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 1.19 scrub starts 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 1.19 scrub ok 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 8.0 scrub starts 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4147/100257 objects degraded (4.136%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: osdmap e923: 12 total, 11 up, 12 in 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1665980052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 1.1d scrub starts 2024-08-30T23:56:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:46 smithi136 ceph-mon[20207]: 1.1d scrub ok 2024-08-30T23:56:47.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:56:47.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.140 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:47.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:56:47.240 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:56:47.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:56:47.241 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:47.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:47.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:47.421 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:47.523 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:47.557 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:47.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:47.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:47.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:47.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:47 smithi106 ceph-mon[16603]: pgmap v2049: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s; 4147/95430 objects degraded (4.346%) 2024-08-30T23:56:47.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:47 smithi106 ceph-mon[16603]: 8.0 scrub ok 2024-08-30T23:56:47.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:47 smithi106 ceph-mon[16603]: 8.c scrub starts 2024-08-30T23:56:47.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:47 smithi173 ceph-mon[20554]: pgmap v2049: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s; 4147/95430 objects degraded (4.346%) 2024-08-30T23:56:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:47 smithi173 ceph-mon[20554]: 8.0 scrub ok 2024-08-30T23:56:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:47 smithi173 ceph-mon[20554]: 8.c scrub starts 2024-08-30T23:56:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.825 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:47.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:47.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:47.958 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:47.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:48.032 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:48.032 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:47 smithi136 ceph-mon[20207]: pgmap v2049: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 708 op/s; 4147/95430 objects degraded (4.346%) 2024-08-30T23:56:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:47 smithi136 ceph-mon[20207]: 8.0 scrub ok 2024-08-30T23:56:48.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:47 smithi136 ceph-mon[20207]: 8.c scrub starts 2024-08-30T23:56:48.060 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:48.089 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:48.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:48.215 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:48.317 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:48.346 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:48.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:48.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:48.607 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:48.687 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:48.688 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:48.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:48.715 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:56:48.752 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:48.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:48 smithi106 ceph-mon[16603]: 8.c scrub ok 2024-08-30T23:56:48.779 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:48 smithi106 ceph-mon[16603]: 8.f scrub starts 2024-08-30T23:56:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:48 smithi173 ceph-mon[20554]: 8.c scrub ok 2024-08-30T23:56:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:48 smithi173 ceph-mon[20554]: 8.f scrub starts 2024-08-30T23:56:48.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:48.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:48.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:48.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:49.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:48 smithi136 ceph-mon[20207]: 8.c scrub ok 2024-08-30T23:56:49.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:48 smithi136 ceph-mon[20207]: 8.f scrub starts 2024-08-30T23:56:49.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:49.136 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:49.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:49.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:49.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:49.401 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:49.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:49.525 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:49.627 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:49.658 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:49.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:49 smithi106 ceph-mon[16603]: pgmap v2051: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 3607/87609 objects degraded (4.117%) 2024-08-30T23:56:49.692 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:49 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:56:49.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:49 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:56:49.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:49 smithi106 ceph-mon[16603]: 8.f scrub ok 2024-08-30T23:56:49.693 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:49 smithi106 ceph-mon[16603]: 8.7 scrub starts 2024-08-30T23:56:49.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:56:49.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:49.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:49 smithi173 ceph-mon[20554]: pgmap v2051: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 3607/87609 objects degraded (4.117%) 2024-08-30T23:56:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:49 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:56:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:49 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:56:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:49 smithi173 ceph-mon[20554]: 8.f scrub ok 2024-08-30T23:56:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:49 smithi173 ceph-mon[20554]: 8.7 scrub starts 2024-08-30T23:56:49.863 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:49.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:49.866 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:49.908 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:56:49.908 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:56:49.908 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:56:49.909 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:56:49.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.002 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:50.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:49 smithi136 ceph-mon[20207]: pgmap v2051: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 909 op/s; 3607/87609 objects degraded (4.117%) 2024-08-30T23:56:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:49 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:56:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:49 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:56:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:49 smithi136 ceph-mon[20207]: 8.f scrub ok 2024-08-30T23:56:50.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:49 smithi136 ceph-mon[20207]: 8.7 scrub starts 2024-08-30T23:56:50.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.145 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:50.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:50.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:50.273 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.274 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:50.374 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:50.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:50.499 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:50.521 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:50.561 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:56:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:56:50.571 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:56:50.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:56:50.623 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:50.650 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:50.752 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:50.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.781 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:50.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:50 smithi106 ceph-mon[16603]: 8.b scrub starts 2024-08-30T23:56:50.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3199310139' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:50.803 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:50 smithi106 ceph-mon[16603]: 8.7 scrub ok 2024-08-30T23:56:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:50 smithi173 ceph-mon[20554]: 8.b scrub starts 2024-08-30T23:56:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3199310139' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:50 smithi173 ceph-mon[20554]: 8.7 scrub ok 2024-08-30T23:56:50.882 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:50.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:50.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:50.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:51.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:51.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:50 smithi136 ceph-mon[20207]: 8.b scrub starts 2024-08-30T23:56:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3199310139' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:56:51.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:50 smithi136 ceph-mon[20207]: 8.7 scrub ok 2024-08-30T23:56:51.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:51.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:51.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:51.297 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:51.399 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:51.430 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:51.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:51.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:51.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:51.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.729 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:51.736 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:56:51.736 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: 8.b scrub ok 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: pgmap v2052: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s; 3607/84420 objects degraded (4.273%) 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3199310139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: osdmap e924: 12 total, 11 up, 12 in 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3607/84420 objects degraded (4.273%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:51.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:51 smithi106 ceph-mon[16603]: osdmap e925: 12 total, 11 up, 12 in 2024-08-30T23:56:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: 8.b scrub ok 2024-08-30T23:56:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: pgmap v2052: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s; 3607/84420 objects degraded (4.273%) 2024-08-30T23:56:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3199310139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: osdmap e924: 12 total, 11 up, 12 in 2024-08-30T23:56:51.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3607/84420 objects degraded (4.273%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:51.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:51 smithi173 ceph-mon[20554]: osdmap e925: 12 total, 11 up, 12 in 2024-08-30T23:56:51.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:51.862 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:51.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:51.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:51.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:52.023 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: 8.b scrub ok 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: pgmap v2052: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 790 op/s; 3607/84420 objects degraded (4.273%) 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3199310139' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: osdmap e924: 12 total, 11 up, 12 in 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3607/84420 objects degraded (4.273%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:52.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:51 smithi136 ceph-mon[20207]: osdmap e925: 12 total, 11 up, 12 in 2024-08-30T23:56:52.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:52.115 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:52.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:52.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:52.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.367 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:52.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:52.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:52.491 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:52.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:56:52.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:52.620 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:52.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:52.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:52.761 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:52.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:52.878 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:52.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:52.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:53.001 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:53.060 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:56:53.060 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:56:53.087 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:56:53.087 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:56:53.103 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:53.111 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:56:53.111 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:56:53.125 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:53.141 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:56:53.141 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:56:53.172 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:56:53.173 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:56:53.197 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:56:53.197 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:56:53.224 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:56:53.225 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:56:53.225 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:56:53.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:53.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:56:53.258 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:56:53.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:53.289 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:56:53.289 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:56:53.327 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:56:53.328 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:56:53.357 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:56:53.358 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:56:53.361 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:53.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:53.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:53.509 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:53.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:56:53.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:53 smithi106 ceph-mon[16603]: pgmap v2055: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 3125/78270 objects degraded (3.993%) 2024-08-30T23:56:53.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2553557477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:53.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:53.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:53.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:53.761 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:53 smithi173 ceph-mon[20554]: pgmap v2055: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 3125/78270 objects degraded (3.993%) 2024-08-30T23:56:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2553557477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:53.841 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:53 smithi136 ceph-mon[20207]: pgmap v2055: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 3125/78270 objects degraded (3.993%) 2024-08-30T23:56:53.841 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2553557477' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:56:53.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:53.870 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:56:53.870 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:56:53.870 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:56:53.870 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:56:53.871 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:56:53.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:53.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:53.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:54.023 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:54.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:54.153 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:54.255 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:54.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:54.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:54.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:54.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:54.544 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:54.546 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:54.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:54.672 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:54.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:54.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:54.906 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:54.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:54.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:55.045 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:55.068 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:56:55.084 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:56:55.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:56:55.127 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:55.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:55.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.204 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:55.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:55.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:56:55.329 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:56:55.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:55.452 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:55.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:55.575 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.577 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:55.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:55 smithi106 ceph-mon[16603]: pgmap v2056: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 3125/75234 objects degraded (4.154%) 2024-08-30T23:56:55.678 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:55.730 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:55 smithi173 ceph-mon[20554]: pgmap v2056: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 3125/75234 objects degraded (4.154%) 2024-08-30T23:56:55.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:55.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:55.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:55.897 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:55 smithi136 ceph-mon[20207]: pgmap v2056: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s; 3125/75234 objects degraded (4.154%) 2024-08-30T23:56:55.971 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:56.001 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:56.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:56.139 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:56.202 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:56:56.202 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:199538 6.c 6:39c64173:::benchmark_data_smithi173_114000_object99740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.043898+0000", 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 489.882230945, 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50821048800000002, 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:56:56.203 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199538 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043898+0000", 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043898+0000", 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043899+0000", 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8899999999999998e-07 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043904+0000", 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8960000000000002e-06 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.205 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043919+0000", 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4664000000000001e-05 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043928+0000", 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5480000000000007e-06 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.043980+0000", 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1727999999999997e-05 2024-08-30T23:56:56.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.044056+0000", 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5982999999999994e-05 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,8", 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.044134+0000", 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7832999999999998e-05 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.207 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.048295+0000", 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041609580000000002 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.049339+0000", 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010443379999999999 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.552032+0000", 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50269233999999996 2024-08-30T23:56:56.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.552072+0000", 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0627000000000003e-05 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.552109+0000", 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6584000000000001e-05 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27034.0:199538 6.c e720/619 ondisk, result = 0)", 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.048059+0000", 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 489.87807064600003, 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50405534900000004, 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.048059+0000", 2024-08-30T23:56:56.210 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.048059+0000", 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.048059+0000", 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1900000000000002e-07 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.211 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.048064+0000", 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3680000000000004e-06 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.551935+0000", 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50387167700000002 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.551944+0000", 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9889999999999998e-06 2024-08-30T23:56:56.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.552018+0000", 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3147999999999998e-05 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.552026+0000", 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5699999999999993e-06 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.213 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.552114+0000", 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7878000000000001e-05 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:199547 6.c 6:3a2ea6ac:::benchmark_data_smithi173_114000_object99749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.058178+0000", 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 489.86795144500002, 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.87962152100000002, 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.214 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199547 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058178+0000", 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.215 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058178+0000", 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058179+0000", 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1000000000000001e-06 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058185+0000", 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8379999999999998e-06 2024-08-30T23:56:56.216 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058197+0000", 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1875999999999999e-05 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058204+0000", 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5109999999999997e-06 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.217 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058252+0000", 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7799000000000001e-05 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058336+0000", 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3633000000000004e-05 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 7,8", 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.058412+0000", 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6587000000000004e-05 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.218 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.063130+0000", 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0047173500000000004 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.063140+0000", 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0730000000000001e-05 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937672+0000", 2024-08-30T23:56:56.219 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.874532117 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937741+0000", 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9047000000000005e-05 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937799+0000", 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7933000000000003e-05 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.220 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27034.0:199547 6.c e720/619 ondisk, result = 0)", 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.062359+0000", 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 489.86377027499998, 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.87544755799999996, 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.221 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.062359+0000", 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.062359+0000", 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.062360+0000", 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8400000000000003e-07 2024-08-30T23:56:56.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.062366+0000", 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9030000000000001e-06 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937569+0000", 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.87520300699999998 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.223 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937577+0000", 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9310000000000008e-06 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937650+0000", 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3585999999999996e-05 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937663+0000", 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3077e-05 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.937807+0000", 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014327 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27034.0:199589 6.0 6:08f52e91:::benchmark_data_smithi173_114000_object99791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.152542+0000", 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 489.77358727299998, 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72903101999999997, 2024-08-30T23:56:56.225 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27034", 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 199589 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152542+0000", 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152542+0000", 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152543+0000", 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2699999999999999e-06 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152549+0000", 2024-08-30T23:56:56.227 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6540000000000004e-06 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152560+0000", 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1488000000000001e-05 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152567+0000", 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1929999999999998e-06 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.228 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152614+0000", 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7034999999999999e-05 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152737+0000", 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012375099999999999 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.152818+0000", 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0518000000000004e-05 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.158177+0000", 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0053585200000000003 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.158468+0000", 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00029133799999999998 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881470+0000", 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72300222400000003 2024-08-30T23:56:56.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881530+0000", 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0040000000000001e-05 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881573+0000", 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.2988999999999998e-05 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27034.0:199589 6.0 e720/619 ondisk, result = 0)", 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:48:46.158178+0000", 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 489.76795151800002, 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72339978299999996, 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.158178+0000", 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.158178+0000", 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.158179+0000", 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9599999999999999e-07 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.158181+0000", 2024-08-30T23:56:56.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.931e-06 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881385+0000", 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72320337499999998 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881394+0000", 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4269999999999997e-06 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881450+0000", 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6134999999999997e-05 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881461+0000", 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0971999999999999e-05 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:48:46.881578+0000", 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116247 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.235 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.6 765/762 [PushReplyOp(7:6ec96c14:::benchmark_data_smithi173_129296_object3483:head)])", 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:30.462410+0000", 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 385.46371976500001, 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62992348499999995, 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.462410+0000", 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.462410+0000", 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.462411+0000", 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.043e-06 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.462417+0000", 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4989999999999999e-06 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.086088+0000", 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62367124799999996 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.086096+0000", 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9980000000000003e-06 2024-08-30T23:56:56.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.086158+0000", 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1439999999999995e-05 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.092333+0000", 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006175257 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.a 766/756 [PushOp(7:5141fea4:::benchmark_data_smithi173_129296_object12246:head, version: 730'735, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5141fea4:::benchmark_data_smithi173_129296_object12246:head@730'735, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:31.788613+0000", 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 384.13751617100002, 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75060152599999996, 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.788613+0000", 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.788613+0000", 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.788614+0000", 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2640000000000001e-06 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.788646+0000", 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1708000000000002e-05 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:32.537876+0000", 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74923014200000004 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:32.537895+0000", 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9194999999999998e-05 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:32.538012+0000", 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000116783 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:32.538023+0000", 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.083e-05 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:32.539215+0000", 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001191604 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.243 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.a 768/756 [PushOp(7:5cf3d093:::benchmark_data_smithi173_129296_object19271:head, version: 734'1164, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:5cf3d093:::benchmark_data_smithi173_129296_object19271:head@734'1164, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:38.901696+0000", 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 377.024433741, 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81187862799999999, 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.901696+0000", 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.901696+0000", 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.901698+0000", 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9350000000000001e-06 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.901734+0000", 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6337000000000001e-05 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:39.711906+0000", 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81017255499999996 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:39.711920+0000", 2024-08-30T23:56:56.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3828999999999999e-05 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:39.711992+0000", 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1439000000000005e-05 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:39.712000+0000", 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6770000000000001e-06 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:39.713574+0000", 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001573856 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.6 769/762 [PushReplyOp(7:6577fb0b:::benchmark_data_smithi173_129296_object14079:head)])", 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:45.267550+0000", 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 370.65857913899998, 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45030445899999999, 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.248 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.267550+0000", 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.267550+0000", 2024-08-30T23:56:56.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.267551+0000", 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8700000000000004e-07 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.267556+0000", 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1359999999999996e-06 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.712898+0000", 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.445341875 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.712908+0000", 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.8539999999999992e-06 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.712973+0000", 2024-08-30T23:56:56.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4987999999999995e-05 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:45.717855+0000", 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0048817189999999996 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.a 787/756 [PushOp(7:50360385:::benchmark_data_smithi173_129296_object56226:head, version: 744'3471, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:50360385:::benchmark_data_smithi173_129296_object56226:head@744'3471, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:56:56.252 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:42.229135+0000", 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 313.69699450000002, 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.53249119199999995, 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.229135+0000", 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.229135+0000", 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.229135+0000", 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7800000000000001e-07 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.229163+0000", 2024-08-30T23:56:56.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7804e-05 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.758715+0000", 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.52955158099999999 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.758733+0000", 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8578999999999999e-05 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.758842+0000", 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109108 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.758858+0000", 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5744999999999999e-05 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.761626+0000", 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027677969999999998 2024-08-30T23:56:56.256 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:117770 7.0 7:02bb615e:::benchmark_data_smithi173_129296_object30654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e789)", 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.981222+0000", 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 311.94490724500002, 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28644283100000001, 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.257 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 117770 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.981222+0000", 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.981222+0000", 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.981222+0000", 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1400000000000001e-07 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.981223+0000", 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0610000000000001e-06 2024-08-30T23:56:56.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.981226+0000", 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6989999999999999e-06 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.981228+0000", 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4090000000000001e-06 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.982782+0000", 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00155349 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for peered", 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.982789+0000", 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9530000000000002e-06 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.228120+0000", 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.245331356 2024-08-30T23:56:56.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.228277+0000", 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015649699999999999 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.228427+0000", 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000150228 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.239245+0000", 2024-08-30T23:56:56.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.010818196 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.239259+0000", 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4256e-05 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.264738+0000", 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.025478786 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.264791+0000", 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2441999999999997e-05 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:44.267665+0000", 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0028742440000000002 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.6 795/762 [PushReplyOp(7:6562d669:::benchmark_data_smithi173_129296_object58724:head)])", 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:02.808209+0000", 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 293.11792031499999, 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28362149599999997, 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.808209+0000", 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.808209+0000", 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.808210+0000", 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5099999999999998e-07 2024-08-30T23:56:56.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.808216+0000", 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7239999999999997e-06 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:03.089730+0000", 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28151421700000001 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:03.089737+0000", 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1409999999999999e-06 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:03.089836+0000", 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9065000000000001e-05 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:03.091831+0000", 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019943980000000001 2024-08-30T23:56:56.268 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.0 810/788 [PushReplyOp(7:034c0a32:::benchmark_data_smithi173_129296_object65457:head)])", 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.583947+0000", 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 258.342182283, 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.386556917, 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.583947+0000", 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.583947+0000", 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.583948+0000", 2024-08-30T23:56:56.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.097e-06 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.583954+0000", 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.925e-06 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.968998+0000", 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38504373600000003 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.969007+0000", 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1630000000000002e-06 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.969083+0000", 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6334999999999993e-05 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.970504+0000", 2024-08-30T23:56:56.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014206609999999999 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:161591 7.c e823/792 7:3ef003dd:::benchmark_data_smithi173_129296_object74475:head v 823'9948, mlcod=823'9945)", 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:12.394269+0000", 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 223.53186045300001, 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23875217500000001, 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.273 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.394269+0000", 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.394269+0000", 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.394270+0000", 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1519999999999999e-06 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.394278+0000", 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3720000000000005e-06 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.622922+0000", 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.228643507 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.622927+0000", 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5400000000000003e-06 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.623014+0000", 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6883000000000001e-05 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.623034+0000", 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0098999999999998e-05 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.632976+0000", 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0099413709999999992 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:12.633021+0000", 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5250999999999998e-05 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.277 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36272 8.0 8:01a4ab1b:::benchmark_data_smithi173_143816_object36271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:24.299406+0000", 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.62672328100001, 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22687872000000001, 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:56:56.278 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36272 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299406+0000", 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299406+0000", 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299408+0000", 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.939e-06 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299444+0000", 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6256e-05 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299461+0000", 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7184e-05 2024-08-30T23:56:56.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299469+0000", 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3950000000000003e-06 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299534+0000", 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4725000000000001e-05 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299684+0000", 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00015013 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.299843+0000", 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000159557 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.301088+0000", 2024-08-30T23:56:56.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001245091 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.301394+0000", 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030570499999999997 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.526223+0000", 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.22482919800000001 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.526256+0000", 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3192999999999999e-05 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.526285+0000", 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8347000000000002e-05 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36277 8.0 8:0e910301:::benchmark_data_smithi173_143816_object36276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e861)", 2024-08-30T23:56:56.284 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:24.310418+0000", 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 151.61571131299999, 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24962490600000001, 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36277 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310418+0000", 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310418+0000", 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310425+0000", 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1389999999999998e-06 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310459+0000", 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3361000000000001e-05 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310475+0000", 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6104e-05 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310482+0000", 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.888e-06 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310546+0000", 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4425999999999996e-05 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310690+0000", 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014440000000000001 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.310845+0000", 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000154559 2024-08-30T23:56:56.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.312113+0000", 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001267904 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.312259+0000", 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000146587 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.559972+0000", 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24771269600000001 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.560012+0000", 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9520000000000001e-05 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:24.560043+0000", 2024-08-30T23:56:56.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1322000000000003e-05 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "replica_scrub(pg: 1.16,from:0'0,to:0'0,epoch:880/879,start:1:68000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:55:04.219220+0000", 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 111.706909264, 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.98180668599999998, 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:56:56.291 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:04.219220+0000", 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:04.219220+0000", 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:04.219221+0000", 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8400000000000004e-07 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:04.219226+0000", 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1669999999999998e-06 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.200906+0000", 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.98168038000000002 2024-08-30T23:56:56.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.200917+0000", 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0876999999999999e-05 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.200983+0000", 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5680999999999997e-05 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.200996+0000", 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3559e-05 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:55:05.201027+0000", 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0338e-05 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:146291 8.c e913/866 8:3bf8623b:::benchmark_data_smithi173_143816_object47977:head v 913'9088, mlcod=913'9084)", 2024-08-30T23:56:56.295 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:56:20.563861+0000", 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 35.362267887000002, 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.174730465, 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.563861+0000", 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.563861+0000", 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.563869+0000", 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2899999999999997e-06 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.563876+0000", 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5089999999999996e-06 2024-08-30T23:56:56.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.735067+0000", 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17119086 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.735080+0000", 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3165e-05 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.735151+0000", 2024-08-30T23:56:56.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0729999999999995e-05 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.735171+0000", 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.0486e-05 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.738555+0000", 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003383259 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.299 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.738592+0000", 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7166000000000003e-05 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:146295 8.0 8:05e346d7:::benchmark_data_smithi173_143816_object47981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e913)", 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:56:20.567507+0000", 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 35.35862186, 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.175439506, 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:56:56.300 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 146295 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567507+0000", 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:56:56.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567507+0000", 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567509+0000", 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3599999999999999e-06 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567515+0000", 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0599999999999996e-06 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:56:56.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567530+0000", 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5074e-05 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567538+0000", 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3289999999999994e-06 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567605+0000", 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6205999999999996e-05 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567722+0000", 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000117816 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.567823+0000", 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010021100000000001 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.572310+0000", 2024-08-30T23:56:56.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044879079999999997 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.572381+0000", 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0635000000000004e-05 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.742861+0000", 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17048028900000001 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:56:56.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.742912+0000", 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0813999999999998e-05 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:56:20.742947+0000", 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4804000000000001e-05 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:56:56.306 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:56:56.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:56:56.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:56:56.442 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:56.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:56.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:56.616 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:56.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:56.756 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3125/75234 objects degraded (4.154%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2410711491' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:56 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:56.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:56.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:56.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:56.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:56:57.005 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:56:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3125/75234 objects degraded (4.154%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:57.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2410711491' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:57.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:56 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:57.043 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:56:57.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3125/75234 objects degraded (4.154%), 3 pgs degraded (PG_DEGRADED) 2024-08-30T23:56:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2410711491' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:57.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:56 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:56:57.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:57.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:56:57.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:56:57.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:57.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.261 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:57.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:57.395 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:57.497 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:57.524 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:57.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:57 smithi106 ceph-mon[16603]: pgmap v2057: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 3125/70854 objects degraded (4.410%) 2024-08-30T23:56:57.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:57 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:57.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:57 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:57.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:57 smithi106 ceph-mon[16603]: osdmap e926: 12 total, 11 up, 12 in 2024-08-30T23:56:57.625 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:57.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.655 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:57.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:57.785 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:57 smithi173 ceph-mon[20554]: pgmap v2057: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 3125/70854 objects degraded (4.410%) 2024-08-30T23:56:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:57 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:57 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:57 smithi173 ceph-mon[20554]: osdmap e926: 12 total, 11 up, 12 in 2024-08-30T23:56:57.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:57.915 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:57.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:58.017 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:57 smithi136 ceph-mon[20207]: pgmap v2057: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 697 op/s; 3125/70854 objects degraded (4.410%) 2024-08-30T23:56:58.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:57 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:56:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:57 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:56:58.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:57 smithi136 ceph-mon[20207]: osdmap e926: 12 total, 11 up, 12 in 2024-08-30T23:56:58.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:58.144 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:56:58.173 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:56:58.275 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:56:58.280 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:56:58.304 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.305 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:56:58.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:56:58.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:58.428 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:58.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:58.563 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.564 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:58.599 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:56:58.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:56:58.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:56:58.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:58.829 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:58.878 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:56:58.878 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:56:58.878 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:56:58.878 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:56:58.878 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:56:58.878 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:56:58.879 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:56:58.879 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:56:58.879 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:56:58.879 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:56:58.879 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:56:58.879 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:56:58.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:56:58.953 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:58.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:56:59.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:59.080 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:59.157 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:56:59.157 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:56:59.157 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:56:59.157 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:56:59.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:56:59.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:56:59.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:56:59.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:56:59.373 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:56:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:56:59.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:59.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:59.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:59.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:59.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:56:59.771 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:56:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:56:59.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:56:59 smithi106 ceph-mon[16603]: pgmap v2059: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 2608/63690 objects degraded (4.095%) 2024-08-30T23:56:59.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:56:59.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:56:59.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:56:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:56:59.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:56:59.946 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:56:59 smithi136 ceph-mon[20207]: pgmap v2059: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 2608/63690 objects degraded (4.095%) 2024-08-30T23:56:59.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:56:59.997 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:00.048 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:00.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:56:59 smithi173 ceph-mon[20554]: pgmap v2059: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 950 op/s; 2608/63690 objects degraded (4.095%) 2024-08-30T23:57:00.150 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:00.178 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:00.257 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:00.281 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:00.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:00.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:00.436 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:00.538 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:00.559 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:00.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:00.685 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:00 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:57:00.688 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:00.776 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:00 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:57:00.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:00.802 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:57:00.802 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:57:00.802 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:57:00.803 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:57:00.803 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:57:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:00.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:00.822 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:00.823 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:57:00.892 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:00.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:00.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:00.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:01.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:01.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:00 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:57:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.086 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:01.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:01.188 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:01.209 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:01.311 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:01.341 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:01.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:01.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:57:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.476 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:01.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:01.545 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:01.546 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:01.572 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:01.573 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:01.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:01.603 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:01.605 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:01.605 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:01.619 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:01.635 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:01.635 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:01.670 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:01.671 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:01.671 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:01.701 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:01.702 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:01.705 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:01.732 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:01.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:01.732 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:01.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:01.759 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:01.760 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:01.760 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:01.786 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:01.787 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: pgmap v2060: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 2608/60639 objects degraded (4.301%) 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2608/60639 objects degraded (4.301%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: osdmap e927: 12 total, 11 up, 12 in 2024-08-30T23:57:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3690001280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:01.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: pgmap v2060: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 2608/60639 objects degraded (4.301%) 2024-08-30T23:57:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:57:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:57:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:57:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2608/60639 objects degraded (4.301%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:01.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: osdmap e927: 12 total, 11 up, 12 in 2024-08-30T23:57:01.860 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3690001280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:01.862 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:01.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:01.990 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:01.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:02.013 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:02.013 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:02.041 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:02.041 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: pgmap v2060: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 2608/60639 objects degraded (4.301%) 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2608/60639 objects degraded (4.301%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: osdmap e927: 12 total, 11 up, 12 in 2024-08-30T23:57:02.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3690001280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:02.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:02.117 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:02.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:02.241 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:02.319 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:02.319 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:02.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:02.373 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:02.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:02.501 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:02.603 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:02.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:02.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:02.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:02.866 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:02.893 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:02.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:02.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:03.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:03.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:57:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:03.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:03.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:03.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:03.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:03.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:03.305 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:03.306 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:03.308 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:03.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:03.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:03.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.534 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:03.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:03.591 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:57:03.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:03.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:03.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:03 smithi136 ceph-mon[20207]: pgmap v2062: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s; 2124/54747 objects degraded (3.880%) 2024-08-30T23:57:03.704 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3803593307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:03.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:03.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:03.814 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:03.814 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:03 smithi106 ceph-mon[16603]: pgmap v2062: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s; 2124/54747 objects degraded (3.880%) 2024-08-30T23:57:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3803593307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:03.894 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:03.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:03.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:04.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:04.069 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:04.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:03 smithi173 ceph-mon[20554]: pgmap v2062: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 853 op/s; 2124/54747 objects degraded (3.880%) 2024-08-30T23:57:04.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3803593307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:04.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:04.193 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:04.194 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-30T23:57:04.198 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.199 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:04.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:04.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:04.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:04.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:04.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:04.589 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:04.635 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:04.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:57:04.690 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.719 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:04.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:04 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3803593307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:04 smithi136 ceph-mon[20207]: osdmap e928: 12 total, 11 up, 12 in 2024-08-30T23:57:04.820 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:04.845 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:04.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:04 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3803593307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:04 smithi106 ceph-mon[16603]: osdmap e928: 12 total, 11 up, 12 in 2024-08-30T23:57:04.947 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:04.955 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:04.971 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:04.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:05.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:57:05.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:05.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:04 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:05.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3803593307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:05.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:04 smithi173 ceph-mon[20554]: osdmap e928: 12 total, 11 up, 12 in 2024-08-30T23:57:05.096 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:05.198 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:05.226 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:05.286 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:05.287 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:05.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.354 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:05.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:05.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:05.478 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:05.521 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:05.522 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:05.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:05.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:05.707 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:05.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:05.806 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:05 smithi136 ceph-mon[20207]: pgmap v2064: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 2124/51873 objects degraded (4.095%) 2024-08-30T23:57:05.807 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:05 smithi136 ceph-mon[20207]: 8.3 scrub starts 2024-08-30T23:57:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:05 smithi106 ceph-mon[16603]: pgmap v2064: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 2124/51873 objects degraded (4.095%) 2024-08-30T23:57:05.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:05 smithi106 ceph-mon[16603]: 8.3 scrub starts 2024-08-30T23:57:05.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:05.944 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:05.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:06.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:05 smithi173 ceph-mon[20554]: pgmap v2064: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 685 op/s; 2124/51873 objects degraded (4.095%) 2024-08-30T23:57:06.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:05 smithi173 ceph-mon[20554]: 8.3 scrub starts 2024-08-30T23:57:06.082 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:06.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:06.220 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:06.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:57:06.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:06.348 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:06.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:06.474 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:06.518 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:06.573 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:06.573 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-30T23:57:06.575 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:06.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:06.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:57:06.627 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:06.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:06.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:06.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:06 smithi106 ceph-mon[16603]: 8.3 scrub ok 2024-08-30T23:57:06.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2124/51873 objects degraded (4.095%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:06.804 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:06 smithi106 ceph-mon[16603]: osdmap e929: 12 total, 11 up, 12 in 2024-08-30T23:57:06.834 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:06.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:06.947 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:06 smithi136 ceph-mon[20207]: 8.3 scrub ok 2024-08-30T23:57:06.947 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2124/51873 objects degraded (4.095%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:06.947 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:06 smithi136 ceph-mon[20207]: osdmap e929: 12 total, 11 up, 12 in 2024-08-30T23:57:06.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:06.994 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:06.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:07.068 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:07.068 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:07.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:06 smithi173 ceph-mon[20554]: 8.3 scrub ok 2024-08-30T23:57:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2124/51873 objects degraded (4.095%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:06 smithi173 ceph-mon[20554]: osdmap e929: 12 total, 11 up, 12 in 2024-08-30T23:57:07.096 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:07.131 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:07.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:07.257 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:07.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:07.265 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:07.265 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:07.277 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:07.305 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:07.305 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:07.337 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:07.337 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:07.365 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:07.366 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:07.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:07.387 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:07.388 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.390 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:07.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:07.410 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:07.410 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:07.442 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:07.442 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:07.442 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:07.471 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:07.471 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:07.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:07.502 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:07.502 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:07.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:07.538 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:07.538 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:07.574 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:07.574 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:07.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:07.650 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:07.752 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:07.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:57:07.780 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:07.883 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:07.908 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:07.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:08.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:07 smithi136 ceph-mon[20207]: pgmap v2065: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 988 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2124/47427 objects degraded (4.478%) 2024-08-30T23:57:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1126857615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:07 smithi136 ceph-mon[20207]: 8.a scrub starts 2024-08-30T23:57:08.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:07 smithi136 ceph-mon[20207]: 8.a scrub ok 2024-08-30T23:57:08.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:07 smithi173 ceph-mon[20554]: pgmap v2065: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 988 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2124/47427 objects degraded (4.478%) 2024-08-30T23:57:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1126857615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:07 smithi173 ceph-mon[20554]: 8.a scrub starts 2024-08-30T23:57:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:07 smithi173 ceph-mon[20554]: 8.a scrub ok 2024-08-30T23:57:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:07 smithi106 ceph-mon[16603]: pgmap v2065: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 988 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 2124/47427 objects degraded (4.478%) 2024-08-30T23:57:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1126857615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:07 smithi106 ceph-mon[16603]: 8.a scrub starts 2024-08-30T23:57:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:07 smithi106 ceph-mon[16603]: 8.a scrub ok 2024-08-30T23:57:08.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:08.151 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:08.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:08.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:08.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:08.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:08.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:08.518 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:08.518 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:08.518 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:08.518 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:08.519 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:08.556 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:08.577 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:08.634 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:57:08.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:08.701 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:08.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:08.805 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:08.805 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:57:08.805 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:57:08.806 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:08.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:08.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:08.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:08 smithi106 ceph-mon[16603]: osdmap e930: 12 total, 11 up, 12 in 2024-08-30T23:57:08.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:08 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:57:08.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:08 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:57:08.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3203226034' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:08.838 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:08.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:08.969 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:08.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:09.024 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:09.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:08 smithi136 ceph-mon[20207]: osdmap e930: 12 total, 11 up, 12 in 2024-08-30T23:57:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:08 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:57:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:08 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:57:09.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3203226034' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:09.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:08 smithi173 ceph-mon[20554]: osdmap e930: 12 total, 11 up, 12 in 2024-08-30T23:57:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:08 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:57:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:08 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:57:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3203226034' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:09.095 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:09.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:09.219 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:09.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:09.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.357 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:09.458 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:09.480 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:09.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:57:09.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:09.605 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:09.697 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:09 smithi106 ceph-mon[16603]: pgmap v2068: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 824 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 1594/39528 objects degraded (4.033%) 2024-08-30T23:57:09.698 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:09 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:09.698 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3203226034' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:09.698 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:09 smithi106 ceph-mon[16603]: osdmap e931: 12 total, 11 up, 12 in 2024-08-30T23:57:09.706 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:09.727 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:09.735 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:09.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-30T23:57:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:09 smithi173 ceph-mon[20554]: pgmap v2068: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 824 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 1594/39528 objects degraded (4.033%) 2024-08-30T23:57:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:09 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3203226034' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:09.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:09 smithi173 ceph-mon[20554]: osdmap e931: 12 total, 11 up, 12 in 2024-08-30T23:57:09.829 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:09.867 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:09.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:09.918 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:09.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:10.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.005 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:10.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:09 smithi136 ceph-mon[20207]: pgmap v2068: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 824 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s; 1594/39528 objects degraded (4.033%) 2024-08-30T23:57:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:09 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3203226034' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:10.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:09 smithi136 ceph-mon[20207]: osdmap e931: 12 total, 11 up, 12 in 2024-08-30T23:57:10.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:10.133 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:10.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:10.267 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:10.369 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:10.397 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:10.485 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:10.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:10.520 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:10.528 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:57:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:57:10.528 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:57:10.528 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:10.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:10.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:10.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:10.776 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:10.878 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:10.907 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:10.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:10.997 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:11.007 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:11.025 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:57:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.030 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:11.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:11.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:11.159 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.160 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:11.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:11.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:11.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:11.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:11.496 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:57:11.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.542 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:11.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:11.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:11.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:11.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:11.776 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:11.776 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-30T23:57:11.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:11.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:11 smithi173 ceph-mon[20554]: pgmap v2070: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 757 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 1594/36303 objects degraded (4.391%) 2024-08-30T23:57:11.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1594/36303 objects degraded (4.391%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:11.878 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:11.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:11.937 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:11.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:11.939 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:12.005 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:11 smithi106 ceph-mon[16603]: pgmap v2070: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 757 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 1594/36303 objects degraded (4.391%) 2024-08-30T23:57:12.005 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1594/36303 objects degraded (4.391%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:12.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:11 smithi136 ceph-mon[20207]: pgmap v2070: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 757 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 1594/36303 objects degraded (4.391%) 2024-08-30T23:57:12.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1594/36303 objects degraded (4.391%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:12.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:12.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:12.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:12.191 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:12.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:12.323 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:12.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:57:12.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:12.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27034.0:199570 6.1 6:86daa506:::benchmark_data_smithi173_114000_object99772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 506.43190457100002, 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65846682899999998, 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.547 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27034", 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 199570 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112290+0000", 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112292+0000", 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.198e-06 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112301+0000", 2024-08-30T23:57:12.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0219999999999993e-06 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112321+0000", 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9842000000000001e-05 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112331+0000", 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8949999999999993e-06 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112400+0000", 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8448999999999995e-05 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112494+0000", 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4380000000000001e-05 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.112570+0000", 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5704999999999994e-05 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.117141+0000", 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045717809999999996 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.117454+0000", 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031256200000000001 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770689+0000", 2024-08-30T23:57:12.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.653235393 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770728+0000", 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8445999999999997e-05 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770757+0000", 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9156e-05 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.553 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27034.0:199570 6.1 e720/715 ondisk, result = 0)", 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 506.42578472299999, 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65235226300000004, 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:57:12.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118410+0000", 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118411+0000", 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2900000000000004e-07 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.118414+0000", 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8670000000000004e-06 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770637+0000", 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.65222257100000003 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770642+0000", 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3920000000000002e-06 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770680+0000", 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7886999999999998e-05 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770686+0000", 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5670000000000002e-06 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:48:46.770762+0000", 2024-08-30T23:57:12.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6450000000000002e-05 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 765/761 [PushReplyOp(7:f2a3ccc8:::benchmark_data_smithi173_129296_object3685:head)])", 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 402.05713877800002, 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63433747900000004, 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487057+0000", 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0219999999999999e-06 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487063+0000", 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0469999999999999e-06 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115718+0000", 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62865490300000004 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115728+0000", 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9080000000000007e-06 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115812+0000", 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3713000000000003e-05 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.121393+0000", 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055818860000000003 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.561 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 769/761 [PushReplyOp(7:f0c70446:::benchmark_data_smithi173_129296_object15083:head)])", 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 389.12324388500002, 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97997442000000001, 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420952+0000", 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.062e-06 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420958+0000", 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395034+0000", 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97407640699999998 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395059+0000", 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4338999999999999e-05 2024-08-30T23:57:12.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395138+0000", 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9306000000000003e-05 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.400925+0000", 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057873259999999998 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:106574 7.7 7:e13bf739:::benchmark_data_smithi173_129296_object19458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 350.07138263899998, 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29613937600000001, 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 106574 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.566 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472813+0000", 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3600000000000003e-07 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472817+0000", 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2119999999999997e-06 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472827+0000", 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.91e-06 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472833+0000", 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5310000000000001e-06 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476588+0000", 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037553450000000002 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476692+0000", 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103875 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476803+0000", 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011145999999999999 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.480260+0000", 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034566480000000001 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.486997+0000", 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067370900000000003 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768836+0000", 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28183939600000002 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768874+0000", 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7537000000000003e-05 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768952+0000", 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7636e-05 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.571 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 792/761 [PushReplyOp(7:ff5d9ae4:::benchmark_data_smithi173_129296_object65310:head)])", 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 318.19360055300001, 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79968256000000004, 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350595+0000", 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9299999999999998e-07 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.573 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350601+0000", 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0379999999999996e-06 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148049+0000", 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79744800299999996 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148074+0000", 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5244000000000001e-05 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.574 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148181+0000", 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106742 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.150277+0000", 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020959400000000001 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.575 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:27874 8.c 8:385c40a0:::benchmark_data_smithi173_143816_object27873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:16.646546+0000", 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 175.89764888600001, 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31994099999999998, 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.577 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 27874 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646546+0000", 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646546+0000", 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646547+0000", 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5899999999999995e-07 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646588+0000", 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1168000000000003e-05 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646600+0000", 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1986e-05 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646606+0000", 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4760000000000003e-06 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646647+0000", 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0408999999999999e-05 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646725+0000", 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8450999999999998e-05 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646807+0000", 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1458999999999994e-05 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647769+0000", 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000962457 2024-08-30T23:57:12.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.803588+0000", 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15581879400000001 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966370+0000", 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16278235299999999 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966419+0000", 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8578999999999997e-05 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966487+0000", 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8009000000000006e-05 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.583 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:27874 8.c e857/840 ondisk, result = 0)", 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:16.647975+0000", 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 175.896219884, 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31851893199999998, 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647975+0000", 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647975+0000", 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647976+0000", 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647979+0000", 2024-08-30T23:57:12.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4860000000000002e-06 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966319+0000", 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31833969099999998 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966325+0000", 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2650000000000002e-06 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966362+0000", 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7240000000000003e-05 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966367+0000", 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5929999999999999e-06 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966494+0000", 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012689299999999999 2024-08-30T23:57:12.587 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:146297 8.b 8:d66af36d:::benchmark_data_smithi173_143816_object47983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e913)", 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:20.568778+0000", 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 51.975417200000003, 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.212557265, 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.588 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 146297 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568778+0000", 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568778+0000", 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568779+0000", 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.578e-06 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568785+0000", 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0599999999999996e-06 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.590 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568802+0000", 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7278e-05 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568811+0000", 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6510000000000007e-06 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568882+0000", 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0706999999999999e-05 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.569137+0000", 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025551399999999998 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.569436+0000", 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000298249 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.573731+0000", 2024-08-30T23:57:12.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042953380000000001 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.573818+0000", 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6705000000000004e-05 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781137+0000", 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.207319486 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781210+0000", 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2924999999999997e-05 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781335+0000", 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012477399999999999 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:146297 8.b e913/840 ondisk, result = 0)", 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 51.969783178, 2024-08-30T23:57:12.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206941876, 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.92e-07 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574419+0000", 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2330000000000003e-06 2024-08-30T23:57:12.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.779467+0000", 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20504845799999999 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.779473+0000", 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4670000000000001e-06 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781118+0000", 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016446900000000001 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781128+0000", 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5089999999999999e-06 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781353+0000", 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000225827 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:151473 8.b 8:df64a229:::benchmark_data_smithi173_143816_object53159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e915)", 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:26.890437+0000", 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 45.653757327000001, 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.14080226500000001, 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.599 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 151473 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.890437+0000", 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.890437+0000", 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.890439+0000", 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1200000000000001e-06 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.890444+0000", 2024-08-30T23:57:12.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4539999999999998e-06 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.890458+0000", 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.397e-05 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.890605+0000", 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014651599999999999 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.896108+0000", 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055033790000000001 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.896239+0000", 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013067299999999999 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.896345+0000", 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106198 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.901314+0000", 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049689369999999997 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:26.901630+0000", 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00031595899999999999 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:27.031117+0000", 2024-08-30T23:57:12.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.129487193 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:27.031199+0000", 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1741000000000002e-05 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:27.031240+0000", 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1124999999999997e-05 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.605 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:154039 8.b 8:da704aff:::benchmark_data_smithi173_143816_object55725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:29.822143+0000", 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 42.722052177999998, 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112447664, 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.606 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 154039 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822143+0000", 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822143+0000", 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822144+0000", 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7129999999999999e-06 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822152+0000", 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1919999999999999e-06 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822166+0000", 2024-08-30T23:57:12.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4643e-05 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822174+0000", 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1699999999999997e-06 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822494+0000", 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000319685 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822635+0000", 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000140712 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.822752+0000", 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117568 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.827057+0000", 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043047980000000003 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.828450+0000", 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013926170000000001 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.934370+0000", 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10591993800000001 2024-08-30T23:57:12.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.934468+0000", 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8689000000000003e-05 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:29.934590+0000", 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000121939 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.612 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:164857 8.b 8:dace5ab8:::benchmark_data_smithi173_143816_object66543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:42.836600+0000", 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.707595033, 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11674317200000001, 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.613 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 164857 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836600+0000", 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836600+0000", 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836601+0000", 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5799999999999999e-06 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836608+0000", 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0790000000000003e-06 2024-08-30T23:57:12.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836626+0000", 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7186000000000001e-05 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836634+0000", 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9160000000000001e-06 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836704+0000", 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0675999999999999e-05 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836833+0000", 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012845099999999999 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.836960+0000", 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012736600000000001 2024-08-30T23:57:12.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.841400+0000", 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044402230000000001 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.843225+0000", 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001824591 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953232+0000", 2024-08-30T23:57:12.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11000718399999999 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953286+0000", 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3743999999999999e-05 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953343+0000", 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7176000000000003e-05 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.619 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:164857 8.b e921/840 ondisk, result = 0)", 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:42.840696+0000", 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 29.703499072, 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.112667765, 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.840696+0000", 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.840696+0000", 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.840697+0000", 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0100000000000003e-07 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.840701+0000", 2024-08-30T23:57:12.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6319999999999999e-06 2024-08-30T23:57:12.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953158+0000", 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.11245682 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953165+0000", 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9450000000000004e-06 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953216+0000", 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1319000000000001e-05 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953226+0000", 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7999999999999993e-06 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:42.953363+0000", 2024-08-30T23:57:12.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000137348 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:166169 8.a 8:58ac550a:::benchmark_data_smithi173_143816_object67855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:44.460702+0000", 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 28.083492952, 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.113439234, 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.625 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 166169 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.460702+0000", 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.626 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.460702+0000", 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.460703+0000", 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.544e-06 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.460710+0000", 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2400000000000004e-06 2024-08-30T23:57:12.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.460725+0000", 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5143999999999999e-05 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.460733+0000", 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9589999999999995e-06 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.462544+0000", 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018110190000000001 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.462673+0000", 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000129667 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.462804+0000", 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013033700000000001 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.466817+0000", 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040133570000000004 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.467079+0000", 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00026222700000000002 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.574042+0000", 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.10696240999999999 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.574098+0000", 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5865999999999997e-05 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:44.574141+0000", 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3464000000000003e-05 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.631 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:174992 8.b 8:d69338c6:::benchmark_data_smithi173_143816_object76678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:55.280554+0000", 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.263640280000001, 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.113628848, 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 174992 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.632 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280554+0000", 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280554+0000", 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280556+0000", 2024-08-30T23:57:12.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8390000000000001e-06 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280563+0000", 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7050000000000002e-06 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280589+0000", 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5959999999999999e-05 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280598+0000", 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8580000000000006e-06 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280665+0000", 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7423999999999997e-05 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280806+0000", 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014029900000000001 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.635 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.280938+0000", 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013232700000000001 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.285918+0000", 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0049797959999999999 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.298322+0000", 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.01240403 2024-08-30T23:57:12.636 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.394098+0000", 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.095776719999999996 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.394133+0000", 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4652e-05 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.637 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.394183+0000", 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0238000000000002e-05 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:175001 8.b 8:d669ac38:::benchmark_data_smithi173_143816_object76687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:55.293143+0000", 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.251051834999998, 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15613911599999999, 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.638 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 175001 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293143+0000", 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.639 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293143+0000", 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293144+0000", 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9399999999999993e-07 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.640 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293149+0000", 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2959999999999998e-06 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293164+0000", 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4922999999999999e-05 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293172+0000", 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9650000000000008e-06 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.641 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293328+0000", 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015552599999999999 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293438+0000", 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011014 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.642 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.293540+0000", 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101795 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.298077+0000", 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0045370089999999998 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.313167+0000", 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0150902 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.643 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.449187+0000", 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13602011899999999 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.449240+0000", 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3284e-05 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.449282+0000", 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1865e-05 2024-08-30T23:57:12.644 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:175094 8.b 8:d68d4d7e:::benchmark_data_smithi173_143816_object76780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:55.443378+0000", 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.100816630000001, 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19880268600000001, 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.645 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 175094 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443378+0000", 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.646 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443378+0000", 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443379+0000", 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.540000000000001e-07 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443384+0000", 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6779999999999999e-06 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.647 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443396+0000", 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2354999999999999e-05 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443404+0000", 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.746e-06 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443455+0000", 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1088000000000003e-05 2024-08-30T23:57:12.648 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443535+0000", 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9568999999999997e-05 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.443606+0000", 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1385999999999997e-05 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.447297+0000", 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036910290000000002 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.649 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.641332+0000", 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.194035084 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642080+0000", 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00074841800000000004 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642136+0000", 2024-08-30T23:57:12.650 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5646000000000002e-05 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642181+0000", 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4733e-05 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:175094 8.b e925/840 ondisk, result = 0)", 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:55.447788+0000", 2024-08-30T23:57:12.651 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 17.096407209999999, 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19440006200000001, 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.447788+0000", 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.652 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.447788+0000", 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.447788+0000", 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3399999999999999e-07 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.447792+0000", 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.405e-06 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.653 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642022+0000", 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.19423017000000001 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642027+0000", 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6899999999999997e-06 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642068+0000", 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0309999999999999e-05 2024-08-30T23:57:12.654 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642074+0000", 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6850000000000004e-06 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.642188+0000", 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113268 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.655 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:178947 8.a 8:523510ea:::benchmark_data_smithi173_143816_object80633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:00.400595+0000", 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 12.143600212999999, 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23827178600000001, 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 178947 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.656 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400595+0000", 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400595+0000", 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:12.657 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400596+0000", 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4900000000000004e-07 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400601+0000", 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0030000000000002e-06 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400614+0000", 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3423e-05 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:12.658 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400622+0000", 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9489999999999996e-06 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400683+0000", 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1047e-05 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400798+0000", 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011467399999999999 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.659 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400910+0000", 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111944 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.405687+0000", 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047770850000000004 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.406106+0000", 2024-08-30T23:57:12.660 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041911999999999999 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.638694+0000", 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23258803 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.638822+0000", 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000128157 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:12.661 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.638866+0000", 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4404999999999999e-05 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:12.662 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.663 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:12.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:12.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:12.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:12.664 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:12.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:12.664 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:12.665 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:12.665 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1109899988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:12.665 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:12.665 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:12.697 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:12.698 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:12.720 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:12.730 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:12.730 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:12.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:12.769 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:12.769 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:12.770 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:12.799 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:12.800 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:12.806 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:12.824 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:12.824 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:12.856 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:12.856 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:12.884 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:12.884 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:12.908 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:12.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:12.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:13.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1109899988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:13.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:13.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:13.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1109899988' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:13.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:13.203 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.204 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:13.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:13.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:13.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:13.459 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:13.550 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:57:13.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:13.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:13.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:13.694 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:13.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:13 smithi173 ceph-mon[20554]: pgmap v2071: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 631 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 1088/30267 objects degraded (3.595%) 2024-08-30T23:57:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:13 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:13.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:13 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:13.846 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:13 smithi106 ceph-mon[16603]: pgmap v2071: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 631 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 1088/30267 objects degraded (3.595%) 2024-08-30T23:57:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:13 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:13.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:13 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:13.934 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:13.947 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:13.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:13.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:14.019 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:13 smithi136 ceph-mon[20207]: pgmap v2071: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 631 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 953 op/s; 1088/30267 objects degraded (3.595%) 2024-08-30T23:57:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:13 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:14.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:13 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:14.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:14.096 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:14.097 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.107 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:14.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:14.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:14.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:14.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:14.360 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:14.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:14.498 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:14.522 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:14.522 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:14.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:14.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:14.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:14.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:14.850 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:14.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:14.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:14.938 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:14 smithi106 ceph-mon[16603]: osdmap e932: 12 total, 11 up, 12 in 2024-08-30T23:57:14.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:14 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:57:14.940 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:14 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:57:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:14.969 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:14.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-30T23:57:14.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:15.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:14 smithi136 ceph-mon[20207]: osdmap e932: 12 total, 11 up, 12 in 2024-08-30T23:57:15.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:14 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:57:15.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:14 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:57:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:14 smithi173 ceph-mon[20554]: osdmap e932: 12 total, 11 up, 12 in 2024-08-30T23:57:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:14 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:57:15.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:14 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:57:15.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:15.168 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.169 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:15.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:15.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:57:15.300 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:15.312 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:15.403 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.436 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:15.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:15.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:15.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:15.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:57:15.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:15.686 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:57:15.693 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:15.702 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:15 smithi106 ceph-mon[16603]: pgmap v2073: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 563 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 1088/27024 objects degraded (4.026%) 2024-08-30T23:57:15.702 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1034678142' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:15.702 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:15 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:15.703 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:15.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:15.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:15.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:15.899 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:15.929 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:15.954 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:15.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:15 smithi136 ceph-mon[20207]: pgmap v2073: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 563 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 1088/27024 objects degraded (4.026%) 2024-08-30T23:57:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1034678142' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:16.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:15 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:16.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:16.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:15 smithi173 ceph-mon[20554]: pgmap v2073: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 563 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 671 op/s; 1088/27024 objects degraded (4.026%) 2024-08-30T23:57:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1034678142' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:16.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:15 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:16.077 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:16.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.202 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:16.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:16.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:16.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:16.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:16.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:16.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:16.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:16.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:16.713 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:16.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:16.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:16 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:16 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:57:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:16 smithi106 ceph-mon[16603]: osdmap e933: 12 total, 11 up, 12 in 2024-08-30T23:57:16.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1088/27024 objects degraded (4.026%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:16.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:16.961 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:16.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:16 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:16 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:57:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:16 smithi136 ceph-mon[20207]: osdmap e933: 12 total, 11 up, 12 in 2024-08-30T23:57:17.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1088/27024 objects degraded (4.026%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:17.063 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:16 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:16 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:57:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:16 smithi173 ceph-mon[20554]: osdmap e933: 12 total, 11 up, 12 in 2024-08-30T23:57:17.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1088/27024 objects degraded (4.026%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:17.088 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:17.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:17.212 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:17.239 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:17.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-08-30T23:57:17.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:17.338 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:17.355 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:17.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:17.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:17.583 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:17.604 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:17.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:17.739 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:17.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:17.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:17.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:17.931 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:17.937 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:17.937 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:17.971 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:17.971 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:17.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:18.002 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:17 smithi106 ceph-mon[16603]: pgmap v2075: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 471 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 1088/22599 objects degraded (4.814%) 2024-08-30T23:57:18.003 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:18.003 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:18.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:18.030 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:18.030 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:18.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:17 smithi136 ceph-mon[20207]: pgmap v2075: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 471 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 1088/22599 objects degraded (4.814%) 2024-08-30T23:57:18.042 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:17 smithi173 ceph-mon[20554]: pgmap v2075: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 471 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 783 op/s; 1088/22599 objects degraded (4.814%) 2024-08-30T23:57:18.063 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:18.063 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:18.090 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:18.090 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:18.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:18.122 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:18.122 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:18.122 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:18.150 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:18.150 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:18.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:18.186 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:18.186 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:18.209 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:18.209 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:57:18.209 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:57:18.209 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:57:18.209 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:18.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:18.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:18.245 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:18.246 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:18.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:18.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:18.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:18.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:18.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:18.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:57:18.518 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:18.519 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:18.526 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:18.628 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:18.653 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.654 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:18.755 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.777 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:18.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:18.829 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2413542837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:18.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:18.897 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2413542837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:18.903 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:18.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:18.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:57:19.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:19.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2413542837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:19.056 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:19.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:19.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:19.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:19.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:19.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:19.381 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:19.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:19.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:19.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:19.590 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:19.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.591 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:19.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:19.718 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:19.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:19.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:19 smithi173 ceph-mon[20554]: pgmap v2076: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 324 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 602/15546 objects degraded (3.872%) 2024-08-30T23:57:19.846 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:19.927 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:19 smithi106 ceph-mon[16603]: pgmap v2076: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 324 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 602/15546 objects degraded (3.872%) 2024-08-30T23:57:19.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:19.966 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:19.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-30T23:57:19.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:19.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:19.982 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86701 7.5 7:aeb8975f:::benchmark_data_smithi173_129296_object86700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776359+0000", 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24443434199998, 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.525555945000001, 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.025 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86701 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776359+0000", 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776359+0000", 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776361+0000", 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1409999999999999e-06 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776384+0000", 2024-08-30T23:57:20.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.349e-05 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776400+0000", 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5396e-05 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776408+0000", 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3420000000000008e-06 2024-08-30T23:57:20.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.293759+0000", 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.517351353 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.301915+0000", 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0081552229999999996 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86740 7.5 7:ae1ce8c3:::benchmark_data_smithi173_129296_object86739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776429+0000", 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24436408999998, 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.599940781000001, 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86740 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.030 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776429+0000", 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776429+0000", 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776429+0000", 2024-08-30T23:57:20.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.89e-07 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776444+0000", 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4876e-05 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776450+0000", 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0750000000000003e-06 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776454+0000", 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6030000000000001e-06 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.302640+0000", 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.526186638 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.312142+0000", 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0095015849999999999 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.325632+0000", 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013489663000000001 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374800+0000", 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.049168346000000002 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374828+0000", 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7585000000000001e-05 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374853+0000", 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4899e-05 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374893+0000", 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0710999999999997e-05 2024-08-30T23:57:20.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376370+0000", 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001476411 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86786 7.5 7:aacb3b1b:::benchmark_data_smithi173_129296_object86785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776459+0000", 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24433395599999, 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.601068818, 2024-08-30T23:57:20.036 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86786 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776459+0000", 2024-08-30T23:57:20.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776459+0000", 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776459+0000", 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.53e-07 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776474+0000", 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4507e-05 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776479+0000", 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7899999999999999e-06 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776483+0000", 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.002e-06 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.334845+0000", 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.558362949999999 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.342691+0000", 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078453540000000006 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.354770+0000", 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012079608 2024-08-30T23:57:20.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376431+0000", 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021660723 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376447+0000", 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6036999999999999e-05 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376492+0000", 2024-08-30T23:57:20.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4329999999999997e-05 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376526+0000", 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4403999999999998e-05 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377528+0000", 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00100186 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86883 7.5 7:aef44bcd:::benchmark_data_smithi173_129296_object86882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776487+0000", 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24430595500002, 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.625095855, 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86883 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.043 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776487+0000", 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776487+0000", 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776487+0000", 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2999999999999999e-07 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776502+0000", 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4394e-05 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776505+0000", 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8539999999999999e-06 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776508+0000", 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9349999999999999e-06 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377638+0000", 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.601129986 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377740+0000", 2024-08-30T23:57:20.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101199 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377847+0000", 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107093 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.384770+0000", 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069229469999999996 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386449+0000", 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00167946 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.400138+0000", 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013688735 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.400177+0000", 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.892e-05 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.401583+0000", 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001406102 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86907 7.5 7:a6e37ca7:::benchmark_data_smithi173_129296_object86906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776512+0000", 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24428069200002, 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.626529162000001, 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.049 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86907 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776512+0000", 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776512+0000", 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776512+0000", 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.36e-07 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776567+0000", 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4041e-05 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776572+0000", 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6389999999999997e-06 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776576+0000", 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6069999999999999e-06 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.380129+0000", 2024-08-30T23:57:20.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.6035532 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.380233+0000", 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010365300000000001 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.054 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:19 smithi136 ceph-mon[20207]: pgmap v2076: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 324 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 864 op/s; 602/15546 objects degraded (3.872%) 2024-08-30T23:57:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.380332+0000", 2024-08-30T23:57:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9827999999999994e-05 2024-08-30T23:57:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386463+0000", 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0061304899999999997 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386503+0000", 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9678000000000002e-05 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.401641+0000", 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015137904000000001 2024-08-30T23:57:20.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.401701+0000", 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0798000000000003e-05 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403041+0000", 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013400879999999999 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86923 7.5 7:a7631e73:::benchmark_data_smithi173_129296_object86922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776581+0000", 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24421236699999, 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.634007578, 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86923 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.057 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776581+0000", 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776581+0000", 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776581+0000", 2024-08-30T23:57:20.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1899999999999998e-07 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776613+0000", 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2311999999999998e-05 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776619+0000", 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3920000000000002e-06 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776622+0000", 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6909999999999999e-06 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.381838+0000", 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.605215756 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.381938+0000", 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9461000000000002e-05 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.382040+0000", 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102277 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386470+0000", 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044304080000000003 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397958+0000", 2024-08-30T23:57:20.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011487314 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.408624+0000", 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010666893 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.408675+0000", 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0816000000000003e-05 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.062 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.410588+0000", 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912939 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86963 7.5 7:a6c5bc79:::benchmark_data_smithi173_129296_object86962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.063 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776626+0000", 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24416662300001, 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.635543245000001, 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86963 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.064 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776626+0000", 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776626+0000", 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776627+0000", 2024-08-30T23:57:20.065 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2999999999999999e-07 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776666+0000", 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9297e-05 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776672+0000", 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6999999999999996e-06 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.066 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776675+0000", 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5219999999999999e-06 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.383270+0000", 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.606594899999999 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.383362+0000", 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1893999999999998e-05 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.067 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.383459+0000", 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7600000000000001e-05 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388100+0000", 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046405580000000004 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397998+0000", 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0098980519999999992 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.068 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.410658+0000", 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012660217 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.410723+0000", 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4547000000000004e-05 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.069 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.412170+0000", 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001446728 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86984 7.5 7:ae63c7dc:::benchmark_data_smithi173_129296_object86983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776680+0000", 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24411303699998, 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.642668217000001, 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.070 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86984 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776680+0000", 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.071 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776680+0000", 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776680+0000", 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.96e-07 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776714+0000", 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4e-05 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.072 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776720+0000", 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4410000000000001e-06 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776723+0000", 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.174e-06 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.384804+0000", 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.608080960000001 2024-08-30T23:57:20.073 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.384895+0000", 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1519000000000002e-05 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.074 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.385002+0000", 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010621299999999999 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390722+0000", 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057209729999999999 2024-08-30T23:57:20.076 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403094+0000", 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012371715 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.417626+0000", 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014531662000000001 2024-08-30T23:57:20.077 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.417666+0000", 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0287999999999998e-05 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.419348+0000", 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001681976 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.078 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86989 7.5 7:a5519cde:::benchmark_data_smithi173_129296_object86988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776727+0000", 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.244065707, 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.644000823000001, 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.079 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86989 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776727+0000", 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.080 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776727+0000", 2024-08-30T23:57:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.081 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776728+0000", 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5800000000000001e-07 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776768+0000", 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0256999999999998e-05 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776773+0000", 2024-08-30T23:57:20.082 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4260000000000002e-06 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776776+0000", 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2789999999999999e-06 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386532+0000", 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.609755765999999 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.083 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386621+0000", 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8885e-05 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386719+0000", 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8142999999999999e-05 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.084 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393565+0000", 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068459940000000002 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403147+0000", 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.009581895 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.419414+0000", 2024-08-30T23:57:20.085 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016267001 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.419455+0000", 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0911999999999997e-05 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.420728+0000", 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001273007 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.086 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86997 7.5 7:a5e72096:::benchmark_data_smithi173_129296_object86996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776782+0000", 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24401141999999, 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.652112274, 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.087 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86997 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776782+0000", 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.088 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776782+0000", 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776782+0000", 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4900000000000002e-07 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.089 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776813+0000", 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1024e-05 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776818+0000", 2024-08-30T23:57:20.090 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8199999999999996e-06 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776821+0000", 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1149999999999998e-06 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388130+0000", 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.611309732 2024-08-30T23:57:20.091 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388211+0000", 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0389000000000006e-05 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388303+0000", 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2146999999999997e-05 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.092 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393573+0000", 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052695320000000004 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403193+0000", 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.009620356 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.093 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.426952+0000", 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023759399 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.427029+0000", 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7062000000000002e-05 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.094 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.428894+0000", 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001864449 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87015 7.5 7:accfa1ec:::benchmark_data_smithi173_129296_object87014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776825+0000", 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24396823000001, 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.653428977000001, 2024-08-30T23:57:20.095 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87015 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.096 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776825+0000", 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776825+0000", 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776825+0000", 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3300000000000001e-07 2024-08-30T23:57:20.097 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776855+0000", 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0103e-05 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776860+0000", 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8520000000000003e-06 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.098 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776863+0000", 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1530000000000001e-06 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.389401+0000", 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.612538358 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.099 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.389492+0000", 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0104000000000003e-05 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.389584+0000", 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2057000000000003e-05 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396323+0000", 2024-08-30T23:57:20.100 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067397749999999999 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407266+0000", 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010943122 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.428946+0000", 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021679684000000001 2024-08-30T23:57:20.101 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.428995+0000", 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8427000000000002e-05 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.430254+0000", 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001259109 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.102 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87022 7.5 7:a6c9f5ab:::benchmark_data_smithi173_129296_object87021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776867+0000", 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24392611899998, 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.668171694, 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.103 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87022 2024-08-30T23:57:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.105 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776867+0000", 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776867+0000", 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.106 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776867+0000", 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2700000000000001e-07 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776899+0000", 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2437000000000001e-05 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.107 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776904+0000", 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.775e-06 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776907+0000", 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.005e-06 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.108 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390754+0000", 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.613846801999999 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390847+0000", 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2781999999999994e-05 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.109 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390943+0000", 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6514000000000005e-05 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396332+0000", 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053883819999999997 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.110 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407305+0000", 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010973478 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.442821+0000", 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035515821000000003 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.442890+0000", 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8573999999999998e-05 2024-08-30T23:57:20.111 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.445039+0000", 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021488969999999999 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87068 7.5 7:a60f23eb:::benchmark_data_smithi173_129296_object87067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776911+0000", 2024-08-30T23:57:20.112 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.243881821, 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.669716318999999, 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87068 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.113 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776911+0000", 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776911+0000", 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.114 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776911+0000", 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776940+0000", 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8428999999999998e-05 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776945+0000", 2024-08-30T23:57:20.115 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8119999999999998e-06 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776948+0000", 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6859999999999999e-06 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.392300+0000", 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.615352217 2024-08-30T23:57:20.116 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.392384+0000", 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3766999999999999e-05 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.392482+0000", 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7633000000000005e-05 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.117 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397918+0000", 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054359220000000001 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407361+0000", 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0094433550000000005 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.118 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.445097+0000", 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037735746000000001 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.445166+0000", 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9369000000000006e-05 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.446627+0000", 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014611649999999999 2024-08-30T23:57:20.119 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87085 7.5 7:a025fb98:::benchmark_data_smithi173_129296_object87084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776952+0000", 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24384082300003, 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.671139907000001, 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.120 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87085 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776952+0000", 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.121 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776952+0000", 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776952+0000", 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.122 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777026+0000", 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3590999999999994e-05 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777032+0000", 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2899999999999999e-06 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.123 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777036+0000", 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.121e-06 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393604+0000", 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.616567779 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393691+0000", 2024-08-30T23:57:20.124 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7140999999999998e-05 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393785+0000", 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3696000000000006e-05 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397926+0000", 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041405649999999997 2024-08-30T23:57:20.125 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407400+0000", 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0094743429999999997 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.446703+0000", 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039303005000000002 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.126 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.446766+0000", 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3028999999999996e-05 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.448092+0000", 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013261289999999999 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.127 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87102 7.5 7:a44aa11d:::benchmark_data_smithi173_129296_object87101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.777041+0000", 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.243751723, 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.672397555, 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.128 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87102 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777041+0000", 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.129 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777041+0000", 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777041+0000", 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6199999999999999e-07 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777074+0000", 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2558999999999998e-05 2024-08-30T23:57:20.130 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777079+0000", 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0390000000000004e-06 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777083+0000", 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0280000000000002e-06 2024-08-30T23:57:20.131 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.394924+0000", 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.617841357 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.395027+0000", 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010244800000000001 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.132 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.395123+0000", 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6083999999999997e-05 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.398010+0000", 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028866870000000002 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.133 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.412227+0000", 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014217483 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.448151+0000", 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035924229000000002 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.448194+0000", 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2203999999999998e-05 2024-08-30T23:57:20.134 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.449439+0000", 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012451750000000001 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87113 7.5 7:a933b630:::benchmark_data_smithi173_129296_object87112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.777087+0000", 2024-08-30T23:57:20.135 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 401.24370572399999, 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.673579998999999, 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87113 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.136 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777087+0000", 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777087+0000", 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.137 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777087+0000", 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5499999999999999e-07 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777119+0000", 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1924000000000001e-05 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.138 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777124+0000", 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8590000000000004e-06 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777128+0000", 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2590000000000001e-06 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.139 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396367+0000", 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.619239438999999 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396459+0000", 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2467999999999996e-05 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.140 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396557+0000", 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8011999999999996e-05 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.399549+0000", 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029911429999999999 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.412277+0000", 2024-08-30T23:57:20.141 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012728868000000001 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.449514+0000", 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037236242000000003 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.449574+0000", 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0139000000000001e-05 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.142 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.450667+0000", 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001093391 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.5 e833,761 7:ad5d727b:::benchmark_data_smithi173_129296_object8348:head,775'5941)", 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:53:39.158185+0000", 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 220.862608443, 2024-08-30T23:57:20.143 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55608631399999997, 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158185+0000", 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.144 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158185+0000", 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158185+0000", 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158189+0000", 2024-08-30T23:57:20.145 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.89e-06 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.712823+0000", 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55463346099999999 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.712834+0000", 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1821000000000001e-05 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.146 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.712901+0000", 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6729000000000004e-05 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.714271+0000", 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001369756 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.147 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.5 e835,761 7:a30b13d9:::benchmark_data_smithi173_129296_object15052:head,778'6360)", 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:53:41.823307+0000", 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 218.197485589, 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52982797100000001, 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.148 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.823307+0000", 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.823307+0000", 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.823308+0000", 2024-08-30T23:57:20.149 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9699999999999996e-07 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.823314+0000", 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.648e-06 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.352028+0000", 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.52871427999999998 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.150 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.352041+0000", 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3105000000000001e-05 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.352109+0000", 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8125000000000003e-05 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.151 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.353135+0000", 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010262159999999999 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.5 e835,761 7:a36f90bf:::benchmark_data_smithi173_129296_object15109:head,778'6361)", 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:53:41.829526+0000", 2024-08-30T23:57:20.152 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 218.191266931, 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53125486300000002, 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.829526+0000", 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.153 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.829526+0000", 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.829527+0000", 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0900000000000001e-07 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:41.829532+0000", 2024-08-30T23:57:20.154 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.711e-06 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.360104+0000", 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.53057199700000002 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.360113+0000", 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3459999999999994e-06 2024-08-30T23:57:20.155 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.360158+0000", 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5299000000000001e-05 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.360781+0000", 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062290100000000003 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.156 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.5 e835,761 7:a218f32c:::benchmark_data_smithi173_129296_object15307:head,778'6377)", 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:53:42.441178+0000", 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 217.57961490299999, 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.320975283, 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.157 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.441178+0000", 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.441178+0000", 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.441185+0000", 2024-08-30T23:57:20.158 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5429999999999999e-06 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.441195+0000", 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0186999999999999e-05 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.761530+0000", 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32033568600000001 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.159 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.761553+0000", 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2889000000000001e-05 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.761603+0000", 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9580000000000003e-05 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:42.762153+0000", 2024-08-30T23:57:20.160 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055039800000000001 2024-08-30T23:57:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:20.161 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:20.161 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:20.188 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:20.291 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:20.322 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:20.424 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.460 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:20.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:20.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:20.570 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:20.584 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:20.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:20.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:20.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:20.849 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:20.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:20.975 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:20.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:21.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:21.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:21.184 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:57:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:57:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:57:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:57:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:57:21.189 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:57:21.190 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:57:21.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:21.208 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:57:21.230 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:21.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:21.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:21.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:21.478 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:21.494 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:21.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:21.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:21.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:21.727 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:21.743 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:57:21.793 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:21 smithi106 ceph-mon[16603]: pgmap v2077: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 264 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 602/12672 objects degraded (4.751%) 2024-08-30T23:57:21.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1077022528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:21.794 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 602/12672 objects degraded (4.751%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:57:21.820 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:57:21.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:21.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:21 smithi173 ceph-mon[20554]: pgmap v2077: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 264 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 602/12672 objects degraded (4.751%) 2024-08-30T23:57:21.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1077022528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:21.830 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 602/12672 objects degraded (4.751%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:21.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:21.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:21.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:21.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:21 smithi136 ceph-mon[20207]: pgmap v2077: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 264 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 732 op/s; 602/12672 objects degraded (4.751%) 2024-08-30T23:57:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1077022528' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:22.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 602/12672 objects degraded (4.751%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:22.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:22.126 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:22.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:22.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:22.356 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.378 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:22.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:22.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:22.512 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:22.534 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:22.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:22.636 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:22.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:22.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:22.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:22.890 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:22.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:22.961 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:196146 8.4 e934/840 8:2aa7018e:::benchmark_data_smithi173_143816_object97832:head v 934'12250, mlcod=934'12246)", 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:22.342427+0000", 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.011527147, 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011556009000000001, 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:22.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.342427+0000", 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.342427+0000", 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.342428+0000", 2024-08-30T23:57:22.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7600000000000006e-07 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.342435+0000", 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.029e-06 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.343810+0000", 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013751340000000001 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:22.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.343817+0000", 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8419999999999999e-06 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.348607+0000", 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.004789967 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.348625+0000", 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8117999999999998e-05 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.965 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:196151 8.9 8:91bba08d:::benchmark_data_smithi173_143816_object97837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e934)", 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:22.345683+0000", 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0082709340000000006, 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.008335917, 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:22.966 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 196151 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345683+0000", 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345683+0000", 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345685+0000", 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3629999999999999e-06 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345689+0000", 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5750000000000002e-06 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345705+0000", 2024-08-30T23:57:22.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5345e-05 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345713+0000", 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0250000000000002e-06 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.346729+0000", 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0010163049999999999 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:22.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.347054+0000", 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000324822 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.347454+0000", 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00039974299999999998 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.353472+0000", 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0060184949999999996 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.353592+0000", 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000119707 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:196147 8.4 e934/840 8:2cc49474:::benchmark_data_smithi173_143816_object97833:head v 934'12251, mlcod=934'12246)", 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:22.345472+0000", 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0084826459999999999, 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0085878069999999994, 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:22.971 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345472+0000", 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345472+0000", 2024-08-30T23:57:22.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345472+0000", 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8699999999999995e-07 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.345478+0000", 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0329999999999997e-06 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.351238+0000", 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.005759363 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.351246+0000", 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6729999999999998e-06 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.353373+0000", 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002126317 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:22.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:22.353392+0000", 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9468999999999999e-05 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 3 2024-08-30T23:57:22.975 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:22.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:23.003 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:22 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:23.003 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1077022528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:23.003 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:22 smithi106 ceph-mon[16603]: osdmap e934: 12 total, 11 up, 12 in 2024-08-30T23:57:23.013 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:23.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:22 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:23.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1077022528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:23.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:22 smithi136 ceph-mon[20207]: osdmap e934: 12 total, 11 up, 12 in 2024-08-30T23:57:23.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:22 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:23.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1077022528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:23.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:22 smithi173 ceph-mon[20554]: osdmap e934: 12 total, 11 up, 12 in 2024-08-30T23:57:23.111 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:23.111 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:23.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:23.130 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:23.130 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:23.137 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:23.152 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:23.152 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:23.170 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:23.171 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:23.192 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:23.192 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:23.212 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:23.212 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:23.233 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:23.233 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:23.233 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:23.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:23.254 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:23.254 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:23.259 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:23.275 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:23.275 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:23.296 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:23.296 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:23.317 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:23.317 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:23.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:23.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:23.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:23.507 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:23.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:23.632 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:23.655 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 98311 objects 2024-08-30T23:57:23.655 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :117.223 2024-08-30T23:57:23.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.758 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:23.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:23.859 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:23.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:23.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:23.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:23.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:24.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:23 smithi136 ceph-mon[20207]: pgmap v2079: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 148 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 134/7107 objects degraded (1.885%) 2024-08-30T23:57:24.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1678727630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:24.073 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:23 smithi173 ceph-mon[20554]: pgmap v2079: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 148 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 134/7107 objects degraded (1.885%) 2024-08-30T23:57:24.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1678727630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:24.105 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:23 smithi106 ceph-mon[16603]: pgmap v2079: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 148 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 829 op/s; 134/7107 objects degraded (1.885%) 2024-08-30T23:57:24.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1678727630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:24.126 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:24.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.251 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:24.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.375 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:24.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:24.482 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:57:24.498 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:24.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.622 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:24.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:24.726 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:24.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:24.848 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:24.869 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:24.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:57:24.970 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:24.991 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:24.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:25.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.118 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:25.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:25.148 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:25.218 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:25.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:25.342 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.364 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:25.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:25.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.487 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:25.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:25.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:25.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:25.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:25.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:25.759 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:57:25.798 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:25.798 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:25.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:25.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:25.902 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:57:25.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:25.982 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:25.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:25 smithi136 ceph-mon[20207]: pgmap v2080: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 91 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 134/4332 objects degraded (3.093%) 2024-08-30T23:57:26.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3274387788' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:25 smithi173 ceph-mon[20554]: pgmap v2080: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 91 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 134/4332 objects degraded (3.093%) 2024-08-30T23:57:26.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3274387788' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:26.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:25 smithi106 ceph-mon[16603]: pgmap v2080: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 91 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s; 134/4332 objects degraded (3.093%) 2024-08-30T23:57:26.081 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3274387788' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:26.085 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:26.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:26.210 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.296 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:26.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:26.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:26.420 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:26.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:57:26.523 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:26.545 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:26.647 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:26.668 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:26.713 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:26.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:26.790 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:26.821 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:57:26.821 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:57:26.821 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-08-30T23:57:26.821 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-30T23:57:26.892 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:26.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:26.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:27.015 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:27.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:27.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3274387788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:27.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:26 smithi136 ceph-mon[20207]: osdmap e935: 12 total, 11 up, 12 in 2024-08-30T23:57:27.042 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 134/4332 objects degraded (3.093%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:27.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3274387788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:27.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:26 smithi173 ceph-mon[20554]: osdmap e935: 12 total, 11 up, 12 in 2024-08-30T23:57:27.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 134/4332 objects degraded (3.093%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3274387788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:26 smithi106 ceph-mon[16603]: osdmap e935: 12 total, 11 up, 12 in 2024-08-30T23:57:27.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 134/4332 objects degraded (3.093%), 3 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:27.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:27.165 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:27.188 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:27.189 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:27.266 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:27.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:27.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:27.411 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:27.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:27.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:27.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:27.638 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:27.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.661 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:27.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:57:27.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:27.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:27.790 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: pgmap v2082: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 36 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 134/1689 objects degraded (7.934%) 2024-08-30T23:57:27.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: 8.2 scrub starts 2024-08-30T23:57:27.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: 8.2 scrub ok 2024-08-30T23:57:27.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: 8.8 scrub starts 2024-08-30T23:57:27.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: 8.8 scrub ok 2024-08-30T23:57:27.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: 1.16 scrub starts 2024-08-30T23:57:27.791 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:27 smithi106 ceph-mon[16603]: 1.16 scrub ok 2024-08-30T23:57:27.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:27.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:27.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:27.981 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:28.014 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: pgmap v2082: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 36 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 134/1689 objects degraded (7.934%) 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: 8.2 scrub starts 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: 8.2 scrub ok 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: 8.8 scrub starts 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: 8.8 scrub ok 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: 1.16 scrub starts 2024-08-30T23:57:28.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:27 smithi136 ceph-mon[20207]: 1.16 scrub ok 2024-08-30T23:57:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.041 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:28.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:28.069 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: pgmap v2082: 47 pgs: 11 active+undersized, 3 active+undersized+degraded, 33 active+clean; 36 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 134/1689 objects degraded (7.934%) 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: 8.2 scrub starts 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: 8.2 scrub ok 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: 8.8 scrub starts 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: 8.8 scrub ok 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: 1.16 scrub starts 2024-08-30T23:57:28.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:27 smithi173 ceph-mon[20554]: 1.16 scrub ok 2024-08-30T23:57:28.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:28.164 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 935, 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-30T23:57:25.748639+0000", 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:55:58.153661+0000", 2024-08-30T23:57:28.171 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:54:10.077277+0000", 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5804032, 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "nodeep-scrub", 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 29, 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-30T23:57:28.172 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 8, 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "luminous", 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:57:28.173 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 31, 2024-08-30T23:57:28.174 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 31, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 31, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 31, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 31, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.175 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "879", 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "877", 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:57:28.176 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:57:28.177 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:57:28.178 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.6100000143051147, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.6100000143051147, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-30T23:57:28.179 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 1.7699999809265137, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 1.7699999809265137, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.68016940355300903, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.81999999284744263, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.68000000715255737 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 8, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_6", 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:53:47.812719+0000", 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-30T23:57:28.180 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 16, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 16, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 16, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 16, 2024-08-30T23:57:28.181 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 16, 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "844", 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "0", 2024-08-30T23:57:28.182 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-30T23:57:28.183 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-30T23:57:28.184 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-30T23:57:28.185 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 3.130000114440918, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 3.130000114440918, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 3.130000114440918, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 3.130000114440918, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.68016940355300903, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.81999999284744263, 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.68000000715255737 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.186 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 800, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.187 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.188 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.189 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.189 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.189 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.189 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.189 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.189 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.190 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.191 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-30T23:57:28.192 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-30T23:57:28.193 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.194 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.195 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.196 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.197 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.198 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 863, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.199 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.200 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.201 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.202 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-30T23:57:28.203 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:57:28.204 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 877, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.205 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.206 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.207 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.208 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.209 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.210 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.211 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.212 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.213 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.214 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.215 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 566, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 593, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 649, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-30T23:57:28.216 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-30T23:57:28.217 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-30T23:57:28.218 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-30T23:57:28.219 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2739515404", 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2739515404", 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2739515404", 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2739515404", 2024-08-30T23:57:28.220 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.221 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.222 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.223 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.224 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.225 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.226 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.1631317138671875, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 903, 2024-08-30T23:57:28.227 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.228 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.229 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.230 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-30T23:57:28.231 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-30T23:57:28.232 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.233 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-30T23:57:28.234 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-30T23:57:28.235 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.236 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.237 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.701446533203125, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:57:28.238 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.239 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-30T23:57:28.240 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-30T23:57:28.241 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-30T23:57:28.242 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.243 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.243 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-30T23:57:28.243 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-30T23:57:28.243 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-30T23:57:28.243 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:57:28.244 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.245 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.246 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.249 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-30T23:57:28.250 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 903, 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.252 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.253 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-30T23:57:28.254 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:57:28.255 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:19.117919+0000", 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.256 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:57:13.212920+0000", 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.257 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:37.654615+0000", 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:02.657335+0000", 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.258 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:28.304555+0000", 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:57:07.333430+0000", 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-30T23:57:28.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:27.519160+0000", 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.260 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:55:09.475333+0000", 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 902 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:33.620287+0000", 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:35.973521+0000", 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.262 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:36.968806+0000", 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:56:07.517807+0000", 2024-08-30T23:57:28.263 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [ 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.0", 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: 7, 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: 11 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1", 2024-08-30T23:57:28.264 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [], 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:57:28.265 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-30T23:57:28.266 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-30T23:57:28.267 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-30T23:57:28.268 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:28.268 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:57:28.268 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:28.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.290 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:28.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:28.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:28.496 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:28.497 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:28.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:28.535 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:28.637 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:28.663 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:28.765 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:28.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:28.834 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:28.835 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:28.835 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:28.835 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:28.835 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:28.835 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:28.844 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:28.844 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:28.867 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:28.867 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:28.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:28.888 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:28.888 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:28.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:28.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:28.910 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:28.911 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:28.911 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:28.928 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-08-30T23:57:28.928 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-08-30T23:57:28.931 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:28.931 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:28.953 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:28.953 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:28.973 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:28.974 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:28.995 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:28.995 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:29.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:29.029 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:28 smithi106 ceph-mon[16603]: 8.5 scrub starts 2024-08-30T23:57:29.030 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:28 smithi106 ceph-mon[16603]: 8.5 scrub ok 2024-08-30T23:57:29.030 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/364265301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:57:29.030 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/684304171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:29.033 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:29.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:28 smithi136 ceph-mon[20207]: 8.5 scrub starts 2024-08-30T23:57:29.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:28 smithi136 ceph-mon[20207]: 8.5 scrub ok 2024-08-30T23:57:29.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/364265301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:57:29.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/684304171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:29.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:28 smithi173 ceph-mon[20554]: 8.5 scrub starts 2024-08-30T23:57:29.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:28 smithi173 ceph-mon[20554]: 8.5 scrub ok 2024-08-30T23:57:29.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/364265301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-30T23:57:29.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/684304171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:29.135 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:29.156 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:29.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:57:29.258 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:29.284 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:29.333 INFO:tasks.thrashosds.thrasher:j is {'epoch': 935, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-30T23:57:25.748639+0000', 'last_up_change': '2024-08-30T23:55:58.153661+0000', 'last_in_change': '2024-08-30T23:54:10.077277+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '879', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '877', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6100000143051147, 'score_stable': 1.6100000143051147, 'optimal_score': 1.100000023841858, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.680169403553009, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.6800000071525574}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-08-30T23:53:47.812719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '844', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.130000114440918, 'score_stable': 3.130000114440918, 'optimal_score': 1, 'raw_score_acting': 3.130000114440918, 'raw_score_stable': 3.130000114440918, 'primary_affinity_weighted': 0.680169403553009, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.6800000071525574}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 800, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 879, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 863, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 877, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 566, 'last_clean_end': 593, 'up_from': 649, 'up_thru': 903, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2739515404}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2739515404}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2739515404}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2739515404}]}, 'public_addr': '172.21.15.136:6809/2739515404', 'cluster_addr': '172.21.15.136:6811/2739515404', 'heartbeat_back_addr': '172.21.15.136:6815/2739515404', 'heartbeat_front_addr': '172.21.15.136:6813/2739515404', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.1631317138671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 879, 'down_at': 903, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 879, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.701446533203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 879, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 903, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:19.117919+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:57:13.212920+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:37.654615+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:02.657335+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:28.304555+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:57:07.333430+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:27.519160+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:55:09.475333+0000', 'dead_epoch': 902}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:33.620287+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:35.973521+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:36.968806+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:56:07.517807+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 7, 11]}, {'pgid': '1.1', 'osds': [2, 0, 10]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-30T23:57:29.333 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:57:29.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:57:29.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.409 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:29.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:29.425 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:29.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:29.533 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:29.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.657 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:29.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:29.759 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:29.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:29 smithi173 ceph-mon[20554]: pgmap v2083: 47 pgs: 13 active+undersized, 1 active+undersized+degraded, 33 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 1/6 objects degraded (16.667%) 2024-08-30T23:57:29.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:29.904 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:29.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:29.930 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:29.930 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:57:29.931 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:57:29.931 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:57:29.931 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:29.960 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:29 smithi106 ceph-mon[16603]: pgmap v2083: 47 pgs: 13 active+undersized, 1 active+undersized+degraded, 33 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 1/6 objects degraded (16.667%) 2024-08-30T23:57:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:29.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:29.994 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:30.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:30.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:30.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:29 smithi136 ceph-mon[20207]: pgmap v2083: 47 pgs: 13 active+undersized, 1 active+undersized+degraded, 33 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 527 op/s; 1/6 objects degraded (16.667%) 2024-08-30T23:57:30.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:30.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:30.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:30.277 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:30.379 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:30.400 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:30.468 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:30.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:30.503 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:30.523 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:30.610 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:57:30.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:30.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:30.686 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:57:30.686 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:57:30.748 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.770 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:30.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:30.848 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:57:30.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:30.891 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:30.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:30.993 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.016 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:31.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:31.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:31.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:31.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:31.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:31.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:31.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37699 8.d 8:b57c4cbd:::benchmark_data_smithi173_143816_object37698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:57:31.396 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26438371, 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93994113199999996, 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37699 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:57:31.397 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121523+0000", 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3480000000000001e-06 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.398 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121550+0000", 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7135e-05 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121561+0000", 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.081e-05 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121569+0000", 2024-08-30T23:57:31.399 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6990000000000003e-06 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3838999999999996e-05 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.400 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121654+0000", 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1416999999999999e-05 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061408+0000", 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93975412199999997 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.401 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061463+0000", 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4762000000000002e-05 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37724 8.d 8:bb898ecc:::benchmark_data_smithi173_143816_object37723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:57:31.402 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26431467500001, 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94419919900000004, 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37724 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.403 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.404 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121616+0000", 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4661999999999999e-05 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.405 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121625+0000", 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6009999999999995e-06 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121631+0000", 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500000000000001e-06 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121691+0000", 2024-08-30T23:57:31.406 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9901000000000001e-05 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121695+0000", 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6849999999999999e-06 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061884+0000", 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94018891100000002 2024-08-30T23:57:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061957+0000", 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2855999999999995e-05 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.062054+0000", 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7165999999999998e-05 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065642+0000", 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035878989999999999 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065656+0000", 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4899000000000001e-05 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065728+0000", 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1124999999999994e-05 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065764+0000", 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6154e-05 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.410 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065790+0000", 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6273000000000001e-05 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37735 8.d 8:bd387e6f:::benchmark_data_smithi173_143816_object37734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:57:31.411 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.264262092, 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94428094500000004, 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37735 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121644+0000", 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9500000000000002e-07 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121680+0000", 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6387000000000002e-05 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121688+0000", 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5839999999999997e-06 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121694+0000", 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3119999999999999e-06 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121745+0000", 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0837999999999997e-05 2024-08-30T23:57:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121749+0000", 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6150000000000001e-06 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063145+0000", 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94139648399999998 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063233+0000", 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199e-05 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063310+0000", 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6536e-05 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065662+0000", 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023518760000000001 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065685+0000", 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2708999999999999e-05 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065841+0000", 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015618500000000001 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065890+0000", 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.888e-05 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065924+0000", 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4745e-05 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37745 8.d 8:b0c83413:::benchmark_data_smithi173_143816_object37744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.264184247, 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94438678600000003, 2024-08-30T23:57:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37745 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:57:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:57:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121722+0000", 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6700000000000003e-07 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121746+0000", 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4036000000000002e-05 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121752+0000", 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2570000000000004e-06 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121756+0000", 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9620000000000004e-06 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121787+0000", 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0587999999999999e-05 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121790+0000", 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3519999999999998e-06 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063974+0000", 2024-08-30T23:57:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94218355200000004 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064046+0000", 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1983999999999994e-05 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064119+0000", 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3266e-05 2024-08-30T23:57:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065799+0000", 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001680186 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065979+0000", 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179552 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066033+0000", 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4085999999999997e-05 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066077+0000", 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4382000000000003e-05 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066108+0000", 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1016000000000003e-05 2024-08-30T23:57:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37783 8.d 8:bd0ef9a3:::benchmark_data_smithi173_143816_object37782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.264141185, 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94672526099999998, 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37783 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121765+0000", 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4799999999999999e-07 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121789+0000", 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4697000000000001e-05 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121795+0000", 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1429999999999997e-06 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121798+0000", 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.472e-06 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121834+0000", 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5528000000000003e-05 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121842+0000", 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3159999999999997e-06 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064918+0000", 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.943076154 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064992+0000", 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3893999999999994e-05 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065076+0000", 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3681000000000001e-05 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065991+0000", 2024-08-30T23:57:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091558099999999995 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068378+0000", 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023872390000000002 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068428+0000", 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9329999999999997e-05 2024-08-30T23:57:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068464+0000", 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5917999999999997e-05 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068490+0000", 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5959999999999999e-05 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37794 8.d 8:b33c5833:::benchmark_data_smithi173_143816_object37793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26409952700001, 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94686707599999997, 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37794 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5900000000000003e-07 2024-08-30T23:57:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121850+0000", 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3831000000000001e-05 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121856+0000", 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2670000000000003e-06 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121861+0000", 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8799999999999999e-06 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121892+0000", 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1179000000000001e-05 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121896+0000", 2024-08-30T23:57:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3780000000000001e-06 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066144+0000", 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94424862499999995 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066245+0000", 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010058800000000001 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066373+0000", 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012802199999999999 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068538+0000", 2024-08-30T23:57:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021646870000000002 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068591+0000", 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3541000000000001e-05 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068604+0000", 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2607e-05 2024-08-30T23:57:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068648+0000", 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3781e-05 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068673+0000", 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5330999999999999e-05 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37798 8.d 8:b32c1d3f:::benchmark_data_smithi173_143816_object37797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26403620799999, 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96212161299999999, 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37798 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121870+0000", 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0800000000000001e-07 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121910+0000", 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0589000000000001e-05 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121916+0000", 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.981e-06 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121920+0000", 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5719999999999999e-06 2024-08-30T23:57:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121956+0000", 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6510000000000001e-05 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121970+0000", 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3285000000000001e-05 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067392+0000", 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94542252100000002 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067488+0000", 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5859000000000005e-05 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067570+0000", 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068681+0000", 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011110720000000001 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074538+0000", 2024-08-30T23:57:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058571530000000004 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083860+0000", 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0093212599999999996 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083936+0000", 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6007999999999995e-05 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083991+0000", 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5384999999999999e-05 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37806 8.d 8:b567e58a:::benchmark_data_smithi173_143816_object37805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.263977012, 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96731055799999999, 2024-08-30T23:57:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37806 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:57:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121929+0000", 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.22e-07 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122024+0000", 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5063999999999996e-05 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122031+0000", 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2060000000000001e-06 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122035+0000", 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7960000000000002e-06 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122067+0000", 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2122999999999997e-05 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122072+0000", 2024-08-30T23:57:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5140000000000003e-06 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068710+0000", 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94663852100000001 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068855+0000", 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014504599999999999 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068990+0000", 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135177 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074497+0000", 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055070910000000004 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074576+0000", 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109000000000004e-05 2024-08-30T23:57:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089138+0000", 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014561274000000001 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089193+0000", 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4864000000000002e-05 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089239+0000", 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6451e-05 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.464 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.465 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37807 8.d 8:bd227052:::benchmark_data_smithi173_143816_object37806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.263862367, 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96736628899999999, 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37807 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122079+0000", 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5790000000000001e-05 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122085+0000", 2024-08-30T23:57:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9170000000000003e-06 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122088+0000", 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.084e-06 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122122+0000", 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3738999999999997e-05 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2359999999999998e-06 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.072882+0000", 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95075315999999999 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073020+0000", 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013803900000000001 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073142+0000", 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122043 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077477+0000", 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043347180000000004 2024-08-30T23:57:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077486+0000", 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0759999999999992e-06 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089319+0000", 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011832666 2024-08-30T23:57:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089368+0000", 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9446000000000001e-05 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089409+0000", 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1060999999999999e-05 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37812 8.d 8:bad38160:::benchmark_data_smithi173_143816_object37811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.263811734, 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.967454445, 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37812 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8599999999999999e-07 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122123+0000", 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8960000000000001e-05 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5520000000000004e-06 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122132+0000", 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5379999999999998e-06 2024-08-30T23:57:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122161+0000", 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9351000000000001e-05 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122166+0000", 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3440000000000003e-06 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074606+0000", 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95243996799999997 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074746+0000", 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014037899999999999 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074867+0000", 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120969 2024-08-30T23:57:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077539+0000", 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026718330000000002 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078510+0000", 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097061800000000002 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089459+0000", 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010949181000000001 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089509+0000", 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0547999999999998e-05 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089548+0000", 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8918000000000002e-05 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37816 8.d 8:bceb9fa7:::benchmark_data_smithi173_143816_object37815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.263767702, 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96754795299999996, 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37816 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6e-07 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122168+0000", 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9663e-05 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122172+0000", 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-06 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122176+0000", 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2400000000000001e-06 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122205+0000", 2024-08-30T23:57:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9184999999999999e-05 2024-08-30T23:57:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122212+0000", 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1849999999999998e-06 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076169+0000", 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95395610799999997 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076291+0000", 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122633 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076448+0000", 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015706699999999999 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079517+0000", 2024-08-30T23:57:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003068777 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079533+0000", 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5656000000000001e-05 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089594+0000", 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010061151000000001 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089648+0000", 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4110999999999997e-05 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089686+0000", 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7762000000000002e-05 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37819 8.d 8:b27be7f0:::benchmark_data_smithi173_143816_object37818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26372390500001, 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96865621300000004, 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37819 2024-08-30T23:57:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.72e-07 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122214+0000", 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1964000000000001e-05 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122218+0000", 2024-08-30T23:57:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1649999999999999e-06 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122222+0000", 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7179999999999998e-06 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122251+0000", 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8810999999999999e-05 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122254+0000", 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8410000000000002e-06 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077587+0000", 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95533246100000002 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077667+0000", 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0438000000000005e-05 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077751+0000", 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4217999999999999e-05 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079574+0000", 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018224580000000001 2024-08-30T23:57:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080426+0000", 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085253 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090743+0000", 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010316967 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090798+0000", 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4866e-05 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090838+0000", 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9504e-05 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37827 8.d 8:bc586ec9:::benchmark_data_smithi173_143816_object37826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26367710400001, 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96986781300000002, 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37827 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122229+0000", 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8799999999999998e-07 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122263+0000", 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3946999999999999e-05 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122268+0000", 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122271+0000", 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1879999999999998e-06 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122300+0000", 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8824e-05 2024-08-30T23:57:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122303+0000", 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4249999999999998e-06 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078551+0000", 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95624797399999995 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078639+0000", 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078716+0000", 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6747000000000002e-05 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081587+0000", 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028711209999999999 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081602+0000", 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4864e-05 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.091996+0000", 2024-08-30T23:57:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010393823 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092053+0000", 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7574000000000001e-05 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092096+0000", 2024-08-30T23:57:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2870000000000001e-05 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37830 8.d 8:b56bd01c:::benchmark_data_smithi173_143816_object37829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26362781500001, 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97002787499999998, 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37830 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8500000000000002e-07 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122306+0000", 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8231000000000001e-05 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122311+0000", 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122314+0000", 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-06 2024-08-30T23:57:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122348+0000", 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3624999999999997e-05 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122359+0000", 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0685e-05 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079614+0000", 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95725496799999998 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079729+0000", 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114969 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079807+0000", 2024-08-30T23:57:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8337000000000006e-05 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083762+0000", 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039546900000000003 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083812+0000", 2024-08-30T23:57:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0781000000000001e-05 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092177+0000", 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083641789999999994 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092265+0000", 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8883999999999998e-05 2024-08-30T23:57:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092306+0000", 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0092999999999997e-05 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37833 8.d 8:bd5be583:::benchmark_data_smithi173_143816_object37832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26358400699999, 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97017067000000001, 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37833 2024-08-30T23:57:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122322+0000", 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122349+0000", 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7472999999999999e-05 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122356+0000", 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7889999999999997e-06 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122360+0000", 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8330000000000004e-06 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122389+0000", 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.879e-05 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122393+0000", 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1980000000000003e-06 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080473+0000", 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95808044999999997 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080581+0000", 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010725700000000001 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080712+0000", 2024-08-30T23:57:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013142199999999999 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084054+0000", 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003342434 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084069+0000", 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4487e-05 2024-08-30T23:57:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092412+0000", 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083428129999999993 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092458+0000", 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6440999999999998e-05 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092492+0000", 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4019000000000001e-05 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37856 8.d 8:b1ef2347:::benchmark_data_smithi173_143816_object37855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 183.26353948100001, 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97027297800000001, 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37856 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122397+0000", 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994000000000002e-05 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122401+0000", 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1439999999999996e-06 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122406+0000", 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122438+0000", 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2008000000000003e-05 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122445+0000", 2024-08-30T23:57:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081647+0000", 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95920211399999999 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081737+0000", 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0278999999999994e-05 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081881+0000", 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144036 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084081+0000", 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021993759999999998 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084140+0000", 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9562999999999999e-05 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092544+0000", 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084042449999999994 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092598+0000", 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3677999999999997e-05 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092639+0000", 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1013999999999997e-05 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45699 8.7 8:eae4016d:::benchmark_data_smithi173_143816_object45698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 174.38479679100001, 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64772118199999995, 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45699 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:57:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9100000000000003e-07 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001164+0000", 2024-08-30T23:57:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4604000000000001e-05 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001173+0000", 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9269999999999994e-06 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001177+0000", 2024-08-30T23:57:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4950000000000002e-06 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001209+0000", 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1758999999999999e-05 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001213+0000", 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9600000000000002e-06 2024-08-30T23:57:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642148+0000", 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64093464099999997 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642302+0000", 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015428999999999999 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:57:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642473+0000", 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000170843 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646115+0000", 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036422680000000002 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648167+0000", 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002051911 2024-08-30T23:57:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648704+0000", 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053669899999999997 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648748+0000", 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4098999999999999e-05 2024-08-30T23:57:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648830+0000", 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1995000000000004e-05 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45725 8.7 8:e95f37b8:::benchmark_data_smithi173_143816_object45724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:57:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 174.384718625, 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64797183599999997, 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45725 2024-08-30T23:57:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001188+0000", 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9999999999999997e-07 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001240+0000", 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120999999999998e-05 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001248+0000", 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6430000000000001e-06 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001253+0000", 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4989999999999996e-06 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001286+0000", 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3328999999999999e-05 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001290+0000", 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0090000000000001e-06 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.643899+0000", 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64260842600000001 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644052+0000", 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015375600000000001 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644192+0000", 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013943899999999999 2024-08-30T23:57:31.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646441+0000", 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022493819999999999 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648867+0000", 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024262860000000002 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.550 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649035+0000", 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167052 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649081+0000", 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6409999999999998e-05 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649159+0000", 2024-08-30T23:57:31.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7783999999999999e-05 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:175078 8.6 8:64dc8411:::benchmark_data_smithi173_143816_object76764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e925)", 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:55.402412+0000", 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.983493600999999, 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.218996363, 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.552 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 175078 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.402412+0000", 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.402412+0000", 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.402413+0000", 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.540000000000001e-07 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.402417+0000", 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.527e-06 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.402429+0000", 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1949e-05 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.402436+0000", 2024-08-30T23:57:31.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6440000000000003e-06 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.405641+0000", 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0032047640000000001 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.405759+0000", 2024-08-30T23:57:31.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118751 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 10", 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.405898+0000", 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000138584 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.409918+0000", 2024-08-30T23:57:31.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0040199720000000001 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.621233+0000", 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21131501999999999 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.621314+0000", 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0560000000000001e-05 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:55.621408+0000", 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4637999999999998e-05 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:178942 8.d 8:b6aa5353:::benchmark_data_smithi173_143816_object80628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:00.397092+0000", 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 30.988813263000001, 2024-08-30T23:57:31.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22032190400000001, 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 178942 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397092+0000", 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397092+0000", 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:57:31.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397093+0000", 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0100000000000001e-06 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397098+0000", 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6659999999999999e-06 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:57:31.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397110+0000", 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2252999999999999e-05 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397117+0000", 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9410000000000001e-06 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397164+0000", 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6616000000000002e-05 2024-08-30T23:57:31.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397242+0000", 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8189000000000006e-05 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2", 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.397312+0000", 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0247999999999996e-05 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.403115+0000", 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058023609999999998 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.617324+0000", 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21420994600000001 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:57:31.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.617359+0000", 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4431999999999998e-05 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.617414+0000", 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5241999999999997e-05 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.566 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:57:31.567 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:57:31.567 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:57:31.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:31.569 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2084,"stamp":"2024-08-30T23:57:29.839404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":196632,"num_write_kb":6292051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6907,"num_bytes_recovered":452741299,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163576,"ondisk_log_size":163576,"up":127,"acting":127,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8014488,"kb_used_data":600356,"kb_used_omap":109,"kb_used_meta":7413778,"kb_avail":1116927336,"statfs":{"total":1151940427776,"available":1143733592064,"internally_reserved":0,"allocated":614764544,"data_stored":606562912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":7591709336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-168034328,"num_objects":-2565,"num_object_clones":0,"num_object_copies":-7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2565,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.094945"},"pg_stats":[{"pgid":"1.1e","version":"23'92","reported_seq":135,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134739+0000","last_change":"2024-08-30T23:55:59.172632+0000","last_active":"2024-08-30T23:57:26.134739+0000","last_peered":"2024-08-30T23:57:26.134739+0000","last_clean":"2024-08-30T23:55:57.160064+0000","last_became_active":"2024-08-30T23:55:59.172632+0000","last_became_peered":"2024-08-30T23:55:59.172632+0000","last_unstale":"2024-08-30T23:57:26.134739+0000","last_undegraded":"2024-08-30T23:57:26.134739+0000","last_fullsized":"2024-08-30T23:55:59.167996+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.344869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.344869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":138,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134387+0000","last_change":"2024-08-30T23:56:46.421822+0000","last_active":"2024-08-30T23:57:26.134387+0000","last_peered":"2024-08-30T23:57:26.134387+0000","last_clean":"2024-08-30T23:57:26.134387+0000","last_became_active":"2024-08-30T23:55:03.321559+0000","last_became_peered":"2024-08-30T23:55:03.321559+0000","last_unstale":"2024-08-30T23:57:26.134387+0000","last_undegraded":"2024-08-30T23:57:26.134387+0000","last_fullsized":"2024-08-30T23:57:26.134387+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:46.421676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:46.421676+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:16.405067+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":136,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-08-30T23:57:21.836211+0000","last_change":"2024-08-30T23:56:16.491110+0000","last_active":"2024-08-30T23:57:21.836211+0000","last_peered":"2024-08-30T23:57:21.836211+0000","last_clean":"2024-08-30T23:57:21.836211+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-30T23:57:21.836211+0000","last_undegraded":"2024-08-30T23:57:21.836211+0000","last_fullsized":"2024-08-30T23:57:21.836211+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.491037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.491037+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:41.126506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":136,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-08-30T23:57:21.836482+0000","last_change":"2024-08-30T23:56:17.466832+0000","last_active":"2024-08-30T23:57:21.836482+0000","last_peered":"2024-08-30T23:57:21.836482+0000","last_clean":"2024-08-30T23:57:21.836482+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-30T23:57:21.836482+0000","last_undegraded":"2024-08-30T23:57:21.836482+0000","last_fullsized":"2024-08-30T23:57:21.836482+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:17.466761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:17.466761+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":136,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:25.760896+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-30T23:57:25.760896+0000","last_peered":"2024-08-30T23:57:25.760896+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-30T23:57:25.760896+0000","last_undegraded":"2024-08-30T23:57:25.760896+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:27.175887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":138,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.760537+0000","last_change":"2024-08-30T23:56:45.191296+0000","last_active":"2024-08-30T23:57:25.760537+0000","last_peered":"2024-08-30T23:57:25.760537+0000","last_clean":"2024-08-30T23:57:25.760537+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-30T23:57:25.760537+0000","last_undegraded":"2024-08-30T23:57:25.760537+0000","last_fullsized":"2024-08-30T23:57:25.760537+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.191245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.191245+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:49.183507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":360,"reported_epoch":935,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:26.134546+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-30T23:57:26.134546+0000","last_peered":"2024-08-30T23:57:26.134546+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-30T23:57:26.134546+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":136,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-08-30T23:57:21.836760+0000","last_change":"2024-08-30T23:56:40.272914+0000","last_active":"2024-08-30T23:57:21.836760+0000","last_peered":"2024-08-30T23:57:21.836760+0000","last_clean":"2024-08-30T23:57:21.836760+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-30T23:57:21.836760+0000","last_undegraded":"2024-08-30T23:57:21.836760+0000","last_fullsized":"2024-08-30T23:57:21.836760+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:40.272848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:40.272848+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:09.954699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":138,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134904+0000","last_change":"2024-08-30T23:56:16.212856+0000","last_active":"2024-08-30T23:57:26.134904+0000","last_peered":"2024-08-30T23:57:26.134904+0000","last_clean":"2024-08-30T23:57:26.134904+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-30T23:57:26.134904+0000","last_undegraded":"2024-08-30T23:57:26.134904+0000","last_fullsized":"2024-08-30T23:57:26.134904+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.212785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.212785+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":74,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134402+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-30T23:57:26.134402+0000","last_peered":"2024-08-30T23:57:26.134402+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-30T23:57:26.134402+0000","last_undegraded":"2024-08-30T23:57:26.134402+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:57:31.084955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":194,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134332+0000","last_change":"2024-08-30T23:57:14.423376+0000","last_active":"2024-08-30T23:57:26.134332+0000","last_peered":"2024-08-30T23:57:26.134332+0000","last_clean":"2024-08-30T23:57:26.134332+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:57:26.134332+0000","last_undegraded":"2024-08-30T23:57:26.134332+0000","last_fullsized":"2024-08-30T23:57:26.134332+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:14.423316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:14.423316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.319991+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":335,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134280+0000","last_change":"2024-08-30T23:56:21.163157+0000","last_active":"2024-08-30T23:57:26.134280+0000","last_peered":"2024-08-30T23:57:26.134280+0000","last_clean":"2024-08-30T23:57:26.134280+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:57:26.134280+0000","last_undegraded":"2024-08-30T23:57:26.134280+0000","last_fullsized":"2024-08-30T23:57:26.134280+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:21.163095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:21.163095+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:30.401960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":953,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-08-30T23:57:21.836173+0000","last_change":"2024-08-30T23:56:39.308813+0000","last_active":"2024-08-30T23:57:21.836173+0000","last_peered":"2024-08-30T23:57:21.836173+0000","last_clean":"2024-08-30T23:57:21.836173+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:57:21.836173+0000","last_undegraded":"2024-08-30T23:57:21.836173+0000","last_fullsized":"2024-08-30T23:57:21.836173+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:39.308738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:39.308738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:58.601526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":331,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134670+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-30T23:57:26.134670+0000","last_peered":"2024-08-30T23:57:26.134670+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-30T23:57:26.134670+0000","last_undegraded":"2024-08-30T23:57:26.134670+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:28.157831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"934'12210","reported_seq":12499,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.760539+0000","last_change":"2024-08-30T23:56:39.745288+0000","last_active":"2024-08-30T23:57:25.760539+0000","last_peered":"2024-08-30T23:57:25.760539+0000","last_clean":"2024-08-30T23:57:25.760539+0000","last_became_active":"2024-08-30T23:53:49.706310+0000","last_became_peered":"2024-08-30T23:53:49.706310+0000","last_unstale":"2024-08-30T23:57:25.760539+0000","last_undegraded":"2024-08-30T23:57:25.760539+0000","last_fullsized":"2024-08-30T23:57:25.760539+0000","mapping_epoch":840,"log_start":"860'2200","ondisk_log_start":"860'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'10088","last_scrub_stamp":"2024-08-30T23:56:39.745119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:39.745119+0000","objects_scrubbed":2143,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:53.129080+0000","scrub_duration":586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12210,"num_write_kb":390720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":746,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134149+0000","last_change":"2024-08-30T23:55:59.173538+0000","last_active":"2024-08-30T23:57:26.134149+0000","last_peered":"2024-08-30T23:57:26.134149+0000","last_clean":"2024-08-30T23:55:57.750202+0000","last_became_active":"2024-08-30T23:55:59.173538+0000","last_became_peered":"2024-08-30T23:55:59.173538+0000","last_unstale":"2024-08-30T23:57:26.134149+0000","last_undegraded":"2024-08-30T23:57:26.134149+0000","last_fullsized":"2024-08-30T23:55:59.169546+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:53.668489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:53.668489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:26.876086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"934'12448","reported_seq":15478,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134649+0000","last_change":"2024-08-30T23:57:26.134649+0000","last_active":"2024-08-30T23:57:26.134649+0000","last_peered":"2024-08-30T23:57:26.134649+0000","last_clean":"2024-08-30T23:55:58.160493+0000","last_became_active":"2024-08-30T23:55:59.170857+0000","last_became_peered":"2024-08-30T23:55:59.170857+0000","last_unstale":"2024-08-30T23:57:26.134649+0000","last_undegraded":"2024-08-30T23:57:26.134649+0000","last_fullsized":"2024-08-30T23:55:59.169519+0000","mapping_epoch":903,"log_start":"863'2400","ondisk_log_start":"863'2400","created":840,"last_epoch_clean":865,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'6608","last_scrub_stamp":"2024-08-30T23:55:35.179078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:55:35.179078+0000","objects_scrubbed":5896,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:26.467121+0000","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12448,"num_write_kb":398273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":89522176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":["11","2"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":198,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134521+0000","last_change":"2024-08-30T23:55:59.171246+0000","last_active":"2024-08-30T23:57:26.134521+0000","last_peered":"2024-08-30T23:57:26.134521+0000","last_clean":"2024-08-30T23:55:57.160009+0000","last_became_active":"2024-08-30T23:55:59.171246+0000","last_became_peered":"2024-08-30T23:55:59.171246+0000","last_unstale":"2024-08-30T23:57:26.134521+0000","last_undegraded":"2024-08-30T23:57:26.134521+0000","last_fullsized":"2024-08-30T23:55:59.168006+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:36.270718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:36.270718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:27.455252+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"934'12210","reported_seq":12511,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134427+0000","last_change":"2024-08-30T23:56:47.162288+0000","last_active":"2024-08-30T23:57:26.134427+0000","last_peered":"2024-08-30T23:57:26.134427+0000","last_clean":"2024-08-30T23:57:26.134427+0000","last_became_active":"2024-08-30T23:54:34.127322+0000","last_became_peered":"2024-08-30T23:54:34.127322+0000","last_unstale":"2024-08-30T23:57:26.134427+0000","last_undegraded":"2024-08-30T23:57:26.134427+0000","last_fullsized":"2024-08-30T23:57:26.134427+0000","mapping_epoch":866,"log_start":"860'2200","ondisk_log_start":"860'2200","created":840,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'10446","last_scrub_stamp":"2024-08-30T23:56:47.162157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:47.162157+0000","objects_scrubbed":1779,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:00.049951+0000","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12210,"num_write_kb":390720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":540,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.760646+0000","last_change":"2024-08-30T23:56:48.173739+0000","last_active":"2024-08-30T23:57:25.760646+0000","last_peered":"2024-08-30T23:57:25.760646+0000","last_clean":"2024-08-30T23:57:25.760646+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-30T23:57:25.760646+0000","last_undegraded":"2024-08-30T23:57:25.760646+0000","last_fullsized":"2024-08-30T23:57:25.760646+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:48.173668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:48.173668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:13.591624+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"934'12144","reported_seq":15183,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.135079+0000","last_change":"2024-08-30T23:56:49.069675+0000","last_active":"2024-08-30T23:57:26.135079+0000","last_peered":"2024-08-30T23:57:26.135079+0000","last_clean":"2024-08-30T23:57:26.135079+0000","last_became_active":"2024-08-30T23:54:35.995990+0000","last_became_peered":"2024-08-30T23:54:35.995990+0000","last_unstale":"2024-08-30T23:57:26.135079+0000","last_undegraded":"2024-08-30T23:57:26.135079+0000","last_fullsized":"2024-08-30T23:57:26.135079+0000","mapping_epoch":868,"log_start":"859'2100","ondisk_log_start":"859'2100","created":840,"last_epoch_clean":869,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'10456","last_scrub_stamp":"2024-08-30T23:56:49.069367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:49.069367+0000","objects_scrubbed":1698,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:14.369745+0000","scrub_duration":674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12144,"num_write_kb":388608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1369,"num_bytes_recovered":89718784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":514,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.760864+0000","last_change":"2024-08-30T23:56:16.234435+0000","last_active":"2024-08-30T23:57:25.760864+0000","last_peered":"2024-08-30T23:57:25.760864+0000","last_clean":"2024-08-30T23:57:25.760864+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-30T23:57:25.760864+0000","last_undegraded":"2024-08-30T23:57:25.760864+0000","last_fullsized":"2024-08-30T23:57:25.760864+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.234371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.234371+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:36.272119+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"934'12104","reported_seq":12372,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.761041+0000","last_change":"2024-08-30T23:55:31.202809+0000","last_active":"2024-08-30T23:57:25.761041+0000","last_peered":"2024-08-30T23:57:25.761041+0000","last_clean":"2024-08-30T23:57:25.761041+0000","last_became_active":"2024-08-30T23:53:49.705978+0000","last_became_peered":"2024-08-30T23:53:49.705978+0000","last_unstale":"2024-08-30T23:57:25.761041+0000","last_undegraded":"2024-08-30T23:57:25.761041+0000","last_fullsized":"2024-08-30T23:57:25.761041+0000","mapping_epoch":840,"log_start":"859'2100","ondisk_log_start":"859'2100","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"892'6239","last_scrub_stamp":"2024-08-30T23:55:31.202699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.202699+0000","objects_scrubbed":5865,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:28.135335+0000","scrub_duration":1644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":387328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1153,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134397+0000","last_change":"2024-08-30T23:56:20.789107+0000","last_active":"2024-08-30T23:57:26.134397+0000","last_peered":"2024-08-30T23:57:26.134397+0000","last_clean":"2024-08-30T23:57:26.134397+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-30T23:57:26.134397+0000","last_undegraded":"2024-08-30T23:57:26.134397+0000","last_fullsized":"2024-08-30T23:57:26.134397+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:20.789023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:20.789023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:50.461299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"934'12242","reported_seq":12513,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:27.089861+0000","last_change":"2024-08-30T23:57:27.089861+0000","last_active":"2024-08-30T23:57:27.089861+0000","last_peered":"2024-08-30T23:57:27.089861+0000","last_clean":"2024-08-30T23:57:27.089861+0000","last_became_active":"2024-08-30T23:53:49.707708+0000","last_became_peered":"2024-08-30T23:53:49.707708+0000","last_unstale":"2024-08-30T23:57:27.089861+0000","last_undegraded":"2024-08-30T23:57:27.089861+0000","last_fullsized":"2024-08-30T23:57:27.089861+0000","mapping_epoch":840,"log_start":"863'2200","ondisk_log_start":"863'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"934'12242","last_scrub_stamp":"2024-08-30T23:57:27.089803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:57:27.089803+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:38.228179+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12242,"num_write_kb":391744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":499,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.760659+0000","last_change":"2024-08-30T23:57:08.049165+0000","last_active":"2024-08-30T23:57:25.760659+0000","last_peered":"2024-08-30T23:57:25.760659+0000","last_clean":"2024-08-30T23:57:25.760659+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:57:25.760659+0000","last_undegraded":"2024-08-30T23:57:25.760659+0000","last_fullsized":"2024-08-30T23:57:25.760659+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:08.049028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:08.049028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:28.561467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"934'12256","reported_seq":12544,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-08-30T23:57:22.856656+0000","last_change":"2024-08-30T23:56:49.721984+0000","last_active":"2024-08-30T23:57:22.856656+0000","last_peered":"2024-08-30T23:57:22.856656+0000","last_clean":"2024-08-30T23:57:22.856656+0000","last_became_active":"2024-08-30T23:53:49.671637+0000","last_became_peered":"2024-08-30T23:53:49.671637+0000","last_unstale":"2024-08-30T23:57:22.856656+0000","last_undegraded":"2024-08-30T23:57:22.856656+0000","last_fullsized":"2024-08-30T23:57:22.856656+0000","mapping_epoch":840,"log_start":"859'2200","ondisk_log_start":"859'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'10652","last_scrub_stamp":"2024-08-30T23:56:49.721795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:49.721795+0000","objects_scrubbed":1613,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:51.130344+0000","scrub_duration":441,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12256,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1986,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:25.760970+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-30T23:57:25.760970+0000","last_peered":"2024-08-30T23:57:25.760970+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-30T23:57:25.760970+0000","last_undegraded":"2024-08-30T23:57:25.760970+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"934'12218","reported_seq":12497,"reported_epoch":934,"state":"active+clean","last_fresh":"2024-08-30T23:57:22.917160+0000","last_change":"2024-08-30T23:57:07.395049+0000","last_active":"2024-08-30T23:57:22.917160+0000","last_peered":"2024-08-30T23:57:22.917160+0000","last_clean":"2024-08-30T23:57:22.917160+0000","last_became_active":"2024-08-30T23:53:49.708926+0000","last_became_peered":"2024-08-30T23:53:49.708926+0000","last_unstale":"2024-08-30T23:57:22.917160+0000","last_undegraded":"2024-08-30T23:57:22.917160+0000","last_fullsized":"2024-08-30T23:57:22.917160+0000","mapping_epoch":840,"log_start":"860'2200","ondisk_log_start":"860'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'11484","last_scrub_stamp":"2024-08-30T23:57:07.394904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:57:07.394904+0000","objects_scrubbed":737,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:12.024046+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12218,"num_write_kb":390976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":310,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134312+0000","last_change":"2024-08-30T23:55:55.752282+0000","last_active":"2024-08-30T23:57:26.134312+0000","last_peered":"2024-08-30T23:57:26.134312+0000","last_clean":"2024-08-30T23:57:26.134312+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:57:26.134312+0000","last_undegraded":"2024-08-30T23:57:26.134312+0000","last_fullsized":"2024-08-30T23:57:26.134312+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:55.752214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:55.752214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:27.908400+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"934'12340","reported_seq":12646,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.760616+0000","last_change":"2024-08-30T23:56:15.144364+0000","last_active":"2024-08-30T23:57:25.760616+0000","last_peered":"2024-08-30T23:57:25.760616+0000","last_clean":"2024-08-30T23:57:25.760616+0000","last_became_active":"2024-08-30T23:53:49.670369+0000","last_became_peered":"2024-08-30T23:53:49.670369+0000","last_unstale":"2024-08-30T23:57:25.760616+0000","last_undegraded":"2024-08-30T23:57:25.760616+0000","last_fullsized":"2024-08-30T23:57:25.760616+0000","mapping_epoch":840,"log_start":"860'2300","ondisk_log_start":"860'2300","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"910'8914","last_scrub_stamp":"2024-08-30T23:56:15.144304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:15.144304+0000","objects_scrubbed":3451,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12340,"num_write_kb":394880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":972,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.762532+0000","last_change":"2024-08-30T23:56:17.340413+0000","last_active":"2024-08-30T23:57:25.762532+0000","last_peered":"2024-08-30T23:57:25.762532+0000","last_clean":"2024-08-30T23:57:25.762532+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-30T23:57:25.762532+0000","last_undegraded":"2024-08-30T23:57:25.762532+0000","last_fullsized":"2024-08-30T23:57:25.762532+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:17.340337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:17.340337+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:31.809756+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"934'12284","reported_seq":12588,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134270+0000","last_change":"2024-08-30T23:56:17.717912+0000","last_active":"2024-08-30T23:57:26.134270+0000","last_peered":"2024-08-30T23:57:26.134270+0000","last_clean":"2024-08-30T23:57:26.134270+0000","last_became_active":"2024-08-30T23:53:49.706915+0000","last_became_peered":"2024-08-30T23:53:49.706915+0000","last_unstale":"2024-08-30T23:57:26.134270+0000","last_undegraded":"2024-08-30T23:57:26.134270+0000","last_fullsized":"2024-08-30T23:57:26.134270+0000","mapping_epoch":840,"log_start":"861'2200","ondisk_log_start":"861'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"911'8870","last_scrub_stamp":"2024-08-30T23:56:17.717693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:17.717693+0000","objects_scrubbed":3428,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12284,"num_write_kb":393088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":956,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:25.760970+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-30T23:57:25.760970+0000","last_peered":"2024-08-30T23:57:25.760970+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-30T23:57:25.760970+0000","last_undegraded":"2024-08-30T23:57:25.760970+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.7","version":"934'12426","reported_seq":15550,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134951+0000","last_change":"2024-08-30T23:56:50.136277+0000","last_active":"2024-08-30T23:57:26.134951+0000","last_peered":"2024-08-30T23:57:26.134951+0000","last_clean":"2024-08-30T23:57:26.134951+0000","last_became_active":"2024-08-30T23:54:37.621282+0000","last_became_peered":"2024-08-30T23:54:37.621282+0000","last_unstale":"2024-08-30T23:57:26.134951+0000","last_undegraded":"2024-08-30T23:57:26.134951+0000","last_fullsized":"2024-08-30T23:57:26.134951+0000","mapping_epoch":869,"log_start":"863'2400","ondisk_log_start":"863'2400","created":840,"last_epoch_clean":870,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'10874","last_scrub_stamp":"2024-08-30T23:56:50.136173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:50.136173+0000","objects_scrubbed":1570,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:05.353955+0000","scrub_duration":700,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":397632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1410,"num_bytes_recovered":92405760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":887,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:25.761125+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-30T23:57:25.761125+0000","last_peered":"2024-08-30T23:57:25.761125+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-30T23:57:25.761125+0000","last_undegraded":"2024-08-30T23:57:25.761125+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"934'12500","reported_seq":15614,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134884+0000","last_change":"2024-08-30T23:57:22.938085+0000","last_active":"2024-08-30T23:57:26.134884+0000","last_peered":"2024-08-30T23:57:26.134884+0000","last_clean":"2024-08-30T23:55:58.154828+0000","last_became_active":"2024-08-30T23:55:59.172863+0000","last_became_peered":"2024-08-30T23:55:59.172863+0000","last_unstale":"2024-08-30T23:57:26.134884+0000","last_undegraded":"2024-08-30T23:57:26.134884+0000","last_fullsized":"2024-08-30T23:55:59.170206+0000","mapping_epoch":903,"log_start":"865'2400","ondisk_log_start":"865'2400","created":840,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"885'5405","last_scrub_stamp":"2024-08-30T23:55:12.514923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:55:12.514923+0000","objects_scrubbed":5373,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12500,"num_write_kb":400000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":92798976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10],"acting":[11,10],"avail_no_missing":["11","10"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1157,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134439+0000","last_change":"2024-08-30T23:56:45.888377+0000","last_active":"2024-08-30T23:57:26.134439+0000","last_peered":"2024-08-30T23:57:26.134439+0000","last_clean":"2024-08-30T23:57:26.134439+0000","last_became_active":"2024-08-30T23:45:50.826113+0000","last_became_peered":"2024-08-30T23:45:50.826113+0000","last_unstale":"2024-08-30T23:57:26.134439+0000","last_undegraded":"2024-08-30T23:57:26.134439+0000","last_fullsized":"2024-08-30T23:57:26.134439+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.888321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.888321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:15.843140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1","version":"934'12236","reported_seq":12539,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134451+0000","last_change":"2024-08-30T23:56:16.511183+0000","last_active":"2024-08-30T23:57:26.134451+0000","last_peered":"2024-08-30T23:57:26.134451+0000","last_clean":"2024-08-30T23:57:26.134451+0000","last_became_active":"2024-08-30T23:53:49.706070+0000","last_became_peered":"2024-08-30T23:53:49.706070+0000","last_unstale":"2024-08-30T23:57:26.134451+0000","last_undegraded":"2024-08-30T23:57:26.134451+0000","last_fullsized":"2024-08-30T23:57:26.134451+0000","mapping_epoch":840,"log_start":"860'2200","ondisk_log_start":"860'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"911'8790","last_scrub_stamp":"2024-08-30T23:56:16.511106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.511106+0000","objects_scrubbed":3480,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:46.243328+0000","scrub_duration":1414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":391552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":353,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134655+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-30T23:57:26.134655+0000","last_peered":"2024-08-30T23:57:26.134655+0000","last_clean":"2024-08-30T23:57:26.134655+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-30T23:57:26.134655+0000","last_undegraded":"2024-08-30T23:57:26.134655+0000","last_fullsized":"2024-08-30T23:57:26.134655+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:53.707888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"934'12558","reported_seq":12849,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:25.762386+0000","last_change":"2024-08-30T23:56:45.841147+0000","last_active":"2024-08-30T23:57:25.762386+0000","last_peered":"2024-08-30T23:57:25.762386+0000","last_clean":"2024-08-30T23:57:25.762386+0000","last_became_active":"2024-08-30T23:53:49.706543+0000","last_became_peered":"2024-08-30T23:53:49.706543+0000","last_unstale":"2024-08-30T23:57:25.762386+0000","last_undegraded":"2024-08-30T23:57:25.762386+0000","last_fullsized":"2024-08-30T23:57:25.762386+0000","mapping_epoch":840,"log_start":"865'2500","ondisk_log_start":"865'2500","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"922'10679","last_scrub_stamp":"2024-08-30T23:56:45.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.840847+0000","objects_scrubbed":1893,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:55.612904+0000","scrub_duration":534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":401856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":950,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134699+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-30T23:57:26.134699+0000","last_peered":"2024-08-30T23:57:26.134699+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-30T23:57:26.134699+0000","last_undegraded":"2024-08-30T23:57:26.134699+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:57:27.657059+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"934'12152","reported_seq":15140,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134327+0000","last_change":"2024-08-30T23:57:05.628363+0000","last_active":"2024-08-30T23:57:26.134327+0000","last_peered":"2024-08-30T23:57:26.134327+0000","last_clean":"2024-08-30T23:57:26.134327+0000","last_became_active":"2024-08-30T23:54:25.996662+0000","last_became_peered":"2024-08-30T23:54:25.996662+0000","last_unstale":"2024-08-30T23:57:26.134327+0000","last_undegraded":"2024-08-30T23:57:26.134327+0000","last_fullsized":"2024-08-30T23:57:26.134327+0000","mapping_epoch":862,"log_start":"859'2100","ondisk_log_start":"859'2100","created":840,"last_epoch_clean":863,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'11330","last_scrub_stamp":"2024-08-30T23:57:05.628256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:57:05.628256+0000","objects_scrubbed":824,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:19.554930+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12152,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":88211456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1088,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134519+0000","last_change":"2024-08-30T23:56:19.712394+0000","last_active":"2024-08-30T23:57:26.134519+0000","last_peered":"2024-08-30T23:57:26.134519+0000","last_clean":"2024-08-30T23:57:26.134519+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-30T23:57:26.134519+0000","last_undegraded":"2024-08-30T23:57:26.134519+0000","last_fullsized":"2024-08-30T23:57:26.134519+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:19.712327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:19.712327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"934'12296","reported_seq":12572,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.830286+0000","last_change":"2024-08-30T23:57:26.830286+0000","last_active":"2024-08-30T23:57:26.830286+0000","last_peered":"2024-08-30T23:57:26.830286+0000","last_clean":"2024-08-30T23:57:26.830286+0000","last_became_active":"2024-08-30T23:53:49.706194+0000","last_became_peered":"2024-08-30T23:53:49.706194+0000","last_unstale":"2024-08-30T23:57:26.830286+0000","last_undegraded":"2024-08-30T23:57:26.830286+0000","last_fullsized":"2024-08-30T23:57:26.830286+0000","mapping_epoch":840,"log_start":"860'2200","ondisk_log_start":"860'2200","created":840,"last_epoch_clean":841,"parent":"0.0","parent_split_bits":0,"last_scrub":"934'12296","last_scrub_stamp":"2024-08-30T23:57:26.830238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:53:48.650807+0000","last_clean_scrub_stamp":"2024-08-30T23:57:26.830238+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:46.229260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12296,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":200,"reported_epoch":935,"state":"active+clean","last_fresh":"2024-08-30T23:57:26.134531+0000","last_change":"2024-08-30T23:56:13.242208+0000","last_active":"2024-08-30T23:57:26.134531+0000","last_peered":"2024-08-30T23:57:26.134531+0000","last_clean":"2024-08-30T23:57:26.134531+0000","last_became_active":"2024-08-30T23:55:01.289376+0000","last_became_peered":"2024-08-30T23:55:01.289376+0000","last_unstale":"2024-08-30T23:57:26.134531+0000","last_undegraded":"2024-08-30T23:57:26.134531+0000","last_fullsized":"2024-08-30T23:57:26.134531+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:13.242035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:13.242035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:35.066814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1043,"reported_epoch":935,"state":"active+undersized","last_fresh":"2024-08-30T23:57:26.134602+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-30T23:57:26.134602+0000","last_peered":"2024-08-30T23:57:26.134602+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-30T23:57:26.134602+0000","last_undegraded":"2024-08-30T23:57:26.134602+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:29.675972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196624,"num_write_kb":6291905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6907,"num_bytes_recovered":452657152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":675221504,"data_stored":675217432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":46,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":81,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187851,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646616,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098536,"statfs":{"total":95995035648,"available":95332900864,"internally_reserved":0,"allocated":1839104,"data_stored":1168763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":660273604},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449493,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":628580,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93116572,"statfs":{"total":95995035648,"available":95351369728,"internally_reserved":0,"allocated":1851392,"data_stored":1176569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":641792448},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711122,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718820,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93026332,"statfs":{"total":95995035648,"available":95258963968,"internally_reserved":0,"allocated":1851392,"data_stored":1176569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":734198208},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972764,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":506392,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93238760,"statfs":{"total":95995035648,"available":95476490240,"internally_reserved":0,"allocated":1839104,"data_stored":1168763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528742,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484516,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93260636,"statfs":{"total":95995035648,"available":95498891264,"internally_reserved":0,"allocated":2179072,"data_stored":1504249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775245,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259756,"kb_used_data":2008,"kb_used_omap":5,"kb_used_meta":257722,"kb_avail":93485396,"statfs":{"total":95995035648,"available":95729045504,"internally_reserved":0,"allocated":2056192,"data_stored":1377817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":263907402},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953905,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709092,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93036060,"statfs":{"total":95995035648,"available":95268925440,"internally_reserved":0,"allocated":1851392,"data_stored":1176569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150131,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831588,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92913564,"statfs":{"total":95995035648,"available":95143489536,"internally_reserved":0,"allocated":2179072,"data_stored":1504249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":849344967},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411769,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642668,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93102484,"statfs":{"total":95995035648,"available":95336943616,"internally_reserved":0,"allocated":2056192,"data_stored":1377817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830585,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":494808,"kb_used_data":1796,"kb_used_omap":87,"kb_used_meta":492904,"kb_avail":93250344,"statfs":{"total":95995035648,"available":95488352256,"internally_reserved":0,"allocated":1839104,"data_stored":1168763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":504733855},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738819,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":787172,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92957980,"statfs":{"total":95995035648,"available":95188971520,"internally_reserved":0,"allocated":2179072,"data_stored":1504249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":88211456,"data_stored":88211456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":587010048,"data_stored":587005976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:57:31.571 INFO:tasks.thrashosds.thrasher:Setting 1.f to [0, 6, 9] 2024-08-30T23:57:31.571 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.f', '0', '6', '9'] 2024-08-30T23:57:31.572 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 1.f 0 6 9 2024-08-30T23:57:31.629 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:31.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:31.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:31.780 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_6' removed 2024-08-30T23:57:31.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: pgmap v2084: 47 pgs: 13 active+undersized, 1 active+undersized+degraded, 33 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 1/6 objects degraded (16.667%) 2024-08-30T23:57:31.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: from='client.19113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:31.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:31.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:31.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3761086572' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:57:31.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:31.781 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:31 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:31.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:31.877 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:31.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:31.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:31.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:57:32.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: pgmap v2084: 47 pgs: 13 active+undersized, 1 active+undersized+degraded, 33 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 1/6 objects degraded (16.667%) 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: from='client.19113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3761086572' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:32.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:31 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: pgmap v2084: 47 pgs: 13 active+undersized, 1 active+undersized+degraded, 33 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 316 op/s; 1/6 objects degraded (16.667%) 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: from='client.19113 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3761086572' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:32.063 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:31 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:32.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:32.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:32.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.253 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:32.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:32.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:57:32.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:32.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:32.476 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:32.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:32.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.623 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:32.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:32.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:32.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:32.851 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:57:32.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:32.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:32.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:32.978 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.001 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:33.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3761086572' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:57:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:32 smithi136 ceph-mon[20207]: osdmap e936: 12 total, 11 up, 12 in 2024-08-30T23:57:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:32 smithi136 ceph-mon[20207]: pgmap v2086: 31 pgs: 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:32 smithi136 ceph-mon[20207]: 1.1b scrub starts 2024-08-30T23:57:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:32 smithi136 ceph-mon[20207]: 1.1b scrub ok 2024-08-30T23:57:33.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:32 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3693490694' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3761086572' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:57:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:32 smithi173 ceph-mon[20554]: osdmap e936: 12 total, 11 up, 12 in 2024-08-30T23:57:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:32 smithi173 ceph-mon[20554]: pgmap v2086: 31 pgs: 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:32 smithi173 ceph-mon[20554]: 1.1b scrub starts 2024-08-30T23:57:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:32 smithi173 ceph-mon[20554]: 1.1b scrub ok 2024-08-30T23:57:33.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:32 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3693490694' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:33.103 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3761086572' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-08-30T23:57:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:32 smithi106 ceph-mon[16603]: osdmap e936: 12 total, 11 up, 12 in 2024-08-30T23:57:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:32 smithi106 ceph-mon[16603]: pgmap v2086: 31 pgs: 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:32 smithi106 ceph-mon[16603]: 1.1b scrub starts 2024-08-30T23:57:33.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:32 smithi106 ceph-mon[16603]: 1.1b scrub ok 2024-08-30T23:57:33.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:32 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3693490694' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:33.124 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:33.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:33.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:33.350 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:33.372 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:33.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:33.497 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:33.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:33.607 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-30T23:57:33.608 INFO:tasks.radosbench:Beginning radosbench... 2024-08-30T23:57:33.608 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-08-30T23:57:33.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-08-30T23:57:33.625 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:33.727 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:33.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:33.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:33 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3693490694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:33 smithi106 ceph-mon[16603]: osdmap e937: 12 total, 11 up, 12 in 2024-08-30T23:57:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:33 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:57:33.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:33 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:57:33.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:33.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:33.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.000 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:34.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:34.039 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:33 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3693490694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:33 smithi136 ceph-mon[20207]: osdmap e937: 12 total, 11 up, 12 in 2024-08-30T23:57:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:33 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:57:34.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:33 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:57:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:33 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3693490694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:33 smithi173 ceph-mon[20554]: osdmap e937: 12 total, 11 up, 12 in 2024-08-30T23:57:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:33 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:57:34.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:33 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:57:34.086 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:34.093 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:34.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:34.129 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:34.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:34.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:34.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.254 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:34.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:34.356 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:34.379 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:34.481 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:34.505 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:34.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:34.633 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:34.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:34.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:34.784 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:34 smithi173 ceph-mon[20554]: pgmap v2088: 31 pgs: 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:34.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:34.880 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:34.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:34.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:34.998 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:34 smithi106 ceph-mon[16603]: pgmap v2088: 31 pgs: 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:35.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:35.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:34 smithi136 ceph-mon[20207]: pgmap v2088: 31 pgs: 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:35.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:35.130 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:35.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:35.253 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:35.355 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:35.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:35.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:35.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:35.503 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:35.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:35.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:35.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:35.740 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:57:35.740 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:57:35.740 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:57:35.740 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:57:35.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:35.799 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_7' created 2024-08-30T23:57:35.799 INFO:teuthology.orchestra.run.smithi106.stderr:set 1.f pg_upmap mapping to [0,6,9] 2024-08-30T23:57:35.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:35.876 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:35.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:35.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:36.005 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:36.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:57:36.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:35 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:57:36.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:35 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:57:36.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/469853237' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.f", "id": [0, 6, 9]} : dispatch 2024-08-30T23:57:36.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:35 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-30T23:57:36.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:35 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-30T23:57:36.061 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2276317846' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-08-30T23:57:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:35 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:57:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:35 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:57:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/469853237' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.f", "id": [0, 6, 9]} : dispatch 2024-08-30T23:57:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:35 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-30T23:57:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:35 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-30T23:57:36.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2276317846' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-08-30T23:57:36.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:36.128 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:36.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:36.251 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:35 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:57:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:35 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:57:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/469853237' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.f", "id": [0, 6, 9]} : dispatch 2024-08-30T23:57:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:35 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-30T23:57:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:35 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-30T23:57:36.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2276317846' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-08-30T23:57:36.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:36.376 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:36.476 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:36.477 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:36.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:36.498 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:36.498 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:36.499 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:36.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:36.518 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:36.538 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:36.538 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:36.561 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:36.561 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:36.582 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:36.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:36.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:36.603 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:36.604 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:36.604 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:36.621 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:36.626 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:36.626 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:36.648 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:36.648 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:36.670 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:36.670 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:36.691 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:36.691 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:36.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:36.749 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:36.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:36.880 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:36.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:36.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:36.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.010 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:37.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/469853237' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.f", "id": [0, 6, 9]}]': finished 2024-08-30T23:57:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2276317846' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-08-30T23:57:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: osdmap e938: 12 total, 11 up, 12 in 2024-08-30T23:57:37.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: pgmap v2090: 47 pgs: 16 unknown, 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:37.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:37.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: osdmap e939: 12 total, 11 up, 12 in 2024-08-30T23:57:37.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2454813694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/469853237' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.f", "id": [0, 6, 9]}]': finished 2024-08-30T23:57:37.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2276317846' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-08-30T23:57:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: osdmap e938: 12 total, 11 up, 12 in 2024-08-30T23:57:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: pgmap v2090: 47 pgs: 16 unknown, 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: osdmap e939: 12 total, 11 up, 12 in 2024-08-30T23:57:37.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2454813694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:37.110 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.137 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:37.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:37.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:37.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/469853237' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.f", "id": [0, 6, 9]}]': finished 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2276317846' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: osdmap e938: 12 total, 11 up, 12 in 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: pgmap v2090: 47 pgs: 16 unknown, 11 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: osdmap e939: 12 total, 11 up, 12 in 2024-08-30T23:57:37.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2454813694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:37.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:37.394 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:37.434 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:37.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:37.519 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:37.621 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:37.647 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:37.749 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:37.770 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:37.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:37.894 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:37.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:37.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:37.996 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:38.022 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:38.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:37 smithi106 ceph-mon[16603]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:57:38.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:37 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:38.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:37 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:38.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:37 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:38.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:37 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:38.046 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:37 smithi106 ceph-mon[16603]: osdmap e940: 12 total, 11 up, 12 in 2024-08-30T23:57:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:37 smithi173 ceph-mon[20554]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:57:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:37 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:37 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:38.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:37 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:38.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:37 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:38.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:37 smithi173 ceph-mon[20554]: osdmap e940: 12 total, 11 up, 12 in 2024-08-30T23:57:38.075 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-08-30T23:57:38.124 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:38.149 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:38.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:38.272 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:38.292 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:37 smithi136 ceph-mon[20207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-30T23:57:38.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:37 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:38.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:37 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:38.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:37 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:38.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:37 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:38.293 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:37 smithi136 ceph-mon[20207]: osdmap e940: 12 total, 11 up, 12 in 2024-08-30T23:57:38.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:38.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:38.488 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:57:38.489 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:38.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:38.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:38.630 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:38.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:38.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:38.780 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:38.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:38.905 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:38.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:38.962 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:38.992 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:39.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:39.033 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.034 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:38 smithi173 ceph-mon[20554]: pgmap v2093: 47 pgs: 10 unknown, 14 active+undersized, 1 active+undersized+degraded, 22 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:38 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:39.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:38 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:39.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:38 smithi106 ceph-mon[16603]: pgmap v2093: 47 pgs: 10 unknown, 14 active+undersized, 1 active+undersized+degraded, 22 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:39.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:38 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:39.084 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:38 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:39.135 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:39.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:39.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:39.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:39.289 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:38 smithi136 ceph-mon[20207]: pgmap v2093: 47 pgs: 10 unknown, 14 active+undersized, 1 active+undersized+degraded, 22 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:38 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:39.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:38 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:39.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:39.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:39.435 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:57:39.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:39.542 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:39.645 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:39.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:39.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:39.791 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:39.817 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:57:39.834 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:39.835 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:39.893 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:39.915 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:39.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:40.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:40.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:39 smithi106 ceph-mon[16603]: osdmap e941: 12 total, 11 up, 12 in 2024-08-30T23:57:40.023 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/483424881' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:40.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:39 smithi173 ceph-mon[20554]: osdmap e941: 12 total, 11 up, 12 in 2024-08-30T23:57:40.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/483424881' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:40.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:40.172 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:40.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:40.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:39 smithi136 ceph-mon[20207]: osdmap e941: 12 total, 11 up, 12 in 2024-08-30T23:57:40.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/483424881' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:57:40.296 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:40.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:40.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.425 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:40.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:40.552 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:40.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:40.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:40.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.806 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:40.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:40.818 INFO:teuthology.orchestra.run.smithi106.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-08-30T23:57:40.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:40.930 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:40.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:41.032 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:41.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/483424881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: osdmap e942: 12 total, 11 up, 12 in 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/412296409' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:57:41.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:40 smithi173 ceph-mon[20554]: pgmap v2096: 47 pgs: 1 unknown, 16 active+undersized, 1 active+undersized+degraded, 29 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/483424881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: osdmap e942: 12 total, 11 up, 12 in 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/412296409' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:57:41.091 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:40 smithi106 ceph-mon[16603]: pgmap v2096: 47 pgs: 1 unknown, 16 active+undersized, 1 active+undersized+degraded, 29 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:41.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:57:41.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.184 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:41.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:41.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/483424881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: osdmap e942: 12 total, 11 up, 12 in 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/412296409' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-30T23:57:41.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:40 smithi136 ceph-mon[20207]: pgmap v2096: 47 pgs: 1 unknown, 16 active+undersized, 1 active+undersized+degraded, 29 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:41.311 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:41.413 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:41.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:41.486 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:41.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:41.563 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:41.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:41.690 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:41.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:41.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:41.819 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:57:41.819 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:57:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/412296409' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:57:41.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:41 smithi106 ceph-mon[16603]: osdmap e943: 12 total, 11 up, 12 in 2024-08-30T23:57:41.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:41.944 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:41.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:42.046 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:42.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:42.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.074 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/412296409' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:57:42.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:41 smithi173 ceph-mon[20554]: osdmap e943: 12 total, 11 up, 12 in 2024-08-30T23:57:42.176 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:42.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/412296409' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-30T23:57:42.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:41 smithi136 ceph-mon[20207]: osdmap e943: 12 total, 11 up, 12 in 2024-08-30T23:57:42.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:42.326 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:42.414 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:42.414 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:42.414 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:42.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:42.439 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:42.439 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:42.451 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:42.460 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:42.460 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:42.480 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:42.480 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:42.502 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:42.502 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:42.524 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:42.524 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:42.545 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:42.546 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:42.546 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:42.553 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:42.567 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:42.567 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:42.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.580 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:42.589 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:42.589 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:42.611 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:42.611 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:42.632 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:42.632 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:42.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:57:42.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:42.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:42.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:42.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:42.914 DEBUG:teuthology.orchestra.run.smithi173:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-08-30T23:57:42.915 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-08-30T23:57:42.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:57:42.918 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-30T23:57:42.918 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-30T23:57:42.937 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:42.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:42.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:42.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:hints = 1 2024-08-30T23:57:42.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-30T23:57:42.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object prefix: benchmark_data_smithi173_160189 2024-08-30T23:57:42.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:57:42.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 0 0 0 0 0 0 - 0 2024-08-30T23:57:43.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:42 smithi106 ceph-mon[16603]: pgmap v2098: 47 pgs: 16 active+undersized, 1 active+undersized+degraded, 30 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:43.058 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3920521417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:43.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:43.092 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:43.196 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.219 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:43.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:42 smithi136 ceph-mon[20207]: pgmap v2098: 47 pgs: 16 active+undersized, 1 active+undersized+degraded, 30 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:43.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3920521417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:43.319 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:43.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:42 smithi173 ceph-mon[20554]: pgmap v2098: 47 pgs: 16 active+undersized, 1 active+undersized+degraded, 30 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:43.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3920521417' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.341 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:43.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:43.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:43.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:43.463 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.b 763/756 [PushReplyOp(7:d4034f8a:::benchmark_data_smithi173_129296_object8907:head)])", 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 435.16729100200001, 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61886441400000003, 2024-08-30T23:57:43.557 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.558 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9800000000000003e-07 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387988+0000", 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9800000000000003e-06 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.559 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001726+0000", 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61373751200000004 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001738+0000", 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1746000000000001e-05 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001796+0000", 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8470000000000001e-05 2024-08-30T23:57:43.560 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.006846+0000", 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050499079999999997 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e833,800 7:ec17d7ba:::benchmark_data_smithi173_129296_object15625:head,778'6427)", 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:36.708683+0000", 2024-08-30T23:57:43.561 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.846589158, 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27911429900000001, 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.708683+0000", 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.708683+0000", 2024-08-30T23:57:43.562 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.708684+0000", 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2699999999999998e-07 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.708690+0000", 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8619999999999999e-06 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.563 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.987387+0000", 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27869647199999997 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.987395+0000", 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0660000000000004e-06 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.987447+0000", 2024-08-30T23:57:43.564 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2463999999999998e-05 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.987798+0000", 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035050800000000003 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e833,800 7:eb051ac9:::benchmark_data_smithi173_129296_object15680:head,778'6428)", 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:36.713983+0000", 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 246.841289832, 2024-08-30T23:57:43.565 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.280422159, 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.713983+0000", 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.713983+0000", 2024-08-30T23:57:43.566 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.713984+0000", 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7599999999999996e-07 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.713989+0000", 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1669999999999998e-06 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.567 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.994024+0000", 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28003519999999998 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.994031+0000", 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0849999999999999e-06 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.994079+0000", 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8418999999999998e-05 2024-08-30T23:57:43.568 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:36.994405+0000", 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00032551200000000002 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e833,800 7:e3e0581a:::benchmark_data_smithi173_129296_object20938:head,781'6741)", 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 244.647965086, 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56565642400000005, 2024-08-30T23:57:43.569 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.570 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.92e-07 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907313+0000", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5199999999999999e-06 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472540+0000", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56522729500000002 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.571 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472548+0000", 2024-08-30T23:57:43.572 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9780000000000005e-06 2024-08-30T23:57:43.572 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.573 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472616+0000", 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7745999999999998e-05 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472964+0000", 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034819300000000001 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.574 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,800 7:e6a7a338:::benchmark_data_smithi173_129296_object26093:head,783'7057)", 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:41.361662+0000", 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 242.19361047300001, 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29360009100000001, 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361662+0000", 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.575 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361662+0000", 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361663+0000", 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4499999999999997e-07 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361666+0000", 2024-08-30T23:57:43.576 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.579e-06 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654885+0000", 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29321898600000001 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654896+0000", 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0654e-05 2024-08-30T23:57:43.577 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654958+0000", 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2126999999999997e-05 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.655262+0000", 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030420000000000002 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.578 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,800 7:e80dc891:::benchmark_data_smithi173_129296_object31110:head,789'7374)", 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 240.48017209400001, 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70067806200000005, 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.579 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9999999999999997e-07 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075106+0000", 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6249999999999998e-06 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.580 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775466+0000", 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70036012700000005 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775475+0000", 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6789999999999994e-06 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.581 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775526+0000", 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1054000000000003e-05 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775779+0000", 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025287699999999998 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:6057 8.b e847/840 8:dba6764f:::benchmark_data_smithi173_143816_object6056:head v 847'367, mlcod=847'362)", 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:59.027172+0000", 2024-08-30T23:57:43.582 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 224.528100808, 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25005487799999998, 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.027172+0000", 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.027172+0000", 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.583 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.027172+0000", 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9900000000000001e-07 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.027267+0000", 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4788000000000001e-05 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.584 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.275946+0000", 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24867924699999999 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.275952+0000", 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8930000000000002e-06 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.276017+0000", 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4729999999999999e-05 2024-08-30T23:57:43.585 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.276036+0000", 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9409e-05 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.277183+0000", 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011464909999999999 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.586 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.277227+0000", 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3921000000000002e-05 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:27836 8.3 8:c5a8f20c:::benchmark_data_smithi173_143816_object27835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:16.607303+0000", 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 206.947969989, 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.282388315, 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.587 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 27836 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607303+0000", 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607303+0000", 2024-08-30T23:57:43.588 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607304+0000", 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.173e-06 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607369+0000", 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5629000000000005e-05 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.589 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607381+0000", 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1141000000000001e-05 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607388+0000", 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8399999999999995e-06 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607434+0000", 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.579e-05 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.590 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607529+0000", 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4813000000000002e-05 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 5,11", 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.607635+0000", 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000105752 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609098+0000", 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014627500000000001 2024-08-30T23:57:43.591 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609122+0000", 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4459000000000001e-05 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.888925+0000", 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27980312299999999 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.592 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.888968+0000", 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3031000000000002e-05 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.889691+0000", 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00072281400000000001 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.593 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:27838 8.7 8:ee12ce7e:::benchmark_data_smithi173_143816_object27837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 206.94619906, 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28304243600000001, 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 27838 2024-08-30T23:57:43.594 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.595 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8999999999999995e-07 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609179+0000", 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104453 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.596 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609187+0000", 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1680000000000004e-06 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609193+0000", 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609238+0000", 2024-08-30T23:57:43.597 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5324999999999997e-05 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609365+0000", 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012742100000000001 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609466+0000", 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100251 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.598 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.611050+0000", 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015842580000000001 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.648761+0000", 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037710942999999997 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.891285+0000", 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24252380600000001 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.599 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.891319+0000", 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4653999999999997e-05 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.892116+0000", 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00079671699999999995 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.600 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36528 8.7 8:ed4e2712:::benchmark_data_smithi173_143816_object36527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.367630+0000", 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 198.18764281599999, 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46017543799999999, 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36528 2024-08-30T23:57:43.601 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367630+0000", 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367630+0000", 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.602 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367631+0000", 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.548e-06 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367685+0000", 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4001e-05 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367695+0000", 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6150000000000003e-06 2024-08-30T23:57:43.603 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367700+0000", 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5650000000000003e-06 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368384+0000", 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068396600000000004 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.604 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368520+0000", 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013615799999999999 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368643+0000", 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012345699999999999 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.370618+0000", 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001974632 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.605 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.373944+0000", 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033259520000000001 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.826917+0000", 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.452973338 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.826991+0000", 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3951999999999999e-05 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.606 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.827805+0000", 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00081425400000000004 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36543 8.7 8:ea468d17:::benchmark_data_smithi173_143816_object36542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.549506+0000", 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 198.00576634999999, 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28538659799999999, 2024-08-30T23:57:43.607 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36543 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549506+0000", 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.608 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549506+0000", 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549508+0000", 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9429999999999999e-06 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549547+0000", 2024-08-30T23:57:43.609 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8903999999999999e-05 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549559+0000", 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2114e-05 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549567+0000", 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.610 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549967+0000", 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040007299999999998 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.550137+0000", 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016991200000000001 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.550336+0000", 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00019918599999999999 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.611 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.554084+0000", 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037478749999999999 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.558679+0000", 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045950710000000001 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.612 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.834074+0000", 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27539524700000001 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.834122+0000", 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7425000000000001e-05 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.834893+0000", 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00077132999999999998 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.613 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37734 8.1 8:87d95214:::benchmark_data_smithi173_143816_object37733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.39961868899999, 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33538572700000002, 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37734 2024-08-30T23:57:43.614 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:57:43.615 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2300000000000001e-07 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155716+0000", 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1576999999999997e-05 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155723+0000", 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8770000000000004e-06 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.616 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.617 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155733+0000", 2024-08-30T23:57:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9280000000000005e-06 2024-08-30T23:57:43.618 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.156800+0000", 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010672489999999999 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.156972+0000", 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000171516 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.619 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.157064+0000", 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2851999999999998e-05 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.157928+0000", 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086371099999999997 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.164297+0000", 2024-08-30T23:57:43.620 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063691900000000003 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490984+0000", 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32668678800000001 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491017+0000", 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3030000000000001e-05 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.621 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491040+0000", 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2486000000000001e-05 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37734 8.1 e864/840 ondisk, result = 0)", 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.39721686999999, 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33298887799999999, 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.622 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.623 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8500000000000002e-07 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158060+0000", 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3620000000000002e-06 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490938+0000", 2024-08-30T23:57:43.624 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33287868399999998 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490943+0000", 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.481e-06 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490976+0000", 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3006999999999998e-05 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.625 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490981+0000", 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0710000000000002e-06 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491045+0000", 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3788000000000007e-05 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.626 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37770 8.1 8:8a7dfc05:::benchmark_data_smithi173_143816_object37769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.228037+0000", 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.32723528299999, 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.393374581, 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37770 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.627 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228037+0000", 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228037+0000", 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228039+0000", 2024-08-30T23:57:43.628 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2550000000000001e-06 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228154+0000", 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115809 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228166+0000", 2024-08-30T23:57:43.629 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1505000000000001e-05 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228171+0000", 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0499999999999999e-06 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228250+0000", 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9462000000000006e-05 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.630 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228340+0000", 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9864000000000006e-05 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228442+0000", 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101697 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229583+0000", 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011410179999999999 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.631 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.264158+0000", 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034575084999999998 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621315+0000", 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35715717000000002 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621360+0000", 2024-08-30T23:57:43.632 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4866e-05 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621412+0000", 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1799999999999999e-05 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37770 8.1 e864/840 ondisk, result = 0)", 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.229920+0000", 2024-08-30T23:57:43.633 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.32535267099999, 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39149838599999998, 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229920+0000", 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229920+0000", 2024-08-30T23:57:43.634 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229921+0000", 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229924+0000", 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3330000000000001e-06 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.635 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621242+0000", 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39131752400000003 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621251+0000", 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4860000000000003e-06 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621303+0000", 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2337999999999999e-05 2024-08-30T23:57:43.636 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621311+0000", 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4320000000000002e-06 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621418+0000", 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107575 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.637 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37815 8.1 8:8b9e7ed9:::benchmark_data_smithi173_143816_object37814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.488921+0000", 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.066351645, 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26600679599999999, 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37815 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.638 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.488921+0000", 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.488921+0000", 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.639 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.488922+0000", 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4139999999999999e-06 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.488971+0000", 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8593e-05 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.488982+0000", 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.065e-05 2024-08-30T23:57:43.640 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.488988+0000", 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0410000000000003e-06 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.489033+0000", 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5679000000000001e-05 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.641 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.489128+0000", 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4210999999999996e-05 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.489216+0000", 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8277999999999997e-05 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490337+0000", 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011212730000000001 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.642 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490736+0000", 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00039897899999999999 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754757+0000", 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26402059100000003 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754857+0000", 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100265 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.643 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754928+0000", 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0821999999999994e-05 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37815 8.1 e864/840 ondisk, result = 0)", 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.490921+0000", 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 196.06435127500001, 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26401236700000003, 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.644 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490921+0000", 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490921+0000", 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.645 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490922+0000", 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.75e-07 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490925+0000", 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.213e-06 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754604+0000", 2024-08-30T23:57:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.26367897099999998 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754617+0000", 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3063e-05 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754731+0000", 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113688 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754747+0000", 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6432000000000001e-05 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.754934+0000", 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018642499999999999 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(8.f 864/854 [PushReplyOp(8:ff509c29:::benchmark_data_smithi173_143816_object7126:head)])", 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:28.072822+0000", 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 195.48245066300001, 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46479312699999997, 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072822+0000", 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072822+0000", 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072823+0000", 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072831+0000", 2024-08-30T23:57:43.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1890000000000001e-06 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536520+0000", 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46368978100000002 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536534+0000", 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3192e-05 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536618+0000", 2024-08-30T23:57:43.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4377999999999997e-05 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.537615+0000", 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00099712900000000003 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:45729 8.1 8:842ca331:::benchmark_data_smithi173_143816_object45728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-08-30T23:57:43.652 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:36.750932+0000", 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.80434063199999, 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.51209708499999995, 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 45729 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750932+0000", 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750932+0000", 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750934+0000", 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6789999999999999e-06 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750972+0000", 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8312000000000002e-05 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750988+0000", 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6388e-05 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750996+0000", 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4989999999999997e-06 2024-08-30T23:57:43.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751056+0000", 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9786e-05 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751197+0000", 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140879 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751340+0000", 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014324499999999999 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752873+0000", 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00153348 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.753371+0000", 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049803800000000002 2024-08-30T23:57:43.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262925+0000", 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50955419099999999 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262977+0000", 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1261999999999998e-05 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.263029+0000", 2024-08-30T23:57:43.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2326e-05 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:45729 8.1 e869/840 ondisk, result = 0)", 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:36.752989+0000", 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 186.80228377500001, 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.51004729900000001, 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752989+0000", 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752989+0000", 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752990+0000", 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0100000000000004e-07 2024-08-30T23:57:43.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752993+0000", 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.619e-06 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262804+0000", 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50981123900000003 2024-08-30T23:57:43.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262811+0000", 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6780000000000003e-06 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262906+0000", 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4585000000000003e-05 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:43.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262918+0000", 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2184000000000001e-05 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.263036+0000", 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118293 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:43.663 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:57:43.664 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:43.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:43.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:43.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:43.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:43.914 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:43.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:43.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 1 16 1357 1341 83.8079 83.8125 0.0128356 0.0118879 2024-08-30T23:57:44.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:44.049 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:43 smithi106 ceph-mon[16603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:57:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:44.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:44.177 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:44.279 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:43 smithi136 ceph-mon[20207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:57:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:44.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:44.305 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:44.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:44.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:43 smithi173 ceph-mon[20554]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-30T23:57:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:44.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:44.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:44.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:44.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:44.557 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:44.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:57:44.652 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:44.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:44.678 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:44.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:44.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:44.818 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:44.902 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:44.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:44.946 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:44.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:44.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 2 16 3676 3660 114.367 144.938 0.00110316 0.00861304 2024-08-30T23:57:45.025 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:44 smithi136 ceph-mon[20207]: osdmap e944: 12 total, 11 up, 12 in 2024-08-30T23:57:45.026 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:44 smithi136 ceph-mon[20207]: pgmap v2100: 47 pgs: 16 active+undersized, 1 active+undersized+degraded, 30 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:45.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:45.073 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:45.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:44 smithi106 ceph-mon[16603]: osdmap e944: 12 total, 11 up, 12 in 2024-08-30T23:57:45.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:44 smithi106 ceph-mon[16603]: pgmap v2100: 47 pgs: 16 active+undersized, 1 active+undersized+degraded, 30 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:45.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.217 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:45.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:45.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:44 smithi173 ceph-mon[20554]: osdmap e944: 12 total, 11 up, 12 in 2024-08-30T23:57:45.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:44 smithi173 ceph-mon[20554]: pgmap v2100: 47 pgs: 16 active+undersized, 1 active+undersized+degraded, 30 active+clean; 616 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-30T23:57:45.342 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:45.349 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:4091 9.f 9:f8dc2dea:::benchmark_data_smithi173_160189_object4090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:45.392326+0000", 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.047518391, 2024-08-30T23:57:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.047547988999999999, 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 4091 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:57:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392326+0000", 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392326+0000", 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392327+0000", 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7800000000000001e-07 2024-08-30T23:57:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392375+0000", 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8136999999999997e-05 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392385+0000", 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9080000000000007e-06 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392390+0000", 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1449999999999999e-06 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392426+0000", 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5808e-05 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392504+0000", 2024-08-30T23:57:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7681000000000004e-05 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,9", 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392580+0000", 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6106000000000006e-05 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.392917+0000", 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033764199999999998 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:57:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:4094 9.d 9:b84f9b0d:::benchmark_data_smithi173_160189_object4093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:45.395244+0000", 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.044600106, 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044674508000000002, 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:57:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 4094 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395244+0000", 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395244+0000", 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395245+0000", 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3600000000000003e-07 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395301+0000", 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5637999999999998e-05 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395310+0000", 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1800000000000002e-06 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395315+0000", 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7090000000000001e-06 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395361+0000", 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6618e-05 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395443+0000", 2024-08-30T23:57:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1804999999999993e-05 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.395521+0000", 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8310000000000001e-05 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.396097+0000", 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057578900000000005 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:57:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:4098 9.a 9:5eea45f9:::benchmark_data_smithi173_160189_object4097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:45.400558+0000", 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.039286861999999999, 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039398162, 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:57:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 4098 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400558+0000", 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400558+0000", 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400559+0000", 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5900000000000005e-07 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400600+0000", 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1362999999999997e-05 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400611+0000", 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0917999999999999e-05 2024-08-30T23:57:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400616+0000", 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5350000000000004e-06 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400657+0000", 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0445999999999999e-05 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400737+0000", 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0255999999999999e-05 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.400816+0000", 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8838000000000006e-05 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:57:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.401202+0000", 2024-08-30T23:57:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000385869 2024-08-30T23:57:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:57:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:4079 9.6 9:64960e39:::benchmark_data_smithi173_160189_object4078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:45.380110+0000", 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.059734203999999999, 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.059891131, 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 4079 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:57:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380110+0000", 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380110+0000", 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380111+0000", 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6499999999999999e-07 2024-08-30T23:57:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380177+0000", 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5945000000000006e-05 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380186+0000", 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7509999999999997e-06 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380190+0000", 2024-08-30T23:57:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380221+0000", 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0576e-05 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380290+0000", 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9480000000000006e-05 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:57:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380370+0000", 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9524999999999995e-05 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.380777+0000", 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040712700000000002 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.396284+0000", 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015507228 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:57:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:4086 9.6 9:62dc48e5:::benchmark_data_smithi173_160189_object4085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:45.388226+0000", 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.051618445999999998, 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.051815272000000002, 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:57:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 4086 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388226+0000", 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388226+0000", 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388228+0000", 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0870000000000001e-06 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388262+0000", 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4137000000000002e-05 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388281+0000", 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8360999999999999e-05 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388288+0000", 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2409999999999998e-06 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388336+0000", 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7958999999999999e-05 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:57:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388420+0000", 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3905000000000004e-05 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388501+0000", 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1680000000000004e-05 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.388962+0000", 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00046057800000000001 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.401809+0000", 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012846959999999999 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:4087 9.f 9:fe02da23:::benchmark_data_smithi173_160189_object4086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:45.389626+0000", 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.050218173999999997, 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050451861000000001, 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:57:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 4087 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389626+0000", 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389626+0000", 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389627+0000", 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6300000000000004e-07 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389665+0000", 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7540999999999999e-05 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389673+0000", 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7120000000000006e-06 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389678+0000", 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.386e-06 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389716+0000", 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8010999999999999e-05 2024-08-30T23:57:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389793+0000", 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6979999999999998e-05 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,9", 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.389872+0000", 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8957999999999999e-05 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:57:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.390349+0000", 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00047774900000000002 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:45.393322+0000", 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029721140000000001 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-30T23:57:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 6 2024-08-30T23:57:45.477 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:57:45.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:45.502 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:45.604 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:45.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:45.649 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:57:45.650 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:57:45.737 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:45.761 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:45.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:45.890 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:45.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:45.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 3 16 5329 5313 110.68 103.312 0.00129248 0.00901121 2024-08-30T23:57:45.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:46.019 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:46.057 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:57:46.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:45 smithi106 ceph-mon[16603]: from='client.19176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:46.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:46.152 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:46.254 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:46.271 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:45 smithi136 ceph-mon[20207]: from='client.19176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:46.280 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:46.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:57:46.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:45 smithi173 ceph-mon[20554]: from='client.19176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:46.344 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2100,"stamp":"2024-08-30T23:57:43.842778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":124,"acting":124,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8055712,"kb_used_data":601312,"kb_used_omap":109,"kb_used_meta":7454034,"kb_avail":1116886112,"statfs":{"total":1151940427776,"available":1143691378688,"internally_reserved":0,"allocated":615743488,"data_stored":607165779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":7632931480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.030788"},"pg_stats":[{"pgid":"1.1e","version":"23'92","reported_seq":149,"reported_epoch":942,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.426345+0000","last_change":"2024-08-30T23:55:59.172632+0000","last_active":"2024-08-30T23:57:40.426345+0000","last_peered":"2024-08-30T23:57:40.426345+0000","last_clean":"2024-08-30T23:55:57.160064+0000","last_became_active":"2024-08-30T23:55:59.172632+0000","last_became_peered":"2024-08-30T23:55:59.172632+0000","last_unstale":"2024-08-30T23:57:40.426345+0000","last_undegraded":"2024-08-30T23:57:40.426345+0000","last_fullsized":"2024-08-30T23:55:59.167996+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.344869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.344869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:45.291659+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":153,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:40.426316+0000","last_change":"2024-08-30T23:56:46.421822+0000","last_active":"2024-08-30T23:57:40.426316+0000","last_peered":"2024-08-30T23:57:40.426316+0000","last_clean":"2024-08-30T23:57:40.426316+0000","last_became_active":"2024-08-30T23:55:03.321559+0000","last_became_peered":"2024-08-30T23:55:03.321559+0000","last_unstale":"2024-08-30T23:57:40.426316+0000","last_undegraded":"2024-08-30T23:57:40.426316+0000","last_fullsized":"2024-08-30T23:57:40.426316+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:46.421676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:46.421676+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:16.405067+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":153,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.819235+0000","last_change":"2024-08-30T23:56:16.491110+0000","last_active":"2024-08-30T23:57:39.819235+0000","last_peered":"2024-08-30T23:57:39.819235+0000","last_clean":"2024-08-30T23:57:39.819235+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-30T23:57:39.819235+0000","last_undegraded":"2024-08-30T23:57:39.819235+0000","last_fullsized":"2024-08-30T23:57:39.819235+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.491037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.491037+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:41.126506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":161,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.819224+0000","last_change":"2024-08-30T23:57:32.122253+0000","last_active":"2024-08-30T23:57:39.819224+0000","last_peered":"2024-08-30T23:57:39.819224+0000","last_clean":"2024-08-30T23:57:39.819224+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-30T23:57:39.819224+0000","last_undegraded":"2024-08-30T23:57:39.819224+0000","last_fullsized":"2024-08-30T23:57:39.819224+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.122186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.122186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.813541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":153,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.821082+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-30T23:57:40.821082+0000","last_peered":"2024-08-30T23:57:40.821082+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-30T23:57:40.821082+0000","last_undegraded":"2024-08-30T23:57:40.821082+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":155,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538514+0000","last_change":"2024-08-30T23:56:45.191296+0000","last_active":"2024-08-30T23:57:41.538514+0000","last_peered":"2024-08-30T23:57:41.538514+0000","last_clean":"2024-08-30T23:57:41.538514+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-30T23:57:41.538514+0000","last_undegraded":"2024-08-30T23:57:41.538514+0000","last_fullsized":"2024-08-30T23:57:41.538514+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.191245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.191245+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:49.183507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":372,"reported_epoch":941,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:39.237428+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-30T23:57:39.237428+0000","last_peered":"2024-08-30T23:57:39.237428+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-30T23:57:39.237428+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:43.132539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":153,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.819387+0000","last_change":"2024-08-30T23:56:40.272914+0000","last_active":"2024-08-30T23:57:39.819387+0000","last_peered":"2024-08-30T23:57:39.819387+0000","last_clean":"2024-08-30T23:57:39.819387+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-30T23:57:39.819387+0000","last_undegraded":"2024-08-30T23:57:39.819387+0000","last_fullsized":"2024-08-30T23:57:39.819387+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:40.272848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:40.272848+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:09.954699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":163,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538211+0000","last_change":"2024-08-30T23:57:27.164513+0000","last_active":"2024-08-30T23:57:41.538211+0000","last_peered":"2024-08-30T23:57:41.538211+0000","last_clean":"2024-08-30T23:57:41.538211+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-30T23:57:41.538211+0000","last_undegraded":"2024-08-30T23:57:41.538211+0000","last_fullsized":"2024-08-30T23:57:41.538211+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:27.164462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:27.164462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:47.956715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":86,"reported_epoch":941,"state":"active+undersized","last_fresh":"2024-08-30T23:57:39.237702+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-30T23:57:39.237702+0000","last_peered":"2024-08-30T23:57:39.237702+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-30T23:57:39.237702+0000","last_undegraded":"2024-08-30T23:57:39.237702+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:57:42.089652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":208,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:40.426233+0000","last_change":"2024-08-30T23:57:14.423376+0000","last_active":"2024-08-30T23:57:40.426233+0000","last_peered":"2024-08-30T23:57:40.426233+0000","last_clean":"2024-08-30T23:57:40.426233+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:57:40.426233+0000","last_undegraded":"2024-08-30T23:57:40.426233+0000","last_fullsized":"2024-08-30T23:57:40.426233+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:14.423316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:14.423316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.319991+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":355,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.237481+0000","last_change":"2024-08-30T23:57:35.093839+0000","last_active":"2024-08-30T23:57:39.237481+0000","last_peered":"2024-08-30T23:57:39.237481+0000","last_clean":"2024-08-30T23:57:39.237481+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:57:39.237481+0000","last_undegraded":"2024-08-30T23:57:39.237481+0000","last_fullsized":"2024-08-30T23:57:39.237481+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:35.093787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.093787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:01.863839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":969,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.819203+0000","last_change":"2024-08-30T23:56:39.308813+0000","last_active":"2024-08-30T23:57:39.819203+0000","last_peered":"2024-08-30T23:57:39.819203+0000","last_clean":"2024-08-30T23:57:39.819203+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:57:39.819203+0000","last_undegraded":"2024-08-30T23:57:39.819203+0000","last_fullsized":"2024-08-30T23:57:39.819203+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:39.308738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:39.308738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:58.601526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":343,"reported_epoch":941,"state":"active+undersized","last_fresh":"2024-08-30T23:57:39.237712+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-30T23:57:39.237712+0000","last_peered":"2024-08-30T23:57:39.237712+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-30T23:57:39.237712+0000","last_undegraded":"2024-08-30T23:57:39.237712+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.238026+0000","last_change":"2024-08-30T23:57:36.063942+0000","last_active":"2024-08-30T23:57:39.238026+0000","last_peered":"2024-08-30T23:57:39.238026+0000","last_clean":"2024-08-30T23:57:39.238026+0000","last_became_active":"2024-08-30T23:57:36.062663+0000","last_became_peered":"2024-08-30T23:57:36.062663+0000","last_unstale":"2024-08-30T23:57:39.238026+0000","last_undegraded":"2024-08-30T23:57:39.238026+0000","last_fullsized":"2024-08-30T23:57:39.238026+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:40.630582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":762,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:41.538419+0000","last_change":"2024-08-30T23:55:59.173538+0000","last_active":"2024-08-30T23:57:41.538419+0000","last_peered":"2024-08-30T23:57:41.538419+0000","last_clean":"2024-08-30T23:55:57.750202+0000","last_became_active":"2024-08-30T23:55:59.173538+0000","last_became_peered":"2024-08-30T23:55:59.173538+0000","last_unstale":"2024-08-30T23:57:41.538419+0000","last_undegraded":"2024-08-30T23:57:41.538419+0000","last_fullsized":"2024-08-30T23:55:59.169546+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:53.668489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:53.668489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:38.812245+0000","last_change":"2024-08-30T23:57:36.062999+0000","last_active":"2024-08-30T23:57:38.812245+0000","last_peered":"2024-08-30T23:57:38.812245+0000","last_clean":"2024-08-30T23:57:38.812245+0000","last_became_active":"2024-08-30T23:57:36.062695+0000","last_became_peered":"2024-08-30T23:57:36.062695+0000","last_unstale":"2024-08-30T23:57:38.812245+0000","last_undegraded":"2024-08-30T23:57:38.812245+0000","last_fullsized":"2024-08-30T23:57:38.812245+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:44.412539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":212,"reported_epoch":942,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.426278+0000","last_change":"2024-08-30T23:55:59.171246+0000","last_active":"2024-08-30T23:57:40.426278+0000","last_peered":"2024-08-30T23:57:40.426278+0000","last_clean":"2024-08-30T23:55:57.160009+0000","last_became_active":"2024-08-30T23:55:59.171246+0000","last_became_peered":"2024-08-30T23:55:59.171246+0000","last_unstale":"2024-08-30T23:57:40.426278+0000","last_undegraded":"2024-08-30T23:57:40.426278+0000","last_fullsized":"2024-08-30T23:55:59.168006+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:36.270718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:36.270718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":941,"state":"active+undersized","last_fresh":"2024-08-30T23:57:39.237716+0000","last_change":"2024-08-30T23:57:36.059533+0000","last_active":"2024-08-30T23:57:39.237716+0000","last_peered":"2024-08-30T23:57:39.237716+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.059533+0000","last_became_peered":"2024-08-30T23:57:36.059533+0000","last_unstale":"2024-08-30T23:57:39.237716+0000","last_undegraded":"2024-08-30T23:57:39.237716+0000","last_fullsized":"2024-08-30T23:57:36.057835+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:05.071870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":556,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538590+0000","last_change":"2024-08-30T23:56:48.173739+0000","last_active":"2024-08-30T23:57:41.538590+0000","last_peered":"2024-08-30T23:57:41.538590+0000","last_clean":"2024-08-30T23:57:41.538590+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-30T23:57:41.538590+0000","last_undegraded":"2024-08-30T23:57:41.538590+0000","last_fullsized":"2024-08-30T23:57:41.538590+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:48.173668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:48.173668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:13.591624+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":18,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:41.538676+0000","last_change":"2024-08-30T23:57:36.061016+0000","last_active":"2024-08-30T23:57:41.538676+0000","last_peered":"2024-08-30T23:57:41.538676+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.061016+0000","last_became_peered":"2024-08-30T23:57:36.061016+0000","last_unstale":"2024-08-30T23:57:41.538676+0000","last_undegraded":"2024-08-30T23:57:41.538676+0000","last_fullsized":"2024-08-30T23:57:36.054773+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:45.036298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":["10","1"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":530,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538280+0000","last_change":"2024-08-30T23:56:16.234435+0000","last_active":"2024-08-30T23:57:41.538280+0000","last_peered":"2024-08-30T23:57:41.538280+0000","last_clean":"2024-08-30T23:57:41.538280+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-30T23:57:41.538280+0000","last_undegraded":"2024-08-30T23:57:41.538280+0000","last_fullsized":"2024-08-30T23:57:41.538280+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.234371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.234371+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.237806+0000","last_change":"2024-08-30T23:57:36.063702+0000","last_active":"2024-08-30T23:57:39.237806+0000","last_peered":"2024-08-30T23:57:39.237806+0000","last_clean":"2024-08-30T23:57:39.237806+0000","last_became_active":"2024-08-30T23:57:36.063425+0000","last_became_peered":"2024-08-30T23:57:36.063425+0000","last_unstale":"2024-08-30T23:57:39.237806+0000","last_undegraded":"2024-08-30T23:57:39.237806+0000","last_fullsized":"2024-08-30T23:57:39.237806+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.204762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1169,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538208+0000","last_change":"2024-08-30T23:56:20.789107+0000","last_active":"2024-08-30T23:57:41.538208+0000","last_peered":"2024-08-30T23:57:41.538208+0000","last_clean":"2024-08-30T23:57:41.538208+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-30T23:57:41.538208+0000","last_undegraded":"2024-08-30T23:57:41.538208+0000","last_fullsized":"2024-08-30T23:57:41.538208+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:20.789023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:20.789023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:50.461299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":21,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538645+0000","last_change":"2024-08-30T23:57:36.064940+0000","last_active":"2024-08-30T23:57:41.538645+0000","last_peered":"2024-08-30T23:57:41.538645+0000","last_clean":"2024-08-30T23:57:41.538645+0000","last_became_active":"2024-08-30T23:57:36.064599+0000","last_became_peered":"2024-08-30T23:57:36.064599+0000","last_unstale":"2024-08-30T23:57:41.538645+0000","last_undegraded":"2024-08-30T23:57:41.538645+0000","last_fullsized":"2024-08-30T23:57:41.538645+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:01.169354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":515,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538669+0000","last_change":"2024-08-30T23:57:08.049165+0000","last_active":"2024-08-30T23:57:41.538669+0000","last_peered":"2024-08-30T23:57:41.538669+0000","last_clean":"2024-08-30T23:57:41.538669+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:57:41.538669+0000","last_undegraded":"2024-08-30T23:57:41.538669+0000","last_fullsized":"2024-08-30T23:57:41.538669+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:08.049028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:08.049028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:28.561467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":941,"state":"active+undersized","last_fresh":"2024-08-30T23:57:39.237567+0000","last_change":"2024-08-30T23:57:36.059522+0000","last_active":"2024-08-30T23:57:39.237567+0000","last_peered":"2024-08-30T23:57:39.237567+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.059522+0000","last_became_peered":"2024-08-30T23:57:36.059522+0000","last_unstale":"2024-08-30T23:57:39.237567+0000","last_undegraded":"2024-08-30T23:57:39.237567+0000","last_fullsized":"2024-08-30T23:57:36.057850+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:58.552624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":2002,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.821083+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-30T23:57:40.821083+0000","last_peered":"2024-08-30T23:57:40.821083+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-30T23:57:40.821083+0000","last_undegraded":"2024-08-30T23:57:40.821083+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":18,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.819279+0000","last_change":"2024-08-30T23:57:36.061979+0000","last_active":"2024-08-30T23:57:39.819279+0000","last_peered":"2024-08-30T23:57:39.819279+0000","last_clean":"2024-08-30T23:57:39.819279+0000","last_became_active":"2024-08-30T23:57:36.061597+0000","last_became_peered":"2024-08-30T23:57:36.061597+0000","last_unstale":"2024-08-30T23:57:39.819279+0000","last_undegraded":"2024-08-30T23:57:39.819279+0000","last_fullsized":"2024-08-30T23:57:39.819279+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:48.863769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":334,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538750+0000","last_change":"2024-08-30T23:57:34.913534+0000","last_active":"2024-08-30T23:57:41.538750+0000","last_peered":"2024-08-30T23:57:41.538750+0000","last_clean":"2024-08-30T23:57:41.538750+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:57:41.538750+0000","last_undegraded":"2024-08-30T23:57:41.538750+0000","last_fullsized":"2024-08-30T23:57:41.538750+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:34.913482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:34.913482+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.935529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":21,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538122+0000","last_change":"2024-08-30T23:57:36.060551+0000","last_active":"2024-08-30T23:57:41.538122+0000","last_peered":"2024-08-30T23:57:41.538122+0000","last_clean":"2024-08-30T23:57:41.538122+0000","last_became_active":"2024-08-30T23:57:36.060190+0000","last_became_peered":"2024-08-30T23:57:36.060190+0000","last_unstale":"2024-08-30T23:57:41.538122+0000","last_undegraded":"2024-08-30T23:57:41.538122+0000","last_fullsized":"2024-08-30T23:57:41.538122+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:59.409650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":992,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:38.812276+0000","last_change":"2024-08-30T23:57:32.972014+0000","last_active":"2024-08-30T23:57:38.812276+0000","last_peered":"2024-08-30T23:57:38.812276+0000","last_clean":"2024-08-30T23:57:38.812276+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-30T23:57:38.812276+0000","last_undegraded":"2024-08-30T23:57:38.812276+0000","last_fullsized":"2024-08-30T23:57:38.812276+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.971963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.971963+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:59.480355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":15,"reported_epoch":942,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.426296+0000","last_change":"2024-08-30T23:57:36.063471+0000","last_active":"2024-08-30T23:57:40.426296+0000","last_peered":"2024-08-30T23:57:40.426296+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.063471+0000","last_became_peered":"2024-08-30T23:57:36.063471+0000","last_unstale":"2024-08-30T23:57:40.426296+0000","last_undegraded":"2024-08-30T23:57:40.426296+0000","last_fullsized":"2024-08-30T23:57:36.061798+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:52.618644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","5"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":972,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.821080+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-30T23:57:40.821080+0000","last_peered":"2024-08-30T23:57:40.821080+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-30T23:57:40.821080+0000","last_undegraded":"2024-08-30T23:57:40.821080+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:42.038763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.237720+0000","last_change":"2024-08-30T23:57:36.063587+0000","last_active":"2024-08-30T23:57:39.237720+0000","last_peered":"2024-08-30T23:57:39.237720+0000","last_clean":"2024-08-30T23:57:39.237720+0000","last_became_active":"2024-08-30T23:57:36.063215+0000","last_became_peered":"2024-08-30T23:57:36.063215+0000","last_unstale":"2024-08-30T23:57:39.237720+0000","last_undegraded":"2024-08-30T23:57:39.237720+0000","last_fullsized":"2024-08-30T23:57:39.237720+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.864690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":903,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:40.821090+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-30T23:57:40.821090+0000","last_peered":"2024-08-30T23:57:40.821090+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-30T23:57:40.821090+0000","last_undegraded":"2024-08-30T23:57:40.821090+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":18,"reported_epoch":943,"state":"active+undersized","last_fresh":"2024-08-30T23:57:41.538831+0000","last_change":"2024-08-30T23:57:36.056134+0000","last_active":"2024-08-30T23:57:41.538831+0000","last_peered":"2024-08-30T23:57:41.538831+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.056134+0000","last_became_peered":"2024-08-30T23:57:36.056134+0000","last_unstale":"2024-08-30T23:57:41.538831+0000","last_undegraded":"2024-08-30T23:57:41.538831+0000","last_fullsized":"2024-08-30T23:57:36.054771+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:46.658959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","3"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1180,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.539126+0000","last_change":"2024-08-30T23:57:36.159072+0000","last_active":"2024-08-30T23:57:41.539126+0000","last_peered":"2024-08-30T23:57:41.539126+0000","last_clean":"2024-08-30T23:57:41.539126+0000","last_became_active":"2024-08-30T23:57:36.158625+0000","last_became_peered":"2024-08-30T23:57:36.158625+0000","last_unstale":"2024-08-30T23:57:41.539126+0000","last_undegraded":"2024-08-30T23:57:41.539126+0000","last_fullsized":"2024-08-30T23:57:41.539126+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.888321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.888321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:47.316661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":21,"reported_epoch":943,"state":"active+clean","last_fresh":"2024-08-30T23:57:41.538339+0000","last_change":"2024-08-30T23:57:36.061082+0000","last_active":"2024-08-30T23:57:41.538339+0000","last_peered":"2024-08-30T23:57:41.538339+0000","last_clean":"2024-08-30T23:57:41.538339+0000","last_became_active":"2024-08-30T23:57:36.060760+0000","last_became_peered":"2024-08-30T23:57:36.060760+0000","last_unstale":"2024-08-30T23:57:41.538339+0000","last_undegraded":"2024-08-30T23:57:41.538339+0000","last_fullsized":"2024-08-30T23:57:41.538339+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:05.712674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":365,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.237536+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-30T23:57:39.237536+0000","last_peered":"2024-08-30T23:57:39.237536+0000","last_clean":"2024-08-30T23:57:39.237536+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-30T23:57:39.237536+0000","last_undegraded":"2024-08-30T23:57:39.237536+0000","last_fullsized":"2024-08-30T23:57:39.237536+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:53.707888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:38.812236+0000","last_change":"2024-08-30T23:57:36.061220+0000","last_active":"2024-08-30T23:57:38.812236+0000","last_peered":"2024-08-30T23:57:38.812236+0000","last_clean":"2024-08-30T23:57:38.812236+0000","last_became_active":"2024-08-30T23:57:36.060876+0000","last_became_peered":"2024-08-30T23:57:36.060876+0000","last_unstale":"2024-08-30T23:57:38.812236+0000","last_undegraded":"2024-08-30T23:57:38.812236+0000","last_fullsized":"2024-08-30T23:57:38.812236+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.757644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":962,"reported_epoch":941,"state":"active+undersized","last_fresh":"2024-08-30T23:57:39.238116+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-30T23:57:39.238116+0000","last_peered":"2024-08-30T23:57:39.238116+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-30T23:57:39.238116+0000","last_undegraded":"2024-08-30T23:57:39.238116+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:57:43.570855+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:38.812248+0000","last_change":"2024-08-30T23:57:36.060493+0000","last_active":"2024-08-30T23:57:38.812248+0000","last_peered":"2024-08-30T23:57:38.812248+0000","last_clean":"2024-08-30T23:57:38.812248+0000","last_became_active":"2024-08-30T23:57:36.060205+0000","last_became_peered":"2024-08-30T23:57:36.060205+0000","last_unstale":"2024-08-30T23:57:38.812248+0000","last_undegraded":"2024-08-30T23:57:38.812248+0000","last_fullsized":"2024-08-30T23:57:38.812248+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:48.787551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1100,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.238114+0000","last_change":"2024-08-30T23:56:19.712394+0000","last_active":"2024-08-30T23:57:39.238114+0000","last_peered":"2024-08-30T23:57:39.238114+0000","last_clean":"2024-08-30T23:57:39.238114+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-30T23:57:39.238114+0000","last_undegraded":"2024-08-30T23:57:39.238114+0000","last_fullsized":"2024-08-30T23:57:39.238114+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:19.712327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:19.712327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":16,"reported_epoch":941,"state":"active+clean","last_fresh":"2024-08-30T23:57:39.238245+0000","last_change":"2024-08-30T23:57:36.063456+0000","last_active":"2024-08-30T23:57:39.238245+0000","last_peered":"2024-08-30T23:57:39.238245+0000","last_clean":"2024-08-30T23:57:39.238245+0000","last_became_active":"2024-08-30T23:57:36.063092+0000","last_became_peered":"2024-08-30T23:57:36.063092+0000","last_unstale":"2024-08-30T23:57:39.238245+0000","last_undegraded":"2024-08-30T23:57:39.238245+0000","last_fullsized":"2024-08-30T23:57:39.238245+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:46.389881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":214,"reported_epoch":942,"state":"active+clean","last_fresh":"2024-08-30T23:57:40.426267+0000","last_change":"2024-08-30T23:56:13.242208+0000","last_active":"2024-08-30T23:57:40.426267+0000","last_peered":"2024-08-30T23:57:40.426267+0000","last_clean":"2024-08-30T23:57:40.426267+0000","last_became_active":"2024-08-30T23:55:01.289376+0000","last_became_peered":"2024-08-30T23:55:01.289376+0000","last_unstale":"2024-08-30T23:57:40.426267+0000","last_undegraded":"2024-08-30T23:57:40.426267+0000","last_fullsized":"2024-08-30T23:57:40.426267+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:13.242035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:13.242035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1055,"reported_epoch":941,"state":"active+undersized","last_fresh":"2024-08-30T23:57:39.238484+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-30T23:57:39.238484+0000","last_peered":"2024-08-30T23:57:39.238484+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-30T23:57:39.238484+0000","last_undegraded":"2024-08-30T23:57:39.238484+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":43,"acting":43,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":81,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664764,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93080388,"statfs":{"total":95995035648,"available":95314317312,"internally_reserved":0,"allocated":1941504,"data_stored":1232000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":678754756},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449496,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":628680,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93116472,"statfs":{"total":95995035648,"available":95351267328,"internally_reserved":0,"allocated":1953792,"data_stored":1240179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":641792448},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711124,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718888,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93026264,"statfs":{"total":95995035648,"available":95258894336,"internally_reserved":0,"allocated":1916928,"data_stored":1216248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":734198208},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972767,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":510656,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93234496,"statfs":{"total":95995035648,"available":95472123904,"internally_reserved":0,"allocated":1941504,"data_stored":1232000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528745,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":484616,"kb_used_data":2228,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93260536,"statfs":{"total":95995035648,"available":95498788864,"internally_reserved":0,"allocated":2281472,"data_stored":1567859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775247,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259820,"kb_used_data":2072,"kb_used_omap":5,"kb_used_meta":257722,"kb_avail":93485332,"statfs":{"total":95995035648,"available":95728979968,"internally_reserved":0,"allocated":2121728,"data_stored":1417496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":263907402},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709196,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":93035956,"statfs":{"total":95995035648,"available":95268818944,"internally_reserved":0,"allocated":1953792,"data_stored":1240179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150133,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831652,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92913500,"statfs":{"total":95995035648,"available":95143424000,"internally_reserved":0,"allocated":2244608,"data_stored":1543928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":849344967},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411772,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660816,"kb_used_data":2108,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93084336,"statfs":{"total":95995035648,"available":95318360064,"internally_reserved":0,"allocated":2158592,"data_stored":1441427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":494908,"kb_used_data":1896,"kb_used_omap":87,"kb_used_meta":492904,"kb_avail":93250244,"statfs":{"total":95995035648,"available":95488249856,"internally_reserved":0,"allocated":1941504,"data_stored":1232000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":504733855},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":787236,"kb_used_data":2192,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92957916,"statfs":{"total":95995035648,"available":95188905984,"internally_reserved":0,"allocated":2244608,"data_stored":1543928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:57:46.348 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-30T23:57:46.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_7 pg_num 26 2024-08-30T23:57:46.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:46.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:46.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:46.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:46.592 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:46.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:46.654 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:46.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:46.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:46.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:46.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:46.909 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:46.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:46.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 4 16 7411 7395 115.537 130.125 0.00113088 0.00858167 2024-08-30T23:57:47.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:47.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:46 smithi136 ceph-mon[20207]: pgmap v2101: 47 pgs: 14 active+undersized, 3 active+undersized+degraded, 30 active+clean; 110 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 291 op/s; 413/5268 objects degraded (7.840%) 2024-08-30T23:57:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:46 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3679277949' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:46 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:46 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:46 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:47.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:46 smithi136 ceph-mon[20207]: osdmap e945: 12 total, 11 up, 12 in 2024-08-30T23:57:47.054 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:46 smithi106 ceph-mon[16603]: pgmap v2101: 47 pgs: 14 active+undersized, 3 active+undersized+degraded, 30 active+clean; 110 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 291 op/s; 413/5268 objects degraded (7.840%) 2024-08-30T23:57:47.054 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:46 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3679277949' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:47.054 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:46 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:47.054 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:46 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:47.055 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:46 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:47.055 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:46 smithi106 ceph-mon[16603]: osdmap e945: 12 total, 11 up, 12 in 2024-08-30T23:57:47.103 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:47.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:47.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:47.174 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:57:47.175 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:57:47.175 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:57:47.175 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:57:47.175 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:57:47.263 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:47.289 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:46 smithi173 ceph-mon[20554]: pgmap v2101: 47 pgs: 14 active+undersized, 3 active+undersized+degraded, 30 active+clean; 110 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 291 op/s; 413/5268 objects degraded (7.840%) 2024-08-30T23:57:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:46 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3679277949' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:46 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:46 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:46 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:47.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:46 smithi173 ceph-mon[20554]: osdmap e945: 12 total, 11 up, 12 in 2024-08-30T23:57:47.391 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:47.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:47.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:47.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:47.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:47.671 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:47.695 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:47.695 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:47.698 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:47.725 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:47.725 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:47.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:47.754 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:47.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:47.780 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:47.780 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:47.797 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:47.810 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:47.810 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:47.836 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:47.836 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:47.862 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:47.862 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:47.862 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:47.888 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:47.888 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:47.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:47.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:57:47.917 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:47.917 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:47.926 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:47.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:47.942 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:47.943 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:47.961 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:47 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 413/5268 objects degraded (7.840%), 3 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:47.962 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/397074636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:47.971 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:47.972 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:47.979 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 5 16 9555 9539 119.227 134 0.0105975 0.00837389 2024-08-30T23:57:48.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:48.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.052 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:48.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:48.179 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:48.219 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:57:48.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:48.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:47 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 413/5268 objects degraded (7.840%), 3 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:48.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/397074636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:48.306 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:47 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 413/5268 objects degraded (7.840%), 3 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:48.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/397074636' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:48.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:48.434 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:48.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:48.560 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:48.662 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:48.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:48.753 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:48.753 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:48.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:48.820 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:48.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:48.945 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:48.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:48.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 6 16 11661 11645 121.292 131.625 0.00938643 0.00822387 2024-08-30T23:57:49.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:49.071 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:49.095 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:57:49.162 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:48 smithi106 ceph-mon[16603]: pgmap v2103: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 283 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s; 1809/13575 objects degraded (13.326%) 2024-08-30T23:57:49.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:49.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:49.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:49.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:48 smithi136 ceph-mon[20207]: pgmap v2103: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 283 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s; 1809/13575 objects degraded (13.326%) 2024-08-30T23:57:49.299 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:49.323 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:49.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:48 smithi173 ceph-mon[20554]: pgmap v2103: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 283 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s; 1809/13575 objects degraded (13.326%) 2024-08-30T23:57:49.362 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:49.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.448 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:49.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:49.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:57:49.549 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:49.574 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:49.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:49.698 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:49.800 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:49.824 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:49.906 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 9 pg_num to 26 2024-08-30T23:57:49.926 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:49.956 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:49.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:49.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 7 16 13031 13015 116.196 85.625 0.0191382 0.00859529 2024-08-30T23:57:50.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:50.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:50.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1309931835' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-08-30T23:57:50.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:50.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:50.289 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1309931835' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-08-30T23:57:50.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:50.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1309931835' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-08-30T23:57:50.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:50.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:50.451 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:50.553 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:50.577 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:50.679 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:50.703 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:50.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:50.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:50.855 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:50.927 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:57:50.930 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:50.959 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:50.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:50.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 8 16 14735 14719 114.983 106.5 0.0139563 0.00868109 2024-08-30T23:57:51.062 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:51.085 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:50 smithi106 ceph-mon[16603]: pgmap v2104: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 621 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 2982/29763 objects degraded (10.019%) 2024-08-30T23:57:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1309931835' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-08-30T23:57:51.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:50 smithi106 ceph-mon[16603]: osdmap e946: 12 total, 11 up, 12 in 2024-08-30T23:57:51.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2995049942' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:51.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:50 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:51.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:51.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:57:51.214 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:51.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:51.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:50 smithi173 ceph-mon[20554]: pgmap v2104: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 621 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 2982/29763 objects degraded (10.019%) 2024-08-30T23:57:51.224 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1309931835' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-08-30T23:57:51.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:50 smithi173 ceph-mon[20554]: osdmap e946: 12 total, 11 up, 12 in 2024-08-30T23:57:51.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2995049942' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:51.225 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:50 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:50 smithi136 ceph-mon[20207]: pgmap v2104: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 621 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 2982/29763 objects degraded (10.019%) 2024-08-30T23:57:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1309931835' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-08-30T23:57:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:50 smithi136 ceph-mon[20207]: osdmap e946: 12 total, 11 up, 12 in 2024-08-30T23:57:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2995049942' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:51.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:50 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:57:51.316 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:51.352 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:51.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:51.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:51.548 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:57:51.548 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:57:51.589 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:51.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.621 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:51.722 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:51.749 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:51.851 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:51.879 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:51.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:51.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 9 16 17195 17179 119.289 153.75 0.00243008 0.00837385 2024-08-30T23:57:51.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:52.007 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:52.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:52.130 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: osdmap e947: 12 total, 11 up, 12 in 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:52.204 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:51 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:52.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:52.253 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:52.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:52.255 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: osdmap e947: 12 total, 11 up, 12 in 2024-08-30T23:57:52.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:52.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:57:52.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:57:52.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:52.256 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:51 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:52.257 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: osdmap e947: 12 total, 11 up, 12 in 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:52.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:51 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:52.356 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:52.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:52.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:52.500 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:52.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:52.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:52.683 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:57:52.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:52.750 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:52.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:52.874 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:52.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:52.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:52.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 10 16 17391 17375 108.584 12.25 0.0022367 0.00839592 2024-08-30T23:57:52.999 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:53.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:53.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:53.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:52 smithi106 ceph-mon[16603]: pgmap v2107: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 695 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s; 3553/33333 objects degraded (10.659%) 2024-08-30T23:57:53.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:52 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3553/33333 objects degraded (10.659%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:53.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:52 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-30T23:57:53.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:52 smithi106 ceph-mon[16603]: osdmap e948: 12 total, 11 up, 12 in 2024-08-30T23:57:53.143 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:53.189 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:57:53.189 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:57:53.191 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:52 smithi136 ceph-mon[20207]: pgmap v2107: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 695 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s; 3553/33333 objects degraded (10.659%) 2024-08-30T23:57:53.191 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:52 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3553/33333 objects degraded (10.659%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:53.191 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:52 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-30T23:57:53.191 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:52 smithi136 ceph-mon[20207]: osdmap e948: 12 total, 11 up, 12 in 2024-08-30T23:57:53.192 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:53.221 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:57:53.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:57:53.245 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:53.249 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:57:53.249 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:57:53.275 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:53.275 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:57:53.276 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:57:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:53.304 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:57:53.304 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:57:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:52 smithi173 ceph-mon[20554]: pgmap v2107: 47 pgs: 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 695 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s; 3553/33333 objects degraded (10.659%) 2024-08-30T23:57:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:52 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3553/33333 objects degraded (10.659%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:52 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-08-30T23:57:53.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:52 smithi173 ceph-mon[20554]: osdmap e948: 12 total, 11 up, 12 in 2024-08-30T23:57:53.330 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:57:53.331 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:57:53.353 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:57:53.353 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:57:53.353 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:57:53.374 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:57:53.374 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:57:53.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:53.396 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:57:53.396 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:57:53.400 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:53.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:57:53.417 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:57:53.437 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:57:53.438 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:57:53.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:53.522 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.523 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:53.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:53.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:53.754 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:53.783 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:53.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:53.914 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:53.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:53.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 11 16 19102 19086 108.434 106.938 0.00576094 0.00921263 2024-08-30T23:57:54.016 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:54.046 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:54.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:54.169 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.170 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:54.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:57:54.253 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:57:54.253 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:57:54.253 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: osdmap e949: 12 total, 11 up, 12 in 2024-08-30T23:57:54.253 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/921879413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:54.253 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: 1.1c scrub starts 2024-08-30T23:57:54.253 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: 1.1c scrub ok 2024-08-30T23:57:54.254 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: 9.4 scrub starts 2024-08-30T23:57:54.254 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: 9.4 scrub ok 2024-08-30T23:57:54.254 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:54.254 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:53 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:54.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: osdmap e949: 12 total, 11 up, 12 in 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/921879413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: 1.1c scrub starts 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: 1.1c scrub ok 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: 9.4 scrub starts 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: 9.4 scrub ok 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:54.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:53 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:54.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:54.323 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: osdmap e949: 12 total, 11 up, 12 in 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/921879413' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: 1.1c scrub starts 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: 1.1c scrub ok 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: 9.4 scrub starts 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: 9.4 scrub ok 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:54.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:53 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-08-30T23:57:54.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:54.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.415 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:54.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:54.538 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:54.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:54.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:54.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:54.788 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:54.890 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:54.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:54.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:54.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 12 16 19285 19269 100.351 11.4375 0.00201691 0.00940944 2024-08-30T23:57:55.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:55.035 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.036 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.3 scrub starts 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.3 scrub ok 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: pgmap v2110: 57 pgs: 3 peering, 10 unknown, 11 active+undersized, 5 active+undersized+degraded, 28 active+clean; 883 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 3676/42375 objects degraded (8.675%) 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.0 scrub starts 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.2 scrub starts 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.0 scrub ok 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.2 scrub ok 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: 9.6 scrub starts 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-30T23:57:55.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:54 smithi136 ceph-mon[20207]: osdmap e950: 12 total, 11 up, 12 in 2024-08-30T23:57:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:57:55.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:57:55.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:55.158 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.3 scrub starts 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.3 scrub ok 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: pgmap v2110: 57 pgs: 3 peering, 10 unknown, 11 active+undersized, 5 active+undersized+degraded, 28 active+clean; 883 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 3676/42375 objects degraded (8.675%) 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.0 scrub starts 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.2 scrub starts 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.0 scrub ok 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.2 scrub ok 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: 9.6 scrub starts 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-30T23:57:55.238 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:54 smithi106 ceph-mon[16603]: osdmap e950: 12 total, 11 up, 12 in 2024-08-30T23:57:55.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:57:55.281 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.3 scrub starts 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.3 scrub ok 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: pgmap v2110: 57 pgs: 3 peering, 10 unknown, 11 active+undersized, 5 active+undersized+degraded, 28 active+clean; 883 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 3676/42375 objects degraded (8.675%) 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.0 scrub starts 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.2 scrub starts 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.0 scrub ok 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.2 scrub ok 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: 9.6 scrub starts 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-08-30T23:57:55.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:54 smithi173 ceph-mon[20554]: osdmap e950: 12 total, 11 up, 12 in 2024-08-30T23:57:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:57:55.354 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:57:55.354 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:57:55.354 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:57:55.354 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:57:55.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:55.404 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:55.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:55.528 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:55.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:55.629 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:55.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:55.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:55.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:55.784 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:57:55.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:57:55.785 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-30T23:57:55.785 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:57:55.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:55.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:55.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:55.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 13 16 19285 19269 92.6309 0 - 0.00940944 2024-08-30T23:57:55.997 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:57:56.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:56.039 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:56.068 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:56.079 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:56.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:56.205 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:56.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:56.308 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: 9.8 scrub starts 2024-08-30T23:57:56.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: 9.9 scrub starts 2024-08-30T23:57:56.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: 9.1 scrub starts 2024-08-30T23:57:56.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: 9.8 scrub ok 2024-08-30T23:57:56.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: 9.1 scrub ok 2024-08-30T23:57:56.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: 9.6 scrub ok 2024-08-30T23:57:56.309 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:56 smithi106 ceph-mon[16603]: osdmap e951: 12 total, 11 up, 12 in 2024-08-30T23:57:56.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:56.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:56.460 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: 9.8 scrub starts 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: 9.9 scrub starts 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: 9.1 scrub starts 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: 9.8 scrub ok 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: 9.1 scrub ok 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: 9.6 scrub ok 2024-08-30T23:57:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:56 smithi136 ceph-mon[20207]: osdmap e951: 12 total, 11 up, 12 in 2024-08-30T23:57:56.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: 9.8 scrub starts 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: 9.9 scrub starts 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: 9.1 scrub starts 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: 9.8 scrub ok 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: 9.1 scrub ok 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: 9.6 scrub ok 2024-08-30T23:57:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:56 smithi173 ceph-mon[20554]: osdmap e951: 12 total, 11 up, 12 in 2024-08-30T23:57:56.588 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:56.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:56.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:56.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:56.845 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:56.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:56.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:56.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 14 16 20445 20429 91.1926 36.25 0.00782938 0.010954 2024-08-30T23:57:56.999 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:57:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:57.056 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:57:57.104 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:57.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:57.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:57.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:57.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:57 smithi106 ceph-mon[16603]: pgmap v2113: 57 pgs: 1 active+clean+scrubbing, 3 peering, 9 unknown, 11 active+undersized, 5 active+undersized+degraded, 28 active+clean; 998 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.06k op/s; 4464/47889 objects degraded (9.322%) 2024-08-30T23:57:57.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:57 smithi106 ceph-mon[16603]: osdmap e952: 12 total, 11 up, 12 in 2024-08-30T23:57:57.347 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:57 smithi106 ceph-mon[16603]: 9.9 scrub ok 2024-08-30T23:57:57.363 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:57.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:57.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:57.517 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:57.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:57 smithi136 ceph-mon[20207]: pgmap v2113: 57 pgs: 1 active+clean+scrubbing, 3 peering, 9 unknown, 11 active+undersized, 5 active+undersized+degraded, 28 active+clean; 998 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.06k op/s; 4464/47889 objects degraded (9.322%) 2024-08-30T23:57:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:57 smithi136 ceph-mon[20207]: osdmap e952: 12 total, 11 up, 12 in 2024-08-30T23:57:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:57 smithi136 ceph-mon[20207]: 9.9 scrub ok 2024-08-30T23:57:57.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:57 smithi173 ceph-mon[20554]: pgmap v2113: 57 pgs: 1 active+clean+scrubbing, 3 peering, 9 unknown, 11 active+undersized, 5 active+undersized+degraded, 28 active+clean; 998 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.06k op/s; 4464/47889 objects degraded (9.322%) 2024-08-30T23:57:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:57 smithi173 ceph-mon[20554]: osdmap e952: 12 total, 11 up, 12 in 2024-08-30T23:57:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:57 smithi173 ceph-mon[20554]: 9.9 scrub ok 2024-08-30T23:57:57.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:57.641 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:57.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:57.771 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:57.799 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:57:57.800 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:57:57.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:57.896 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:57.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:57.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 15 16 21606 21590 89.9502 72.5625 0.011742 0.0111095 2024-08-30T23:57:57.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:58.029 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:58.096 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:57:58.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:58.158 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:58.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:57:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:57:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:58 smithi106 ceph-mon[16603]: 9.11 scrub starts 2024-08-30T23:57:58.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:58 smithi106 ceph-mon[16603]: 9.11 scrub ok 2024-08-30T23:57:58.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/592958444' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:58.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:58 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:58.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:57:58.421 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.422 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:57:58.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:58 smithi136 ceph-mon[20207]: 9.11 scrub starts 2024-08-30T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:58 smithi136 ceph-mon[20207]: 9.11 scrub ok 2024-08-30T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/592958444' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:58 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:58.543 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2114,"stamp":"2024-08-30T23:57:56.087791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1046716253,"num_objects":15963,"num_object_clones":0,"num_object_copies":47889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15963,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":15375,"num_write_kb":983698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25259,"ondisk_log_size":25259,"up":138,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11228036,"kb_used_data":3580660,"kb_used_omap":109,"kb_used_meta":7646994,"kb_avail":1113713788,"statfs":{"total":1151940427776,"available":1140442918912,"internally_reserved":0,"allocated":3666595840,"data_stored":3657712814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":7830522520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":277938176,"num_objects":4240,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":233408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.023747"},"pg_stats":[{"pgid":"9.17","version":"947'1072","reported_seq":1100,"reported_epoch":947,"state":"unknown","last_fresh":"2024-08-30T23:57:52.060959+0000","last_change":"2024-08-30T23:57:52.142562+0000","last_active":"2024-08-30T23:57:52.142562+0000","last_peered":"2024-08-30T23:57:52.060959+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.056134+0000","last_became_peered":"2024-08-30T23:57:36.056134+0000","last_unstale":"2024-08-30T23:57:52.060959+0000","last_undegraded":"2024-08-30T23:57:43.003836+0000","last_fullsized":"2024-08-30T23:57:36.054771+0000","mapping_epoch":948,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":536,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:57:52.055408+0000","last_change":"2024-08-30T23:57:52.055408+0000","last_active":"2024-08-30T23:57:52.055408+0000","last_peered":"2024-08-30T23:57:52.055408+0000","last_clean":"2024-08-30T23:57:52.055408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:57:52.055408+0000","last_undegraded":"2024-08-30T23:57:52.055408+0000","last_fullsized":"2024-08-30T23:57:52.055408+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.055408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":159,"reported_epoch":947,"state":"active+undersized","last_fresh":"2024-08-30T23:57:50.930686+0000","last_change":"2024-08-30T23:55:59.172632+0000","last_active":"2024-08-30T23:57:50.930686+0000","last_peered":"2024-08-30T23:57:50.930686+0000","last_clean":"2024-08-30T23:55:57.160064+0000","last_became_active":"2024-08-30T23:55:59.172632+0000","last_became_peered":"2024-08-30T23:55:59.172632+0000","last_unstale":"2024-08-30T23:57:50.930686+0000","last_undegraded":"2024-08-30T23:57:50.930686+0000","last_fullsized":"2024-08-30T23:55:59.167996+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.344869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.344869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:54.199567+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:57:52.055408+0000","last_change":"2024-08-30T23:57:52.055408+0000","last_active":"2024-08-30T23:57:52.055408+0000","last_peered":"2024-08-30T23:57:52.055408+0000","last_clean":"2024-08-30T23:57:52.055408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:57:52.055408+0000","last_undegraded":"2024-08-30T23:57:52.055408+0000","last_fullsized":"2024-08-30T23:57:52.055408+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.055408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":163,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-08-30T23:57:50.931190+0000","last_change":"2024-08-30T23:56:46.421822+0000","last_active":"2024-08-30T23:57:50.931190+0000","last_peered":"2024-08-30T23:57:50.931190+0000","last_clean":"2024-08-30T23:57:50.931190+0000","last_became_active":"2024-08-30T23:55:03.321559+0000","last_became_peered":"2024-08-30T23:55:03.321559+0000","last_unstale":"2024-08-30T23:57:50.931190+0000","last_undegraded":"2024-08-30T23:57:50.931190+0000","last_fullsized":"2024-08-30T23:57:50.931190+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:46.421676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:46.421676+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:16.405067+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:57:52.055408+0000","last_change":"2024-08-30T23:57:52.055408+0000","last_active":"2024-08-30T23:57:52.055408+0000","last_peered":"2024-08-30T23:57:52.055408+0000","last_clean":"2024-08-30T23:57:52.055408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:57:52.055408+0000","last_undegraded":"2024-08-30T23:57:52.055408+0000","last_fullsized":"2024-08-30T23:57:52.055408+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.055408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":179,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-08-30T23:57:55.074302+0000","last_change":"2024-08-30T23:57:53.388159+0000","last_active":"2024-08-30T23:57:55.074302+0000","last_peered":"2024-08-30T23:57:55.074302+0000","last_clean":"2024-08-30T23:57:55.074302+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-30T23:57:55.074302+0000","last_undegraded":"2024-08-30T23:57:55.074302+0000","last_fullsized":"2024-08-30T23:57:55.074302+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:53.388110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.388110+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:03.016237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:57:52.055408+0000","last_change":"2024-08-30T23:57:52.055408+0000","last_active":"2024-08-30T23:57:52.055408+0000","last_peered":"2024-08-30T23:57:52.055408+0000","last_clean":"2024-08-30T23:57:52.055408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:57:52.055408+0000","last_undegraded":"2024-08-30T23:57:52.055408+0000","last_fullsized":"2024-08-30T23:57:52.055408+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.055408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":179,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-08-30T23:57:55.074330+0000","last_change":"2024-08-30T23:57:32.122253+0000","last_active":"2024-08-30T23:57:55.074330+0000","last_peered":"2024-08-30T23:57:55.074330+0000","last_clean":"2024-08-30T23:57:55.074330+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-30T23:57:55.074330+0000","last_undegraded":"2024-08-30T23:57:55.074330+0000","last_fullsized":"2024-08-30T23:57:55.074330+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.122186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.122186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.813541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":950,"state":"unknown","last_fresh":"2024-08-30T23:57:54.079515+0000","last_change":"2024-08-30T23:57:54.796749+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:57:54.079515+0000","last_undegraded":"2024-08-30T23:57:54.079515+0000","last_fullsized":"2024-08-30T23:57:54.079515+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":161,"reported_epoch":947,"state":"active+undersized","last_fresh":"2024-08-30T23:57:50.920658+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-30T23:57:50.920658+0000","last_peered":"2024-08-30T23:57:50.920658+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-30T23:57:50.920658+0000","last_undegraded":"2024-08-30T23:57:50.920658+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:52.231227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"949'1106","reported_seq":1148,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-08-30T23:57:53.939629+0000","last_change":"2024-08-30T23:57:53.079552+0000","last_active":"2024-08-30T23:57:53.939629+0000","last_peered":"2024-08-30T23:57:53.939629+0000","last_clean":"2024-08-30T23:57:53.939629+0000","last_became_active":"2024-08-30T23:57:53.079018+0000","last_became_peered":"2024-08-30T23:57:53.079018+0000","last_unstale":"2024-08-30T23:57:53.939629+0000","last_undegraded":"2024-08-30T23:57:53.939629+0000","last_fullsized":"2024-08-30T23:57:53.939629+0000","mapping_epoch":948,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":579,"num_write_kb":37056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":165,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.067971+0000","last_change":"2024-08-30T23:56:45.191296+0000","last_active":"2024-08-30T23:57:52.067971+0000","last_peered":"2024-08-30T23:57:52.067971+0000","last_clean":"2024-08-30T23:57:52.067971+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-30T23:57:52.067971+0000","last_undegraded":"2024-08-30T23:57:52.067971+0000","last_fullsized":"2024-08-30T23:57:52.067971+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.191245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.191245+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:55.010360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"947'1096","reported_seq":1125,"reported_epoch":947,"state":"unknown","last_fresh":"2024-08-30T23:57:52.061372+0000","last_change":"2024-08-30T23:57:52.328135+0000","last_active":"2024-08-30T23:57:52.328135+0000","last_peered":"2024-08-30T23:57:52.061372+0000","last_clean":"2024-08-30T23:57:52.061372+0000","last_became_active":"2024-08-30T23:57:36.060760+0000","last_became_peered":"2024-08-30T23:57:36.060760+0000","last_unstale":"2024-08-30T23:57:52.061372+0000","last_undegraded":"2024-08-30T23:57:52.061372+0000","last_fullsized":"2024-08-30T23:57:52.061372+0000","mapping_epoch":948,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1096,"log_dups_size":0,"ondisk_log_size":1096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":390,"reported_epoch":950,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:54.076898+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-30T23:57:54.076898+0000","last_peered":"2024-08-30T23:57:54.076898+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-30T23:57:54.076898+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:54.113307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":171,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-08-30T23:57:55.074522+0000","last_change":"2024-08-30T23:56:40.272914+0000","last_active":"2024-08-30T23:57:55.074522+0000","last_peered":"2024-08-30T23:57:55.074522+0000","last_clean":"2024-08-30T23:57:55.074522+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-30T23:57:55.074522+0000","last_undegraded":"2024-08-30T23:57:55.074522+0000","last_fullsized":"2024-08-30T23:57:55.074522+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:40.272848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:40.272848+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:09.954699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":171,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-08-30T23:57:50.928157+0000","last_change":"2024-08-30T23:57:27.164513+0000","last_active":"2024-08-30T23:57:50.928157+0000","last_peered":"2024-08-30T23:57:50.928157+0000","last_clean":"2024-08-30T23:57:50.928157+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-30T23:57:50.928157+0000","last_undegraded":"2024-08-30T23:57:50.928157+0000","last_fullsized":"2024-08-30T23:57:50.928157+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:27.164462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:27.164462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:47.956715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":104,"reported_epoch":950,"state":"active+undersized","last_fresh":"2024-08-30T23:57:54.078532+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-30T23:57:54.078532+0000","last_peered":"2024-08-30T23:57:54.078532+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-30T23:57:54.078532+0000","last_undegraded":"2024-08-30T23:57:54.078532+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:57:58.089296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":218,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-08-30T23:57:50.930649+0000","last_change":"2024-08-30T23:57:14.423376+0000","last_active":"2024-08-30T23:57:50.930649+0000","last_peered":"2024-08-30T23:57:50.930649+0000","last_clean":"2024-08-30T23:57:50.930649+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:57:50.930649+0000","last_undegraded":"2024-08-30T23:57:50.930649+0000","last_fullsized":"2024-08-30T23:57:50.930649+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:14.423316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:14.423316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.319991+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":373,"reported_epoch":950,"state":"active+clean","last_fresh":"2024-08-30T23:57:54.078883+0000","last_change":"2024-08-30T23:57:35.093839+0000","last_active":"2024-08-30T23:57:54.078883+0000","last_peered":"2024-08-30T23:57:54.078883+0000","last_clean":"2024-08-30T23:57:54.078883+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:57:54.078883+0000","last_undegraded":"2024-08-30T23:57:54.078883+0000","last_fullsized":"2024-08-30T23:57:54.078883+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:35.093787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.093787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:01.863839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"947'1076","reported_seq":1106,"reported_epoch":947,"state":"unknown","last_fresh":"2024-08-30T23:57:52.067334+0000","last_change":"2024-08-30T23:57:52.142532+0000","last_active":"2024-08-30T23:57:52.142532+0000","last_peered":"2024-08-30T23:57:52.067334+0000","last_clean":"2024-08-30T23:57:52.067334+0000","last_became_active":"2024-08-30T23:57:36.064599+0000","last_became_peered":"2024-08-30T23:57:36.064599+0000","last_unstale":"2024-08-30T23:57:52.067334+0000","last_undegraded":"2024-08-30T23:57:52.067334+0000","last_fullsized":"2024-08-30T23:57:52.067334+0000","mapping_epoch":948,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":538,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":987,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-08-30T23:57:55.074001+0000","last_change":"2024-08-30T23:56:39.308813+0000","last_active":"2024-08-30T23:57:55.074001+0000","last_peered":"2024-08-30T23:57:55.074001+0000","last_clean":"2024-08-30T23:57:55.074001+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:57:55.074001+0000","last_undegraded":"2024-08-30T23:57:55.074001+0000","last_fullsized":"2024-08-30T23:57:55.074001+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:39.308738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:39.308738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:58.601526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-30T23:57:52.055408+0000","last_change":"2024-08-30T23:57:52.055408+0000","last_active":"2024-08-30T23:57:52.055408+0000","last_peered":"2024-08-30T23:57:52.055408+0000","last_clean":"2024-08-30T23:57:52.055408+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-30T23:57:52.055408+0000","last_undegraded":"2024-08-30T23:57:52.055408+0000","last_fullsized":"2024-08-30T23:57:52.055408+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:52.055408+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.055408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":361,"reported_epoch":950,"state":"active+undersized","last_fresh":"2024-08-30T23:57:54.078708+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-30T23:57:54.078708+0000","last_peered":"2024-08-30T23:57:54.078708+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-30T23:57:54.078708+0000","last_undegraded":"2024-08-30T23:57:54.078708+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:55.121805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"949'1156","reported_seq":1200,"reported_epoch":950,"state":"active+clean","last_fresh":"2024-08-30T23:57:54.079213+0000","last_change":"2024-08-30T23:57:53.077990+0000","last_active":"2024-08-30T23:57:54.079213+0000","last_peered":"2024-08-30T23:57:54.079213+0000","last_clean":"2024-08-30T23:57:54.079213+0000","last_became_active":"2024-08-30T23:57:53.076655+0000","last_became_peered":"2024-08-30T23:57:53.076655+0000","last_unstale":"2024-08-30T23:57:54.079213+0000","last_undegraded":"2024-08-30T23:57:54.079213+0000","last_fullsized":"2024-08-30T23:57:54.079213+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1156,"log_dups_size":0,"ondisk_log_size":1156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39354368,"num_objects":600,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":600,"num_write_kb":38432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":772,"reported_epoch":948,"state":"active+undersized","last_fresh":"2024-08-30T23:57:52.077078+0000","last_change":"2024-08-30T23:55:59.173538+0000","last_active":"2024-08-30T23:57:52.077078+0000","last_peered":"2024-08-30T23:57:52.077078+0000","last_clean":"2024-08-30T23:55:57.750202+0000","last_became_active":"2024-08-30T23:55:59.173538+0000","last_became_peered":"2024-08-30T23:55:59.173538+0000","last_unstale":"2024-08-30T23:57:52.077078+0000","last_undegraded":"2024-08-30T23:57:52.077078+0000","last_fullsized":"2024-08-30T23:55:59.169546+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:53.668489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:53.668489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:54.840254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"949'1192","reported_seq":1224,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-08-30T23:57:53.920614+0000","last_change":"2024-08-30T23:57:36.062999+0000","last_active":"2024-08-30T23:57:53.920614+0000","last_peered":"2024-08-30T23:57:53.920614+0000","last_clean":"2024-08-30T23:57:53.920614+0000","last_became_active":"2024-08-30T23:57:36.062695+0000","last_became_peered":"2024-08-30T23:57:36.062695+0000","last_unstale":"2024-08-30T23:57:53.920614+0000","last_undegraded":"2024-08-30T23:57:53.920614+0000","last_fullsized":"2024-08-30T23:57:53.920614+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1192,"log_dups_size":0,"ondisk_log_size":1192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:41.939616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78118912,"num_objects":1192,"num_object_clones":0,"num_object_copies":3576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1192,"num_write_kb":76288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":222,"reported_epoch":947,"state":"active+undersized","last_fresh":"2024-08-30T23:57:50.930649+0000","last_change":"2024-08-30T23:55:59.171246+0000","last_active":"2024-08-30T23:57:50.930649+0000","last_peered":"2024-08-30T23:57:50.930649+0000","last_clean":"2024-08-30T23:55:57.160009+0000","last_became_active":"2024-08-30T23:55:59.171246+0000","last_became_peered":"2024-08-30T23:55:59.171246+0000","last_unstale":"2024-08-30T23:57:50.930649+0000","last_undegraded":"2024-08-30T23:57:50.930649+0000","last_fullsized":"2024-08-30T23:55:59.168006+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:36.270718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:36.270718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"950'1157","reported_seq":1192,"reported_epoch":950,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:54.087576+0000","last_change":"2024-08-30T23:57:42.988451+0000","last_active":"2024-08-30T23:57:54.087576+0000","last_peered":"2024-08-30T23:57:54.087576+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.059533+0000","last_became_peered":"2024-08-30T23:57:36.059533+0000","last_unstale":"2024-08-30T23:57:54.087576+0000","last_undegraded":"2024-08-30T23:57:42.988196+0000","last_fullsized":"2024-08-30T23:57:36.057835+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:03.589784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75825152,"num_objects":1157,"num_object_clones":0,"num_object_copies":3471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1157,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1157}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":566,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.068036+0000","last_change":"2024-08-30T23:56:48.173739+0000","last_active":"2024-08-30T23:57:52.068036+0000","last_peered":"2024-08-30T23:57:52.068036+0000","last_clean":"2024-08-30T23:57:52.068036+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-30T23:57:52.068036+0000","last_undegraded":"2024-08-30T23:57:52.068036+0000","last_fullsized":"2024-08-30T23:57:52.068036+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:48.173668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:48.173668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:13.591624+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"947'1092","reported_seq":1122,"reported_epoch":948,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:52.068372+0000","last_change":"2024-08-30T23:57:43.002149+0000","last_active":"2024-08-30T23:57:52.068372+0000","last_peered":"2024-08-30T23:57:52.068372+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.061016+0000","last_became_peered":"2024-08-30T23:57:36.061016+0000","last_unstale":"2024-08-30T23:57:52.068372+0000","last_undegraded":"2024-08-30T23:57:42.987440+0000","last_fullsized":"2024-08-30T23:57:36.054773+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1092,"log_dups_size":0,"ondisk_log_size":1092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:38.514908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71565312,"num_objects":1092,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1092,"num_write_kb":69888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":["10","1"],"object_location_counts":[{"shards":"1,10","objects":1092}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":540,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.068612+0000","last_change":"2024-08-30T23:56:16.234435+0000","last_active":"2024-08-30T23:57:52.068612+0000","last_peered":"2024-08-30T23:57:52.068612+0000","last_clean":"2024-08-30T23:57:52.068612+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-30T23:57:52.068612+0000","last_undegraded":"2024-08-30T23:57:52.068612+0000","last_fullsized":"2024-08-30T23:57:52.068612+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.234371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.234371+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:54.050780+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"950'1246","reported_seq":1284,"reported_epoch":950,"state":"active+clean","last_fresh":"2024-08-30T23:57:54.078414+0000","last_change":"2024-08-30T23:57:36.063702+0000","last_active":"2024-08-30T23:57:54.078414+0000","last_peered":"2024-08-30T23:57:54.078414+0000","last_clean":"2024-08-30T23:57:54.078414+0000","last_became_active":"2024-08-30T23:57:36.063425+0000","last_became_peered":"2024-08-30T23:57:36.063425+0000","last_unstale":"2024-08-30T23:57:54.078414+0000","last_undegraded":"2024-08-30T23:57:54.078414+0000","last_fullsized":"2024-08-30T23:57:54.078414+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1246,"log_dups_size":0,"ondisk_log_size":1246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:00.507197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81657856,"num_objects":1246,"num_object_clones":0,"num_object_copies":3738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1246,"num_write_kb":79744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1179,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.068545+0000","last_change":"2024-08-30T23:56:20.789107+0000","last_active":"2024-08-30T23:57:52.068545+0000","last_peered":"2024-08-30T23:57:52.068545+0000","last_clean":"2024-08-30T23:57:52.068545+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-30T23:57:52.068545+0000","last_undegraded":"2024-08-30T23:57:52.068545+0000","last_fullsized":"2024-08-30T23:57:52.068545+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:20.789023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:20.789023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"947'1076","reported_seq":1107,"reported_epoch":948,"state":"peering","last_fresh":"2024-08-30T23:57:52.080709+0000","last_change":"2024-08-30T23:57:52.080709+0000","last_active":"2024-08-30T23:57:52.080709+0000","last_peered":"2024-08-30T23:57:52.067334+0000","last_clean":"2024-08-30T23:57:52.067334+0000","last_became_active":"2024-08-30T23:57:36.064599+0000","last_became_peered":"2024-08-30T23:57:36.064599+0000","last_unstale":"2024-08-30T23:57:52.080709+0000","last_undegraded":"2024-08-30T23:57:52.080709+0000","last_fullsized":"2024-08-30T23:57:52.080709+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":538,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,11],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":525,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.068106+0000","last_change":"2024-08-30T23:57:08.049165+0000","last_active":"2024-08-30T23:57:52.068106+0000","last_peered":"2024-08-30T23:57:52.068106+0000","last_clean":"2024-08-30T23:57:52.068106+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:57:52.068106+0000","last_undegraded":"2024-08-30T23:57:52.068106+0000","last_fullsized":"2024-08-30T23:57:52.068106+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:08.049028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:08.049028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:28.561467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"950'1216","reported_seq":1251,"reported_epoch":950,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:54.076939+0000","last_change":"2024-08-30T23:57:43.007699+0000","last_active":"2024-08-30T23:57:54.076939+0000","last_peered":"2024-08-30T23:57:54.076939+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.059522+0000","last_became_peered":"2024-08-30T23:57:36.059522+0000","last_unstale":"2024-08-30T23:57:54.076939+0000","last_undegraded":"2024-08-30T23:57:43.002973+0000","last_fullsized":"2024-08-30T23:57:36.057850+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1216,"log_dups_size":0,"ondisk_log_size":1216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:43.386225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":1216,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1216,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1216}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":2010,"reported_epoch":947,"state":"active+undersized","last_fresh":"2024-08-30T23:57:50.920664+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-30T23:57:50.920664+0000","last_peered":"2024-08-30T23:57:50.920664+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-30T23:57:50.920664+0000","last_undegraded":"2024-08-30T23:57:50.920664+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:54.156759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"950'1261","reported_seq":1301,"reported_epoch":951,"state":"active+clean","last_fresh":"2024-08-30T23:57:55.074209+0000","last_change":"2024-08-30T23:57:36.061979+0000","last_active":"2024-08-30T23:57:55.074209+0000","last_peered":"2024-08-30T23:57:55.074209+0000","last_clean":"2024-08-30T23:57:55.074209+0000","last_became_active":"2024-08-30T23:57:36.061597+0000","last_became_peered":"2024-08-30T23:57:36.061597+0000","last_unstale":"2024-08-30T23:57:55.074209+0000","last_undegraded":"2024-08-30T23:57:55.074209+0000","last_fullsized":"2024-08-30T23:57:55.074209+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:04.889278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":80704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":344,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.068427+0000","last_change":"2024-08-30T23:57:34.913534+0000","last_active":"2024-08-30T23:57:52.068427+0000","last_peered":"2024-08-30T23:57:52.068427+0000","last_clean":"2024-08-30T23:57:52.068427+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:57:52.068427+0000","last_undegraded":"2024-08-30T23:57:52.068427+0000","last_fullsized":"2024-08-30T23:57:52.068427+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:34.913482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:34.913482+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.935529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"947'1020","reported_seq":1050,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-08-30T23:57:51.861749+0000","last_change":"2024-08-30T23:57:36.060551+0000","last_active":"2024-08-30T23:57:51.861749+0000","last_peered":"2024-08-30T23:57:51.861749+0000","last_clean":"2024-08-30T23:57:51.861749+0000","last_became_active":"2024-08-30T23:57:36.060190+0000","last_became_peered":"2024-08-30T23:57:36.060190+0000","last_unstale":"2024-08-30T23:57:51.861749+0000","last_undegraded":"2024-08-30T23:57:51.861749+0000","last_fullsized":"2024-08-30T23:57:51.861749+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:53.930981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":1008,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-08-30T23:57:53.066444+0000","last_change":"2024-08-30T23:57:32.972014+0000","last_active":"2024-08-30T23:57:53.066444+0000","last_peered":"2024-08-30T23:57:53.066444+0000","last_clean":"2024-08-30T23:57:53.066444+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-30T23:57:53.066444+0000","last_undegraded":"2024-08-30T23:57:53.066444+0000","last_fullsized":"2024-08-30T23:57:53.066444+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.971963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.971963+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:59.480355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"947'998","reported_seq":1024,"reported_epoch":947,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:57:51.325030+0000","last_change":"2024-08-30T23:57:43.005762+0000","last_active":"2024-08-30T23:57:51.325030+0000","last_peered":"2024-08-30T23:57:51.325030+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.063471+0000","last_became_peered":"2024-08-30T23:57:36.063471+0000","last_unstale":"2024-08-30T23:57:51.325030+0000","last_undegraded":"2024-08-30T23:57:43.005511+0000","last_fullsized":"2024-08-30T23:57:36.061798+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":998,"log_dups_size":0,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:50.837092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65404928,"num_objects":998,"num_object_clones":0,"num_object_copies":2994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":998,"num_write_kb":63872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"5,11","objects":998}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":980,"reported_epoch":947,"state":"active+undersized","last_fresh":"2024-08-30T23:57:50.920742+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-30T23:57:50.920742+0000","last_peered":"2024-08-30T23:57:50.920742+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-30T23:57:50.920742+0000","last_undegraded":"2024-08-30T23:57:50.920742+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"950'1118","reported_seq":1167,"reported_epoch":950,"state":"active+clean+scrubbing","last_fresh":"2024-08-30T23:57:54.633032+0000","last_change":"2024-08-30T23:57:54.067814+0000","last_active":"2024-08-30T23:57:54.633032+0000","last_peered":"2024-08-30T23:57:54.633032+0000","last_clean":"2024-08-30T23:57:54.633032+0000","last_became_active":"2024-08-30T23:57:53.081131+0000","last_became_peered":"2024-08-30T23:57:53.081131+0000","last_unstale":"2024-08-30T23:57:54.633032+0000","last_undegraded":"2024-08-30T23:57:54.633032+0000","last_fullsized":"2024-08-30T23:57:54.633032+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":199,"log_size":1118,"log_dups_size":0,"ondisk_log_size":1118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38404096,"num_objects":586,"num_object_clones":0,"num_object_copies":1758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":586,"num_write_kb":37504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":911,"reported_epoch":947,"state":"active+undersized","last_fresh":"2024-08-30T23:57:50.920820+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-30T23:57:50.920820+0000","last_peered":"2024-08-30T23:57:50.920820+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-30T23:57:50.920820+0000","last_undegraded":"2024-08-30T23:57:50.920820+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:57:51.195105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"947'1072","reported_seq":1101,"reported_epoch":948,"state":"peering","last_fresh":"2024-08-30T23:57:52.081220+0000","last_change":"2024-08-30T23:57:52.081220+0000","last_active":"2024-08-30T23:57:52.081220+0000","last_peered":"2024-08-30T23:57:52.060959+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.056134+0000","last_became_peered":"2024-08-30T23:57:36.056134+0000","last_unstale":"2024-08-30T23:57:52.081220+0000","last_undegraded":"2024-08-30T23:57:52.081220+0000","last_fullsized":"2024-08-30T23:57:52.081220+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":536,"num_write_kb":34304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1190,"reported_epoch":948,"state":"active+clean","last_fresh":"2024-08-30T23:57:52.068425+0000","last_change":"2024-08-30T23:57:36.159072+0000","last_active":"2024-08-30T23:57:52.068425+0000","last_peered":"2024-08-30T23:57:52.068425+0000","last_clean":"2024-08-30T23:57:52.068425+0000","last_became_active":"2024-08-30T23:57:36.158625+0000","last_became_peered":"2024-08-30T23:57:36.158625+0000","last_unstale":"2024-08-30T23:57:52.068425+0000","last_undegraded":"2024-08-30T23:57:52.068425+0000","last_fullsized":"2024-08-30T23:57:52.068425+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.888321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.888321+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"947'1096","reported_seq":1126,"reported_epoch":948,"state":"peering","last_fresh":"2024-08-30T23:57:52.081473+0000","last_change":"2024-08-30T23:57:52.081473+0000","last_active":"2024-08-30T23:57:52.081473+0000","last_peered":"2024-08-30T23:57:52.061372+0000","last_clean":"2024-08-30T23:57:52.061372+0000","last_became_active":"2024-08-30T23:57:36.060760+0000","last_became_peered":"2024-08-30T23:57:36.060760+0000","last_unstale":"2024-08-30T23:57:52.081473+0000","last_undegraded":"2024-08-30T23:57:52.081473+0000","last_fullsized":"2024-08-30T23:57:52.081473+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1096,"log_dups_size":0,"ondisk_log_size":1096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,11],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":383,"reported_epoch":950,"state":"active+clean","last_fresh":"2024-08-30T23:57:54.076807+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-30T23:57:54.076807+0000","last_peered":"2024-08-30T23:57:54.076807+0000","last_clean":"2024-08-30T23:57:54.076807+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-30T23:57:54.076807+0000","last_undegraded":"2024-08-30T23:57:54.076807+0000","last_fullsized":"2024-08-30T23:57:54.076807+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"949'1099","reported_seq":1141,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-08-30T23:57:53.909678+0000","last_change":"2024-08-30T23:57:53.073871+0000","last_active":"2024-08-30T23:57:53.909678+0000","last_peered":"2024-08-30T23:57:53.909678+0000","last_clean":"2024-08-30T23:57:53.909678+0000","last_became_active":"2024-08-30T23:57:53.072273+0000","last_became_peered":"2024-08-30T23:57:53.072273+0000","last_unstale":"2024-08-30T23:57:53.909678+0000","last_undegraded":"2024-08-30T23:57:53.909678+0000","last_fullsized":"2024-08-30T23:57:53.909678+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37486592,"num_objects":572,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":572,"num_write_kb":36608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":980,"reported_epoch":950,"state":"active+undersized","last_fresh":"2024-08-30T23:57:54.079486+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-30T23:57:54.079486+0000","last_peered":"2024-08-30T23:57:54.079486+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-30T23:57:54.079486+0000","last_undegraded":"2024-08-30T23:57:54.079486+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:57:54.609639+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"949'1107","reported_seq":1149,"reported_epoch":949,"state":"active+clean","last_fresh":"2024-08-30T23:57:53.900450+0000","last_change":"2024-08-30T23:57:53.072500+0000","last_active":"2024-08-30T23:57:53.900450+0000","last_peered":"2024-08-30T23:57:53.900450+0000","last_clean":"2024-08-30T23:57:53.900450+0000","last_became_active":"2024-08-30T23:57:53.070711+0000","last_became_peered":"2024-08-30T23:57:53.070711+0000","last_unstale":"2024-08-30T23:57:53.900450+0000","last_undegraded":"2024-08-30T23:57:53.900450+0000","last_fullsized":"2024-08-30T23:57:53.900450+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1107,"log_dups_size":0,"ondisk_log_size":1107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37978112,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":579,"num_write_kb":37088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1118,"reported_epoch":950,"state":"active+clean","last_fresh":"2024-08-30T23:57:54.079819+0000","last_change":"2024-08-30T23:56:19.712394+0000","last_active":"2024-08-30T23:57:54.079819+0000","last_peered":"2024-08-30T23:57:54.079819+0000","last_clean":"2024-08-30T23:57:54.079819+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-30T23:57:54.079819+0000","last_undegraded":"2024-08-30T23:57:54.079819+0000","last_fullsized":"2024-08-30T23:57:54.079819+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:19.712327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:19.712327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"949'1151","reported_seq":1210,"reported_epoch":950,"state":"active+clean","last_fresh":"2024-08-30T23:57:54.078689+0000","last_change":"2024-08-30T23:57:53.737093+0000","last_active":"2024-08-30T23:57:54.078689+0000","last_peered":"2024-08-30T23:57:54.078689+0000","last_clean":"2024-08-30T23:57:54.078689+0000","last_became_active":"2024-08-30T23:57:53.074222+0000","last_became_peered":"2024-08-30T23:57:53.074222+0000","last_unstale":"2024-08-30T23:57:54.078689+0000","last_undegraded":"2024-08-30T23:57:54.078689+0000","last_fullsized":"2024-08-30T23:57:54.078689+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1132","last_scrub_stamp":"2024-08-30T23:57:53.736917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.736917+0000","objects_scrubbed":594,"log_size":1151,"log_dups_size":0,"ondisk_log_size":1151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:14.720979+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":619,"num_object_clones":0,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":224,"reported_epoch":947,"state":"active+clean","last_fresh":"2024-08-30T23:57:50.930709+0000","last_change":"2024-08-30T23:56:13.242208+0000","last_active":"2024-08-30T23:57:50.930709+0000","last_peered":"2024-08-30T23:57:50.930709+0000","last_clean":"2024-08-30T23:57:50.930709+0000","last_became_active":"2024-08-30T23:55:01.289376+0000","last_became_peered":"2024-08-30T23:55:01.289376+0000","last_unstale":"2024-08-30T23:57:50.930709+0000","last_undegraded":"2024-08-30T23:57:50.930709+0000","last_fullsized":"2024-08-30T23:57:50.930709+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:13.242035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:13.242035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1073,"reported_epoch":950,"state":"active+undersized","last_fresh":"2024-08-30T23:57:54.078820+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-30T23:57:54.078820+0000","last_peered":"2024-08-30T23:57:54.078820+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-30T23:57:54.078820+0000","last_undegraded":"2024-08-30T23:57:54.078820+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":1046085632,"num_objects":15961,"num_object_clones":0,"num_object_copies":47883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15367,"num_write_kb":983552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3168337920,"data_stored":3168337920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22407,"ondisk_log_size":22407,"up":57,"acting":57,"num_store_stats":9},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":81,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907516,"kb_used_data":244644,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92837636,"statfs":{"total":95995035648,"available":95065739264,"internally_reserved":0,"allocated":250515456,"data_stored":249784437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":678754756},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449498,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993336,"kb_used_data":344356,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92751816,"statfs":{"total":95995035648,"available":94977859584,"internally_reserved":0,"allocated":352620544,"data_stored":351889525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711127,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202136,"kb_used_data":440700,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92543016,"statfs":{"total":95995035648,"available":94764048384,"internally_reserved":0,"allocated":451276800,"data_stored":450537337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972770,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":657776,"kb_used_data":157332,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93087376,"statfs":{"total":95995035648,"available":95321473024,"internally_reserved":0,"allocated":161107968,"data_stored":160360591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528747,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626360,"kb_used_data":135652,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93118792,"statfs":{"total":95995035648,"available":95353643008,"internally_reserved":0,"allocated":138907648,"data_stored":138176629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775251,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1001944,"kb_used_data":699780,"kb_used_omap":5,"kb_used_meta":302138,"kb_avail":92743208,"statfs":{"total":95995035648,"available":94969044992,"internally_reserved":0,"allocated":716574720,"data_stored":715831705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":309389386},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":819260,"kb_used_data":120292,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92925892,"statfs":{"total":95995035648,"available":95156113408,"internally_reserved":0,"allocated":123179008,"data_stored":122447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150136,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1233280,"kb_used_data":345520,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92511872,"statfs":{"total":95995035648,"available":94732156928,"internally_reserved":0,"allocated":353812480,"data_stored":353077410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411774,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660876,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":93084276,"statfs":{"total":95995035648,"available":95318298624,"internally_reserved":0,"allocated":2220032,"data_stored":1481410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033724,"kb_used_data":508772,"kb_used_omap":87,"kb_used_meta":524840,"kb_avail":92711428,"statfs":{"total":95995035648,"available":94936502272,"internally_reserved":0,"allocated":520982528,"data_stored":520251509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":537436319},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":787348,"kb_used_data":2300,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92957804,"statfs":{"total":95995035648,"available":95188791296,"internally_reserved":0,"allocated":2355200,"data_stored":1615737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":549519360,"data_stored":549519360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":384565248,"data_stored":384565248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":133496832,"data_stored":133496832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":715587584,"data_stored":715587584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140902400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":159055872,"data_stored":159055872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":473235456,"data_stored":473235456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":355860480,"data_stored":355860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":256114688,"data_stored":256114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:57:58.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-08-30T23:57:58.551 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:57:58.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:58 smithi173 ceph-mon[20554]: 9.11 scrub starts 2024-08-30T23:57:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:58 smithi173 ceph-mon[20554]: 9.11 scrub ok 2024-08-30T23:57:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/592958444' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:58 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:57:58.654 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:57:58.681 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:57:58.713 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:57:58.713 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:57:58.713 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:57:58.713 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:57:58.713 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.713 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.9 760/755 [PushOp(7:92ccacc7:::benchmark_data_smithi173_129296_object5107:head, version: 730'310, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:92ccacc7:::benchmark_data_smithi173_129296_object5107:head@730'310, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 454.64814773900002, 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93410173799999996, 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.714 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061773+0000", 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061775+0000", 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4950000000000001e-06 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.061824+0000", 2024-08-30T23:57:58.715 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9017000000000002e-05 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993785+0000", 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.93196087299999997 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993806+0000", 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1152e-05 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.716 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993912+0000", 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000106251 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.993928+0000", 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6387000000000001e-05 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:24.995875+0000", 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019465629999999999 2024-08-30T23:57:58.717 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 765/762 [PushOp(7:6c4bd07a:::benchmark_data_smithi173_129296_object3500:head, version: 730'216, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6c4bd07a:::benchmark_data_smithi173_129296_object3500:head@730'216, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 448.23635461399999, 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62968416199999999, 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.718 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473566+0000", 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473567+0000", 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.6799999999999999e-07 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.719 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.473595+0000", 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7571000000000001e-05 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101453+0000", 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.62785843200000002 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101470+0000", 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7164e-05 2024-08-30T23:57:58.720 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101579+0000", 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000109252 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.101595+0000", 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5449000000000001e-05 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:31.103250+0000", 2024-08-30T23:57:58.721 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001655526 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 767/762 [PushOp(7:2ff57e78:::benchmark_data_smithi173_129296_object4738:head, version: 730'300, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2ff57e78:::benchmark_data_smithi173_129296_object4738:head@730'300, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 444.96529379399999, 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.95161098399999999, 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.722 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744627+0000", 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.723 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744628+0000", 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1999999999999999e-07 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.744662+0000", 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3967000000000002e-05 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694267+0000", 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.94960559300000003 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.724 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694286+0000", 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.844e-05 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694373+0000", 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7074e-05 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.725 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.694383+0000", 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7389999999999994e-06 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.696238+0000", 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018554509999999999 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 767/762 [PushOp(7:603ba3a2:::benchmark_data_smithi173_129296_object7838:head, version: 730'501, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:603ba3a2:::benchmark_data_smithi173_129296_object7838:head@730'501, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.726 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 443.10594581300001, 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.99532398200000005, 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.727 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603975+0000", 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.603976+0000", 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.071e-06 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.604007+0000", 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0408000000000001e-05 2024-08-30T23:57:58.728 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597608+0000", 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.99360170000000003 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597625+0000", 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7184999999999999e-05 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597718+0000", 2024-08-30T23:57:58.729 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2985999999999994e-05 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.597729+0000", 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0227999999999999e-05 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:36.599299+0000", 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015704040000000001 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.730 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 768/762 [PushOp(7:6615b48e:::benchmark_data_smithi173_129296_object8839:head, version: 730'559, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6615b48e:::benchmark_data_smithi173_129296_object8839:head@730'559, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 441.248385799, 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63695423500000004, 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.731 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461535+0000", 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461536+0000", 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1519999999999999e-06 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.732 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:37.461566+0000", 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.9569000000000001e-05 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097105+0000", 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63553892199999995 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097118+0000", 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2814e-05 2024-08-30T23:57:58.733 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097203+0000", 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5037999999999995e-05 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.097213+0000", 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9930000000000007e-06 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.098489+0000", 2024-08-30T23:57:58.734 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001276747 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 768/762 [PushOp(7:2ce0b962:::benchmark_data_smithi173_129296_object7093:head, version: 730'443, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2ce0b962:::benchmark_data_smithi173_129296_object7093:head@730'443, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 440.66457964599999, 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.819240623, 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045341+0000", 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045342+0000", 2024-08-30T23:57:58.736 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8800000000000002e-07 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.045405+0000", 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2865000000000002e-05 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.861929+0000", 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.81652422599999996 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.737 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.861952+0000", 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2768999999999998e-05 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.862068+0000", 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115807 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.862081+0000", 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3247e-05 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.738 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:38.864582+0000", 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002501021 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 769/762 [PushOp(7:6b3af24c:::benchmark_data_smithi173_129296_object11749:head, version: 730'750, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:6b3af24c:::benchmark_data_smithi173_129296_object11749:head@730'750, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 437.50238752199999, 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.72094049199999999, 2024-08-30T23:57:58.739 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.740 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207533+0000", 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207535+0000", 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6759999999999999e-06 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.741 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.207575+0000", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0224e-05 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927098+0000", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.71952308300000001 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927109+0000", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0771000000000001e-05 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927189+0000", 2024-08-30T23:57:58.742 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0348999999999999e-05 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.927199+0000", 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3209999999999994e-06 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.928474+0000", 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012750680000000001 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.743 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.6 769/762 [PushOp(7:668cd238:::benchmark_data_smithi173_129296_object13196:head, version: 730'849, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:668cd238:::benchmark_data_smithi173_129296_object13196:head@730'849, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 435.03872788199999, 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96579458500000004, 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.744 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671193+0000", 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671194+0000", 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.297e-06 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:43.671225+0000", 2024-08-30T23:57:58.745 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.0312e-05 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.634913+0000", 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.96368826900000004 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.634929+0000", 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6371999999999998e-05 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.746 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.635041+0000", 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111471 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.635052+0000", 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1195e-05 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:44.636988+0000", 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019356690000000001 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.747 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.748 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.748 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.748 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.f 771/761 [PushOp(7:fed971b1:::benchmark_data_smithi173_129296_object18024:head, version: 733'1120, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:fed971b1:::benchmark_data_smithi173_129296_object18024:head@733'1120, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.748 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:57:58.748 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 431.132464379, 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58954947000000002, 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.749 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577457+0000", 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577458+0000", 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.577571+0000", 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000112908 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.750 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164827+0000", 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.58725667999999998 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164846+0000", 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8417e-05 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164937+0000", 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0884000000000006e-05 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.751 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.164946+0000", 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7380000000000006e-06 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.167006+0000", 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020595629999999999 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.4 771/762 [PushOp(7:241f6387:::benchmark_data_smithi173_129296_object16791:head, version: 731'1054, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:241f6387:::benchmark_data_smithi173_129296_object16791:head@731'1054, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.752 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 431.08378038400002, 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44453464500000001, 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626141+0000", 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.753 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626142+0000", 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3740000000000001e-06 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.626194+0000", 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1793999999999999e-05 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068657+0000", 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44246287299999998 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.754 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068671+0000", 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4783999999999999e-05 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068746+0000", 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4684999999999994e-05 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.068756+0000", 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5750000000000007e-06 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.755 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.070675+0000", 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019195600000000001 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.f 787/761 [PushOp(7:f52af3f4:::benchmark_data_smithi173_129296_object52699:head, version: 743'3327, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f52af3f4:::benchmark_data_smithi173_129296_object52699:head@743'3327, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 376.41554328299998, 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.66126428199999998, 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.756 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294378+0000", 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.757 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294379+0000", 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.28e-06 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.294427+0000", 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8597999999999998e-05 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952418+0000", 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.65799002699999998 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.758 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952437+0000", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9136000000000001e-05 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952574+0000", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013768 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.952590+0000", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5860000000000001e-05 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:42.955642+0000", 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.003051701 2024-08-30T23:57:58.759 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27487.0:142397 7.d 7:bed45d06:::benchmark_data_smithi173_129296_object55281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 321.08300193399998, 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.48225094800000001, 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:57:58.760 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27487", 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 142397 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626919+0000", 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.761 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626921+0000", 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4219999999999999e-06 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626930+0000", 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5599999999999994e-06 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.762 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626945+0000", 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5495999999999999e-05 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.626955+0000", 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6190000000000006e-06 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.628796+0000", 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0018411910000000001 2024-08-30T23:57:58.763 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.628976+0000", 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000179777 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,6", 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.629127+0000", 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000150848 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.633350+0000", 2024-08-30T23:57:58.764 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042234500000000001 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.635612+0000", 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022614369999999998 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107166+0000", 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.471554168 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:57:58.765 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107232+0000", 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5927999999999996e-05 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.109170+0000", 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001938052 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27487.0:142397 7.d e810/800 ondisk, result = 0)", 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:57:58.766 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 321.07512508399998, 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.474397232, 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.767 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634796+0000", 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5099999999999999e-07 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.634801+0000", 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0550000000000001e-06 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.768 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.102650+0000", 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.467849444 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.102656+0000", 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8649999999999998e-06 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107133+0000", 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0044769750000000002 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.769 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.107145+0000", 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2656999999999999e-05 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.109193+0000", 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020475850000000002 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.770 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.9 e833,762 7:9b279d9f:::benchmark_data_smithi173_129296_object38265:head,794'7954)", 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 261.52300132400001, 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.880300788, 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.771 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186920+0000", 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.61e-07 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:37.186925+0000", 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4730000000000002e-06 2024-08-30T23:57:58.772 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067100+0000", 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.88017544199999997 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067106+0000", 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9030000000000001e-06 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.773 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067157+0000", 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0556000000000002e-05 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067162+0000", 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.587e-06 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.067220+0000", 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8066000000000003e-05 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.774 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e835,800 7:bbbda60d:::benchmark_data_smithi173_129296_object15076:head,778'6374)", 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 257.46310710300003, 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74636665599999996, 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:57:58.775 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246814+0000", 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246816+0000", 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.716e-06 2024-08-30T23:57:58.776 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.246828+0000", 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.27e-05 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992675+0000", 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.74584704999999996 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.777 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992682+0000", 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.241e-06 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.992757+0000", 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5322999999999997e-05 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.993180+0000", 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00042362600000000003 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.778 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e836,800 7:b1ca300e:::benchmark_data_smithi173_129296_object19401:head,781'6635)", 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 254.50642959499999, 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45044139700000002, 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.779 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203491+0000", 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203492+0000", 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7000000000000004e-07 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.780 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.203497+0000", 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8380000000000001e-06 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653230+0000", 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.44973348699999999 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653238+0000", 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1329999999999999e-06 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.781 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653353+0000", 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114485 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.653933+0000", 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00057978399999999996 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.5 e837,761 7:ae197e78:::benchmark_data_smithi173_129296_object20842:head,781'6740)", 2024-08-30T23:57:58.782 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 254.0468314, 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.45953577400000001, 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.783 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663089+0000", 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663090+0000", 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2699999999999998e-07 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.663096+0000", 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4179999999999996e-06 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.784 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122458+0000", 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.459362202 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122466+0000", 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.7600000000000002e-06 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122530+0000", 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3917000000000005e-05 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.785 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122541+0000", 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0824e-05 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.122625+0000", 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4726000000000002e-05 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDelete(7.5 e837,761 7:a846cf89:::benchmark_data_smithi173_129296_object20850:head,781'6741)", 2024-08-30T23:57:58.786 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 254.04043728900001, 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46070137100000003, 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.787 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669484+0000", 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669485+0000", 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.759e-06 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.669491+0000", 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7590000000000003e-06 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.788 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130069+0000", 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.460578184 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130075+0000", 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.541e-06 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130126+0000", 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1564000000000003e-05 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.789 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130133+0000", 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6730000000000003e-06 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.130185+0000", 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1891000000000002e-05 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.790 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e837,800 7:b35fe8f8:::benchmark_data_smithi173_129296_object19689:head,781'6653)", 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 253.95341490499999, 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42114565399999998, 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.791 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756506+0000", 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756507+0000", 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8899999999999999e-07 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.756510+0000", 2024-08-30T23:57:58.792 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.839e-06 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177074+0000", 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42056344499999998 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177083+0000", 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.397e-06 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.793 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177141+0000", 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7311000000000001e-05 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.177652+0000", 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00051107300000000002 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.d e837,800 7:b5944d16:::benchmark_data_smithi173_129296_object19693:head,781'6654)", 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:57:58.794 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 253.948821936, 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42161228299999998, 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.795 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761099+0000", 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.39e-07 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:44.761102+0000", 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8049999999999998e-06 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182311+0000", 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42120909499999998 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.796 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182317+0000", 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0440000000000001e-06 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182359+0000", 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1857000000000003e-05 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:45.182711+0000", 2024-08-30T23:57:58.797 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00035204300000000002 2024-08-30T23:57:58.798 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.798 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.798 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.798 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:57:58.798 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:57:58.798 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:57:58.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:58.827 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:58.929 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:57:58.958 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:58.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:57:58.980 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 16 16 23075 23059 90.0661 91.8125 0.00780227 0.01109 2024-08-30T23:57:59.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:57:59.087 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:57:59.190 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:59.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:59.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:57:59.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:57:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:59 smithi106 ceph-mon[16603]: from='client.19218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:59 smithi106 ceph-mon[16603]: pgmap v2115: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 6 unknown, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s wr, 120 op/s; 5581/51357 objects degraded (10.867%); 3210/51357 objects misplaced (6.250%); 2.1 MiB/s, 34 objects/s recovering 2024-08-30T23:57:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:59 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 5581/51357 objects degraded (10.867%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:59 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:59 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:57:59 smithi106 ceph-mon[16603]: osdmap e953: 12 total, 11 up, 12 in 2024-08-30T23:57:59.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:57:59.482 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:57:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:57:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:59 smithi136 ceph-mon[20207]: from='client.19218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:59 smithi136 ceph-mon[20207]: pgmap v2115: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 6 unknown, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s wr, 120 op/s; 5581/51357 objects degraded (10.867%); 3210/51357 objects misplaced (6.250%); 2.1 MiB/s, 34 objects/s recovering 2024-08-30T23:57:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:59 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 5581/51357 objects degraded (10.867%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:59 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:59 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:57:59 smithi136 ceph-mon[20207]: osdmap e953: 12 total, 11 up, 12 in 2024-08-30T23:57:59.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:59 smithi173 ceph-mon[20554]: from='client.19218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:57:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:59 smithi173 ceph-mon[20554]: pgmap v2115: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 6 unknown, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s wr, 120 op/s; 5581/51357 objects degraded (10.867%); 3210/51357 objects misplaced (6.250%); 2.1 MiB/s, 34 objects/s recovering 2024-08-30T23:57:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:59 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 5581/51357 objects degraded (10.867%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:57:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:59 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:57:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:59 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:57:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:57:59 smithi173 ceph-mon[20554]: osdmap e953: 12 total, 11 up, 12 in 2024-08-30T23:57:59.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:59.612 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:59.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:57:59.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:57:59.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:57:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:57:59.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:57:59.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:57:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:57:59.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:57:59.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 17 16 24773 24757 91.0103 106.125 0.00657958 0.0109768 2024-08-30T23:57:59.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:00.025 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:00.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:00.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:00.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:00.286 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:00.334 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:00.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:00.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:00.521 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:00.550 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:00.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:00.675 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:00.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:00.810 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:00.874 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:00.878 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:00.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:00.932 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:58:00.932 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:58:00.939 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:00.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:00.965 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:58:00.965 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:58:00.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 18 16 26177 26161 90.8288 87.75 0.0103928 0.0110014 2024-08-30T23:58:00.995 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:58:00.995 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:58:01.032 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:58:01.032 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:58:01.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:01.063 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:58:01.063 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:58:01.069 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:01.098 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:58:01.098 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:58:01.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:01.183 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:01 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:01.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:01 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:01.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:01 smithi106 ceph-mon[16603]: pgmap v2117: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 307 op/s; 8066/67902 objects degraded (11.879%); 7911/67902 objects misplaced (11.651%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:58:01.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:01 smithi106 ceph-mon[16603]: osdmap e954: 12 total, 11 up, 12 in 2024-08-30T23:58:01.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2041234140' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:01.184 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:01 smithi106 ceph-mon[16603]: osdmap e955: 12 total, 11 up, 12 in 2024-08-30T23:58:01.195 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:01.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:01.325 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:01.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.326 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:01.333 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:58:01.333 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:58:01.334 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:58:01.369 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:58:01.370 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:58:01.408 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:58:01.408 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:58:01.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:01.446 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:58:01.446 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:58:01.475 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:58:01.475 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:58:01.480 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:01 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:01 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:01 smithi136 ceph-mon[20207]: pgmap v2117: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 307 op/s; 8066/67902 objects degraded (11.879%); 7911/67902 objects misplaced (11.651%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:58:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:01 smithi136 ceph-mon[20207]: osdmap e954: 12 total, 11 up, 12 in 2024-08-30T23:58:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2041234140' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:01 smithi136 ceph-mon[20207]: osdmap e955: 12 total, 11 up, 12 in 2024-08-30T23:58:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:01 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:01 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:01 smithi173 ceph-mon[20554]: pgmap v2117: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 307 op/s; 8066/67902 objects degraded (11.879%); 7911/67902 objects misplaced (11.651%); 18 MiB/s, 286 objects/s recovering 2024-08-30T23:58:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:01 smithi173 ceph-mon[20554]: osdmap e954: 12 total, 11 up, 12 in 2024-08-30T23:58:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2041234140' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:01 smithi173 ceph-mon[20554]: osdmap e955: 12 total, 11 up, 12 in 2024-08-30T23:58:01.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:01.613 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:01.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.746 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:01.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:01.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:01.875 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:01.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:01.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:01.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 19 16 27634 27618 90.8407 91.0625 0.00994378 0.0110021 2024-08-30T23:58:02.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:02.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:02.138 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:02.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:02.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:02.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:58:02.336 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:02 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:58:02.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:02.388 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:02.490 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:02.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:02.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:02 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:58:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:02 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:58:02.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:02.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:02.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:02.760 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:02.862 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:02.884 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:02.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:02.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:58:02.981841+0000 min lat: 0.000982985 max lat: 2.30386 avg lat: 0.0110824 2024-08-30T23:58:02.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:58:02.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 20 16 28059 28043 87.6267 26.5625 0.0017661 0.0110824 2024-08-30T23:58:02.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:03.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:03.109 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pgp_num to 31 2024-08-30T23:58:03.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:03.140 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:03.242 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:03.264 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:03.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:03 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:03 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:03 smithi106 ceph-mon[16603]: pgmap v2120: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 429 op/s; 8291/69687 objects degraded (11.897%); 7911/69687 objects misplaced (11.352%); 18 MiB/s, 287 objects/s recovering 2024-08-30T23:58:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:03 smithi106 ceph-mon[16603]: osdmap e956: 12 total, 11 up, 12 in 2024-08-30T23:58:03.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2678948251' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-30T23:58:03.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:03.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:03.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:03.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:03 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:03 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:03 smithi136 ceph-mon[20207]: pgmap v2120: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 429 op/s; 8291/69687 objects degraded (11.897%); 7911/69687 objects misplaced (11.352%); 18 MiB/s, 287 objects/s recovering 2024-08-30T23:58:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:03 smithi136 ceph-mon[20207]: osdmap e956: 12 total, 11 up, 12 in 2024-08-30T23:58:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2678948251' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-30T23:58:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:03 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:03 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:03 smithi173 ceph-mon[20554]: pgmap v2120: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 429 op/s; 8291/69687 objects degraded (11.897%); 7911/69687 objects misplaced (11.352%); 18 MiB/s, 287 objects/s recovering 2024-08-30T23:58:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:03 smithi173 ceph-mon[20554]: osdmap e956: 12 total, 11 up, 12 in 2024-08-30T23:58:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2678948251' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-30T23:58:03.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:03.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:03.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:03.756 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:03.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:03.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:03.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:03.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:03.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 21 16 28212 28196 83.9094 9.5625 0.00169524 0.0115164 2024-08-30T23:58:03.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:04.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:04.104 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:04.134 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:04.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:04.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:04.359 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:04.381 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:04.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2678948251' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-30T23:58:04.462 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: osdmap e957: 12 total, 11 up, 12 in 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: 9.13 scrub starts 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: 9.13 scrub ok 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: 9.15 scrub starts 2024-08-30T23:58:04.463 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:04 smithi173 ceph-mon[20554]: osdmap e958: 12 total, 11 up, 12 in 2024-08-30T23:58:04.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:04.504 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2678948251' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: osdmap e957: 12 total, 11 up, 12 in 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: 9.13 scrub starts 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: 9.13 scrub ok 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: 9.15 scrub starts 2024-08-30T23:58:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:04 smithi136 ceph-mon[20207]: osdmap e958: 12 total, 11 up, 12 in 2024-08-30T23:58:04.606 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:04.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2678948251' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: osdmap e957: 12 total, 11 up, 12 in 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: 9.13 scrub starts 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: 9.13 scrub ok 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: 9.15 scrub starts 2024-08-30T23:58:04.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:04 smithi106 ceph-mon[16603]: osdmap e958: 12 total, 11 up, 12 in 2024-08-30T23:58:04.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:04.730 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:04.751 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:04.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:04.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:04.876 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:04.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:04.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:04.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 22 16 28351 28335 80.4899 8.6875 0.00162041 0.0120042 2024-08-30T23:58:05.000 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:05.101 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:05.125 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:05.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:05.260 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:05.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:05.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:05.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:05.519 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 9.15 scrub ok 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: pgmap v2123: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 658 op/s; 9204/75807 objects degraded (12.141%); 5219/75807 objects misplaced (6.885%); 21 MiB/s, 336 objects/s recovering 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9204/75807 objects degraded (12.141%), 10 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 9.12 scrub starts 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 9.12 scrub ok 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 9.14 scrub starts 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 9.14 scrub ok 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:58:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:05 smithi136 ceph-mon[20207]: osdmap e959: 12 total, 11 up, 12 in 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 9.15 scrub ok 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: pgmap v2123: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 658 op/s; 9204/75807 objects degraded (12.141%); 5219/75807 objects misplaced (6.885%); 21 MiB/s, 336 objects/s recovering 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9204/75807 objects degraded (12.141%), 10 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 9.12 scrub starts 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 9.12 scrub ok 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 9.14 scrub starts 2024-08-30T23:58:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 9.14 scrub ok 2024-08-30T23:58:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:58:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:58:05.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:05 smithi173 ceph-mon[20554]: osdmap e959: 12 total, 11 up, 12 in 2024-08-30T23:58:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 9.15 scrub ok 2024-08-30T23:58:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: pgmap v2123: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+remapped, 5 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 30 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 658 op/s; 9204/75807 objects degraded (12.141%); 5219/75807 objects misplaced (6.885%); 21 MiB/s, 336 objects/s recovering 2024-08-30T23:58:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9204/75807 objects degraded (12.141%), 10 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:05.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 9.12 scrub starts 2024-08-30T23:58:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 9.12 scrub ok 2024-08-30T23:58:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 9.14 scrub starts 2024-08-30T23:58:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 9.14 scrub ok 2024-08-30T23:58:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:58:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:58:05.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:05 smithi106 ceph-mon[16603]: osdmap e959: 12 total, 11 up, 12 in 2024-08-30T23:58:05.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:05.646 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:05.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:05.774 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:05.876 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:05.899 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:05.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:05.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 23 16 30033 30017 81.5606 105.125 0.0621771 0.0122478 2024-08-30T23:58:06.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:06.024 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:06.126 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:06.149 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:06.251 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:06.272 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:06 smithi106 ceph-mon[16603]: 9.19 scrub starts 2024-08-30T23:58:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:06 smithi106 ceph-mon[16603]: 9.10 scrub starts 2024-08-30T23:58:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:06 smithi106 ceph-mon[16603]: 9.19 scrub ok 2024-08-30T23:58:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:06 smithi106 ceph-mon[16603]: 9.10 scrub ok 2024-08-30T23:58:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:06 smithi106 ceph-mon[16603]: 9.17 scrub starts 2024-08-30T23:58:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:06 smithi106 ceph-mon[16603]: osdmap e960: 12 total, 11 up, 12 in 2024-08-30T23:58:06.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:06.395 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.396 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:06.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:06.518 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:06.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:06 smithi136 ceph-mon[20207]: 9.19 scrub starts 2024-08-30T23:58:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:06 smithi136 ceph-mon[20207]: 9.10 scrub starts 2024-08-30T23:58:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:06 smithi136 ceph-mon[20207]: 9.19 scrub ok 2024-08-30T23:58:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:06 smithi136 ceph-mon[20207]: 9.10 scrub ok 2024-08-30T23:58:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:06 smithi136 ceph-mon[20207]: 9.17 scrub starts 2024-08-30T23:58:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:06 smithi136 ceph-mon[20207]: osdmap e960: 12 total, 11 up, 12 in 2024-08-30T23:58:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:06 smithi173 ceph-mon[20554]: 9.19 scrub starts 2024-08-30T23:58:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:06 smithi173 ceph-mon[20554]: 9.10 scrub starts 2024-08-30T23:58:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:06 smithi173 ceph-mon[20554]: 9.19 scrub ok 2024-08-30T23:58:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:06 smithi173 ceph-mon[20554]: 9.10 scrub ok 2024-08-30T23:58:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:06 smithi173 ceph-mon[20554]: 9.17 scrub starts 2024-08-30T23:58:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:06 smithi173 ceph-mon[20554]: osdmap e960: 12 total, 11 up, 12 in 2024-08-30T23:58:06.620 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:06.641 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:06.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:06.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:06.867 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:06.888 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:06.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:06.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 24 16 30440 30424 79.2219 25.4375 0.00212223 0.0123451 2024-08-30T23:58:06.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:07.015 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:07.117 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:07.152 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:07.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:07.278 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:07.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:07.405 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:07.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:07.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:07 smithi136 ceph-mon[20207]: pgmap v2126: 57 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 31 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 778 op/s; 8809/84486 objects degraded (10.427%); 3916/84486 objects misplaced (4.635%); 21 MiB/s, 334 objects/s recovering 2024-08-30T23:58:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:07 smithi136 ceph-mon[20207]: 9.16 scrub starts 2024-08-30T23:58:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:07 smithi136 ceph-mon[20207]: 9.17 scrub ok 2024-08-30T23:58:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:07 smithi136 ceph-mon[20207]: 9.16 scrub ok 2024-08-30T23:58:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:07 smithi136 ceph-mon[20207]: osdmap e961: 12 total, 11 up, 12 in 2024-08-30T23:58:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:07 smithi173 ceph-mon[20554]: pgmap v2126: 57 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 31 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 778 op/s; 8809/84486 objects degraded (10.427%); 3916/84486 objects misplaced (4.635%); 21 MiB/s, 334 objects/s recovering 2024-08-30T23:58:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:07 smithi173 ceph-mon[20554]: 9.16 scrub starts 2024-08-30T23:58:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:07 smithi173 ceph-mon[20554]: 9.17 scrub ok 2024-08-30T23:58:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:07 smithi173 ceph-mon[20554]: 9.16 scrub ok 2024-08-30T23:58:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:07 smithi173 ceph-mon[20554]: osdmap e961: 12 total, 11 up, 12 in 2024-08-30T23:58:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:07 smithi106 ceph-mon[16603]: pgmap v2126: 57 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+clean+remapped, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 31 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 778 op/s; 8809/84486 objects degraded (10.427%); 3916/84486 objects misplaced (4.635%); 21 MiB/s, 334 objects/s recovering 2024-08-30T23:58:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:07 smithi106 ceph-mon[16603]: 9.16 scrub starts 2024-08-30T23:58:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:07 smithi106 ceph-mon[16603]: 9.17 scrub ok 2024-08-30T23:58:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:07 smithi106 ceph-mon[16603]: 9.16 scrub ok 2024-08-30T23:58:07.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:07 smithi106 ceph-mon[16603]: osdmap e961: 12 total, 11 up, 12 in 2024-08-30T23:58:07.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:07.656 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:07.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:07.780 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:07.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:07.906 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:07.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:07.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 25 16 32088 32072 80.1727 103 0.00762245 0.0124656 2024-08-30T23:58:08.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:08.031 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:08.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:08.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.156 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:08.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:08.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:08.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:08.409 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:08.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:08.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:08.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:08.656 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:08.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:08.767 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:08.767 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:08.781 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:08.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:08.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:08.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:08.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 26 16 33307 33291 80.0192 76.1875 0.00370068 0.0124849 2024-08-30T23:58:09.006 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:09.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:09.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:09.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:09.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:09.287 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:09.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:09.418 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.419 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:09.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:09 smithi136 ceph-mon[20207]: pgmap v2129: 57 pgs: 2 active+clean+scrubbing, 1 peering, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 33 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7355/88059 objects degraded (8.352%); 3775/88059 objects misplaced (4.287%) 2024-08-30T23:58:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:09 smithi136 ceph-mon[20207]: 9.18 scrub starts 2024-08-30T23:58:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:09 smithi136 ceph-mon[20207]: 9.18 scrub ok 2024-08-30T23:58:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:09 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:58:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:09 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:58:09.543 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:09.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:09 smithi173 ceph-mon[20554]: pgmap v2129: 57 pgs: 2 active+clean+scrubbing, 1 peering, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 33 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7355/88059 objects degraded (8.352%); 3775/88059 objects misplaced (4.287%) 2024-08-30T23:58:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:09 smithi173 ceph-mon[20554]: 9.18 scrub starts 2024-08-30T23:58:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:09 smithi173 ceph-mon[20554]: 9.18 scrub ok 2024-08-30T23:58:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:09 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:58:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:09 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:58:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:09 smithi106 ceph-mon[16603]: pgmap v2129: 57 pgs: 2 active+clean+scrubbing, 1 peering, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 33 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7355/88059 objects degraded (8.352%); 3775/88059 objects misplaced (4.287%) 2024-08-30T23:58:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:09 smithi106 ceph-mon[16603]: 9.18 scrub starts 2024-08-30T23:58:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:09 smithi106 ceph-mon[16603]: 9.18 scrub ok 2024-08-30T23:58:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:09 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:58:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:09 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:58:09.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:09.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:58:09.746 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:58:09.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:09.788 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:09.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:09.913 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:09.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:09.981 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 27 16 34707 34691 80.296 87.5 0.0113795 0.0124475 2024-08-30T23:58:10.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:10.039 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:10.141 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:10.168 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:10.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:58:10.213 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:10.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:10.293 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:10.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:10.422 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:10.524 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:10 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8136/96675 objects degraded (8.416%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:10.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:10.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:10.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:10 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8136/96675 objects degraded (8.416%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:10.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:10 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8136/96675 objects degraded (8.416%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:10.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:10.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:10.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:10.794 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:10.896 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:10.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:10.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:10.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 28 16 35942 35926 80.1846 77.1875 0.00216624 0.0124165 2024-08-30T23:58:11.020 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:11.041 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:11.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:11.155 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:11.165 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:11.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:11.290 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:11.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:11.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:11.420 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:11.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:11 smithi106 ceph-mon[16603]: pgmap v2130: 57 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 34 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s wr, 110 op/s; 8136/96675 objects degraded (8.416%); 3927/96675 objects misplaced (4.062%) 2024-08-30T23:58:11.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:11 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:58:11.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:11 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:58:11.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:11 smithi136 ceph-mon[20207]: pgmap v2130: 57 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 34 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s wr, 110 op/s; 8136/96675 objects degraded (8.416%); 3927/96675 objects misplaced (4.062%) 2024-08-30T23:58:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:11 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:58:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:11 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:58:11.543 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:11 smithi173 ceph-mon[20554]: pgmap v2130: 57 pgs: 2 active+clean+scrubbing, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 34 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s wr, 110 op/s; 8136/96675 objects degraded (8.416%); 3927/96675 objects misplaced (4.062%) 2024-08-30T23:58:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:11 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:58:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:11 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:58:11.644 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:11.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:11.676 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:11.676 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:11.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:58:11.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:11.819 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:11.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:11.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:11.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:11.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 29 16 37137 37121 79.9948 74.6875 0.00252408 0.0124822 2024-08-30T23:58:12.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:12.076 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:12.178 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:12.188 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:58:12.201 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:12.303 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:12.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:12 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:58:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:12 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:58:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/65413314' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:58:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:12 smithi106 ceph-mon[16603]: 1.17 scrub starts 2024-08-30T23:58:12.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:12 smithi106 ceph-mon[16603]: 1.17 scrub ok 2024-08-30T23:58:12.417 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:58:12.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:12.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:12 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:58:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:12 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:58:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/65413314' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:58:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:12 smithi136 ceph-mon[20207]: 1.17 scrub starts 2024-08-30T23:58:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:12 smithi136 ceph-mon[20207]: 1.17 scrub ok 2024-08-30T23:58:12.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:12.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:12 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:58:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:12 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:58:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/65413314' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:58:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:12 smithi173 ceph-mon[20554]: 1.17 scrub starts 2024-08-30T23:58:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:12 smithi173 ceph-mon[20554]: 1.17 scrub ok 2024-08-30T23:58:12.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:12.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:12.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:12.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:12.832 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:12.934 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:12.961 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:12.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:12.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 30 16 38584 38568 80.3427 90.4375 0.0018174 0.0124373 2024-08-30T23:58:13.062 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:13.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:13.134 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:13.160 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:13.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:13.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.216 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:13.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:13.303 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:13 smithi106 ceph-mon[16603]: pgmap v2131: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8302/101265 objects degraded (8.198%); 4341/101265 objects misplaced (4.287%) 2024-08-30T23:58:13.304 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:13 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:13.304 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/65413314' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:58:13.304 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:13 smithi106 ceph-mon[16603]: osdmap e962: 12 total, 11 up, 12 in 2024-08-30T23:58:13.317 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:13.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:13.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:13.466 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:13 smithi136 ceph-mon[20207]: pgmap v2131: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8302/101265 objects degraded (8.198%); 4341/101265 objects misplaced (4.287%) 2024-08-30T23:58:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:13 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/65413314' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:58:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:13 smithi136 ceph-mon[20207]: osdmap e962: 12 total, 11 up, 12 in 2024-08-30T23:58:13.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:13 smithi173 ceph-mon[20554]: pgmap v2131: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8302/101265 objects degraded (8.198%); 4341/101265 objects misplaced (4.287%) 2024-08-30T23:58:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:13 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/65413314' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:58:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:13 smithi173 ceph-mon[20554]: osdmap e962: 12 total, 11 up, 12 in 2024-08-30T23:58:13.593 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:13.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:13.723 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:13.744 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:58:13.745 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:58:13.772 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:58:13.772 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:58:13.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:58:13.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:58:13.823 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:58:13.823 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:58:13.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:13.852 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:58:13.852 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:58:13.853 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:13.881 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:58:13.882 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:58:13.912 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:58:13.912 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:58:13.912 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:58:13.942 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:58:13.942 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:58:13.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:13.970 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:58:13.970 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:58:13.981 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:13.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:13.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 31 16 40345 40329 81.3011 110.062 0.00311923 0.0122921 2024-08-30T23:58:13.999 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:58:13.999 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:58:14.025 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:58:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:58:14.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:14.108 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:14.210 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.232 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:14.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:14.333 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:14.360 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:14.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:14.487 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:14.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:14.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4232600901' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4232600901' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:14.589 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4232600901' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:14.611 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:14.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:14.738 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:14.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:14.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:14.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:14.898 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:58:14.898 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:58:14.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 763/756 [PushReplyOp(7:59a9bd86:::benchmark_data_smithi173_129296_object9245:head)])", 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:28.337999+0000", 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 466.55939180399997, 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35323532299999999, 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.337999+0000", 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.337999+0000", 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.338001+0000", 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1659999999999999e-06 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.338007+0000", 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2140000000000001e-06 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.686245+0000", 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34823851900000002 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.686252+0000", 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0690000000000004e-06 2024-08-30T23:58:14.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.686323+0000", 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0389999999999995e-05 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.691235+0000", 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049119649999999999 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 764/756 [PushOp(7:3f8ad8ed:::benchmark_data_smithi173_129296_object10539:head, version: 730'677, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3f8ad8ed:::benchmark_data_smithi173_129296_object10539:head@730'677, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:29.900241+0000", 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 464.99715020000002, 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69259163499999998, 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900241+0000", 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900241+0000", 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900242+0000", 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.06e-07 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.900310+0000", 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8467999999999996e-05 2024-08-30T23:58:14.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591335+0000", 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.69102508299999998 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591348+0000", 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2687000000000001e-05 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591489+0000", 2024-08-30T23:58:14.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014089800000000001 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.591503+0000", 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3661e-05 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:30.592833+0000", 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013301319999999999 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 767/756 [PushOp(7:3c0798f8:::benchmark_data_smithi173_129296_object12781:head, version: 730'820, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3c0798f8:::benchmark_data_smithi173_129296_object12781:head@730'820, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:33.363684+0000", 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 461.53370683499998, 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.463505677, 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363684+0000", 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363684+0000", 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363691+0000", 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5100000000000004e-06 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.363732+0000", 2024-08-30T23:58:14.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0989999999999999e-05 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825755+0000", 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46202335 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825774+0000", 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8624999999999999e-05 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825887+0000", 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000113225 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.825898+0000", 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1107000000000001e-05 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:33.827190+0000", 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012918700000000001 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.911 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 767/756 [PushOp(7:3b8bff84:::benchmark_data_smithi173_129296_object14349:head, version: 731'902, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3b8bff84:::benchmark_data_smithi173_129296_object14349:head@731'902, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.911 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:34.792891+0000", 2024-08-30T23:58:14.911 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 460.10450020399998, 2024-08-30T23:58:14.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75540186399999998, 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792891+0000", 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792891+0000", 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792893+0000", 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4959999999999999e-06 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:34.792941+0000", 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8745999999999997e-05 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.545895+0000", 2024-08-30T23:58:14.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75295403400000005 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.545914+0000", 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8332e-05 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.546016+0000", 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010255399999999999 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.546031+0000", 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4878000000000001e-05 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:35.548293+0000", 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002261824 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.915 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 768/756 [PushOp(7:335c6484:::benchmark_data_smithi173_129296_object20286:head, version: 734'1249, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:335c6484:::benchmark_data_smithi173_129296_object20286:head@734'1249, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:40.509141+0000", 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 454.38825039599999, 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.42131471799999998, 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509141+0000", 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509141+0000", 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509142+0000", 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1800000000000004e-07 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.509210+0000", 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7805999999999994e-05 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928275+0000", 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.41906584200000002 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928303+0000", 2024-08-30T23:58:14.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.7571000000000001e-05 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928418+0000", 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000114824 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.928433+0000", 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4754e-05 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:40.930456+0000", 2024-08-30T23:58:14.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0020230030000000002 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 769/756 [PushReplyOp(7:5d5301f9:::benchmark_data_smithi173_129296_object21928:head)])", 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:41.792253+0000", 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 453.10513853700002, 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63873635799999995, 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792253+0000", 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792253+0000", 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792254+0000", 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.046e-06 2024-08-30T23:58:14.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:41.792260+0000", 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7849999999999997e-06 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.426542+0000", 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.63428287000000005 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.426551+0000", 2024-08-30T23:58:14.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9460000000000004e-06 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.426617+0000", 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5765000000000005e-05 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:42.430989+0000", 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0043719459999999998 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 770/756 [PushOp(7:34377c92:::benchmark_data_smithi173_129296_object26126:head, version: 734'1608, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:34377c92:::benchmark_data_smithi173_129296_object26126:head@734'1608, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:46.509140+0000", 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 448.38825158200001, 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75758949099999995, 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509140+0000", 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509140+0000", 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509141+0000", 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4750000000000001e-06 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.509183+0000", 2024-08-30T23:58:14.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1538000000000001e-05 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265266+0000", 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.75608357800000003 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265280+0000", 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3713000000000001e-05 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265360+0000", 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9806000000000001e-05 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.265369+0000", 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4630000000000007e-06 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.266729+0000", 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0013599180000000001 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 770/756 [PushReplyOp(7:5610cbb5:::benchmark_data_smithi173_129296_object27066:head)])", 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 448.34417671400001, 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80890829099999995, 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:58:14.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553215+0000", 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4e-07 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:46.553220+0000", 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3499999999999999e-06 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.357935+0000", 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.80471498500000005 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.357944+0000", 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.5680000000000005e-06 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.358006+0000", 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1835999999999995e-05 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.362123+0000", 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041167119999999998 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 771/756 [PushOp(7:3420d111:::benchmark_data_smithi173_129296_object27164:head, version: 735'1670, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3420d111:::benchmark_data_smithi173_129296_object27164:head@735'1670, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:47.927136+0000", 2024-08-30T23:58:14.931 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 446.97025504700002, 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61515216699999997, 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927136+0000", 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927136+0000", 2024-08-30T23:58:14.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927138+0000", 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.322e-06 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:47.927187+0000", 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.9691999999999998e-05 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.540898+0000", 2024-08-30T23:58:14.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.613710423 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.540917+0000", 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9219999999999999e-05 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.541014+0000", 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7189999999999996e-05 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.541027+0000", 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3355e-05 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.542288+0000", 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001260965 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 771/756 [PushReplyOp(7:5694c7fb:::benchmark_data_smithi173_129296_object28842:head)])", 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:48.976044+0000", 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 445.92134714600002, 2024-08-30T23:58:14.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.55948937200000004, 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976044+0000", 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976044+0000", 2024-08-30T23:58:14.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976045+0000", 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2200000000000003e-07 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:48.976050+0000", 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7960000000000003e-06 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.520874+0000", 2024-08-30T23:58:14.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.54482465000000002 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.520882+0000", 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.469e-06 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.526943+0000", 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0060610630000000002 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:49.535534+0000", 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0085906720000000006 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 788/756 [PushOp(7:3133a158:::benchmark_data_smithi173_129296_object77006:head, version: 749'4738, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3133a158:::benchmark_data_smithi173_129296_object77006:head@749'4738, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.467818+0000", 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 391.42957318200001, 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.41297604199999999, 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.939 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467818+0000", 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467818+0000", 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467820+0000", 2024-08-30T23:58:14.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.747e-06 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.467883+0000", 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2741000000000001e-05 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878536+0000", 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.410653885 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878552+0000", 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5569e-05 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878644+0000", 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2165999999999998e-05 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.878660+0000", 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.596e-05 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.880794+0000", 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0021339739999999999 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPush(7.c 788/756 [PushOp(7:3db12067:::benchmark_data_smithi173_129296_object77016:head, version: 749'4739, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3db12067:::benchmark_data_smithi173_129296_object77016:head@749'4739, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.474899+0000", 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 391.42249205799999, 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.41134347300000002, 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.943 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474899+0000", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474899+0000", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474901+0000", 2024-08-30T23:58:14.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9870000000000002e-06 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.474952+0000", 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1276000000000002e-05 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883728+0000", 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.40877536199999998 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883747+0000", 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9368000000000002e-05 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883844+0000", 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6424999999999999e-05 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.883857+0000", 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3427e-05 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.886243+0000", 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0023856279999999999 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.a 788/756 [PushReplyOp(7:59265772:::benchmark_data_smithi173_129296_object57027:head)])", 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:51:43.476365+0000", 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 391.42102598700001, 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.510494157, 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.947 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:58:14.948 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476365+0000", 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476365+0000", 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476367+0000", 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2699999999999999e-06 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.476375+0000", 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.2579999999999995e-06 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.983888+0000", 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50751275399999995 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.983897+0000", 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6409999999999997e-06 2024-08-30T23:58:14.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.983992+0000", 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4927999999999996e-05 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:51:43.986859+0000", 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002867306 2024-08-30T23:58:14.951 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:126882 7.c e795/792 7:35900c7d:::benchmark_data_smithi173_129296_object39766:head v 795'7834, mlcod=794'7831)", 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:02.133357+0000", 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 372.76403442600002, 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56243918699999995, 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133357+0000", 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133357+0000", 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133358+0000", 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.353e-06 2024-08-30T23:58:14.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.133367+0000", 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2790000000000004e-06 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693400+0000", 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56003286100000005 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693406+0000", 2024-08-30T23:58:14.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0619999999999997e-06 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693461+0000", 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4506999999999998e-05 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.693479+0000", 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.7861999999999999e-05 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.695751+0000", 2024-08-30T23:58:14.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0022726489999999998 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:02.695796+0000", 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4613999999999997e-05 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27487.0:142377 7.8 e810/725 7:18815242:::benchmark_data_smithi173_129296_object55261:head v 810'8843, mlcod=810'8841)", 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:52:37.395840+0000", 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 337.501550931, 2024-08-30T23:58:14.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61643008899999996, 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395840+0000", 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395840+0000", 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395842+0000", 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3999999999999999e-06 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:37.395850+0000", 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9079999999999995e-06 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007839+0000", 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61198960800000002 2024-08-30T23:58:14.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007848+0000", 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6619999999999994e-06 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007916+0000", 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7676999999999996e-05 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.007937+0000", 2024-08-30T23:58:14.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.1053999999999999e-05 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.012201+0000", 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0042639740000000002 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:52:38.012270+0000", 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9806000000000002e-05 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:6054 8.e e847/840 8:7a0156cd:::benchmark_data_smithi173_143816_object6053:head v 847'378, mlcod=847'375)", 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:59.025739+0000", 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 255.871652138, 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23799822800000001, 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025739+0000", 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025739+0000", 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025740+0000", 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8899999999999999e-07 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.025910+0000", 2024-08-30T23:58:14.962 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00017013099999999999 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262718+0000", 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23680768099999999 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262724+0000", 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5150000000000003e-06 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.963 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262826+0000", 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101853 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.262845+0000", 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9255000000000001e-05 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.263700+0000", 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00085512499999999998 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.964 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:59.263737+0000", 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7079000000000002e-05 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:36537 8.e e862/840 8:78cdc0b0:::benchmark_data_smithi173_143816_object36536:head v 862'2321, mlcod=862'2318)", 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.400646+0000", 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 229.49674523100001, 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39470844900000002, 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.965 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400646+0000", 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400646+0000", 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.966 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400647+0000", 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7699999999999992e-07 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.400688+0000", 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1428999999999999e-05 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793696+0000", 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39300789699999999 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.967 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793705+0000", 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.132e-06 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793778+0000", 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2804000000000003e-05 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.793805+0000", 2024-08-30T23:58:14.968 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6543999999999999e-05 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.795319+0000", 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001514154 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.795354+0000", 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5512000000000002e-05 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:37753 8.a e864/840 8:57fc5b26:::benchmark_data_smithi173_143816_object37752:head v 864'2348, mlcod=864'2345)", 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.183034+0000", 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 227.71435724599999, 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25250152199999998, 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183034+0000", 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.970 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183034+0000", 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183035+0000", 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3099999999999996e-07 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.183092+0000", 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6889999999999999e-05 2024-08-30T23:58:14.971 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434193+0000", 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.25110125 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434202+0000", 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7280000000000001e-06 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434259+0000", 2024-08-30T23:58:14.972 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.7534999999999997e-05 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.434276+0000", 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6269999999999998e-05 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.435485+0000", 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0012090029999999999 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.973 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.435536+0000", 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1014999999999999e-05 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.27853.0:178947 8.a e926/840 8:523510ea:::benchmark_data_smithi173_143816_object80633:head v 926'11130, mlcod=926'11127)", 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:00.401008+0000", 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 74.496383532999999, 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23612587299999999, 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.974 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.401008+0000", 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.401008+0000", 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.401009+0000", 2024-08-30T23:58:14.975 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.7899999999999997e-07 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.401015+0000", 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.567e-06 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.633140+0000", 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.23212490099999999 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.976 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.633146+0000", 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2219999999999999e-06 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.633204+0000", 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8128000000000003e-05 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.633223+0000", 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8695e-05 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.977 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.637102+0000", 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0038785780000000001 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:00.637134+0000", 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.1903000000000003e-05 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.978 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.28405.0:40538 9.16 e962/958 9:6c8a89cf:::benchmark_data_smithi173_160189_object40537:head v 962'1813, mlcod=962'1808)", 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:14.148161+0000", 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.74923032000000001, 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17199956599999999, 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.148161+0000", 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.979 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.148161+0000", 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.148162+0000", 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.08e-06 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.148216+0000", 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3921000000000001e-05 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.980 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.319246+0000", 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.17103004499999999 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.319254+0000", 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5599999999999996e-06 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.319308+0000", 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4700999999999997e-05 2024-08-30T23:58:14.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.319327+0000", 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8875000000000001e-05 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.320119+0000", 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00079168499999999998 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:14.320161+0000", 2024-08-30T23:58:14.982 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1699000000000002e-05 2024-08-30T23:58:14.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.983 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:14.983 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:14.983 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:58:14.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:14.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 32 16 41536 41520 81.0865 74.4375 0.00848711 0.0123252 2024-08-30T23:58:15.007 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:15.086 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:58:15.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:15.131 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:15.233 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:15.257 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:15.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:15 smithi106 ceph-mon[16603]: pgmap v2133: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s wr, 121 op/s; 8934/106236 objects degraded (8.410%); 4560/106236 objects misplaced (4.292%) 2024-08-30T23:58:15.359 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:15.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:15.484 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:15.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:15 smithi136 ceph-mon[20207]: pgmap v2133: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s wr, 121 op/s; 8934/106236 objects degraded (8.410%); 4560/106236 objects misplaced (4.292%) 2024-08-30T23:58:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:15 smithi173 ceph-mon[20554]: pgmap v2133: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s wr, 121 op/s; 8934/106236 objects degraded (8.410%); 4560/106236 objects misplaced (4.292%) 2024-08-30T23:58:15.614 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:15.630 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:58:15.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:58:15.630 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-08-30T23:58:15.631 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:58:15.644 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:15.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:58:15.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:58:15.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:15.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:15.903 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:15.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:15.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 33 16 43232 43216 81.8411 106 0.00376653 0.0122132 2024-08-30T23:58:16.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:16.031 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:16.085 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:16.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:16.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:16.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:16.288 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:16.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:16.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:16.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:16.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8934/106236 objects degraded (8.410%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:16.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:16 smithi136 ceph-mon[20207]: osdmap e963: 12 total, 11 up, 12 in 2024-08-30T23:58:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:16.541 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8934/106236 objects degraded (8.410%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:16 smithi173 ceph-mon[20554]: osdmap e963: 12 total, 11 up, 12 in 2024-08-30T23:58:16.606 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8934/106236 objects degraded (8.410%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:16.606 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:16 smithi106 ceph-mon[16603]: osdmap e963: 12 total, 11 up, 12 in 2024-08-30T23:58:16.641 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:58:16.642 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:16.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:16.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:16.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:16.894 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:16.895 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:16.920 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:16.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:16.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 34 16 44622 44606 81.989 86.875 0.00338713 0.0121872 2024-08-30T23:58:17.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:17.047 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:17.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:17.175 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:17.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:17.301 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:17.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:17.432 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:17.498 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:17 smithi106 ceph-mon[16603]: pgmap v2134: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 798 op/s; 9759/118797 objects degraded (8.215%); 5009/118797 objects misplaced (4.216%) 2024-08-30T23:58:17.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:17.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:17 smithi136 ceph-mon[20207]: pgmap v2134: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 798 op/s; 9759/118797 objects degraded (8.215%); 5009/118797 objects misplaced (4.216%) 2024-08-30T23:58:17.561 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:17 smithi173 ceph-mon[20554]: pgmap v2134: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 798 op/s; 9759/118797 objects degraded (8.215%); 5009/118797 objects misplaced (4.216%) 2024-08-30T23:58:17.662 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:17.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:17.667 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:58:17.668 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:58:17.685 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:17.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:58:17.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.811 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:17.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:17.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:17.935 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:17.936 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:17.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 35 16 46121 46105 82.3229 93.6875 0.0305415 0.0121214 2024-08-30T23:58:18.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:18.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:18.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.186 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:18.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:18.207 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:58:18.288 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:18.310 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:18.404 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2135,"stamp":"2024-08-30T23:58:16.064506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2595659613,"num_objects":39599,"num_object_clones":0,"num_object_copies":118797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9759,"num_objects_misplaced":5009,"num_objects_unfound":0,"num_objects_dirty":39599,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":26099,"num_write_kb":1669970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":149683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53210,"ondisk_log_size":53210,"up":150,"acting":154,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15381300,"kb_used_data":7614628,"kb_used_omap":109,"kb_used_meta":7766354,"kb_avail":1109560524,"statfs":{"total":1151940427776,"available":1136189976576,"internally_reserved":0,"allocated":7797379072,"data_stored":7787883410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":7952747160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":671514624,"num_objects":10246,"num_object_clones":0,"num_object_copies":30738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2404,"num_objects_misplaced":1234,"num_objects_unfound":0,"num_objects_dirty":10246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":417248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2285,"num_bytes_recovered":-149749760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.214997"},"pg_stats":[{"pgid":"9.17","version":"962'1871","reported_seq":4056,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:15.703011+0000","last_change":"2024-08-30T23:58:06.151958+0000","last_active":"2024-08-30T23:58:15.703011+0000","last_peered":"2024-08-30T23:58:15.703011+0000","last_clean":"2024-08-30T23:58:15.703011+0000","last_became_active":"2024-08-30T23:58:05.121618+0000","last_became_peered":"2024-08-30T23:58:05.121618+0000","last_unstale":"2024-08-30T23:58:15.703011+0000","last_undegraded":"2024-08-30T23:58:15.703011+0000","last_fullsized":"2024-08-30T23:58:15.703011+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'1460","last_scrub_stamp":"2024-08-30T23:58:06.151812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:06.151812+0000","objects_scrubbed":907,"log_size":1871,"log_dups_size":0,"ondisk_log_size":1871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:30.365434+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86835200,"num_objects":1325,"num_object_clones":0,"num_object_copies":3975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":418,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"961'1642","reported_seq":4089,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:11.215010+0000","last_change":"2024-08-30T23:58:06.471559+0000","last_active":"2024-08-30T23:58:11.215010+0000","last_peered":"2024-08-30T23:58:11.215010+0000","last_clean":"2024-08-30T23:58:11.215010+0000","last_became_active":"2024-08-30T23:58:05.118500+0000","last_became_peered":"2024-08-30T23:58:05.118500+0000","last_unstale":"2024-08-30T23:58:11.215010+0000","last_undegraded":"2024-08-30T23:58:11.215010+0000","last_fullsized":"2024-08-30T23:58:11.215010+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'1469","last_scrub_stamp":"2024-08-30T23:58:06.471505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:06.471505+0000","objects_scrubbed":927,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:10.422556+0000","scrub_duration":392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72155136,"num_objects":1101,"num_object_clones":0,"num_object_copies":3303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":188,"reported_epoch":961,"state":"active+undersized","last_fresh":"2024-08-30T23:58:07.074476+0000","last_change":"2024-08-30T23:55:59.172632+0000","last_active":"2024-08-30T23:58:07.074476+0000","last_peered":"2024-08-30T23:58:07.074476+0000","last_clean":"2024-08-30T23:55:57.160064+0000","last_became_active":"2024-08-30T23:55:59.172632+0000","last_became_peered":"2024-08-30T23:55:59.172632+0000","last_unstale":"2024-08-30T23:58:07.074476+0000","last_undegraded":"2024-08-30T23:58:07.074476+0000","last_fullsized":"2024-08-30T23:55:59.167996+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.344869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.344869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:12.066513+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.15","version":"962'1944","reported_seq":3975,"reported_epoch":962,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:15.760774+0000","last_change":"2024-08-30T23:58:03.667578+0000","last_active":"2024-08-30T23:58:15.760774+0000","last_peered":"2024-08-30T23:58:15.760774+0000","last_clean":"2024-08-30T23:58:15.760774+0000","last_became_active":"2024-08-30T23:58:03.116440+0000","last_became_peered":"2024-08-30T23:58:03.116440+0000","last_unstale":"2024-08-30T23:58:15.760774+0000","last_undegraded":"2024-08-30T23:58:15.760774+0000","last_fullsized":"2024-08-30T23:58:15.760774+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'1441","last_scrub_stamp":"2024-08-30T23:58:03.667484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:03.667484+0000","objects_scrubbed":908,"log_size":1944,"log_dups_size":0,"ondisk_log_size":1944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:17.986331+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92471296,"num_objects":1411,"num_object_clones":0,"num_object_copies":4233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1411,"num_objects_unfound":0,"num_objects_dirty":1411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":503,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4,5],"avail_no_missing":["8","4","5"],"object_location_counts":[{"shards":"4,5,8","objects":1411}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":192,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.074458+0000","last_change":"2024-08-30T23:56:46.421822+0000","last_active":"2024-08-30T23:58:07.074458+0000","last_peered":"2024-08-30T23:58:07.074458+0000","last_clean":"2024-08-30T23:58:07.074458+0000","last_became_active":"2024-08-30T23:55:03.321559+0000","last_became_peered":"2024-08-30T23:55:03.321559+0000","last_unstale":"2024-08-30T23:58:07.074458+0000","last_undegraded":"2024-08-30T23:58:07.074458+0000","last_fullsized":"2024-08-30T23:58:07.074458+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:46.421676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:46.421676+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.14","version":"961'1658","reported_seq":3465,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:11.846588+0000","last_change":"2024-08-30T23:58:04.503326+0000","last_active":"2024-08-30T23:58:11.846588+0000","last_peered":"2024-08-30T23:58:11.846588+0000","last_clean":"2024-08-30T23:58:11.846588+0000","last_became_active":"2024-08-30T23:58:04.118179+0000","last_became_peered":"2024-08-30T23:58:04.118179+0000","last_unstale":"2024-08-30T23:58:11.846588+0000","last_undegraded":"2024-08-30T23:58:11.846588+0000","last_fullsized":"2024-08-30T23:58:11.846588+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1388","last_scrub_stamp":"2024-08-30T23:58:04.503242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.503242+0000","objects_scrubbed":845,"log_size":1658,"log_dups_size":0,"ondisk_log_size":1658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:23.237911+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73072640,"num_objects":1115,"num_object_clones":0,"num_object_copies":3345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":270,"num_write_kb":17280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":202,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.211580+0000","last_change":"2024-08-30T23:57:53.388159+0000","last_active":"2024-08-30T23:58:12.211580+0000","last_peered":"2024-08-30T23:58:12.211580+0000","last_clean":"2024-08-30T23:58:12.211580+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-30T23:58:12.211580+0000","last_undegraded":"2024-08-30T23:58:12.211580+0000","last_fullsized":"2024-08-30T23:58:12.211580+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:53.388110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.388110+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:00.379789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"961'1668","reported_seq":3523,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:11.336892+0000","last_change":"2024-08-30T23:58:03.245375+0000","last_active":"2024-08-30T23:58:11.336892+0000","last_peered":"2024-08-30T23:58:11.336892+0000","last_clean":"2024-08-30T23:58:11.336892+0000","last_became_active":"2024-08-30T23:58:03.115334+0000","last_became_peered":"2024-08-30T23:58:03.115334+0000","last_unstale":"2024-08-30T23:58:11.336892+0000","last_undegraded":"2024-08-30T23:58:11.336892+0000","last_fullsized":"2024-08-30T23:58:11.336892+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'1406","last_scrub_stamp":"2024-08-30T23:58:03.245266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:03.245266+0000","objects_scrubbed":849,"log_size":1668,"log_dups_size":0,"ondisk_log_size":1668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:21.629292+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72941568,"num_objects":1113,"num_object_clones":0,"num_object_copies":3339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":264,"num_write_kb":16896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,5],"acting":[11,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":202,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.211521+0000","last_change":"2024-08-30T23:57:32.122253+0000","last_active":"2024-08-30T23:58:12.211521+0000","last_peered":"2024-08-30T23:58:12.211521+0000","last_clean":"2024-08-30T23:58:12.211521+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-30T23:58:12.211521+0000","last_undegraded":"2024-08-30T23:58:12.211521+0000","last_fullsized":"2024-08-30T23:58:12.211521+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.122186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.122186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:53.855972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"962'1787","reported_seq":3708,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:14.785105+0000","last_change":"2024-08-30T23:58:04.315602+0000","last_active":"2024-08-30T23:58:14.785105+0000","last_peered":"2024-08-30T23:58:14.785105+0000","last_clean":"2024-08-30T23:58:14.785105+0000","last_became_active":"2024-08-30T23:58:04.118740+0000","last_became_peered":"2024-08-30T23:58:04.118740+0000","last_unstale":"2024-08-30T23:58:14.785105+0000","last_undegraded":"2024-08-30T23:58:14.785105+0000","last_fullsized":"2024-08-30T23:58:14.785105+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1428","last_scrub_stamp":"2024-08-30T23:58:04.315539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.315539+0000","objects_scrubbed":909,"log_size":1787,"log_dups_size":0,"ondisk_log_size":1787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:30.240827+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83099648,"num_objects":1268,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":359,"num_write_kb":22976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":190,"reported_epoch":961,"state":"active+undersized","last_fresh":"2024-08-30T23:58:07.104006+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-30T23:58:07.104006+0000","last_peered":"2024-08-30T23:58:07.104006+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-30T23:58:07.104006+0000","last_undegraded":"2024-08-30T23:58:07.104006+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:08.979632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"962'1748","reported_seq":1837,"reported_epoch":962,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:13.908266+0000","last_change":"2024-08-30T23:57:57.163576+0000","last_active":"2024-08-30T23:58:13.908266+0000","last_peered":"2024-08-30T23:58:13.908266+0000","last_clean":"2024-08-30T23:58:13.908266+0000","last_became_active":"2024-08-30T23:57:56.118862+0000","last_became_peered":"2024-08-30T23:57:56.118862+0000","last_unstale":"2024-08-30T23:58:13.908266+0000","last_undegraded":"2024-08-30T23:58:13.908266+0000","last_fullsized":"2024-08-30T23:58:13.908266+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"952'1150","last_scrub_stamp":"2024-08-30T23:57:57.163327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:57.163327+0000","objects_scrubbed":628,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:14.650398+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609280,"num_objects":1230,"num_object_clones":0,"num_object_copies":3690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1230,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":602,"num_write_kb":38528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,9],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"3,6,9","objects":1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":192,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.069500+0000","last_change":"2024-08-30T23:56:45.191296+0000","last_active":"2024-08-30T23:58:07.069500+0000","last_peered":"2024-08-30T23:58:07.069500+0000","last_clean":"2024-08-30T23:58:07.069500+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-30T23:58:07.069500+0000","last_undegraded":"2024-08-30T23:58:07.069500+0000","last_fullsized":"2024-08-30T23:58:07.069500+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.191245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.191245+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"961'1708","reported_seq":3567,"reported_epoch":961,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:11.746897+0000","last_change":"2024-08-30T23:58:05.325967+0000","last_active":"2024-08-30T23:58:11.746897+0000","last_peered":"2024-08-30T23:58:11.746897+0000","last_clean":"2024-08-30T23:58:11.746897+0000","last_became_active":"2024-08-30T23:58:04.118964+0000","last_became_peered":"2024-08-30T23:58:04.118964+0000","last_unstale":"2024-08-30T23:58:11.746897+0000","last_undegraded":"2024-08-30T23:58:11.746897+0000","last_fullsized":"2024-08-30T23:58:11.746897+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'1457","last_scrub_stamp":"2024-08-30T23:58:05.325770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:05.325770+0000","objects_scrubbed":883,"log_size":1708,"log_dups_size":0,"ondisk_log_size":1708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:20.384655+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75038720,"num_objects":1145,"num_object_clones":0,"num_object_copies":3435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1145,"num_objects_unfound":0,"num_objects_dirty":1145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":262,"num_write_kb":16768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":1145}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":415,"reported_epoch":962,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:12.191518+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-30T23:58:12.191518+0000","last_peered":"2024-08-30T23:58:12.191518+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-30T23:58:12.191518+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:14.186836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":202,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.211754+0000","last_change":"2024-08-30T23:58:11.457897+0000","last_active":"2024-08-30T23:58:12.211754+0000","last_peered":"2024-08-30T23:58:12.211754+0000","last_clean":"2024-08-30T23:58:12.211754+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-30T23:58:12.211754+0000","last_undegraded":"2024-08-30T23:58:12.211754+0000","last_fullsized":"2024-08-30T23:58:12.211754+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:11.457846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:11.457846+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:28.511611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":200,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.072050+0000","last_change":"2024-08-30T23:57:27.164513+0000","last_active":"2024-08-30T23:58:07.072050+0000","last_peered":"2024-08-30T23:58:07.072050+0000","last_clean":"2024-08-30T23:58:07.072050+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-30T23:58:07.072050+0000","last_undegraded":"2024-08-30T23:58:07.072050+0000","last_fullsized":"2024-08-30T23:58:07.072050+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:27.164462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:27.164462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:31.046456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":129,"reported_epoch":962,"state":"active+undersized","last_fresh":"2024-08-30T23:58:12.192018+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-30T23:58:12.192018+0000","last_peered":"2024-08-30T23:58:12.192018+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-30T23:58:12.192018+0000","last_undegraded":"2024-08-30T23:58:12.192018+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:13.203621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":247,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.074323+0000","last_change":"2024-08-30T23:57:14.423376+0000","last_active":"2024-08-30T23:58:07.074323+0000","last_peered":"2024-08-30T23:58:07.074323+0000","last_clean":"2024-08-30T23:58:07.074323+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:58:07.074323+0000","last_undegraded":"2024-08-30T23:58:07.074323+0000","last_fullsized":"2024-08-30T23:58:07.074323+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:14.423316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:14.423316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.050397+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":398,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.192162+0000","last_change":"2024-08-30T23:57:35.093839+0000","last_active":"2024-08-30T23:58:12.192162+0000","last_peered":"2024-08-30T23:58:12.192162+0000","last_clean":"2024-08-30T23:58:12.192162+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:58:12.192162+0000","last_undegraded":"2024-08-30T23:58:12.192162+0000","last_fullsized":"2024-08-30T23:58:12.192162+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:35.093787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.093787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:54.153963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"962'1802","reported_seq":3587,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:13.922950+0000","last_change":"2024-08-30T23:58:04.958546+0000","last_active":"2024-08-30T23:58:13.922950+0000","last_peered":"2024-08-30T23:58:13.922950+0000","last_clean":"2024-08-30T23:58:13.922950+0000","last_became_active":"2024-08-30T23:58:04.122431+0000","last_became_peered":"2024-08-30T23:58:04.122431+0000","last_unstale":"2024-08-30T23:58:13.922950+0000","last_undegraded":"2024-08-30T23:58:13.922950+0000","last_fullsized":"2024-08-30T23:58:13.922950+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1412","last_scrub_stamp":"2024-08-30T23:58:04.958468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.958468+0000","objects_scrubbed":844,"log_size":1802,"log_dups_size":0,"ondisk_log_size":1802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:27.300422+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80871424,"num_objects":1234,"num_object_clones":0,"num_object_copies":3702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":390,"num_write_kb":24960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":1019,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.211520+0000","last_change":"2024-08-30T23:58:04.552055+0000","last_active":"2024-08-30T23:58:12.211520+0000","last_peered":"2024-08-30T23:58:12.211520+0000","last_clean":"2024-08-30T23:58:12.211520+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:58:12.211520+0000","last_undegraded":"2024-08-30T23:58:12.211520+0000","last_fullsized":"2024-08-30T23:58:12.211520+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:04.552013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.552013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:20.924737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.18","version":"961'1772","reported_seq":3088,"reported_epoch":961,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:12.115750+0000","last_change":"2024-08-30T23:58:08.120423+0000","last_active":"2024-08-30T23:58:12.115750+0000","last_peered":"2024-08-30T23:58:12.115750+0000","last_clean":"2024-08-30T23:58:12.115750+0000","last_became_active":"2024-08-30T23:58:07.098389+0000","last_became_peered":"2024-08-30T23:58:07.098389+0000","last_unstale":"2024-08-30T23:58:12.115750+0000","last_undegraded":"2024-08-30T23:58:12.115750+0000","last_fullsized":"2024-08-30T23:58:12.115750+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"961'1604","last_scrub_stamp":"2024-08-30T23:58:08.120271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:08.120271+0000","objects_scrubbed":1049,"log_size":1772,"log_dups_size":0,"ondisk_log_size":1772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:30.455542+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150528,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1223,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":174,"num_write_kb":11136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5,9],"avail_no_missing":["10","5","9"],"object_location_counts":[{"shards":"5,9,10","objects":1223}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":386,"reported_epoch":962,"state":"active+undersized","last_fresh":"2024-08-30T23:58:12.191751+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-30T23:58:12.191751+0000","last_peered":"2024-08-30T23:58:12.191751+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-30T23:58:12.191751+0000","last_undegraded":"2024-08-30T23:58:12.191751+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"962'1847","reported_seq":1927,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:14.461157+0000","last_change":"2024-08-30T23:57:54.976951+0000","last_active":"2024-08-30T23:58:14.461157+0000","last_peered":"2024-08-30T23:58:14.461157+0000","last_clean":"2024-08-30T23:58:14.461157+0000","last_became_active":"2024-08-30T23:57:53.076655+0000","last_became_peered":"2024-08-30T23:57:53.076655+0000","last_unstale":"2024-08-30T23:58:14.461157+0000","last_undegraded":"2024-08-30T23:58:14.461157+0000","last_fullsized":"2024-08-30T23:58:14.461157+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1157","last_scrub_stamp":"2024-08-30T23:57:54.976868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:54.976868+0000","objects_scrubbed":595,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:07.211647+0000","scrub_duration":399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84213760,"num_objects":1285,"num_object_clones":0,"num_object_copies":3855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":690,"num_write_kb":44160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":801,"reported_epoch":962,"state":"active+undersized","last_fresh":"2024-08-30T23:58:12.207861+0000","last_change":"2024-08-30T23:55:59.173538+0000","last_active":"2024-08-30T23:58:12.207861+0000","last_peered":"2024-08-30T23:58:12.207861+0000","last_clean":"2024-08-30T23:55:57.750202+0000","last_became_active":"2024-08-30T23:55:59.173538+0000","last_became_peered":"2024-08-30T23:55:59.173538+0000","last_unstale":"2024-08-30T23:58:12.207861+0000","last_undegraded":"2024-08-30T23:58:12.207861+0000","last_fullsized":"2024-08-30T23:55:59.169546+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:53.668489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:53.668489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:12.890370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"962'2483","reported_seq":2545,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:13.930019+0000","last_change":"2024-08-30T23:57:36.062999+0000","last_active":"2024-08-30T23:58:13.930019+0000","last_peered":"2024-08-30T23:58:13.930019+0000","last_clean":"2024-08-30T23:58:13.930019+0000","last_became_active":"2024-08-30T23:57:36.062695+0000","last_became_peered":"2024-08-30T23:57:36.062695+0000","last_unstale":"2024-08-30T23:58:13.930019+0000","last_undegraded":"2024-08-30T23:58:13.930019+0000","last_fullsized":"2024-08-30T23:58:13.930019+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:05.096732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162725888,"num_objects":2483,"num_object_clones":0,"num_object_copies":7449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":158912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":251,"reported_epoch":961,"state":"active+undersized","last_fresh":"2024-08-30T23:58:07.074369+0000","last_change":"2024-08-30T23:55:59.171246+0000","last_active":"2024-08-30T23:58:07.074369+0000","last_peered":"2024-08-30T23:58:07.074369+0000","last_clean":"2024-08-30T23:55:57.160009+0000","last_became_active":"2024-08-30T23:55:59.171246+0000","last_became_peered":"2024-08-30T23:55:59.171246+0000","last_unstale":"2024-08-30T23:58:07.074369+0000","last_undegraded":"2024-08-30T23:58:07.074369+0000","last_fullsized":"2024-08-30T23:55:59.168006+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:36.270718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:36.270718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"962'2547","reported_seq":2606,"reported_epoch":962,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:14.802413+0000","last_change":"2024-08-30T23:57:42.988451+0000","last_active":"2024-08-30T23:58:14.802413+0000","last_peered":"2024-08-30T23:58:14.802413+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.059533+0000","last_became_peered":"2024-08-30T23:57:36.059533+0000","last_unstale":"2024-08-30T23:58:14.802413+0000","last_undegraded":"2024-08-30T23:57:42.988196+0000","last_fullsized":"2024-08-30T23:57:36.057835+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2547,"log_dups_size":0,"ondisk_log_size":2547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:03.589784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166920192,"num_objects":2547,"num_object_clones":0,"num_object_copies":7641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2547,"num_write_kb":163008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2547}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":593,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.069572+0000","last_change":"2024-08-30T23:56:48.173739+0000","last_active":"2024-08-30T23:58:07.069572+0000","last_peered":"2024-08-30T23:58:07.069572+0000","last_clean":"2024-08-30T23:58:07.069572+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-30T23:58:07.069572+0000","last_undegraded":"2024-08-30T23:58:07.069572+0000","last_fullsized":"2024-08-30T23:58:07.069572+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:48.173668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:48.173668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:13.790263+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"961'2375","reported_seq":2432,"reported_epoch":961,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:12.136668+0000","last_change":"2024-08-30T23:57:43.002149+0000","last_active":"2024-08-30T23:58:12.136668+0000","last_peered":"2024-08-30T23:58:12.136668+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.061016+0000","last_became_peered":"2024-08-30T23:57:36.061016+0000","last_unstale":"2024-08-30T23:58:12.136668+0000","last_undegraded":"2024-08-30T23:57:42.987440+0000","last_fullsized":"2024-08-30T23:57:36.054773+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2375,"log_dups_size":0,"ondisk_log_size":2375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:01.832597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155648000,"num_objects":2375,"num_object_clones":0,"num_object_copies":7125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2375,"num_write_kb":152000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":["10","1"],"object_location_counts":[{"shards":"1,10","objects":2375}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":567,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.069749+0000","last_change":"2024-08-30T23:56:16.234435+0000","last_active":"2024-08-30T23:58:07.069749+0000","last_peered":"2024-08-30T23:58:07.069749+0000","last_clean":"2024-08-30T23:58:07.069749+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-30T23:58:07.069749+0000","last_undegraded":"2024-08-30T23:58:07.069749+0000","last_fullsized":"2024-08-30T23:58:07.069749+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.234371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.234371+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"962'2657","reported_seq":2719,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:14.778607+0000","last_change":"2024-08-30T23:57:36.063702+0000","last_active":"2024-08-30T23:58:14.778607+0000","last_peered":"2024-08-30T23:58:14.778607+0000","last_clean":"2024-08-30T23:58:14.778607+0000","last_became_active":"2024-08-30T23:57:36.063425+0000","last_became_peered":"2024-08-30T23:57:36.063425+0000","last_unstale":"2024-08-30T23:58:14.778607+0000","last_undegraded":"2024-08-30T23:58:14.778607+0000","last_fullsized":"2024-08-30T23:58:14.778607+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2657,"log_dups_size":0,"ondisk_log_size":2657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:00.507197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174129152,"num_objects":2657,"num_object_clones":0,"num_object_copies":7971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":170048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1216,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.208150+0000","last_change":"2024-08-30T23:58:10.841604+0000","last_active":"2024-08-30T23:58:12.208150+0000","last_peered":"2024-08-30T23:58:12.208150+0000","last_clean":"2024-08-30T23:58:12.208150+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-30T23:58:12.208150+0000","last_undegraded":"2024-08-30T23:58:12.208150+0000","last_fullsized":"2024-08-30T23:58:12.208150+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:10.841542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:10.841542+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:36.684389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"961'1708","reported_seq":1788,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.117615+0000","last_change":"2024-08-30T23:57:56.128716+0000","last_active":"2024-08-30T23:58:12.117615+0000","last_peered":"2024-08-30T23:58:12.117615+0000","last_clean":"2024-08-30T23:58:12.117615+0000","last_became_active":"2024-08-30T23:57:53.081012+0000","last_became_peered":"2024-08-30T23:57:53.081012+0000","last_unstale":"2024-08-30T23:58:12.117615+0000","last_undegraded":"2024-08-30T23:58:12.117615+0000","last_fullsized":"2024-08-30T23:58:12.117615+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1143","last_scrub_stamp":"2024-08-30T23:57:56.128615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:56.128615+0000","objects_scrubbed":635,"log_size":1708,"log_dups_size":0,"ondisk_log_size":1708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:01.452832+0000","scrub_duration":1261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78643200,"num_objects":1200,"num_object_clones":0,"num_object_copies":3600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":565,"num_write_kb":36160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":552,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:07.069620+0000","last_change":"2024-08-30T23:57:08.049165+0000","last_active":"2024-08-30T23:58:07.069620+0000","last_peered":"2024-08-30T23:58:07.069620+0000","last_clean":"2024-08-30T23:58:07.069620+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:58:07.069620+0000","last_undegraded":"2024-08-30T23:58:07.069620+0000","last_fullsized":"2024-08-30T23:58:07.069620+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:08.049028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:08.049028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:18.034921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"962'2588","reported_seq":2647,"reported_epoch":962,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:14.785270+0000","last_change":"2024-08-30T23:57:43.007699+0000","last_active":"2024-08-30T23:58:14.785270+0000","last_peered":"2024-08-30T23:58:14.785270+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:36.059522+0000","last_became_peered":"2024-08-30T23:57:36.059522+0000","last_unstale":"2024-08-30T23:58:14.785270+0000","last_undegraded":"2024-08-30T23:57:43.002973+0000","last_fullsized":"2024-08-30T23:57:36.057850+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:43.386225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2588}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":2039,"reported_epoch":961,"state":"active+undersized","last_fresh":"2024-08-30T23:58:07.104020+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-30T23:58:07.104020+0000","last_peered":"2024-08-30T23:58:07.104020+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-30T23:58:07.104020+0000","last_undegraded":"2024-08-30T23:58:07.104020+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.b","version":"962'2699","reported_seq":2760,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:15.750292+0000","last_change":"2024-08-30T23:57:36.061979+0000","last_active":"2024-08-30T23:58:15.750292+0000","last_peered":"2024-08-30T23:58:15.750292+0000","last_clean":"2024-08-30T23:58:15.750292+0000","last_became_active":"2024-08-30T23:57:36.061597+0000","last_became_peered":"2024-08-30T23:57:36.061597+0000","last_unstale":"2024-08-30T23:58:15.750292+0000","last_undegraded":"2024-08-30T23:58:15.750292+0000","last_fullsized":"2024-08-30T23:58:15.750292+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2699,"log_dups_size":0,"ondisk_log_size":2699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:04.889278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176881664,"num_objects":2699,"num_object_clones":0,"num_object_copies":8097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":172736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":373,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.208012+0000","last_change":"2024-08-30T23:57:34.913534+0000","last_active":"2024-08-30T23:58:12.208012+0000","last_peered":"2024-08-30T23:58:12.208012+0000","last_clean":"2024-08-30T23:58:12.208012+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:58:12.208012+0000","last_undegraded":"2024-08-30T23:58:12.208012+0000","last_fullsized":"2024-08-30T23:58:12.208012+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:34.913482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:34.913482+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.633745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"961'1621","reported_seq":1700,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:11.822463+0000","last_change":"2024-08-30T23:57:53.615819+0000","last_active":"2024-08-30T23:58:11.822463+0000","last_peered":"2024-08-30T23:58:11.822463+0000","last_clean":"2024-08-30T23:58:11.822463+0000","last_became_active":"2024-08-30T23:57:53.070921+0000","last_became_peered":"2024-08-30T23:57:53.070921+0000","last_unstale":"2024-08-30T23:58:11.822463+0000","last_undegraded":"2024-08-30T23:58:11.822463+0000","last_fullsized":"2024-08-30T23:58:11.822463+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1072","last_scrub_stamp":"2024-08-30T23:57:53.615713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.615713+0000","objects_scrubbed":564,"log_size":1621,"log_dups_size":0,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:11.833234+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73072640,"num_objects":1115,"num_object_clones":0,"num_object_copies":3345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":551,"num_write_kb":35264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":1035,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.200590+0000","last_change":"2024-08-30T23:57:32.972014+0000","last_active":"2024-08-30T23:58:12.200590+0000","last_peered":"2024-08-30T23:58:12.200590+0000","last_clean":"2024-08-30T23:58:12.200590+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-30T23:58:12.200590+0000","last_undegraded":"2024-08-30T23:58:12.200590+0000","last_fullsized":"2024-08-30T23:58:12.200590+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.971963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.971963+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.937780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"961'1676","reported_seq":1740,"reported_epoch":961,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:11.237605+0000","last_change":"2024-08-30T23:57:53.082784+0000","last_active":"2024-08-30T23:58:11.237605+0000","last_peered":"2024-08-30T23:58:11.237605+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:53.076653+0000","last_became_peered":"2024-08-30T23:57:53.076653+0000","last_unstale":"2024-08-30T23:58:11.237605+0000","last_undegraded":"2024-08-30T23:57:53.073210+0000","last_fullsized":"2024-08-30T23:57:53.072333+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1676,"log_dups_size":0,"ondisk_log_size":1676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:15.108653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73531392,"num_objects":1122,"num_object_clones":0,"num_object_copies":3366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":71808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"5,11","objects":1122}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":1009,"reported_epoch":961,"state":"active+undersized","last_fresh":"2024-08-30T23:58:07.104281+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-30T23:58:07.104281+0000","last_peered":"2024-08-30T23:58:07.104281+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-30T23:58:07.104281+0000","last_undegraded":"2024-08-30T23:58:07.104281+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:10.919337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"962'1807","reported_seq":1887,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:14.743524+0000","last_change":"2024-08-30T23:57:55.058608+0000","last_active":"2024-08-30T23:58:14.743524+0000","last_peered":"2024-08-30T23:58:14.743524+0000","last_clean":"2024-08-30T23:58:14.743524+0000","last_became_active":"2024-08-30T23:57:53.081131+0000","last_became_peered":"2024-08-30T23:57:53.081131+0000","last_unstale":"2024-08-30T23:58:14.743524+0000","last_undegraded":"2024-08-30T23:58:14.743524+0000","last_fullsized":"2024-08-30T23:58:14.743524+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1118","last_scrub_stamp":"2024-08-30T23:57:55.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:55.058465+0000","objects_scrubbed":594,"log_size":1807,"log_dups_size":0,"ondisk_log_size":1807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:04.899523+0000","scrub_duration":990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148224,"num_objects":1284,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":690,"num_write_kb":44160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":940,"reported_epoch":961,"state":"active+undersized","last_fresh":"2024-08-30T23:58:07.104278+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-30T23:58:07.104278+0000","last_peered":"2024-08-30T23:58:07.104278+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-30T23:58:07.104278+0000","last_undegraded":"2024-08-30T23:58:07.104278+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"961'1662","reported_seq":1723,"reported_epoch":961,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:12.028636+0000","last_change":"2024-08-30T23:57:53.077387+0000","last_active":"2024-08-30T23:58:12.028636+0000","last_peered":"2024-08-30T23:58:12.028636+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:53.077387+0000","last_became_peered":"2024-08-30T23:57:53.077387+0000","last_unstale":"2024-08-30T23:58:12.028636+0000","last_undegraded":"2024-08-30T23:57:53.068813+0000","last_fullsized":"2024-08-30T23:57:53.068689+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":1662,"log_dups_size":0,"ondisk_log_size":1662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73793536,"num_objects":1126,"num_object_clones":0,"num_object_copies":3378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1126,"num_write_kb":72064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","3"],"object_location_counts":[{"shards":"3,10","objects":1126}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1227,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.208141+0000","last_change":"2024-08-30T23:57:52.872669+0000","last_active":"2024-08-30T23:58:12.208141+0000","last_peered":"2024-08-30T23:58:12.208141+0000","last_clean":"2024-08-30T23:58:12.208141+0000","last_became_active":"2024-08-30T23:57:36.158625+0000","last_became_peered":"2024-08-30T23:57:36.158625+0000","last_unstale":"2024-08-30T23:58:12.208141+0000","last_undegraded":"2024-08-30T23:58:12.208141+0000","last_fullsized":"2024-08-30T23:58:12.208141+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:52.872607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.872607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:21.985915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"962'1721","reported_seq":1804,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.244927+0000","last_change":"2024-08-30T23:57:53.985025+0000","last_active":"2024-08-30T23:58:12.244927+0000","last_peered":"2024-08-30T23:58:12.244927+0000","last_clean":"2024-08-30T23:58:12.244927+0000","last_became_active":"2024-08-30T23:57:53.077332+0000","last_became_peered":"2024-08-30T23:57:53.077332+0000","last_unstale":"2024-08-30T23:58:12.244927+0000","last_undegraded":"2024-08-30T23:58:12.244927+0000","last_fullsized":"2024-08-30T23:58:12.244927+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1159","last_scrub_stamp":"2024-08-30T23:57:53.984940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.984940+0000","objects_scrubbed":625,"log_size":1721,"log_dups_size":0,"ondisk_log_size":1721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:08.737107+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77856768,"num_objects":1188,"num_object_clones":0,"num_object_copies":3564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":563,"num_write_kb":36032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":409,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.191634+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-30T23:58:12.191634+0000","last_peered":"2024-08-30T23:58:12.191634+0000","last_clean":"2024-08-30T23:58:12.191634+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-30T23:58:12.191634+0000","last_undegraded":"2024-08-30T23:58:12.191634+0000","last_fullsized":"2024-08-30T23:58:12.191634+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"962'1781","reported_seq":1863,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:13.939662+0000","last_change":"2024-08-30T23:57:54.994039+0000","last_active":"2024-08-30T23:58:13.939662+0000","last_peered":"2024-08-30T23:58:13.939662+0000","last_clean":"2024-08-30T23:58:13.939662+0000","last_became_active":"2024-08-30T23:57:53.072273+0000","last_became_peered":"2024-08-30T23:57:53.072273+0000","last_unstale":"2024-08-30T23:58:13.939662+0000","last_undegraded":"2024-08-30T23:58:13.939662+0000","last_fullsized":"2024-08-30T23:58:13.939662+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1110","last_scrub_stamp":"2024-08-30T23:57:54.993958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:54.993958+0000","objects_scrubbed":574,"log_size":1781,"log_dups_size":0,"ondisk_log_size":1781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:12.322103+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81592320,"num_objects":1245,"num_object_clones":0,"num_object_copies":3735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":671,"num_write_kb":42944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1006,"reported_epoch":962,"state":"active+undersized","last_fresh":"2024-08-30T23:58:12.197972+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-30T23:58:12.197972+0000","last_peered":"2024-08-30T23:58:12.197972+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-30T23:58:12.197972+0000","last_undegraded":"2024-08-30T23:58:12.197972+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:14.463916+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"962'1752","reported_seq":1835,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:13.895112+0000","last_change":"2024-08-30T23:57:54.047513+0000","last_active":"2024-08-30T23:58:13.895112+0000","last_peered":"2024-08-30T23:58:13.895112+0000","last_clean":"2024-08-30T23:58:13.895112+0000","last_became_active":"2024-08-30T23:57:53.070711+0000","last_became_peered":"2024-08-30T23:57:53.070711+0000","last_unstale":"2024-08-30T23:58:13.895112+0000","last_undegraded":"2024-08-30T23:58:13.895112+0000","last_fullsized":"2024-08-30T23:58:13.895112+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1114","last_scrub_stamp":"2024-08-30T23:57:54.047415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:54.047415+0000","objects_scrubbed":576,"log_size":1752,"log_dups_size":0,"ondisk_log_size":1752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:15.252096+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":1216,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":640,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1151,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:12.197873+0000","last_change":"2024-08-30T23:58:08.502691+0000","last_active":"2024-08-30T23:58:12.197873+0000","last_peered":"2024-08-30T23:58:12.197873+0000","last_clean":"2024-08-30T23:58:12.197873+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-30T23:58:12.197873+0000","last_undegraded":"2024-08-30T23:58:12.197873+0000","last_fullsized":"2024-08-30T23:58:12.197873+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:08.502643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:08.502643+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:23.323808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"962'1834","reported_seq":1917,"reported_epoch":962,"state":"active+clean","last_fresh":"2024-08-30T23:58:14.449411+0000","last_change":"2024-08-30T23:57:53.737093+0000","last_active":"2024-08-30T23:58:14.449411+0000","last_peered":"2024-08-30T23:58:14.449411+0000","last_clean":"2024-08-30T23:58:14.449411+0000","last_became_active":"2024-08-30T23:57:53.074222+0000","last_became_peered":"2024-08-30T23:57:53.074222+0000","last_unstale":"2024-08-30T23:58:14.449411+0000","last_undegraded":"2024-08-30T23:58:14.449411+0000","last_fullsized":"2024-08-30T23:58:14.449411+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1132","last_scrub_stamp":"2024-08-30T23:57:53.736917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.736917+0000","objects_scrubbed":594,"log_size":1834,"log_dups_size":0,"ondisk_log_size":1834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:14.720979+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85327872,"num_objects":1302,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":708,"num_write_kb":45312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":261,"reported_epoch":961,"state":"active+clean","last_fresh":"2024-08-30T23:58:11.144659+0000","last_change":"2024-08-30T23:58:11.144659+0000","last_active":"2024-08-30T23:58:11.144659+0000","last_peered":"2024-08-30T23:58:11.144659+0000","last_clean":"2024-08-30T23:58:11.144659+0000","last_became_active":"2024-08-30T23:55:01.289376+0000","last_became_peered":"2024-08-30T23:55:01.289376+0000","last_unstale":"2024-08-30T23:58:11.144659+0000","last_undegraded":"2024-08-30T23:58:11.144659+0000","last_fullsized":"2024-08-30T23:58:11.144659+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:11.144606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:11.144606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:32.000093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1098,"reported_epoch":962,"state":"active+undersized","last_fresh":"2024-08-30T23:58:12.198107+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-30T23:58:12.198107+0000","last_peered":"2024-08-30T23:58:12.198107+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-30T23:58:12.198107+0000","last_undegraded":"2024-08-30T23:58:12.198107+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:12.534527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":2595028992,"num_objects":39597,"num_object_clones":0,"num_object_copies":118791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9758,"num_objects_misplaced":5009,"num_objects_unfound":0,"num_objects_dirty":39597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26091,"num_write_kb":1669824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7263158272,"data_stored":7263158272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50358,"ondisk_log_size":50358,"up":69,"acting":73,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":81,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172956,"kb_used_data":501772,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92572196,"statfs":{"total":95995035648,"available":94793928704,"internally_reserved":0,"allocated":513814528,"data_stored":513025643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":687274436},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449502,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1387100,"kb_used_data":742284,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92358052,"statfs":{"total":95995035648,"available":94574645248,"internally_reserved":0,"allocated":760098816,"data_stored":759309931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1665836,"kb_used_data":900248,"kb_used_omap":1,"kb_used_meta":765566,"kb_avail":92079316,"statfs":{"total":95995035648,"available":94289219584,"internally_reserved":0,"allocated":921853952,"data_stored":921060750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":783940032},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972774,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":929512,"kb_used_data":424920,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92815640,"statfs":{"total":95995035648,"available":95043215360,"internally_reserved":0,"allocated":435118080,"data_stored":434324878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528751,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859100,"kb_used_data":368396,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92886052,"statfs":{"total":95995035648,"available":95115317248,"internally_reserved":0,"allocated":377237504,"data_stored":376448619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775255,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851248,"kb_used_data":1508832,"kb_used_omap":5,"kb_used_meta":342394,"kb_avail":91893904,"statfs":{"total":95995035648,"available":94099357696,"internally_reserved":0,"allocated":1545043968,"data_stored":1544247214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":350611530},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953914,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1083548,"kb_used_data":384588,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92661604,"statfs":{"total":95995035648,"available":94885482496,"internally_reserved":0,"allocated":393818112,"data_stored":393029227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150140,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1640616,"kb_used_data":725080,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92104536,"statfs":{"total":95995035648,"available":94315044864,"internally_reserved":0,"allocated":742481920,"data_stored":741688718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":937490887},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745268,"kb_used_data":78240,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92999884,"statfs":{"total":95995035648,"available":95231881216,"internally_reserved":0,"allocated":80117760,"data_stored":79321006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830594,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1722588,"kb_used_data":1167116,"kb_used_omap":87,"kb_used_meta":555368,"kb_avail":92022564,"statfs":{"total":95995035648,"available":94231105536,"internally_reserved":0,"allocated":1195126784,"data_stored":1194337899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":568696991},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738829,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019048,"kb_used_data":234008,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92726104,"statfs":{"total":95995035648,"available":94951530496,"internally_reserved":0,"allocated":239624192,"data_stored":238830990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243335168,"data_stored":243335168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1210056704,"data_stored":1210056704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79822848,"data_stored":79822848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741277696,"data_stored":741277696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":398196736,"data_stored":398196736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1569062912,"data_stored":1569062912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":380305408,"data_stored":380305408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":440205312,"data_stored":440205312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":925564928,"data_stored":925564928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":761069568,"data_stored":761069568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":514260992,"data_stored":514260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:58:18.406 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-30T23:58:18.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_7 pg_num 36 2024-08-30T23:58:18.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:18.434 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:18 smithi106 ceph-mon[16603]: from='client.19248 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:58:18.434 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1038442989' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:18.434 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:18 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:18.438 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:18 smithi136 ceph-mon[20207]: from='client.19248 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:58:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1038442989' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:18 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:18.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:18.561 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:18 smithi173 ceph-mon[20554]: from='client.19248 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:58:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1038442989' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:18.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:18 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:18.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:18.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:18.791 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:18.816 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:18.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:18.947 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:18.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:18.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:18.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 36 16 47502 47486 82.4336 86.3125 0.00203755 0.0121232 2024-08-30T23:58:19.049 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:19.075 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:19.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:19.177 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:19.203 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:19.305 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:19.326 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:19.342 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:19 smithi106 ceph-mon[16603]: pgmap v2136: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 925 op/s; 10732/127548 objects degraded (8.414%); 5493/127548 objects misplaced (4.307%) 2024-08-30T23:58:19.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:19 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:19.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:19 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:58:19.343 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:19 smithi106 ceph-mon[16603]: osdmap e964: 12 total, 11 up, 12 in 2024-08-30T23:58:19.428 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:19.455 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:19.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:19 smithi136 ceph-mon[20207]: pgmap v2136: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 925 op/s; 10732/127548 objects degraded (8.414%); 5493/127548 objects misplaced (4.307%) 2024-08-30T23:58:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:19 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:19 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:58:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:19 smithi136 ceph-mon[20207]: osdmap e964: 12 total, 11 up, 12 in 2024-08-30T23:58:19.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:19.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:19 smithi173 ceph-mon[20554]: pgmap v2136: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 925 op/s; 10732/127548 objects degraded (8.414%); 5493/127548 objects misplaced (4.307%) 2024-08-30T23:58:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:19 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:19 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:58:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:19 smithi173 ceph-mon[20554]: osdmap e964: 12 total, 11 up, 12 in 2024-08-30T23:58:19.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:19.644 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:58:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:58:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.644 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:48284 9.c 9:383d80f6:::benchmark_data_smithi173_160189_object48283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:19.634406+0000", 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.008977545, 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0090124780000000008, 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48284 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634406+0000", 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634406+0000", 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634409+0000", 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.3709999999999998e-06 2024-08-30T23:58:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634457+0000", 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8634000000000002e-05 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634476+0000", 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.8779000000000001e-05 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634485+0000", 2024-08-30T23:58:19.647 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.3989999999999993e-06 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634556+0000", 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0704999999999995e-05 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634748+0000", 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000191845 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.634930+0000", 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00018208699999999999 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.636901+0000", 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0019711970000000001 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.637357+0000", 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00045623400000000002 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:48292 9.c 9:38a2fed2:::benchmark_data_smithi173_160189_object48291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:19.641035+0000", 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0023489359999999998, 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0024676849999999998, 2024-08-30T23:58:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48292 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641035+0000", 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641035+0000", 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641036+0000", 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3799999999999999e-06 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641096+0000", 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0149999999999998e-05 2024-08-30T23:58:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641112+0000", 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.5435999999999999e-05 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641120+0000", 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3999999999999992e-06 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641179+0000", 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8567999999999999e-05 2024-08-30T23:58:19.653 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641447+0000", 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00026817799999999999 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,9", 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.641572+0000", 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012465499999999999 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.654 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.28405.0:48292 9.c e964/938)", 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:19.643356+0000", 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 2.7929999999999999e-05, 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00020305, 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643356+0000", 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.655 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643356+0000", 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643357+0000", 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4e-07 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:19.656 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643362+0000", 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6339999999999998e-06 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643379+0000", 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6705999999999999e-05 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643386+0000", 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5590000000000002e-06 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:19.657 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.28405.0:48284 9.c e964/938)", 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:19.643488+0000", 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 4294967295.9998956, 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1909000000000004e-05, 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:19.658 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643488+0000", 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643488+0000", 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.659 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643489+0000", 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5000000000000003e-07 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643492+0000", 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8999999999999999e-06 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643509+0000", 2024-08-30T23:58:19.660 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6977e-05 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.643516+0000", 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4760000000000003e-06 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:48252 9.2 9:442189f7:::benchmark_data_smithi173_160189_object48251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:19.610903+0000", 2024-08-30T23:58:19.661 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.032481072, 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.032705157999999998, 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48252 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:19.662 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.610903+0000", 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.610903+0000", 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.610903+0000", 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6400000000000002e-07 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.663 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.610964+0000", 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0946999999999998e-05 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.610972+0000", 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9370000000000004e-06 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.610977+0000", 2024-08-30T23:58:19.664 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6920000000000001e-06 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.611022+0000", 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5055000000000001e-05 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.611117+0000", 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4993000000000003e-05 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,9", 2024-08-30T23:58:19.665 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.611203+0000", 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6136000000000003e-05 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.612162+0000", 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00095869200000000003 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.628401+0000", 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.016239144 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.666 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:48252 9.2 9:442189f7:::benchmark_data_smithi173_160189_object48251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e964)", 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:19.616510+0000", 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.026873871000000001, 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.027150507000000001, 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for ondisk", 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:19.667 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 48252 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616510+0000", 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616510+0000", 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616510+0000", 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.96e-07 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616571+0000", 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0627000000000001e-05 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616576+0000", 2024-08-30T23:58:19.669 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5759999999999996e-06 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616580+0000", 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.6890000000000002e-06 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616607+0000", 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6896000000000001e-05 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for ondisk", 2024-08-30T23:58:19.670 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:19.616635+0000", 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.815e-05 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 6 2024-08-30T23:58:19.671 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:58:19.682 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:19.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.709 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:19.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:19.837 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:19.877 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:19.877 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:19.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:19.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:19.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:19.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 37 16 48924 48908 82.6075 88.875 0.00217696 0.0120989 2024-08-30T23:58:20.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:20.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:20.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:20.216 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:20.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:20.345 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:20.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:20.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:20.477 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:20.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:20.609 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:20.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:20.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:20.738 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:20.840 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:20.868 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:20.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:20.982 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 38 15 50909 50894 83.6998 124.125 0.0522779 0.0119389 2024-08-30T23:58:20.994 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:20.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 9 pg_num to 36 2024-08-30T23:58:21.095 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:21.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:58:21.116 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:58:21.121 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:21.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:21.256 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:21.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:21.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:21 smithi106 ceph-mon[16603]: pgmap v2138: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 11613/138318 objects degraded (8.396%); 5745/138318 objects misplaced (4.153%) 2024-08-30T23:58:21.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3443866735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"} : dispatch 2024-08-30T23:58:21.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:21 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"} : dispatch 2024-08-30T23:58:21.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 11613/138318 objects degraded (8.396%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:21.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:21 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"}]': finished 2024-08-30T23:58:21.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:21 smithi106 ceph-mon[16603]: osdmap e965: 12 total, 11 up, 12 in 2024-08-30T23:58:21.387 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.389 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:21.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:21.515 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:21.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:21 smithi136 ceph-mon[20207]: pgmap v2138: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 11613/138318 objects degraded (8.396%); 5745/138318 objects misplaced (4.153%) 2024-08-30T23:58:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3443866735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"} : dispatch 2024-08-30T23:58:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:21 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"} : dispatch 2024-08-30T23:58:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 11613/138318 objects degraded (8.396%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:21 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"}]': finished 2024-08-30T23:58:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:21 smithi136 ceph-mon[20207]: osdmap e965: 12 total, 11 up, 12 in 2024-08-30T23:58:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:21 smithi173 ceph-mon[20554]: pgmap v2138: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s; 11613/138318 objects degraded (8.396%); 5745/138318 objects misplaced (4.153%) 2024-08-30T23:58:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3443866735' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"} : dispatch 2024-08-30T23:58:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:21 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"} : dispatch 2024-08-30T23:58:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 11613/138318 objects degraded (8.396%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:21 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "36"}]': finished 2024-08-30T23:58:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:21 smithi173 ceph-mon[20554]: osdmap e965: 12 total, 11 up, 12 in 2024-08-30T23:58:21.617 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:21.641 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:21.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:21.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:21.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:21.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:21.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:21.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 39 16 52672 52656 84.3771 110.125 0.00584875 0.0118419 2024-08-30T23:58:21.999 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:22.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.031 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:22.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.156 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:22.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:22.220 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:58:22.221 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:58:22.258 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:22.260 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:22.260 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:58:22.260 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:58:22.276 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:22.299 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:58:22.299 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:58:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.325 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:22.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:22.337 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:58:22.337 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:58:22.358 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:58:22.358 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:58:22.378 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:58:22.379 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:58:22.399 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:58:22.400 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:58:22.400 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:58:22.412 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:22 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:22.412 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:22 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:22.412 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3180175661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:22.427 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:22.448 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:22.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:22.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:58:22.518 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:58:22.539 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:58:22.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:58:22.583 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:58:22.583 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:58:22.584 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:58:22.584 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:58:22.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:22.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:22 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:22.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:22 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:22.585 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3180175661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:22.585 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:22 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:22.585 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:22 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:22.586 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3180175661' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:22.606 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:22.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:22.730 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:22.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:22.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:22.955 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:22.978 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:22.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:22.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:58:22.983666+0000 min lat: 0.000982985 max lat: 2.30386 avg lat: 0.0118511 2024-08-30T23:58:22.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:58:22.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 40 16 53079 53063 82.9035 25.4375 0.384883 0.0118511 2024-08-30T23:58:23.079 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:23.102 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:23.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:23.218 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:58:23.229 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:23.332 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:23.361 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:58:23.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:58:23.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:23.488 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:23 smithi136 ceph-mon[20207]: pgmap v2140: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s; 11858/144153 objects degraded (8.226%); 6246/144153 objects misplaced (4.333%) 2024-08-30T23:58:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:23 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"}]': finished 2024-08-30T23:58:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:23 smithi136 ceph-mon[20207]: osdmap e966: 12 total, 11 up, 12 in 2024-08-30T23:58:23.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2171146737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:58:23.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:23 smithi173 ceph-mon[20554]: pgmap v2140: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s; 11858/144153 objects degraded (8.226%); 6246/144153 objects misplaced (4.333%) 2024-08-30T23:58:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:23 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"}]': finished 2024-08-30T23:58:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:23 smithi173 ceph-mon[20554]: osdmap e966: 12 total, 11 up, 12 in 2024-08-30T23:58:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2171146737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:58:23.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:23 smithi106 ceph-mon[16603]: pgmap v2140: 57 pgs: 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s; 11858/144153 objects degraded (8.226%); 6246/144153 objects misplaced (4.333%) 2024-08-30T23:58:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:23 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "36"}]': finished 2024-08-30T23:58:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:23 smithi106 ceph-mon[16603]: osdmap e966: 12 total, 11 up, 12 in 2024-08-30T23:58:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2171146737' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:58:23.614 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:23.716 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:23.741 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:23.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:23.872 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:23.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:23.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:23.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 41 16 54225 54209 82.6282 71.625 0.0127902 0.0120975 2024-08-30T23:58:24.002 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:24.104 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:24.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:24.234 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:24.260 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:24.282 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:24 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:24.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2171146737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:58:24.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:24 smithi136 ceph-mon[20207]: osdmap e967: 12 total, 11 up, 12 in 2024-08-30T23:58:24.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:24 smithi136 ceph-mon[20207]: 9.b scrub starts 2024-08-30T23:58:24.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:24 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:24.283 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:24 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:24 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2171146737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:58:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:24 smithi106 ceph-mon[16603]: osdmap e967: 12 total, 11 up, 12 in 2024-08-30T23:58:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:24 smithi106 ceph-mon[16603]: 9.b scrub starts 2024-08-30T23:58:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:24 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:24 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:24.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:24.384 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:24.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.510 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:24.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:24.548 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:24.548 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:24 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2171146737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:24 smithi173 ceph-mon[20554]: osdmap e967: 12 total, 11 up, 12 in 2024-08-30T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:24 smithi173 ceph-mon[20554]: 9.b scrub starts 2024-08-30T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:24 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:24 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"} : dispatch 2024-08-30T23:58:24.612 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.635 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:24.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:24.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:24.759 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:24.861 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:24.882 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:24.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:24.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 42 16 54666 54650 81.317 27.5625 0.00257893 0.0121289 2024-08-30T23:58:24.985 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:25.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:25.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:25.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:25.232 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:25.255 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:25.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: 9.c scrub starts 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: pgmap v2143: 67 pgs: 1 peering, 10 unknown, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 35 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 12662/150249 objects degraded (8.427%); 6519/150249 objects misplaced (4.339%) 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: 9.b scrub ok 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: 9.0 scrub starts 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"}]': finished 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: osdmap e968: 12 total, 11 up, 12 in 2024-08-30T23:58:25.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: 9.3 scrub starts 2024-08-30T23:58:25.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: 9.0 scrub ok 2024-08-30T23:58:25.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:25 smithi136 ceph-mon[20207]: 9.3 scrub ok 2024-08-30T23:58:25.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:25.386 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:25.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:58:25.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:25.510 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: 9.c scrub starts 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: pgmap v2143: 67 pgs: 1 peering, 10 unknown, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 35 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 12662/150249 objects degraded (8.427%); 6519/150249 objects misplaced (4.339%) 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: 9.b scrub ok 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: 9.0 scrub starts 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"}]': finished 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: osdmap e968: 12 total, 11 up, 12 in 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: 9.3 scrub starts 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: 9.0 scrub ok 2024-08-30T23:58:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:25 smithi173 ceph-mon[20554]: 9.3 scrub ok 2024-08-30T23:58:25.595 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:25.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: 9.c scrub starts 2024-08-30T23:58:25.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: pgmap v2143: 67 pgs: 1 peering, 10 unknown, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 35 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 12662/150249 objects degraded (8.427%); 6519/150249 objects misplaced (4.339%) 2024-08-30T23:58:25.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: 9.b scrub ok 2024-08-30T23:58:25.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: 9.0 scrub starts 2024-08-30T23:58:25.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:25.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:25.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "36"}]': finished 2024-08-30T23:58:25.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: osdmap e968: 12 total, 11 up, 12 in 2024-08-30T23:58:25.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: 9.3 scrub starts 2024-08-30T23:58:25.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: 9.0 scrub ok 2024-08-30T23:58:25.602 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:25 smithi106 ceph-mon[16603]: 9.3 scrub ok 2024-08-30T23:58:25.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:25.637 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:25.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:25.739 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:25.761 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:25.863 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:25.885 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:25.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:25.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 43 16 54666 54650 79.426 0 - 0.0121289 2024-08-30T23:58:25.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.009 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:26.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:26.099 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:26.099 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:26.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:26.134 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:26.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:26.270 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:26.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:26.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:26.467 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:58:26.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: 9.2 scrub starts 2024-08-30T23:58:26.503 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: 9.c scrub ok 2024-08-30T23:58:26.503 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: 9.2 scrub ok 2024-08-30T23:58:26.503 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: 9.f scrub starts 2024-08-30T23:58:26.503 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: osdmap e969: 12 total, 11 up, 12 in 2024-08-30T23:58:26.503 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 12662/150249 objects degraded (8.427%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:26.503 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:26 smithi106 ceph-mon[16603]: osdmap e970: 12 total, 11 up, 12 in 2024-08-30T23:58:26.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: 9.2 scrub starts 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: 9.c scrub ok 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: 9.2 scrub ok 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: 9.f scrub starts 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: osdmap e969: 12 total, 11 up, 12 in 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 12662/150249 objects degraded (8.427%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:26 smithi136 ceph-mon[20207]: osdmap e970: 12 total, 11 up, 12 in 2024-08-30T23:58:26.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: 9.2 scrub starts 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: 9.c scrub ok 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: 9.2 scrub ok 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: 9.f scrub starts 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: osdmap e969: 12 total, 11 up, 12 in 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 12662/150249 objects degraded (8.427%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:26 smithi173 ceph-mon[20554]: osdmap e970: 12 total, 11 up, 12 in 2024-08-30T23:58:26.653 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:26.682 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:26.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:26.807 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:26.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:26.817 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:26.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-30T23:58:26.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:26.941 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:26.942 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:58:26.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:58:26.942 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-30T23:58:26.942 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-30T23:58:26.942 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-30T23:58:26.942 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-30T23:58:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:26.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:26.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 44 16 55325 55309 78.5569 20.5938 0.00884042 0.0127181 2024-08-30T23:58:27.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:27.068 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:27.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:27.178 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:27.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:27.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:27.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:27.429 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:27.464 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:27.501 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:27 smithi136 ceph-mon[20207]: pgmap v2146: 67 pgs: 2 active+clean+scrubbing, 1 peering, 9 unknown, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 34 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9995/140736 objects degraded (7.102%); 6837/140736 objects misplaced (4.858%) 2024-08-30T23:58:27.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:27 smithi136 ceph-mon[20207]: 9.1 scrub starts 2024-08-30T23:58:27.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:27 smithi136 ceph-mon[20207]: 9.1 scrub ok 2024-08-30T23:58:27.502 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:27 smithi136 ceph-mon[20207]: 9.f scrub ok 2024-08-30T23:58:27.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:27.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:27 smithi106 ceph-mon[16603]: pgmap v2146: 67 pgs: 2 active+clean+scrubbing, 1 peering, 9 unknown, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 34 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9995/140736 objects degraded (7.102%); 6837/140736 objects misplaced (4.858%) 2024-08-30T23:58:27.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:27 smithi106 ceph-mon[16603]: 9.1 scrub starts 2024-08-30T23:58:27.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:27 smithi106 ceph-mon[16603]: 9.1 scrub ok 2024-08-30T23:58:27.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:27 smithi106 ceph-mon[16603]: 9.f scrub ok 2024-08-30T23:58:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:27 smithi173 ceph-mon[20554]: pgmap v2146: 67 pgs: 2 active+clean+scrubbing, 1 peering, 9 unknown, 4 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 34 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9995/140736 objects degraded (7.102%); 6837/140736 objects misplaced (4.858%) 2024-08-30T23:58:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:27 smithi173 ceph-mon[20554]: 9.1 scrub starts 2024-08-30T23:58:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:27 smithi173 ceph-mon[20554]: 9.1 scrub ok 2024-08-30T23:58:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:27 smithi173 ceph-mon[20554]: 9.f scrub ok 2024-08-30T23:58:27.594 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:27.691 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:27.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:27.737 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:27.744 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:27.758 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:58:27.758 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:58:27.765 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:27.766 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:27.789 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:58:27.789 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:58:27.826 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:58:27.826 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:58:27.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:27.861 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:58:27.861 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:58:27.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:27.895 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:58:27.895 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:58:27.925 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:58:27.925 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:58:27.957 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:58:27.958 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:58:27.959 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:58:27.969 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:27.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 45 16 56490 56474 78.4291 72.8125 0.0222632 0.0127433 2024-08-30T23:58:27.994 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:58:27.995 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:58:27.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:27.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:28.026 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:58:28.026 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:58:28.054 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:58:28.055 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:58:28.087 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:58:28.087 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:58:28.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:28.132 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:28.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:28.262 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1872471189' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:28.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.397 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:28.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:28.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-30T23:58:28.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:28.532 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:28.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1872471189' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:28.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1872471189' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:28.634 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:28.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:28.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:28.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:28.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:28.924 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:28.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:28.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 46 16 57666 57650 78.3218 73.5 0.00216179 0.0127574 2024-08-30T23:58:29.014 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:58:29.014 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "" 2024-08-30T23:58:29.014 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:58:29.026 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:29.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:29.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:29.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.195 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:29.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:29.270 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:58:29.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:29.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:29.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:29.425 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:29.457 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:29 smithi136 ceph-mon[20207]: pgmap v2148: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 5 unknown, 4 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 35 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12022/154155 objects degraded (7.799%); 19413/154155 objects misplaced (12.593%); 1.6 MiB/s, 25 objects/s recovering 2024-08-30T23:58:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2034894044' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:29.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:29.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:29 smithi173 ceph-mon[20554]: pgmap v2148: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 5 unknown, 4 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 35 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12022/154155 objects degraded (7.799%); 19413/154155 objects misplaced (12.593%); 1.6 MiB/s, 25 objects/s recovering 2024-08-30T23:58:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2034894044' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:29.588 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:29.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:29 smithi106 ceph-mon[16603]: pgmap v2148: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 5 unknown, 4 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 35 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 12022/154155 objects degraded (7.799%); 19413/154155 objects misplaced (12.593%); 1.6 MiB/s, 25 objects/s recovering 2024-08-30T23:58:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2034894044' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:58:29.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:29.716 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:29.818 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:29.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:29.944 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:29.968 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:29.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:29.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 47 16 58568 58552 77.8547 56.375 0.00239683 0.0128131 2024-08-30T23:58:30.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:30.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:30.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:30.212 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:30.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:30.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:30.358 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:30.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:30.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:30 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:58:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2034894044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:58:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:30 smithi136 ceph-mon[20207]: osdmap e971: 12 total, 11 up, 12 in 2024-08-30T23:58:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:30 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:58:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2034894044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:58:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:30 smithi173 ceph-mon[20554]: osdmap e971: 12 total, 11 up, 12 in 2024-08-30T23:58:30.587 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:30.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:30 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:58:30.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2034894044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:58:30.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:30 smithi106 ceph-mon[16603]: osdmap e971: 12 total, 11 up, 12 in 2024-08-30T23:58:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.622 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:30.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:30.724 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.751 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:30.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:30.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:30.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:30.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:30.960 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:58:30.960 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:58:30.960 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:58:30.960 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:58:30.960 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:58:30.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:30.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 48 16 59856 59840 77.9097 80.5 0.00350721 0.0128297 2024-08-30T23:58:31.013 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:31.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:31.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:31.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:31.276 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:31.378 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:31.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:31.512 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:31.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:31 smithi136 ceph-mon[20207]: pgmap v2150: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22418/169761 objects degraded (13.206%); 24333/169761 objects misplaced (14.334%); 11 MiB/s, 174 objects/s recovering 2024-08-30T23:58:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 22418/169761 objects degraded (13.206%), 13 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:31 smithi173 ceph-mon[20554]: pgmap v2150: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22418/169761 objects degraded (13.206%); 24333/169761 objects misplaced (14.334%); 11 MiB/s, 174 objects/s recovering 2024-08-30T23:58:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 22418/169761 objects degraded (13.206%), 13 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:31 smithi106 ceph-mon[16603]: pgmap v2150: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22418/169761 objects degraded (13.206%); 24333/169761 objects misplaced (14.334%); 11 MiB/s, 174 objects/s recovering 2024-08-30T23:58:31.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 22418/169761 objects degraded (13.206%), 13 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:31.635 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:31.657 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:31.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-30T23:58:31.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:31.786 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:31.888 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:31.919 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:31.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:31.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 49 16 60876 60860 77.6206 63.75 0.00232089 0.0128777 2024-08-30T23:58:31.985 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:32.022 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:32.053 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:32.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.190 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:32.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:32.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:32.316 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:32.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:32.441 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:32.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:32.578 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:32.587 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:58:32.587 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 763/762 [PushReplyOp(7:23574efc:::benchmark_data_smithi173_129296_object75:head)])", 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 485.297548592, 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71604419900000005, 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0699999999999997e-07 2024-08-30T23:58:32.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288323+0000", 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3490000000000002e-06 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001046+0000", 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71272310000000005 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001053+0000", 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0230000000000004e-06 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001113+0000", 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0402000000000002e-05 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.004362+0000", 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032488180000000001 2024-08-30T23:58:32.591 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 766/762 [PushReplyOp(7:21ab2b56:::benchmark_data_smithi173_129296_object2772:head)])", 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 481.43012721100001, 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92479003900000001, 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155740+0000", 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0780000000000001e-06 2024-08-30T23:58:32.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155745+0000", 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077308+0000", 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92156320800000002 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077323+0000", 2024-08-30T23:58:32.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4443e-05 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077429+0000", 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105683 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.080529+0000", 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031005659999999999 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.595 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 767/762 [PushReplyOp(7:26f21d66:::benchmark_data_smithi173_129296_object4816:head)])", 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 477.75268213200002, 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0014494869999999, 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833185+0000", 2024-08-30T23:58:32.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0499999999999999e-06 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833190+0000", 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0830000000000003e-06 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830783+0000", 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99759255400000002 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830792+0000", 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7779999999999996e-06 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830874+0000", 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2557000000000003e-05 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.834634+0000", 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003759465 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.599 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.b 769/756 [PushOp(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head, version: 734'1560, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head@734'1560, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 468.03270961200002, 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60120657700000002, 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553158+0000", 2024-08-30T23:58:32.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3799999999999999e-06 2024-08-30T23:58:32.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553211+0000", 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2676999999999997e-05 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153021+0000", 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59981003200000005 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153032+0000", 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1415e-05 2024-08-30T23:58:32.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153139+0000", 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107075 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153149+0000", 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9629999999999993e-06 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.154363+0000", 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214035 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:123065 7.1 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e792)", 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 398.09835263600002, 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64777179500000004, 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 123065 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487515+0000", 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.477e-06 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487521+0000", 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487533+0000", 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.234e-05 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487541+0000", 2024-08-30T23:58:32.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487594+0000", 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3134000000000003e-05 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487681+0000", 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7437999999999999e-05 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487753+0000", 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1632999999999997e-05 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.490939+0000", 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031855550000000001 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.492073+0000", 2024-08-30T23:58:32.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011345039999999999 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135209+0000", 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64313627299999998 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135251+0000", 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1378000000000003e-05 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135285+0000", 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4730999999999997e-05 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:142377 7.8 7:18815242:::benchmark_data_smithi173_129296_object55261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 355.19029410799999, 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61694005799999996, 2024-08-30T23:58:32.612 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 142377 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395573+0000", 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395576+0000", 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395586+0000", 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4360000000000008e-06 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395591+0000", 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8910000000000003e-06 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395638+0000", 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7027000000000002e-05 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395723+0000", 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5181999999999999e-05 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395790+0000", 2024-08-30T23:58:32.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7430999999999999e-05 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400118+0000", 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043280920000000004 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400495+0000", 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037659899999999998 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012439+0000", 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61194435700000005 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012480+0000", 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1180000000000002e-05 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012512+0000", 2024-08-30T23:58:32.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1869999999999998e-05 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145475 7.4 7:2251fdb7:::benchmark_data_smithi173_129296_object58359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.41285880499998, 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61919969799999997, 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145475 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173010+0000", 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7190000000000001e-06 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173019+0000", 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6289999999999999e-06 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173041+0000", 2024-08-30T23:58:32.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658000000000002e-05 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173051+0000", 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0139999999999999e-05 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173114+0000", 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3289999999999999e-05 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173214+0000", 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100282 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173291+0000", 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6983999999999993e-05 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.177105+0000", 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003813355 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.178585+0000", 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001480608 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792092+0000", 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61350662700000003 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792154+0000", 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1884000000000006e-05 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792207+0000", 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3522e-05 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145497 7.4 7:2208bbfd:::benchmark_data_smithi173_129296_object58381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.37747342300003, 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60666463800000003, 2024-08-30T23:58:32.625 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145497 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208395+0000", 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.66e-06 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208403+0000", 2024-08-30T23:58:32.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2309999999999996e-06 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208421+0000", 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7825999999999998e-05 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208428+0000", 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0539999999999997e-06 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208486+0000", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8517000000000002e-05 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208625+0000", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138517 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208748+0000", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012329600000000001 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.212469+0000", 2024-08-30T23:58:32.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037205519999999998 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.213491+0000", 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010220489999999999 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.814962+0000", 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.601471588 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815023+0000", 2024-08-30T23:58:32.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0732000000000001e-05 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815058+0000", 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4616000000000002e-05 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145502 7.4 7:2d669461:::benchmark_data_smithi173_129296_object58386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.36999034000002, 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59929806799999996, 2024-08-30T23:58:32.631 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145502 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215878+0000", 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6220000000000001e-06 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215886+0000", 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9219999999999997e-06 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215902+0000", 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6677e-05 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215910+0000", 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8499999999999994e-06 2024-08-30T23:58:32.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215967+0000", 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6898999999999999e-05 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216108+0000", 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014146699999999999 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216238+0000", 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012992900000000001 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223310+0000", 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070715079999999998 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224153+0000", 2024-08-30T23:58:32.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084321899999999998 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815104+0000", 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59095095600000003 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815137+0000", 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2759000000000003e-05 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815174+0000", 2024-08-30T23:58:32.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7259999999999999e-05 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145504 7.4 7:2d76a73e:::benchmark_data_smithi173_129296_object58388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.36622567199998, 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59561452400000003, 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.638 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145504 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219642+0000", 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.782e-06 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219653+0000", 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0723999999999999e-05 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219670+0000", 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.681e-05 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219679+0000", 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219829+0000", 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149863 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219920+0000", 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1633999999999997e-05 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.220015+0000", 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4344000000000003e-05 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223383+0000", 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033687449999999998 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224195+0000", 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081209999999999995 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815208+0000", 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59101230400000004 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815235+0000", 2024-08-30T23:58:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7310000000000001e-05 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815255+0000", 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0075e-05 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145510 7.4 7:2d7edcbd:::benchmark_data_smithi173_129296_object58394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.34537117600001, 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.57487306000000005, 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145510 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240497+0000", 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.381e-06 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240502+0000", 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.575e-06 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240514+0000", 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1996e-05 2024-08-30T23:58:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240522+0000", 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7079999999999997e-06 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240565+0000", 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3095e-05 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240644+0000", 2024-08-30T23:58:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9509000000000001e-05 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240712+0000", 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7859000000000002e-05 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.244050+0000", 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003337727 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.245826+0000", 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017758229999999999 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815284+0000", 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56945848700000001 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815331+0000", 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6276999999999998e-05 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815368+0000", 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7623000000000002e-05 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145529 7.4 7:2339dd6d:::benchmark_data_smithi173_129296_object58413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.29079029000002, 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53035033899999995, 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145529 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295077+0000", 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.47e-07 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295085+0000", 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9750000000000007e-06 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295097+0000", 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2391999999999999e-05 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295103+0000", 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8529999999999997e-06 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295150+0000", 2024-08-30T23:58:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6684000000000002e-05 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295235+0000", 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4892e-05 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295302+0000", 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6934999999999995e-05 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.298587+0000", 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032851030000000002 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.299413+0000", 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00082625299999999999 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825357+0000", 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52594381499999998 2024-08-30T23:58:32.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825395+0000", 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7898999999999997e-05 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825426+0000", 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1591000000000003e-05 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145538 7.4 7:2666a83b:::benchmark_data_smithi173_129296_object58422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.26377627900001, 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51489389200000002, 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145538 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322092+0000", 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.683e-06 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322100+0000", 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3119999999999994e-06 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322119+0000", 2024-08-30T23:58:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9368000000000002e-05 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322128+0000", 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1599999999999998e-06 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322189+0000", 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0976e-05 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322326+0000", 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013735099999999999 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322456+0000", 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012996399999999999 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326052+0000", 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035965490000000001 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328903+0000", 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028504350000000001 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836909+0000", 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.508006556 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836949+0000", 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9864000000000003e-05 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836984+0000", 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4674e-05 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145540 7.4 7:22c5e8c4:::benchmark_data_smithi173_129296_object58424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.25960838700001, 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52274398799999999, 2024-08-30T23:58:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145540 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326259+0000", 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2440000000000001e-06 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326268+0000", 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3760000000000008e-06 2024-08-30T23:58:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326280+0000", 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2665e-05 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326288+0000", 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326325+0000", 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6508999999999999e-05 2024-08-30T23:58:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326403+0000", 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7877e-05 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326473+0000", 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9820999999999994e-05 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328987+0000", 2024-08-30T23:58:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025140520000000001 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.330596+0000", 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001609367 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848889+0000", 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51829265199999996 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848951+0000", 2024-08-30T23:58:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2806999999999997e-05 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849002+0000", 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0544000000000003e-05 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145558 7.4 7:2fe416e2:::benchmark_data_smithi173_129296_object58442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.18018246999998, 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44345267799999999, 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145558 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405686+0000", 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4059999999999999e-06 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405696+0000", 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0073e-05 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405726+0000", 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9686999999999999e-05 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405740+0000", 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4072e-05 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405842+0000", 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010199700000000001 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405928+0000", 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5784000000000004e-05 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.406015+0000", 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7096999999999996e-05 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.409500+0000", 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034854629999999998 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.411030+0000", 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015296820000000001 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849046+0000", 2024-08-30T23:58:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43801564300000001 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849097+0000", 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1424000000000001e-05 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849137+0000", 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9350000000000001e-05 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145576 7.4 7:2d26fcfb:::benchmark_data_smithi173_129296_object58460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 349.02800161499999, 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29138068, 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145576 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557865+0000", 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557867+0000", 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9910000000000001e-06 2024-08-30T23:58:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557875+0000", 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557897+0000", 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2021000000000001e-05 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557906+0000", 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3360000000000001e-06 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.557987+0000", 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1238999999999999e-05 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.558139+0000", 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000151368 2024-08-30T23:58:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.558263+0000", 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123824 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.561781+0000", 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035184819999999999 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.562154+0000", 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037294599999999998 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849181+0000", 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28702652299999998 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849221+0000", 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0670000000000002e-05 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849245+0000", 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4122e-05 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21811 8.5 e853/840 8:a2778f2f:::benchmark_data_smithi173_143816_object21810:head v 853'1377, mlcod=853'1375)", 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.953091+0000", 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 261.63277514999999, 2024-08-30T23:58:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34313867100000001, 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953091+0000", 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953091+0000", 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953093+0000", 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.81e-06 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953143+0000", 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0099000000000003e-05 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295091+0000", 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34194817500000002 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295107+0000", 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6229999999999999e-05 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295194+0000", 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6767999999999993e-05 2024-08-30T23:58:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295221+0000", 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6829e-05 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.296190+0000", 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096933700000000002 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.296230+0000", 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9422999999999998e-05 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:36537 8.e 8:78cdc0b0:::benchmark_data_smithi173_143816_object36536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:25.400191+0000", 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 247.18567489700001, 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.395384505, 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36537 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400191+0000", 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400191+0000", 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400193+0000", 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0690000000000001e-06 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400228+0000", 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5423000000000003e-05 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400241+0000", 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2695e-05 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400248+0000", 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999993e-06 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400300+0000", 2024-08-30T23:58:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1440000000000002e-05 2024-08-30T23:58:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400391+0000", 2024-08-30T23:58:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1564999999999995e-05 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400477+0000", 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5532999999999996e-05 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.401514+0000", 2024-08-30T23:58:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037428 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.402322+0000", 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080706599999999995 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795505+0000", 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.393183318 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795541+0000", 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6442000000000001e-05 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795576+0000", 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4666000000000003e-05 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:58511 9.10 9:09617f02:::benchmark_data_smithi173_160189_object58510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e971)", 2024-08-30T23:58:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:29.776506+0000", 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 2.8093604230000002, 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30996779400000002, 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 58511 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776506+0000", 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776506+0000", 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776508+0000", 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.64e-06 2024-08-30T23:58:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776597+0000", 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9516000000000001e-05 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776612+0000", 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4632e-05 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776619+0000", 2024-08-30T23:58:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4950000000000002e-06 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776665+0000", 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6106999999999997e-05 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776794+0000", 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012893799999999999 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.776931+0000", 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013702199999999999 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.778108+0000", 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011764119999999999 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:29.778380+0000", 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027249700000000002 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:30.086376+0000", 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30799552699999999 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:30.086412+0000", 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5809000000000002e-05 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:30.086474+0000", 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2199000000000005e-05 2024-08-30T23:58:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(9.22 971/969 [PushOp(9:45b69309:::benchmark_data_smithi173_160189_object53417:head, version: 967'2169, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:45b69309:::benchmark_data_smithi173_160189_object53417:head@967'2169, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:31.409507+0000", 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 1.1763598040000001, 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.270074745, 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.409507+0000", 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.409507+0000", 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.409507+0000", 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000003e-07 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.409586+0000", 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9363999999999994e-05 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.678890+0000", 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26930338999999998 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.678910+0000", 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0126999999999999e-05 2024-08-30T23:58:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.679028+0000", 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011788799999999999 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.679042+0000", 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4324e-05 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:31.679581+0000", 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053914000000000002 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:32.709 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:58:32.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:32.735 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:32.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:32.871 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:58:32.877 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:58:32.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:32.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 50 16 62246 62230 77.7806 85.625 0.00289952 0.0128457 2024-08-30T23:58:32.996 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:32.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:32.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:33.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:33.119 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:33.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:33.242 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:33.344 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:33.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:33.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:33.494 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:33.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:33 smithi136 ceph-mon[20207]: pgmap v2151: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 232 op/s; 22587/173256 objects degraded (13.037%); 24651/173256 objects misplaced (14.228%); 9.3 MiB/s, 148 objects/s recovering 2024-08-30T23:58:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:33 smithi173 ceph-mon[20554]: pgmap v2151: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 232 op/s; 22587/173256 objects degraded (13.037%); 24651/173256 objects misplaced (14.228%); 9.3 MiB/s, 148 objects/s recovering 2024-08-30T23:58:33.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-30T23:58:33.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:33 smithi106 ceph-mon[16603]: pgmap v2151: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 232 op/s; 22587/173256 objects degraded (13.037%); 24651/173256 objects misplaced (14.228%); 9.3 MiB/s, 148 objects/s recovering 2024-08-30T23:58:33.621 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:33.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:33.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:33.849 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:33.873 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:33.878 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:33.975 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:33.983 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 51 16 62734 62718 76.8535 30.5 0.0847394 0.0129747 2024-08-30T23:58:33.999 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:33.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:34.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:34.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:34.227 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:34.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.250 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:34.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:34.374 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:34.383 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:58:34.383 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:58:34.383 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:58:34.383 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:58:34.475 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:34.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:34.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:34 smithi136 ceph-mon[20207]: osdmap e972: 12 total, 11 up, 12 in 2024-08-30T23:58:34.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:34 smithi173 ceph-mon[20554]: osdmap e972: 12 total, 11 up, 12 in 2024-08-30T23:58:34.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:34 smithi106 ceph-mon[16603]: osdmap e972: 12 total, 11 up, 12 in 2024-08-30T23:58:34.607 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:34.633 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:34.735 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:34.763 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:34.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:34.877 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:58:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:34.886 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:34.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:34.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 52 16 63456 63440 76.2432 45.125 0.00784985 0.0131105 2024-08-30T23:58:34.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:35.012 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:35.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-30T23:58:35.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:35.135 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:35.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:35.250 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:58:35.250 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:58:35.250 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:58:35.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:35.290 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:35 smithi106 ceph-mon[16603]: pgmap v2153: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 23317/178608 objects degraded (13.055%); 22316/178608 objects misplaced (12.494%); 17 MiB/s, 274 objects/s recovering 2024-08-30T23:58:35.373 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:35.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:35.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:35.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:35.541 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.542 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:35 smithi173 ceph-mon[20554]: pgmap v2153: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 23317/178608 objects degraded (13.055%); 22316/178608 objects misplaced (12.494%); 17 MiB/s, 274 objects/s recovering 2024-08-30T23:58:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:35 smithi173 ceph-mon[20554]: osdmap e973: 12 total, 11 up, 12 in 2024-08-30T23:58:35.597 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:35 smithi136 ceph-mon[20207]: pgmap v2153: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+clean+remapped, 4 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 820 op/s; 23317/178608 objects degraded (13.055%); 22316/178608 objects misplaced (12.494%); 17 MiB/s, 274 objects/s recovering 2024-08-30T23:58:35.597 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:35 smithi136 ceph-mon[20207]: osdmap e973: 12 total, 11 up, 12 in 2024-08-30T23:58:35.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:35 smithi106 ceph-mon[16603]: osdmap e973: 12 total, 11 up, 12 in 2024-08-30T23:58:35.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:35.673 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:35.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:35.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:35.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:35.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:35.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:35.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 53 16 64627 64611 76.1855 73.1875 0.0222039 0.0131207 2024-08-30T23:58:36.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:36.059 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:36.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:36.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:36.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:36.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:36.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: 9.22 scrub starts 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: 9.22 scrub ok 2024-08-30T23:58:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 23317/178608 objects degraded (13.055%), 13 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:36.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:36.573 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: 9.22 scrub starts 2024-08-30T23:58:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:58:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:58:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:58:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:58:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:58:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:58:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:58:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: 9.22 scrub ok 2024-08-30T23:58:36.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 23317/178608 objects degraded (13.055%), 13 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: 9.22 scrub starts 2024-08-30T23:58:36.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: 9.22 scrub ok 2024-08-30T23:58:36.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 23317/178608 objects degraded (13.055%), 13 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:36.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:36.707 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:36.809 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:36.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:36.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:36.964 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:36.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:36.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 54 16 65727 65711 76.0477 68.75 0.00229882 0.0131429 2024-08-30T23:58:37.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:37.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:37.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:37.230 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:37.332 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:37.368 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:37.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:37.495 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:37.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:37 smithi136 ceph-mon[20207]: pgmap v2155: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s; 22470/186792 objects degraded (12.029%); 22438/186792 objects misplaced (12.012%); 24 MiB/s, 390 objects/s recovering 2024-08-30T23:58:37.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:37 smithi106 ceph-mon[16603]: pgmap v2155: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s; 22470/186792 objects degraded (12.029%); 22438/186792 objects misplaced (12.012%); 24 MiB/s, 390 objects/s recovering 2024-08-30T23:58:37.622 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.623 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:37.724 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:37.755 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:37 smithi173 ceph-mon[20554]: pgmap v2155: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 938 op/s; 22470/186792 objects degraded (12.029%); 22438/186792 objects misplaced (12.012%); 24 MiB/s, 390 objects/s recovering 2024-08-30T23:58:37.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:37.878 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:37.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:37.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:37.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 55 16 66641 66625 75.7035 57.125 0.00252485 0.013204 2024-08-30T23:58:38.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:38.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:38.144 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:58:38.145 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:58:38.145 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:58:38.145 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:58:38.145 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:58:38.145 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:38.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:38.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:38.407 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:38.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:38.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:38.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:38.672 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:38.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:38.806 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:38.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:38.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:38.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:38.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 56 15 68087 68072 75.9665 90.4375 0.00344108 0.0131594 2024-08-30T23:58:39.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:39.066 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:39.168 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:39.194 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:39.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:39.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:39.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:39.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:39.443 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:39.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:39.568 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:39.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:39 smithi106 ceph-mon[16603]: pgmap v2156: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s; 23249/193269 objects degraded (12.029%); 20478/193269 objects misplaced (10.596%); 22 MiB/s, 357 objects/s recovering 2024-08-30T23:58:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:39 smithi106 ceph-mon[16603]: 1.1d scrub starts 2024-08-30T23:58:39.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:39 smithi106 ceph-mon[16603]: 1.1d scrub ok 2024-08-30T23:58:39.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:39.699 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:39 smithi136 ceph-mon[20207]: pgmap v2156: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s; 23249/193269 objects degraded (12.029%); 20478/193269 objects misplaced (10.596%); 22 MiB/s, 357 objects/s recovering 2024-08-30T23:58:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:39 smithi136 ceph-mon[20207]: 1.1d scrub starts 2024-08-30T23:58:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:39 smithi136 ceph-mon[20207]: 1.1d scrub ok 2024-08-30T23:58:39.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:39 smithi173 ceph-mon[20554]: pgmap v2156: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 979 op/s; 23249/193269 objects degraded (12.029%); 20478/193269 objects misplaced (10.596%); 22 MiB/s, 357 objects/s recovering 2024-08-30T23:58:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:39 smithi173 ceph-mon[20554]: 1.1d scrub starts 2024-08-30T23:58:39.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:39 smithi173 ceph-mon[20554]: 1.1d scrub ok 2024-08-30T23:58:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:39.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:39.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:39.961 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:39.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:39.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 57 16 69288 69272 75.9494 75 0.0095641 0.0131614 2024-08-30T23:58:40.064 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:40.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:40.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:40.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:40.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:40.352 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:40.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:40.483 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.484 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:40.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:40.609 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:40.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:40.738 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:40.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:40.832 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:40.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:40.841 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:40.868 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:40.869 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:40.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:40.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 58 16 70588 70572 76.0407 81.25 0.00776316 0.013144 2024-08-30T23:58:41.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:41.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:41.136 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:41.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:41.261 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.262 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:41.363 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:41.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.393 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:41.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:41.517 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:41 smithi106 ceph-mon[16603]: pgmap v2157: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 23253/200889 objects degraded (11.575%); 19504/200889 objects misplaced (9.709%); 21 MiB/s, 334 objects/s recovering 2024-08-30T23:58:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 23253/200889 objects degraded (11.575%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:41.617 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:41 smithi136 ceph-mon[20207]: pgmap v2157: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 23253/200889 objects degraded (11.575%); 19504/200889 objects misplaced (9.709%); 21 MiB/s, 334 objects/s recovering 2024-08-30T23:58:41.618 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 23253/200889 objects degraded (11.575%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:41.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:41.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:41.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-30T23:58:41.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:41.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:41.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:41 smithi173 ceph-mon[20554]: pgmap v2157: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s; 23253/200889 objects degraded (11.575%); 19504/200889 objects misplaced (9.709%); 21 MiB/s, 334 objects/s recovering 2024-08-30T23:58:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 23253/200889 objects degraded (11.575%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:41.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:41.897 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:41.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:41.926 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:58:41.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 59 16 71416 71400 75.6289 51.75 0.00348319 0.0131576 2024-08-30T23:58:41.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:42.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:42.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:42.143 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:42.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:42.269 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:42.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:42.405 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:42.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:42.534 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:42 smithi106 ceph-mon[16603]: osdmap e974: 12 total, 11 up, 12 in 2024-08-30T23:58:42.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:42.655 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:42 smithi136 ceph-mon[20207]: osdmap e974: 12 total, 11 up, 12 in 2024-08-30T23:58:42.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27034.0:199583 6.3 6:cfed5daf:::benchmark_data_smithi173_114000_object99785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e720)", 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.140717+0000", 2024-08-30T23:58:42.692 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 596.54866463999997, 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71268692700000003, 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27034", 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/2869872015", 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 199583 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140717+0000", 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140717+0000", 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140719+0000", 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4759999999999999e-06 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140728+0000", 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140743+0000", 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5667000000000001e-05 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140750+0000", 2024-08-30T23:58:42.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.742e-06 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140802+0000", 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1521999999999999e-05 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140895+0000", 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3666999999999996e-05 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 7,11", 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.140982+0000", 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6316000000000005e-05 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.144913+0000", 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003931197 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146322+0000", 2024-08-30T23:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001408786 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853242+0000", 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70692048699999999 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853334+0000", 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2075000000000002e-05 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853404+0000", 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0159000000000004e-05 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27034.0:199583 6.3 e720/657 ondisk, result = 0)", 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 596.54328647, 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70731964599999997, 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.699 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146096+0000", 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4000000000000001e-07 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.146100+0000", 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1679999999999997e-06 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853064+0000", 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.70696384899999998 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853077+0000", 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2799999999999999e-05 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853219+0000", 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141734 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853233+0000", 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4042e-05 2024-08-30T23:58:42.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:48:46.853415+0000", 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000182313 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.f 760/756 [PushOp(7:f4cf17c7:::benchmark_data_smithi173_129296_object4282:head, version: 730'263, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:f4cf17c7:::benchmark_data_smithi173_129296_object4282:head@730'263, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:24.161126+0000", 2024-08-30T23:58:42.703 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 498.528256344, 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53930470500000005, 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161126+0000", 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161126+0000", 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161127+0000", 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.161173+0000", 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5976000000000002e-05 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699030+0000", 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53785763600000003 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699044+0000", 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3576e-05 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699155+0000", 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011168299999999999 2024-08-30T23:58:42.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.699165+0000", 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5820000000000008e-06 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.700430+0000", 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00126541 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 760/756 [PushReplyOp(7:3fbe9260:::benchmark_data_smithi173_129296_object4371:head)])", 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:24.540715+0000", 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 498.14866733700001, 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52877982700000004, 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540715+0000", 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540715+0000", 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540716+0000", 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:24.540720+0000", 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5770000000000004e-06 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.065515+0000", 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.524794439 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.065533+0000", 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8516e-05 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.065702+0000", 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016925999999999999 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:25.069495+0000", 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037920779999999999 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.711 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 766/756 [PushReplyOp(7:323ebbfd:::benchmark_data_smithi173_129296_object11913:head)])", 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:31.914679+0000", 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 490.77470284600003, 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63426543899999999, 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914679+0000", 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914679+0000", 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914680+0000", 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.181e-06 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.914687+0000", 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8299999999999998e-06 2024-08-30T23:58:42.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.543684+0000", 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.62899660499999999 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.543700+0000", 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5726000000000002e-05 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.543829+0000", 2024-08-30T23:58:42.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012904400000000001 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.548945+0000", 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051160529999999997 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 769/756 [PushReplyOp(7:3b2647a8:::benchmark_data_smithi173_129296_object24066:head)])", 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:43.867979+0000", 2024-08-30T23:58:42.715 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 478.82140287300001, 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.84462474499999995, 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867979+0000", 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867979+0000", 2024-08-30T23:58:42.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867980+0000", 2024-08-30T23:58:42.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17e-06 2024-08-30T23:58:42.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:43.867986+0000", 2024-08-30T23:58:42.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5890000000000002e-06 2024-08-30T23:58:42.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.709251+0000", 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.841265076 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.709278+0000", 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6812999999999999e-05 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.709368+0000", 2024-08-30T23:58:42.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0655999999999994e-05 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.712604+0000", 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032354409999999999 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 769/756 [PushReplyOp(7:3f75d97c:::benchmark_data_smithi173_129296_object24809:head)])", 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 477.60852851099997, 2024-08-30T23:58:42.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.66064861900000005, 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080854+0000", 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9899999999999999e-07 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.080861+0000", 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6080000000000001e-06 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.738045+0000", 2024-08-30T23:58:42.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.65718375100000004 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.738061+0000", 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6336000000000001e-05 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.738224+0000", 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016336000000000001 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.741502+0000", 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032776649999999999 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 787/756 [PushReplyOp(7:38fe4451:::benchmark_data_smithi173_129296_object76141:head)])", 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 420.420815506, 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30744231999999999, 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268567+0000", 2024-08-30T23:58:42.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5700000000000002e-07 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.268572+0000", 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8579999999999999e-06 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.575023+0000", 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.306450746 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.575031+0000", 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.575086+0000", 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5577e-05 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:42.576009+0000", 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00092242300000000001 2024-08-30T23:58:42.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.c 788/756 [PushReplyOp(7:3655d6ab:::benchmark_data_smithi173_129296_object76872:head)])", 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:43.176815+0000", 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 419.51256662100002, 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.254290707, 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176815+0000", 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176815+0000", 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176816+0000", 2024-08-30T23:58:42.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9400000000000004e-07 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.176824+0000", 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6620000000000005e-06 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.430155+0000", 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.253331419 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.430163+0000", 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4080000000000001e-06 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.430257+0000", 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4209999999999994e-05 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:43.431106+0000", 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084921400000000005 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.734 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:126882 7.c 7:35900c7d:::benchmark_data_smithi173_129296_object39766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:02.132879+0000", 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 400.55650304099998, 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56331322800000005, 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-30T23:58:42.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 126882 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132879+0000", 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132879+0000", 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132881+0000", 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1299999999999999e-06 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132890+0000", 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.636e-06 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132912+0000", 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1744999999999999e-05 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.132921+0000", 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8449999999999998e-06 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.133008+0000", 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6881999999999999e-05 2024-08-30T23:58:42.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.133146+0000", 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013792899999999999 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,3", 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.133269+0000", 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012271100000000001 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:42.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.136955+0000", 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003686506 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.139302+0000", 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0023464520000000002 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.696119+0000", 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.55681687499999999 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.696158+0000", 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9539999999999998e-05 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.696192+0000", 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3976999999999997e-05 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.7 e833,800 7:ead71d3e:::benchmark_data_smithi173_129296_object19426:head,781'6655)", 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 304.74451523099998, 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.558258531, 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944867+0000", 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6199999999999998e-07 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:37.944871+0000", 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7969999999999998e-06 2024-08-30T23:58:42.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.502987+0000", 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.55811628899999999 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503016+0000", 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8617999999999999e-05 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503073+0000", 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6978999999999998e-05 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503083+0000", 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0237999999999999e-05 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:38.503125+0000", 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2147999999999997e-05 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e833,800 7:bc8b5569:::benchmark_data_smithi173_129296_object11595:head,777'6152)", 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:39.231853+0000", 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 303.45752873800001, 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45119504599999999, 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231853+0000", 2024-08-30T23:58:42.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231853+0000", 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231855+0000", 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.663e-06 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.231865+0000", 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0277e-05 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682696+0000", 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45083097300000002 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682714+0000", 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.751e-05 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682880+0000", 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016595600000000001 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.682909+0000", 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9620000000000001e-05 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.683048+0000", 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139047 2024-08-30T23:58:42.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.7 e833,800 7:eeb0af75:::benchmark_data_smithi173_129296_object21006:head,781'6745)", 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:39.488031+0000", 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 303.20135090700001, 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25498426000000002, 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488031+0000", 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488031+0000", 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488033+0000", 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5600000000000001e-06 2024-08-30T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.488043+0000", 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.057e-05 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742898+0000", 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25485495600000002 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742903+0000", 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742971+0000", 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7314e-05 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.742977+0000", 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6150000000000002e-06 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:39.743015+0000", 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8027e-05 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e834,800 7:b3496454:::benchmark_data_smithi173_129296_object13177:head,778'6249)", 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:40.366733+0000", 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 302.32264865399998, 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26684242000000002, 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366733+0000", 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366733+0000", 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366734+0000", 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.366738+0000", 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1910000000000002e-06 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633427+0000", 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.26668847600000001 2024-08-30T23:58:42.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633434+0000", 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2429999999999999e-06 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633516+0000", 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1892999999999997e-05 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633524+0000", 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7319999999999998e-06 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.633576+0000", 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2286e-05 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.7 e834,800 7:e204aafa:::benchmark_data_smithi173_129296_object23666:head,782'6902)", 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:40.447246+0000", 2024-08-30T23:58:42.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 302.24213570900002, 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25044494499999997, 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447246+0000", 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447246+0000", 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447247+0000", 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0200000000000002e-07 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.447251+0000", 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9689999999999996e-06 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697523+0000", 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.250271937 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697545+0000", 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2374999999999998e-05 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697635+0000", 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9909999999999998e-05 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697643+0000", 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1969999999999996e-06 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:40.697691+0000", 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7954999999999997e-05 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e835,800 7:b9e4363b:::benchmark_data_smithi173_129296_object16399:head,779'6452)", 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:42.354445+0000", 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 300.33493669900002, 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.946729178, 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354445+0000", 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354445+0000", 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354446+0000", 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6199999999999998e-07 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:42.354449+0000", 2024-08-30T23:58:42.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2940000000000001e-06 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301025+0000", 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.94657636899999997 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301031+0000", 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4519999999999996e-06 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301123+0000", 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1745999999999999e-05 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301129+0000", 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.657e-06 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:43.301175+0000", 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5198000000000003e-05 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGRecoveryDelete(7.d e837,800 7:b580be49:::benchmark_data_smithi173_129296_object20352:head,781'6705)", 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 297.05209432300001, 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36897256299999998, 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:58:42.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637288+0000", 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6499999999999999e-07 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:45.637295+0000", 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.353e-06 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:46.006128+0000", 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.36883366099999998 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:46.006147+0000", 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8706999999999999e-05 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:53:46.006260+0000", 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113377 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:47868 9.0 9:03e11011:::benchmark_data_smithi173_160189_object47867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e964)", 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:19.193877+0000", 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.495505189999999, 2024-08-30T23:58:42.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25837180100000001, 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 47868 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.193877+0000", 2024-08-30T23:58:42.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.193877+0000", 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.193878+0000", 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.435e-06 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.193941+0000", 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2483000000000005e-05 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.193955+0000", 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4276e-05 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.193963+0000", 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5950000000000001e-06 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.194026+0000", 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3597999999999997e-05 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.194139+0000", 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011247300000000001 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,11", 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.194248+0000", 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109134 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.195196+0000", 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00094779900000000004 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.202530+0000", 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0073343660000000001 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.452189+0000", 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24965885800000001 2024-08-30T23:58:42.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.452221+0000", 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2361999999999999e-05 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.452249+0000", 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7421999999999999e-05 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:47888 9.13 e964/956 9:ced6ee84:::benchmark_data_smithi173_160189_object47887:head v 964'2024, mlcod=964'2022)", 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:19.202369+0000", 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.487012545999999, 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21475876499999999, 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.202369+0000", 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.202369+0000", 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.202370+0000", 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1299999999999999e-07 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.202437+0000", 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6575999999999994e-05 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.416039+0000", 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.21360232200000001 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.416047+0000", 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5050000000000001e-06 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.416105+0000", 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8458999999999997e-05 2024-08-30T23:58:42.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.416126+0000", 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0959999999999999e-05 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.417088+0000", 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096204899999999998 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:19.417128+0000", 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0181e-05 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:62816 9.19 e972/957 9:9f8d97ec:::benchmark_data_smithi173_160189_object62815:head v 972'2514, mlcod=972'2512)", 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:34.148110+0000", 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.5412719549999991, 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25386350400000002, 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:42.781 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.148110+0000", 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.148110+0000", 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.148111+0000", 2024-08-30T23:58:42.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.148232+0000", 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121205 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.400587+0000", 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25235486600000001 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:58:42.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.400594+0000", 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9870000000000002e-06 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.400679+0000", 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5418000000000001e-05 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.400700+0000", 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1209999999999999e-05 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.401931+0000", 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001231146 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:34.401974+0000", 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2119000000000001e-05 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:58:42.785 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:58:42.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:42.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:58:42.815 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:42.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:42 smithi173 ceph-mon[20554]: osdmap e974: 12 total, 11 up, 12 in 2024-08-30T23:58:42.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:42.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:42.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:42.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:58:42.985353+0000 min lat: 0.000982985 max lat: 2.30386 avg lat: 0.0132772 2024-08-30T23:58:42.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:58:42.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 60 16 72272 72256 75.2601 53.5 0.00804453 0.0132772 2024-08-30T23:58:43.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:43.076 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:43.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:43.205 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:43.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:43.337 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:43.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:43.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:43 smithi106 ceph-mon[16603]: pgmap v2159: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 23436/204849 objects degraded (11.441%); 19872/204849 objects misplaced (9.701%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:58:43.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:43.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:43.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:43 smithi106 ceph-mon[16603]: osdmap e975: 12 total, 11 up, 12 in 2024-08-30T23:58:43.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:43.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:43.461 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:43 smithi136 ceph-mon[20207]: pgmap v2159: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 23436/204849 objects degraded (11.441%); 19872/204849 objects misplaced (9.701%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:43 smithi136 ceph-mon[20207]: osdmap e975: 12 total, 11 up, 12 in 2024-08-30T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:43.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:43.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:43.587 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:43.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:43.695 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-30T23:58:43.713 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:43.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:43 smithi173 ceph-mon[20554]: pgmap v2159: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s; 23436/204849 objects degraded (11.441%); 19872/204849 objects misplaced (9.701%); 12 MiB/s, 195 objects/s recovering 2024-08-30T23:58:43.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:43.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:43.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:43 smithi173 ceph-mon[20554]: osdmap e975: 12 total, 11 up, 12 in 2024-08-30T23:58:43.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:43.738 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:43.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:43.840 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:43.943 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:43.982 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:43.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:43.984 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 61 16 73255 73239 75.0334 61.4375 0.0081854 0.0133224 2024-08-30T23:58:44.055 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:44.084 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:44.124 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:44.210 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:58:44.211 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:58:44.226 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.273 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:44.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:44.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:44.406 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:44.482 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:44 smithi173 ceph-mon[20554]: 9.21 scrub starts 2024-08-30T23:58:44.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:44 smithi173 ceph-mon[20554]: 9.21 scrub ok 2024-08-30T23:58:44.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:44 smithi173 ceph-mon[20554]: 9.20 scrub starts 2024-08-30T23:58:44.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:44 smithi173 ceph-mon[20554]: 9.20 scrub ok 2024-08-30T23:58:44.483 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:44 smithi173 ceph-mon[20554]: osdmap e976: 12 total, 11 up, 12 in 2024-08-30T23:58:44.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:44.532 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:44.604 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:58:44.604 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:58:44.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.604 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:74006 9.17 9:e874843b:::benchmark_data_smithi173_160189_object74005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e976)", 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:44.554898+0000", 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.048785006999999998, 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.048811806999999999, 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 74006 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:44.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.554898+0000", 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.554898+0000", 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.554899+0000", 2024-08-30T23:58:44.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.159e-06 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.554917+0000", 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8104000000000001e-05 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.554930+0000", 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2595999999999999e-05 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:44.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.554937+0000", 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4749999999999996e-06 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.555002+0000", 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4947e-05 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.555140+0000", 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013851099999999999 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.555286+0000", 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014512800000000001 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.556643+0000", 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013576700000000001 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:44.557152+0000", 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00050885899999999996 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:44.609 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:44.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:44.610 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:44.610 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:58:44.610 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-08-30T23:58:44.610 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:58:44.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:44 smithi106 ceph-mon[16603]: 9.21 scrub starts 2024-08-30T23:58:44.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:44 smithi106 ceph-mon[16603]: 9.21 scrub ok 2024-08-30T23:58:44.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:44 smithi106 ceph-mon[16603]: 9.20 scrub starts 2024-08-30T23:58:44.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:44 smithi106 ceph-mon[16603]: 9.20 scrub ok 2024-08-30T23:58:44.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:44 smithi106 ceph-mon[16603]: osdmap e976: 12 total, 11 up, 12 in 2024-08-30T23:58:44.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:44.664 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:44.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:44.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:44 smithi136 ceph-mon[20207]: 9.21 scrub starts 2024-08-30T23:58:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:44 smithi136 ceph-mon[20207]: 9.21 scrub ok 2024-08-30T23:58:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:44 smithi136 ceph-mon[20207]: 9.20 scrub starts 2024-08-30T23:58:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:44 smithi136 ceph-mon[20207]: 9.20 scrub ok 2024-08-30T23:58:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:44 smithi136 ceph-mon[20207]: osdmap e976: 12 total, 11 up, 12 in 2024-08-30T23:58:44.798 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:44.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:44.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:44.921 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:44.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:44.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 62 16 74356 74340 74.9329 68.8125 0.0248294 0.0133336 2024-08-30T23:58:45.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:45.055 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:45.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:45.181 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:45.283 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:45.315 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:45.321 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-30T23:58:45.418 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:45.447 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:45.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:45.572 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:45 smithi173 ceph-mon[20554]: pgmap v2162: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 812 op/s; 24134/209856 objects degraded (11.500%); 18243/209856 objects misplaced (8.693%); 7.4 MiB/s, 118 objects/s recovering 2024-08-30T23:58:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:45 smithi173 ceph-mon[20554]: from='client.28513 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:58:45.656 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:45.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:45.702 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:45 smithi136 ceph-mon[20207]: pgmap v2162: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 812 op/s; 24134/209856 objects degraded (11.500%); 18243/209856 objects misplaced (8.693%); 7.4 MiB/s, 118 objects/s recovering 2024-08-30T23:58:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:45 smithi136 ceph-mon[20207]: from='client.28513 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:58:45.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:45.828 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:45 smithi106 ceph-mon[16603]: pgmap v2162: 67 pgs: 3 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 812 op/s; 24134/209856 objects degraded (11.500%); 18243/209856 objects misplaced (8.693%); 7.4 MiB/s, 118 objects/s recovering 2024-08-30T23:58:45.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:45 smithi106 ceph-mon[16603]: from='client.28513 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:58:45.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:45.959 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:45.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:45.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 63 16 75585 75569 74.9627 76.8125 0.0210223 0.0133315 2024-08-30T23:58:46.061 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:46.084 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:46.186 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:46.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:46.325 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:58:46.326 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:58:46.326 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:58:46.326 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:58:46.326 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:58:46.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:46.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:46.470 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:46.498 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 24134/209856 objects degraded (11.500%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:46 smithi173 ceph-mon[20554]: osdmap e977: 12 total, 11 up, 12 in 2024-08-30T23:58:46.599 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:46.623 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:46.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:46.746 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:46.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 24134/209856 objects degraded (11.500%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:46 smithi136 ceph-mon[20207]: osdmap e977: 12 total, 11 up, 12 in 2024-08-30T23:58:46.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 24134/209856 objects degraded (11.500%), 12 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:46.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:46 smithi106 ceph-mon[16603]: osdmap e977: 12 total, 11 up, 12 in 2024-08-30T23:58:46.875 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:46.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:46.977 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:46.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 64 16 76509 76493 74.6936 57.75 0.00284055 0.0133819 2024-08-30T23:58:47.008 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:47.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:47.140 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:47.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:47.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-30T23:58:47.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:47.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:47.399 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:47.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:47.531 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:47 smithi173 ceph-mon[20554]: pgmap v2163: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 725 op/s; 20689/218925 objects degraded (9.450%); 19670/218925 objects misplaced (8.985%); 3.2 MiB/s, 51 objects/s recovering 2024-08-30T23:58:47.616 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:47.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:47.665 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.666 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:47.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:47.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:47 smithi136 ceph-mon[20207]: pgmap v2163: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 725 op/s; 20689/218925 objects degraded (9.450%); 19670/218925 objects misplaced (8.985%); 3.2 MiB/s, 51 objects/s recovering 2024-08-30T23:58:47.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:47.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:47 smithi106 ceph-mon[16603]: pgmap v2163: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 725 op/s; 20689/218925 objects degraded (9.450%); 19670/218925 objects misplaced (8.985%); 3.2 MiB/s, 51 objects/s recovering 2024-08-30T23:58:47.898 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:47.926 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:47.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:47.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 65 16 77838 77822 74.8223 83.0625 0.0615166 0.0133564 2024-08-30T23:58:48.028 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:48.054 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:48.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:48.180 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:48.252 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 765/761 [PushReplyOp(7:f2a3ccc8:::benchmark_data_smithi173_129296_object3685:head)])", 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 497.76143484699998, 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63433747900000004, 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.253 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487056+0000", 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487057+0000", 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0219999999999999e-06 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:30.487063+0000", 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0469999999999999e-06 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115718+0000", 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62865490300000004 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115728+0000", 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9080000000000007e-06 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.115812+0000", 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3713000000000003e-05 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:31.121393+0000", 2024-08-30T23:58:48.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055818860000000003 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 769/761 [PushReplyOp(7:f0c70446:::benchmark_data_smithi173_129296_object15083:head)])", 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 484.82753995399997, 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97997442000000001, 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.257 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420951+0000", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420952+0000", 2024-08-30T23:58:48.258 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.062e-06 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:43.420958+0000", 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9800000000000003e-06 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395034+0000", 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97407640699999998 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395059+0000", 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4338999999999999e-05 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.395138+0000", 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9306000000000003e-05 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:44.400925+0000", 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057873259999999998 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:106574 7.7 7:e13bf739:::benchmark_data_smithi173_129296_object19458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 445.77567870799999, 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29613937600000001, 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.261 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 106574 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472812+0000", 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472813+0000", 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3600000000000003e-07 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472817+0000", 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2119999999999997e-06 2024-08-30T23:58:48.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472827+0000", 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.91e-06 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.472833+0000", 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5310000000000001e-06 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476588+0000", 2024-08-30T23:58:48.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037553450000000002 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476692+0000", 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000103875 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,3", 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.476803+0000", 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011145999999999999 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.480260+0000", 2024-08-30T23:58:48.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0034566480000000001 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.486997+0000", 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067370900000000003 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768836+0000", 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.28183939600000002 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768874+0000", 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7537000000000003e-05 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:22.768952+0000", 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7636e-05 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGPushReply(7.f 792/761 [PushReplyOp(7:ff5d9ae4:::benchmark_data_smithi173_129296_object65310:head)])", 2024-08-30T23:58:48.267 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 413.89789662200002, 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79968256000000004, 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350594+0000", 2024-08-30T23:58:48.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350595+0000", 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9299999999999998e-07 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.350601+0000", 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0379999999999996e-06 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.269 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148049+0000", 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.79744800299999996 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148074+0000", 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5244000000000001e-05 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.148181+0000", 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106742 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.150277+0000", 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020959400000000001 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:27874 8.c 8:385c40a0:::benchmark_data_smithi173_143816_object27873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:16.646546+0000", 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.60194495500002, 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31994099999999998, 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.271 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 27874 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646546+0000", 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646546+0000", 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646547+0000", 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5899999999999995e-07 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646588+0000", 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1168000000000003e-05 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646600+0000", 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1986e-05 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646606+0000", 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4760000000000003e-06 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646647+0000", 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0408999999999999e-05 2024-08-30T23:58:48.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646725+0000", 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8450999999999998e-05 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.646807+0000", 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1458999999999994e-05 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647769+0000", 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000962457 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.803588+0000", 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.15581879400000001 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966370+0000", 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16278235299999999 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.276 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966419+0000", 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8578999999999997e-05 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966487+0000", 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8009000000000006e-05 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.277 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:27874 8.c e857/840 ondisk, result = 0)", 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:16.647975+0000", 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 271.60051595300001, 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31851893199999998, 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647975+0000", 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647975+0000", 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647976+0000", 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6400000000000001e-07 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.647979+0000", 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4860000000000002e-06 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966319+0000", 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31833969099999998 2024-08-30T23:58:48.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966325+0000", 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2650000000000002e-06 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966362+0000", 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7240000000000003e-05 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966367+0000", 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5929999999999999e-06 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:16.966494+0000", 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012689299999999999 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.282 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:146297 8.b 8:d66af36d:::benchmark_data_smithi173_143816_object47983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e913)", 2024-08-30T23:58:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:20.568778+0000", 2024-08-30T23:58:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 147.67971326899999, 2024-08-30T23:58:48.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.212557265, 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 146297 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568778+0000", 2024-08-30T23:58:48.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568778+0000", 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568779+0000", 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.578e-06 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568785+0000", 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0599999999999996e-06 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568802+0000", 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7278e-05 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568811+0000", 2024-08-30T23:58:48.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6510000000000007e-06 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.568882+0000", 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0706999999999999e-05 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.569137+0000", 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00025551399999999998 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.569436+0000", 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000298249 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.573731+0000", 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0042953380000000001 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.573818+0000", 2024-08-30T23:58:48.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6705000000000004e-05 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781137+0000", 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.207319486 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781210+0000", 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2924999999999997e-05 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781335+0000", 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012477399999999999 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:146297 8.b e913/840 ondisk, result = 0)", 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 147.67407924700001, 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206941876, 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.291 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574412+0000", 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.92e-07 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.574419+0000", 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2330000000000003e-06 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.779467+0000", 2024-08-30T23:58:48.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20504845799999999 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.779473+0000", 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4670000000000001e-06 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781118+0000", 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016446900000000001 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781128+0000", 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5089999999999999e-06 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:56:20.781353+0000", 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000225827 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.295 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:178947 8.a 8:523510ea:::benchmark_data_smithi173_143816_object80633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:00.400595+0000", 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 107.84789628199999, 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23827178600000001, 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 178947 2024-08-30T23:58:48.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400595+0000", 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400595+0000", 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400596+0000", 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.4900000000000004e-07 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400601+0000", 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0030000000000002e-06 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400614+0000", 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3423e-05 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400622+0000", 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9489999999999996e-06 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400683+0000", 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1047e-05 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400798+0000", 2024-08-30T23:58:48.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011467399999999999 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.400910+0000", 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000111944 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.405687+0000", 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0047770850000000004 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.406106+0000", 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041911999999999999 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.638694+0000", 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.23258803 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.638822+0000", 2024-08-30T23:58:48.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000128157 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:00.638866+0000", 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4404999999999999e-05 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:195458 8.b 8:d9fed7f1:::benchmark_data_smithi173_143816_object97144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e933)", 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:21.262301+0000", 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 86.986189846000002, 2024-08-30T23:58:48.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.223549313, 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 195458 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262301+0000", 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262301+0000", 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262302+0000", 2024-08-30T23:58:48.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4139999999999999e-06 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262309+0000", 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0950000000000001e-06 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262321+0000", 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2449000000000001e-05 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262328+0000", 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5050000000000001e-06 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.262881+0000", 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055236499999999997 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.263055+0000", 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017370499999999999 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.263179+0000", 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000124148 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.269952+0000", 2024-08-30T23:58:48.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067737350000000003 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.270940+0000", 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00098715000000000001 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.485733+0000", 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.214793924 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.485787+0000", 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3939e-05 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:21.485850+0000", 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2884000000000003e-05 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:2805 9.b 9:d71f65c6:::benchmark_data_smithi173_160189_object2804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:44.408396+0000", 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.840094975, 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.21076450299999999, 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 2805 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.310 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408396+0000", 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408396+0000", 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408398+0000", 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.046e-06 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408427+0000", 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9147000000000001e-05 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408447+0000", 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9760000000000001e-05 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408569+0000", 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012230699999999999 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408654+0000", 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4491000000000001e-05 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408821+0000", 2024-08-30T23:58:48.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167362 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.408969+0000", 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014835099999999999 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.409590+0000", 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00062063400000000001 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410265+0000", 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00067537100000000002 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619093+0000", 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20882731400000001 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619131+0000", 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8204999999999998e-05 2024-08-30T23:58:48.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619160+0000", 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9515000000000001e-05 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:2811 9.b 9:d069d54c:::benchmark_data_smithi173_160189_object2810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:44.410797+0000", 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.837694237000001, 2024-08-30T23:58:48.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.208520237, 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 2811 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410797+0000", 2024-08-30T23:58:48.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410797+0000", 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410799+0000", 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0040000000000002e-06 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410824+0000", 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5585999999999999e-05 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410839+0000", 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4943000000000001e-05 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410847+0000", 2024-08-30T23:58:48.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3819999999999998e-06 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.410915+0000", 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8727999999999997e-05 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411049+0000", 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013331000000000001 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411163+0000", 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011409299999999999 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.412159+0000", 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00099632400000000004 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.412487+0000", 2024-08-30T23:58:48.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00032812100000000001 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619213+0000", 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.206725728 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619273+0000", 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0257000000000003e-05 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619317+0000", 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3761000000000003e-05 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:2812 9.b 9:d996f27e:::benchmark_data_smithi173_160189_object2811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e944)", 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:44.411069+0000", 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 63.837421786999997, 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20850690899999999, 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.323 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 2812 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411069+0000", 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411069+0000", 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411070+0000", 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2900000000000002e-07 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411094+0000", 2024-08-30T23:58:48.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4019000000000002e-05 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411105+0000", 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1085e-05 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411113+0000", 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5209999999999997e-06 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411491+0000", 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00037843000000000002 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411710+0000", 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000219111 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.411847+0000", 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013693700000000001 2024-08-30T23:58:48.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.412338+0000", 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00049131200000000004 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.412533+0000", 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019451099999999999 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619495+0000", 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20696184400000001 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619539+0000", 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4283000000000003e-05 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:44.619576+0000", 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6927000000000001e-05 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.329 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:19265 9.17 9.412dacb7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:54.177039+0000", 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 54.071451510000003, 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89744186699999995, 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.330 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 19265 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177039+0000", 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177039+0000", 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177041+0000", 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0980000000000001e-06 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177092+0000", 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0578000000000003e-05 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177117+0000", 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5134e-05 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177127+0000", 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0117e-05 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177202+0000", 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4265999999999997e-05 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.177213+0000", 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1019000000000001e-05 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.074471+0000", 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.89725853200000005 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.074481+0000", 2024-08-30T23:58:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0122999999999999e-05 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:40957 9.15 9:a9730c56:::benchmark_data_smithi173_160189_object40956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e962)", 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:14.487113+0000", 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 33.761378174000001, 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26376061099999998, 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.335 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 40957 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487113+0000", 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487113+0000", 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487114+0000", 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1620000000000001e-06 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487156+0000", 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2076999999999997e-05 2024-08-30T23:58:48.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487168+0000", 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1640999999999999e-05 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487174+0000", 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0379999999999996e-06 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487216+0000", 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2304e-05 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487353+0000", 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013759500000000001 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,5", 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.487517+0000", 2024-08-30T23:58:48.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000163249 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.488455+0000", 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00093787999999999999 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.489231+0000", 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077658299999999996 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.750705+0000", 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.26147389500000001 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.750738+0000", 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2657999999999998e-05 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:14.750873+0000", 2024-08-30T23:58:48.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013552900000000001 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:62825 9.b 9:d26bcb74:::benchmark_data_smithi173_160189_object62824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e972)", 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:34.168861+0000", 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 14.079630046, 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22930445199999999, 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.342 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 62825 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.168861+0000", 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.168861+0000", 2024-08-30T23:58:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.168863+0000", 2024-08-30T23:58:48.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.08e-06 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.168917+0000", 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3687999999999998e-05 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.168936+0000", 2024-08-30T23:58:48.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9100999999999999e-05 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.168945+0000", 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0710000000000001e-06 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.169020+0000", 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5575999999999996e-05 2024-08-30T23:58:48.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.169170+0000", 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014941499999999999 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,10", 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.169325+0000", 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015544699999999999 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.170665+0000", 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013395010000000001 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.373086+0000", 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.20242158299999999 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.398058+0000", 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.024972138000000001 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.398120+0000", 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1260999999999995e-05 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:34.398165+0000", 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5590999999999998e-05 2024-08-30T23:58:48.350 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:72700 9.15 9:a8341c5d:::benchmark_data_smithi173_160189_object72699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e976)", 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:43.433387+0000", 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.8151035980000003, 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22451834900000001, 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.351 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 72700 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433387+0000", 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433387+0000", 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433389+0000", 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6589999999999999e-06 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433412+0000", 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3135000000000001e-05 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433425+0000", 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2670999999999999e-05 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433433+0000", 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4130000000000006e-06 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433495+0000", 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1508999999999997e-05 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433629+0000", 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013453299999999999 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,5", 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.433770+0000", 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014106599999999999 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.435214+0000", 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014439629999999999 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.435996+0000", 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00078136899999999997 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657793+0000", 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.221797196 2024-08-30T23:58:48.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657835+0000", 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2568999999999998e-05 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657906+0000", 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0265999999999994e-05 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:72700 9.15 e976/956 ondisk, result = 0)", 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:43.435683+0000", 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 4.8128083149999998, 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.22222930199999999, 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.435683+0000", 2024-08-30T23:58:48.358 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.435683+0000", 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.435683+0000", 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3900000000000003e-07 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.359 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.435688+0000", 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6249999999999998e-06 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657748+0000", 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.222059859 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657752+0000", 2024-08-30T23:58:48.360 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0219999999999998e-06 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657783+0000", 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1337999999999997e-05 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657789+0000", 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7699999999999998e-06 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.361 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:43.657912+0000", 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012274900000000001 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:76127 9.15 9:aee9cf86:::benchmark_data_smithi173_160189_object76126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e977)", 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:46.672721+0000", 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.575769558, 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.165222124, 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.362 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 76127 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672721+0000", 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.363 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672721+0000", 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672722+0000", 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3300000000000002e-07 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.364 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672770+0000", 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8114e-05 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672777+0000", 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3900000000000004e-06 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672783+0000", 2024-08-30T23:58:48.365 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9610000000000002e-06 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672842+0000", 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9111999999999999e-05 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.672977+0000", 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013489099999999999 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.366 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,5", 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.673140+0000", 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000162893 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.674649+0000", 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0015083340000000001 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.367 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.676065+0000", 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001416327 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.837806+0000", 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.16174160500000001 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.837854+0000", 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7262999999999998e-05 2024-08-30T23:58:48.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.837943+0000", 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9701000000000001e-05 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:76352 9.15 9:abd70b6b:::benchmark_data_smithi173_160189_object76351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e977)", 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:46.892096+0000", 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 1.356394527, 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13860604300000001, 2024-08-30T23:58:48.369 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 76352 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892096+0000", 2024-08-30T23:58:48.370 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892096+0000", 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892098+0000", 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2979999999999999e-06 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.371 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892150+0000", 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2318000000000003e-05 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892165+0000", 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5387e-05 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:48.372 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892172+0000", 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0990000000000001e-06 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892534+0000", 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00036204599999999998 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892682+0000", 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014799100000000001 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.373 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 4,5", 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.892857+0000", 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00017448499999999999 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.894533+0000", 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001675733 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:46.894763+0000", 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000230626 2024-08-30T23:58:48.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:47.030530+0000", 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.13576660400000001 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:47.030584+0000", 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4107000000000002e-05 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:47.030702+0000", 2024-08-30T23:58:48.375 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118349 2024-08-30T23:58:48.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.376 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.376 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:48.376 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:48.376 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:58:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:48.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:48.381 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:58:48.382 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:58:48.417 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:58:48.417 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:58:48.448 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:58:48.449 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:58:48.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:48.479 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:58:48.479 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:58:48.509 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:48.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:48.637 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:48.713 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:58:48.713 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:58:48.714 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2702867579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:48.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:48.749 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:58:48.749 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:58:48.767 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:48.774 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:58:48.774 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:58:48.775 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:58:48.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2702867579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:48.797 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:58:48.797 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:58:48.820 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:58:48.821 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:58:48.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2702867579' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:48.842 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:58:48.842 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:58:48.863 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:58:48.864 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:58:48.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:48.891 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:48.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:48.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 66 15 78838 78823 74.6364 62.5625 0.00184699 0.0133934 2024-08-30T23:58:48.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:49.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:49.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:49.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:49.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:49.269 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-30T23:58:49.283 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:49.385 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:49.414 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:49.515 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:49.545 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:49 smithi173 ceph-mon[20554]: pgmap v2165: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 859 op/s; 21492/225285 objects degraded (9.540%); 17881/225285 objects misplaced (7.937%); 14 MiB/s, 224 objects/s recovering 2024-08-30T23:58:49.604 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:49.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:49.674 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:49.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:49.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:49 smithi136 ceph-mon[20207]: pgmap v2165: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 859 op/s; 21492/225285 objects degraded (9.540%); 17881/225285 objects misplaced (7.937%); 14 MiB/s, 224 objects/s recovering 2024-08-30T23:58:49.802 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:49.838 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:49 smithi106 ceph-mon[16603]: pgmap v2165: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 859 op/s; 21492/225285 objects degraded (9.540%); 17881/225285 objects misplaced (7.937%); 14 MiB/s, 224 objects/s recovering 2024-08-30T23:58:49.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:49.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:49.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:49.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 67 16 80171 80155 74.7649 83.25 0.0032134 0.013369 2024-08-30T23:58:50.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:50.055 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:50.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:50.189 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:50.195 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:50.195 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:50.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:50.323 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:50.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:50.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:50.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:50.564 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2162,"stamp":"2024-08-30T23:58:43.859539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4584775517,"num_objects":69952,"num_object_clones":0,"num_object_copies":209856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24134,"num_objects_misplaced":18243,"num_objects_unfound":0,"num_objects_dirty":69952,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":48298,"num_write_kb":3090578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4555,"num_bytes_recovered":298600627,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112198,"ondisk_log_size":112198,"up":177,"acting":175,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":216,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":21859276,"kb_used_data":13653884,"kb_used_omap":109,"kb_used_meta":8205074,"kb_avail":1103082548,"statfs":{"total":1151940427776,"available":1129556529152,"internally_reserved":0,"allocated":13981577216,"data_stored":13971554737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":8401996440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,2,1,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":362348544,"num_objects":5529,"num_object_clones":0,"num_object_copies":16587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":885,"num_objects_misplaced":-2235,"num_objects_unfound":0,"num_objects_dirty":5529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":312192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":712,"num_bytes_recovered":46661632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001736"},"pg_stats":[{"pgid":"9.23","version":"973'2420","reported_seq":2536,"reported_epoch":973,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:58:38.916782+0000","last_change":"2024-08-30T23:58:26.434809+0000","last_active":"2024-08-30T23:58:38.916782+0000","last_peered":"2024-08-30T23:58:38.916782+0000","last_clean":"2024-08-30T23:58:24.066816+0000","last_became_active":"2024-08-30T23:58:26.433256+0000","last_became_peered":"2024-08-30T23:58:26.433256+0000","last_unstale":"2024-08-30T23:58:38.916782+0000","last_undegraded":"2024-08-30T23:58:26.367753+0000","last_fullsized":"2024-08-30T23:58:26.234529+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":6,"last_scrub":"949'1132","last_scrub_stamp":"2024-08-30T23:57:53.736917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.736917+0000","objects_scrubbed":594,"log_size":2420,"log_dups_size":0,"ondisk_log_size":2420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68976640,"num_objects":1053,"num_object_clones":0,"num_object_copies":3159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":849,"num_objects_misplaced":849,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":756,"num_write_kb":48352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[3,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":849},{"shards":"3,9","objects":204}],"blocked_by":[],"up_primary":10,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"973'2393","reported_seq":4144,"reported_epoch":973,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:39.253412+0000","last_change":"2024-08-30T23:58:35.493120+0000","last_active":"2024-08-30T23:58:39.253412+0000","last_peered":"2024-08-30T23:58:39.253412+0000","last_clean":"2024-08-30T23:58:39.253412+0000","last_became_active":"2024-08-30T23:58:34.338653+0000","last_became_peered":"2024-08-30T23:58:34.338653+0000","last_unstale":"2024-08-30T23:58:39.253412+0000","last_undegraded":"2024-08-30T23:58:39.253412+0000","last_fullsized":"2024-08-30T23:58:39.253412+0000","mapping_epoch":972,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":6,"last_scrub":"973'2327","last_scrub_stamp":"2024-08-30T23:58:35.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:35.492788+0000","objects_scrubbed":939,"log_size":2393,"log_dups_size":0,"ondisk_log_size":2393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:58.878728+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65929216,"num_objects":1006,"num_object_clones":0,"num_object_copies":3018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1006,"num_objects_unfound":0,"num_objects_dirty":1006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":4288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4,1],"avail_no_missing":["9","1","4"],"object_location_counts":[{"shards":"1,4,9","objects":1006}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.21","version":"973'2445","reported_seq":5039,"reported_epoch":973,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:58:38.943502+0000","last_change":"2024-08-30T23:58:26.501579+0000","last_active":"2024-08-30T23:58:38.943502+0000","last_peered":"2024-08-30T23:58:38.943502+0000","last_clean":"2024-08-30T23:58:24.221645+0000","last_became_active":"2024-08-30T23:58:26.472904+0000","last_became_peered":"2024-08-30T23:58:26.472904+0000","last_unstale":"2024-08-30T23:58:38.943502+0000","last_undegraded":"2024-08-30T23:58:38.943502+0000","last_fullsized":"2024-08-30T23:58:26.233563+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":6,"last_scrub":"950'1110","last_scrub_stamp":"2024-08-30T23:57:54.993958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:54.993958+0000","objects_scrubbed":574,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70615040,"num_objects":1078,"num_object_clones":0,"num_object_copies":3234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":695,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":50592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":619,"num_bytes_recovered":40566784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[3,1],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"1,3","objects":846},{"shards":"2,3,6,9","objects":1},{"shards":"3,6,9","objects":231}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.20","version":"973'2402","reported_seq":4512,"reported_epoch":973,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:58:36.907906+0000","last_change":"2024-08-30T23:58:26.358571+0000","last_active":"2024-08-30T23:58:36.907906+0000","last_peered":"2024-08-30T23:58:36.907906+0000","last_clean":"2024-08-30T23:58:24.171559+0000","last_became_active":"2024-08-30T23:58:26.343409+0000","last_became_peered":"2024-08-30T23:58:26.343409+0000","last_unstale":"2024-08-30T23:58:36.907906+0000","last_undegraded":"2024-08-30T23:58:36.907906+0000","last_fullsized":"2024-08-30T23:58:26.233641+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":6,"last_scrub":"949'1159","last_scrub_stamp":"2024-08-30T23:57:53.984940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.984940+0000","objects_scrubbed":625,"log_size":2402,"log_dups_size":0,"ondisk_log_size":2402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67829760,"num_objects":1035,"num_object_clones":0,"num_object_copies":3105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":1035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":46240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":32636928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[1,0],"avail_no_missing":["1","6","11"],"object_location_counts":[{"shards":"0,1","objects":711},{"shards":"1,6,11","objects":324}],"blocked_by":[],"up_primary":8,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1c","version":"973'3740","reported_seq":3834,"reported_epoch":973,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:58:38.851986+0000","last_change":"2024-08-30T23:58:26.426581+0000","last_active":"2024-08-30T23:58:38.851986+0000","last_peered":"2024-08-30T23:58:38.851986+0000","last_clean":"2024-08-30T23:58:24.213324+0000","last_became_active":"2024-08-30T23:58:26.422320+0000","last_became_peered":"2024-08-30T23:58:26.422320+0000","last_unstale":"2024-08-30T23:58:38.851986+0000","last_undegraded":"2024-08-30T23:58:26.356599+0000","last_fullsized":"2024-08-30T23:58:26.234157+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3740,"log_dups_size":0,"ondisk_log_size":3740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137822208,"num_objects":2103,"num_object_clones":0,"num_object_copies":6309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":134592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,3],"acting":[3,0],"avail_no_missing":["3","4","9"],"object_location_counts":[{"shards":"0,3","objects":422},{"shards":"3,4,9","objects":1681}],"blocked_by":[],"up_primary":11,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1120,"reported_epoch":973,"state":"active+undersized","last_fresh":"2024-08-30T23:58:34.291463+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-30T23:58:34.291463+0000","last_peered":"2024-08-30T23:58:34.291463+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-30T23:58:34.291463+0000","last_undegraded":"2024-08-30T23:58:34.291463+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":288,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.337431+0000","last_change":"2024-08-30T23:58:11.144659+0000","last_active":"2024-08-30T23:58:41.337431+0000","last_peered":"2024-08-30T23:58:41.337431+0000","last_clean":"2024-08-30T23:58:41.337431+0000","last_became_active":"2024-08-30T23:55:01.289376+0000","last_became_peered":"2024-08-30T23:55:01.289376+0000","last_unstale":"2024-08-30T23:58:41.337431+0000","last_undegraded":"2024-08-30T23:58:41.337431+0000","last_fullsized":"2024-08-30T23:58:41.337431+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:11.144606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:11.144606+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:32.000093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.3","version":"973'2431","reported_seq":2563,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:39.243175+0000","last_change":"2024-08-30T23:58:25.101873+0000","last_active":"2024-08-30T23:58:39.243175+0000","last_peered":"2024-08-30T23:58:39.243175+0000","last_clean":"2024-08-30T23:58:39.243175+0000","last_became_active":"2024-08-30T23:58:23.229111+0000","last_became_peered":"2024-08-30T23:58:23.229111+0000","last_unstale":"2024-08-30T23:58:39.243175+0000","last_undegraded":"2024-08-30T23:58:39.243175+0000","last_fullsized":"2024-08-30T23:58:39.243175+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'2230","last_scrub_stamp":"2024-08-30T23:58:25.101767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:25.101767+0000","objects_scrubbed":870,"log_size":2431,"log_dups_size":0,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:38.278227+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70189056,"num_objects":1071,"num_object_clones":0,"num_object_copies":3213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":201,"num_write_kb":12864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1173,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.291825+0000","last_change":"2024-08-30T23:58:08.502691+0000","last_active":"2024-08-30T23:58:34.291825+0000","last_peered":"2024-08-30T23:58:34.291825+0000","last_clean":"2024-08-30T23:58:34.291825+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-30T23:58:34.291825+0000","last_undegraded":"2024-08-30T23:58:34.291825+0000","last_fullsized":"2024-08-30T23:58:34.291825+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:08.502643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:08.502643+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:23.323808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"973'2410","reported_seq":2541,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:38.650726+0000","last_change":"2024-08-30T23:58:24.986319+0000","last_active":"2024-08-30T23:58:38.650726+0000","last_peered":"2024-08-30T23:58:38.650726+0000","last_clean":"2024-08-30T23:58:38.650726+0000","last_became_active":"2024-08-30T23:58:23.225981+0000","last_became_peered":"2024-08-30T23:58:23.225981+0000","last_unstale":"2024-08-30T23:58:38.650726+0000","last_undegraded":"2024-08-30T23:58:38.650726+0000","last_fullsized":"2024-08-30T23:58:38.650726+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'2200","last_scrub_stamp":"2024-08-30T23:58:24.986029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:24.986029+0000","objects_scrubbed":883,"log_size":2410,"log_dups_size":0,"ondisk_log_size":2410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:38.998643+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71630848,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":13440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1028,"reported_epoch":973,"state":"active+undersized","last_fresh":"2024-08-30T23:58:34.291365+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-30T23:58:34.291365+0000","last_peered":"2024-08-30T23:58:34.291365+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-30T23:58:34.291365+0000","last_undegraded":"2024-08-30T23:58:34.291365+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:37.258998+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"973'2444","reported_seq":2573,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:38.767714+0000","last_change":"2024-08-30T23:58:25.993565+0000","last_active":"2024-08-30T23:58:38.767714+0000","last_peered":"2024-08-30T23:58:38.767714+0000","last_clean":"2024-08-30T23:58:38.767714+0000","last_became_active":"2024-08-30T23:58:23.226284+0000","last_became_peered":"2024-08-30T23:58:23.226284+0000","last_unstale":"2024-08-30T23:58:38.767714+0000","last_undegraded":"2024-08-30T23:58:38.767714+0000","last_fullsized":"2024-08-30T23:58:38.767714+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'2220","last_scrub_stamp":"2024-08-30T23:58:25.993421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:25.993421+0000","objects_scrubbed":860,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:33.490251+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":14336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":431,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.289962+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-30T23:58:34.289962+0000","last_peered":"2024-08-30T23:58:34.289962+0000","last_clean":"2024-08-30T23:58:34.289962+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-30T23:58:34.289962+0000","last_undegraded":"2024-08-30T23:58:34.289962+0000","last_fullsized":"2024-08-30T23:58:34.289962+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"974'2506","reported_seq":2639,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.767220+0000","last_change":"2024-08-30T23:58:24.279278+0000","last_active":"2024-08-30T23:58:41.767220+0000","last_peered":"2024-08-30T23:58:41.767220+0000","last_clean":"2024-08-30T23:58:41.767220+0000","last_became_active":"2024-08-30T23:58:23.225864+0000","last_became_peered":"2024-08-30T23:58:23.225864+0000","last_unstale":"2024-08-30T23:58:41.767220+0000","last_undegraded":"2024-08-30T23:58:41.767220+0000","last_fullsized":"2024-08-30T23:58:41.767220+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'2225","last_scrub_stamp":"2024-08-30T23:58:24.279213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:24.279213+0000","objects_scrubbed":904,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:28.409101+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77725696,"num_objects":1186,"num_object_clones":0,"num_object_copies":3558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":282,"num_write_kb":18048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1251,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.336079+0000","last_change":"2024-08-30T23:57:52.872669+0000","last_active":"2024-08-30T23:58:41.336079+0000","last_peered":"2024-08-30T23:58:41.336079+0000","last_clean":"2024-08-30T23:58:41.336079+0000","last_became_active":"2024-08-30T23:57:36.158625+0000","last_became_peered":"2024-08-30T23:57:36.158625+0000","last_unstale":"2024-08-30T23:58:41.336079+0000","last_undegraded":"2024-08-30T23:58:41.336079+0000","last_fullsized":"2024-08-30T23:58:41.336079+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:52.872607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.872607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:21.985915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"974'2773","reported_seq":2863,"reported_epoch":974,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:41.842575+0000","last_change":"2024-08-30T23:57:53.077387+0000","last_active":"2024-08-30T23:58:41.842575+0000","last_peered":"2024-08-30T23:58:41.842575+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:53.077387+0000","last_became_peered":"2024-08-30T23:57:53.077387+0000","last_unstale":"2024-08-30T23:58:41.842575+0000","last_undegraded":"2024-08-30T23:57:53.068813+0000","last_fullsized":"2024-08-30T23:57:53.068689+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2773,"log_dups_size":0,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:46.780311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146604032,"num_objects":2237,"num_object_clones":0,"num_object_copies":6711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2237,"num_write_kb":143168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","3"],"object_location_counts":[{"shards":"3,10","objects":2237}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":967,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.334222+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-30T23:58:41.334222+0000","last_peered":"2024-08-30T23:58:41.334222+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-30T23:58:41.334222+0000","last_undegraded":"2024-08-30T23:58:41.334222+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:43.884669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"973'2669","reported_seq":2775,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:39.768979+0000","last_change":"2024-08-30T23:57:55.058608+0000","last_active":"2024-08-30T23:58:39.768979+0000","last_peered":"2024-08-30T23:58:39.768979+0000","last_clean":"2024-08-30T23:58:39.768979+0000","last_became_active":"2024-08-30T23:57:53.081131+0000","last_became_peered":"2024-08-30T23:57:53.081131+0000","last_unstale":"2024-08-30T23:58:39.768979+0000","last_undegraded":"2024-08-30T23:58:39.768979+0000","last_fullsized":"2024-08-30T23:58:39.768979+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1118","last_scrub_stamp":"2024-08-30T23:57:55.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:55.058465+0000","objects_scrubbed":594,"log_size":2669,"log_dups_size":0,"ondisk_log_size":2669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:04.286867+0000","scrub_duration":990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140640256,"num_objects":2146,"num_object_clones":0,"num_object_copies":6438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1552,"num_write_kb":99328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":1036,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.334697+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-30T23:58:41.334697+0000","last_peered":"2024-08-30T23:58:41.334697+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-30T23:58:41.334697+0000","last_undegraded":"2024-08-30T23:58:41.334697+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.847432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"974'2759","reported_seq":2852,"reported_epoch":974,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:41.346949+0000","last_change":"2024-08-30T23:57:53.082784+0000","last_active":"2024-08-30T23:58:41.346949+0000","last_peered":"2024-08-30T23:58:41.346949+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:53.076653+0000","last_became_peered":"2024-08-30T23:57:53.076653+0000","last_unstale":"2024-08-30T23:58:41.346949+0000","last_undegraded":"2024-08-30T23:57:53.073210+0000","last_fullsized":"2024-08-30T23:57:53.072333+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":2759,"log_dups_size":0,"ondisk_log_size":2759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:46.328002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144506880,"num_objects":2205,"num_object_clones":0,"num_object_copies":6615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2205,"num_write_kb":141120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"5,11","objects":2205}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":1057,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.337467+0000","last_change":"2024-08-30T23:57:32.972014+0000","last_active":"2024-08-30T23:58:34.337467+0000","last_peered":"2024-08-30T23:58:34.337467+0000","last_clean":"2024-08-30T23:58:34.337467+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-30T23:58:34.337467+0000","last_undegraded":"2024-08-30T23:58:34.337467+0000","last_fullsized":"2024-08-30T23:58:34.337467+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.971963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.971963+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:39.937780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"974'2647","reported_seq":2755,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.687196+0000","last_change":"2024-08-30T23:57:53.615819+0000","last_active":"2024-08-30T23:58:41.687196+0000","last_peered":"2024-08-30T23:58:41.687196+0000","last_clean":"2024-08-30T23:58:41.687196+0000","last_became_active":"2024-08-30T23:57:53.070921+0000","last_became_peered":"2024-08-30T23:57:53.070921+0000","last_unstale":"2024-08-30T23:58:41.687196+0000","last_undegraded":"2024-08-30T23:58:41.687196+0000","last_fullsized":"2024-08-30T23:58:41.687196+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1072","last_scrub_stamp":"2024-08-30T23:57:53.615713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.615713+0000","objects_scrubbed":564,"log_size":2647,"log_dups_size":0,"ondisk_log_size":2647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:13.784595+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140312576,"num_objects":2141,"num_object_clones":0,"num_object_copies":6423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1577,"num_write_kb":100928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":398,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.335962+0000","last_change":"2024-08-30T23:57:34.913534+0000","last_active":"2024-08-30T23:58:41.335962+0000","last_peered":"2024-08-30T23:58:41.335962+0000","last_clean":"2024-08-30T23:58:41.335962+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:58:41.335962+0000","last_undegraded":"2024-08-30T23:58:41.335962+0000","last_fullsized":"2024-08-30T23:58:41.335962+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:34.913482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:34.913482+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.633745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"973'3980","reported_seq":4101,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:40.724873+0000","last_change":"2024-08-30T23:58:23.894116+0000","last_active":"2024-08-30T23:58:40.724873+0000","last_peered":"2024-08-30T23:58:40.724873+0000","last_clean":"2024-08-30T23:58:40.724873+0000","last_became_active":"2024-08-30T23:58:23.233769+0000","last_became_peered":"2024-08-30T23:58:23.233769+0000","last_unstale":"2024-08-30T23:58:40.724873+0000","last_undegraded":"2024-08-30T23:58:40.724873+0000","last_fullsized":"2024-08-30T23:58:40.724873+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'3430","last_scrub_stamp":"2024-08-30T23:58:23.893952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:23.893952+0000","objects_scrubbed":1718,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:41.108321+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":558,"num_write_kb":35712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":2065,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.334234+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-30T23:58:41.334234+0000","last_peered":"2024-08-30T23:58:41.334234+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-30T23:58:41.334234+0000","last_undegraded":"2024-08-30T23:58:41.334234+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"973'3792","reported_seq":3880,"reported_epoch":973,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:39.802856+0000","last_change":"2024-08-30T23:58:23.233914+0000","last_active":"2024-08-30T23:58:39.802856+0000","last_peered":"2024-08-30T23:58:39.802856+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.233914+0000","last_became_peered":"2024-08-30T23:58:23.233914+0000","last_unstale":"2024-08-30T23:58:39.802856+0000","last_undegraded":"2024-08-30T23:58:23.221431+0000","last_fullsized":"2024-08-30T23:58:23.221296+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141754368,"num_objects":2163,"num_object_clones":0,"num_object_copies":6489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":138432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2163}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":578,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.340224+0000","last_change":"2024-08-30T23:57:08.049165+0000","last_active":"2024-08-30T23:58:41.340224+0000","last_peered":"2024-08-30T23:58:41.340224+0000","last_clean":"2024-08-30T23:58:41.340224+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:58:41.340224+0000","last_undegraded":"2024-08-30T23:58:41.340224+0000","last_fullsized":"2024-08-30T23:58:41.340224+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:08.049028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:08.049028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"974'2781","reported_seq":2890,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.899728+0000","last_change":"2024-08-30T23:57:56.128716+0000","last_active":"2024-08-30T23:58:41.899728+0000","last_peered":"2024-08-30T23:58:41.899728+0000","last_clean":"2024-08-30T23:58:41.899728+0000","last_became_active":"2024-08-30T23:57:53.081012+0000","last_became_peered":"2024-08-30T23:57:53.081012+0000","last_unstale":"2024-08-30T23:58:41.899728+0000","last_undegraded":"2024-08-30T23:58:41.899728+0000","last_fullsized":"2024-08-30T23:58:41.899728+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1143","last_scrub_stamp":"2024-08-30T23:57:56.128615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:56.128615+0000","objects_scrubbed":635,"log_size":2781,"log_dups_size":0,"ondisk_log_size":2781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:58.044791+0000","scrub_duration":1261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148963328,"num_objects":2273,"num_object_clones":0,"num_object_copies":6819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1638,"num_write_kb":104832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1240,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.336193+0000","last_change":"2024-08-30T23:58:10.841604+0000","last_active":"2024-08-30T23:58:41.336193+0000","last_peered":"2024-08-30T23:58:41.336193+0000","last_clean":"2024-08-30T23:58:41.336193+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-30T23:58:41.336193+0000","last_undegraded":"2024-08-30T23:58:41.336193+0000","last_fullsized":"2024-08-30T23:58:41.336193+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:10.841542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:10.841542+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:36.684389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"973'3854","reported_seq":3973,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:39.773588+0000","last_change":"2024-08-30T23:58:26.120565+0000","last_active":"2024-08-30T23:58:39.773588+0000","last_peered":"2024-08-30T23:58:39.773588+0000","last_clean":"2024-08-30T23:58:39.773588+0000","last_became_active":"2024-08-30T23:58:23.234078+0000","last_became_peered":"2024-08-30T23:58:23.234078+0000","last_unstale":"2024-08-30T23:58:39.773588+0000","last_undegraded":"2024-08-30T23:58:39.773588+0000","last_fullsized":"2024-08-30T23:58:39.773588+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'3422","last_scrub_stamp":"2024-08-30T23:58:26.120459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:26.120459+0000","objects_scrubbed":1763,"log_size":3854,"log_dups_size":0,"ondisk_log_size":3854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:27.599850+0000","scrub_duration":1088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":6585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":593,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.341124+0000","last_change":"2024-08-30T23:56:16.234435+0000","last_active":"2024-08-30T23:58:41.341124+0000","last_peered":"2024-08-30T23:58:41.341124+0000","last_clean":"2024-08-30T23:58:41.341124+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-30T23:58:41.341124+0000","last_undegraded":"2024-08-30T23:58:41.341124+0000","last_fullsized":"2024-08-30T23:58:41.341124+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:16.234371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:16.234371+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"974'4021","reported_seq":4110,"reported_epoch":974,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:41.930377+0000","last_change":"2024-08-30T23:58:23.240351+0000","last_active":"2024-08-30T23:58:41.930377+0000","last_peered":"2024-08-30T23:58:41.930377+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.240351+0000","last_became_peered":"2024-08-30T23:58:23.240351+0000","last_unstale":"2024-08-30T23:58:41.930377+0000","last_undegraded":"2024-08-30T23:58:23.225188+0000","last_fullsized":"2024-08-30T23:58:23.224931+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":4021,"log_dups_size":0,"ondisk_log_size":4021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151715840,"num_objects":2315,"num_object_clones":0,"num_object_copies":6945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":148160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":["10","1"],"object_location_counts":[{"shards":"1,10","objects":2315}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":619,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.340413+0000","last_change":"2024-08-30T23:56:48.173739+0000","last_active":"2024-08-30T23:58:41.340413+0000","last_peered":"2024-08-30T23:58:41.340413+0000","last_clean":"2024-08-30T23:58:41.340413+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-30T23:58:41.340413+0000","last_undegraded":"2024-08-30T23:58:41.340413+0000","last_fullsized":"2024-08-30T23:58:41.340413+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:48.173668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:48.173668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"973'3710","reported_seq":3798,"reported_epoch":973,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:39.801518+0000","last_change":"2024-08-30T23:58:23.224228+0000","last_active":"2024-08-30T23:58:39.801518+0000","last_peered":"2024-08-30T23:58:39.801518+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.224228+0000","last_became_peered":"2024-08-30T23:58:23.224228+0000","last_unstale":"2024-08-30T23:58:39.801518+0000","last_undegraded":"2024-08-30T23:58:23.222413+0000","last_fullsized":"2024-08-30T23:58:23.221376+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:43.989407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136871936,"num_objects":2088,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":133664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":2088}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":278,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.337624+0000","last_change":"2024-08-30T23:55:59.171246+0000","last_active":"2024-08-30T23:58:41.337624+0000","last_peered":"2024-08-30T23:58:41.337624+0000","last_clean":"2024-08-30T23:55:57.160009+0000","last_became_active":"2024-08-30T23:55:59.171246+0000","last_became_peered":"2024-08-30T23:55:59.171246+0000","last_unstale":"2024-08-30T23:58:41.337624+0000","last_undegraded":"2024-08-30T23:58:41.337624+0000","last_fullsized":"2024-08-30T23:55:59.168006+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:36.270718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:36.270718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.c","version":"973'3718","reported_seq":3838,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:38.913741+0000","last_change":"2024-08-30T23:58:24.964811+0000","last_active":"2024-08-30T23:58:38.913741+0000","last_peered":"2024-08-30T23:58:38.913741+0000","last_clean":"2024-08-30T23:58:38.913741+0000","last_became_active":"2024-08-30T23:58:23.224459+0000","last_became_peered":"2024-08-30T23:58:23.224459+0000","last_unstale":"2024-08-30T23:58:38.913741+0000","last_undegraded":"2024-08-30T23:58:38.913741+0000","last_fullsized":"2024-08-30T23:58:38.913741+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'3325","last_scrub_stamp":"2024-08-30T23:58:24.964745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:24.964745+0000","objects_scrubbed":1690,"log_size":3718,"log_dups_size":0,"ondisk_log_size":3718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:25.808541+0000","scrub_duration":1112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136642560,"num_objects":2085,"num_object_clones":0,"num_object_copies":6255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":395,"num_write_kb":25280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":825,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.336313+0000","last_change":"2024-08-30T23:55:59.173538+0000","last_active":"2024-08-30T23:58:41.336313+0000","last_peered":"2024-08-30T23:58:41.336313+0000","last_clean":"2024-08-30T23:55:57.750202+0000","last_became_active":"2024-08-30T23:55:59.173538+0000","last_became_peered":"2024-08-30T23:55:59.173538+0000","last_unstale":"2024-08-30T23:58:41.336313+0000","last_undegraded":"2024-08-30T23:58:41.336313+0000","last_fullsized":"2024-08-30T23:55:59.169546+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:53.668489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:53.668489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":["6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"973'2737","reported_seq":2843,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:39.393833+0000","last_change":"2024-08-30T23:57:54.976951+0000","last_active":"2024-08-30T23:58:39.393833+0000","last_peered":"2024-08-30T23:58:39.393833+0000","last_clean":"2024-08-30T23:58:39.393833+0000","last_became_active":"2024-08-30T23:57:53.076655+0000","last_became_peered":"2024-08-30T23:57:53.076655+0000","last_unstale":"2024-08-30T23:58:39.393833+0000","last_undegraded":"2024-08-30T23:58:39.393833+0000","last_fullsized":"2024-08-30T23:58:39.393833+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1157","last_scrub_stamp":"2024-08-30T23:57:54.976868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:54.976868+0000","objects_scrubbed":595,"log_size":2737,"log_dups_size":0,"ondisk_log_size":2737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:01.148633+0000","scrub_duration":399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142540800,"num_objects":2175,"num_object_clones":0,"num_object_copies":6525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1580,"num_write_kb":101120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":408,"reported_epoch":973,"state":"active+undersized","last_fresh":"2024-08-30T23:58:34.290186+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-30T23:58:34.290186+0000","last_peered":"2024-08-30T23:58:34.290186+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-30T23:58:34.290186+0000","last_undegraded":"2024-08-30T23:58:34.290186+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.18","version":"974'2820","reported_seq":4166,"reported_epoch":974,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:41.823853+0000","last_change":"2024-08-30T23:58:08.120423+0000","last_active":"2024-08-30T23:58:41.823853+0000","last_peered":"2024-08-30T23:58:41.823853+0000","last_clean":"2024-08-30T23:58:41.823853+0000","last_became_active":"2024-08-30T23:58:07.098389+0000","last_became_peered":"2024-08-30T23:58:07.098389+0000","last_unstale":"2024-08-30T23:58:41.823853+0000","last_undegraded":"2024-08-30T23:58:41.823853+0000","last_fullsized":"2024-08-30T23:58:41.823853+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"961'1604","last_scrub_stamp":"2024-08-30T23:58:08.120271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:08.120271+0000","objects_scrubbed":1049,"log_size":2820,"log_dups_size":0,"ondisk_log_size":2820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:27.133050+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148832256,"num_objects":2271,"num_object_clones":0,"num_object_copies":6813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2271,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1222,"num_write_kb":78208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5,9],"avail_no_missing":["10","5","9"],"object_location_counts":[{"shards":"5,9,10","objects":2271}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":1041,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.339265+0000","last_change":"2024-08-30T23:58:04.552055+0000","last_active":"2024-08-30T23:58:34.339265+0000","last_peered":"2024-08-30T23:58:34.339265+0000","last_clean":"2024-08-30T23:58:34.339265+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:58:34.339265+0000","last_undegraded":"2024-08-30T23:58:34.339265+0000","last_fullsized":"2024-08-30T23:58:34.339265+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:04.552013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.552013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:20.924737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.19","version":"973'2672","reported_seq":4482,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:38.852546+0000","last_change":"2024-08-30T23:58:04.958546+0000","last_active":"2024-08-30T23:58:38.852546+0000","last_peered":"2024-08-30T23:58:38.852546+0000","last_clean":"2024-08-30T23:58:38.852546+0000","last_became_active":"2024-08-30T23:58:04.122431+0000","last_became_peered":"2024-08-30T23:58:04.122431+0000","last_unstale":"2024-08-30T23:58:38.852546+0000","last_undegraded":"2024-08-30T23:58:38.852546+0000","last_fullsized":"2024-08-30T23:58:38.852546+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1412","last_scrub_stamp":"2024-08-30T23:58:04.958468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.958468+0000","objects_scrubbed":844,"log_size":2672,"log_dups_size":0,"ondisk_log_size":2672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:14.446451+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137887744,"num_objects":2104,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":80640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":420,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.290381+0000","last_change":"2024-08-30T23:57:35.093839+0000","last_active":"2024-08-30T23:58:34.290381+0000","last_peered":"2024-08-30T23:58:34.290381+0000","last_clean":"2024-08-30T23:58:34.290381+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:58:34.290381+0000","last_undegraded":"2024-08-30T23:58:34.290381+0000","last_fullsized":"2024-08-30T23:58:34.290381+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:35.093787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.093787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:54.153963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"974'3810","reported_seq":6196,"reported_epoch":974,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:58:41.912169+0000","last_change":"2024-08-30T23:58:26.526706+0000","last_active":"2024-08-30T23:58:41.912169+0000","last_peered":"2024-08-30T23:58:41.912169+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:26.500213+0000","last_became_peered":"2024-08-30T23:58:26.500213+0000","last_unstale":"2024-08-30T23:58:41.912169+0000","last_undegraded":"2024-08-30T23:58:26.352427+0000","last_fullsized":"2024-08-30T23:58:26.234149+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3810,"log_dups_size":0,"ondisk_log_size":3810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142934016,"num_objects":2181,"num_object_clones":0,"num_object_copies":6543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2181,"num_objects_misplaced":1798,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2181,"num_write_kb":139584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":50135040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[1,2],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,2","objects":1282},{"shards":"1,5","objects":899}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":273,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.337275+0000","last_change":"2024-08-30T23:57:14.423376+0000","last_active":"2024-08-30T23:58:41.337275+0000","last_peered":"2024-08-30T23:58:41.337275+0000","last_clean":"2024-08-30T23:58:41.337275+0000","last_became_active":"2024-08-30T23:54:10.991428+0000","last_became_peered":"2024-08-30T23:54:10.991428+0000","last_unstale":"2024-08-30T23:58:41.337275+0000","last_undegraded":"2024-08-30T23:58:41.337275+0000","last_fullsized":"2024-08-30T23:58:41.337275+0000","mapping_epoch":853,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":854,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:14.423316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:14.423316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:42.050397+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.1b","version":"973'3897","reported_seq":3994,"reported_epoch":973,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-30T23:58:39.684879+0000","last_change":"2024-08-30T23:58:26.439811+0000","last_active":"2024-08-30T23:58:39.684879+0000","last_peered":"2024-08-30T23:58:39.684879+0000","last_clean":"2024-08-30T23:58:24.214529+0000","last_became_active":"2024-08-30T23:58:26.436975+0000","last_became_peered":"2024-08-30T23:58:26.436975+0000","last_unstale":"2024-08-30T23:58:39.684879+0000","last_undegraded":"2024-08-30T23:58:26.270806+0000","last_fullsized":"2024-08-30T23:58:26.233511+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144310272,"num_objects":2202,"num_object_clones":0,"num_object_copies":6606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":140928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[5,0],"avail_no_missing":["5","8","10"],"object_location_counts":[{"shards":"0,5","objects":450},{"shards":"5,8,10","objects":1752}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"973'3758","reported_seq":6185,"reported_epoch":973,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:58:39.796195+0000","last_change":"2024-08-30T23:58:26.397437+0000","last_active":"2024-08-30T23:58:39.796195+0000","last_peered":"2024-08-30T23:58:39.796195+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:26.379579+0000","last_became_peered":"2024-08-30T23:58:26.379579+0000","last_unstale":"2024-08-30T23:58:39.796195+0000","last_undegraded":"2024-08-30T23:58:26.277017+0000","last_fullsized":"2024-08-30T23:58:26.233800+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3758,"log_dups_size":0,"ondisk_log_size":3758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140017664,"num_objects":2137,"num_object_clones":0,"num_object_copies":6411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":936,"num_objects_misplaced":936,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2137,"num_write_kb":136736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":779,"num_bytes_recovered":51052544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,0],"acting":[5,0],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"0,5","objects":1201},{"shards":"1,5","objects":936}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":152,"reported_epoch":973,"state":"active+undersized","last_fresh":"2024-08-30T23:58:34.292012+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-30T23:58:34.292012+0000","last_peered":"2024-08-30T23:58:34.292012+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-30T23:58:34.292012+0000","last_undegraded":"2024-08-30T23:58:34.292012+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:58:37.013785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1e","version":"974'3999","reported_seq":7012,"reported_epoch":974,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-30T23:58:42.137576+0000","last_change":"2024-08-30T23:58:26.377778+0000","last_active":"2024-08-30T23:58:42.137576+0000","last_peered":"2024-08-30T23:58:42.137576+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:26.360032+0000","last_became_peered":"2024-08-30T23:58:26.360032+0000","last_unstale":"2024-08-30T23:58:42.137576+0000","last_undegraded":"2024-08-30T23:58:26.289401+0000","last_fullsized":"2024-08-30T23:58:26.233805+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150274048,"num_objects":2293,"num_object_clones":0,"num_object_copies":6879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2293,"num_objects_misplaced":1582,"num_objects_unfound":0,"num_objects_dirty":2293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2293,"num_write_kb":146752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":974,"num_bytes_recovered":63832064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[10,0],"avail_no_missing":["10","1"],"object_location_counts":[{"shards":"0,10","objects":1502},{"shards":"1,10","objects":791}],"blocked_by":[],"up_primary":5,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":226,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.340558+0000","last_change":"2024-08-30T23:57:27.164513+0000","last_active":"2024-08-30T23:58:41.340558+0000","last_peered":"2024-08-30T23:58:41.340558+0000","last_clean":"2024-08-30T23:58:41.340558+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-30T23:58:41.340558+0000","last_undegraded":"2024-08-30T23:58:41.340558+0000","last_fullsized":"2024-08-30T23:58:41.340558+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:27.164462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:27.164462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1f","version":"974'3956","reported_seq":6808,"reported_epoch":974,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:58:41.912080+0000","last_change":"2024-08-30T23:58:26.456243+0000","last_active":"2024-08-30T23:58:41.912080+0000","last_peered":"2024-08-30T23:58:41.912080+0000","last_clean":"2024-08-30T23:58:24.200223+0000","last_became_active":"2024-08-30T23:58:26.364334+0000","last_became_peered":"2024-08-30T23:58:26.364334+0000","last_unstale":"2024-08-30T23:58:41.912080+0000","last_undegraded":"2024-08-30T23:58:41.912080+0000","last_fullsized":"2024-08-30T23:58:26.234874+0000","mapping_epoch":969,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148733952,"num_objects":2270,"num_object_clones":0,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1563,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":145248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":918,"num_bytes_recovered":60162048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,6],"avail_no_missing":["1","5","9"],"object_location_counts":[{"shards":"1,5,9","objects":781},{"shards":"1,5,9,10","objects":1},{"shards":"1,6","objects":1488}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":224,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.339746+0000","last_change":"2024-08-30T23:58:11.457897+0000","last_active":"2024-08-30T23:58:34.339746+0000","last_peered":"2024-08-30T23:58:34.339746+0000","last_clean":"2024-08-30T23:58:34.339746+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-30T23:58:34.339746+0000","last_undegraded":"2024-08-30T23:58:34.339746+0000","last_fullsized":"2024-08-30T23:58:34.339746+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:11.457846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:11.457846+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:28.511611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":437,"reported_epoch":973,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:58:34.289898+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-30T23:58:34.289898+0000","last_peered":"2024-08-30T23:58:34.289898+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-30T23:58:34.289898+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.10","version":"974'2763","reported_seq":4651,"reported_epoch":974,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:41.689627+0000","last_change":"2024-08-30T23:58:05.325967+0000","last_active":"2024-08-30T23:58:41.689627+0000","last_peered":"2024-08-30T23:58:41.689627+0000","last_clean":"2024-08-30T23:58:41.689627+0000","last_became_active":"2024-08-30T23:58:04.118964+0000","last_became_peered":"2024-08-30T23:58:04.118964+0000","last_unstale":"2024-08-30T23:58:41.689627+0000","last_undegraded":"2024-08-30T23:58:41.689627+0000","last_fullsized":"2024-08-30T23:58:41.689627+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'1457","last_scrub_stamp":"2024-08-30T23:58:05.325770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:05.325770+0000","objects_scrubbed":883,"log_size":2763,"log_dups_size":0,"ondisk_log_size":2763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:12.092182+0000","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144179200,"num_objects":2200,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2200,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1317,"num_write_kb":84288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":2200}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":218,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.340075+0000","last_change":"2024-08-30T23:56:45.191296+0000","last_active":"2024-08-30T23:58:41.340075+0000","last_peered":"2024-08-30T23:58:41.340075+0000","last_clean":"2024-08-30T23:58:41.340075+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-30T23:58:41.340075+0000","last_undegraded":"2024-08-30T23:58:41.340075+0000","last_fullsized":"2024-08-30T23:58:41.340075+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.191245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.191245+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.11","version":"973'2634","reported_seq":2748,"reported_epoch":973,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:38.881263+0000","last_change":"2024-08-30T23:57:57.163576+0000","last_active":"2024-08-30T23:58:38.881263+0000","last_peered":"2024-08-30T23:58:38.881263+0000","last_clean":"2024-08-30T23:58:38.881263+0000","last_became_active":"2024-08-30T23:57:56.118862+0000","last_became_peered":"2024-08-30T23:57:56.118862+0000","last_unstale":"2024-08-30T23:58:38.881263+0000","last_undegraded":"2024-08-30T23:58:38.881263+0000","last_fullsized":"2024-08-30T23:58:38.881263+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"952'1150","last_scrub_stamp":"2024-08-30T23:57:57.163327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:57.163327+0000","objects_scrubbed":628,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:18.954084+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138674176,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2116,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":95232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,9],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"3,6,9","objects":2116}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":216,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.334232+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-30T23:58:41.334232+0000","last_peered":"2024-08-30T23:58:41.334232+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-30T23:58:41.334232+0000","last_undegraded":"2024-08-30T23:58:41.334232+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"973'2626","reported_seq":4572,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:39.792762+0000","last_change":"2024-08-30T23:58:04.315602+0000","last_active":"2024-08-30T23:58:39.792762+0000","last_peered":"2024-08-30T23:58:39.792762+0000","last_clean":"2024-08-30T23:58:39.792762+0000","last_became_active":"2024-08-30T23:58:04.118740+0000","last_became_peered":"2024-08-30T23:58:04.118740+0000","last_unstale":"2024-08-30T23:58:39.792762+0000","last_undegraded":"2024-08-30T23:58:39.792762+0000","last_fullsized":"2024-08-30T23:58:39.792762+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1428","last_scrub_stamp":"2024-08-30T23:58:04.315539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.315539+0000","objects_scrubbed":909,"log_size":2626,"log_dups_size":0,"ondisk_log_size":2626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:09.159598+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138084352,"num_objects":2107,"num_object_clones":0,"num_object_copies":6321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1198,"num_write_kb":76672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":224,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.339480+0000","last_change":"2024-08-30T23:57:32.122253+0000","last_active":"2024-08-30T23:58:34.339480+0000","last_peered":"2024-08-30T23:58:34.339480+0000","last_clean":"2024-08-30T23:58:34.339480+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-30T23:58:34.339480+0000","last_undegraded":"2024-08-30T23:58:34.339480+0000","last_fullsized":"2024-08-30T23:58:34.339480+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.122186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.122186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:58:53.855972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"974'2748","reported_seq":4631,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.340794+0000","last_change":"2024-08-30T23:58:03.245375+0000","last_active":"2024-08-30T23:58:41.340794+0000","last_peered":"2024-08-30T23:58:41.340794+0000","last_clean":"2024-08-30T23:58:41.340794+0000","last_became_active":"2024-08-30T23:58:03.115334+0000","last_became_peered":"2024-08-30T23:58:03.115334+0000","last_unstale":"2024-08-30T23:58:41.340794+0000","last_undegraded":"2024-08-30T23:58:41.340794+0000","last_fullsized":"2024-08-30T23:58:41.340794+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'1406","last_scrub_stamp":"2024-08-30T23:58:03.245266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:03.245266+0000","objects_scrubbed":849,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:11.172139+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143720448,"num_objects":2193,"num_object_clones":0,"num_object_copies":6579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1344,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,5],"acting":[11,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":224,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:34.339298+0000","last_change":"2024-08-30T23:57:53.388159+0000","last_active":"2024-08-30T23:58:34.339298+0000","last_peered":"2024-08-30T23:58:34.339298+0000","last_clean":"2024-08-30T23:58:34.339298+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-30T23:58:34.339298+0000","last_undegraded":"2024-08-30T23:58:34.339298+0000","last_fullsized":"2024-08-30T23:58:34.339298+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:53.388110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.388110+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:00.379789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.14","version":"974'2755","reported_seq":4592,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.878263+0000","last_change":"2024-08-30T23:58:04.503326+0000","last_active":"2024-08-30T23:58:41.878263+0000","last_peered":"2024-08-30T23:58:41.878263+0000","last_clean":"2024-08-30T23:58:41.878263+0000","last_became_active":"2024-08-30T23:58:04.118179+0000","last_became_peered":"2024-08-30T23:58:04.118179+0000","last_unstale":"2024-08-30T23:58:41.878263+0000","last_undegraded":"2024-08-30T23:58:41.878263+0000","last_fullsized":"2024-08-30T23:58:41.878263+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1388","last_scrub_stamp":"2024-08-30T23:58:04.503242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.503242+0000","objects_scrubbed":845,"log_size":2755,"log_dups_size":0,"ondisk_log_size":2755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:07.877140+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144965632,"num_objects":2212,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":87488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":226,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.337319+0000","last_change":"2024-08-30T23:58:38.228006+0000","last_active":"2024-08-30T23:58:41.337319+0000","last_peered":"2024-08-30T23:58:41.337319+0000","last_clean":"2024-08-30T23:58:41.337319+0000","last_became_active":"2024-08-30T23:55:03.321559+0000","last_became_peered":"2024-08-30T23:55:03.321559+0000","last_unstale":"2024-08-30T23:58:41.337319+0000","last_undegraded":"2024-08-30T23:58:41.337319+0000","last_fullsized":"2024-08-30T23:58:41.337319+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:38.227924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:38.227924+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:52.979059+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.15","version":"973'2788","reported_seq":4847,"reported_epoch":973,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:58:40.688150+0000","last_change":"2024-08-30T23:58:03.667578+0000","last_active":"2024-08-30T23:58:40.688150+0000","last_peered":"2024-08-30T23:58:40.688150+0000","last_clean":"2024-08-30T23:58:40.688150+0000","last_became_active":"2024-08-30T23:58:03.116440+0000","last_became_peered":"2024-08-30T23:58:03.116440+0000","last_unstale":"2024-08-30T23:58:40.688150+0000","last_undegraded":"2024-08-30T23:58:40.688150+0000","last_fullsized":"2024-08-30T23:58:40.688150+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'1441","last_scrub_stamp":"2024-08-30T23:58:03.667484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:03.667484+0000","objects_scrubbed":908,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:17.324288+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147783680,"num_objects":2255,"num_object_clones":0,"num_object_copies":6765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2255,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1347,"num_write_kb":86208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4,5],"avail_no_missing":["8","4","5"],"object_location_counts":[{"shards":"4,5,8","objects":2255}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":214,"reported_epoch":974,"state":"active+undersized","last_fresh":"2024-08-30T23:58:41.337452+0000","last_change":"2024-08-30T23:55:59.172632+0000","last_active":"2024-08-30T23:58:41.337452+0000","last_peered":"2024-08-30T23:58:41.337452+0000","last_clean":"2024-08-30T23:55:57.160064+0000","last_became_active":"2024-08-30T23:55:59.172632+0000","last_became_peered":"2024-08-30T23:55:59.172632+0000","last_unstale":"2024-08-30T23:58:41.337452+0000","last_undegraded":"2024-08-30T23:58:41.337452+0000","last_fullsized":"2024-08-30T23:55:59.167996+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.344869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.344869+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8],"avail_no_missing":["11","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.16","version":"973'2790","reported_seq":5267,"reported_epoch":974,"state":"active+clean","last_fresh":"2024-08-30T23:58:41.337338+0000","last_change":"2024-08-30T23:58:06.471559+0000","last_active":"2024-08-30T23:58:41.337338+0000","last_peered":"2024-08-30T23:58:41.337338+0000","last_clean":"2024-08-30T23:58:41.337338+0000","last_became_active":"2024-08-30T23:58:05.118500+0000","last_became_peered":"2024-08-30T23:58:05.118500+0000","last_unstale":"2024-08-30T23:58:41.337338+0000","last_undegraded":"2024-08-30T23:58:41.337338+0000","last_fullsized":"2024-08-30T23:58:41.337338+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'1469","last_scrub_stamp":"2024-08-30T23:58:06.471505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:06.471505+0000","objects_scrubbed":927,"log_size":2790,"log_dups_size":0,"ondisk_log_size":2790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:22.684820+0000","scrub_duration":392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147390464,"num_objects":2249,"num_object_clones":0,"num_object_copies":6747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1322,"num_write_kb":84608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.17","version":"973'2698","reported_seq":4909,"reported_epoch":973,"state":"active+clean","last_fresh":"2024-08-30T23:58:40.645042+0000","last_change":"2024-08-30T23:58:06.151958+0000","last_active":"2024-08-30T23:58:40.645042+0000","last_peered":"2024-08-30T23:58:40.645042+0000","last_clean":"2024-08-30T23:58:40.645042+0000","last_became_active":"2024-08-30T23:58:05.121618+0000","last_became_peered":"2024-08-30T23:58:05.121618+0000","last_unstale":"2024-08-30T23:58:40.645042+0000","last_undegraded":"2024-08-30T23:58:40.645042+0000","last_fullsized":"2024-08-30T23:58:40.645042+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'1460","last_scrub_stamp":"2024-08-30T23:58:06.151812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:06.151812+0000","objects_scrubbed":907,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:24.403703+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141033472,"num_objects":2152,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1245,"num_write_kb":79680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":36,"stat_sum":{"num_bytes":4584144896,"num_objects":69950,"num_object_clones":0,"num_object_copies":209850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24133,"num_objects_misplaced":18243,"num_objects_unfound":0,"num_objects_dirty":69950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48290,"num_write_kb":3090432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4555,"num_bytes_recovered":298516480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13500809216,"data_stored":13500809216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109346,"ondisk_log_size":109346,"up":96,"acting":94,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":81,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187867,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1723308,"kb_used_data":1016028,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92021844,"statfs":{"total":95995035648,"available":94230368256,"internally_reserved":0,"allocated":1040412672,"data_stored":1039575002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":724236740},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449508,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2072952,"kb_used_data":1383720,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91672200,"statfs":{"total":95995035648,"available":93872332800,"internally_reserved":0,"allocated":1416929280,"data_stored":1416087587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":705755584},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2357996,"kb_used_data":1514652,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":91387156,"statfs":{"total":95995035648,"available":93580447744,"internally_reserved":0,"allocated":1551003648,"data_stored":1550165978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":863566272},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972779,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1268140,"kb_used_data":727452,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92477012,"statfs":{"total":95995035648,"available":94696460288,"internally_reserved":0,"allocated":744910848,"data_stored":744073178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528757,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294200,"kb_used_data":775720,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92450952,"statfs":{"total":95995035648,"available":94669774848,"internally_reserved":0,"allocated":794337280,"data_stored":793495587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775260,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2930612,"kb_used_data":2516004,"kb_used_omap":5,"kb_used_meta":414586,"kb_avail":90814540,"statfs":{"total":95995035648,"available":92994088960,"internally_reserved":0,"allocated":2576388096,"data_stored":2575546874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":424536138},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953920,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475820,"kb_used_data":732444,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92269332,"statfs":{"total":95995035648,"available":94483795968,"internally_reserved":0,"allocated":750022656,"data_stored":749184986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150145,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2121580,"kb_used_data":1183836,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":91623572,"statfs":{"total":95995035648,"available":93822537728,"internally_reserved":0,"allocated":1212248064,"data_stored":1211410394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":960231879},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411784,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972052,"kb_used_data":309180,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92773100,"statfs":{"total":95995035648,"available":94999654400,"internally_reserved":0,"allocated":316600320,"data_stored":315751060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830600,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2861420,"kb_used_data":2268444,"kb_used_omap":87,"kb_used_meta":592872,"kb_avail":90883732,"statfs":{"total":95995035648,"available":93064941568,"internally_reserved":0,"allocated":2322886656,"data_stored":2322048986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":607101087},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738834,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1476716,"kb_used_data":647260,"kb_used_omap":1,"kb_used_meta":829438,"kb_avail":92268436,"statfs":{"total":95995035648,"available":94482878464,"internally_reserved":0,"allocated":662794240,"data_stored":661956570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":849344965},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":678428672,"data_stored":678428672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2329542656,"data_stored":2329542656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317784064,"data_stored":317784064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1246101504,"data_stored":1246101504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":757071872,"data_stored":757071872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2610495488,"data_stored":2610495488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":794492928,"data_stored":794492928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":756744192,"data_stored":756744192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1553924096,"data_stored":1553924096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1417412608,"data_stored":1417412608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1038811136,"data_stored":1038811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:58:50.573 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:50.676 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:50.698 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:50.801 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:50.828 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:50.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-30T23:58:50.930 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:50.956 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:50.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:50.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 68 16 80994 80978 74.4218 51.4375 0.00436034 0.0133921 2024-08-30T23:58:51.057 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:51.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:58:51.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:58:51.185 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:51.192 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:51.219 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:51.253 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:51.322 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:51.347 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:51.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:51.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:51.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:51 smithi106 ceph-mon[16603]: pgmap v2166: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 21391/233424 objects degraded (9.164%); 17169/233424 objects misplaced (7.355%); 9.8 MiB/s, 157 objects/s recovering 2024-08-30T23:58:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:51 smithi106 ceph-mon[16603]: osdmap e978: 12 total, 11 up, 12 in 2024-08-30T23:58:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 21391/233424 objects degraded (9.164%), 11 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:51.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:51 smithi106 ceph-mon[16603]: osdmap e979: 12 total, 11 up, 12 in 2024-08-30T23:58:51.611 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:51.652 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:51 smithi173 ceph-mon[20554]: pgmap v2166: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 21391/233424 objects degraded (9.164%); 17169/233424 objects misplaced (7.355%); 9.8 MiB/s, 157 objects/s recovering 2024-08-30T23:58:51.652 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:51 smithi173 ceph-mon[20554]: osdmap e978: 12 total, 11 up, 12 in 2024-08-30T23:58:51.652 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 21391/233424 objects degraded (9.164%), 11 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:51.653 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:51 smithi173 ceph-mon[20554]: osdmap e979: 12 total, 11 up, 12 in 2024-08-30T23:58:51.713 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:51.750 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:51 smithi136 ceph-mon[20207]: pgmap v2166: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.7 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 824 op/s; 21391/233424 objects degraded (9.164%); 17169/233424 objects misplaced (7.355%); 9.8 MiB/s, 157 objects/s recovering 2024-08-30T23:58:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:51 smithi136 ceph-mon[20207]: osdmap e978: 12 total, 11 up, 12 in 2024-08-30T23:58:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 21391/233424 objects degraded (9.164%), 11 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:51 smithi136 ceph-mon[20207]: osdmap e979: 12 total, 11 up, 12 in 2024-08-30T23:58:51.800 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:51.800 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:51.852 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:51.893 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:51.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:51.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 69 16 82391 82375 74.6085 87.3125 0.00279057 0.0133987 2024-08-30T23:58:51.994 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:52.023 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:52.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:52.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:52.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:52.288 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:52.390 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:52.391 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:58:52.428 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:52.520 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-30T23:58:52.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:52.554 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:52.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:52.679 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:52.721 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2214861934' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:58:52.764 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:52.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:52.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2214861934' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:58:52.803 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:52.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2214861934' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:58:52.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:52.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:52.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:52.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 70 16 83204 83188 74.2685 50.8125 0.0119524 0.0133975 2024-08-30T23:58:53.030 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:53.052 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:53.140 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:58:53.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:53.175 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:53.262 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-30T23:58:53.316 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:58:53.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:53.338 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:53.398 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:53.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.475 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:53.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: pgmap v2169: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 21545/237237 objects degraded (9.082%); 17521/237237 objects misplaced (7.385%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: 9.1d scrub starts 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2214861934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: osdmap e980: 12 total, 11 up, 12 in 2024-08-30T23:58:53.535 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:53 smithi106 ceph-mon[16603]: osdmap e981: 12 total, 11 up, 12 in 2024-08-30T23:58:53.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:53.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: pgmap v2169: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 21545/237237 objects degraded (9.082%); 17521/237237 objects misplaced (7.385%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: 9.1d scrub starts 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2214861934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: osdmap e980: 12 total, 11 up, 12 in 2024-08-30T23:58:53.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:53 smithi136 ceph-mon[20207]: osdmap e981: 12 total, 11 up, 12 in 2024-08-30T23:58:53.706 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:53.756 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: pgmap v2169: 67 pgs: 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s; 21545/237237 objects degraded (9.082%); 17521/237237 objects misplaced (7.385%); 18 MiB/s, 294 objects/s recovering 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: 9.1d scrub starts 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2214861934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:58:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: osdmap e980: 12 total, 11 up, 12 in 2024-08-30T23:58:53.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:53 smithi173 ceph-mon[20554]: osdmap e981: 12 total, 11 up, 12 in 2024-08-30T23:58:53.858 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:53.884 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:53.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:53.941 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-30T23:58:53.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 71 16 84096 84080 74.0076 55.75 0.009397 0.0135063 2024-08-30T23:58:53.986 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:54.015 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:54.022 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:58:54.023 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:58:54.055 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:58:54.056 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:58:54.087 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:58:54.087 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:58:54.118 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:54.121 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:58:54.121 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:58:54.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:54.152 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:58:54.152 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:58:54.183 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:58:54.183 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:58:54.216 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:58:54.216 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:58:54.217 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:58:54.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:54.247 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:58:54.247 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:58:54.261 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:54.274 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:54.277 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:58:54.277 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:58:54.307 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:58:54.307 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:58:54.337 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:58:54.337 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:58:54.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:54.404 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:54.466 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:54 smithi136 ceph-mon[20207]: 9.1f scrub starts 2024-08-30T23:58:54.466 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/72630005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:54.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:54.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:54.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:54.664 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:54 smithi173 ceph-mon[20554]: 9.1f scrub starts 2024-08-30T23:58:54.664 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/72630005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:54.666 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:54.768 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:54.774 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.28405.0:84865 9.14 e982/957 9:28545254:::benchmark_data_smithi173_160189_object84864:head v 982'3238, mlcod=982'3236)", 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:54.772839+0000", 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0016395260000000001, 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001666135, 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:54.775 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:54.776 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:54.776 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.776 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:54.776 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.772839+0000", 2024-08-30T23:58:54.776 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:54.776 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.777 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.777 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:54.777 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.772839+0000", 2024-08-30T23:58:54.777 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:54.777 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.772840+0000", 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5799999999999998e-07 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.773046+0000", 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000205985 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.778 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.773069+0000", 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3059000000000001e-05 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.773074+0000", 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9459999999999997e-06 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.779 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.773116+0000", 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1677000000000001e-05 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:54.773136+0000", 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9749999999999999e-05 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:54.780 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:54.781 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-30T23:58:54.781 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 1 2024-08-30T23:58:54.781 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:58:54.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:54 smithi106 ceph-mon[16603]: 9.1f scrub starts 2024-08-30T23:58:54.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/72630005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:58:54.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:54.921 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:54.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:54.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 72 16 84948 84932 73.7193 53.25 0.097563 0.0135218 2024-08-30T23:58:55.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:55.048 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:55.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:55.171 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:55.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:58:55.294 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.295 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:58:55.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:58:55.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:58:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.419 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:55.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:58:55.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-30T23:58:55.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:55.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:55.568 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:58:55.568 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:58:55.569 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-30T23:58:55.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd out 11 2024-08-30T23:58:55.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:55 smithi106 ceph-mon[16603]: pgmap v2172: 67 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 773 op/s; 19657/242709 objects degraded (8.099%); 17982/242709 objects misplaced (7.409%); 19 MiB/s, 296 objects/s recovering 2024-08-30T23:58:55.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:55 smithi106 ceph-mon[16603]: osdmap e982: 12 total, 11 up, 12 in 2024-08-30T23:58:55.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:58:55.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:55 smithi173 ceph-mon[20554]: pgmap v2172: 67 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 773 op/s; 19657/242709 objects degraded (8.099%); 17982/242709 objects misplaced (7.409%); 19 MiB/s, 296 objects/s recovering 2024-08-30T23:58:55.663 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:55 smithi173 ceph-mon[20554]: osdmap e982: 12 total, 11 up, 12 in 2024-08-30T23:58:55.691 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:58:55.742 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:55 smithi136 ceph-mon[20207]: pgmap v2172: 67 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 38 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 773 op/s; 19657/242709 objects degraded (8.099%); 17982/242709 objects misplaced (7.409%); 19 MiB/s, 296 objects/s recovering 2024-08-30T23:58:55.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:55 smithi136 ceph-mon[20207]: osdmap e982: 12 total, 11 up, 12 in 2024-08-30T23:58:55.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:55.819 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:55.921 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:55.946 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:55.947 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:55.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 73 16 85702 85686 73.3549 47.125 0.00839358 0.0136271 2024-08-30T23:58:56.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:56.072 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:56.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:56.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:56.260 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:58:56.261 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:58:56.261 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:58:56.261 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:58:56.261 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:58:56.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:56.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:56.432 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:56.459 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:56.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:58:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:56 smithi173 ceph-mon[20554]: osdmap e983: 12 total, 11 up, 12 in 2024-08-30T23:58:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 19657/242709 objects degraded (8.099%), 10 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:56.593 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.594 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:58:56.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:56.724 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:56 smithi136 ceph-mon[20207]: osdmap e983: 12 total, 11 up, 12 in 2024-08-30T23:58:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 19657/242709 objects degraded (8.099%), 10 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:56.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:56.850 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:56 smithi106 ceph-mon[16603]: osdmap e983: 12 total, 11 up, 12 in 2024-08-30T23:58:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 19657/242709 objects degraded (8.099%), 10 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:58:56.952 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:56.965 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-30T23:58:56.972 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:56.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:56.985 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 74 16 87018 87002 73.475 82.25 0.00220833 0.0135903 2024-08-30T23:58:57.074 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:57.097 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:57.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:57.224 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:57.265 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:57.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:57.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:57.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:57.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:57.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:57.620 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:57.677 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:57 smithi173 ceph-mon[20554]: pgmap v2175: 67 pgs: 1 active+recovery_wait+remapped, 4 active+recovering+undersized+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 39 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 15548/251160 objects degraded (6.190%); 21147/251160 objects misplaced (8.420%); 31 MiB/s, 502 objects/s recovering 2024-08-30T23:58:57.677 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:57 smithi173 ceph-mon[20554]: 9.1a scrub starts 2024-08-30T23:58:57.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:57.748 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:57.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:57 smithi136 ceph-mon[20207]: pgmap v2175: 67 pgs: 1 active+recovery_wait+remapped, 4 active+recovering+undersized+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 39 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 15548/251160 objects degraded (6.190%); 21147/251160 objects misplaced (8.420%); 31 MiB/s, 502 objects/s recovering 2024-08-30T23:58:57.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:57 smithi136 ceph-mon[20207]: 9.1a scrub starts 2024-08-30T23:58:57.815 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86717 7.e 7:7cc26902:::benchmark_data_smithi173_129296_object86716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e755)", 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:17.100563+0000", 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 520.70334982199995, 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45683305899999999, 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.816 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86717 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100563+0000", 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100563+0000", 2024-08-30T23:58:57.817 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100564+0000", 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100582+0000", 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.8272999999999999e-05 2024-08-30T23:58:57.818 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100591+0000", 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7930000000000003e-06 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100595+0000", 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6970000000000001e-06 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100638+0000", 2024-08-30T23:58:57.819 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2515999999999997e-05 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100737+0000", 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9041000000000003e-05 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.100828+0000", 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1121999999999999e-05 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.820 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.821 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103141+0000", 2024-08-30T23:58:57.821 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023128519999999998 2024-08-30T23:58:57.821 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.821 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.821 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.822 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103779+0000", 2024-08-30T23:58:57.822 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00063816000000000003 2024-08-30T23:58:57.822 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.822 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.822 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557286+0000", 2024-08-30T23:58:57.823 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45350701999999998 2024-08-30T23:58:57.823 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.823 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.823 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.823 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557348+0000", 2024-08-30T23:58:57.824 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2018000000000002e-05 2024-08-30T23:58:57.824 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.824 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.824 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.824 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557396+0000", 2024-08-30T23:58:57.825 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7963000000000001e-05 2024-08-30T23:58:57.825 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.825 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.825 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.825 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.825 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.826 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27487.0:86717 7.e e755/725 ondisk, result = 0)", 2024-08-30T23:58:57.826 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:17.103595+0000", 2024-08-30T23:58:57.826 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 520.70031762500003, 2024-08-30T23:58:57.826 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45380901800000001, 2024-08-30T23:58:57.826 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.826 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.827 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:57.827 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.827 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.827 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.827 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103595+0000", 2024-08-30T23:58:57.827 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103595+0000", 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103596+0000", 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2099999999999996e-07 2024-08-30T23:58:57.828 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.829 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.829 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.103600+0000", 2024-08-30T23:58:57.829 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3919999999999996e-06 2024-08-30T23:58:57.829 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.829 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557219+0000", 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45361822299999999 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557226+0000", 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6750000000000002e-06 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557268+0000", 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1674000000000001e-05 2024-08-30T23:58:57.830 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557279+0000", 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0969e-05 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:17.557404+0000", 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012536399999999999 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.831 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87834 7.e 7:7ff39eeb:::benchmark_data_smithi173_129296_object718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:50.625937+0000", 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 487.17797551199999, 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36790855099999997, 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87834 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.832 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625937+0000", 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625937+0000", 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625938+0000", 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5700000000000002e-07 2024-08-30T23:58:57.833 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625942+0000", 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.393e-06 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625951+0000", 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8600000000000005e-06 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625957+0000", 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.834 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627053+0000", 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001096334 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627171+0000", 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011757899999999999 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.835 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627278+0000", 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010718299999999999 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.629558+0000", 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022799309999999998 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.836 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.630384+0000", 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082581199999999999 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993774+0000", 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.363390306 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993807+0000", 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3179999999999997e-05 2024-08-30T23:58:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993846+0000", 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8665999999999999e-05 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27487.0:87834 7.e e771/725 ondisk, result = 0)", 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:50.634204+0000", 2024-08-30T23:58:57.838 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 487.16970854900001, 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35964531900000002, 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634204+0000", 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.839 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634204+0000", 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634205+0000", 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634208+0000", 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9459999999999998e-06 2024-08-30T23:58:57.840 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993718+0000", 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.359510361 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993722+0000", 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0999999999999997e-06 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.841 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993764+0000", 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1885999999999998e-05 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993769+0000", 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2140000000000004e-06 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993850+0000", 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0378999999999997e-05 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.842 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27487.0:123065 7.1 e792/725 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head v 792'7770, mlcod=792'7767)", 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:54.487833+0000", 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 423.31607984099998, 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64708236699999999, 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487833+0000", 2024-08-30T23:58:57.843 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487833+0000", 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487834+0000", 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9999999999999996e-07 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.844 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487840+0000", 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2280000000000003e-06 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130467+0000", 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64262676900000004 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130489+0000", 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2246e-05 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.845 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130670+0000", 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018123 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130713+0000", 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2786e-05 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.134863+0000", 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041495330000000004 2024-08-30T23:58:57.846 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.134915+0000", 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2775000000000002e-05 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.847 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.848 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.848 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.848 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:127005 7.e 7:7c88bf46:::benchmark_data_smithi173_129296_object39889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-08-30T23:58:57.848 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:52:02.842372+0000", 2024-08-30T23:58:57.848 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 414.96154044799999, 2024-08-30T23:58:57.848 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.363980886, 2024-08-30T23:58:57.849 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.849 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.849 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.849 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.849 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-30T23:58:57.850 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-30T23:58:57.850 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127005 2024-08-30T23:58:57.850 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.850 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.850 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.850 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842372+0000", 2024-08-30T23:58:57.851 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.851 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.851 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.851 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.851 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842372+0000", 2024-08-30T23:58:57.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.852 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.852 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.852 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.852 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842374+0000", 2024-08-30T23:58:57.852 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5570000000000001e-06 2024-08-30T23:58:57.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.853 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.853 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.853 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842381+0000", 2024-08-30T23:58:57.853 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3799999999999996e-06 2024-08-30T23:58:57.853 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842395+0000", 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.855 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842405+0000", 2024-08-30T23:58:57.855 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5689999999999994e-06 2024-08-30T23:58:57.855 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.855 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.855 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842460+0000", 2024-08-30T23:58:57.856 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5458000000000003e-05 2024-08-30T23:58:57.856 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.856 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.856 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.856 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842547+0000", 2024-08-30T23:58:57.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-08-30T23:58:57.857 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.857 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.857 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:58:57.857 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842621+0000", 2024-08-30T23:58:57.857 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3875000000000007e-05 2024-08-30T23:58:57.860 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.860 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.860 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.860 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.846407+0000", 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037859090000000001 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.846902+0000", 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000495124 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.861 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206262+0000", 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35936010899999998 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206314+0000", 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1866000000000001e-05 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206353+0000", 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9736e-05 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.862 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:21765 8.9 8:9868b323:::benchmark_data_smithi173_143816_object21764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:10.078955+0000", 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 287.72495780100002, 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.30126246000000001, 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:57.863 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21765 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078955+0000", 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078955+0000", 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.864 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078956+0000", 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.049e-06 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078983+0000", 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6998000000000001e-05 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.078994+0000", 2024-08-30T23:58:57.865 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1265e-05 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079000+0000", 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3399999999999997e-06 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079044+0000", 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4563e-05 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.866 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079162+0000", 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000117231 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.079249+0000", 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7340000000000001e-05 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080081+0000", 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00083194900000000001 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.867 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.080390+0000", 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00030953600000000001 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380119+0000", 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.29972854700000001 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380176+0000", 2024-08-30T23:58:57.868 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6737999999999998e-05 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.380218+0000", 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1903999999999997e-05 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:21811 8.5 8:a2778f2f:::benchmark_data_smithi173_143816_object21810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:58:57.869 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 286.85132637100003, 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34422484399999997, 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 21811 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952587+0000", 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5899999999999995e-07 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.871 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952609+0000", 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1517000000000001e-05 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952621+0000", 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2041999999999999e-05 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952628+0000", 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8589999999999999e-06 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.873 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:57.874 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:57 smithi106 ceph-mon[16603]: pgmap v2175: 67 pgs: 1 active+recovery_wait+remapped, 4 active+recovering+undersized+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 39 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s; 15548/251160 objects degraded (6.190%); 21147/251160 objects misplaced (8.420%); 31 MiB/s, 502 objects/s recovering 2024-08-30T23:58:57.875 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:57 smithi106 ceph-mon[16603]: 9.1a scrub starts 2024-08-30T23:58:57.875 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952666+0000", 2024-08-30T23:58:57.875 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8696999999999999e-05 2024-08-30T23:58:57.875 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952746+0000", 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9958999999999998e-05 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.952828+0000", 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2070000000000005e-05 2024-08-30T23:58:57.876 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.953606+0000", 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00077772200000000001 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:10.954157+0000", 2024-08-30T23:58:57.877 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00055049299999999999 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:11.296625+0000", 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34246814399999997 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.878 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:11.296725+0000", 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010038400000000001 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:11.296811+0000", 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6198000000000004e-05 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.879 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36149 8.6 e860/854)", 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.055209+0000", 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 273.74870435999998, 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41090775600000001, 2024-08-30T23:58:57.880 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055209+0000", 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.881 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055209+0000", 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.882 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055210+0000", 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0380000000000001e-06 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055243+0000", 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3021000000000001e-05 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.883 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.465880+0000", 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.410637788 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.465894+0000", 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3969e-05 2024-08-30T23:58:57.884 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.466085+0000", 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019035300000000001 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.885 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.466116+0000", 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1587000000000001e-05 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36527 8.5 8:a4f61870:::benchmark_data_smithi173_143816_object36526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.366820+0000", 2024-08-30T23:58:57.886 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 272.43709318499998, 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42580382900000002, 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36527 2024-08-30T23:58:57.887 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366820+0000", 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.888 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366820+0000", 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366821+0000", 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.889 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366840+0000", 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9429e-05 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366849+0000", 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3600000000000002e-06 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.890 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366854+0000", 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4880000000000001e-06 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366888+0000", 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4350000000000001e-05 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366994+0000", 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010563700000000001 2024-08-30T23:58:57.891 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.367078+0000", 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3735999999999999e-05 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.367886+0000", 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00080812600000000005 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368184+0000", 2024-08-30T23:58:57.892 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029837400000000001 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792529+0000", 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42434479800000002 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792588+0000", 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8922000000000003e-05 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.893 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792624+0000", 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5751999999999999e-05 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36527 8.5 e862/840 ondisk, result = 0)", 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.368528+0000", 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 272.43538462499998, 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.424101687, 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.894 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368528+0000", 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368528+0000", 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.895 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368529+0000", 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9100000000000004e-07 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368532+0000", 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8169999999999999e-06 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792461+0000", 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.423929155 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.896 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792467+0000", 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8019999999999997e-06 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792512+0000", 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5624999999999998e-05 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792521+0000", 2024-08-30T23:58:57.897 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1540000000000008e-06 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792630+0000", 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108643 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36528 8.7 e862/854 8:ed4e2712:::benchmark_data_smithi173_143816_object36527:head v 862'2374, mlcod=845'96)", 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.368738+0000", 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 272.43517455099999, 2024-08-30T23:58:57.898 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45802140800000002, 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368738+0000", 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.899 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368738+0000", 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368739+0000", 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5300000000000004e-07 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368831+0000", 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2077999999999995e-05 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.900 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825237+0000", 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45640643600000003 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825246+0000", 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0749999999999998e-06 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825305+0000", 2024-08-30T23:58:57.901 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9893999999999999e-05 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825332+0000", 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6239000000000001e-05 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.826715+0000", 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013836230000000001 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.902 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.826760+0000", 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4509999999999999e-05 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36535 8.9 8:9fb542f2:::benchmark_data_smithi173_143816_object36534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.395482+0000", 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 272.40843077, 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41703795100000002, 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.903 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36535 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395482+0000", 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395482+0000", 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395484+0000", 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.562e-06 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395507+0000", 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3461000000000001e-05 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395523+0000", 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6069000000000001e-05 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395531+0000", 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0039999999999999e-06 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395580+0000", 2024-08-30T23:58:57.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8621000000000001e-05 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395686+0000", 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106156 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395786+0000", 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9801000000000002e-05 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.396991+0000", 2024-08-30T23:58:57.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012049350000000001 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.397402+0000", 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041134199999999998 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812443+0000", 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41504124100000001 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812477+0000", 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4093e-05 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812520+0000", 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2666000000000001e-05 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37809 8.2 8:45f707ad:::benchmark_data_smithi173_143816_object37808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:27.427735+0000", 2024-08-30T23:58:57.909 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 270.37617757599998, 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37169286800000001, 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37809 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427735+0000", 2024-08-30T23:58:57.910 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427735+0000", 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427737+0000", 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1850000000000001e-06 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.911 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427757+0000", 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0934999999999999e-05 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427771+0000", 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3672999999999999e-05 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427779+0000", 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7999999999999999e-06 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427829+0000", 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9795e-05 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427918+0000", 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9624999999999997e-05 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.428017+0000", 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8312000000000003e-05 2024-08-30T23:58:57.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.429169+0000", 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011526660000000001 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.456373+0000", 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027204145999999998 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799342+0000", 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34296861200000001 2024-08-30T23:58:57.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799393+0000", 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1048999999999999e-05 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799428+0000", 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5070000000000001e-05 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:45691 8.5 e869/840 ondisk, result = 0)", 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:36.332347+0000", 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 261.47156622699998, 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32774751699999999, 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.332347+0000", 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.332347+0000", 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.332347+0000", 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4300000000000003e-07 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.332350+0000", 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0819999999999999e-06 2024-08-30T23:58:57.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.659885+0000", 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.32753507100000001 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.659900+0000", 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4643e-05 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.659999+0000", 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8560000000000005e-05 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.660007+0000", 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9160000000000001e-06 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:36.660094+0000", 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7602000000000006e-05 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:46104 9.18 9:1d3ec02e:::benchmark_data_smithi173_160189_object46103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e963)", 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:17.917134+0000", 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 39.886778772, 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31642663199999999, 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.920 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 46104 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917134+0000", 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917134+0000", 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917135+0000", 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.249e-06 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917156+0000", 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0610000000000001e-05 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917167+0000", 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1056000000000001e-05 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917173+0000", 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4419999999999997e-06 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917215+0000", 2024-08-30T23:58:57.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2758e-05 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917312+0000", 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6317999999999995e-05 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,9", 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.917447+0000", 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013543 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.918638+0000", 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001191103 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:17.918798+0000", 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00016015800000000001 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:18.233353+0000", 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.31455483000000001 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:18.233420+0000", 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7248000000000004e-05 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:18.233561+0000", 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014043000000000001 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.926 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:52974 9.9 9:944ce93d:::benchmark_data_smithi173_160189_object52973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.119545+0000", 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.684367881, 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36679148, 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.927 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 52974 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119545+0000", 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119545+0000", 2024-08-30T23:58:57.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119546+0000", 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.850000000000001e-07 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119566+0000", 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9652000000000001e-05 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119578+0000", 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2533e-05 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119586+0000", 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9200000000000004e-06 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119678+0000", 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1805999999999995e-05 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119825+0000", 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014685300000000001 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.120027+0000", 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020249100000000001 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121088+0000", 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001060985 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121342+0000", 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000254045 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486212+0000", 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36486941099999998 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486282+0000", 2024-08-30T23:58:57.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9850000000000004e-05 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486337+0000", 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4948999999999999e-05 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:52974 9.9 e965/948 ondisk, result = 0)", 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.121475+0000", 2024-08-30T23:58:57.933 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.682438378000001, 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36486892399999998, 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121475+0000", 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121475+0000", 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121475+0000", 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7700000000000004e-07 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121479+0000", 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1330000000000001e-06 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486082+0000", 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36460314399999999 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486095+0000", 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2636999999999999e-05 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486195+0000", 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9485e-05 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486204+0000", 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8630000000000003e-06 2024-08-30T23:58:57.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486343+0000", 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000138985 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:53033 9.9 9:9754497a:::benchmark_data_smithi173_160189_object53032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.170814+0000", 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.633099055000002, 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38473522700000001, 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.937 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 53033 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170814+0000", 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.938 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170814+0000", 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170815+0000", 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1400000000000001e-06 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170836+0000", 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1291999999999999e-05 2024-08-30T23:58:57.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170849+0000", 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3023999999999999e-05 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170858+0000", 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2020000000000004e-06 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170912+0000", 2024-08-30T23:58:57.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4877999999999999e-05 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.171067+0000", 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154122 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.171197+0000", 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013034400000000001 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172497+0000", 2024-08-30T23:58:57.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012998510000000001 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172647+0000", 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015048199999999999 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555462+0000", 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38281501800000001 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555502+0000", 2024-08-30T23:58:57.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9671e-05 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555549+0000", 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7203000000000002e-05 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:53033 9.9 e965/948 ondisk, result = 0)", 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.172911+0000", 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 35.631001918999999, 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38264409500000002, 2024-08-30T23:58:57.943 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172911+0000", 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172911+0000", 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172912+0000", 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5199999999999996e-07 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172915+0000", 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8399999999999997e-06 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555370+0000", 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.382454501 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555375+0000", 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2619999999999998e-06 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555451+0000", 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6042000000000001e-05 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555458+0000", 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2019999999999998e-06 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:58:57.946 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555555+0000", 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6595999999999995e-05 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:58:57.947 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:58:57.971 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:57.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:57.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 75 16 88277 88261 73.5444 78.6875 0.0262624 0.0135924 2024-08-30T23:58:58.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:58:58.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:58:58.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:58.225 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:58.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:58.351 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:58.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:58.478 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:58.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:58:58.603 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:58:58.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:58.729 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:58.807 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-30T23:58:58.830 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:58.882 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:58.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:58.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:58.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 76 16 89495 89479 73.5783 76.125 0.00471661 0.013586 2024-08-30T23:58:59.010 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:59.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:59.136 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:59.145 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:58:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:58:59.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:58:59.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:58:59.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:58:59.290 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:59.296 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:58:59.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:58:59.389 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:58:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:58:59.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:58:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.520 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:59.520 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:58:59.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:59 smithi173 ceph-mon[20554]: pgmap v2176: 67 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s; 16003/256419 objects degraded (6.241%); 21423/256419 objects misplaced (8.355%); 9.6 MiB/s, 152 objects/s recovering 2024-08-30T23:58:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:58:59 smithi173 ceph-mon[20554]: 9.1d scrub starts 2024-08-30T23:58:59.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.646 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:59.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:59.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:59 smithi106 ceph-mon[16603]: pgmap v2176: 67 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s; 16003/256419 objects degraded (6.241%); 21423/256419 objects misplaced (8.355%); 9.6 MiB/s, 152 objects/s recovering 2024-08-30T23:58:59.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:58:59 smithi106 ceph-mon[16603]: 9.1d scrub starts 2024-08-30T23:58:59.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:58:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.773 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:58:59.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:58:59.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:59 smithi136 ceph-mon[20207]: pgmap v2176: 67 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+remapped, 5 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s; 16003/256419 objects degraded (6.241%); 21423/256419 objects misplaced (8.355%); 9.6 MiB/s, 152 objects/s recovering 2024-08-30T23:58:59.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:58:59 smithi136 ceph-mon[20207]: 9.1d scrub starts 2024-08-30T23:58:59.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:58:59.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:58:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:58:59.902 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:58:59.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 77 16 90569 90553 73.4944 67.125 0.00304385 0.0135995 2024-08-30T23:59:00.003 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:59:00.003 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:59:00.003 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 0 2024-08-30T23:59:00.003 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:59:00.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:00.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:00.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:00.163 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:00.265 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.297 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:00.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:00.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:00.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:marked out osd.11. 2024-08-30T23:59:00.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:00.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:00.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:00.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-30T23:59:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.671 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:00.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:00.692 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:00 smithi173 ceph-mon[20554]: 9.1f scrub starts 2024-08-30T23:59:00.692 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/244474277' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:59:00.693 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:00 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:59:00.693 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:00 smithi173 ceph-mon[20554]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-30T23:59:00.773 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:00.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:00 smithi136 ceph-mon[20207]: 9.1f scrub starts 2024-08-30T23:59:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/244474277' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:59:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:00 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:59:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:00 smithi136 ceph-mon[20207]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-30T23:59:00.795 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:00 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:59:00.371+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e983 definitely_dead 0 2024-08-30T23:59:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:00 smithi106 ceph-mon[16603]: 9.1f scrub starts 2024-08-30T23:59:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/244474277' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:59:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:00 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-08-30T23:59:00.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:00 smithi106 ceph-mon[16603]: Client client.admin marked osd.11 out, while it was still marked up 2024-08-30T23:59:00.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:00.910 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:59:00.918 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:00.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:00.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 78 16 91188 91172 73.0481 38.6875 0.335949 0.0136023 2024-08-30T23:59:01.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:01.042 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:01.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:01.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:01.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:01.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:59:01.207 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:59:01.270 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:01.292 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:01.394 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:01.416 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:01.518 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:01.540 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.541 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:01.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:01.664 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:01.726 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: pgmap v2177: 67 pgs: 3 active+recovering+undersized+remapped, 7 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 13804/265269 objects degraded (5.204%); 22049/265269 objects misplaced (8.312%); 21 MiB/s, 328 objects/s recovering 2024-08-30T23:59:01.726 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: osdmap e984: 12 total, 11 up, 11 in 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1509830512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 13804/265269 objects degraded (5.204%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:01.727 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:01 smithi173 ceph-mon[20554]: osdmap e985: 12 total, 11 up, 11 in 2024-08-30T23:59:01.735 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:59:01.735 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-30T23:59:01.735 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-30T23:59:01.735 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-30T23:59:01.735 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:59:01.766 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.787 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:01.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: pgmap v2177: 67 pgs: 3 active+recovering+undersized+remapped, 7 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 13804/265269 objects degraded (5.204%); 22049/265269 objects misplaced (8.312%); 21 MiB/s, 328 objects/s recovering 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: osdmap e984: 12 total, 11 up, 11 in 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1509830512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 13804/265269 objects degraded (5.204%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:01.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:01 smithi136 ceph-mon[20207]: osdmap e985: 12 total, 11 up, 11 in 2024-08-30T23:59:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: pgmap v2177: 67 pgs: 3 active+recovering+undersized+remapped, 7 active+clean+remapped, 11 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s; 13804/265269 objects degraded (5.204%); 22049/265269 objects misplaced (8.312%); 21 MiB/s, 328 objects/s recovering 2024-08-30T23:59:01.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: osdmap e984: 12 total, 11 up, 11 in 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1509830512' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 13804/265269 objects degraded (5.204%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:01 smithi106 ceph-mon[16603]: osdmap e985: 12 total, 11 up, 11 in 2024-08-30T23:59:01.889 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:01.910 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:01.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:01.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 79 16 91211 91195 72.1416 1.4375 0.0022973 0.0136443 2024-08-30T23:59:02.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:02.034 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:02.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:02.158 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:02.260 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:02.290 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:02.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:02.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:02.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-30T23:59:02.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:02.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:02.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:02.686 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:02.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:02.807 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-30T23:59:02.813 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:02.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:02.941 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:02.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:02.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:2024-08-30T23:59:02.987121+0000 min lat: 0.000982985 max lat: 2.30386 avg lat: 0.013892 2024-08-30T23:59:02.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-30T23:59:02.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 80 16 92113 92097 71.9445 56.375 0.00693268 0.013892 2024-08-30T23:59:03.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:03.070 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:03.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:03 smithi106 ceph-mon[16603]: 9.23 scrub starts 2024-08-30T23:59:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:03 smithi106 ceph-mon[16603]: pgmap v2180: 67 pgs: 2 unknown, 3 active+recovering+undersized+remapped, 7 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.4 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 317 B/s rd, 58 MiB/s wr, 920 op/s; 13804/267615 objects degraded (5.158%); 22357/267615 objects misplaced (8.354%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:59:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:03 smithi106 ceph-mon[16603]: osdmap e986: 12 total, 11 up, 11 in 2024-08-30T23:59:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:03 smithi106 ceph-mon[16603]: 9.1a scrub starts 2024-08-30T23:59:03.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:03 smithi106 ceph-mon[16603]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:59:03.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:03.194 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:03 smithi173 ceph-mon[20554]: 9.23 scrub starts 2024-08-30T23:59:03.194 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:03 smithi173 ceph-mon[20554]: pgmap v2180: 67 pgs: 2 unknown, 3 active+recovering+undersized+remapped, 7 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.4 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 317 B/s rd, 58 MiB/s wr, 920 op/s; 13804/267615 objects degraded (5.158%); 22357/267615 objects misplaced (8.354%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:59:03.194 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:03 smithi173 ceph-mon[20554]: osdmap e986: 12 total, 11 up, 11 in 2024-08-30T23:59:03.194 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:03 smithi173 ceph-mon[20554]: 9.1a scrub starts 2024-08-30T23:59:03.194 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:03 smithi173 ceph-mon[20554]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:59:03.200 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:03.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:03.331 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:03.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:03.458 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37699 8.d 8:b57c4cbd:::benchmark_data_smithi173_143816_object37698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36831806100002, 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93994113199999996, 2024-08-30T23:59:03.492 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37699 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121523+0000", 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3480000000000001e-06 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121550+0000", 2024-08-30T23:59:03.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7135e-05 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121561+0000", 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.081e-05 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121569+0000", 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6990000000000003e-06 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:59:03.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3838999999999996e-05 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121654+0000", 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1416999999999999e-05 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061408+0000", 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93975412199999997 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061463+0000", 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4762000000000002e-05 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37724 8.d 8:bb898ecc:::benchmark_data_smithi173_143816_object37723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.368249026, 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94419919900000004, 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.497 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37724 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121616+0000", 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4661999999999999e-05 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121625+0000", 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6009999999999995e-06 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121631+0000", 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500000000000001e-06 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121691+0000", 2024-08-30T23:59:03.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9901000000000001e-05 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121695+0000", 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6849999999999999e-06 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061884+0000", 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94018891100000002 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061957+0000", 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2855999999999995e-05 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.062054+0000", 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7165999999999998e-05 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065642+0000", 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035878989999999999 2024-08-30T23:59:03.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065656+0000", 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4899000000000001e-05 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065728+0000", 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1124999999999994e-05 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065764+0000", 2024-08-30T23:59:03.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6154e-05 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065790+0000", 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6273000000000001e-05 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37735 8.d 8:bd387e6f:::benchmark_data_smithi173_143816_object37734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:59:03.504 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36819644299999, 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94428094500000004, 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37735 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:59:03.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121644+0000", 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9500000000000002e-07 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121680+0000", 2024-08-30T23:59:03.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6387000000000002e-05 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121688+0000", 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5839999999999997e-06 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121694+0000", 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3119999999999999e-06 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121745+0000", 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0837999999999997e-05 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121749+0000", 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6150000000000001e-06 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063145+0000", 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94139648399999998 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.509 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063233+0000", 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199e-05 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063310+0000", 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6536e-05 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065662+0000", 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023518760000000001 2024-08-30T23:59:03.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065685+0000", 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2708999999999999e-05 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065841+0000", 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015618500000000001 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065890+0000", 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.888e-05 2024-08-30T23:59:03.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065924+0000", 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4745e-05 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37745 8.d 8:b0c83413:::benchmark_data_smithi173_143816_object37744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36811859800002, 2024-08-30T23:59:03.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94438678600000003, 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37745 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121722+0000", 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6700000000000003e-07 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121746+0000", 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4036000000000002e-05 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121752+0000", 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2570000000000004e-06 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121756+0000", 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9620000000000004e-06 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121787+0000", 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0587999999999999e-05 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121790+0000", 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3519999999999998e-06 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063974+0000", 2024-08-30T23:59:03.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94218355200000004 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064046+0000", 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1983999999999994e-05 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064119+0000", 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3266e-05 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065799+0000", 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001680186 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065979+0000", 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179552 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066033+0000", 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4085999999999997e-05 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066077+0000", 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4382000000000003e-05 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066108+0000", 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1016000000000003e-05 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37783 8.d 8:bd0ef9a3:::benchmark_data_smithi173_143816_object37782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36807553599999, 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94672526099999998, 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37783 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.520 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121765+0000", 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4799999999999999e-07 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121789+0000", 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4697000000000001e-05 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121795+0000", 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1429999999999997e-06 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121798+0000", 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.472e-06 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121834+0000", 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5528000000000003e-05 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121842+0000", 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3159999999999997e-06 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064918+0000", 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.943076154 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064992+0000", 2024-08-30T23:59:03.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3893999999999994e-05 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065076+0000", 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3681000000000001e-05 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065991+0000", 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091558099999999995 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068378+0000", 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023872390000000002 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068428+0000", 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9329999999999997e-05 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068464+0000", 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5917999999999997e-05 2024-08-30T23:59:03.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068490+0000", 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5959999999999999e-05 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37794 8.d 8:b33c5833:::benchmark_data_smithi173_143816_object37793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36803387800001, 2024-08-30T23:59:03.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94686707599999997, 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37794 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:59:03.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5900000000000003e-07 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121850+0000", 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3831000000000001e-05 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121856+0000", 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2670000000000003e-06 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121861+0000", 2024-08-30T23:59:03.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8799999999999999e-06 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121892+0000", 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1179000000000001e-05 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121896+0000", 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3780000000000001e-06 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066144+0000", 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94424862499999995 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066245+0000", 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010058800000000001 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066373+0000", 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012802199999999999 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068538+0000", 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021646870000000002 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068591+0000", 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3541000000000001e-05 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068604+0000", 2024-08-30T23:59:03.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2607e-05 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068648+0000", 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3781e-05 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068673+0000", 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5330999999999999e-05 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37798 8.d 8:b32c1d3f:::benchmark_data_smithi173_143816_object37797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36797055900001, 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96212161299999999, 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37798 2024-08-30T23:59:03.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121870+0000", 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0800000000000001e-07 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121910+0000", 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0589000000000001e-05 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121916+0000", 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.981e-06 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121920+0000", 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5719999999999999e-06 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121956+0000", 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6510000000000001e-05 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121970+0000", 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3285000000000001e-05 2024-08-30T23:59:03.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067392+0000", 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94542252100000002 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067488+0000", 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5859000000000005e-05 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067570+0000", 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068681+0000", 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011110720000000001 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074538+0000", 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058571530000000004 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083860+0000", 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0093212599999999996 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083936+0000", 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6007999999999995e-05 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083991+0000", 2024-08-30T23:59:03.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5384999999999999e-05 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37806 8.d 8:b567e58a:::benchmark_data_smithi173_143816_object37805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36791136300002, 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96731055799999999, 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.542 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37806 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121929+0000", 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.22e-07 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122024+0000", 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5063999999999996e-05 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122031+0000", 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2060000000000001e-06 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122035+0000", 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7960000000000002e-06 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122067+0000", 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2122999999999997e-05 2024-08-30T23:59:03.545 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122072+0000", 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5140000000000003e-06 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068710+0000", 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94663852100000001 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068855+0000", 2024-08-30T23:59:03.546 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014504599999999999 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068990+0000", 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135177 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074497+0000", 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055070910000000004 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.547 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074576+0000", 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109000000000004e-05 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089138+0000", 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014561274000000001 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.548 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089193+0000", 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4864000000000002e-05 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089239+0000", 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6451e-05 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.549 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:03 smithi136 ceph-mon[20207]: 9.23 scrub starts 2024-08-30T23:59:03.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:03 smithi136 ceph-mon[20207]: pgmap v2180: 67 pgs: 2 unknown, 3 active+recovering+undersized+remapped, 7 active+clean+remapped, 9 active+undersized, 6 active+undersized+degraded, 40 active+clean; 5.4 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 317 B/s rd, 58 MiB/s wr, 920 op/s; 13804/267615 objects degraded (5.158%); 22357/267615 objects misplaced (8.354%); 13 MiB/s, 202 objects/s recovering 2024-08-30T23:59:03.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:03 smithi136 ceph-mon[20207]: osdmap e986: 12 total, 11 up, 11 in 2024-08-30T23:59:03.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:03 smithi136 ceph-mon[20207]: 9.1a scrub starts 2024-08-30T23:59:03.550 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:03 smithi136 ceph-mon[20207]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-08-30T23:59:03.551 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37807 8.d 8:bd227052:::benchmark_data_smithi173_143816_object37806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.551 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:59:03.551 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36779671800002, 2024-08-30T23:59:03.551 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96736628899999999, 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37807 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:59:03.552 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.553 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122079+0000", 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5790000000000001e-05 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122085+0000", 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9170000000000003e-06 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122088+0000", 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.084e-06 2024-08-30T23:59:03.554 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122122+0000", 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3738999999999997e-05 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2359999999999998e-06 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.072882+0000", 2024-08-30T23:59:03.555 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95075315999999999 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073020+0000", 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013803900000000001 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073142+0000", 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122043 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.556 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077477+0000", 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043347180000000004 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077486+0000", 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0759999999999992e-06 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089319+0000", 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011832666 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.557 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089368+0000", 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9446000000000001e-05 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089409+0000", 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1060999999999999e-05 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37812 8.d 8:bad38160:::benchmark_data_smithi173_143816_object37811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.558 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36774608500002, 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.967454445, 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37812 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:59:03.559 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8599999999999999e-07 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.560 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122123+0000", 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8960000000000001e-05 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5520000000000004e-06 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122132+0000", 2024-08-30T23:59:03.561 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5379999999999998e-06 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122161+0000", 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9351000000000001e-05 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122166+0000", 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3440000000000003e-06 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.562 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074606+0000", 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95243996799999997 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074746+0000", 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014037899999999999 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074867+0000", 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120969 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.563 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077539+0000", 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026718330000000002 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078510+0000", 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097061800000000002 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089459+0000", 2024-08-30T23:59:03.564 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010949181000000001 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089509+0000", 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0547999999999998e-05 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089548+0000", 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8918000000000002e-05 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.565 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37816 8.d 8:bceb9fa7:::benchmark_data_smithi173_143816_object37815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36770205300002, 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96754795299999996, 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37816 2024-08-30T23:59:03.566 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-30T23:59:03.567 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6e-07 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122168+0000", 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9663e-05 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122172+0000", 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-06 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.568 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122176+0000", 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2400000000000001e-06 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122205+0000", 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9184999999999999e-05 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122212+0000", 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1849999999999998e-06 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.569 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076169+0000", 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95395610799999997 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076291+0000", 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122633 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076448+0000", 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015706699999999999 2024-08-30T23:59:03.570 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079517+0000", 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003068777 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079533+0000", 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5656000000000001e-05 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089594+0000", 2024-08-30T23:59:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010061151000000001 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089648+0000", 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4110999999999997e-05 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089686+0000", 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7762000000000002e-05 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.572 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37819 8.d 8:b27be7f0:::benchmark_data_smithi173_143816_object37818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36765825600003, 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96865621300000004, 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.573 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37819 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.576 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.72e-07 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122214+0000", 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1964000000000001e-05 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.577 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122218+0000", 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1649999999999999e-06 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122222+0000", 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7179999999999998e-06 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122251+0000", 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8810999999999999e-05 2024-08-30T23:59:03.578 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122254+0000", 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8410000000000002e-06 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077587+0000", 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95533246100000002 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.579 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077667+0000", 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0438000000000005e-05 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077751+0000", 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4217999999999999e-05 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079574+0000", 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018224580000000001 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.580 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080426+0000", 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085253 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090743+0000", 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010316967 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090798+0000", 2024-08-30T23:59:03.581 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4866e-05 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090838+0000", 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9504e-05 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37827 8.d 8:bc586ec9:::benchmark_data_smithi173_143816_object37826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:59:03.582 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36761145499997, 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96986781300000002, 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37827 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.583 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122229+0000", 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8799999999999998e-07 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.584 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122263+0000", 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3946999999999999e-05 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122268+0000", 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3000000000000001e-06 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122271+0000", 2024-08-30T23:59:03.585 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1879999999999998e-06 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122300+0000", 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8824e-05 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122303+0000", 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4249999999999998e-06 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.586 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078551+0000", 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95624797399999995 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078639+0000", 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7868000000000006e-05 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078716+0000", 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6747000000000002e-05 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.587 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081587+0000", 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028711209999999999 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081602+0000", 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4864e-05 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.091996+0000", 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010393823 2024-08-30T23:59:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092053+0000", 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7574000000000001e-05 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092096+0000", 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2870000000000001e-05 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.589 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37830 8.d 8:b56bd01c:::benchmark_data_smithi173_143816_object37829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36756216600003, 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97002787499999998, 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37830 2024-08-30T23:59:03.590 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.591 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8500000000000002e-07 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122306+0000", 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8231000000000001e-05 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122311+0000", 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2179999999999998e-06 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.592 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122314+0000", 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-06 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122348+0000", 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3624999999999997e-05 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122359+0000", 2024-08-30T23:59:03.593 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0685e-05 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079614+0000", 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95725496799999998 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079729+0000", 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114969 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.594 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079807+0000", 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8337000000000006e-05 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083762+0000", 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039546900000000003 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083812+0000", 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0781000000000001e-05 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.595 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092177+0000", 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083641789999999994 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092265+0000", 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8883999999999998e-05 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092306+0000", 2024-08-30T23:59:03.596 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0092999999999997e-05 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37833 8.d 8:bd5be583:::benchmark_data_smithi173_143816_object37832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36751835799998, 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97017067000000001, 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.597 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37833 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.598 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122322+0000", 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122349+0000", 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7472999999999999e-05 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.599 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122356+0000", 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7889999999999997e-06 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122360+0000", 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8330000000000004e-06 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122389+0000", 2024-08-30T23:59:03.600 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.879e-05 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122393+0000", 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1980000000000003e-06 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080473+0000", 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95808044999999997 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.601 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080581+0000", 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010725700000000001 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080712+0000", 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013142199999999999 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084054+0000", 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003342434 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.602 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084069+0000", 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4487e-05 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092412+0000", 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083428129999999993 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092458+0000", 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6440999999999998e-05 2024-08-30T23:59:03.603 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092492+0000", 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4019000000000001e-05 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37856 8.d 8:b1ef2347:::benchmark_data_smithi173_143816_object37855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 275.36747383199997, 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97027297800000001, 2024-08-30T23:59:03.604 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37856 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.605 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122397+0000", 2024-08-30T23:59:03.606 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994000000000002e-05 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122401+0000", 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1439999999999996e-06 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122406+0000", 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2579999999999997e-06 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.607 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122438+0000", 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2008000000000003e-05 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122445+0000", 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081647+0000", 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95920211399999999 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.608 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081737+0000", 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0278999999999994e-05 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081881+0000", 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144036 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084081+0000", 2024-08-30T23:59:03.609 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021993759999999998 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084140+0000", 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9562999999999999e-05 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092544+0000", 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084042449999999994 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.610 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092598+0000", 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3677999999999997e-05 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092639+0000", 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1013999999999997e-05 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45699 8.7 8:eae4016d:::benchmark_data_smithi173_143816_object45698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:59:03.611 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 266.48873114200001, 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64772118199999995, 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45699 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.612 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9100000000000003e-07 2024-08-30T23:59:03.613 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001164+0000", 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4604000000000001e-05 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001173+0000", 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9269999999999994e-06 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.614 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001177+0000", 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4950000000000002e-06 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001209+0000", 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1758999999999999e-05 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001213+0000", 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9600000000000002e-06 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.615 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642148+0000", 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64093464099999997 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642302+0000", 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015428999999999999 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642473+0000", 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000170843 2024-08-30T23:59:03.616 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646115+0000", 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036422680000000002 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648167+0000", 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002051911 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648704+0000", 2024-08-30T23:59:03.617 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053669899999999997 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648748+0000", 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4098999999999999e-05 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648830+0000", 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1995000000000004e-05 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.618 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45725 8.7 8:e95f37b8:::benchmark_data_smithi173_143816_object45724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 266.48865297600003, 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64797183599999997, 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45725 2024-08-30T23:59:03.619 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.620 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001188+0000", 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9999999999999997e-07 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001240+0000", 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120999999999998e-05 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001248+0000", 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6430000000000001e-06 2024-08-30T23:59:03.621 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001253+0000", 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4989999999999996e-06 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001286+0000", 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3328999999999999e-05 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001290+0000", 2024-08-30T23:59:03.622 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0090000000000001e-06 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.643899+0000", 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64260842600000001 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644052+0000", 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015375600000000001 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-30T23:59:03.623 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644192+0000", 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013943899999999999 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646441+0000", 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022493819999999999 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648867+0000", 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024262860000000002 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.624 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649035+0000", 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167052 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649081+0000", 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6409999999999998e-05 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649159+0000", 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7783999999999999e-05 2024-08-30T23:59:03.625 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:19279 9.13 9.d9822fb3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:54.641293+0000", 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 68.848547296000007, 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.434407932, 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-30T23:59:03.626 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 19279 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641293+0000", 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641293+0000", 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.627 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641294+0000", 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2240000000000001e-06 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641316+0000", 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2464e-05 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641330+0000", 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3390000000000001e-05 2024-08-30T23:59:03.628 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641336+0000", 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2639999999999997e-06 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641385+0000", 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8779000000000002e-05 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641391+0000", 2024-08-30T23:59:03.629 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.426e-06 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.075688+0000", 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43429733300000001 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.075700+0000", 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2052000000000001e-05 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.630 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 9.9,from:0'0,to:949'1135,epoch:950/948,start:9:96404f94:::benchmark_data_smithi173_160189_object9101:0,end:9:97883b83:::benchmark_data_smithi173_160189_object15908:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:55.017414+0000", 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 68.472426162000005, 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78106588499999996, 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017414+0000", 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.631 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017414+0000", 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017414+0000", 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-30T23:59:03.632 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017420+0000", 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4809999999999997e-06 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798214+0000", 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78079391099999995 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798234+0000", 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0608999999999999e-05 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.633 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798420+0000", 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000185353 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798437+0000", 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6782999999999998e-05 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798480+0000", 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2942999999999999e-05 2024-08-30T23:59:03.634 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-30T23:59:03.635 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-30T23:59:03.635 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-30T23:59:03.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:03.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:03.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:03.797 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:03.829 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:03.899 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:03.927 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:03.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:03.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 81 16 93165 93149 71.8679 65.75 0.00387271 0.0139039 2024-08-30T23:59:04.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:04.052 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:04.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:04.183 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:04.268 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:04 smithi106 ceph-mon[16603]: 9.1e scrub starts 2024-08-30T23:59:04.268 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:04 smithi106 ceph-mon[16603]: 1.1e scrub starts 2024-08-30T23:59:04.285 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:04.313 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:04.382 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:04 smithi173 ceph-mon[20554]: 9.1e scrub starts 2024-08-30T23:59:04.382 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:04 smithi173 ceph-mon[20554]: 1.1e scrub starts 2024-08-30T23:59:04.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:04.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-30T23:59:04.432 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:04.432 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:04.441 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:04.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:04.469 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:04.470 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:04.503 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:04.503 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:04.535 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:04.536 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:04 smithi136 ceph-mon[20207]: 9.1e scrub starts 2024-08-30T23:59:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:04 smithi136 ceph-mon[20207]: 1.1e scrub starts 2024-08-30T23:59:04.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:04.566 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:04.566 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:04.569 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:04.601 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:04.602 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:04.635 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:04.635 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:04.636 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:04.671 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:04.672 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:04.672 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:04.709 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:04.709 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:04.711 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:04.753 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:04.753 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:04.788 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:04.788 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:04.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:04.841 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:04.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:59:04.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:04.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:04.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:04.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 82 16 94230 94214 71.8032 66.5625 0.0122544 0.0139224 2024-08-30T23:59:05.072 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:05.111 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:05.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:05.245 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:05.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:05.348 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: pgmap v2182: 67 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 2 unknown, 4 active+recovering+undersized+remapped, 8 active+clean+remapped, 8 active+undersized, 6 active+undersized+degraded, 37 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 341 B/s rd, 51 MiB/s wr, 816 op/s; 16989/271098 objects degraded (6.267%); 26561/271098 objects misplaced (9.798%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:59:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:05.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: 9.1d scrub starts 2024-08-30T23:59:05.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: osdmap e987: 12 total, 11 up, 11 in 2024-08-30T23:59:05.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/8892653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:05.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:05 smithi106 ceph-mon[16603]: 9.1f scrub starts 2024-08-30T23:59:05.377 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: pgmap v2182: 67 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 2 unknown, 4 active+recovering+undersized+remapped, 8 active+clean+remapped, 8 active+undersized, 6 active+undersized+degraded, 37 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 341 B/s rd, 51 MiB/s wr, 816 op/s; 16989/271098 objects degraded (6.267%); 26561/271098 objects misplaced (9.798%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: 9.1d scrub starts 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: osdmap e987: 12 total, 11 up, 11 in 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/8892653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:05.474 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:05 smithi136 ceph-mon[20207]: 9.1f scrub starts 2024-08-30T23:59:05.479 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:05.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:05.514 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:59:05.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: pgmap v2182: 67 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 2 unknown, 4 active+recovering+undersized+remapped, 8 active+clean+remapped, 8 active+undersized, 6 active+undersized+degraded, 37 active+clean; 5.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 341 B/s rd, 51 MiB/s wr, 816 op/s; 16989/271098 objects degraded (6.267%); 26561/271098 objects misplaced (9.798%); 15 MiB/s, 236 objects/s recovering 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: 9.1d scrub starts 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: osdmap e987: 12 total, 11 up, 11 in 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/8892653' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:05 smithi173 ceph-mon[20554]: 9.1f scrub starts 2024-08-30T23:59:05.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:05.643 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:05.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:05.775 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.776 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:05.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:05.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:05.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:05.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 83 16 95331 95315 71.7671 68.8125 0.019853 0.0139289 2024-08-30T23:59:06.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:06.038 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:06.093 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:59:06.128 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:59:06.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:06.168 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:06.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:06 smithi106 ceph-mon[16603]: 1.1d scrub starts 2024-08-30T23:59:06.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:06.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:06.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:06 smithi106 ceph-mon[16603]: osdmap e988: 12 total, 11 up, 11 in 2024-08-30T23:59:06.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 16989/271098 objects degraded (6.267%), 7 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:06.229 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:06 smithi106 ceph-mon[16603]: osdmap e989: 12 total, 11 up, 11 in 2024-08-30T23:59:06.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:06.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:06.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:06.424 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:06.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:06 smithi136 ceph-mon[20207]: 1.1d scrub starts 2024-08-30T23:59:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:06 smithi136 ceph-mon[20207]: osdmap e988: 12 total, 11 up, 11 in 2024-08-30T23:59:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 16989/271098 objects degraded (6.267%), 7 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:06 smithi136 ceph-mon[20207]: osdmap e989: 12 total, 11 up, 11 in 2024-08-30T23:59:06.554 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:06 smithi173 ceph-mon[20554]: 1.1d scrub starts 2024-08-30T23:59:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:06 smithi173 ceph-mon[20554]: osdmap e988: 12 total, 11 up, 11 in 2024-08-30T23:59:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 16989/271098 objects degraded (6.267%), 7 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:06 smithi173 ceph-mon[20554]: osdmap e989: 12 total, 11 up, 11 in 2024-08-30T23:59:06.656 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:06.681 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:06.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:06.814 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:06.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:06.938 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:06.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:06.986 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 84 16 96286 96270 71.6232 59.6875 0.0513413 0.0139385 2024-08-30T23:59:07.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:07.061 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:07.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:07.195 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:07.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-30T23:59:07.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:07.327 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.328 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:07.423 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:07 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:07.423 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:07 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:07.423 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:07 smithi106 ceph-mon[16603]: pgmap v2185: 67 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 5.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 52 MiB/s wr, 835 op/s; 17431/279207 objects degraded (6.243%); 36195/279207 objects misplaced (12.964%); 15 MiB/s, 241 objects/s recovering 2024-08-30T23:59:07.424 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:07 smithi106 ceph-mon[16603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-08-30T23:59:07.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:07.456 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:07 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:07 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:07 smithi136 ceph-mon[20207]: pgmap v2185: 67 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 5.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 52 MiB/s wr, 835 op/s; 17431/279207 objects degraded (6.243%); 36195/279207 objects misplaced (12.964%); 15 MiB/s, 241 objects/s recovering 2024-08-30T23:59:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:07 smithi136 ceph-mon[20207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-08-30T23:59:07.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:59:07.558 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:07 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:07 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:07 smithi173 ceph-mon[20554]: pgmap v2185: 67 pgs: 1 activating+undersized+degraded+remapped, 1 remapped+peering, 5 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 5.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 52 MiB/s wr, 835 op/s; 17431/279207 objects degraded (6.243%); 36195/279207 objects misplaced (12.964%); 15 MiB/s, 241 objects/s recovering 2024-08-30T23:59:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:07 smithi173 ceph-mon[20554]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-08-30T23:59:07.630 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:07.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:07.732 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:07.766 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:07.868 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:07.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:07.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:07.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 85 16 97272 97256 71.5055 61.625 0.0148163 0.0139764 2024-08-30T23:59:07.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:08.025 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:08.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:08.153 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:59:08.162 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:08.221 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:59:08.221 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:59:08.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:08.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:08 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:08 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:08.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:08 smithi106 ceph-mon[16603]: osdmap e990: 12 total, 11 up, 11 in 2024-08-30T23:59:08.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:08 smithi106 ceph-mon[16603]: 9.1a scrub starts 2024-08-30T23:59:08.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:08 smithi106 ceph-mon[16603]: 9.23 scrub starts 2024-08-30T23:59:08.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/244346432' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:59:08.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:08.425 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:08.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:08 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:08 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:08 smithi136 ceph-mon[20207]: osdmap e990: 12 total, 11 up, 11 in 2024-08-30T23:59:08.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:08 smithi136 ceph-mon[20207]: 9.1a scrub starts 2024-08-30T23:59:08.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:08 smithi136 ceph-mon[20207]: 9.23 scrub starts 2024-08-30T23:59:08.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/244346432' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:59:08.556 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.557 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:08 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:08 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:08 smithi173 ceph-mon[20554]: osdmap e990: 12 total, 11 up, 11 in 2024-08-30T23:59:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:08 smithi173 ceph-mon[20554]: 9.1a scrub starts 2024-08-30T23:59:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:08 smithi173 ceph-mon[20554]: 9.23 scrub starts 2024-08-30T23:59:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/244346432' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:59:08.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:08.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] out_osds: [11] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:59:08.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:59:08.687 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-08-30T23:59:08.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd in 11 2024-08-30T23:59:08.693 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:08.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:08.820 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:08.922 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:08.965 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:08.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:08.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 86 16 98283 98267 71.4087 63.1875 0.0062067 0.0139979 2024-08-30T23:59:09.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:09.093 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:09.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:09.222 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:09.324 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:09.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:09.455 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.490 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:09.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: pgmap v2188: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 14827/284142 objects degraded (5.218%); 41162/284142 objects misplaced (14.486%); 35 MiB/s, 553 objects/s recovering 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: 1.1e scrub starts 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/244346432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: osdmap e991: 12 total, 11 up, 11 in 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:09 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: pgmap v2188: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 14827/284142 objects degraded (5.218%); 41162/284142 objects misplaced (14.486%); 35 MiB/s, 553 objects/s recovering 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: 1.1e scrub starts 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/244346432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: osdmap e991: 12 total, 11 up, 11 in 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:09 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:09.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: pgmap v2188: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 5.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 14827/284142 objects degraded (5.218%); 41162/284142 objects misplaced (14.486%); 35 MiB/s, 553 objects/s recovering 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: 1.1e scrub starts 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/244346432' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: osdmap e991: 12 total, 11 up, 11 in 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:09.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:09 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:09.613 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:09.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:09.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:09.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:09.873 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:09.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:09.975 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:09.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 87 16 99238 99222 71.2739 59.6875 0.00692474 0.0140244 2024-08-30T23:59:10.013 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:10.115 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:10.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:10.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-30T23:59:10.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:10.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:10.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:10.408 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:10.510 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:10.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: osdmap e992: 12 total, 11 up, 11 in 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:59:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:10 smithi136 ceph-mon[20207]: 9.1f scrub starts 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: osdmap e992: 12 total, 11 up, 11 in 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:59:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:10 smithi173 ceph-mon[20554]: 9.1f scrub starts 2024-08-30T23:59:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:59:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:59:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-30T23:59:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: osdmap e992: 12 total, 11 up, 11 in 2024-08-30T23:59:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-30T23:59:10.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-30T23:59:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-30T23:59:10.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:10 smithi106 ceph-mon[16603]: 9.1f scrub starts 2024-08-30T23:59:10.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:10.666 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:10.768 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:10.793 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:10.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:10.928 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:10.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:10.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 88 16 100185 100169 71.1366 59.1875 0.00775548 0.0140523 2024-08-30T23:59:11.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:11.058 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:11.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:11.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:11.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:11.320 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:11.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:11.423 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:11 smithi106 ceph-mon[16603]: pgmap v2191: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 15265/292887 objects degraded (5.212%); 40051/292887 objects misplaced (13.675%); 40 MiB/s, 635 objects/s recovering 2024-08-30T23:59:11.424 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:11 smithi106 ceph-mon[16603]: 9.1e scrub starts 2024-08-30T23:59:11.424 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:11 smithi106 ceph-mon[16603]: 9.1f scrub ok 2024-08-30T23:59:11.424 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 15265/292887 objects degraded (5.212%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:11.424 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:11 smithi106 ceph-mon[16603]: osdmap e993: 12 total, 11 up, 11 in 2024-08-30T23:59:11.438 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:11.444 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.445 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:11.467 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:11 smithi136 ceph-mon[20207]: pgmap v2191: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 15265/292887 objects degraded (5.212%); 40051/292887 objects misplaced (13.675%); 40 MiB/s, 635 objects/s recovering 2024-08-30T23:59:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:11 smithi136 ceph-mon[20207]: 9.1e scrub starts 2024-08-30T23:59:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:11 smithi136 ceph-mon[20207]: 9.1f scrub ok 2024-08-30T23:59:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 15265/292887 objects degraded (5.212%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:11 smithi136 ceph-mon[20207]: osdmap e993: 12 total, 11 up, 11 in 2024-08-30T23:59:11.546 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:11 smithi173 ceph-mon[20554]: pgmap v2191: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s; 15265/292887 objects degraded (5.212%); 40051/292887 objects misplaced (13.675%); 40 MiB/s, 635 objects/s recovering 2024-08-30T23:59:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:11 smithi173 ceph-mon[20554]: 9.1e scrub starts 2024-08-30T23:59:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:11 smithi173 ceph-mon[20554]: 9.1f scrub ok 2024-08-30T23:59:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 15265/292887 objects degraded (5.212%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:11 smithi173 ceph-mon[20554]: osdmap e993: 12 total, 11 up, 11 in 2024-08-30T23:59:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:11.676 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:11.699 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:11.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:11.829 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:11.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:11.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:11.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:11.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 89 16 100964 100948 70.8843 48.6875 0.0318607 0.0141026 2024-08-30T23:59:12.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:12.092 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:12.194 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.225 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:12.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:12.304 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:12 smithi106 ceph-mon[16603]: 9.1e scrub ok 2024-08-30T23:59:12.305 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:59:12.305 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:59:12.305 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:12 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:59:12.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:12.364 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:12.407 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:12.408 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:12.452 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:12.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:12.466 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:12.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.494 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:12.497 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:12.497 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:12.499 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:12.536 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:12.536 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:12.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:12 smithi136 ceph-mon[20207]: 9.1e scrub ok 2024-08-30T23:59:12.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:59:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:59:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:12 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:59:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:12 smithi173 ceph-mon[20554]: 9.1e scrub ok 2024-08-30T23:59:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-30T23:59:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-30T23:59:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:12 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-30T23:59:12.576 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:12.576 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:12.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:12.616 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:12.616 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:12.629 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:12.648 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:12.648 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:12.648 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:12.682 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:12.682 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:12.715 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:12.716 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:12.731 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:12.750 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:12.750 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:12.765 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:12.789 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:12.789 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:12.827 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:12 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-30T23:59:12.601+0000 7f04ca6f6640 -1 mon.a@0(leader).osd e993 definitely_dead 0 2024-08-30T23:59:12.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:12.895 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:12.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:12.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:12.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 90 6 102001 101995 70.8237 65.4375 0.0091918 0.0141141 2024-08-30T23:59:12.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:13.019 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:13.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:13.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:13.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:13.168 INFO:teuthology.orchestra.run.smithi106.stderr:marked in osd.11. 2024-08-30T23:59:13.229 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:59:13.229 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-30T23:59:13.229 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-30T23:59:13.229 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(7.b 763/756 [PushReplyOp(7:d4034f8a:::benchmark_data_smithi173_129296_object8907:head)])", 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 524.83305769599997, 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61886441400000003, 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.230 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387982+0000", 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9800000000000003e-07 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:28.387988+0000", 2024-08-30T23:59:13.231 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9800000000000003e-06 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001726+0000", 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.61373751200000004 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001738+0000", 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1746000000000001e-05 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.232 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.001796+0000", 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8470000000000001e-05 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:50:29.006846+0000", 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0050499079999999997 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.233 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e833,800 7:e3e0581a:::benchmark_data_smithi173_129296_object20938:head,781'6741)", 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 334.31373178000001, 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56565642400000005, 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.234 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.92e-07 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.235 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907313+0000", 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5199999999999999e-06 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472540+0000", 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56522729500000002 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472548+0000", 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9780000000000005e-06 2024-08-30T23:59:13.236 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472616+0000", 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7745999999999998e-05 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472964+0000", 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034819300000000001 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.237 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,800 7:e6a7a338:::benchmark_data_smithi173_129296_object26093:head,783'7057)", 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:41.361662+0000", 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 331.85937716699999, 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29360009100000001, 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361662+0000", 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.238 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361662+0000", 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361663+0000", 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4499999999999997e-07 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361666+0000", 2024-08-30T23:59:13.239 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.579e-06 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654885+0000", 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29321898600000001 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654896+0000", 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0654e-05 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.240 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654958+0000", 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2126999999999997e-05 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.655262+0000", 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030420000000000002 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.241 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,800 7:e80dc891:::benchmark_data_smithi173_129296_object31110:head,789'7374)", 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 330.14593878800002, 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70067806200000005, 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.242 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9999999999999997e-07 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075106+0000", 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6249999999999998e-06 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.243 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775466+0000", 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70036012700000005 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775475+0000", 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6789999999999994e-06 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775526+0000", 2024-08-30T23:59:13.244 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1054000000000003e-05 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775779+0000", 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025287699999999998 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:27838 8.7 8:ee12ce7e:::benchmark_data_smithi173_143816_object27837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e857)", 2024-08-30T23:59:13.245 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 296.61196575399998, 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28304243600000001, 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 27838 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.246 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609074+0000", 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8999999999999995e-07 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.247 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609179+0000", 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000104453 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609187+0000", 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1680000000000004e-06 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609193+0000", 2024-08-30T23:59:13.248 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.5500000000000002e-06 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609238+0000", 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5324999999999997e-05 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609365+0000", 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012742100000000001 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.609466+0000", 2024-08-30T23:59:13.249 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000100251 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.611050+0000", 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015842580000000001 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.648761+0000", 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.037710942999999997 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.250 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.891285+0000", 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.24252380600000001 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.891319+0000", 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.4653999999999997e-05 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:16.892116+0000", 2024-08-30T23:59:13.251 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00079671699999999995 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36528 8.7 8:ed4e2712:::benchmark_data_smithi173_143816_object36527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.367630+0000", 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 287.85340951000001, 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46017543799999999, 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.252 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36528 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367630+0000", 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.253 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367630+0000", 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367631+0000", 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.548e-06 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.254 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367685+0000", 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4001e-05 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367695+0000", 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6150000000000003e-06 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367700+0000", 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5650000000000003e-06 2024-08-30T23:59:13.255 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368384+0000", 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068396600000000004 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368520+0000", 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013615799999999999 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-30T23:59:13.256 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368643+0000", 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012345699999999999 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.370618+0000", 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001974632 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.373944+0000", 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033259520000000001 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.257 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.826917+0000", 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.452973338 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.826991+0000", 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3951999999999999e-05 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.827805+0000", 2024-08-30T23:59:13.258 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00081425400000000004 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36543 8.7 8:ea468d17:::benchmark_data_smithi173_143816_object36542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.549506+0000", 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 287.671533044, 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.28538659799999999, 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.259 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36543 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549506+0000", 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.260 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549506+0000", 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549508+0000", 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9429999999999999e-06 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549547+0000", 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8903999999999999e-05 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.261 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549559+0000", 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2114e-05 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549567+0000", 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5179999999999998e-06 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.549967+0000", 2024-08-30T23:59:13.262 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00040007299999999998 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.550137+0000", 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00016991200000000001 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.550336+0000", 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00019918599999999999 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.263 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.554084+0000", 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0037478749999999999 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.558679+0000", 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0045950710000000001 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.834074+0000", 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.27539524700000001 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.264 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.834122+0000", 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.7425000000000001e-05 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.834893+0000", 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00077132999999999998 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.265 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37734 8.1 8:87d95214:::benchmark_data_smithi173_143816_object37733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 286.06538538299998, 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33538572700000002, 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37734 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.266 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2300000000000001e-07 2024-08-30T23:59:13.267 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155716+0000", 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1576999999999997e-05 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155723+0000", 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8770000000000004e-06 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.268 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155733+0000", 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9280000000000005e-06 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.156800+0000", 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010672489999999999 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.156972+0000", 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000171516 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.269 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.157064+0000", 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2851999999999998e-05 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.157928+0000", 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086371099999999997 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.164297+0000", 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063691900000000003 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.270 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490984+0000", 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32668678800000001 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491017+0000", 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3030000000000001e-05 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491040+0000", 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2486000000000001e-05 2024-08-30T23:59:13.271 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37734 8.1 e864/840 ondisk, result = 0)", 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 286.06298356399998, 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33298887799999999, 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.272 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.273 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:59:13.273 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.273 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.273 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.273 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:13.274 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:59:13.274 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8500000000000002e-07 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158060+0000", 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3620000000000002e-06 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.275 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490938+0000", 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33287868399999998 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490943+0000", 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.481e-06 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490976+0000", 2024-08-30T23:59:13.276 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3006999999999998e-05 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490981+0000", 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0710000000000002e-06 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491045+0000", 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3788000000000007e-05 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.277 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37770 8.1 8:8a7dfc05:::benchmark_data_smithi173_143816_object37769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.228037+0000", 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 285.99300197700001, 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.393374581, 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37770 2024-08-30T23:59:13.278 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228037+0000", 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228037+0000", 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.279 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228039+0000", 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2550000000000001e-06 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228154+0000", 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115809 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228166+0000", 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1505000000000001e-05 2024-08-30T23:59:13.280 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228171+0000", 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0499999999999999e-06 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228250+0000", 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9462000000000006e-05 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228340+0000", 2024-08-30T23:59:13.281 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9864000000000006e-05 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228442+0000", 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101697 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229583+0000", 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011410179999999999 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.282 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.264158+0000", 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034575084999999998 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621315+0000", 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35715717000000002 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621360+0000", 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4866e-05 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.283 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621412+0000", 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1799999999999999e-05 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37770 8.1 e864/840 ondisk, result = 0)", 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.229920+0000", 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 285.99111936499997, 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39149838599999998, 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.284 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229920+0000", 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229920+0000", 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.285 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229921+0000", 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9800000000000003e-07 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229924+0000", 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3330000000000001e-06 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621242+0000", 2024-08-30T23:59:13.286 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39131752400000003 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621251+0000", 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4860000000000003e-06 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621303+0000", 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2337999999999999e-05 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.287 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621311+0000", 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4320000000000002e-06 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621418+0000", 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107575 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(8.f 864/854 [PushReplyOp(8:ff509c29:::benchmark_data_smithi173_143816_object7126:head)])", 2024-08-30T23:59:13.288 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:28.072822+0000", 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 285.14821735700002, 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46479312699999997, 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072822+0000", 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.289 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072822+0000", 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072823+0000", 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072831+0000", 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1890000000000001e-06 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.290 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536520+0000", 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46368978100000002 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536534+0000", 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3192e-05 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536618+0000", 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4377999999999997e-05 2024-08-30T23:59:13.291 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.537615+0000", 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00099712900000000003 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:45729 8.1 8:842ca331:::benchmark_data_smithi173_143816_object45728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:36.750932+0000", 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.470107326, 2024-08-30T23:59:13.292 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.51209708499999995, 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 45729 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750932+0000", 2024-08-30T23:59:13.293 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750932+0000", 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750934+0000", 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6789999999999999e-06 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.294 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750972+0000", 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8312000000000002e-05 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750988+0000", 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6388e-05 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750996+0000", 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4989999999999997e-06 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.295 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751056+0000", 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9786e-05 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751197+0000", 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140879 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751340+0000", 2024-08-30T23:59:13.296 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014324499999999999 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752873+0000", 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00153348 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.753371+0000", 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049803800000000002 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.297 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262925+0000", 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50955419099999999 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262977+0000", 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1261999999999998e-05 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.263029+0000", 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2326e-05 2024-08-30T23:59:13.298 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:45729 8.1 e869/840 ondisk, result = 0)", 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:36.752989+0000", 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 276.46805046899999, 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.51004729900000001, 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.299 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752989+0000", 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752989+0000", 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752990+0000", 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0100000000000004e-07 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.300 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752993+0000", 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.619e-06 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262804+0000", 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50981123900000003 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.301 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262811+0000", 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6780000000000003e-06 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262906+0000", 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4585000000000003e-05 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262918+0000", 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2184000000000001e-05 2024-08-30T23:59:13.302 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.263036+0000", 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118293 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:19261 9.4 9:27013f3f:::benchmark_data_smithi173_160189_object19260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.083444+0000", 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 79.137594999000001, 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46293038199999997, 2024-08-30T23:59:13.303 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 19261 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083444+0000", 2024-08-30T23:59:13.304 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083444+0000", 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083446+0000", 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.215e-06 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.305 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083483+0000", 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7916000000000001e-05 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083495+0000", 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.199e-05 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083501+0000", 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1229999999999999e-06 2024-08-30T23:59:13.306 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083540+0000", 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9263e-05 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083624+0000", 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3987999999999996e-05 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-30T23:59:13.307 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083719+0000", 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4710000000000006e-05 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084202+0000", 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00048355400000000003 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.138227+0000", 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054025380999999997 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.308 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546203+0000", 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.407975486 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546303+0000", 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010033400000000001 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546375+0000", 2024-08-30T23:59:13.309 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1421999999999995e-05 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.28405.0:19261 9.4 e950/948 ondisk, result = 0)", 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.084652+0000", 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 79.136386845000004, 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46173567799999998, 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:59:13.310 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084652+0000", 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084652+0000", 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.311 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084653+0000", 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1500000000000005e-07 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084656+0000", 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.118e-06 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546041+0000", 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46138474899999998 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.312 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546057+0000", 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.662e-05 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546176+0000", 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011835 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546193+0000", 2024-08-30T23:59:13.313 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6986999999999998e-05 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546388+0000", 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00019533899999999999 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.28405.0:19270 9.f e950/938 9:f74eb61c:::benchmark_data_smithi173_160189_object19269:head v 950'1246, mlcod=949'1244)", 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.328695+0000", 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.892344589000004, 2024-08-30T23:59:13.314 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31213600899999999, 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328695+0000", 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328695+0000", 2024-08-30T23:59:13.315 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328696+0000", 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.536e-06 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328785+0000", 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8706000000000001e-05 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.316 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639878+0000", 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31109302799999999 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639884+0000", 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4629999999999998e-06 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639941+0000", 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6628000000000001e-05 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.317 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639961+0000", 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.944e-05 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.640802+0000", 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00084156699999999999 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.640831+0000", 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8640999999999999e-05 2024-08-30T23:59:13.318 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "replica_scrub(pg: 9.6,from:0'0,to:949'1111,epoch:950/948,start:9:6299959b:::benchmark_data_smithi173_160189_object13888:0,end:9:63fa1766:::benchmark_data_smithi173_160189_object12833:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.639844+0000", 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 78.581195730000005, 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.306055256, 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.319 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639844+0000", 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639844+0000", 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639845+0000", 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.540000000000001e-07 2024-08-30T23:59:13.320 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639852+0000", 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0559999999999999e-06 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945749+0000", 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.30589699599999998 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.321 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945758+0000", 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0739999999999999e-06 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945829+0000", 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1110000000000002e-05 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945845+0000", 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6690000000000001e-05 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.322 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945899+0000", 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3375999999999999e-05 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.28405.0:57585 9.22 e970/969 9:447b4f71:::benchmark_data_smithi173_160189_object57584:head v 970'2231, mlcod=953'1283)", 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:28.894879+0000", 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 44.326160096999999, 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38515502400000001, 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.323 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894879+0000", 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894879+0000", 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.324 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894880+0000", 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5099999999999998e-07 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894961+0000", 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1088999999999996e-05 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.276804+0000", 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38184307299999998 2024-08-30T23:59:13.325 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.276811+0000", 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6200000000000001e-06 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.277218+0000", 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000407484 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:13.326 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.277243+0000", 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4284999999999999e-05 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.279951+0000", 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027079740000000001 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.280034+0000", 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3647999999999996e-05 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.327 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(9.20 973/969 [PushReplyOp(9:05c2bc44:::benchmark_data_smithi173_160189_object27957:head)])", 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:34.746380+0000", 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 38.474659660999997, 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31719325700000001, 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.328 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746380+0000", 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746380+0000", 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746380+0000", 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2099999999999998e-07 2024-08-30T23:59:13.329 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746383+0000", 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8459999999999999e-06 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.062098+0000", 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31571540199999998 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.062107+0000", 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1300000000000007e-06 2024-08-30T23:59:13.330 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.062815+0000", 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070730900000000004 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.063573+0000", 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00075824900000000001 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:13.331 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:59:13.353 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:13.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:59:13.450 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:13 smithi106 ceph-mon[16603]: pgmap v2193: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 61 MiB/s wr, 972 op/s; 15265/294951 objects degraded (5.175%); 40375/294951 objects misplaced (13.689%); 30 MiB/s, 486 objects/s recovering 2024-08-30T23:59:13.450 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:13 smithi106 ceph-mon[16603]: 9.1d scrub starts 2024-08-30T23:59:13.450 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/761068463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:13.450 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:13 smithi106 ceph-mon[16603]: 9.23 scrub starts 2024-08-30T23:59:13.450 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3928550217' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-08-30T23:59:13.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:13.478 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:13 smithi136 ceph-mon[20207]: pgmap v2193: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 61 MiB/s wr, 972 op/s; 15265/294951 objects degraded (5.175%); 40375/294951 objects misplaced (13.689%); 30 MiB/s, 486 objects/s recovering 2024-08-30T23:59:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:13 smithi136 ceph-mon[20207]: 9.1d scrub starts 2024-08-30T23:59:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/761068463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:13 smithi136 ceph-mon[20207]: 9.23 scrub starts 2024-08-30T23:59:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3928550217' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-08-30T23:59:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:13 smithi173 ceph-mon[20554]: pgmap v2193: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 61 MiB/s wr, 972 op/s; 15265/294951 objects degraded (5.175%); 40375/294951 objects misplaced (13.689%); 30 MiB/s, 486 objects/s recovering 2024-08-30T23:59:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:13 smithi173 ceph-mon[20554]: 9.1d scrub starts 2024-08-30T23:59:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/761068463' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:13 smithi173 ceph-mon[20554]: 9.23 scrub starts 2024-08-30T23:59:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3928550217' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-08-30T23:59:13.580 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:13.600 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:13.702 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:13.728 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:13.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:13.856 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:13.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:13.980 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:13.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:13.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout: 91 1 102001 102000 70.0488 0.3125 0.813383 0.0141271 2024-08-30T23:59:14.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:14.104 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:14.206 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:14.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.237 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:14.338 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:14.360 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:14.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:14.486 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:14.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:14.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: 9.23 scrub ok 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: 9.1d scrub ok 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: 1.1e scrub starts 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: 1.1e scrub ok 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3928550217' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: osdmap e994: 12 total, 11 up, 12 in 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: 9.1a scrub starts 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: 9.1a scrub ok 2024-08-30T23:59:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:14 smithi136 ceph-mon[20207]: osdmap e995: 12 total, 11 up, 12 in 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: 9.23 scrub ok 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: 9.1d scrub ok 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: 1.1e scrub starts 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: 1.1e scrub ok 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3928550217' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: osdmap e994: 12 total, 11 up, 12 in 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: 9.1a scrub starts 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: 9.1a scrub ok 2024-08-30T23:59:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:14 smithi173 ceph-mon[20554]: osdmap e995: 12 total, 11 up, 12 in 2024-08-30T23:59:14.588 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: 9.23 scrub ok 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: 9.1d scrub ok 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: 1.1e scrub starts 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: 1.1e scrub ok 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3928550217' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: osdmap e994: 12 total, 11 up, 12 in 2024-08-30T23:59:14.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: 9.1a scrub starts 2024-08-30T23:59:14.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: 9.1a scrub ok 2024-08-30T23:59:14.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:14 smithi106 ceph-mon[16603]: osdmap e995: 12 total, 11 up, 12 in 2024-08-30T23:59:14.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.611 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:14.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-30T23:59:14.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:14.732 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:14.834 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:14.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:14.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:14.911 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:59:14.925 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-08-30T23:59:14.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:14.978 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:14.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:14.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total time run: 91.2516 2024-08-30T23:59:14.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Total writes made: 102001 2024-08-30T23:59:14.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Write size: 65536 2024-08-30T23:59:14.987 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Object size: 65536 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Bandwidth (MB/sec): 69.8624 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Bandwidth: 32.8247 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max bandwidth (MB/sec): 153.75 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min bandwidth (MB/sec): 0 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average IOPS: 1117 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev IOPS: 525.203 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max IOPS: 2460 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min IOPS: 0 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Average Latency(s): 0.0141393 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Stddev Latency(s): 0.0548926 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Max latency(s): 2.30386 2024-08-30T23:59:14.988 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Min latency(s): 0.000982985 2024-08-30T23:59:14.996 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Cleaning up (deleting benchmark objects) 2024-08-30T23:59:15.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:15.110 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:15.212 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:15.243 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:15 smithi136 ceph-mon[20207]: pgmap v2195: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 19 MiB/s wr, 308 op/s; 15570/298809 objects degraded (5.211%); 39072/298809 objects misplaced (13.076%); 12 MiB/s, 196 objects/s recovering 2024-08-30T23:59:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:15 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-30T23:59:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:15 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-30T23:59:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:15 smithi136 ceph-mon[20207]: osdmap e996: 12 total, 11 up, 12 in 2024-08-30T23:59:15.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:15.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:59:15.434 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:59:15.450 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:15.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:15.496 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:15 smithi173 ceph-mon[20554]: pgmap v2195: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 19 MiB/s wr, 308 op/s; 15570/298809 objects degraded (5.211%); 39072/298809 objects misplaced (13.076%); 12 MiB/s, 196 objects/s recovering 2024-08-30T23:59:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:15 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-30T23:59:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:15 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-30T23:59:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:15 smithi173 ceph-mon[20554]: osdmap e996: 12 total, 11 up, 12 in 2024-08-30T23:59:15.575 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:15 smithi106 ceph-mon[16603]: pgmap v2195: 67 pgs: 7 active+recovering+undersized+remapped, 11 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 19 MiB/s wr, 308 op/s; 15570/298809 objects degraded (5.211%); 39072/298809 objects misplaced (13.076%); 12 MiB/s, 196 objects/s recovering 2024-08-30T23:59:15.575 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:15 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-30T23:59:15.575 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:15 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-30T23:59:15.575 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:15 smithi106 ceph-mon[16603]: osdmap e996: 12 total, 11 up, 12 in 2024-08-30T23:59:15.598 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:15.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:15.722 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:15.747 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:15.848 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:15.870 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:15.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:15.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:15.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:16.098 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:16.121 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:16.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:16.223 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:16.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:16.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.377 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:16.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:16.443 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:16 smithi136 ceph-mon[20207]: 1.1b scrub starts 2024-08-30T23:59:16.443 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:16 smithi136 ceph-mon[20207]: 1.1b scrub ok 2024-08-30T23:59:16.444 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:16 smithi136 ceph-mon[20207]: 1.1d scrub ok 2024-08-30T23:59:16.444 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 15570/298809 objects degraded (5.211%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:16.444 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:16 smithi136 ceph-mon[20207]: osdmap e997: 12 total, 11 up, 12 in 2024-08-30T23:59:16.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:16.502 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:16.540 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:59:16.540 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:59:16.540 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-30T23:59:16.540 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:59:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:16 smithi173 ceph-mon[20554]: 1.1b scrub starts 2024-08-30T23:59:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:16 smithi173 ceph-mon[20554]: 1.1b scrub ok 2024-08-30T23:59:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:16 smithi173 ceph-mon[20554]: 1.1d scrub ok 2024-08-30T23:59:16.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 15570/298809 objects degraded (5.211%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:16.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:16 smithi173 ceph-mon[20554]: osdmap e997: 12 total, 11 up, 12 in 2024-08-30T23:59:16.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:16 smithi106 ceph-mon[16603]: 1.1b scrub starts 2024-08-30T23:59:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:16 smithi106 ceph-mon[16603]: 1.1b scrub ok 2024-08-30T23:59:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:16 smithi106 ceph-mon[16603]: 1.1d scrub ok 2024-08-30T23:59:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 15570/298809 objects degraded (5.211%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:16.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:16 smithi106 ceph-mon[16603]: osdmap e997: 12 total, 11 up, 12 in 2024-08-30T23:59:16.626 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:16.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:16.752 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:16.854 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:16.876 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:16.877 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:16.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:17.000 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:17.083 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:59:17.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:17.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:17.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:17.258 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:17.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:17.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-30T23:59:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: pgmap v2198: 67 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 5 active+recovering+undersized+remapped, 10 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.2 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 16148/304761 objects degraded (5.299%); 30337/304761 objects misplaced (9.954%) 2024-08-30T23:59:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-30T23:59:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-30T23:59:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: 1.1c scrub starts 2024-08-30T23:59:17.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: 1.1c scrub ok 2024-08-30T23:59:17.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1866027935' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:17.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:59:17.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1866027935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:17.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:17 smithi106 ceph-mon[16603]: osdmap e998: 12 total, 11 up, 12 in 2024-08-30T23:59:17.360 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:17.383 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:17.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:59:17.485 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:17.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: pgmap v2198: 67 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 5 active+recovering+undersized+remapped, 10 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.2 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 16148/304761 objects degraded (5.299%); 30337/304761 objects misplaced (9.954%) 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: 1.1c scrub starts 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: 1.1c scrub ok 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1866027935' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:59:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1866027935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:17.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:17 smithi136 ceph-mon[20207]: osdmap e998: 12 total, 11 up, 12 in 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: pgmap v2198: 67 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 5 active+recovering+undersized+remapped, 10 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 8 active+undersized, 5 active+undersized+degraded, 35 active+clean; 6.2 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 16148/304761 objects degraded (5.299%); 30337/304761 objects misplaced (9.954%) 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: 1.1c scrub starts 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: 1.1c scrub ok 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1866027935' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1866027935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:17 smithi173 ceph-mon[20554]: osdmap e998: 12 total, 11 up, 12 in 2024-08-30T23:59:17.600 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:59:17.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:17.631 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:17.733 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:17.757 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:17.860 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:17.881 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:17.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:17.983 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.006 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:18.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:18.108 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:18.129 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:18.134 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:59:18.134 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-30T23:59:18.134 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-30T23:59:18.134 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-30T23:59:18.134 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:59:18.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:18.255 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:18.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:18.381 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:18.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:18.507 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:18.609 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:18.633 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:18.734 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:18.758 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:18.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:18.878 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-30T23:59:18.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:18.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:18.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:18.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:19.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:19.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:19.135 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-30T23:59:19.135 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:19.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:19.257 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:19 smithi136 ceph-mon[20207]: 9.8 scrub starts 2024-08-30T23:59:19.257 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:19 smithi136 ceph-mon[20207]: pgmap v2201: 67 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 remapped+peering, 1 active+recovering+undersized+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 36 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16209/304818 objects degraded (5.318%); 20802/304818 objects misplaced (6.824%) 2024-08-30T23:59:19.257 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:19 smithi136 ceph-mon[20207]: osdmap e999: 12 total, 11 up, 12 in 2024-08-30T23:59:19.257 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:19 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-30T23:59:19.257 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:19 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-30T23:59:19.257 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:19 smithi136 ceph-mon[20207]: 9.1b scrub starts 2024-08-30T23:59:19.265 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:19 smithi106 ceph-mon[16603]: 9.8 scrub starts 2024-08-30T23:59:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:19 smithi106 ceph-mon[16603]: pgmap v2201: 67 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 remapped+peering, 1 active+recovering+undersized+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 36 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16209/304818 objects degraded (5.318%); 20802/304818 objects misplaced (6.824%) 2024-08-30T23:59:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:19 smithi106 ceph-mon[16603]: osdmap e999: 12 total, 11 up, 12 in 2024-08-30T23:59:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:19 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-30T23:59:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:19 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-30T23:59:19.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:19 smithi106 ceph-mon[16603]: 9.1b scrub starts 2024-08-30T23:59:19.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:19.393 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:19.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:19.521 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:19 smithi173 ceph-mon[20554]: 9.8 scrub starts 2024-08-30T23:59:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:19 smithi173 ceph-mon[20554]: pgmap v2201: 67 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 1 remapped+peering, 1 active+recovering+undersized+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 36 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16209/304818 objects degraded (5.318%); 20802/304818 objects misplaced (6.824%) 2024-08-30T23:59:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:19 smithi173 ceph-mon[20554]: osdmap e999: 12 total, 11 up, 12 in 2024-08-30T23:59:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:19 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-30T23:59:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:19 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-30T23:59:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:19 smithi173 ceph-mon[20554]: 9.1b scrub starts 2024-08-30T23:59:19.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:19.648 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:19.669 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-30T23:59:19.669 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-30T23:59:19.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.0 795/788 [PushOp(7:0859067a:::benchmark_data_smithi173_129296_object41869:head, version: 739'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:0859067a:::benchmark_data_smithi173_129296_object41869:head@739'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 436.85426860199999, 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.2782153, 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:59:19.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814176+0000", 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7199999999999998e-07 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814204+0000", 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7526e-05 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090173+0000", 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27596965699999998 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090195+0000", 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2075000000000001e-05 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090304+0000", 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010874699999999999 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090321+0000", 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7149000000000001e-05 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.092391+0000", 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002069474 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21763 8.4 8:2f5d79f6:::benchmark_data_smithi173_143816_object21762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 309.59050803700001, 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30212884200000001, 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:59:19.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21763 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077938+0000", 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0480000000000001e-06 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077983+0000", 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5318999999999998e-05 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078009+0000", 2024-08-30T23:59:19.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5554999999999998e-05 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078016+0000", 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4769999999999997e-06 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078078+0000", 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1748999999999994e-05 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078236+0000", 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000157434 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078373+0000", 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013750000000000001 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079802+0000", 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001428653 2024-08-30T23:59:19.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079818+0000", 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6597999999999999e-05 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379984+0000", 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30016611799999998 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380032+0000", 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7268000000000002e-05 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380065+0000", 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3123000000000001e-05 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21763 8.4 e852/840 ondisk, result = 0)", 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 309.58494776100002, 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.296575003, 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083497+0000", 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1500000000000004e-07 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083505+0000", 2024-08-30T23:59:19.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-30T23:59:19.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379828+0000", 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29632308899999998 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379844+0000", 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5665999999999999e-05 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379970+0000", 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012600999999999999 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379979+0000", 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9770000000000006e-06 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380071+0000", 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2387999999999997e-05 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21778 8.8 8:10e57910:::benchmark_data_smithi173_143816_object21777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 309.44112105900001, 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28839689299999999, 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21778 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-30T23:59:19.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227333+0000", 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5489999999999995e-06 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227385+0000", 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2265999999999997e-05 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227427+0000", 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2255000000000001e-05 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227452+0000", 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4443999999999998e-05 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227520+0000", 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8916000000000002e-05 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227528+0000", 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228639+0000", 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011110810000000001 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228736+0000", 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7001000000000002e-05 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228817+0000", 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1038000000000006e-05 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.229951+0000", 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001133319 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230356+0000", 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040563399999999999 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515551+0000", 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28519461400000001 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515669+0000", 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117731 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515720+0000", 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1390000000000001e-05 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21778 8.8 e853/840 ondisk, result = 0)", 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 309.43807146500001, 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28535590100000002, 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:59:19.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0399999999999996e-07 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230376+0000", 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2480000000000001e-06 2024-08-30T23:59:19.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515346+0000", 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28496944699999999 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515361+0000", 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5573000000000002e-05 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515505+0000", 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515533+0000", 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7696e-05 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515728+0000", 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019533300000000001 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21807 8.4 8:2833fcc7:::benchmark_data_smithi173_143816_object21806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-30T23:59:19.696 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 309.06559709099997, 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34692570299999997, 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21807 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602848+0000", 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009e-06 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602878+0000", 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0278999999999999e-05 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602891+0000", 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2371000000000001e-05 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602897+0000", 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602941+0000", 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4617000000000003e-05 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603031+0000", 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0060000000000002e-05 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603115+0000", 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3387000000000006e-05 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603890+0000", 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077577699999999996 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604297+0000", 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040658900000000001 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949697+0000", 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34539998399999999 2024-08-30T23:59:19.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949739+0000", 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1758000000000002e-05 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949773+0000", 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3970999999999997e-05 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21807 8.4 e853/840 ondisk, result = 0)", 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 309.064213698, 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34559199699999998, 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604231+0000", 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7599999999999999e-07 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604232+0000", 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.105e-06 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949585+0000", 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34535339199999998 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949593+0000", 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.644e-06 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949684+0000", 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1680000000000003e-05 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949693+0000", 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.354e-06 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949822+0000", 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129646 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:61503 8.4 8:29953d46:::benchmark_data_smithi173_143816_object61502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 269.40715655700001, 2024-08-30T23:59:19.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24835605799999999, 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61503 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9299999999999997e-07 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261343+0000", 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5149999999999999e-05 2024-08-30T23:59:19.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261352+0000", 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261356+0000", 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1420000000000003e-06 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261393+0000", 2024-08-30T23:59:19.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899e-05 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261516+0000", 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123299 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261655+0000", 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139195 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263257+0000", 2024-08-30T23:59:19.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016018880000000001 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263968+0000", 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071132499999999996 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509535+0000", 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24556666499999999 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509591+0000", 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6273000000000001e-05 2024-08-30T23:59:19.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509644+0000", 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2148999999999998e-05 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(9.6 e948 from shard 10)", 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:54.088793+0000", 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.579651554999998, 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.544203887, 2024-08-30T23:59:19.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088793+0000", 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088793+0000", 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088793+0000", 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088831+0000", 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7805000000000001e-05 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632788+0000", 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54395688799999997 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632793+0000", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.853e-06 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632843+0000", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0306000000000003e-05 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632856+0000", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3033999999999999e-05 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632996+0000", 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000140381 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:19270 9.f 9:f74eb61c:::benchmark_data_smithi173_160189_object19269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:54.328258+0000", 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.340186013999997, 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.312871391, 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19270 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328258+0000", 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328258+0000", 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328260+0000", 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7859999999999999e-06 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328294+0000", 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4258999999999998e-05 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328314+0000", 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9814e-05 2024-08-30T23:59:19.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328324+0000", 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5959999999999993e-06 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328407+0000", 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328512+0000", 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104817 2024-08-30T23:59:19.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,9", 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328611+0000", 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8474999999999995e-05 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.329444+0000", 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083315400000000001 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.330021+0000", 2024-08-30T23:59:19.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057678899999999997 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.641045+0000", 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31102477299999998 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.641086+0000", 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0747000000000001e-05 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.641129+0000", 2024-08-30T23:59:19.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3408999999999997e-05 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "replica_scrub(pg: 9.9,from:0'0,to:949'1127,epoch:951/948,start:9:97883b83:::benchmark_data_smithi173_160189_object15908:0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:55.808503+0000", 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 83.859941410999994, 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31569409599999998, 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808503+0000", 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808503+0000", 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808505+0000", 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.424e-06 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808521+0000", 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-05 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.122578+0000", 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31405760300000002 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.122590+0000", 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1260000000000001e-05 2024-08-30T23:59:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.124149+0000", 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00155949 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.124163+0000", 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582e-05 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.124197+0000", 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4116999999999999e-05 2024-08-30T23:59:19.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(9.21 971/969 [PushOp(9:85fa522a:::benchmark_data_smithi173_160189_object19232:head, version: 949'1108, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:85fa522a:::benchmark_data_smithi173_160189_object19232:head@949'1108, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:33.089878+0000", 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.578566242999997, 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25101715099999999, 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089878+0000", 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089878+0000", 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089878+0000", 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.75e-07 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089926+0000", 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7298e-05 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340259+0000", 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25033364699999999 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340268+0000", 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1679999999999993e-06 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340345+0000", 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7075000000000003e-05 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340359+0000", 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3258000000000001e-05 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340895+0000", 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053622999999999995 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:62298 9.19 e971/957 9:9be56da5:::benchmark_data_smithi173_160189_object62297:head v 971'2500, mlcod=971'2497)", 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:33.094019+0000", 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.574424604000001, 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24765731099999999, 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094019+0000", 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094019+0000", 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094020+0000", 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3799999999999997e-07 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094123+0000", 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102896 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340587+0000", 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.246463975 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340591+0000", 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6320000000000001e-06 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340632+0000", 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1198000000000001e-05 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340652+0000", 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0171999999999999e-05 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.341651+0000", 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00099860500000000007 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.341677+0000", 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6194999999999999e-05 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:74093 9.21 9:865d22da:::benchmark_data_smithi173_160189_object74092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e976)", 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:44.666003+0000", 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 35.002440759999999, 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25998222799999998, 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74093 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666003+0000", 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666003+0000", 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666004+0000", 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666058+0000", 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3695e-05 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666069+0000", 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.079e-05 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666081+0000", 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.222e-05 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666131+0000", 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0492999999999999e-05 2024-08-30T23:59:19.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666239+0000", 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108045 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666365+0000", 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012594900000000001 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667262+0000", 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00089650099999999996 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.669369+0000", 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021067350000000002 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925924+0000", 2024-08-30T23:59:19.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25655586000000002 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925959+0000", 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4552999999999999e-05 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925986+0000", 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6519000000000001e-05 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.28405.0:74093 9.21 e976/974 ondisk, result = 0)", 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:44.667959+0000", 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 35.000484845000003, 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25803844799999998, 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667959+0000", 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667959+0000", 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667960+0000", 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0200000000000002e-07 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667964+0000", 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0720000000000001e-06 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925769+0000", 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25780540000000002 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925781+0000", 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1717e-05 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925896+0000", 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114911 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925921+0000", 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5085000000000001e-05 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925998+0000", 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6661000000000003e-05 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:91166 9.1d 9:b9e063a0:::benchmark_data_smithi173_160189_object91165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e984)", 2024-08-30T23:59:19.745 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:00.451819+0000", 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.216624871, 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63616116899999997, 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91166 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451819+0000", 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451819+0000", 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451820+0000", 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451875+0000", 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4614000000000003e-05 2024-08-30T23:59:19.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451886+0000", 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1785e-05 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451895+0000", 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6950000000000006e-06 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.453708+0000", 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018123639999999999 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.453830+0000", 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012297299999999999 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.454044+0000", 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021353 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.455288+0000", 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012440859999999999 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.470029+0000", 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014740985999999999 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.087909+0000", 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61788033799999997 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.087947+0000", 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7463999999999999e-05 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.087980+0000", 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3458999999999999e-05 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(9.1b 994/969 [PushReplyOp(9:db66c045:::benchmark_data_smithi173_160189_object46533:head)])", 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:13.482460+0000", 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 6.1859845030000002, 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35726268799999999, 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.751 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482460+0000", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482460+0000", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482460+0000", 2024-08-30T23:59:19.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4499999999999996e-07 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482466+0000", 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3029999999999999e-06 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839039+0000", 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35657290000000003 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839049+0000", 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0334e-05 2024-08-30T23:59:19.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839141+0000", 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2153999999999999e-05 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839722+0000", 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058115200000000004 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(9.1b 995/969 [PushReplyOp(9:da33c270:::benchmark_data_smithi173_160189_object52147:head)])", 2024-08-30T23:59:19.754 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:14.534516+0000", 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.1339282490000002, 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45652447000000002, 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534516+0000", 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534516+0000", 2024-08-30T23:59:19.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534516+0000", 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6199999999999998e-07 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534520+0000", 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6239999999999999e-06 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.990657+0000", 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.456136879 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.990670+0000", 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3463e-05 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.990766+0000", 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5520999999999996e-05 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.991040+0000", 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027442100000000001 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:103252 9.13 e998/997 9:cf2b091d:::benchmark_data_smithi173_160189_object1248:head v 998'3726, mlcod=998'3723)", 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:17.297722+0000", 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 2.3707217840000001, 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37625371200000002, 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297722+0000", 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297722+0000", 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297723+0000", 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9899999999999999e-07 2024-08-30T23:59:19.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297729+0000", 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672211+0000", 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37448169399999998 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672220+0000", 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3950000000000007e-06 2024-08-30T23:59:19.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672285+0000", 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4584000000000004e-05 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672310+0000", 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5624999999999999e-05 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.673937+0000", 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016269050000000001 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.673976+0000", 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8821e-05 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:103268 9.19 e998/957 9:9c47d45a:::benchmark_data_smithi173_160189_object1264:head v 998'3776, mlcod=998'3774)", 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:17.320130+0000", 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 2.3483138600000002, 2024-08-30T23:59:19.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.73841249399999997, 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320130+0000", 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320130+0000", 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320132+0000", 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5579999999999999e-06 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320140+0000", 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9019999999999999e-06 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.056952+0000", 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.73681268300000002 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.056958+0000", 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9229999999999999e-06 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.057012+0000", 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3801000000000002e-05 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.057029+0000", 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6574000000000001e-05 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.058491+0000", 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014624760000000001 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.058543+0000", 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1576999999999998e-05 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-30T23:59:19.766 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-30T23:59:19.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:19.795 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:19.897 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:19.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:19.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:19.926 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:59:19.927 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:59:19.927 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-30T23:59:19.927 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-30T23:59:20.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:20.044 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:20.146 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:20.178 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:20.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:20.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: 9.8 scrub ok 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: 9.1b scrub ok 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: osdmap e1000: 12 total, 11 up, 12 in 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: 9.15 scrub starts 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-30T23:59:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:20 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-30T23:59:20.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:20.428 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:20.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: 9.8 scrub ok 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: 9.1b scrub ok 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: osdmap e1000: 12 total, 11 up, 12 in 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: 9.15 scrub starts 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-30T23:59:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:20 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-30T23:59:20.551 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: 9.8 scrub ok 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: 9.1b scrub ok 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: osdmap e1000: 12 total, 11 up, 12 in 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: 9.15 scrub starts 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-30T23:59:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:20 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-30T23:59:20.653 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:20.685 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:20.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:20.812 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:20.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-30T23:59:20.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:20.943 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:20.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:21.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:21.073 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.074 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:21.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:21.201 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:21.303 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:21.329 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:21.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:21 smithi106 ceph-mon[16603]: pgmap v2204: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 38 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16060/301545 objects degraded (5.326%); 21143/301545 objects misplaced (7.012%) 2024-08-30T23:59:21.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:21 smithi106 ceph-mon[16603]: 9.15 scrub ok 2024-08-30T23:59:21.349 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 16060/301545 objects degraded (5.326%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:21.350 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:21 smithi106 ceph-mon[16603]: osdmap e1001: 12 total, 11 up, 12 in 2024-08-30T23:59:21.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.459 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:21.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:21 smithi173 ceph-mon[20554]: pgmap v2204: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 38 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16060/301545 objects degraded (5.326%); 21143/301545 objects misplaced (7.012%) 2024-08-30T23:59:21.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:21 smithi173 ceph-mon[20554]: 9.15 scrub ok 2024-08-30T23:59:21.474 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 16060/301545 objects degraded (5.326%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:21.475 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:21 smithi173 ceph-mon[20554]: osdmap e1001: 12 total, 11 up, 12 in 2024-08-30T23:59:21.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:21 smithi136 ceph-mon[20207]: pgmap v2204: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 38 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16060/301545 objects degraded (5.326%); 21143/301545 objects misplaced (7.012%) 2024-08-30T23:59:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:21 smithi136 ceph-mon[20207]: 9.15 scrub ok 2024-08-30T23:59:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 16060/301545 objects degraded (5.326%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:21 smithi136 ceph-mon[20207]: osdmap e1001: 12 total, 11 up, 12 in 2024-08-30T23:59:21.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:21.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:21.684 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:21.711 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:21.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:21.846 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:21.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:21.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:21.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:22.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:22.109 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:22.211 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:22.238 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:22.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:22.369 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:22.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.501 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:22.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:22.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:22.630 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:22.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:22.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:22.856 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:22.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:22.979 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:23.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:23.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:23.129 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:23.231 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:23.257 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:23.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:23 smithi106 ceph-mon[16603]: pgmap v2206: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 peering, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 B/s rd, 0 op/s; 15588/298842 objects degraded (5.216%); 20920/298842 objects misplaced (7.000%) 2024-08-30T23:59:23.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:23.371 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:23 smithi136 ceph-mon[20207]: pgmap v2206: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 peering, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 B/s rd, 0 op/s; 15588/298842 objects degraded (5.216%); 20920/298842 objects misplaced (7.000%) 2024-08-30T23:59:23.385 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:23.487 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:23.515 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.516 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:23 smithi173 ceph-mon[20554]: pgmap v2206: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 1 active+recovering+degraded+remapped, 1 peering, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 B/s rd, 0 op/s; 15588/298842 objects degraded (5.216%); 20920/298842 objects misplaced (7.000%) 2024-08-30T23:59:23.617 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:23.639 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:23.741 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:23.767 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:23.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:23.899 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:23.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:24.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:24.047 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:24.074 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:24.122 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:24.151 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:24.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:24.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:24.304 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:24 smithi106 ceph-mon[16603]: 1.19 scrub starts 2024-08-30T23:59:24.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:24 smithi106 ceph-mon[16603]: 1.19 scrub ok 2024-08-30T23:59:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:24 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-30T23:59:24.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:24 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-30T23:59:24.363 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:24.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:24.427 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.429 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:24.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:24.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:24 smithi136 ceph-mon[20207]: 1.19 scrub starts 2024-08-30T23:59:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:24 smithi136 ceph-mon[20207]: 1.19 scrub ok 2024-08-30T23:59:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:24 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-30T23:59:24.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:24 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-30T23:59:24.557 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:24 smithi173 ceph-mon[20554]: 1.19 scrub starts 2024-08-30T23:59:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:24 smithi173 ceph-mon[20554]: 1.19 scrub ok 2024-08-30T23:59:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:24 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-30T23:59:24.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:24 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-30T23:59:24.641 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:24.641 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:24.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:24.663 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:24.663 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:24.682 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:24.687 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:24.688 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:24.716 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:24.716 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:24.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:24.805 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:24.907 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:24.938 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:24.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:24.944 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:24.944 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:24.971 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:24.971 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:25.003 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:25.003 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:25.004 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:25.035 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:25.035 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:25.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:25.062 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:25.099 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:25.099 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:25.145 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:25.146 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:25.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:25.176 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:25.176 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:25.193 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:25.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:25 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-30T23:59:25.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:25 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-30T23:59:25.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:25 smithi106 ceph-mon[16603]: pgmap v2207: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s; 15311/295482 objects degraded (5.182%); 20770/295482 objects misplaced (7.029%) 2024-08-30T23:59:25.207 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1492858582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:25.294 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:25.328 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:25.355 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:59:25.356 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:59:25.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:25 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-30T23:59:25.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:25 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-30T23:59:25.370 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:25 smithi173 ceph-mon[20554]: pgmap v2207: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s; 15311/295482 objects degraded (5.182%); 20770/295482 objects misplaced (7.029%) 2024-08-30T23:59:25.371 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1492858582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:25.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:25.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.456 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:25.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:25 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-30T23:59:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:25 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-30T23:59:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:25 smithi136 ceph-mon[20207]: pgmap v2207: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s; 15311/295482 objects degraded (5.182%); 20770/295482 objects misplaced (7.029%) 2024-08-30T23:59:25.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1492858582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:25.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:25.582 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:25.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:25.706 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:25.795 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-30T23:59:25.796 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-30T23:59:25.808 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:25.831 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:25.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:25.958 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:25.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:26.043 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:59:26.060 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:26.084 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.086 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:26.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:26.208 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:26.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:26.328 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:26 smithi106 ceph-mon[16603]: from='client.28576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:59:26.328 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 15311/295482 objects degraded (5.182%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:26.328 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:26 smithi106 ceph-mon[16603]: osdmap e1002: 12 total, 11 up, 12 in 2024-08-30T23:59:26.329 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-30T23:59:26.332 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:26.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:26.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:26 smithi136 ceph-mon[20207]: from='client.28576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:59:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 15311/295482 objects degraded (5.182%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:26.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:26 smithi136 ceph-mon[20207]: osdmap e1002: 12 total, 11 up, 12 in 2024-08-30T23:59:26.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:26.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:26 smithi173 ceph-mon[20554]: from='client.28576 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-30T23:59:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 15311/295482 objects degraded (5.182%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:26 smithi173 ceph-mon[20554]: osdmap e1002: 12 total, 11 up, 12 in 2024-08-30T23:59:26.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:26.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.705 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:26.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:26.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:26.829 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:26.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:26.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:26.952 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:26.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:26.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:59:26.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:59:27.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:27.077 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:27.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:27.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:27.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:27.338 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:27 smithi106 ceph-mon[16603]: pgmap v2208: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s rd, 0 B/s wr, 494 op/s; 14891/283527 objects degraded (5.252%); 19382/283527 objects misplaced (6.836%) 2024-08-30T23:59:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:27 smithi106 ceph-mon[16603]: osdmap e1003: 12 total, 11 up, 12 in 2024-08-30T23:59:27.440 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:27.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:27.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:27 smithi136 ceph-mon[20207]: pgmap v2208: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s rd, 0 B/s wr, 494 op/s; 14891/283527 objects degraded (5.252%); 19382/283527 objects misplaced (6.836%) 2024-08-30T23:59:27.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:27 smithi136 ceph-mon[20207]: osdmap e1003: 12 total, 11 up, 12 in 2024-08-30T23:59:27.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:27.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:27 smithi173 ceph-mon[20554]: pgmap v2208: 67 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s rd, 0 B/s wr, 494 op/s; 14891/283527 objects degraded (5.252%); 19382/283527 objects misplaced (6.836%) 2024-08-30T23:59:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:27 smithi173 ceph-mon[20554]: osdmap e1003: 12 total, 11 up, 12 in 2024-08-30T23:59:27.596 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:27.678 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2207,"stamp":"2024-08-30T23:59:23.870958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6455369589,"num_objects":98494,"num_object_clones":0,"num_object_copies":295482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15311,"num_objects_misplaced":20770,"num_objects_unfound":0,"num_objects_dirty":98494,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":65150,"num_write_kb":3944563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8695,"num_bytes_recovered":569919667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147912,"ondisk_log_size":147912,"up":177,"acting":187,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":215,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":27716192,"kb_used_data":19362140,"kb_used_omap":109,"kb_used_meta":8353682,"kb_avail":1097225632,"statfs":{"total":1151940427776,"available":1123559047168,"internally_reserved":0,"allocated":19826831360,"data_stored":19815643191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":8554171032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-204013544,"num_objects":-3112,"num_object_clones":0,"num_object_copies":-9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-898,"num_objects_misplaced":-32,"num_objects_unfound":0,"num_objects_dirty":-3112,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":-137,"num_write_kb":-205119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-536,"num_bytes_recovered":-35127296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001579"},"pg_stats":[{"pgid":"9.23","version":"1001'3006","reported_seq":5766,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:22.183760+0000","last_change":"2024-08-30T23:59:12.788853+0000","last_active":"2024-08-30T23:59:22.183760+0000","last_peered":"2024-08-30T23:59:22.183760+0000","last_clean":"2024-08-30T23:59:22.183760+0000","last_became_active":"2024-08-30T23:59:01.084823+0000","last_became_peered":"2024-08-30T23:59:01.084823+0000","last_unstale":"2024-08-30T23:59:22.183760+0000","last_undegraded":"2024-08-30T23:59:22.183760+0000","last_fullsized":"2024-08-30T23:59:22.183760+0000","mapping_epoch":984,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":985,"parent":"0.0","parent_split_bits":6,"last_scrub":"993'2910","last_scrub_stamp":"2024-08-30T23:59:12.788717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:12.788717+0000","objects_scrubbed":1534,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:19.319859+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95223808,"num_objects":1453,"num_object_clones":0,"num_object_copies":4359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.22","version":"1000'2902","reported_seq":4707,"reported_epoch":1000,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:19.348603+0000","last_change":"2024-08-30T23:58:35.493120+0000","last_active":"2024-08-30T23:59:19.348603+0000","last_peered":"2024-08-30T23:59:19.348603+0000","last_clean":"2024-08-30T23:59:19.348603+0000","last_became_active":"2024-08-30T23:58:34.338653+0000","last_became_peered":"2024-08-30T23:58:34.338653+0000","last_unstale":"2024-08-30T23:59:19.348603+0000","last_undegraded":"2024-08-30T23:59:19.348603+0000","last_fullsized":"2024-08-30T23:59:19.348603+0000","mapping_epoch":972,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":6,"last_scrub":"973'2327","last_scrub_stamp":"2024-08-30T23:58:35.492788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:35.492788+0000","objects_scrubbed":939,"log_size":2902,"log_dups_size":0,"ondisk_log_size":2902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:58.878728+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94437376,"num_objects":1441,"num_object_clones":0,"num_object_copies":4323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1441,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":576,"num_write_kb":34496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4,1],"avail_no_missing":["9","1","4"],"object_location_counts":[{"shards":"1,4,9","objects":1441}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.21","version":"1000'3059","reported_seq":6662,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.613012+0000","last_change":"2024-08-30T23:58:43.086940+0000","last_active":"2024-08-30T23:59:19.613012+0000","last_peered":"2024-08-30T23:59:19.613012+0000","last_clean":"2024-08-30T23:59:19.613012+0000","last_became_active":"2024-08-30T23:58:42.354339+0000","last_became_peered":"2024-08-30T23:58:42.354339+0000","last_unstale":"2024-08-30T23:59:19.613012+0000","last_undegraded":"2024-08-30T23:59:19.613012+0000","last_fullsized":"2024-08-30T23:59:19.613012+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":6,"last_scrub":"975'2511","last_scrub_stamp":"2024-08-30T23:58:43.086862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:43.086862+0000","objects_scrubbed":1135,"log_size":3059,"log_dups_size":0,"ondisk_log_size":3059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:47.235977+0000","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103940096,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":549,"num_write_kb":32000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.20","version":"1000'3043","reported_seq":6502,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:20.751940+0000","last_change":"2024-08-30T23:58:43.259931+0000","last_active":"2024-08-30T23:59:20.751940+0000","last_peered":"2024-08-30T23:59:20.751940+0000","last_clean":"2024-08-30T23:59:20.751940+0000","last_became_active":"2024-08-30T23:58:42.350889+0000","last_became_peered":"2024-08-30T23:58:42.350889+0000","last_unstale":"2024-08-30T23:59:20.751940+0000","last_undegraded":"2024-08-30T23:59:20.751940+0000","last_fullsized":"2024-08-30T23:59:20.751940+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":6,"last_scrub":"975'2536","last_scrub_stamp":"2024-08-30T23:58:43.259851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:43.259851+0000","objects_scrubbed":1120,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:52.758062+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98631680,"num_objects":1505,"num_object_clones":0,"num_object_copies":4515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":509,"num_write_kb":28608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.1c","version":"999'4807","reported_seq":10103,"reported_epoch":999,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-30T23:59:18.801269+0000","last_change":"2024-08-30T23:59:16.256984+0000","last_active":"2024-08-30T23:59:18.801269+0000","last_peered":"2024-08-30T23:59:18.801269+0000","last_clean":"2024-08-30T23:58:24.213324+0000","last_became_active":"2024-08-30T23:59:14.201705+0000","last_became_peered":"2024-08-30T23:59:14.201705+0000","last_unstale":"2024-08-30T23:59:18.801269+0000","last_undegraded":"2024-08-30T23:59:18.801269+0000","last_fullsized":"2024-08-30T23:59:14.173680+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201588736,"num_objects":3076,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":712,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":199872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1728,"num_bytes_recovered":113246208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,3],"acting":[3,0],"avail_no_missing":["3","0","4","9"],"object_location_counts":[{"shards":"0,3","objects":2364},{"shards":"0,3,4,9,10","objects":396},{"shards":"0,3,10","objects":316}],"blocked_by":[],"up_primary":11,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1175,"reported_epoch":1000,"state":"active+undersized","last_fresh":"2024-08-30T23:59:19.089746+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-30T23:59:19.089746+0000","last_peered":"2024-08-30T23:59:19.089746+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-30T23:59:19.089746+0000","last_undegraded":"2024-08-30T23:59:19.089746+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":32,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.089333+0000","last_change":"2024-08-30T23:59:14.299069+0000","last_active":"2024-08-30T23:59:21.089333+0000","last_peered":"2024-08-30T23:59:21.089333+0000","last_clean":"2024-08-30T23:59:21.089333+0000","last_became_active":"2024-08-30T23:59:14.186459+0000","last_became_peered":"2024-08-30T23:59:14.186459+0000","last_unstale":"2024-08-30T23:59:21.089333+0000","last_undegraded":"2024-08-30T23:59:21.089333+0000","last_fullsized":"2024-08-30T23:59:21.089333+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:14.299023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:14.299023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:36.419559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.3","version":"1000'2980","reported_seq":3166,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.381670+0000","last_change":"2024-08-30T23:58:25.101873+0000","last_active":"2024-08-30T23:59:19.381670+0000","last_peered":"2024-08-30T23:59:19.381670+0000","last_clean":"2024-08-30T23:59:19.381670+0000","last_became_active":"2024-08-30T23:58:23.229111+0000","last_became_peered":"2024-08-30T23:58:23.229111+0000","last_unstale":"2024-08-30T23:59:19.381670+0000","last_undegraded":"2024-08-30T23:59:19.381670+0000","last_fullsized":"2024-08-30T23:59:19.381670+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'2230","last_scrub_stamp":"2024-08-30T23:58:25.101767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:25.101767+0000","objects_scrubbed":870,"log_size":2980,"log_dups_size":0,"ondisk_log_size":2980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:38.278227+0000","scrub_duration":858,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101974016,"num_objects":1556,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":750,"num_write_kb":45952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1228,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.089533+0000","last_change":"2024-08-30T23:58:08.502691+0000","last_active":"2024-08-30T23:59:19.089533+0000","last_peered":"2024-08-30T23:59:19.089533+0000","last_clean":"2024-08-30T23:59:19.089533+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-30T23:59:19.089533+0000","last_undegraded":"2024-08-30T23:59:19.089533+0000","last_fullsized":"2024-08-30T23:59:19.089533+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:08.502643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:08.502643+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:23.323808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"999'2965","reported_seq":3148,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-08-30T23:59:18.665148+0000","last_change":"2024-08-30T23:58:24.986319+0000","last_active":"2024-08-30T23:59:18.665148+0000","last_peered":"2024-08-30T23:59:18.665148+0000","last_clean":"2024-08-30T23:59:18.665148+0000","last_became_active":"2024-08-30T23:58:23.225981+0000","last_became_peered":"2024-08-30T23:58:23.225981+0000","last_unstale":"2024-08-30T23:59:18.665148+0000","last_undegraded":"2024-08-30T23:59:18.665148+0000","last_fullsized":"2024-08-30T23:59:18.665148+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'2200","last_scrub_stamp":"2024-08-30T23:58:24.986029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:24.986029+0000","objects_scrubbed":883,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:38.998643+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103940096,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":765,"num_write_kb":46976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1082,"reported_epoch":1000,"state":"active+undersized","last_fresh":"2024-08-30T23:59:19.089669+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-30T23:59:19.089669+0000","last_peered":"2024-08-30T23:59:19.089669+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-30T23:59:19.089669+0000","last_undegraded":"2024-08-30T23:59:19.089669+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:59:21.444863+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"999'3009","reported_seq":3190,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-08-30T23:59:18.852877+0000","last_change":"2024-08-30T23:58:25.993565+0000","last_active":"2024-08-30T23:59:18.852877+0000","last_peered":"2024-08-30T23:59:18.852877+0000","last_clean":"2024-08-30T23:59:18.852877+0000","last_became_active":"2024-08-30T23:58:23.226284+0000","last_became_peered":"2024-08-30T23:58:23.226284+0000","last_unstale":"2024-08-30T23:59:18.852877+0000","last_undegraded":"2024-08-30T23:59:18.852877+0000","last_fullsized":"2024-08-30T23:59:18.852877+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'2220","last_scrub_stamp":"2024-08-30T23:58:25.993421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:25.993421+0000","objects_scrubbed":860,"log_size":3009,"log_dups_size":0,"ondisk_log_size":3009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:33.490251+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105447424,"num_objects":1609,"num_object_clones":0,"num_object_copies":4827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":49216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":485,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.087400+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-30T23:59:19.087400+0000","last_peered":"2024-08-30T23:59:19.087400+0000","last_clean":"2024-08-30T23:59:19.087400+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-30T23:59:19.087400+0000","last_undegraded":"2024-08-30T23:59:19.087400+0000","last_fullsized":"2024-08-30T23:59:19.087400+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"1001'3072","reported_seq":5547,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:22.330287+0000","last_change":"2024-08-30T23:59:16.115025+0000","last_active":"2024-08-30T23:59:22.330287+0000","last_peered":"2024-08-30T23:59:22.330287+0000","last_clean":"2024-08-30T23:59:22.330287+0000","last_became_active":"2024-08-30T23:59:16.112131+0000","last_became_peered":"2024-08-30T23:59:16.112131+0000","last_unstale":"2024-08-30T23:59:22.330287+0000","last_undegraded":"2024-08-30T23:59:22.330287+0000","last_fullsized":"2024-08-30T23:59:22.330287+0000","mapping_epoch":996,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'2225","last_scrub_stamp":"2024-08-30T23:58:24.279213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:24.279213+0000","objects_scrubbed":904,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:50.259480+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102367232,"num_objects":1562,"num_object_clones":0,"num_object_copies":4686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":848,"num_write_kb":48192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1135,"num_bytes_recovered":74383360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1305,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.085362+0000","last_change":"2024-08-30T23:57:52.872669+0000","last_active":"2024-08-30T23:59:21.085362+0000","last_peered":"2024-08-30T23:59:21.085362+0000","last_clean":"2024-08-30T23:59:21.085362+0000","last_became_active":"2024-08-30T23:57:36.158625+0000","last_became_peered":"2024-08-30T23:57:36.158625+0000","last_unstale":"2024-08-30T23:59:21.085362+0000","last_undegraded":"2024-08-30T23:59:21.085362+0000","last_fullsized":"2024-08-30T23:59:21.085362+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:52.872607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:52.872607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:21.985915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"1001'3912","reported_seq":4056,"reported_epoch":1001,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:59:22.130930+0000","last_change":"2024-08-30T23:57:53.077387+0000","last_active":"2024-08-30T23:59:22.130930+0000","last_peered":"2024-08-30T23:59:22.130930+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:53.077387+0000","last_became_peered":"2024-08-30T23:57:53.077387+0000","last_unstale":"2024-08-30T23:59:22.130930+0000","last_undegraded":"2024-08-30T23:57:53.068813+0000","last_fullsized":"2024-08-30T23:57:53.068689+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:59:26.451605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198311936,"num_objects":3026,"num_object_clones":0,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":204864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","3"],"object_location_counts":[{"shards":"3,10","objects":3026}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":1021,"reported_epoch":1001,"state":"active+undersized","last_fresh":"2024-08-30T23:59:21.095814+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-30T23:59:21.095814+0000","last_peered":"2024-08-30T23:59:21.095814+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-30T23:59:21.095814+0000","last_undegraded":"2024-08-30T23:59:21.095814+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:22.930465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"1000'3779","reported_seq":3939,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.746382+0000","last_change":"2024-08-30T23:57:55.058608+0000","last_active":"2024-08-30T23:59:19.746382+0000","last_peered":"2024-08-30T23:59:19.746382+0000","last_clean":"2024-08-30T23:59:19.746382+0000","last_became_active":"2024-08-30T23:57:53.081131+0000","last_became_peered":"2024-08-30T23:57:53.081131+0000","last_unstale":"2024-08-30T23:59:19.746382+0000","last_undegraded":"2024-08-30T23:59:19.746382+0000","last_fullsized":"2024-08-30T23:59:19.746382+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1118","last_scrub_stamp":"2024-08-30T23:57:55.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:55.058465+0000","objects_scrubbed":594,"log_size":3779,"log_dups_size":0,"ondisk_log_size":3779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":1090,"reported_epoch":1001,"state":"active+undersized","last_fresh":"2024-08-30T23:59:21.095866+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-30T23:59:21.095866+0000","last_peered":"2024-08-30T23:59:21.095866+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-30T23:59:21.095866+0000","last_undegraded":"2024-08-30T23:59:21.095866+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"1001'3835","reported_seq":7319,"reported_epoch":1001,"state":"active+recovering+degraded+remapped","last_fresh":"2024-08-30T23:59:21.365745+0000","last_change":"2024-08-30T23:59:15.207522+0000","last_active":"2024-08-30T23:59:21.365745+0000","last_peered":"2024-08-30T23:59:21.365745+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:59:15.187163+0000","last_became_peered":"2024-08-30T23:59:15.187163+0000","last_unstale":"2024-08-30T23:59:21.365745+0000","last_undegraded":"2024-08-30T23:59:15.182126+0000","last_fullsized":"2024-08-30T23:59:21.365745+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195756032,"num_objects":2987,"num_object_clones":0,"num_object_copies":8961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3281,"num_write_kb":200576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1666,"num_bytes_recovered":109182976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5,1],"avail_no_missing":["11","5"],"object_location_counts":[{"shards":"1,5,11","objects":1867},{"shards":"5,11","objects":1120}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":1109,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-08-30T23:59:18.088551+0000","last_change":"2024-08-30T23:57:32.972014+0000","last_active":"2024-08-30T23:59:18.088551+0000","last_peered":"2024-08-30T23:59:18.088551+0000","last_clean":"2024-08-30T23:59:18.088551+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-30T23:59:18.088551+0000","last_undegraded":"2024-08-30T23:59:18.088551+0000","last_fullsized":"2024-08-30T23:59:18.088551+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:32.971963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:32.971963+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"1001'3772","reported_seq":3934,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.918994+0000","last_change":"2024-08-30T23:57:53.615819+0000","last_active":"2024-08-30T23:59:21.918994+0000","last_peered":"2024-08-30T23:59:21.918994+0000","last_clean":"2024-08-30T23:59:21.918994+0000","last_became_active":"2024-08-30T23:57:53.070921+0000","last_became_peered":"2024-08-30T23:57:53.070921+0000","last_unstale":"2024-08-30T23:59:21.918994+0000","last_undegraded":"2024-08-30T23:59:21.918994+0000","last_fullsized":"2024-08-30T23:59:21.918994+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"949'1072","last_scrub_stamp":"2024-08-30T23:57:53.615713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:53.615713+0000","objects_scrubbed":564,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193724416,"num_objects":2956,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2702,"num_write_kb":163008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":460,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.085247+0000","last_change":"2024-08-30T23:59:18.160031+0000","last_active":"2024-08-30T23:59:21.085247+0000","last_peered":"2024-08-30T23:59:21.085247+0000","last_clean":"2024-08-30T23:59:21.085247+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-30T23:59:21.085247+0000","last_undegraded":"2024-08-30T23:59:21.085247+0000","last_fullsized":"2024-08-30T23:59:21.085247+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:18.159957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:18.159957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:42.330903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"1000'5064","reported_seq":5239,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:20.713482+0000","last_change":"2024-08-30T23:58:23.894116+0000","last_active":"2024-08-30T23:59:20.713482+0000","last_peered":"2024-08-30T23:59:20.713482+0000","last_clean":"2024-08-30T23:59:20.713482+0000","last_became_active":"2024-08-30T23:58:23.233769+0000","last_became_peered":"2024-08-30T23:58:23.233769+0000","last_unstale":"2024-08-30T23:59:20.713482+0000","last_undegraded":"2024-08-30T23:59:20.713482+0000","last_fullsized":"2024-08-30T23:59:20.713482+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'3430","last_scrub_stamp":"2024-08-30T23:58:23.893952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:23.893952+0000","objects_scrubbed":1718,"log_size":5064,"log_dups_size":0,"ondisk_log_size":5064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:41.108321+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":3136,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1642,"num_write_kb":97920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":2120,"reported_epoch":1001,"state":"active+undersized","last_fresh":"2024-08-30T23:59:21.096176+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-30T23:59:21.096176+0000","last_peered":"2024-08-30T23:59:21.096176+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-30T23:59:21.096176+0000","last_undegraded":"2024-08-30T23:59:21.096176+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"1000'4911","reported_seq":5053,"reported_epoch":1000,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:59:19.751248+0000","last_change":"2024-08-30T23:58:23.233914+0000","last_active":"2024-08-30T23:59:19.751248+0000","last_peered":"2024-08-30T23:59:19.751248+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.233914+0000","last_became_peered":"2024-08-30T23:58:23.233914+0000","last_unstale":"2024-08-30T23:59:19.751248+0000","last_undegraded":"2024-08-30T23:58:23.221431+0000","last_fullsized":"2024-08-30T23:58:23.221296+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206307328,"num_objects":3148,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3282,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":3148}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":633,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.088679+0000","last_change":"2024-08-30T23:57:08.049165+0000","last_active":"2024-08-30T23:59:21.088679+0000","last_peered":"2024-08-30T23:59:21.088679+0000","last_clean":"2024-08-30T23:59:21.088679+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-30T23:59:21.088679+0000","last_undegraded":"2024-08-30T23:59:21.088679+0000","last_fullsized":"2024-08-30T23:59:21.088679+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:08.049028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:08.049028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"1001'3968","reported_seq":6165,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:22.148394+0000","last_change":"2024-08-30T23:59:18.093510+0000","last_active":"2024-08-30T23:59:22.148394+0000","last_peered":"2024-08-30T23:59:22.148394+0000","last_clean":"2024-08-30T23:59:22.148394+0000","last_became_active":"2024-08-30T23:59:18.091017+0000","last_became_peered":"2024-08-30T23:59:18.091017+0000","last_unstale":"2024-08-30T23:59:22.148394+0000","last_undegraded":"2024-08-30T23:59:22.148394+0000","last_fullsized":"2024-08-30T23:59:22.148394+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1143","last_scrub_stamp":"2024-08-30T23:57:56.128615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:56.128615+0000","objects_scrubbed":635,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200671232,"num_objects":3062,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":168064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":66060288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1295,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.085419+0000","last_change":"2024-08-30T23:58:10.841604+0000","last_active":"2024-08-30T23:59:21.085419+0000","last_peered":"2024-08-30T23:59:21.085419+0000","last_clean":"2024-08-30T23:59:21.085419+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-30T23:59:21.085419+0000","last_undegraded":"2024-08-30T23:59:21.085419+0000","last_fullsized":"2024-08-30T23:59:21.085419+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:10.841542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:10.841542+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:36.684389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"1000'4924","reported_seq":5097,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.784039+0000","last_change":"2024-08-30T23:58:26.120565+0000","last_active":"2024-08-30T23:59:19.784039+0000","last_peered":"2024-08-30T23:59:19.784039+0000","last_clean":"2024-08-30T23:59:19.784039+0000","last_became_active":"2024-08-30T23:58:23.234078+0000","last_became_peered":"2024-08-30T23:58:23.234078+0000","last_unstale":"2024-08-30T23:59:19.784039+0000","last_undegraded":"2024-08-30T23:59:19.784039+0000","last_fullsized":"2024-08-30T23:59:19.784039+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'3422","last_scrub_stamp":"2024-08-30T23:58:26.120459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:26.120459+0000","objects_scrubbed":1763,"log_size":4924,"log_dups_size":0,"ondisk_log_size":4924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:27.599850+0000","scrub_duration":1088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203882496,"num_objects":3111,"num_object_clones":0,"num_object_copies":9333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1502,"num_write_kb":91200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":656,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.089006+0000","last_change":"2024-08-30T23:59:19.405781+0000","last_active":"2024-08-30T23:59:21.089006+0000","last_peered":"2024-08-30T23:59:21.089006+0000","last_clean":"2024-08-30T23:59:21.089006+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-30T23:59:21.089006+0000","last_undegraded":"2024-08-30T23:59:21.089006+0000","last_fullsized":"2024-08-30T23:59:21.089006+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:19.405731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:19.405731+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:34.875293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1001'5188","reported_seq":5331,"reported_epoch":1001,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:59:22.197751+0000","last_change":"2024-08-30T23:58:23.240351+0000","last_active":"2024-08-30T23:59:22.197751+0000","last_peered":"2024-08-30T23:59:22.197751+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.240351+0000","last_became_peered":"2024-08-30T23:58:23.240351+0000","last_unstale":"2024-08-30T23:59:22.197751+0000","last_undegraded":"2024-08-30T23:58:23.225188+0000","last_fullsized":"2024-08-30T23:58:23.224931+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":5188,"log_dups_size":0,"ondisk_log_size":5188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:59:22.438264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205783040,"num_objects":3140,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":211904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":["10","1"],"object_location_counts":[{"shards":"1,10","objects":3140}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":674,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.088898+0000","last_change":"2024-08-30T23:56:48.173739+0000","last_active":"2024-08-30T23:59:21.088898+0000","last_peered":"2024-08-30T23:59:21.088898+0000","last_clean":"2024-08-30T23:59:21.088898+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-30T23:59:21.088898+0000","last_undegraded":"2024-08-30T23:59:21.088898+0000","last_fullsized":"2024-08-30T23:59:21.088898+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:48.173668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:48.173668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"1000'4795","reported_seq":4937,"reported_epoch":1000,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:59:19.640302+0000","last_change":"2024-08-30T23:58:23.224228+0000","last_active":"2024-08-30T23:59:19.640302+0000","last_peered":"2024-08-30T23:59:19.640302+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.224228+0000","last_became_peered":"2024-08-30T23:58:23.224228+0000","last_unstale":"2024-08-30T23:59:19.640302+0000","last_undegraded":"2024-08-30T23:58:23.222413+0000","last_fullsized":"2024-08-30T23:58:23.221376+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:59:24.062318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197230592,"num_objects":3009,"num_object_clones":0,"num_object_copies":9027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":197856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":3009}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":317,"reported_epoch":1001,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:21.089538+0000","last_change":"2024-08-30T23:59:15.178934+0000","last_active":"2024-08-30T23:59:21.089538+0000","last_peered":"2024-08-30T23:59:21.089538+0000","last_clean":"2024-08-30T23:59:21.089538+0000","last_became_active":"2024-08-30T23:59:15.177663+0000","last_became_peered":"2024-08-30T23:59:15.177663+0000","last_unstale":"2024-08-30T23:59:21.089538+0000","last_undegraded":"2024-08-30T23:59:21.089538+0000","last_fullsized":"2024-08-30T23:59:21.089538+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:36.270718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:36.270718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8,6],"avail_no_missing":["11","6","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.c","version":"999'4843","reported_seq":5015,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-08-30T23:59:18.734750+0000","last_change":"2024-08-30T23:58:24.964811+0000","last_active":"2024-08-30T23:59:18.734750+0000","last_peered":"2024-08-30T23:59:18.734750+0000","last_clean":"2024-08-30T23:59:18.734750+0000","last_became_active":"2024-08-30T23:58:23.224459+0000","last_became_peered":"2024-08-30T23:58:23.224459+0000","last_unstale":"2024-08-30T23:59:18.734750+0000","last_undegraded":"2024-08-30T23:59:18.734750+0000","last_fullsized":"2024-08-30T23:59:18.734750+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"968'3325","last_scrub_stamp":"2024-08-30T23:58:24.964745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:24.964745+0000","objects_scrubbed":1690,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:25.808541+0000","scrub_duration":1112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201588736,"num_objects":3076,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1520,"num_write_kb":92992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":900,"reported_epoch":1001,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:21.085017+0000","last_change":"2024-08-30T23:59:16.123147+0000","last_active":"2024-08-30T23:59:21.085017+0000","last_peered":"2024-08-30T23:59:21.085017+0000","last_clean":"2024-08-30T23:59:21.085017+0000","last_became_active":"2024-08-30T23:59:15.180603+0000","last_became_peered":"2024-08-30T23:59:15.180603+0000","last_unstale":"2024-08-30T23:59:21.085017+0000","last_undegraded":"2024-08-30T23:59:21.085017+0000","last_fullsized":"2024-08-30T23:59:21.085017+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:16.123028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:16.123028+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:31.368544+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11,9],"avail_no_missing":["6","9","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1000'3844","reported_seq":4055,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.477579+0000","last_change":"2024-08-30T23:59:18.580672+0000","last_active":"2024-08-30T23:59:19.477579+0000","last_peered":"2024-08-30T23:59:19.477579+0000","last_clean":"2024-08-30T23:59:19.477579+0000","last_became_active":"2024-08-30T23:59:15.179731+0000","last_became_peered":"2024-08-30T23:59:15.179731+0000","last_unstale":"2024-08-30T23:59:19.477579+0000","last_undegraded":"2024-08-30T23:59:19.477579+0000","last_fullsized":"2024-08-30T23:59:19.477579+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'3824","last_scrub_stamp":"2024-08-30T23:59:18.580580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:18.580580+0000","objects_scrubbed":3161,"log_size":3844,"log_dups_size":0,"ondisk_log_size":3844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:31.837362+0000","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205651968,"num_objects":3138,"num_object_clones":0,"num_object_copies":9414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":463,"reported_epoch":1000,"state":"active+undersized","last_fresh":"2024-08-30T23:59:19.087550+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-30T23:59:19.087550+0000","last_peered":"2024-08-30T23:59:19.087550+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-30T23:59:19.087550+0000","last_undegraded":"2024-08-30T23:59:19.087550+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.18","version":"1001'3978","reported_seq":5378,"reported_epoch":1001,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:22.198047+0000","last_change":"2024-08-30T23:58:08.120423+0000","last_active":"2024-08-30T23:59:22.198047+0000","last_peered":"2024-08-30T23:59:22.198047+0000","last_clean":"2024-08-30T23:59:22.198047+0000","last_became_active":"2024-08-30T23:58:07.098389+0000","last_became_peered":"2024-08-30T23:58:07.098389+0000","last_unstale":"2024-08-30T23:59:22.198047+0000","last_undegraded":"2024-08-30T23:59:22.198047+0000","last_fullsized":"2024-08-30T23:59:22.198047+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"961'1604","last_scrub_stamp":"2024-08-30T23:58:08.120271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:08.120271+0000","objects_scrubbed":1049,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:27.133050+0000","scrub_duration":247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200605696,"num_objects":3061,"num_object_clones":0,"num_object_copies":9183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3061,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":140544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5,9],"avail_no_missing":["10","5","9"],"object_location_counts":[{"shards":"5,9,10","objects":3061}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":1095,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.104261+0000","last_change":"2024-08-30T23:58:04.552055+0000","last_active":"2024-08-30T23:59:19.104261+0000","last_peered":"2024-08-30T23:59:19.104261+0000","last_clean":"2024-08-30T23:59:19.104261+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-30T23:59:19.104261+0000","last_undegraded":"2024-08-30T23:59:19.104261+0000","last_fullsized":"2024-08-30T23:59:19.104261+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:04.552013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.552013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:20.924737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.19","version":"999'3785","reported_seq":5647,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-08-30T23:59:18.699643+0000","last_change":"2024-08-30T23:58:04.958546+0000","last_active":"2024-08-30T23:59:18.699643+0000","last_peered":"2024-08-30T23:59:18.699643+0000","last_clean":"2024-08-30T23:59:18.699643+0000","last_became_active":"2024-08-30T23:58:04.122431+0000","last_became_peered":"2024-08-30T23:58:04.122431+0000","last_unstale":"2024-08-30T23:59:18.699643+0000","last_undegraded":"2024-08-30T23:59:18.699643+0000","last_fullsized":"2024-08-30T23:59:18.699643+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1412","last_scrub_stamp":"2024-08-30T23:58:04.958468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.958468+0000","objects_scrubbed":844,"log_size":3785,"log_dups_size":0,"ondisk_log_size":3785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205062144,"num_objects":3129,"num_object_clones":0,"num_object_copies":9387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2373,"num_write_kb":149056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":475,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.087914+0000","last_change":"2024-08-30T23:57:35.093839+0000","last_active":"2024-08-30T23:59:19.087914+0000","last_peered":"2024-08-30T23:59:19.087914+0000","last_clean":"2024-08-30T23:59:19.087914+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-30T23:59:19.087914+0000","last_undegraded":"2024-08-30T23:59:19.087914+0000","last_fullsized":"2024-08-30T23:59:19.087914+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:35.093787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.093787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"999'4810","reported_seq":10001,"reported_epoch":999,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:18.737146+0000","last_change":"2024-08-30T23:59:13.562000+0000","last_active":"2024-08-30T23:59:18.737146+0000","last_peered":"2024-08-30T23:59:18.737146+0000","last_clean":"2024-08-30T23:59:18.737146+0000","last_became_active":"2024-08-30T23:58:55.439550+0000","last_became_peered":"2024-08-30T23:58:55.439550+0000","last_unstale":"2024-08-30T23:59:18.737146+0000","last_undegraded":"2024-08-30T23:59:18.737146+0000","last_fullsized":"2024-08-30T23:59:18.737146+0000","mapping_epoch":982,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":983,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'4765","last_scrub_stamp":"2024-08-30T23:59:13.561894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:13.561894+0000","objects_scrubbed":3127,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:43.313485+0000","scrub_duration":338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201981952,"num_objects":3082,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3082,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":3082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":24,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.089413+0000","last_change":"2024-08-30T23:59:14.207893+0000","last_active":"2024-08-30T23:59:21.089413+0000","last_peered":"2024-08-30T23:59:21.089413+0000","last_clean":"2024-08-30T23:59:21.089413+0000","last_became_active":"2024-08-30T23:59:14.207545+0000","last_became_peered":"2024-08-30T23:59:14.207545+0000","last_unstale":"2024-08-30T23:59:21.089413+0000","last_undegraded":"2024-08-30T23:59:21.089413+0000","last_fullsized":"2024-08-30T23:59:21.089413+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:14.423316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:14.423316+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.1b","version":"999'4959","reported_seq":10415,"reported_epoch":999,"state":"active+clean","last_fresh":"2024-08-30T23:59:18.752099+0000","last_change":"2024-08-30T23:59:18.736841+0000","last_active":"2024-08-30T23:59:18.752099+0000","last_peered":"2024-08-30T23:59:18.752099+0000","last_clean":"2024-08-30T23:59:18.752099+0000","last_became_active":"2024-08-30T23:59:18.147982+0000","last_became_peered":"2024-08-30T23:59:18.147982+0000","last_unstale":"2024-08-30T23:59:18.752099+0000","last_undegraded":"2024-08-30T23:59:18.752099+0000","last_fullsized":"2024-08-30T23:59:18.752099+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":5,"last_scrub":"999'4958","last_scrub_stamp":"2024-08-30T23:59:18.736685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:18.736685+0000","objects_scrubbed":3159,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:27.404766+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206766080,"num_objects":3155,"num_object_clones":0,"num_object_copies":9465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"1000'4910","reported_seq":10274,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.766839+0000","last_change":"2024-08-30T23:59:12.818297+0000","last_active":"2024-08-30T23:59:19.766839+0000","last_peered":"2024-08-30T23:59:19.766839+0000","last_clean":"2024-08-30T23:59:19.766839+0000","last_became_active":"2024-08-30T23:58:51.078720+0000","last_became_peered":"2024-08-30T23:58:51.078720+0000","last_unstale":"2024-08-30T23:59:19.766839+0000","last_undegraded":"2024-08-30T23:59:19.766839+0000","last_fullsized":"2024-08-30T23:59:19.766839+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'4811","last_scrub_stamp":"2024-08-30T23:59:12.818133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:12.818133+0000","objects_scrubbed":3206,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:27.523419+0000","scrub_duration":792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206045208,"num_objects":3145,"num_object_clones":0,"num_object_copies":9435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119,"num_write_kb":1793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,0],"acting":[5,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":206,"reported_epoch":1000,"state":"active+undersized","last_fresh":"2024-08-30T23:59:19.087966+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-30T23:59:19.087966+0000","last_peered":"2024-08-30T23:59:19.087966+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-30T23:59:19.087966+0000","last_undegraded":"2024-08-30T23:59:19.087966+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-30T23:59:21.993956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1e","version":"1000'5056","reported_seq":10547,"reported_epoch":1000,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:19.774271+0000","last_change":"2024-08-30T23:59:10.753672+0000","last_active":"2024-08-30T23:59:19.774271+0000","last_peered":"2024-08-30T23:59:19.774271+0000","last_clean":"2024-08-30T23:59:19.774271+0000","last_became_active":"2024-08-30T23:58:53.399195+0000","last_became_peered":"2024-08-30T23:58:53.399195+0000","last_unstale":"2024-08-30T23:59:19.774271+0000","last_undegraded":"2024-08-30T23:59:19.774271+0000","last_fullsized":"2024-08-30T23:59:19.774271+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":983,"parent":"0.0","parent_split_bits":5,"last_scrub":"992'4895","last_scrub_stamp":"2024-08-30T23:59:10.753461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:10.753461+0000","objects_scrubbed":3197,"log_size":5056,"log_dups_size":0,"ondisk_log_size":5056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:32.886648+0000","scrub_duration":721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209911808,"num_objects":3203,"num_object_clones":0,"num_object_copies":9609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3203,"num_objects_unfound":0,"num_objects_dirty":3203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,10],"avail_no_missing":["5","0","10"],"object_location_counts":[{"shards":"0,5,10","objects":3203}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":280,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.086938+0000","last_change":"2024-08-30T23:57:27.164513+0000","last_active":"2024-08-30T23:59:21.086938+0000","last_peered":"2024-08-30T23:59:21.086938+0000","last_clean":"2024-08-30T23:59:21.086938+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-30T23:59:21.086938+0000","last_undegraded":"2024-08-30T23:59:21.086938+0000","last_fullsized":"2024-08-30T23:59:21.086938+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:27.164462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:27.164462+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1f","version":"1001'5110","reported_seq":10417,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.867756+0000","last_change":"2024-08-30T23:59:10.188606+0000","last_active":"2024-08-30T23:59:21.867756+0000","last_peered":"2024-08-30T23:59:21.867756+0000","last_clean":"2024-08-30T23:59:21.867756+0000","last_became_active":"2024-08-30T23:58:53.399542+0000","last_became_peered":"2024-08-30T23:58:53.399542+0000","last_unstale":"2024-08-30T23:59:21.867756+0000","last_undegraded":"2024-08-30T23:59:21.867756+0000","last_fullsized":"2024-08-30T23:59:21.867756+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":5,"last_scrub":"992'4852","last_scrub_stamp":"2024-08-30T23:59:10.188366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:10.188366+0000","objects_scrubbed":3128,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:18.224953+0000","scrub_duration":735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198836224,"num_objects":3034,"num_object_clones":0,"num_object_copies":9102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":268,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":278,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.104747+0000","last_change":"2024-08-30T23:58:11.457897+0000","last_active":"2024-08-30T23:59:19.104747+0000","last_peered":"2024-08-30T23:59:19.104747+0000","last_clean":"2024-08-30T23:59:19.104747+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-30T23:59:19.104747+0000","last_undegraded":"2024-08-30T23:59:19.104747+0000","last_fullsized":"2024-08-30T23:59:19.104747+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:58:11.457846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:58:11.457846+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:28.511611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":491,"reported_epoch":1000,"state":"active+undersized+degraded","last_fresh":"2024-08-30T23:59:19.087333+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-30T23:59:19.087333+0000","last_peered":"2024-08-30T23:59:19.087333+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-30T23:59:19.087333+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.10","version":"1001'3906","reported_seq":5849,"reported_epoch":1001,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:21.754040+0000","last_change":"2024-08-30T23:58:05.325967+0000","last_active":"2024-08-30T23:59:21.754040+0000","last_peered":"2024-08-30T23:59:21.754040+0000","last_clean":"2024-08-30T23:59:21.754040+0000","last_became_active":"2024-08-30T23:58:04.118964+0000","last_became_peered":"2024-08-30T23:58:04.118964+0000","last_unstale":"2024-08-30T23:59:21.754040+0000","last_undegraded":"2024-08-30T23:59:21.754040+0000","last_fullsized":"2024-08-30T23:59:21.754040+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'1457","last_scrub_stamp":"2024-08-30T23:58:05.325770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:05.325770+0000","objects_scrubbed":883,"log_size":3906,"log_dups_size":0,"ondisk_log_size":3906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3037,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2460,"num_write_kb":147648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":3037}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":272,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.088565+0000","last_change":"2024-08-30T23:56:45.191296+0000","last_active":"2024-08-30T23:59:21.088565+0000","last_peered":"2024-08-30T23:59:21.088565+0000","last_clean":"2024-08-30T23:59:21.088565+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-30T23:59:21.088565+0000","last_undegraded":"2024-08-30T23:59:21.088565+0000","last_fullsized":"2024-08-30T23:59:21.088565+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:45.191245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:45.191245+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.11","version":"999'3756","reported_seq":3922,"reported_epoch":999,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:18.749858+0000","last_change":"2024-08-30T23:57:57.163576+0000","last_active":"2024-08-30T23:59:18.749858+0000","last_peered":"2024-08-30T23:59:18.749858+0000","last_clean":"2024-08-30T23:59:18.749858+0000","last_became_active":"2024-08-30T23:57:56.118862+0000","last_became_peered":"2024-08-30T23:57:56.118862+0000","last_unstale":"2024-08-30T23:59:18.749858+0000","last_undegraded":"2024-08-30T23:59:18.749858+0000","last_fullsized":"2024-08-30T23:59:18.749858+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"952'1150","last_scrub_stamp":"2024-08-30T23:57:57.163327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:57.163327+0000","objects_scrubbed":628,"log_size":3756,"log_dups_size":0,"ondisk_log_size":3756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:18.954084+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205258752,"num_objects":3132,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3132,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2610,"num_write_kb":163648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,9],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"3,6,9","objects":3132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":270,"reported_epoch":1001,"state":"active+undersized","last_fresh":"2024-08-30T23:59:21.095892+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-30T23:59:21.095892+0000","last_peered":"2024-08-30T23:59:21.095892+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-30T23:59:21.095892+0000","last_undegraded":"2024-08-30T23:59:21.095892+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:24.954264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"1000'3710","reported_seq":5710,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.765182+0000","last_change":"2024-08-30T23:58:04.315602+0000","last_active":"2024-08-30T23:59:19.765182+0000","last_peered":"2024-08-30T23:59:19.765182+0000","last_clean":"2024-08-30T23:59:19.765182+0000","last_became_active":"2024-08-30T23:58:04.118740+0000","last_became_peered":"2024-08-30T23:58:04.118740+0000","last_unstale":"2024-08-30T23:59:19.765182+0000","last_undegraded":"2024-08-30T23:59:19.765182+0000","last_fullsized":"2024-08-30T23:59:19.765182+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1428","last_scrub_stamp":"2024-08-30T23:58:04.315539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.315539+0000","objects_scrubbed":909,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199032832,"num_objects":3037,"num_object_clones":0,"num_object_copies":9111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":141120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":286,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.104292+0000","last_change":"2024-08-30T23:59:15.110844+0000","last_active":"2024-08-30T23:59:19.104292+0000","last_peered":"2024-08-30T23:59:19.104292+0000","last_clean":"2024-08-30T23:59:19.104292+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-30T23:59:19.104292+0000","last_undegraded":"2024-08-30T23:59:19.104292+0000","last_fullsized":"2024-08-30T23:59:19.104292+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:15.110786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:15.110786+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:20.974253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"1001'3829","reported_seq":7848,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.346259+0000","last_change":"2024-08-30T23:59:17.095185+0000","last_active":"2024-08-30T23:59:21.346259+0000","last_peered":"2024-08-30T23:59:21.346259+0000","last_clean":"2024-08-30T23:59:21.346259+0000","last_became_active":"2024-08-30T23:59:17.091165+0000","last_became_peered":"2024-08-30T23:59:17.091165+0000","last_unstale":"2024-08-30T23:59:21.346259+0000","last_undegraded":"2024-08-30T23:59:21.346259+0000","last_fullsized":"2024-08-30T23:59:21.346259+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'1406","last_scrub_stamp":"2024-08-30T23:58:03.245266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:03.245266+0000","objects_scrubbed":849,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:28.008565+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194772992,"num_objects":2972,"num_object_clones":0,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2425,"num_write_kb":145536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,5],"acting":[11,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":286,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:19.104309+0000","last_change":"2024-08-30T23:59:16.147143+0000","last_active":"2024-08-30T23:59:19.104309+0000","last_peered":"2024-08-30T23:59:19.104309+0000","last_clean":"2024-08-30T23:59:19.104309+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-30T23:59:19.104309+0000","last_undegraded":"2024-08-30T23:59:19.104309+0000","last_fullsized":"2024-08-30T23:59:19.104309+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:16.147096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:16.147096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:33.018431+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.14","version":"1001'3910","reported_seq":7566,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.889311+0000","last_change":"2024-08-30T23:59:18.152107+0000","last_active":"2024-08-30T23:59:21.889311+0000","last_peered":"2024-08-30T23:59:21.889311+0000","last_clean":"2024-08-30T23:59:21.889311+0000","last_became_active":"2024-08-30T23:59:18.148949+0000","last_became_peered":"2024-08-30T23:59:18.148949+0000","last_unstale":"2024-08-30T23:59:21.889311+0000","last_undegraded":"2024-08-30T23:59:21.889311+0000","last_fullsized":"2024-08-30T23:59:21.889311+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1388","last_scrub_stamp":"2024-08-30T23:58:04.503242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.503242+0000","objects_scrubbed":845,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:28.945696+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":9225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":152064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":57278464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":32,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.089538+0000","last_change":"2024-08-30T23:59:15.264826+0000","last_active":"2024-08-30T23:59:21.089538+0000","last_peered":"2024-08-30T23:59:21.089538+0000","last_clean":"2024-08-30T23:59:21.089538+0000","last_became_active":"2024-08-30T23:59:14.206483+0000","last_became_peered":"2024-08-30T23:59:14.206483+0000","last_unstale":"2024-08-30T23:59:21.089538+0000","last_undegraded":"2024-08-30T23:59:21.089538+0000","last_fullsized":"2024-08-30T23:59:21.089538+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:15.264564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:15.264564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:22.020381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.15","version":"1000'3901","reported_seq":6054,"reported_epoch":1000,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:20.762794+0000","last_change":"2024-08-30T23:59:20.137229+0000","last_active":"2024-08-30T23:59:20.762794+0000","last_peered":"2024-08-30T23:59:20.762794+0000","last_clean":"2024-08-30T23:59:20.762794+0000","last_became_active":"2024-08-30T23:58:03.116440+0000","last_became_peered":"2024-08-30T23:58:03.116440+0000","last_unstale":"2024-08-30T23:59:20.762794+0000","last_undegraded":"2024-08-30T23:59:20.762794+0000","last_fullsized":"2024-08-30T23:59:20.762794+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'3865","last_scrub_stamp":"2024-08-30T23:59:20.136835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:20.136835+0000","objects_scrubbed":3160,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:39.001209+0000","scrub_duration":1007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203292672,"num_objects":3102,"num_object_clones":0,"num_object_copies":9306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3102,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2460,"num_write_kb":148928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4,5],"avail_no_missing":["8","4","5"],"object_location_counts":[{"shards":"4,5,8","objects":3102}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":254,"reported_epoch":1001,"state":"active+clean+remapped","last_fresh":"2024-08-30T23:59:21.089405+0000","last_change":"2024-08-30T23:59:15.183405+0000","last_active":"2024-08-30T23:59:21.089405+0000","last_peered":"2024-08-30T23:59:21.089405+0000","last_clean":"2024-08-30T23:59:21.089405+0000","last_became_active":"2024-08-30T23:59:15.181247+0000","last_became_peered":"2024-08-30T23:59:15.181247+0000","last_unstale":"2024-08-30T23:59:21.089405+0000","last_undegraded":"2024-08-30T23:59:21.089405+0000","last_fullsized":"2024-08-30T23:59:21.089405+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:13.150214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:13.150214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:37.026750+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8,10],"avail_no_missing":["11","8","10"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.16","version":"1001'3928","reported_seq":8982,"reported_epoch":1001,"state":"active+clean","last_fresh":"2024-08-30T23:59:21.336014+0000","last_change":"2024-08-30T23:59:19.095535+0000","last_active":"2024-08-30T23:59:21.336014+0000","last_peered":"2024-08-30T23:59:21.336014+0000","last_clean":"2024-08-30T23:59:21.336014+0000","last_became_active":"2024-08-30T23:59:19.090116+0000","last_became_peered":"2024-08-30T23:59:19.090116+0000","last_unstale":"2024-08-30T23:59:21.336014+0000","last_undegraded":"2024-08-30T23:59:21.336014+0000","last_fullsized":"2024-08-30T23:59:21.336014+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":1000,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'1469","last_scrub_stamp":"2024-08-30T23:58:06.471505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:06.471505+0000","objects_scrubbed":927,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:24.771590+0000","scrub_duration":392,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":9225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2460,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":82116608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.17","version":"1000'3834","reported_seq":6099,"reported_epoch":1000,"state":"active+clean","last_fresh":"2024-08-30T23:59:20.771670+0000","last_change":"2024-08-30T23:58:06.151958+0000","last_active":"2024-08-30T23:59:20.771670+0000","last_peered":"2024-08-30T23:59:20.771670+0000","last_clean":"2024-08-30T23:59:20.771670+0000","last_became_active":"2024-08-30T23:58:05.121618+0000","last_became_peered":"2024-08-30T23:58:05.121618+0000","last_unstale":"2024-08-30T23:59:20.771670+0000","last_undegraded":"2024-08-30T23:59:20.771670+0000","last_fullsized":"2024-08-30T23:59:20.771670+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"960'1460","last_scrub_stamp":"2024-08-30T23:58:06.151812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:06.151812+0000","objects_scrubbed":907,"log_size":3834,"log_dups_size":0,"ondisk_log_size":3834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-30T23:59:24.403703+0000","scrub_duration":583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195166208,"num_objects":2978,"num_object_clones":0,"num_object_copies":8934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2381,"num_write_kb":142464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":36,"stat_sum":{"num_bytes":6454738968,"num_objects":98492,"num_object_clones":0,"num_object_copies":295476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15310,"num_objects_misplaced":20770,"num_objects_unfound":0,"num_objects_dirty":98492,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":65142,"num_write_kb":3944417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8695,"num_bytes_recovered":569835520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19081736192,"data_stored":19081723976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145060,"ondisk_log_size":145060,"up":96,"acting":103,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":84,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187875,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2151108,"kb_used_data":1438252,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91594044,"statfs":{"total":95995035648,"available":93792301056,"internally_reserved":0,"allocated":1472770048,"data_stored":1471822751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":729938372},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449516,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2748096,"kb_used_data":2035244,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":90997056,"statfs":{"total":95995035648,"available":93180985344,"internally_reserved":0,"allocated":2084089856,"data_stored":2083142559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":729938368},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711144,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2846072,"kb_used_data":2006884,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":90899080,"statfs":{"total":95995035648,"available":93080657920,"internally_reserved":0,"allocated":2055049216,"data_stored":2054101904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":859306432},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972787,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1426744,"kb_used_data":890208,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92318408,"statfs":{"total":95995035648,"available":94534049792,"internally_reserved":0,"allocated":911572992,"data_stored":910629752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528765,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489728,"kb_used_data":971244,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92255424,"statfs":{"total":95995035648,"available":94469554176,"internally_reserved":0,"allocated":994553856,"data_stored":993606559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775268,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4008120,"kb_used_data":3561568,"kb_used_omap":5,"kb_used_meta":446522,"kb_avail":89737032,"statfs":{"total":95995035648,"available":91890720768,"internally_reserved":0,"allocated":3647045632,"data_stored":3646098767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":457238602},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953928,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1708468,"kb_used_data":977568,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92036684,"statfs":{"total":95995035648,"available":94245564416,"internally_reserved":0,"allocated":1001029632,"data_stored":1000086392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150153,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2947628,"kb_used_data":1977940,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":90797524,"statfs":{"total":95995035648,"available":92976664576,"internally_reserved":0,"allocated":2025410560,"data_stored":2024471319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":992934343},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411792,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1144324,"kb_used_data":481460,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92600828,"statfs":{"total":95995035648,"available":94823247872,"internally_reserved":0,"allocated":493015040,"data_stored":492064191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830608,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3473280,"kb_used_data":2840044,"kb_used_omap":87,"kb_used_meta":633128,"kb_avail":90271872,"statfs":{"total":95995035648,"available":92438396928,"internally_reserved":0,"allocated":2908205056,"data_stored":2907257759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":648323231},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738842,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2468144,"kb_used_data":1602584,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91277008,"statfs":{"total":95995035648,"available":93467656192,"internally_reserved":0,"allocated":1641046016,"data_stored":1640102703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1635323904,"data_stored":1635319832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2885419008,"data_stored":2885419008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":486998016,"data_stored":486998016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2016608256,"data_stored":2016608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":989986816,"data_stored":989986816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3604353024,"data_stored":3604348952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":988020736,"data_stored":988020736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":899481600,"data_stored":899481600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2050887680,"data_stored":2050883608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2056716288,"data_stored":2056716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1467940864,"data_stored":1467940864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-30T23:59:27.681 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-08-30T23:59:27.698 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:27.704 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:27.738 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.740 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:27.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:27.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:27.867 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:27.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:27.970 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:28.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:28.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:28.134 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:28.236 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:28.263 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:28.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:28 smithi106 ceph-mon[16603]: 9.1c scrub starts 2024-08-30T23:59:28.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:28.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:28.492 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:28.514 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:28.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:28 smithi136 ceph-mon[20207]: 9.1c scrub starts 2024-08-30T23:59:28.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:28 smithi173 ceph-mon[20554]: 9.1c scrub starts 2024-08-30T23:59:28.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:28.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:28.740 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:59:28.745 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:59:28.764 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:28.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:28.894 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:28.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:28.990 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:59:28.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:29.022 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:29.032 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:29.124 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:29.140 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-30T23:59:29.160 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:29.262 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:29.285 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:29 smithi106 ceph-mon[16603]: pgmap v2211: 67 pgs: 1 creating+peering, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 14359/277065 objects degraded (5.183%); 18835/277065 objects misplaced (6.798%); 20 MiB/s, 313 objects/s recovering 2024-08-30T23:59:29.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:29 smithi106 ceph-mon[16603]: 9.1c scrub ok 2024-08-30T23:59:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:29 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-30T23:59:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:29 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-30T23:59:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2011693125' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:59:29.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:29 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:59:29.387 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.410 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:29.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:29.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.534 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:29.534 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:29 smithi136 ceph-mon[20207]: pgmap v2211: 67 pgs: 1 creating+peering, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 14359/277065 objects degraded (5.183%); 18835/277065 objects misplaced (6.798%); 20 MiB/s, 313 objects/s recovering 2024-08-30T23:59:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:29 smithi136 ceph-mon[20207]: 9.1c scrub ok 2024-08-30T23:59:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:29 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-30T23:59:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:29 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-30T23:59:29.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2011693125' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:59:29.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:29 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:59:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:29 smithi173 ceph-mon[20554]: pgmap v2211: 67 pgs: 1 creating+peering, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s; 14359/277065 objects degraded (5.183%); 18835/277065 objects misplaced (6.798%); 20 MiB/s, 313 objects/s recovering 2024-08-30T23:59:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:29 smithi173 ceph-mon[20554]: 9.1c scrub ok 2024-08-30T23:59:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:29 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-30T23:59:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:29 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-30T23:59:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2011693125' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:59:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:29 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-30T23:59:29.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-30T23:59:29.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:29.660 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:29.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:29.787 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:29.889 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:29.916 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:29.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:30.019 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:30.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:30.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:30.171 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:30.273 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:30.298 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:30.376 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:30.394 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:30.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:30.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:30.505 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:30 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:30.505 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:30 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:59:30.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:30 smithi106 ceph-mon[16603]: osdmap e1004: 12 total, 11 up, 12 in 2024-08-30T23:59:30.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:30.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:30 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:30 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:59:30.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:30 smithi136 ceph-mon[20207]: osdmap e1004: 12 total, 11 up, 12 in 2024-08-30T23:59:30.553 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:30 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:30 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-30T23:59:30.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:30 smithi173 ceph-mon[20554]: osdmap e1004: 12 total, 11 up, 12 in 2024-08-30T23:59:30.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:30.680 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:30.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:30.810 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.811 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:30.911 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:30.940 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:30.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:31.042 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:31.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:31.087 INFO:teuthology.orchestra.run.smithi106.stderr:set pool 1 pgp_num to 31 2024-08-30T23:59:31.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:31.206 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:31.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:31.334 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:31 smithi106 ceph-mon[16603]: pgmap v2213: 67 pgs: 1 creating+peering, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.46k op/s; 14015/269271 objects degraded (5.205%); 18238/269271 objects misplaced (6.773%); 12 MiB/s, 196 objects/s recovering 2024-08-30T23:59:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1217305337' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-30T23:59:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 14015/269271 objects degraded (5.205%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1217305337' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-30T23:59:31.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:31 smithi106 ceph-mon[16603]: osdmap e1005: 12 total, 11 up, 12 in 2024-08-30T23:59:31.436 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:31.466 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:31 smithi136 ceph-mon[20207]: pgmap v2213: 67 pgs: 1 creating+peering, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.46k op/s; 14015/269271 objects degraded (5.205%); 18238/269271 objects misplaced (6.773%); 12 MiB/s, 196 objects/s recovering 2024-08-30T23:59:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1217305337' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-30T23:59:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 14015/269271 objects degraded (5.205%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1217305337' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-30T23:59:31.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:31 smithi136 ceph-mon[20207]: osdmap e1005: 12 total, 11 up, 12 in 2024-08-30T23:59:31.562 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:59:31.563 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:59:31.568 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:31 smithi173 ceph-mon[20554]: pgmap v2213: 67 pgs: 1 creating+peering, 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.46k op/s; 14015/269271 objects degraded (5.205%); 18238/269271 objects misplaced (6.773%); 12 MiB/s, 196 objects/s recovering 2024-08-30T23:59:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1217305337' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-30T23:59:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 14015/269271 objects degraded (5.205%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1217305337' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-30T23:59:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:31 smithi173 ceph-mon[20554]: osdmap e1005: 12 total, 11 up, 12 in 2024-08-30T23:59:31.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:31.700 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:31.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:31.827 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:31.851 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:31.954 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:31.978 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:31.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:32.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:32.104 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:32.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:32.231 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:32.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:32 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:32.312 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:32 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:32.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:32.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:32.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:32.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:32 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:32.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:32 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:32 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:32 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:32.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:32.609 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:32.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:32.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:32.837 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:32.862 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:32.964 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:32.988 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:32.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:33.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:33.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:33.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-30T23:59:33.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.235 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:33.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:33 smithi173 ceph-mon[20554]: pgmap v2215: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 13829/264597 objects degraded (5.226%); 17863/264597 objects misplaced (6.751%) 2024-08-30T23:59:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:33 smithi173 ceph-mon[20554]: osdmap e1006: 12 total, 11 up, 12 in 2024-08-30T23:59:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:33 smithi173 ceph-mon[20554]: 9.5 scrub starts 2024-08-30T23:59:33.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/494512280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:33.336 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:33.362 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:33.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:33 smithi136 ceph-mon[20207]: pgmap v2215: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 13829/264597 objects degraded (5.226%); 17863/264597 objects misplaced (6.751%) 2024-08-30T23:59:33.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:33 smithi136 ceph-mon[20207]: osdmap e1006: 12 total, 11 up, 12 in 2024-08-30T23:59:33.420 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:33 smithi136 ceph-mon[20207]: 9.5 scrub starts 2024-08-30T23:59:33.421 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/494512280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:33.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:33.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:33.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:33.532 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:33.532 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:33 smithi106 ceph-mon[16603]: pgmap v2215: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 489 op/s; 13829/264597 objects degraded (5.226%); 17863/264597 objects misplaced (6.751%) 2024-08-30T23:59:33.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:33 smithi106 ceph-mon[16603]: osdmap e1006: 12 total, 11 up, 12 in 2024-08-30T23:59:33.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:33 smithi106 ceph-mon[16603]: 9.5 scrub starts 2024-08-30T23:59:33.533 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/494512280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:33.561 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:33.561 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:33.581 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-30T23:59:33.592 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:33.592 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:33.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:33.603 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:33.622 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:33.622 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:33.626 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:33.646 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:33.646 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:33.667 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:33.667 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:33.688 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:33.689 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:33.689 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:33.709 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:33.710 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:33.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:33.731 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:33.732 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:33.751 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:33.755 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:33.755 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:33.783 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:33.784 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:33.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:33.884 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:33.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:33.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:34.008 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:34.111 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:34.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:34.240 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:34.271 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:34.373 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:34.404 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:34.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:34.537 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:34.639 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:34.662 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:34.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:34.764 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:34.786 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:34.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:34.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:34.909 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:34.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:59:35.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:59:35.005 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:35.011 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:35.034 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:35.136 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:35.162 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:35.166 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-30T23:59:35.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:35 smithi106 ceph-mon[16603]: pgmap v2217: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 13410/259509 objects degraded (5.167%); 17647/259509 objects misplaced (6.800%) 2024-08-30T23:59:35.262 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/866896441' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:35.264 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:35.285 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:35.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:35.415 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:35.442 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:35 smithi173 ceph-mon[20554]: pgmap v2217: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 13410/259509 objects degraded (5.167%); 17647/259509 objects misplaced (6.800%) 2024-08-30T23:59:35.442 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/866896441' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:35.518 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:35 smithi136 ceph-mon[20207]: pgmap v2217: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 13410/259509 objects degraded (5.167%); 17647/259509 objects misplaced (6.800%) 2024-08-30T23:59:35.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/866896441' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:35.544 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:35.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:35.670 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:35.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:35.797 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:35.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:35.922 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:35.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:36.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:36.048 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:36.138 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:59:36.138 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:59:36.138 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-30T23:59:36.138 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:59:36.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:36.172 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:36.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:36.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:36.399 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:36.424 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:36.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:36 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:36.441 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/866896441' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:36.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:36 smithi106 ceph-mon[16603]: osdmap e1007: 12 total, 11 up, 12 in 2024-08-30T23:59:36.442 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 13410/259509 objects degraded (5.167%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:36.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:36.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:36 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/866896441' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:36 smithi136 ceph-mon[20207]: osdmap e1007: 12 total, 11 up, 12 in 2024-08-30T23:59:36.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 13410/259509 objects degraded (5.167%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:36.552 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:36.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:36.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:36 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/866896441' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:36 smithi173 ceph-mon[20554]: osdmap e1007: 12 total, 11 up, 12 in 2024-08-30T23:59:36.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 13410/259509 objects degraded (5.167%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:36.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:36.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:36.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:36.805 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:36.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:36.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-30T23:59:36.935 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:36.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:37.002 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:59:37.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:37.062 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:37.153 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-30T23:59:37.153 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-30T23:59:37.153 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-08-30T23:59:37.154 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-08-30T23:59:37.164 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:37.171 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:37 smithi106 ceph-mon[16603]: pgmap v2219: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 13017/247764 objects degraded (5.254%); 16718/247764 objects misplaced (6.748%) 2024-08-30T23:59:37.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:37.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:37.290 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:37.292 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:37.339 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:37.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:37.464 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:37.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:37 smithi136 ceph-mon[20207]: pgmap v2219: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 13017/247764 objects degraded (5.254%); 16718/247764 objects misplaced (6.748%) 2024-08-30T23:59:37.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:37.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:37 smithi173 ceph-mon[20554]: pgmap v2219: 67 pgs: 1 active+recovering+degraded+remapped, 10 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 615 op/s; 13017/247764 objects degraded (5.254%); 16718/247764 objects misplaced (6.748%) 2024-08-30T23:59:37.592 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:37.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:37.726 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:37.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:37.856 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:37.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-30T23:59:37.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:37.984 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:37.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:38.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:38.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:38.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:38.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:38.339 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:38.369 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:38.436 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:59:38.437 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:59:38.437 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:59:38.437 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:59:38.437 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:59:38.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:38.493 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:38.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:38.617 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:38.664 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:38.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:38.744 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:38.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:38.872 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:38.974 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:38.998 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:38.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:39.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:39.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:39.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:39.247 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:39.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-30T23:59:39.295 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:39.300 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:39.349 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:39.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:39.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:39 smithi106 ceph-mon[16603]: pgmap v2220: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s; 10033/240483 objects degraded (4.172%); 18775/240483 objects misplaced (7.807%) 2024-08-30T23:59:39.466 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:39 smithi106 ceph-mon[16603]: 9.5 scrub starts 2024-08-30T23:59:39.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:39.500 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:39 smithi136 ceph-mon[20207]: pgmap v2220: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s; 10033/240483 objects degraded (4.172%); 18775/240483 objects misplaced (7.807%) 2024-08-30T23:59:39.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:39 smithi136 ceph-mon[20207]: 9.5 scrub starts 2024-08-30T23:59:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:39 smithi173 ceph-mon[20554]: pgmap v2220: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s; 10033/240483 objects degraded (4.172%); 18775/240483 objects misplaced (7.807%) 2024-08-30T23:59:39.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:39 smithi173 ceph-mon[20554]: 9.5 scrub starts 2024-08-30T23:59:39.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:39.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:39.726 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:39.756 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:39.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:39.886 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:39.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:39.988 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:40.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:40.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:40.157 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:40.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:40.287 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.288 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:40.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:40.413 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:40.515 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:40.540 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:40.642 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:40.667 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:40.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:40.791 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:40.894 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:40.921 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:40.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:41.023 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:41.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:41.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:41.170 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:41.192 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-30T23:59:41.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.293 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:41.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:41.395 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:41.424 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:41.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:41 smithi136 ceph-mon[20207]: pgmap v2221: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 9638/231375 objects degraded (4.166%); 18068/231375 objects misplaced (7.809%); 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:59:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/391122448' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:59:41.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9638/231375 objects degraded (4.166%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:41.552 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:41.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:41 smithi173 ceph-mon[20554]: pgmap v2221: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 9638/231375 objects degraded (4.166%); 18068/231375 objects misplaced (7.809%); 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:59:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/391122448' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:59:41.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9638/231375 objects degraded (4.166%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:41.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:41 smithi106 ceph-mon[16603]: pgmap v2221: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 9638/231375 objects degraded (4.166%); 18068/231375 objects misplaced (7.809%); 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:59:41.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/391122448' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-30T23:59:41.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9638/231375 objects degraded (4.166%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:41.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:41.678 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:41.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:41.804 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:41.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.805 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:41.906 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:41.934 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:41.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:42.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:42.064 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:42.167 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.194 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:42.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:42.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:42.320 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:42.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.453 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:42.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:42 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/391122448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:59:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:42 smithi136 ceph-mon[20207]: osdmap e1008: 12 total, 11 up, 12 in 2024-08-30T23:59:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:42 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-30T23:59:42.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:42 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-30T23:59:42.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:42 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/391122448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:59:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:42 smithi173 ceph-mon[20554]: osdmap e1008: 12 total, 11 up, 12 in 2024-08-30T23:59:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:42 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-30T23:59:42.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:42 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-30T23:59:42.583 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:42 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-30T23:59:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/391122448' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-30T23:59:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:42 smithi106 ceph-mon[16603]: osdmap e1008: 12 total, 11 up, 12 in 2024-08-30T23:59:42.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:42 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-30T23:59:42.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:42 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-30T23:59:42.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:42.706 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:42.808 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:42.837 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:42.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:42.964 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:42.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:43.068 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:43.095 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:43.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:43.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:43.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:43.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:43.459 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:43.483 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:43 smithi136 ceph-mon[20207]: pgmap v2223: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 9638/226329 objects degraded (4.258%); 17457/226329 objects misplaced (7.713%); 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:59:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:43 smithi136 ceph-mon[20207]: 9.4 scrub starts 2024-08-30T23:59:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:43 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-30T23:59:43.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:43 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-30T23:59:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:43 smithi173 ceph-mon[20554]: pgmap v2223: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 9638/226329 objects degraded (4.258%); 17457/226329 objects misplaced (7.713%); 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:59:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:43 smithi173 ceph-mon[20554]: 9.4 scrub starts 2024-08-30T23:59:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:43 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-30T23:59:43.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:43 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-30T23:59:43.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:43 smithi106 ceph-mon[16603]: pgmap v2223: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.38k op/s; 9638/226329 objects degraded (4.258%); 17457/226329 objects misplaced (7.713%); 1.3 MiB/s, 20 objects/s recovering 2024-08-30T23:59:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:43 smithi106 ceph-mon[16603]: 9.4 scrub starts 2024-08-30T23:59:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:43 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-30T23:59:43.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:43 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-30T23:59:43.610 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:43.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:43.739 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:43.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:43.865 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:43.967 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:43.991 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:43.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:44.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:44.118 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:44.199 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:44.199 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:44.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:44.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:44.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:44.224 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:44.224 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:44.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:44.253 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:44.253 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:44.282 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:44.282 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:44.283 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:44 smithi106 ceph-mon[16603]: 9.4 scrub ok 2024-08-30T23:59:44.283 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:44 smithi106 ceph-mon[16603]: 9.16 scrub starts 2024-08-30T23:59:44.283 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:44 smithi106 ceph-mon[16603]: 9.14 scrub starts 2024-08-30T23:59:44.283 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:44 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-30T23:59:44.283 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:44 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-30T23:59:44.283 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2503268472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:44.308 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:44.308 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:44.334 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:44.335 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:44.335 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:44.353 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:44.366 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:44.366 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:44.382 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:44.397 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:44.397 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:44.426 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:44.426 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:59:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:59:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:59:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:59:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:59:44.450 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:59:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:59:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:59:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:59:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:59:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:59:44.451 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:59:44.455 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:44.456 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:44.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:44.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:44 smithi136 ceph-mon[20207]: 9.4 scrub ok 2024-08-30T23:59:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:44 smithi136 ceph-mon[20207]: 9.16 scrub starts 2024-08-30T23:59:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:44 smithi136 ceph-mon[20207]: 9.14 scrub starts 2024-08-30T23:59:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:44 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-30T23:59:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:44 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-30T23:59:44.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2503268472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:44 smithi173 ceph-mon[20554]: 9.4 scrub ok 2024-08-30T23:59:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:44 smithi173 ceph-mon[20554]: 9.16 scrub starts 2024-08-30T23:59:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:44 smithi173 ceph-mon[20554]: 9.14 scrub starts 2024-08-30T23:59:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:44 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-30T23:59:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:44 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-30T23:59:44.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2503268472' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:44.615 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:44.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:44.744 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:44.769 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:44.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:44.894 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:44.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:44.996 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:45.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:45.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:45.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:59:45.200 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:59:45.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-08-30T23:59:45.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:45.280 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:45.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:45.408 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:45.511 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:45.537 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.538 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: 9.14 scrub ok 2024-08-30T23:59:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: pgmap v2224: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 9223/221334 objects degraded (4.167%); 17221/221334 objects misplaced (7.781%); 1.2 MiB/s, 18 objects/s recovering 2024-08-30T23:59:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: 9.16 scrub ok 2024-08-30T23:59:45.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: 9.5 scrub starts 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: 9.9 scrub starts 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=1183579053,config=1460075751,health=2648089657,kv=2008464973}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891734724}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=170940677}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302602295}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590056036}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3650991838}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322174084}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413251441}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731532496}) 2024-08-30T23:59:45.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=890486135}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986703656}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1275454947}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899178960}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958715535}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1180598078}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318449992}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642402415}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2591672926}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124917928}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559292621}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566701671}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305710582}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916664800}) 2024-08-30T23:59:45.542 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745457311}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4107539271}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3079561798}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3154232169}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550793657}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=264215266}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339410410}) 2024-08-30T23:59:45.543 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025046617}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2257778205}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2781037875}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021162032}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003760109}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86796215}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320259494}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965432864}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236558948}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=457124208}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705479055}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2319449769}) 2024-08-30T23:59:45.544 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4153512443}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3373317710}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=414650273}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2254889311}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690726402}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146643181}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397036762}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980073962}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1052321575}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21256064}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2664939278}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130485706}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500046306}) 2024-08-30T23:59:45.545 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060032921}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2964935946}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717755115}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629227749}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3077091542}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714239953}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807299214}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3166902371}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586699476}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4280849618}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229475536}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3399535876}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095206569}) 2024-08-30T23:59:45.546 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3348581400}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232603330}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500141056}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795409345}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3270547158}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576957771}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=753148364}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2324115800}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700476579}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636626501}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1842193342}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2583857566}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643860804}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=3217913395,mdsmap=425822562,mgr=3725793954}) 2024-08-30T23:59:45.547 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=78} crc {mgr=3093430750,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=1157026795,mon_config_key=3383745689}) 2024-08-30T23:59:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3142339687}) 2024-08-30T23:59:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2038074896}) 2024-08-30T23:59:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=16,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=3715440649,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=985899820,osdmap=2558274691}) 2024-08-30T23:59:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1530718627}) 2024-08-30T23:59:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=287645742}) 2024-08-30T23:59:45.548 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:45 smithi136 ceph-mon[20207]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=2927988781}) 2024-08-30T23:59:45.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: 9.14 scrub ok 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: pgmap v2224: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 9223/221334 objects degraded (4.167%); 17221/221334 objects misplaced (7.781%); 1.2 MiB/s, 18 objects/s recovering 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: 9.16 scrub ok 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: 9.5 scrub starts 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: 9.9 scrub starts 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=1183579053,config=1460075751,health=2648089657,kv=2008464973}) 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891734724}) 2024-08-30T23:59:45.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=170940677}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302602295}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590056036}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3650991838}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322174084}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413251441}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731532496}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=890486135}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986703656}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1275454947}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899178960}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958715535}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1180598078}) 2024-08-30T23:59:45.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318449992}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642402415}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2591672926}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124917928}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559292621}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566701671}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305710582}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916664800}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745457311}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4107539271}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3079561798}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3154232169}) 2024-08-30T23:59:45.577 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550793657}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=264215266}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339410410}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025046617}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2257778205}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2781037875}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021162032}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003760109}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86796215}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320259494}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965432864}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236558948}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=457124208}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705479055}) 2024-08-30T23:59:45.578 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2319449769}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4153512443}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3373317710}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=414650273}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2254889311}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690726402}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146643181}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397036762}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980073962}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1052321575}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21256064}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2664939278}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130485706}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500046306}) 2024-08-30T23:59:45.579 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060032921}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2964935946}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717755115}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629227749}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3077091542}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714239953}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807299214}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3166902371}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586699476}) 2024-08-30T23:59:45.580 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4280849618}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229475536}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3399535876}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095206569}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3348581400}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232603330}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500141056}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795409345}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3270547158}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576957771}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=753148364}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2324115800}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700476579}) 2024-08-30T23:59:45.581 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636626501}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1842193342}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2583857566}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643860804}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=3217913395,mdsmap=425822562,mgr=3725793954}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=78} crc {mgr=3093430750,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=1157026795,mon_config_key=3383745689}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3142339687}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2038074896}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=16,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=3715440649,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=985899820,osdmap=2558274691}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1530718627}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=287645742}) 2024-08-30T23:59:45.582 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:45 smithi173 ceph-mon[20554]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=2927988781}) 2024-08-30T23:59:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: 9.14 scrub ok 2024-08-30T23:59:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: pgmap v2224: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 9223/221334 objects degraded (4.167%); 17221/221334 objects misplaced (7.781%); 1.2 MiB/s, 18 objects/s recovering 2024-08-30T23:59:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: 9.16 scrub ok 2024-08-30T23:59:45.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: 9.5 scrub starts 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: 9.9 scrub starts 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=1183579053,config=1460075751,health=2648089657,kv=2008464973}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891734724}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=170940677}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302602295}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1590056036}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3650991838}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3322174084}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=413251441}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731532496}) 2024-08-30T23:59:45.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=890486135}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986703656}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1275454947}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899178960}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=958715535}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1180598078}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318449992}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2642402415}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2591672926}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1124917928}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2559292621}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566701671}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305710582}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916664800}) 2024-08-30T23:59:45.610 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745457311}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4107539271}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3079561798}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3154232169}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3550793657}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=264215266}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339410410}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1025046617}) 2024-08-30T23:59:45.611 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2257778205}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2781037875}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021162032}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1003760109}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=86796215}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1320259494}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965432864}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4236558948}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=457124208}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705479055}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2319449769}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4153512443}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3373317710}) 2024-08-30T23:59:45.612 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=414650273}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2254889311}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=690726402}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2146643181}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397036762}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2980073962}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1052321575}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21256064}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2664939278}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=130485706}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500046306}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2060032921}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2964935946}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1717755115}) 2024-08-30T23:59:45.613 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3629227749}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3077091542}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714239953}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=807299214}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3166902371}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586699476}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4280849618}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229475536}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3399535876}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1095206569}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3348581400}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=232603330}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500141056}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2795409345}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3270547158}) 2024-08-30T23:59:45.614 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3576957771}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=753148364}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2324115800}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2700476579}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=636626501}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1842193342}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2583857566}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643860804}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=3217913395,mdsmap=425822562,mgr=3725793954}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=78} crc {mgr=3093430750,mgr_command_descs=899306875,mgr_metadata=319223235,mgrstat=1157026795,mon_config_key=3383745689}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3142339687}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2038074896}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=16,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=66} crc {mon_config_key=3715440649,monmap=1193225519,osd_metadata=531677360,osd_pg_creating=985899820,osdmap=2558274691}) 2024-08-30T23:59:45.615 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1530718627}) 2024-08-30T23:59:45.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=287645742}) 2024-08-30T23:59:45.616 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:45 smithi106 ceph-mon[16603]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=96} crc {osdmap=2927988781}) 2024-08-30T23:59:45.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:45.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:45.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:45.798 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:45.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:45.934 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:45.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:46.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:46.060 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:46.162 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:46.190 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:46.293 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:46.317 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:46.419 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:46.443 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:46.468 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: 9.5 scrub ok 2024-08-30T23:59:46.468 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: 9.9 scrub ok 2024-08-30T23:59:46.468 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: 1.16 scrub starts 2024-08-30T23:59:46.468 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: 1.16 scrub ok 2024-08-30T23:59:46.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: 9.3 scrub starts 2024-08-30T23:59:46.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: 9.3 scrub ok 2024-08-30T23:59:46.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 9223/221334 objects degraded (4.167%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:46.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:46 smithi106 ceph-mon[16603]: osdmap e1009: 12 total, 11 up, 12 in 2024-08-30T23:59:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:59:46.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: 9.5 scrub ok 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: 9.9 scrub ok 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: 1.16 scrub starts 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: 1.16 scrub ok 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: 9.3 scrub starts 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: 9.3 scrub ok 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 9223/221334 objects degraded (4.167%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:46.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:46 smithi136 ceph-mon[20207]: osdmap e1009: 12 total, 11 up, 12 in 2024-08-30T23:59:46.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:46.569 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: 9.5 scrub ok 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: 9.9 scrub ok 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: 1.16 scrub starts 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: 1.16 scrub ok 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: 9.3 scrub starts 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: 9.3 scrub ok 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 9223/221334 objects degraded (4.167%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:46.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:46 smithi173 ceph-mon[20554]: osdmap e1009: 12 total, 11 up, 12 in 2024-08-30T23:59:46.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:46.697 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:46.799 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:46.828 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:46.905 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:46.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:46.944 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:46.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:46.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:47.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:47.082 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:47.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:47.229 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:47.331 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:47.356 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:47.408 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:47 smithi106 ceph-mon[16603]: pgmap v2225: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 8810/209046 objects degraded (4.214%); 16269/209046 objects misplaced (7.782%); 1.0 MiB/s, 16 objects/s recovering 2024-08-30T23:59:47.457 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:47.483 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:47 smithi136 ceph-mon[20207]: pgmap v2225: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 8810/209046 objects degraded (4.214%); 16269/209046 objects misplaced (7.782%); 1.0 MiB/s, 16 objects/s recovering 2024-08-30T23:59:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:47 smithi173 ceph-mon[20554]: pgmap v2225: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.29k op/s; 8810/209046 objects degraded (4.214%); 16269/209046 objects misplaced (7.782%); 1.0 MiB/s, 16 objects/s recovering 2024-08-30T23:59:47.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:47.607 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:47.668 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-30T23:59:47.669 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-30T23:59:47.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:47.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:47.837 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:47.864 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:47.928 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:59:47.928 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:59:47.929 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:59:47.966 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:47.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:47.989 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:48.090 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:48.112 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:48.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:48.237 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:48.339 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.365 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:48.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:48.466 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:48.493 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:48.507 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-30T23:59:48.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:48 smithi136 ceph-mon[20207]: 9.18 scrub starts 2024-08-30T23:59:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:48 smithi136 ceph-mon[20207]: 9.18 scrub ok 2024-08-30T23:59:48.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:48 smithi106 ceph-mon[16603]: 9.18 scrub starts 2024-08-30T23:59:48.561 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:48 smithi106 ceph-mon[16603]: 9.18 scrub ok 2024-08-30T23:59:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:48 smithi173 ceph-mon[20554]: 9.18 scrub starts 2024-08-30T23:59:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:48 smithi173 ceph-mon[20554]: 9.18 scrub ok 2024-08-30T23:59:48.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:48.618 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:48.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:48.741 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:48.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:48.866 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:48.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:48.900 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-30T23:59:48.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-30T23:59:48.966 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:49.029 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:49.131 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:49.155 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:49.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:49.283 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:49.385 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:49.417 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:49.473 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:59:49.474 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:59:49.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-08-30T23:59:49.502 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:49 smithi106 ceph-mon[16603]: pgmap v2227: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 8382/200454 objects degraded (4.182%); 15602/200454 objects misplaced (7.783%) 2024-08-30T23:59:49.519 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:49 smithi136 ceph-mon[20207]: pgmap v2227: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 8382/200454 objects degraded (4.182%); 15602/200454 objects misplaced (7.783%) 2024-08-30T23:59:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:49.573 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:49 smithi173 ceph-mon[20554]: pgmap v2227: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s; 8382/200454 objects degraded (4.182%); 15602/200454 objects misplaced (7.783%) 2024-08-30T23:59:49.575 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:49.596 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:49.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:49.667 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:49.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:49.793 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:49.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:49.916 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:49.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:50.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:50.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:50.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:50.169 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:50.271 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:50.295 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:50.398 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:50.403 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:50 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:50.403 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:50 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:50.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:50 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-30T23:59:50.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:50 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-30T23:59:50.404 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:50 smithi106 ceph-mon[16603]: osdmap e1010: 12 total, 11 up, 12 in 2024-08-30T23:59:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.433 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:50.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:50.451 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:50.452 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:50.486 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:50.486 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:50.518 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:50.518 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:50.534 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:50.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:50 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:50 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:50 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-30T23:59:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:50 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-30T23:59:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:50 smithi136 ceph-mon[20207]: osdmap e1010: 12 total, 11 up, 12 in 2024-08-30T23:59:50.549 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:50.550 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:50.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:50 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:50 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:50 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-30T23:59:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:50 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-30T23:59:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:50 smithi173 ceph-mon[20554]: osdmap e1010: 12 total, 11 up, 12 in 2024-08-30T23:59:50.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:50.689 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:50.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:50.782 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:50.782 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:50.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:50.814 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:50.814 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:50.817 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:50.847 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:50.847 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:50.849 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:50.880 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:50.880 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:50.919 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:50.923 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:50.924 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:50.944 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:50.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:50.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:50.960 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:50.960 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:51.002 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:51.002 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:143396 9.11 9:8f595049:::benchmark_data_smithi173_160189_object41392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1010)", 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:50.947454+0000", 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.019653594, 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.019683912000000001, 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:51.022 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143396 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947454+0000", 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.023 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947454+0000", 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947455+0000", 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6789999999999999e-06 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947462+0000", 2024-08-30T23:59:51.024 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7649999999999996e-06 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947479+0000", 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6753999999999999e-05 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947488+0000", 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1560000000000001e-06 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:51.025 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947549+0000", 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.0999000000000003e-05 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947692+0000", 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014317600000000001 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.947811+0000", 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011922 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.026 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.950500+0000", 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.002688251 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954678+0000", 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0041780510000000003 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.027 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:143407 9.1b 9:d82d32fc:::benchmark_data_smithi173_160189_object41403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1010)", 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:50.954039+0000", 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.013068793, 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.01318534, 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143407 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.028 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954039+0000", 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954039+0000", 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954040+0000", 2024-08-30T23:59:51.029 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.305e-06 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954045+0000", 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8570000000000003e-06 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954059+0000", 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.433e-05 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:51.030 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954066+0000", 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1729999999999998e-06 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954124+0000", 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.8214999999999997e-05 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954243+0000", 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011856199999999999 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.031 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 0,5", 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.954346+0000", 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000103414 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.957442+0000", 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0030952419999999998 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.958004+0000", 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00056269700000000002 2024-08-30T23:59:51.032 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:143419 9.19 9.6ea79b59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1010)", 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:50.963934+0000", 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.0031731519999999998, 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033657309999999998, 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "queued for pg", 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:59:51.033 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143419 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.963934+0000", 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.963934+0000", 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.034 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.963936+0000", 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.576e-06 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.963942+0000", 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.562e-06 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.963955+0000", 2024-08-30T23:59:51.035 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.223e-05 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.963962+0000", 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.2919999999999998e-06 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:143421 9.1a 9:581e1e3d:::benchmark_data_smithi173_160189_object41417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1010)", 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:50.965807+0000", 2024-08-30T23:59:51.036 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.001300587, 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0015477100000000001, 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143421 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:51.037 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965807+0000", 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965807+0000", 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965808+0000", 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.7300000000000004e-07 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.038 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965812+0000", 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4900000000000002e-06 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965823+0000", 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0725e-05 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965831+0000", 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9859999999999994e-06 2024-08-30T23:59:51.039 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.965896+0000", 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.5366999999999999e-05 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.966051+0000", 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000155057 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 1,2", 2024-08-30T23:59:51.040 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.966181+0000", 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012938399999999999 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:143360 9.11 9:8f349579:::benchmark_data_smithi173_160189_object41356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1010)", 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:50.925240+0000", 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 0.041867463000000001, 2024-08-30T23:59:51.041 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.042181930999999999, 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "waiting for sub ops", 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 143360 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925240+0000", 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.042 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925240+0000", 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925241+0000", 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3319999999999999e-06 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925246+0000", 2024-08-30T23:59:51.043 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.147e-06 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925257+0000", 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0627e-05 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925264+0000", 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3799999999999996e-06 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-30T23:59:51.044 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925316+0000", 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1238e-05 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925427+0000", 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000111487 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,9", 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.925530+0000", 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010342 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.045 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.930467+0000", 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0049367839999999996 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:50.935582+0000", 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0051146869999999997 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-30T23:59:51.046 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 5 2024-08-30T23:59:51.047 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:59:51.047 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:51.071 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:51.173 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:51.202 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:51.304 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:51.330 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:51.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:51.455 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:51.491 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: pgmap v2229: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 8033/191553 objects degraded (4.194%); 14924/191553 objects misplaced (7.791%) 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: 9.17 scrub starts 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: 9.0 scrub starts 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: 9.17 scrub ok 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: 9.19 scrub starts 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2918142757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: 9.0 scrub ok 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 8033/191553 objects degraded (4.194%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:51.492 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:51 smithi173 ceph-mon[20554]: osdmap e1011: 12 total, 11 up, 12 in 2024-08-30T23:59:51.526 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: pgmap v2229: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 8033/191553 objects degraded (4.194%); 14924/191553 objects misplaced (7.791%) 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: 9.17 scrub starts 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: 9.0 scrub starts 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: 9.17 scrub ok 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: 9.19 scrub starts 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2918142757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: 9.0 scrub ok 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 8033/191553 objects degraded (4.194%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:51 smithi136 ceph-mon[20207]: osdmap e1011: 12 total, 11 up, 12 in 2024-08-30T23:59:51.557 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.579 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:51.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: pgmap v2229: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 925 op/s; 8033/191553 objects degraded (4.194%); 14924/191553 objects misplaced (7.791%) 2024-08-30T23:59:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: 9.17 scrub starts 2024-08-30T23:59:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: 9.0 scrub starts 2024-08-30T23:59:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: 9.17 scrub ok 2024-08-30T23:59:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: 9.19 scrub starts 2024-08-30T23:59:51.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2918142757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:51.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: 9.0 scrub ok 2024-08-30T23:59:51.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 8033/191553 objects degraded (4.194%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:51.609 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:51 smithi106 ceph-mon[16603]: osdmap e1011: 12 total, 11 up, 12 in 2024-08-30T23:59:51.680 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:51.702 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:51.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:51.834 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:51.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:51.962 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:51.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:52.064 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:52.099 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:52.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-30T23:59:52.201 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.226 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:52.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:52.245 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-30T23:59:52.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:52.357 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:52.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: 1.17 scrub starts 2024-08-30T23:59:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: 1.17 scrub ok 2024-08-30T23:59:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-30T23:59:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-30T23:59:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: 9.19 scrub ok 2024-08-30T23:59:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1972285164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:52.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:52 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:52.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:52.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: 1.17 scrub starts 2024-08-30T23:59:52.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: 1.17 scrub ok 2024-08-30T23:59:52.464 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-30T23:59:52.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-30T23:59:52.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: 9.19 scrub ok 2024-08-30T23:59:52.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1972285164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:52.465 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:52 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:52.485 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: 1.17 scrub starts 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: 1.17 scrub ok 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: 9.19 scrub ok 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1972285164' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:52 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-30T23:59:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-30T23:59:52.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-30T23:59:52.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:52.609 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:52.711 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:52.735 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:52.838 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.861 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:52.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:52.962 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:52.993 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:52.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:53.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:53.120 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:53.222 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:53.252 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:53.279 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:53 smithi106 ceph-mon[16603]: pgmap v2231: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 8033/187053 objects degraded (4.295%); 14405/187053 objects misplaced (7.701%) 2024-08-30T23:59:53.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:53 smithi106 ceph-mon[16603]: 9.13 scrub starts 2024-08-30T23:59:53.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:53 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:59:53.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:53 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:53.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:53 smithi106 ceph-mon[16603]: osdmap e1012: 12 total, 11 up, 12 in 2024-08-30T23:59:53.280 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:53 smithi106 ceph-mon[16603]: 9.13 scrub ok 2024-08-30T23:59:53.354 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:53.382 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:53.419 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:59:53.419 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:59:53.419 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:59:53.420 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:59:53.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:53.508 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:53 smithi136 ceph-mon[20207]: pgmap v2231: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 8033/187053 objects degraded (4.295%); 14405/187053 objects misplaced (7.701%) 2024-08-30T23:59:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:53 smithi136 ceph-mon[20207]: 9.13 scrub starts 2024-08-30T23:59:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:53 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:59:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:53 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:53 smithi136 ceph-mon[20207]: osdmap e1012: 12 total, 11 up, 12 in 2024-08-30T23:59:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:53 smithi136 ceph-mon[20207]: 9.13 scrub ok 2024-08-30T23:59:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:53 smithi173 ceph-mon[20554]: pgmap v2231: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 8033/187053 objects degraded (4.295%); 14405/187053 objects misplaced (7.701%) 2024-08-30T23:59:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:53 smithi173 ceph-mon[20554]: 9.13 scrub starts 2024-08-30T23:59:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:53 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-30T23:59:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:53 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-30T23:59:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:53 smithi173 ceph-mon[20554]: osdmap e1012: 12 total, 11 up, 12 in 2024-08-30T23:59:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:53 smithi173 ceph-mon[20554]: 9.13 scrub ok 2024-08-30T23:59:53.610 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:53.634 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:53.736 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:53.761 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:53.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:53.887 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:53.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:53.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:54.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.018 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:59:54.103 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:59:54.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-08-30T23:59:54.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:54.179 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:54.281 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:54.309 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:54 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-30T23:59:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:54 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-30T23:59:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:54 smithi106 ceph-mon[16603]: 9.b scrub starts 2024-08-30T23:59:54.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:54 smithi106 ceph-mon[16603]: osdmap e1013: 12 total, 11 up, 12 in 2024-08-30T23:59:54.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:54 smithi106 ceph-mon[16603]: 9.b scrub ok 2024-08-30T23:59:54.411 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:54.439 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:54 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-30T23:59:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:54 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-30T23:59:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:54 smithi136 ceph-mon[20207]: 9.b scrub starts 2024-08-30T23:59:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:54 smithi136 ceph-mon[20207]: osdmap e1013: 12 total, 11 up, 12 in 2024-08-30T23:59:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:54 smithi136 ceph-mon[20207]: 9.b scrub ok 2024-08-30T23:59:54.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:54.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-30T23:59:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.568 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:54.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:54 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-30T23:59:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:54 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-30T23:59:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:54 smithi173 ceph-mon[20554]: 9.b scrub starts 2024-08-30T23:59:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:54 smithi173 ceph-mon[20554]: osdmap e1013: 12 total, 11 up, 12 in 2024-08-30T23:59:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:54 smithi173 ceph-mon[20554]: 9.b scrub ok 2024-08-30T23:59:54.670 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:54.696 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:54.872 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:54.901 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:54.902 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:55.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:55.027 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:55.129 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.152 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:55.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:55.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:55.280 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:55.382 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:55.410 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.411 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:55.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-30T23:59:55.538 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-30T23:59:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:55 smithi136 ceph-mon[20207]: pgmap v2234: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 7693/183108 objects degraded (4.201%); 14234/183108 objects misplaced (7.774%) 2024-08-30T23:59:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:55 smithi136 ceph-mon[20207]: 9.20 scrub starts 2024-08-30T23:59:55.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:55 smithi136 ceph-mon[20207]: 9.20 scrub ok 2024-08-30T23:59:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:55 smithi173 ceph-mon[20554]: pgmap v2234: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 7693/183108 objects degraded (4.201%); 14234/183108 objects misplaced (7.774%) 2024-08-30T23:59:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:55 smithi173 ceph-mon[20554]: 9.20 scrub starts 2024-08-30T23:59:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:55 smithi173 ceph-mon[20554]: 9.20 scrub ok 2024-08-30T23:59:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:55 smithi106 ceph-mon[16603]: pgmap v2234: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 7693/183108 objects degraded (4.201%); 14234/183108 objects misplaced (7.774%) 2024-08-30T23:59:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:55 smithi106 ceph-mon[16603]: 9.20 scrub starts 2024-08-30T23:59:55.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:55 smithi106 ceph-mon[16603]: 9.20 scrub ok 2024-08-30T23:59:55.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:55.668 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:55.770 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:55.796 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:55.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:55.928 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:55.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:56.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:56.051 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:56.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:56.178 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:56.280 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:56.308 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:56.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:56.438 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7693/183108 objects degraded (4.201%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:56.540 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:56.571 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:56.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7693/183108 objects degraded (4.201%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:56.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7693/183108 objects degraded (4.201%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-30T23:59:56.642 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:56.674 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.714 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:56.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:56.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:56.848 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:56.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:56.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:56.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:57.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:57.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:57.180 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:57.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:57.221 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-30T23:59:57.222 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-30T23:59:57.222 INFO:teuthology.orchestra.run.smithi106.stdout: "complaint_time": 30, 2024-08-30T23:59:57.222 INFO:teuthology.orchestra.run.smithi106.stdout: "num_blocked_ops": 0 2024-08-30T23:59:57.222 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-30T23:59:57.224 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:57.326 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:57.357 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:57.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:57.467 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-30T23:59:57.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:57.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:57 smithi136 ceph-mon[20207]: pgmap v2235: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 7346/173328 objects degraded (4.238%); 13436/173328 objects misplaced (7.752%) 2024-08-30T23:59:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:57 smithi136 ceph-mon[20207]: 9.11 scrub starts 2024-08-30T23:59:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:57 smithi173 ceph-mon[20554]: pgmap v2235: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 7346/173328 objects degraded (4.238%); 13436/173328 objects misplaced (7.752%) 2024-08-30T23:59:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:57 smithi173 ceph-mon[20554]: 9.11 scrub starts 2024-08-30T23:59:57.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:57.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:57 smithi106 ceph-mon[16603]: pgmap v2235: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 7346/173328 objects degraded (4.238%); 13436/173328 objects misplaced (7.752%) 2024-08-30T23:59:57.601 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:57 smithi106 ceph-mon[16603]: 9.11 scrub starts 2024-08-30T23:59:57.617 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:57.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:57.742 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:57.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:57.866 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:57.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:57.991 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:57.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:58.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:58.116 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:58.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-30T23:59:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.250 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:58.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-30T23:59:58.320 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-30T23:59:58.321 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-30T23:59:58.321 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-30T23:59:58.321 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-30T23:59:58.321 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-30T23:59:58.351 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-30T23:59:58.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-30T23:59:58.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:58.475 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:58 smithi106 ceph-mon[16603]: 9.11 scrub ok 2024-08-30T23:59:58.493 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-30T23:59:58.494 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-30T23:59:58.495 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:58.515 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-30T23:59:58.516 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-30T23:59:58.539 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-30T23:59:58.539 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-30T23:59:58.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:58 smithi136 ceph-mon[20207]: 9.11 scrub ok 2024-08-30T23:59:58.573 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-30T23:59:58.574 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-30T23:59:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:58 smithi173 ceph-mon[20554]: 9.11 scrub ok 2024-08-30T23:59:58.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-30T23:59:58.606 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-30T23:59:58.606 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-30T23:59:58.627 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-30T23:59:58.639 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-30T23:59:58.639 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-30T23:59:58.669 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-30T23:59:58.669 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-30T23:59:58.670 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-30T23:59:58.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-30T23:59:58.703 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-30T23:59:58.703 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-30T23:59:58.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:58.739 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-30T23:59:58.739 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-30T23:59:58.754 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:58.768 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-30T23:59:58.768 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-30T23:59:58.802 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-30T23:59:58.802 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-30T23:59:58.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-30T23:59:58.888 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:58.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-30T23:59:58.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-30T23:59:59.017 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-30T23:59:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-30T23:59:59.097 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-30T23:59:59.119 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-30T23:59:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.170 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:59.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-30T23:59:59.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:59.297 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:59 smithi106 ceph-mon[16603]: pgmap v2236: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 6984/165987 objects degraded (4.208%); 12825/165987 objects misplaced (7.727%) 2024-08-30T23:59:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:59 smithi106 ceph-mon[16603]: 9.c scrub starts 2024-08-30T23:59:59.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 30 23:59:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2510974138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:59.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:59.423 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-30T23:59:59.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-30T23:59:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:59 smithi136 ceph-mon[20207]: pgmap v2236: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 6984/165987 objects degraded (4.208%); 12825/165987 objects misplaced (7.727%) 2024-08-30T23:59:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:59 smithi136 ceph-mon[20207]: 9.c scrub starts 2024-08-30T23:59:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 30 23:59:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2510974138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:59.549 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-30T23:59:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:59 smithi173 ceph-mon[20554]: pgmap v2236: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.17k op/s; 6984/165987 objects degraded (4.208%); 12825/165987 objects misplaced (7.727%) 2024-08-30T23:59:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:59 smithi173 ceph-mon[20554]: 9.c scrub starts 2024-08-30T23:59:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 30 23:59:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2510974138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-30T23:59:59.651 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-30T23:59:59.679 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-30T23:59:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-30T23:59:59.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-30T23:59:59.804 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-30T23:59:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-30T23:59:59.852 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-30T23:59:59.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-08-30T23:59:59.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-30T23:59:59.929 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-30T23:59:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-30T23:59:59.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:00.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:00.061 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:00.163 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:00.192 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:00.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:00.318 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: 9.c scrub ok 2024-08-31T00:00:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 6984/165987 objects degraded (4.208%), 5 pgs degraded, 13 pgs undersized 2024-08-31T00:00:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: [WRN] OSD_DOWN: 1 osds down 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: osd.7 (root=default,host=smithi136) is down 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: [WRN] PG_DEGRADED: Degraded data redundancy: 6984/165987 objects degraded (4.208%), 5 pgs degraded, 13 pgs undersized 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.2 is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.9 is stuck undersized for 3m, current state active+undersized, last acting [9,2] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.d is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.e is stuck undersized for 3m, current state active+undersized, last acting [4,3] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.10 is stuck undersized for 3m, current state active+undersized, last acting [5,10] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.14 is stuck undersized for 3m, current state active+undersized, last acting [9,2] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.15 is stuck undersized for 3m, current state active+undersized, last acting [5,8] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.18 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,2] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 1.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 9.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,3] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 9.a is stuck undersized for 94s, current state active+undersized+degraded, last acting [5,1] 2024-08-31T00:00:00.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 9.d is stuck undersized for 94s, current state active+undersized+degraded, last acting [5,1] 2024-08-31T00:00:00.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:00 smithi106 ceph-mon[16603]: pg 9.e is stuck undersized for 94s, current state active+undersized+degraded, last acting [10,1] 2024-08-31T00:00:00.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:00.442 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: 9.c scrub ok 2024-08-31T00:00:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 6984/165987 objects degraded (4.208%), 5 pgs degraded, 13 pgs undersized 2024-08-31T00:00:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: [WRN] OSD_DOWN: 1 osds down 2024-08-31T00:00:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: osd.7 (root=default,host=smithi136) is down 2024-08-31T00:00:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: [WRN] PG_DEGRADED: Degraded data redundancy: 6984/165987 objects degraded (4.208%), 5 pgs degraded, 13 pgs undersized 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.2 is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.9 is stuck undersized for 3m, current state active+undersized, last acting [9,2] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.d is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.e is stuck undersized for 3m, current state active+undersized, last acting [4,3] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.10 is stuck undersized for 3m, current state active+undersized, last acting [5,10] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.14 is stuck undersized for 3m, current state active+undersized, last acting [9,2] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.15 is stuck undersized for 3m, current state active+undersized, last acting [5,8] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.18 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,2] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 1.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 9.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,3] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 9.a is stuck undersized for 94s, current state active+undersized+degraded, last acting [5,1] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 9.d is stuck undersized for 94s, current state active+undersized+degraded, last acting [5,1] 2024-08-31T00:00:00.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:00 smithi136 ceph-mon[20207]: pg 9.e is stuck undersized for 94s, current state active+undersized+degraded, last acting [10,1] 2024-08-31T00:00:00.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:00.566 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: 9.c scrub ok 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 6984/165987 objects degraded (4.208%), 5 pgs degraded, 13 pgs undersized 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: [WRN] OSD_DOWN: 1 osds down 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: osd.7 (root=default,host=smithi136) is down 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: [WRN] PG_DEGRADED: Degraded data redundancy: 6984/165987 objects degraded (4.208%), 5 pgs degraded, 13 pgs undersized 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.2 is stuck undersized for 3m, current state active+undersized, last acting [4,0] 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.9 is stuck undersized for 3m, current state active+undersized, last acting [9,2] 2024-08-31T00:00:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.d is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.e is stuck undersized for 3m, current state active+undersized, last acting [4,3] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.10 is stuck undersized for 3m, current state active+undersized, last acting [5,10] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.14 is stuck undersized for 3m, current state active+undersized, last acting [9,2] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.15 is stuck undersized for 3m, current state active+undersized, last acting [5,8] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.18 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,2] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 1.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 9.7 is stuck undersized for 2m, current state active+undersized+degraded, last acting [10,3] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 9.a is stuck undersized for 94s, current state active+undersized+degraded, last acting [5,1] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 9.d is stuck undersized for 94s, current state active+undersized+degraded, last acting [5,1] 2024-08-31T00:00:00.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:00 smithi173 ceph-mon[20554]: pg 9.e is stuck undersized for 94s, current state active+undersized+degraded, last acting [10,1] 2024-08-31T00:00:00.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:00.691 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:00.777 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:00.793 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:00.816 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:00.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.820 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:00.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:00.951 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:00.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:00.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:01.053 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:01.078 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.079 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:01.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:01.207 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:01.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.344 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:01.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:01.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:01.473 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:01 smithi136 ceph-mon[20207]: pgmap v2237: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 6618/156519 objects degraded (4.228%); 12104/156519 objects misplaced (7.733%) 2024-08-31T00:00:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:01 smithi136 ceph-mon[20207]: 9.22 scrub starts 2024-08-31T00:00:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:01 smithi136 ceph-mon[20207]: 9.22 scrub ok 2024-08-31T00:00:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:01 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6618/156519 objects degraded (4.228%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:01.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:01 smithi173 ceph-mon[20554]: pgmap v2237: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 6618/156519 objects degraded (4.228%); 12104/156519 objects misplaced (7.733%) 2024-08-31T00:00:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:01 smithi173 ceph-mon[20554]: 9.22 scrub starts 2024-08-31T00:00:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:01 smithi173 ceph-mon[20554]: 9.22 scrub ok 2024-08-31T00:00:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:01 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6618/156519 objects degraded (4.228%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:01.576 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:01.605 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:01 smithi106 ceph-mon[16603]: pgmap v2237: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 6618/156519 objects degraded (4.228%); 12104/156519 objects misplaced (7.733%) 2024-08-31T00:00:01.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:01 smithi106 ceph-mon[16603]: 9.22 scrub starts 2024-08-31T00:00:01.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:01 smithi106 ceph-mon[16603]: 9.22 scrub ok 2024-08-31T00:00:01.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:01 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6618/156519 objects degraded (4.228%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:01.629 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:01.705 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:01.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.734 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:01.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:01.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:01.857 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:01.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:01.985 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:01.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:02.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:02.111 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:02.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:02.240 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:02.289 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-31T00:00:02.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:02.373 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:02.406 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:02.406 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:02.476 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:02.502 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:02.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:02 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3078717270' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:02.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:02 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3078717270' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:02.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:02.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:02 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3078717270' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.627 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:02.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:02.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:02.750 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:02.816 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-31T00:00:02.852 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:02.880 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:02.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:02.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:03.011 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:03.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:03.142 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:03.244 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:03.272 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:03.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.273 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:03.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:03.401 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:03.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:03.530 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:03 smithi136 ceph-mon[20207]: pgmap v2238: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 6618/151527 objects degraded (4.368%); 11538/151527 objects misplaced (7.614%) 2024-08-31T00:00:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:03 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:03 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3078717270' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:03 smithi136 ceph-mon[20207]: osdmap e1014: 12 total, 11 up, 12 in 2024-08-31T00:00:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:03 smithi173 ceph-mon[20554]: pgmap v2238: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 6618/151527 objects degraded (4.368%); 11538/151527 objects misplaced (7.614%) 2024-08-31T00:00:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:03 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:03 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3078717270' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:03 smithi173 ceph-mon[20554]: osdmap e1014: 12 total, 11 up, 12 in 2024-08-31T00:00:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:03 smithi106 ceph-mon[16603]: pgmap v2238: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.10k op/s; 6618/151527 objects degraded (4.368%); 11538/151527 objects misplaced (7.614%) 2024-08-31T00:00:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:03 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:03 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3078717270' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:03.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:03 smithi106 ceph-mon[16603]: osdmap e1014: 12 total, 11 up, 12 in 2024-08-31T00:00:03.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:03.658 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:03.760 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.797 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:03.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:03.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:03.924 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:03.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:04.027 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:04.054 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:04.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:04.183 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:04.286 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:04.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-31T00:00:04.313 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:04.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:04.443 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:04.545 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:04.573 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:04.611 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:04.613 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:04.657 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:04.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:04.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:04.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:04.832 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:04.935 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:04.967 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:04.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:05.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:05.099 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:05.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:05.228 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:05.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:05.356 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:05.443 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:05.443 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:05.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.487 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:05.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:05.503 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:05.503 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-08-31T00:00:05.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:05 smithi136 ceph-mon[20207]: pgmap v2240: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 6162/146703 objects degraded (4.200%); 11341/146703 objects misplaced (7.731%) 2024-08-31T00:00:05.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:05 smithi106 ceph-mon[16603]: pgmap v2240: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 6162/146703 objects degraded (4.200%); 11341/146703 objects misplaced (7.731%) 2024-08-31T00:00:05.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:05 smithi173 ceph-mon[20554]: pgmap v2240: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 42 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 6162/146703 objects degraded (4.200%); 11341/146703 objects misplaced (7.731%) 2024-08-31T00:00:05.588 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:05.616 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:05.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:05.740 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.741 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:05.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-31T00:00:05.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:05.869 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:05.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:05.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:06.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:06.044 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:06.044 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-31T00:00:06.069 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:06.069 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:06.094 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:06.094 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:06.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:06.126 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:06.126 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:06.132 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:06.160 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:06.162 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:06.163 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:06.182 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:06.199 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:06.199 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:06.234 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:06.234 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:06.234 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:06.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:06.263 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:06.264 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.265 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:06.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:06.288 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:06.289 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:06.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-31T00:00:06.310 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:06.311 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:06.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:06.387 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:06.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:06 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2913156970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:06.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:06 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 6162/146703 objects degraded (4.200%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:06.458 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:06 smithi106 ceph-mon[16603]: osdmap e1015: 12 total, 11 up, 12 in 2024-08-31T00:00:06.458 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:06.459 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:06.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:06.512 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:06 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2913156970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:06 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 6162/146703 objects degraded (4.200%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:06 smithi136 ceph-mon[20207]: osdmap e1015: 12 total, 11 up, 12 in 2024-08-31T00:00:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:06 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2913156970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:06 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 6162/146703 objects degraded (4.200%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:06 smithi173 ceph-mon[20554]: osdmap e1015: 12 total, 11 up, 12 in 2024-08-31T00:00:06.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:06.638 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.639 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:06.655 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:06.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:06.771 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:06.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:06.911 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:06.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:07.013 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:07.045 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:07.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:07.177 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:07.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:07.297 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-31T00:00:07.305 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:00:07.305 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-31T00:00:07.305 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-31T00:00:07.305 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-31T00:00:07.305 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:00:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.307 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:07.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:07.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:07.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:07 smithi106 ceph-mon[16603]: pgmap v2241: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 5763/135381 objects degraded (4.257%); 10468/135381 objects misplaced (7.732%) 2024-08-31T00:00:07.429 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3193159232' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.430 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:07.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:07.531 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:07.533 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:07 smithi136 ceph-mon[20207]: pgmap v2241: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 5763/135381 objects degraded (4.257%); 10468/135381 objects misplaced (7.732%) 2024-08-31T00:00:07.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3193159232' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:07.563 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:07 smithi173 ceph-mon[20554]: pgmap v2241: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 5763/135381 objects degraded (4.257%); 10468/135381 objects misplaced (7.732%) 2024-08-31T00:00:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3193159232' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:07.597 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:07.597 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:07.597 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:07.598 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:07.665 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:07.692 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:07.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:07.821 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:07.923 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:07.949 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:07.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:08.020 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-31T00:00:08.052 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:08.086 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:08.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:08.211 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:08.311 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:08.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:08.340 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:08.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:08.469 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:08.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:08.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:08 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3193159232' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:08.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:08 smithi173 ceph-mon[20554]: osdmap e1016: 12 total, 11 up, 12 in 2024-08-31T00:00:08.593 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:08 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3193159232' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:08.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:08 smithi106 ceph-mon[16603]: osdmap e1016: 12 total, 11 up, 12 in 2024-08-31T00:00:08.615 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:08 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:08.616 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3193159232' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:08.616 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:08 smithi136 ceph-mon[20207]: osdmap e1016: 12 total, 11 up, 12 in 2024-08-31T00:00:08.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:08.720 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:08.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:08.850 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:08.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 763/762 [PushReplyOp(7:23574efc:::benchmark_data_smithi173_129296_object75:head)])", 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:27.288318+0000", 2024-08-31T00:00:08.882 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 581.591435123, 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71604419900000005, 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288318+0000", 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0699999999999997e-07 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:27.288323+0000", 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3490000000000002e-06 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001046+0000", 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.71272310000000005 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001053+0000", 2024-08-31T00:00:08.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0230000000000004e-06 2024-08-31T00:00:08.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.001113+0000", 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0402000000000002e-05 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:28.004362+0000", 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032488180000000001 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 766/762 [PushReplyOp(7:21ab2b56:::benchmark_data_smithi173_129296_object2772:head)])", 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:31.155739+0000", 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 577.72401374200001, 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92479003900000001, 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155739+0000", 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155740+0000", 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0780000000000001e-06 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:31.155745+0000", 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0610000000000003e-06 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077308+0000", 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92156320800000002 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077323+0000", 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4443e-05 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.077429+0000", 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105683 2024-08-31T00:00:08.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:32.080529+0000", 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031005659999999999 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(7.4 767/762 [PushReplyOp(7:26f21d66:::benchmark_data_smithi173_129296_object4816:head)])", 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:34.833184+0000", 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 574.04656866300002, 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0014494869999999, 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833184+0000", 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833185+0000", 2024-08-31T00:00:08.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0499999999999999e-06 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:34.833190+0000", 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0830000000000003e-06 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830783+0000", 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.99759255400000002 2024-08-31T00:00:08.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830792+0000", 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7779999999999996e-06 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.830874+0000", 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2557000000000003e-05 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:35.834634+0000", 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003759465 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.b 769/756 [PushOp(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head, version: 734'1560, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:dfd66d04:::benchmark_data_smithi173_129296_object25452:head@734'1560, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:50:44.553157+0000", 2024-08-31T00:00:08.898 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 564.32659614299996, 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60120657700000002, 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553157+0000", 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553158+0000", 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3799999999999999e-06 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:44.553211+0000", 2024-08-31T00:00:08.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2676999999999997e-05 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153021+0000", 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59981003200000005 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153032+0000", 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1415e-05 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153139+0000", 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107075 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.153149+0000", 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9629999999999993e-06 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:50:45.154363+0000", 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001214035 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:123065 7.1 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e792)", 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:51:54.487514+0000", 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 494.39223916700001, 2024-08-31T00:00:08.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64777179500000004, 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 123065 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487514+0000", 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487515+0000", 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.477e-06 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487521+0000", 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487533+0000", 2024-08-31T00:00:08.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.234e-05 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487541+0000", 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487594+0000", 2024-08-31T00:00:08.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3134000000000003e-05 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487681+0000", 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7437999999999999e-05 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,10", 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.487753+0000", 2024-08-31T00:00:08.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1632999999999997e-05 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.490939+0000", 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031855550000000001 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:54.492073+0000", 2024-08-31T00:00:08.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011345039999999999 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135209+0000", 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.64313627299999998 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135251+0000", 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1378000000000003e-05 2024-08-31T00:00:08.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:51:55.135285+0000", 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4730999999999997e-05 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:142377 7.8 7:18815242:::benchmark_data_smithi173_129296_object55261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-08-31T00:00:08.911 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:37.395572+0000", 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 451.48418063899999, 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61694005799999996, 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 142377 2024-08-31T00:00:08.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395572+0000", 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395573+0000", 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.99e-07 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395576+0000", 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.394e-06 2024-08-31T00:00:08.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395586+0000", 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4360000000000008e-06 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395591+0000", 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8910000000000003e-06 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395638+0000", 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7027000000000002e-05 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395723+0000", 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5181999999999999e-05 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.395790+0000", 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7430999999999999e-05 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400118+0000", 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043280920000000004 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:37.400495+0000", 2024-08-31T00:00:08.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037659899999999998 2024-08-31T00:00:08.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012439+0000", 2024-08-31T00:00:08.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61194435700000005 2024-08-31T00:00:08.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012480+0000", 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1180000000000002e-05 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:38.012512+0000", 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1869999999999998e-05 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145475 7.4 7:2251fdb7:::benchmark_data_smithi173_129296_object58359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.173008+0000", 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.70674533599998, 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61919969799999997, 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145475 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173008+0000", 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173010+0000", 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7190000000000001e-06 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173019+0000", 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6289999999999999e-06 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173041+0000", 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1658000000000002e-05 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173051+0000", 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0139999999999999e-05 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173114+0000", 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3289999999999999e-05 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173214+0000", 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000100282 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.173291+0000", 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6983999999999993e-05 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.177105+0000", 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003813355 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.178585+0000", 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001480608 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792092+0000", 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61350662700000003 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792154+0000", 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1884000000000006e-05 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.792207+0000", 2024-08-31T00:00:08.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3522e-05 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145497 7.4 7:2208bbfd:::benchmark_data_smithi173_129296_object58381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.208393+0000", 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.67135995400002, 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.60666463800000003, 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.928 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145497 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208393+0000", 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208395+0000", 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.66e-06 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208403+0000", 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2309999999999996e-06 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208421+0000", 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7825999999999998e-05 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208428+0000", 2024-08-31T00:00:08.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0539999999999997e-06 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208486+0000", 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8517000000000002e-05 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208625+0000", 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138517 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.208748+0000", 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012329600000000001 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.212469+0000", 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037205519999999998 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.213491+0000", 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010220489999999999 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.814962+0000", 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.601471588 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815023+0000", 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0732000000000001e-05 2024-08-31T00:00:08.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815058+0000", 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4616000000000002e-05 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145502 7.4 7:2d669461:::benchmark_data_smithi173_129296_object58386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.215876+0000", 2024-08-31T00:00:08.935 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.66387687100001, 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59929806799999996, 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145502 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215876+0000", 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215878+0000", 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6220000000000001e-06 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215886+0000", 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9219999999999997e-06 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215902+0000", 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6677e-05 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215910+0000", 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8499999999999994e-06 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.215967+0000", 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6898999999999999e-05 2024-08-31T00:00:08.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216108+0000", 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014146699999999999 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.216238+0000", 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012992900000000001 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223310+0000", 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070715079999999998 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224153+0000", 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00084321899999999998 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815104+0000", 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59095095600000003 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815137+0000", 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2759000000000003e-05 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815174+0000", 2024-08-31T00:00:08.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7259999999999999e-05 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145504 7.4 7:2d76a73e:::benchmark_data_smithi173_129296_object58388:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.219641+0000", 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.66011220299998, 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59561452400000003, 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.943 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145504 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219641+0000", 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219642+0000", 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.782e-06 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219653+0000", 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0723999999999999e-05 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219670+0000", 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.681e-05 2024-08-31T00:00:08.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219679+0000", 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.833e-06 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219829+0000", 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000149863 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.219920+0000", 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1633999999999997e-05 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.220015+0000", 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4344000000000003e-05 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.223383+0000", 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033687449999999998 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.224195+0000", 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00081209999999999995 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815208+0000", 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.59101230400000004 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815235+0000", 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7310000000000001e-05 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815255+0000", 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0075e-05 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145510 7.4 7:2d7edcbd:::benchmark_data_smithi173_129296_object58394:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.240495+0000", 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.63925770700001, 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.57487306000000005, 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145510 2024-08-31T00:00:08.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240495+0000", 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240497+0000", 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.381e-06 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240502+0000", 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.575e-06 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240514+0000", 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1996e-05 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240522+0000", 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7079999999999997e-06 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240565+0000", 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3095e-05 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240644+0000", 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9509000000000001e-05 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.240712+0000", 2024-08-31T00:00:08.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7859000000000002e-05 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.244050+0000", 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003337727 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.245826+0000", 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017758229999999999 2024-08-31T00:00:08.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815284+0000", 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.56945848700000001 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815331+0000", 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6276999999999998e-05 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.815368+0000", 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7623000000000002e-05 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145529 7.4 7:2339dd6d:::benchmark_data_smithi173_129296_object58413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.295076+0000", 2024-08-31T00:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.58467682100002, 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.53035033899999995, 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145529 2024-08-31T00:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295076+0000", 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295077+0000", 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.47e-07 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295085+0000", 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9750000000000007e-06 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295097+0000", 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2391999999999999e-05 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295103+0000", 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8529999999999997e-06 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295150+0000", 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6684000000000002e-05 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295235+0000", 2024-08-31T00:00:08.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4892e-05 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.295302+0000", 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6934999999999995e-05 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.298587+0000", 2024-08-31T00:00:08.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0032851030000000002 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.299413+0000", 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00082625299999999999 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825357+0000", 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52594381499999998 2024-08-31T00:00:08.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825395+0000", 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7898999999999997e-05 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.825426+0000", 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1591000000000003e-05 2024-08-31T00:00:08.966 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145538 7.4 7:2666a83b:::benchmark_data_smithi173_129296_object58422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.322090+0000", 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.55766281000001, 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51489389200000002, 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.967 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145538 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-31T00:00:08.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322090+0000", 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322092+0000", 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.683e-06 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322100+0000", 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3119999999999994e-06 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322119+0000", 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9368000000000002e-05 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322128+0000", 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1599999999999998e-06 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322189+0000", 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0976e-05 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322326+0000", 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013735099999999999 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.322456+0000", 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012996399999999999 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326052+0000", 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035965490000000001 2024-08-31T00:00:08.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328903+0000", 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028504350000000001 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836909+0000", 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.508006556 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836949+0000", 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9864000000000003e-05 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.836984+0000", 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4674e-05 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145540 7.4 7:22c5e8c4:::benchmark_data_smithi173_129296_object58424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.326258+0000", 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.55349491800001, 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.52274398799999999, 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145540 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326258+0000", 2024-08-31T00:00:08.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326259+0000", 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2440000000000001e-06 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326268+0000", 2024-08-31T00:00:08.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3760000000000008e-06 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326280+0000", 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2665e-05 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326288+0000", 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326325+0000", 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6508999999999999e-05 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326403+0000", 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7877e-05 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.326473+0000", 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9820999999999994e-05 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.328987+0000", 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025140520000000001 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.330596+0000", 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001609367 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848889+0000", 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.51829265199999996 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.848951+0000", 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2806999999999997e-05 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849002+0000", 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0544000000000003e-05 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27487.0:145558 7.4 7:2fe416e2:::benchmark_data_smithi173_129296_object58442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:43.405684+0000", 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 445.47406900099998, 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.44345267799999999, 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27487", 2024-08-31T00:00:08.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 145558 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405684+0000", 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405686+0000", 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4059999999999999e-06 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405696+0000", 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0073e-05 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405726+0000", 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9686999999999999e-05 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405740+0000", 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4072e-05 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405842+0000", 2024-08-31T00:00:08.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010199700000000001 2024-08-31T00:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.405928+0000", 2024-08-31T00:00:08.987 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:08.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5784000000000004e-05 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.406015+0000", 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7096999999999996e-05 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.409500+0000", 2024-08-31T00:00:08.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034854629999999998 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.411030+0000", 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0015296820000000001 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849046+0000", 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.43801564300000001 2024-08-31T00:00:08.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849097+0000", 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1424000000000001e-05 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:43.849137+0000", 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9350000000000001e-05 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.27853.0:21811 8.5 e853/840 8:a2778f2f:::benchmark_data_smithi173_143816_object21810:head v 853'1377, mlcod=853'1375)", 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.953091+0000", 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 357.92666168099998, 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34313867100000001, 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953091+0000", 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953091+0000", 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953093+0000", 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.81e-06 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.953143+0000", 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0099000000000003e-05 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295091+0000", 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34194817500000002 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295107+0000", 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6229999999999999e-05 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295194+0000", 2024-08-31T00:00:08.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6767999999999993e-05 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.295221+0000", 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6829e-05 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.296190+0000", 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096933700000000002 2024-08-31T00:00:08.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:11.296230+0000", 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9422999999999998e-05 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:36537 8.e 8:78cdc0b0:::benchmark_data_smithi173_143816_object36536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-31T00:00:08.997 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:25.400191+0000", 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 343.47956142800001, 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.395384505, 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36537 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.998 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400191+0000", 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400191+0000", 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:08.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400193+0000", 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0690000000000001e-06 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400228+0000", 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5423000000000003e-05 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400241+0000", 2024-08-31T00:00:09.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2695e-05 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400248+0000", 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999993e-06 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400300+0000", 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1440000000000002e-05 2024-08-31T00:00:09.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400391+0000", 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1564999999999995e-05 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,2", 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.400477+0000", 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5532999999999996e-05 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.401514+0000", 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001037428 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.402322+0000", 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00080706599999999995 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795505+0000", 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.393183318 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795541+0000", 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6442000000000001e-05 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:09.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:25.795576+0000", 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4666000000000003e-05 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:62247 9.10 9:0e7ffda9:::benchmark_data_smithi173_160189_object62246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e971)", 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:32.985840+0000", 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.893912503999999, 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35390136500000002, 2024-08-31T00:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62247 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.006 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985840+0000", 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:09.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985840+0000", 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985842+0000", 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7880000000000001e-06 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985884+0000", 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1829999999999998e-05 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985903+0000", 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8671000000000001e-05 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985910+0000", 2024-08-31T00:00:09.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8600000000000004e-06 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.985966+0000", 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6168999999999997e-05 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:09.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.986086+0000", 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120548 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.986197+0000", 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111037 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.987276+0000", 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010784379999999999 2024-08-31T00:00:09.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.987920+0000", 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00064430599999999998 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339634+0000", 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35171409599999998 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339670+0000", 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5475999999999997e-05 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339742+0000", 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2145999999999997e-05 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.28405.0:62247 9.10 e971/957 ondisk, result = 0)", 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:32.987685+0000", 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.892067695999998, 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35206173099999999, 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.987685+0000", 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.987685+0000", 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.987686+0000", 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2500000000000005e-07 2024-08-31T00:00:09.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:32.987689+0000", 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4340000000000001e-06 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339592+0000", 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35190238299999999 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339597+0000", 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.899e-06 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339626+0000", 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9728999999999999e-05 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339630+0000", 2024-08-31T00:00:09.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8650000000000003e-06 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.339747+0000", 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116796 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:91176 9.15 e984/956 9:abfdfcc6:::benchmark_data_smithi173_160189_object91175:head v 984'3415, mlcod=984'3413)", 2024-08-31T00:00:09.018 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:00.565073+0000", 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.314679413999997, 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.78523059200000001, 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.565073+0000", 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:09.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.565073+0000", 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.565075+0000", 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3549999999999999e-06 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.565154+0000", 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8763000000000005e-05 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.348713+0000", 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.78355972100000004 2024-08-31T00:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.348721+0000", 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.875e-06 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.348799+0000", 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8201999999999994e-05 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.348821+0000", 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1364000000000001e-05 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.350263+0000", 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001442235 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:09.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.350304+0000", 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1077e-05 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:09.024 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:00:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.026 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:09.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:09.127 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:09.155 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:09.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:09.279 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:09.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:09 smithi106 ceph-mon[16603]: pgmap v2244: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 5381/128139 objects degraded (4.199%); 9946/128139 objects misplaced (7.762%) 2024-08-31T00:00:09.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:09.402 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:09.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:09.535 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:09 smithi173 ceph-mon[20554]: pgmap v2244: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 5381/128139 objects degraded (4.199%); 9946/128139 objects misplaced (7.762%) 2024-08-31T00:00:09.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:09.667 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.668 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:09.746 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:09 smithi136 ceph-mon[20207]: pgmap v2244: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.30k op/s; 5381/128139 objects degraded (4.199%); 9946/128139 objects misplaced (7.762%) 2024-08-31T00:00:09.769 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:09.790 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-31T00:00:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:09.825 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:09.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-08-31T00:00:09.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:09.929 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:09.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:10.031 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:10.060 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:10.143 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:10.163 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:10.163 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:10.184 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.185 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:10.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:10.311 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:10.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:10.436 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:10.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:10.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:10.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:10.692 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:10.732 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:10.733 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:10.733 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:10.733 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:10.733 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:10.733 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:10.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-31T00:00:10.794 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:10.836 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:10.884 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:00:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-31T00:00:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.884 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:167634 9.19 e1016/957 9:9d013ae8:::benchmark_data_smithi173_160189_object65630:head v 1016'5783, mlcod=1016'5780)", 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-31T00:00:10.875646+0000", 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0081038169999999993, 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0081315740000000008, 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.875646+0000", 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:10.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.875646+0000", 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.875647+0000", 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5499999999999996e-07 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.875653+0000", 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5439999999999998e-06 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880147+0000", 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044944800000000003 2024-08-31T00:00:10.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880155+0000", 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9300000000000003e-06 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880227+0000", 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1178000000000002e-05 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880251+0000", 2024-08-31T00:00:10.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4298999999999999e-05 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:167642 9.19 e1016/957 9:99fe6c83:::benchmark_data_smithi173_160189_object65638:head v 1016'5784, mlcod=1016'5781)", 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-31T00:00:10.880117+0000", 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.003632854, 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003716904, 2024-08-31T00:00:10.889 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880117+0000", 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:10.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880117+0000", 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880118+0000", 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.880125+0000", 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0169999999999999e-06 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.882724+0000", 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025985399999999999 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.882732+0000", 2024-08-31T00:00:10.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2719999999999997e-06 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.882813+0000", 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0754000000000006e-05 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-31T00:00:10.882837+0000", 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4515000000000002e-05 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:10.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-08-31T00:00:10.894 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-08-31T00:00:10.894 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:00:10.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:10.968 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:10.969 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:11.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:11.098 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:11.121 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-31T00:00:11.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:11.227 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:11.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:11.329 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:11.356 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:11.458 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:11.481 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:11.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:11 smithi106 ceph-mon[16603]: pgmap v2245: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 4988/119394 objects degraded (4.178%); 9293/119394 objects misplaced (7.783%) 2024-08-31T00:00:11.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:11 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:11.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:11 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:11.486 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:11 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4988/119394 objects degraded (4.178%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:11.487 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:11 smithi106 ceph-mon[16603]: osdmap e1017: 12 total, 11 up, 12 in 2024-08-31T00:00:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:11.521 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:11.522 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-08-31T00:00:11.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:11 smithi173 ceph-mon[20554]: pgmap v2245: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 4988/119394 objects degraded (4.178%); 9293/119394 objects misplaced (7.783%) 2024-08-31T00:00:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:11 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:11 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:11 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4988/119394 objects degraded (4.178%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:11 smithi173 ceph-mon[20554]: osdmap e1017: 12 total, 11 up, 12 in 2024-08-31T00:00:11.581 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:11.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:11.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:11 smithi136 ceph-mon[20207]: pgmap v2245: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 4988/119394 objects degraded (4.178%); 9293/119394 objects misplaced (7.783%) 2024-08-31T00:00:11.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:11 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:11.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:11 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:11.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:11 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4988/119394 objects degraded (4.178%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:11.604 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:11 smithi136 ceph-mon[20207]: osdmap e1017: 12 total, 11 up, 12 in 2024-08-31T00:00:11.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:11.709 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:11.737 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:11.745 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-31T00:00:11.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:11.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:11.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:11.996 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:11.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:12.068 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:12.099 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:12.122 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:12.224 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.252 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:12.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:12.316 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:12.338 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:12.339 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-31T00:00:12.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:12.372 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:12.373 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:12.402 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:12.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:12.403 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:12.403 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:12.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:12 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:12.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:12 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:12.428 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:12 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:00:12.429 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2612992989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:12.429 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:12.429 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:12.464 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:12.464 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:12.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:12.504 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:12.505 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:12.532 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:12.533 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:12.533 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:12.533 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:12.562 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:12.563 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:12.563 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:12 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:12.563 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:12 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:12.563 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:12 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:00:12.563 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2612992989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:12 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:12 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:12 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:00:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2612992989' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:12.593 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:12.593 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:12.595 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:12.595 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:12.623 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:12.623 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:12.636 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:12.655 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:12.655 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:12.663 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:12.765 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:12.794 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.795 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:12.896 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:12.917 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:12.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:13.019 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:13.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:13.147 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:13.197 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:13.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:13.328 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:13.337 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-31T00:00:13.430 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:13.454 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:13.515 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:13.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:13.573 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:13 smithi173 ceph-mon[20554]: pgmap v2247: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 4988/114651 objects degraded (4.351%); 8750/114651 objects misplaced (7.632%) 2024-08-31T00:00:13.581 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:13 smithi106 ceph-mon[16603]: pgmap v2247: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 4988/114651 objects degraded (4.351%); 8750/114651 objects misplaced (7.632%) 2024-08-31T00:00:13.623 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:13.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:13.707 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:13.749 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:13 smithi136 ceph-mon[20207]: pgmap v2247: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.15k op/s; 4988/114651 objects degraded (4.351%); 8750/114651 objects misplaced (7.632%) 2024-08-31T00:00:13.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:13.835 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:13.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:13.964 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:13.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:14.066 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:14.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:14.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:14.212 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:14.212 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:14.228 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:14.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:14.346 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:14.353 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:14.365 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-31T00:00:14.454 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:14.479 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:00:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:00:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:14 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:00:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/350068144' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:14.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:00:14.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:00:14.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:14 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:00:14.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/350068144' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:00:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:00:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:14 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:00:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/350068144' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:14.581 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:14.607 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:14.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:14.732 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:14.834 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:14.860 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:14.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-31T00:00:14.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:14.985 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:14.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:15.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.117 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:15.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:15.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:15.249 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:15.291 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:15.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:15 smithi173 ceph-mon[20554]: pgmap v2248: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 4583/109929 objects degraded (4.169%); 8578/109929 objects misplaced (7.803%) 2024-08-31T00:00:15.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:15 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:15.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:15 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/350068144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:15.313 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:15 smithi173 ceph-mon[20554]: osdmap e1018: 12 total, 11 up, 12 in 2024-08-31T00:00:15.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:15.377 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:15.377 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-08-31T00:00:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.378 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:15.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:15.480 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:15.506 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.507 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:15.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:15 smithi106 ceph-mon[16603]: pgmap v2248: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 4583/109929 objects degraded (4.169%); 8578/109929 objects misplaced (7.803%) 2024-08-31T00:00:15.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:15 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:15.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:15 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/350068144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:15.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:15 smithi106 ceph-mon[16603]: osdmap e1018: 12 total, 11 up, 12 in 2024-08-31T00:00:15.608 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:15.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.644 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:15.745 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:15.768 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:15.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:15 smithi136 ceph-mon[20207]: pgmap v2248: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 4583/109929 objects degraded (4.169%); 8578/109929 objects misplaced (7.803%) 2024-08-31T00:00:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:15 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:15 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/350068144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:15.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:15 smithi136 ceph-mon[20207]: osdmap e1018: 12 total, 11 up, 12 in 2024-08-31T00:00:15.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:15.896 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:15.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:15.933 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:15.934 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:15.997 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.029 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:16.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:16.129 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:16.154 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:16.256 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:16.282 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:16.384 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:16.414 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:16.470 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:16.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:16.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:16.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:16 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 4583/109929 objects degraded (4.169%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:16.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:16 smithi173 ceph-mon[20554]: osdmap e1019: 12 total, 11 up, 12 in 2024-08-31T00:00:16.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:16 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 4583/109929 objects degraded (4.169%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:16.594 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:16 smithi106 ceph-mon[16603]: osdmap e1019: 12 total, 11 up, 12 in 2024-08-31T00:00:16.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:16.665 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:16.767 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:16.785 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-31T00:00:16.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:16 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 4583/109929 objects degraded (4.169%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:16 smithi136 ceph-mon[20207]: osdmap e1019: 12 total, 11 up, 12 in 2024-08-31T00:00:16.796 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:16.898 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:16.933 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:16.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:17.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:17.044 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:17.045 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.067 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:17.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:17.169 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:17.180 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:17.199 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:17.301 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:17.327 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:17.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.466 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:17.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:17.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:17 smithi173 ceph-mon[20554]: pgmap v2250: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 4252/99327 objects degraded (4.281%); 7741/99327 objects misplaced (7.793%) 2024-08-31T00:00:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:17 smithi173 ceph-mon[20554]: 9.1 scrub starts 2024-08-31T00:00:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:17 smithi173 ceph-mon[20554]: 9.1 scrub ok 2024-08-31T00:00:17.592 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:17 smithi106 ceph-mon[16603]: pgmap v2250: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 4252/99327 objects degraded (4.281%); 7741/99327 objects misplaced (7.793%) 2024-08-31T00:00:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:17 smithi106 ceph-mon[16603]: 9.1 scrub starts 2024-08-31T00:00:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:17 smithi106 ceph-mon[16603]: 9.1 scrub ok 2024-08-31T00:00:17.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:17.715 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:17.745 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:17.745 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:17 smithi136 ceph-mon[20207]: pgmap v2250: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 2.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 4252/99327 objects degraded (4.281%); 7741/99327 objects misplaced (7.793%) 2024-08-31T00:00:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:17 smithi136 ceph-mon[20207]: 9.1 scrub starts 2024-08-31T00:00:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:17 smithi136 ceph-mon[20207]: 9.1 scrub ok 2024-08-31T00:00:17.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:17.844 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:17.849 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-31T00:00:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:17.939 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:17.939 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-08-31T00:00:17.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:17.970 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:17.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:18.072 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.096 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:18.097 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:18.177 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:18.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:18.218 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:18.308 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:18.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:18.341 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:18.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.471 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:18.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:18.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-31T00:00:18.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:18.595 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:18.696 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:18.717 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:18.822 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:18.853 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:18.903 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-31T00:00:18.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:18.980 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:18.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:19.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:19.107 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:19.209 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:19.225 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:19.236 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:19.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:19.362 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:19.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:19.491 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:19 smithi173 ceph-mon[20554]: pgmap v2252: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 3847/91989 objects degraded (4.182%); 7191/91989 objects misplaced (7.817%) 2024-08-31T00:00:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:19 smithi173 ceph-mon[20554]: 9.1f scrub starts 2024-08-31T00:00:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:19 smithi173 ceph-mon[20554]: 9.1f scrub ok 2024-08-31T00:00:19.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:19 smithi106 ceph-mon[16603]: pgmap v2252: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 3847/91989 objects degraded (4.182%); 7191/91989 objects misplaced (7.817%) 2024-08-31T00:00:19.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:19 smithi106 ceph-mon[16603]: 9.1f scrub starts 2024-08-31T00:00:19.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:19 smithi106 ceph-mon[16603]: 9.1f scrub ok 2024-08-31T00:00:19.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:19.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:19.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:19.743 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:19.787 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:19.787 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:19.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:19 smithi136 ceph-mon[20207]: pgmap v2252: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 3847/91989 objects degraded (4.182%); 7191/91989 objects misplaced (7.817%) 2024-08-31T00:00:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:19 smithi136 ceph-mon[20207]: 9.1f scrub starts 2024-08-31T00:00:19.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:19 smithi136 ceph-mon[20207]: 9.1f scrub ok 2024-08-31T00:00:19.845 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:19.866 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:19.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:19.968 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:19.998 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:20.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:20.130 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:20.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:20.259 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:20.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:20.390 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.391 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:20.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:20.518 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:20 smithi173 ceph-mon[20554]: 9.2 scrub starts 2024-08-31T00:00:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:20 smithi173 ceph-mon[20554]: 9.2 scrub ok 2024-08-31T00:00:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:20 smithi106 ceph-mon[16603]: 9.2 scrub starts 2024-08-31T00:00:20.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:20 smithi106 ceph-mon[16603]: 9.2 scrub ok 2024-08-31T00:00:20.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:20.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-31T00:00:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.642 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:20.642 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:20.743 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:20.769 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:20.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:20 smithi136 ceph-mon[20207]: 9.2 scrub starts 2024-08-31T00:00:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:20 smithi136 ceph-mon[20207]: 9.2 scrub ok 2024-08-31T00:00:20.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:20.894 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:20.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:20.936 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:20.995 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:21.018 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:21.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:21.141 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:21.243 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:21.266 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:21.369 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:21.393 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:21.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:21.520 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:21.523 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:21.523 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:21 smithi106 ceph-mon[16603]: pgmap v2253: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 3499/84192 objects degraded (4.156%); 6631/84192 objects misplaced (7.876%) 2024-08-31T00:00:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:21 smithi106 ceph-mon[16603]: 9.23 scrub starts 2024-08-31T00:00:21.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:21 smithi106 ceph-mon[16603]: 9.23 scrub ok 2024-08-31T00:00:21.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:21 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3499/84192 objects degraded (4.156%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:21.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:21.652 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:21.754 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:21.783 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:21 smithi136 ceph-mon[20207]: pgmap v2253: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 3499/84192 objects degraded (4.156%); 6631/84192 objects misplaced (7.876%) 2024-08-31T00:00:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:21 smithi136 ceph-mon[20207]: 9.23 scrub starts 2024-08-31T00:00:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:21 smithi136 ceph-mon[20207]: 9.23 scrub ok 2024-08-31T00:00:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:21 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3499/84192 objects degraded (4.156%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:21 smithi173 ceph-mon[20554]: pgmap v2253: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.27k op/s; 3499/84192 objects degraded (4.156%); 6631/84192 objects misplaced (7.876%) 2024-08-31T00:00:21.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:21 smithi173 ceph-mon[20554]: 9.23 scrub starts 2024-08-31T00:00:21.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:21 smithi173 ceph-mon[20554]: 9.23 scrub ok 2024-08-31T00:00:21.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:21 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3499/84192 objects degraded (4.156%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:21.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:21.907 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:21.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:22.009 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:22.037 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:22.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:22.170 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.172 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:22.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-31T00:00:22.273 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.298 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:22.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:22.340 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-31T00:00:22.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:22.423 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:22.514 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:22.526 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:22.548 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2720831189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:22 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2720831189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:22 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:22.649 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:22.676 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:22.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:22.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2720831189' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:22 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.812 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:22.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:22.813 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:22.814 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:22.814 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:22.814 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:22.814 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:22.913 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:22.937 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:22.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:23.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:23.069 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:23.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:23.083 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:23.083 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:23.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:23.197 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:23.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:23.324 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:23.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:23.455 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:23.538 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:23.538 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-31T00:00:23.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:23.570 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:23.570 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:23.581 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:23.597 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:23.597 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:23.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: pgmap v2254: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 3499/80073 objects degraded (4.370%); 6173/80073 objects misplaced (7.709%) 2024-08-31T00:00:23.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-31T00:00:23.597 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:23.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: osdmap e1020: 12 total, 11 up, 12 in 2024-08-31T00:00:23.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:23.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:23.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:23.598 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:23 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:23.628 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:23.628 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:23.655 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:23.655 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:23.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:23.685 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:23.685 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:23.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: pgmap v2254: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 3499/80073 objects degraded (4.370%); 6173/80073 objects misplaced (7.709%) 2024-08-31T00:00:23.690 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-31T00:00:23.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:23.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: osdmap e1020: 12 total, 11 up, 12 in 2024-08-31T00:00:23.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:23.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:23.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:23.691 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:23 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:23.710 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:23.712 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:23.712 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:23.713 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:23.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: pgmap v2254: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.24k op/s; 3499/80073 objects degraded (4.370%); 6173/80073 objects misplaced (7.709%) 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: osdmap e1020: 12 total, 11 up, 12 in 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:23.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:23 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:23.840 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:23.841 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:23.842 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:23.876 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:23.876 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:23.906 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:23.907 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:23.935 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:23.936 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:23.944 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:23.971 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:23.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:24.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:24.098 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:24.184 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:24.184 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-08-31T00:00:24.200 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:24.222 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:24.297 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-31T00:00:24.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:24.368 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:24.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:24.496 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:24.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:24 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:24.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:24 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:24.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:24 smithi106 ceph-mon[16603]: 1.1b scrub starts 2024-08-31T00:00:24.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:24 smithi106 ceph-mon[16603]: 1.1b scrub ok 2024-08-31T00:00:24.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:24 smithi106 ceph-mon[16603]: osdmap e1021: 12 total, 11 up, 12 in 2024-08-31T00:00:24.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3166476832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:24.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:24 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:24.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:24 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:24.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:24 smithi173 ceph-mon[20554]: 1.1b scrub starts 2024-08-31T00:00:24.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:24 smithi173 ceph-mon[20554]: 1.1b scrub ok 2024-08-31T00:00:24.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:24 smithi173 ceph-mon[20554]: osdmap e1021: 12 total, 11 up, 12 in 2024-08-31T00:00:24.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3166476832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:24.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:24.623 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:24.624 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:24.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:24.753 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:24 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:24 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:24 smithi136 ceph-mon[20207]: 1.1b scrub starts 2024-08-31T00:00:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:24 smithi136 ceph-mon[20207]: 1.1b scrub ok 2024-08-31T00:00:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:24 smithi136 ceph-mon[20207]: osdmap e1021: 12 total, 11 up, 12 in 2024-08-31T00:00:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3166476832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:24.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:24.881 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:24.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:24.983 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:25.011 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:25.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:25.148 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:25.228 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86701 7.5 7:aeb8975f:::benchmark_data_smithi173_129296_object86700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=4 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776359+0000", 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44849049599998, 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.525555945000001, 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.229 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86701 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776359+0000", 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776359+0000", 2024-08-31T00:00:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776361+0000", 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1409999999999999e-06 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776384+0000", 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.349e-05 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.231 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776400+0000", 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5396e-05 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776408+0000", 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3420000000000008e-06 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.293759+0000", 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.517351353 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.232 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.301915+0000", 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0081552229999999996 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86740 7.5 7:ae1ce8c3:::benchmark_data_smithi173_129296_object86739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776429+0000", 2024-08-31T00:00:25.233 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44842024399998, 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.599940781000001, 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86740 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.234 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776429+0000", 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776429+0000", 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776429+0000", 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.89e-07 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.235 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776444+0000", 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4876e-05 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776450+0000", 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0750000000000003e-06 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776454+0000", 2024-08-31T00:00:25.236 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6030000000000001e-06 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.302640+0000", 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.526186638 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.312142+0000", 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0095015849999999999 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.237 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.325632+0000", 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013489663000000001 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374800+0000", 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.049168346000000002 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374828+0000", 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7585000000000001e-05 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.238 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374853+0000", 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4899e-05 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.374893+0000", 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0710999999999997e-05 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376370+0000", 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001476411 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.239 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86786 7.5 7:aacb3b1b:::benchmark_data_smithi173_129296_object86785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776459+0000", 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44839010999999, 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.601068818, 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.240 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86786 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776459+0000", 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776459+0000", 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.241 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776459+0000", 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.53e-07 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776474+0000", 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4507e-05 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776479+0000", 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7899999999999999e-06 2024-08-31T00:00:25.242 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776483+0000", 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.002e-06 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.334845+0000", 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.558362949999999 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.243 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.342691+0000", 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0078453540000000006 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.354770+0000", 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012079608 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376431+0000", 2024-08-31T00:00:25.244 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021660723 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376447+0000", 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6036999999999999e-05 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376492+0000", 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4329999999999997e-05 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.245 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.376526+0000", 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4403999999999998e-05 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377528+0000", 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00100186 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86883 7.5 7:aef44bcd:::benchmark_data_smithi173_129296_object86882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.246 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776487+0000", 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44836210899996, 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.625095855, 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86883 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.247 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776487+0000", 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776487+0000", 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776487+0000", 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2999999999999999e-07 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.248 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776502+0000", 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4394e-05 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776505+0000", 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8539999999999999e-06 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776508+0000", 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9349999999999999e-06 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.249 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377638+0000", 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.601129986 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377740+0000", 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000101199 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.377847+0000", 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107093 2024-08-31T00:00:25.250 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.384770+0000", 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0069229469999999996 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386449+0000", 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00167946 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.400138+0000", 2024-08-31T00:00:25.251 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.013688735 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.400177+0000", 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.892e-05 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.401583+0000", 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001406102 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.252 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86907 7.5 7:a6e37ca7:::benchmark_data_smithi173_129296_object86906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776512+0000", 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44833684599996, 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.626529162000001, 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.253 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86907 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776512+0000", 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.254 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776512+0000", 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776512+0000", 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.36e-07 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776567+0000", 2024-08-31T00:00:25.255 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4041e-05 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776572+0000", 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6389999999999997e-06 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776576+0000", 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6069999999999999e-06 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.256 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.380129+0000", 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.6035532 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.380233+0000", 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010365300000000001 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.257 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.257 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.380332+0000", 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9827999999999994e-05 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386463+0000", 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0061304899999999997 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.259 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386503+0000", 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9678000000000002e-05 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.401641+0000", 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.015137904000000001 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.401701+0000", 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0798000000000003e-05 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.260 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403041+0000", 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013400879999999999 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86923 7.5 7:a7631e73:::benchmark_data_smithi173_129296_object86922:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776581+0000", 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44826852100005, 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.634007578, 2024-08-31T00:00:25.261 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86923 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776581+0000", 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776581+0000", 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776581+0000", 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1899999999999998e-07 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.263 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776613+0000", 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2311999999999998e-05 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776619+0000", 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3920000000000002e-06 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.264 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776622+0000", 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6909999999999999e-06 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.381838+0000", 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.605215756 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.381938+0000", 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9461000000000002e-05 2024-08-31T00:00:25.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.382040+0000", 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000102277 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386470+0000", 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0044304080000000003 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397958+0000", 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011487314 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.408624+0000", 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010666893 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.267 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.408675+0000", 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0816000000000003e-05 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.410588+0000", 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001912939 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.268 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86963 7.5 7:a6c5bc79:::benchmark_data_smithi173_129296_object86962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776626+0000", 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44822277699996, 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.635543245000001, 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86963 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.269 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776626+0000", 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776626+0000", 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.270 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776627+0000", 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2999999999999999e-07 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776666+0000", 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9297e-05 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776672+0000", 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6999999999999996e-06 2024-08-31T00:00:25.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776675+0000", 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5219999999999999e-06 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.383270+0000", 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.606594899999999 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.272 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.383362+0000", 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1893999999999998e-05 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.383459+0000", 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7600000000000001e-05 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388100+0000", 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0046405580000000004 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.273 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397998+0000", 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0098980519999999992 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.410658+0000", 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012660217 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.274 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.410723+0000", 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4547000000000004e-05 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.412170+0000", 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001446728 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.275 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86984 7.5 7:ae63c7dc:::benchmark_data_smithi173_129296_object86983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776680+0000", 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44816919100003, 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.642668217000001, 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86984 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.276 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776680+0000", 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776680+0000", 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776680+0000", 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.96e-07 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776714+0000", 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4e-05 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.278 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776720+0000", 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4410000000000001e-06 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776723+0000", 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.174e-06 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.384804+0000", 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.608080960000001 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.279 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.384895+0000", 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1519000000000002e-05 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.385002+0000", 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010621299999999999 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390722+0000", 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0057209729999999999 2024-08-31T00:00:25.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403094+0000", 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012371715 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.417626+0000", 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014531662000000001 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.281 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.417666+0000", 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0287999999999998e-05 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.419348+0000", 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001681976 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86989 7.5 7:a5519cde:::benchmark_data_smithi173_129296_object86988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776727+0000", 2024-08-31T00:00:25.282 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.448121861, 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.644000823000001, 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86989 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.283 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776727+0000", 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776727+0000", 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776728+0000", 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5800000000000001e-07 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.284 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776768+0000", 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0256999999999998e-05 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776773+0000", 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4260000000000002e-06 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776776+0000", 2024-08-31T00:00:25.285 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2789999999999999e-06 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386532+0000", 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.609755765999999 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386621+0000", 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8885e-05 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.286 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.386719+0000", 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8142999999999999e-05 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393565+0000", 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0068459940000000002 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403147+0000", 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.009581895 2024-08-31T00:00:25.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.419414+0000", 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.016267001 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.419455+0000", 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0911999999999997e-05 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.288 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.420728+0000", 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001273007 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:86997 7.5 7:a5e72096:::benchmark_data_smithi173_129296_object86996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776782+0000", 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44806757399999, 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.652112274, 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.289 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 86997 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776782+0000", 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.290 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776782+0000", 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776782+0000", 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4900000000000002e-07 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.291 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776813+0000", 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1024e-05 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776818+0000", 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8199999999999996e-06 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776821+0000", 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1149999999999998e-06 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.292 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388130+0000", 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.611309732 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388211+0000", 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0389000000000006e-05 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.388303+0000", 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2146999999999997e-05 2024-08-31T00:00:25.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393573+0000", 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0052695320000000004 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.403193+0000", 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.009620356 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.294 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.426952+0000", 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.023759399 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.427029+0000", 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7062000000000002e-05 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.428894+0000", 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001864449 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.295 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87015 7.5 7:accfa1ec:::benchmark_data_smithi173_129296_object87014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776825+0000", 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44802438399995, 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.653428977000001, 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.296 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87015 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776825+0000", 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776825+0000", 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776825+0000", 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3300000000000001e-07 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776855+0000", 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0103e-05 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776860+0000", 2024-08-31T00:00:25.298 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8520000000000003e-06 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776863+0000", 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1530000000000001e-06 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.389401+0000", 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.612538358 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.299 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.389492+0000", 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0104000000000003e-05 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.389584+0000", 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2057000000000003e-05 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396323+0000", 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0067397749999999999 2024-08-31T00:00:25.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407266+0000", 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010943122 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.428946+0000", 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.021679684000000001 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.428995+0000", 2024-08-31T00:00:25.301 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8427000000000002e-05 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.430254+0000", 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001259109 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87022 7.5 7:a6c9f5ab:::benchmark_data_smithi173_129296_object87021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776867+0000", 2024-08-31T00:00:25.302 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44798227299998, 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.668171694, 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87022 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.303 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776867+0000", 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776867+0000", 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776867+0000", 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2700000000000001e-07 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.304 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776899+0000", 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2437000000000001e-05 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776904+0000", 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.775e-06 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776907+0000", 2024-08-31T00:00:25.305 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.005e-06 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390754+0000", 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.613846801999999 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390847+0000", 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2781999999999994e-05 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.306 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.390943+0000", 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6514000000000005e-05 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396332+0000", 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0053883819999999997 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407305+0000", 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010973478 2024-08-31T00:00:25.307 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.442821+0000", 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035515821000000003 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.442890+0000", 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8573999999999998e-05 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.445039+0000", 2024-08-31T00:00:25.308 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021488969999999999 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87068 7.5 7:a60f23eb:::benchmark_data_smithi173_129296_object87067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776911+0000", 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44793797499995, 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.669716318999999, 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.309 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87068 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776911+0000", 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776911+0000", 2024-08-31T00:00:25.310 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776911+0000", 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776940+0000", 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8428999999999998e-05 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.311 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776945+0000", 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8119999999999998e-06 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776948+0000", 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6859999999999999e-06 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.392300+0000", 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.615352217 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.312 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.392384+0000", 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3766999999999999e-05 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.392482+0000", 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7633000000000005e-05 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397918+0000", 2024-08-31T00:00:25.313 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0054359220000000001 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407361+0000", 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0094433550000000005 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.445097+0000", 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037735746000000001 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.314 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.445166+0000", 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9369000000000006e-05 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.446627+0000", 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014611649999999999 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.315 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87085 7.5 7:a025fb98:::benchmark_data_smithi173_129296_object87084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.776952+0000", 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44789697700003, 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.671139907000001, 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87085 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.316 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776952+0000", 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.317 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776952+0000", 2024-08-31T00:00:25.318 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.318 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.318 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.776952+0000", 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1799999999999999e-07 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777026+0000", 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3590999999999994e-05 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777032+0000", 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2899999999999999e-06 2024-08-31T00:00:25.319 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777036+0000", 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.121e-06 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393604+0000", 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.616567779 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393691+0000", 2024-08-31T00:00:25.320 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7140999999999998e-05 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.393785+0000", 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3696000000000006e-05 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.397926+0000", 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041405649999999997 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.321 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.407400+0000", 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0094743429999999997 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.446703+0000", 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.039303005000000002 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.446766+0000", 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3028999999999996e-05 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.322 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.448092+0000", 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013261289999999999 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87102 7.5 7:a44aa11d:::benchmark_data_smithi173_129296_object87101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.777041+0000", 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44780787699995, 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.672397555, 2024-08-31T00:00:25.323 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87102 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777041+0000", 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777041+0000", 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777041+0000", 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6199999999999999e-07 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777074+0000", 2024-08-31T00:00:25.325 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2558999999999998e-05 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777079+0000", 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0390000000000004e-06 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777083+0000", 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0280000000000002e-06 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.326 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.394924+0000", 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.617841357 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.395027+0000", 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010244800000000001 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.395123+0000", 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6083999999999997e-05 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.327 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.398010+0000", 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028866870000000002 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.412227+0000", 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014217483 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.448151+0000", 2024-08-31T00:00:25.328 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.035924229000000002 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.448194+0000", 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2203999999999998e-05 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.449439+0000", 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012451750000000001 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87113 7.5 7:a933b630:::benchmark_data_smithi173_129296_object87112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e763)", 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:38.777087+0000", 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 586.44776187800005, 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.673579998999999, 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87113 2024-08-31T00:00:25.330 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777087+0000", 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777087+0000", 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.331 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777087+0000", 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5499999999999999e-07 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777119+0000", 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1924000000000001e-05 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777124+0000", 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8590000000000004e-06 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.332 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:38.777128+0000", 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2590000000000001e-06 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396367+0000", 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 10.619239438999999 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396459+0000", 2024-08-31T00:00:25.333 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2467999999999996e-05 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,1", 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.396557+0000", 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8011999999999996e-05 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.399549+0000", 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0029911429999999999 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.334 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.412277+0000", 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.012728868000000001 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.449514+0000", 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.037236242000000003 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.449574+0000", 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0139000000000001e-05 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.335 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:49.450667+0000", 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001093391 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDPGRecoveryDeleteReply(7.5 e833,761 7:ad5d727b:::benchmark_data_smithi173_129296_object8348:head,775'5941)", 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:53:39.158185+0000", 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 406.066664597, 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55608631399999997, 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.336 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158185+0000", 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158185+0000", 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.337 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158185+0000", 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5700000000000002e-07 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.158189+0000", 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.89e-06 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.712823+0000", 2024-08-31T00:00:25.338 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.55463346099999999 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.712834+0000", 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1821000000000001e-05 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.712901+0000", 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.6729000000000004e-05 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.339 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:53:39.714271+0000", 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001369756 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.28405.0:91166 9.1d e984/978 9:b9e063a0:::benchmark_data_smithi173_160189_object91165:head v 984'4445, mlcod=983'4443)", 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:00.454158+0000", 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 84.770690985000002, 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.63056721800000004, 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.340 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:00.454158+0000", 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:00.454158+0000", 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.341 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:00.454159+0000", 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1500000000000004e-07 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:00.454242+0000", 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3152000000000006e-05 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.080999+0000", 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.62675706099999995 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.342 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.081006+0000", 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2969999999999998e-06 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.082180+0000", 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011734149999999999 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.082203+0000", 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3357e-05 2024-08-31T00:00:25.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.084679+0000", 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024754180000000001 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.084725+0000", 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6802999999999999e-05 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.344 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:102683 9.8 9:1734cc31:::benchmark_data_smithi173_160189_object679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996)", 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:15.701042+0000", 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 69.523807309000006, 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58673244400000002, 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102683 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.345 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701042+0000", 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701042+0000", 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701044+0000", 2024-08-31T00:00:25.346 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0379999999999998e-06 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701059+0000", 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4762e-05 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701070+0000", 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1284000000000001e-05 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.347 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701075+0000", 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2039999999999996e-06 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701140+0000", 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4627000000000003e-05 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701253+0000", 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000113097 2024-08-31T00:00:25.348 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.701367+0000", 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011372 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.703042+0000", 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016750840000000001 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.703731+0000", 2024-08-31T00:00:25.349 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00068955700000000004 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:16.287680+0000", 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58394865500000004 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:16.287740+0000", 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0640999999999998e-05 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.350 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:16.287774+0000", 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3775000000000001e-05 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:102684 9.8 9:129456f2:::benchmark_data_smithi173_160189_object680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996)", 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:15.705957+0000", 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 69.518891796999995, 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58198216199999997, 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:25.351 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 102684 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.705957+0000", 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.352 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.705957+0000", 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.705958+0000", 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0389999999999999e-06 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.353 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.705971+0000", 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2918000000000001e-05 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.705983+0000", 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1447000000000001e-05 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.705988+0000", 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2320000000000001e-06 2024-08-31T00:00:25.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.706029+0000", 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1312e-05 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.706103+0000", 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4187e-05 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.706174+0000", 2024-08-31T00:00:25.355 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0146000000000002e-05 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.707624+0000", 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0014502429999999999 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:15.707742+0000", 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000118136 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.356 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:16.287833+0000", 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.58009061399999995 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:16.287893+0000", 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0154e-05 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:16.287939+0000", 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6733999999999997e-05 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.357 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.358 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.358 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:25.358 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:25.358 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-31T00:00:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.359 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:25.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:25.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:25.466 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:25.489 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:25.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:25.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:25.606 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:25 smithi173 ceph-mon[20554]: pgmap v2257: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 3188/76359 objects degraded (4.175%); 6027/76359 objects misplaced (7.893%) 2024-08-31T00:00:25.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:25 smithi106 ceph-mon[16603]: pgmap v2257: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 3188/76359 objects degraded (4.175%); 6027/76359 objects misplaced (7.893%) 2024-08-31T00:00:25.613 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.614 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:25.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.743 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:25.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:25 smithi136 ceph-mon[20207]: pgmap v2257: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 983 op/s; 3188/76359 objects degraded (4.175%); 6027/76359 objects misplaced (7.893%) 2024-08-31T00:00:25.844 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:25.869 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:25.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:25.971 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.004 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:26.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:26.020 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:26.020 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:26.021 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:26.105 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:26.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:26.156 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-31T00:00:26.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:26.270 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:26.372 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:26.394 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:26.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:26.497 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:26.517 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:26.618 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:26 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:26.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:26 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:26.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:26 smithi173 ceph-mon[20554]: osdmap e1022: 12 total, 11 up, 12 in 2024-08-31T00:00:26.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:26 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 3188/76359 objects degraded (4.175%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:26.619 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:26 smithi173 ceph-mon[20554]: osdmap e1023: 12 total, 11 up, 12 in 2024-08-31T00:00:26.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:26.644 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:26.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:26.771 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:26 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:26 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:26 smithi136 ceph-mon[20207]: osdmap e1022: 12 total, 11 up, 12 in 2024-08-31T00:00:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:26 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 3188/76359 objects degraded (4.175%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:26 smithi136 ceph-mon[20207]: osdmap e1023: 12 total, 11 up, 12 in 2024-08-31T00:00:26.807 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:26 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:26.807 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:26 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:26.807 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:26 smithi106 ceph-mon[16603]: osdmap e1022: 12 total, 11 up, 12 in 2024-08-31T00:00:26.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:26 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 3188/76359 objects degraded (4.175%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:26.808 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:26 smithi106 ceph-mon[16603]: osdmap e1023: 12 total, 11 up, 12 in 2024-08-31T00:00:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:26.848 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:26.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-08-31T00:00:26.872 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:26.917 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:26.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:26.971 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-31T00:00:27.018 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:27.047 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.28405.0:185570 9.b e1023/966 9:d6b8a982:::benchmark_data_smithi173_160189_object83566:head v 1023'7631, mlcod=1023'7628)", 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-31T00:00:27.050013+0000", 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.0014101420000000001, 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001429112, 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:27.052 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050013+0000", 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050013+0000", 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:27.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050015+0000", 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5340000000000001e-06 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050022+0000", 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6529999999999994e-06 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050042+0000", 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9490000000000001e-05 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050049+0000", 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.029e-06 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050144+0000", 2024-08-31T00:00:27.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5178000000000003e-05 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050169+0000", 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4763999999999999e-05 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:185572 9.9 9:97c5053c:::benchmark_data_smithi173_160189_object83568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1023)", 2024-08-31T00:00:27.056 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-31T00:00:27.050528+0000", 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 0.00089567300000000004, 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00095054199999999997, 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "waiting for sub ops", 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 185572 2024-08-31T00:00:27.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050528+0000", 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:27.058 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050528+0000", 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050529+0000", 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4020000000000001e-06 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:27.059 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050535+0000", 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8869999999999997e-06 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050549+0000", 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3991999999999999e-05 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050556+0000", 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.8360000000000003e-06 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.060 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050611+0000", 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4917000000000003e-05 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050718+0000", 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000107439 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:27.050816+0000", 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7640999999999996e-05 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:27.061 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: ], 2024-08-31T00:00:27.062 INFO:teuthology.orchestra.run.smithi173.stdout: "num_ops": 2 2024-08-31T00:00:27.062 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-31T00:00:27.149 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:27.187 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:27.259 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:27.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.321 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:27.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:27.422 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:27.449 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:27.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:27.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-31T00:00:27.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:27.579 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:27.681 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:27.684 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:27 smithi106 ceph-mon[16603]: pgmap v2259: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s; 2824/66573 objects degraded (4.242%); 5261/66573 objects misplaced (7.903%) 2024-08-31T00:00:27.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:27.766 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:27 smithi173 ceph-mon[20554]: pgmap v2259: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s; 2824/66573 objects degraded (4.242%); 5261/66573 objects misplaced (7.903%) 2024-08-31T00:00:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:27 smithi136 ceph-mon[20207]: pgmap v2259: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 978 op/s; 2824/66573 objects degraded (4.242%); 5261/66573 objects misplaced (7.903%) 2024-08-31T00:00:27.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-31T00:00:27.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:27.831 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:27.933 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:27.960 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:27.961 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:28.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:28.092 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:28.101 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:28.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:28.233 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:28.251 INFO:teuthology.orchestra.run.smithi106.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:28.251 INFO:teuthology.orchestra.run.smithi106.stderr:0 2024-08-31T00:00:28.251 INFO:teuthology.orchestra.run.smithi106.stderr:1 2024-08-31T00:00:28.251 INFO:teuthology.orchestra.run.smithi106.stderr:2 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:abort 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:assert 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:bench [] [] [] [] 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs debug_inject_read_zeros 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs files list 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:bluefs stats 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:bluestore allocator dump block 2024-08-31T00:00:28.252 INFO:teuthology.orchestra.run.smithi106.stderr:Error EINVAL: invalid command 2024-08-31T00:00:28.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:28.362 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:28.464 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:28.490 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:28.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:28.521 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:28.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:28 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-31T00:00:28.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:28 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-31T00:00:28.592 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:28.615 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:28.647 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:28 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-31T00:00:28.647 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:28 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-31T00:00:28.717 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:28.740 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:28 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-31T00:00:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:28 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-31T00:00:28.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:28.868 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:28.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:28.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.006 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:29.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:29.107 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:29.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:29.155 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:29.239 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:29.268 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:29.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:29.375 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-31T00:00:29.402 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:29.504 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:29.527 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:29.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:29.646 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-31T00:00:29.646 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [], 2024-08-31T00:00:29.646 INFO:teuthology.orchestra.run.smithi173.stdout: "complaint_time": 30, 2024-08-31T00:00:29.646 INFO:teuthology.orchestra.run.smithi173.stdout: "num_blocked_ops": 0 2024-08-31T00:00:29.646 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-31T00:00:29.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:29 smithi173 ceph-mon[20554]: pgmap v2261: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s; 2472/59859 objects degraded (4.130%); 4749/59859 objects misplaced (7.934%) 2024-08-31T00:00:29.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:29.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:29.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:29 smithi173 ceph-mon[20554]: osdmap e1024: 12 total, 11 up, 12 in 2024-08-31T00:00:29.647 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1167502619' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:29.654 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:29.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:29 smithi106 ceph-mon[16603]: pgmap v2261: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s; 2472/59859 objects degraded (4.130%); 4749/59859 objects misplaced (7.934%) 2024-08-31T00:00:29.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:29.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:29.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:29 smithi106 ceph-mon[16603]: osdmap e1024: 12 total, 11 up, 12 in 2024-08-31T00:00:29.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1167502619' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:29.756 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:29.759 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-08-31T00:00:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.786 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:29.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:29.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:29 smithi136 ceph-mon[20207]: pgmap v2261: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s; 2472/59859 objects degraded (4.130%); 4749/59859 objects misplaced (7.934%) 2024-08-31T00:00:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:29 smithi136 ceph-mon[20207]: osdmap e1024: 12 total, 11 up, 12 in 2024-08-31T00:00:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1167502619' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:29.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:29.914 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:29.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:30.016 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.043 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:30.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:30.145 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.174 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:30.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:30.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:30.300 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:30.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:30.433 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:30.443 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-31T00:00:30.534 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:30.562 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.563 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:30.664 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.686 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:30.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:30.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:30 smithi106 ceph-mon[16603]: 9.1b scrub starts 2024-08-31T00:00:30.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:30 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:30.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1167502619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:30.716 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:30 smithi106 ceph-mon[16603]: osdmap e1025: 12 total, 11 up, 12 in 2024-08-31T00:00:30.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:30.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:30 smithi136 ceph-mon[20207]: 9.1b scrub starts 2024-08-31T00:00:30.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:30 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:30.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1167502619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:30.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:30 smithi136 ceph-mon[20207]: osdmap e1025: 12 total, 11 up, 12 in 2024-08-31T00:00:30.793 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.819 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:30.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:30 smithi173 ceph-mon[20554]: 9.1b scrub starts 2024-08-31T00:00:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:30 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1167502619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:30 smithi173 ceph-mon[20554]: osdmap e1025: 12 total, 11 up, 12 in 2024-08-31T00:00:30.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:30.943 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:30.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:31.045 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:31.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:31.174 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.209 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:31.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:31.310 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.339 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:31.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:31.408 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-31T00:00:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-31T00:00:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-31T00:00:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-31T00:00:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.408 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:87834 7.e 7:7ff39eeb:::benchmark_data_smithi173_129296_object718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-08-31T00:00:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:50.625937+0000", 2024-08-31T00:00:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 580.779717108, 2024-08-31T00:00:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36790855099999997, 2024-08-31T00:00:31.409 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:31.412 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 87834 2024-08-31T00:00:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.413 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625937+0000", 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625937+0000", 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.414 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625938+0000", 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5700000000000002e-07 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.415 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625942+0000", 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.393e-06 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625951+0000", 2024-08-31T00:00:31.416 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8600000000000005e-06 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.625957+0000", 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5500000000000002e-06 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.417 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627053+0000", 2024-08-31T00:00:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001096334 2024-08-31T00:00:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.418 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627171+0000", 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00011757899999999999 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.627278+0000", 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010718299999999999 2024-08-31T00:00:31.419 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.629558+0000", 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022799309999999998 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.420 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.630384+0000", 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00082581199999999999 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993774+0000", 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.363390306 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.421 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993807+0000", 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3179999999999997e-05 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993846+0000", 2024-08-31T00:00:31.422 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8665999999999999e-05 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27487.0:87834 7.e e771/725 ondisk, result = 0)", 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:50:50.634204+0000", 2024-08-31T00:00:31.423 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 580.77145014500002, 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35964531900000002, 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.424 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634204+0000", 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634204+0000", 2024-08-31T00:00:31.425 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634205+0000", 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3300000000000003e-07 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.426 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.634208+0000", 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9459999999999998e-06 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993718+0000", 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.359510361 2024-08-31T00:00:31.427 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993722+0000", 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0999999999999997e-06 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.428 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993764+0000", 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1885999999999998e-05 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993769+0000", 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2140000000000004e-06 2024-08-31T00:00:31.429 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:50:50.993850+0000", 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0378999999999997e-05 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.430 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27487.0:123065 7.1 e792/725 7:8e5db808:::benchmark_data_smithi173_129296_object35949:head v 792'7770, mlcod=792'7767)", 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:51:54.487833+0000", 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 516.91782143700004, 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64708236699999999, 2024-08-31T00:00:31.431 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487833+0000", 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.432 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487833+0000", 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.433 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487834+0000", 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9999999999999996e-07 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:54.487840+0000", 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2280000000000003e-06 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.434 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130467+0000", 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64262676900000004 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130489+0000", 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2246e-05 2024-08-31T00:00:31.435 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130670+0000", 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00018123 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.436 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.130713+0000", 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2786e-05 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.134863+0000", 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0041495330000000004 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.437 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:51:55.134915+0000", 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2775000000000002e-05 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.438 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27487.0:127005 7.e 7:7c88bf46:::benchmark_data_smithi173_129296_object39889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:52:02.842372+0000", 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 508.563282044, 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.363980886, 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.439 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27487", 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3915389041", 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 127005 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842372+0000", 2024-08-31T00:00:31.440 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842372+0000", 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.441 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842374+0000", 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5570000000000001e-06 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842381+0000", 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3799999999999996e-06 2024-08-31T00:00:31.442 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842395+0000", 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3703000000000001e-05 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.443 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842405+0000", 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5689999999999994e-06 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842460+0000", 2024-08-31T00:00:31.444 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5458000000000003e-05 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842547+0000", 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6600000000000004e-05 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.445 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,9", 2024-08-31T00:00:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.842621+0000", 2024-08-31T00:00:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3875000000000007e-05 2024-08-31T00:00:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.446 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.846407+0000", 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0037859090000000001 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:02.846902+0000", 2024-08-31T00:00:31.447 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000495124 2024-08-31T00:00:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.448 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206262+0000", 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.35936010899999998 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206314+0000", 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1866000000000001e-05 2024-08-31T00:00:31.449 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:52:03.206353+0000", 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9736e-05 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.450 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36149 8.6 e860/854)", 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:24.055209+0000", 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 367.35044595599999, 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41090775600000001, 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.451 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-31T00:00:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055209+0000", 2024-08-31T00:00:31.452 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055209+0000", 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.453 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055210+0000", 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0380000000000001e-06 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.055243+0000", 2024-08-31T00:00:31.454 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3021000000000001e-05 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.465880+0000", 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.410637788 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.455 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.465894+0000", 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3969e-05 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.456 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.466085+0000", 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00019035300000000001 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:24.466116+0000", 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1587000000000001e-05 2024-08-31T00:00:31.457 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36527 8.5 8:a4f61870:::benchmark_data_smithi173_143816_object36526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.366820+0000", 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.03883478099999, 2024-08-31T00:00:31.458 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42580382900000002, 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:31.459 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36527 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366820+0000", 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.460 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366820+0000", 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366821+0000", 2024-08-31T00:00:31.461 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.5700000000000001e-07 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366840+0000", 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9429e-05 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.462 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366849+0000", 2024-08-31T00:00:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.3600000000000002e-06 2024-08-31T00:00:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.463 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.464 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.466 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366854+0000", 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4880000000000001e-06 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366888+0000", 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4350000000000001e-05 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.366994+0000", 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010563700000000001 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.467 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,4", 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.367078+0000", 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3735999999999999e-05 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.367886+0000", 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00080812600000000005 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368184+0000", 2024-08-31T00:00:31.468 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00029837400000000001 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792529+0000", 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.42434479800000002 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792588+0000", 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8922000000000003e-05 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.469 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792624+0000", 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5751999999999999e-05 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.27853.0:36527 8.5 e862/840 ondisk, result = 0)", 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.368528+0000", 2024-08-31T00:00:31.470 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.03712622099999, 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.424101687, 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368528+0000", 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368528+0000", 2024-08-31T00:00:31.471 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368529+0000", 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9100000000000004e-07 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368532+0000", 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8169999999999999e-06 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.472 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792461+0000", 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.423929155 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792467+0000", 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.8019999999999997e-06 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792512+0000", 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5624999999999998e-05 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.473 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792521+0000", 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1540000000000008e-06 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.792630+0000", 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000108643 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.474 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop(client.27853.0:36528 8.7 e862/854 8:ed4e2712:::benchmark_data_smithi173_143816_object36527:head v 862'2374, mlcod=845'96)", 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.368738+0000", 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.036916147, 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45802140800000002, 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368738+0000", 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.475 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368738+0000", 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368739+0000", 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5300000000000004e-07 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.368831+0000", 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.2077999999999995e-05 2024-08-31T00:00:31.476 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825237+0000", 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.45640643600000003 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825246+0000", 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0749999999999998e-06 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825305+0000", 2024-08-31T00:00:31.477 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9893999999999999e-05 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.825332+0000", 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6239000000000001e-05 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.826715+0000", 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0013836230000000001 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.478 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.826760+0000", 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4509999999999999e-05 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:36535 8.9 8:9fb542f2:::benchmark_data_smithi173_143816_object36534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:25.395482+0000", 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 366.01017236600001, 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41703795100000002, 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.479 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 36535 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395482+0000", 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395482+0000", 2024-08-31T00:00:31.480 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395484+0000", 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.562e-06 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395507+0000", 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.3461000000000001e-05 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.481 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395523+0000", 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6069000000000001e-05 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395531+0000", 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0039999999999999e-06 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395580+0000", 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8621000000000001e-05 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.482 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395686+0000", 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000106156 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,5", 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.395786+0000", 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9801000000000002e-05 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.396991+0000", 2024-08-31T00:00:31.483 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012049350000000001 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.397402+0000", 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00041134199999999998 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812443+0000", 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.41504124100000001 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.484 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812477+0000", 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4093e-05 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:25.812520+0000", 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2666000000000001e-05 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37809 8.2 8:45f707ad:::benchmark_data_smithi173_143816_object37808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:31.485 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:27.427735+0000", 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 363.97791917199999, 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.37169286800000001, 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37809 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.486 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427735+0000", 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427735+0000", 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427737+0000", 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1850000000000001e-06 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.487 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427757+0000", 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0934999999999999e-05 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427771+0000", 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3672999999999999e-05 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427779+0000", 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7999999999999999e-06 2024-08-31T00:00:31.488 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427829+0000", 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9795e-05 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.427918+0000", 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9624999999999997e-05 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,4", 2024-08-31T00:00:31.489 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.428017+0000", 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8312000000000003e-05 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.429169+0000", 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011526660000000001 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.456373+0000", 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.027204145999999998 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.490 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799342+0000", 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.34296861200000001 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799393+0000", 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1048999999999999e-05 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:27.799428+0000", 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5070000000000001e-05 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.491 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:52974 9.9 9:944ce93d:::benchmark_data_smithi173_160189_object52973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.119545+0000", 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 129.286109477, 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36679148, 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:31.492 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 52974 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119545+0000", 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119545+0000", 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.493 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119546+0000", 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.850000000000001e-07 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119566+0000", 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9652000000000001e-05 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119578+0000", 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2533e-05 2024-08-31T00:00:31.494 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119586+0000", 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9200000000000004e-06 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119678+0000", 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.1805999999999995e-05 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.495 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.119825+0000", 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014685300000000001 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.120027+0000", 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00020249100000000001 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121088+0000", 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001060985 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.496 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121342+0000", 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000254045 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486212+0000", 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36486941099999998 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486282+0000", 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9850000000000004e-05 2024-08-31T00:00:31.497 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486337+0000", 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4948999999999999e-05 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:52974 9.9 e965/948 ondisk, result = 0)", 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.121475+0000", 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 129.28417997400001, 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36486892399999998, 2024-08-31T00:00:31.498 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121475+0000", 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121475+0000", 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.499 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121475+0000", 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7700000000000004e-07 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.121479+0000", 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1330000000000001e-06 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.500 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486082+0000", 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.36460314399999999 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486095+0000", 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2636999999999999e-05 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486195+0000", 2024-08-31T00:00:31.501 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.9485e-05 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486204+0000", 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.8630000000000003e-06 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.486343+0000", 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000138985 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.502 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:53033 9.9 9:9754497a:::benchmark_data_smithi173_160189_object53032:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.170814+0000", 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 129.23484065100001, 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38473522700000001, 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 53033 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.503 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170814+0000", 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170814+0000", 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.504 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170815+0000", 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1400000000000001e-06 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170836+0000", 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.1291999999999999e-05 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170849+0000", 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3023999999999999e-05 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.505 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170858+0000", 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2020000000000004e-06 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.170912+0000", 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4877999999999999e-05 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.171067+0000", 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154122 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.506 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 5,11", 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.171197+0000", 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013034400000000001 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172497+0000", 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0012998510000000001 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172647+0000", 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015048199999999999 2024-08-31T00:00:31.507 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555462+0000", 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38281501800000001 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555502+0000", 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9671e-05 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.508 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555549+0000", 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7203000000000002e-05 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:53033 9.9 e965/948 ondisk, result = 0)", 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:58:22.172911+0000", 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 129.23274351500001, 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.38264409500000002, 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.509 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172911+0000", 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172911+0000", 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.510 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172912+0000", 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.5199999999999996e-07 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.172915+0000", 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8399999999999997e-06 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555370+0000", 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.382454501 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.511 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555375+0000", 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2619999999999998e-06 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555451+0000", 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6042000000000001e-05 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555458+0000", 2024-08-31T00:00:31.512 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2019999999999998e-06 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:58:22.555555+0000", 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.6595999999999995e-05 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:91158 9.23 9:c71750d6:::benchmark_data_smithi173_160189_object91157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e985)", 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:01.089654+0000", 2024-08-31T00:00:31.513 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 90.316000915999993, 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64359530899999995, 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 91158 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089654+0000", 2024-08-31T00:00:31.514 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089654+0000", 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089654+0000", 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.99e-07 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.515 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089697+0000", 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2352999999999999e-05 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089704+0000", 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8650000000000001e-06 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.516 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089710+0000", 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1370000000000001e-06 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089767+0000", 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7822000000000003e-05 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.089888+0000", 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120909 2024-08-31T00:00:31.517 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,9", 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.090026+0000", 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013724200000000001 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.091658+0000", 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0016329719999999999 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.092390+0000", 2024-08-31T00:00:31.518 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00073108699999999995 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733112+0000", 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64072216100000001 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733189+0000", 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6892999999999997e-05 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.519 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733249+0000", 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.0269000000000002e-05 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:91158 9.23 e985/984 ondisk, result = 0)", 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:01.092548+0000", 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 90.31310646, 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64071003199999998, 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.520 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.092548+0000", 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.092548+0000", 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.092549+0000", 2024-08-31T00:00:31.521 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1200000000000003e-07 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.092552+0000", 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3179999999999998e-06 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733008+0000", 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64045586700000001 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.522 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733017+0000", 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5580000000000006e-06 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733091+0000", 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3576000000000002e-05 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733104+0000", 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2585000000000001e-05 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.523 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:01.733258+0000", 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000154516 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:101968 9.23 9:c78dc1cb:::benchmark_data_smithi173_160189_object101967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e993)", 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:12.956527+0000", 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 78.449127837000006, 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.81312753100000001, 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.524 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 101968 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956527+0000", 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.525 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956527+0000", 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956528+0000", 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.226e-06 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956579+0000", 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1357000000000003e-05 2024-08-31T00:00:31.526 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956591+0000", 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1596999999999999e-05 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956598+0000", 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7610000000000002e-06 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956641+0000", 2024-08-31T00:00:31.527 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3086e-05 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956778+0000", 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013720799999999999 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,9", 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.956937+0000", 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000159481 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.528 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.958993+0000", 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0020556039999999999 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:12.959341+0000", 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00034780599999999998 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:13.769559+0000", 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.81021860599999995 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.529 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:13.769611+0000", 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1267000000000002e-05 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:13.769654+0000", 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3532000000000003e-05 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.530 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "MOSDScrubReserve(1.19,GRANT,e:1001,from:6,reservation_nonce:0)", 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:59:22.431802+0000", 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 68.973852679999993, 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61181212100000004, 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:22.431802+0000", 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.531 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:22.431802+0000", 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:22.431802+0000", 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9299999999999999e-07 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:22.431804+0000", 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.9810000000000002e-06 2024-08-31T00:00:31.532 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:23.043329+0000", 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.61152528500000003 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:23.043336+0000", 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1970000000000001e-06 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:23.043419+0000", 2024-08-31T00:00:31.533 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3398000000000003e-05 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:23.043437+0000", 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.7638999999999998e-05 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:59:23.043614+0000", 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000177228 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.534 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:183221 9.23 9:c6db6861:::benchmark_data_smithi173_160189_object81217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1021)", 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-31T00:00:24.816907+0000", 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.5887474189999997, 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.190920065, 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 183221 2024-08-31T00:00:31.535 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816907+0000", 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816907+0000", 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.536 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816908+0000", 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1599999999999999e-06 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816913+0000", 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.7650000000000001e-06 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816925+0000", 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.163e-05 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.537 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816932+0000", 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.4440000000000002e-06 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.816991+0000", 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9212999999999997e-05 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.817128+0000", 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000136687 2024-08-31T00:00:31.538 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 0,9", 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.817239+0000", 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000110741 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.818871+0000", 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001631794 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.819143+0000", 2024-08-31T00:00:31.539 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00027252500000000002 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007734+0000", 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.188590697 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007783+0000", 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8772e-05 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.540 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007827+0000", 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4637e-05 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_repop_reply(client.28405.0:183221 9.23 e1021/984 ondisk, result = 0)", 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-31T00:00:24.820196+0000", 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 6.5854588090000004, 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.187636689, 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:31.541 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.820196+0000", 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.820196+0000", 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.820196+0000", 2024-08-31T00:00:31.542 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0100000000000004e-07 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:24.820201+0000", 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.605e-06 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007669+0000", 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.18746771300000001 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:31.543 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007673+0000", 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0729999999999998e-06 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007720+0000", 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.757e-05 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007728+0000", 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7980000000000006e-06 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:31.544 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-31T00:00:25.007832+0000", 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010422899999999999 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:31.545 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-31T00:00:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.547 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:31.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:31 smithi173 ceph-mon[20554]: pgmap v2264: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 2147/51942 objects degraded (4.133%); 4170/51942 objects misplaced (8.028%) 2024-08-31T00:00:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/865005892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:31 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 2147/51942 objects degraded (4.133%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:31.647 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:31.670 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:31.772 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:31 smithi136 ceph-mon[20207]: pgmap v2264: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 2147/51942 objects degraded (4.133%); 4170/51942 objects misplaced (8.028%) 2024-08-31T00:00:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/865005892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:31 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 2147/51942 objects degraded (4.133%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:31.794 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:31.846 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:31.846 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-31T00:00:31.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:31 smithi106 ceph-mon[16603]: pgmap v2264: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 2147/51942 objects degraded (4.133%); 4170/51942 objects misplaced (8.028%) 2024-08-31T00:00:31.846 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/865005892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:31.847 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:31 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 2147/51942 objects degraded (4.133%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:31.870 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:31.870 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:31.892 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:31.892 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:31.895 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:31.915 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:31.916 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:31.918 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:31.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:31.937 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:31.938 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:31.959 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:31.960 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:31.988 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:31.988 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:32.015 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:32.015 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:32.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:32.043 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:32.044 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:32.045 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:32.072 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:32.072 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:32.105 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:32.105 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:32.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:32.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:32.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-31T00:00:32.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.324 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:32.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:32.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.451 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:32.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:32.530 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-31T00:00:32.552 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:32.580 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:32.591 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:32.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:32.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:32.807 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:32.839 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:32.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:32.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:32.974 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:32.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:33.048 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:33.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.114 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:33.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:33.163 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:33.163 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:33.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:33.245 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:33.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:33.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:33.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:33.474 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:33 smithi106 ceph-mon[16603]: pgmap v2265: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 990 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 2147/47505 objects degraded (4.520%); 3643/47505 objects misplaced (7.669%) 2024-08-31T00:00:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.505 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:33.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:33.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:33.634 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:33.736 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.769 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:33.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:33 smithi136 ceph-mon[20207]: pgmap v2265: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 990 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 2147/47505 objects degraded (4.520%); 3643/47505 objects misplaced (7.669%) 2024-08-31T00:00:33.801 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:33 smithi173 ceph-mon[20554]: pgmap v2265: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 990 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s; 2147/47505 objects degraded (4.520%); 3643/47505 objects misplaced (7.669%) 2024-08-31T00:00:33.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-31T00:00:33.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:33.899 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:33.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:34.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:34.030 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.031 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:34.132 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:34.159 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:34.188 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:34.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:34.286 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:34.388 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is set 2024-08-31T00:00:34.388 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.416 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:34.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:34.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:34.539 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2842998976' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:34.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.666 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:34.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:34.720 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2842998976' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:34.759 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [7] live_osds: [0, 2, 3, 4, 6, 8, 9, 10, 11, 1, 5] 2024-08-31T00:00:34.760 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-31T00:00:34.760 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-08-31T00:00:34.760 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-08-31T00:00:34.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:34.789 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2842998976' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:34.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:34.912 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:34.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:34.962 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-31T00:00:35.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.045 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:35.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:35.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:35.173 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.174 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:35.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:35.276 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:35.293 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:35.307 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:35.409 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:35.433 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:35.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.560 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:35.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:35.650 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:35 smithi106 ceph-mon[16603]: pgmap v2266: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 897 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 1764/43047 objects degraded (4.098%); 3464/43047 objects misplaced (8.047%) 2024-08-31T00:00:35.651 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:35 smithi106 ceph-mon[16603]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:35.651 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2842998976' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:35.651 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:35 smithi106 ceph-mon[16603]: osdmap e1026: 12 total, 11 up, 12 in 2024-08-31T00:00:35.661 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.689 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:35.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:35.789 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:35.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:35 smithi136 ceph-mon[20207]: pgmap v2266: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 897 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 1764/43047 objects degraded (4.098%); 3464/43047 objects misplaced (8.047%) 2024-08-31T00:00:35.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:35 smithi136 ceph-mon[20207]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:35.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2842998976' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:35.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:35 smithi136 ceph-mon[20207]: osdmap e1026: 12 total, 11 up, 12 in 2024-08-31T00:00:35.813 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:35 smithi173 ceph-mon[20554]: pgmap v2266: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 897 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s; 1764/43047 objects degraded (4.098%); 3464/43047 objects misplaced (8.047%) 2024-08-31T00:00:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:35 smithi173 ceph-mon[20554]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2842998976' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:35 smithi173 ceph-mon[20554]: osdmap e1026: 12 total, 11 up, 12 in 2024-08-31T00:00:35.916 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:35.946 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:35.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:36.048 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.081 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:36.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:36.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:36.209 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:36.209 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-31T00:00:36.213 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:36.240 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:36.240 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:36.270 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:36.270 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:36.298 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:36.298 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:36.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:36.325 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:36.326 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:36.339 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:36.350 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:36.351 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:36.351 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-31T00:00:36.351 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-31T00:00:36.351 INFO:teuthology.orchestra.run.smithi106.stdout: "epoch": 1027, 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "fsid": "eb936aa2-6725-11ef-bcd4-c7b262605968", 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "created": "2024-08-30T23:18:02.086618+0000", 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "modified": "2024-08-31T00:00:36.104712+0000", 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "last_up_change": "2024-08-30T23:55:58.153661+0000", 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "last_in_change": "2024-08-30T23:59:13.167369+0000", 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_num": 5806080, 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_set": [ 2024-08-31T00:00:36.352 INFO:teuthology.orchestra.run.smithi106.stdout: "nodeep-scrub", 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "noscrub", 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "pglog_hardlimit", 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "purged_snapdirs", 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_deletes", 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "sortbitwise" 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_version": 29, 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "full_ratio": 0.94999998807907104, 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_snapshot": "", 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_max": 9, 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "max_osd": 12, 2024-08-31T00:00:36.353 INFO:teuthology.orchestra.run.smithi106.stdout: "require_min_compat_client": "luminous", 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "min_compat_client": "luminous", 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "require_osd_release": "squid", 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "allow_crimson": false, 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "pools": [ 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 1, 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": ".mgr", 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:21:09.548123+0000", 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-31T00:00:36.354 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 31, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 31, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 31, 2024-08-31T00:00:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 31, 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 31, 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "1005", 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-31T00:00:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "877", 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-31T00:00:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-31T00:00:36.358 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-31T00:00:36.359 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-31T00:00:36.359 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-31T00:00:36.359 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-31T00:00:36.359 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-31T00:00:36.359 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-31T00:00:36.359 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-31T00:00:36.360 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-31T00:00:36.360 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-31T00:00:36.360 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "options": { 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_max": 32, 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_min": 1 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "mgr": {} 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 1.6100000143051147, 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 1.6100000143051147, 2024-08-31T00:00:36.361 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 1.7699999809265137, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 1.7699999809265137, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.68016940355300903, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.81999999284744263, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.68000000715255737 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "pool": 9, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_name": "unique_pool_7", 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "create_time": "2024-08-30T23:57:35.187734+0000", 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "flags": 1, 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "flags_names": "hashpspool", 2024-08-31T00:00:36.362 INFO:teuthology.orchestra.run.smithi106.stdout: "type": 1, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 3, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "min_size": 2, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_rule": 0, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_count": 0, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_target": 0, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_barrier": 0, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "is_stretch_pool": false, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "object_hash": 2, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_autoscale_mode": "off", 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num": 36, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num": 36, 2024-08-31T00:00:36.363 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_placement_num_target": 36, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_target": 36, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_num_pending": 36, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_pg_merge_meta": { 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "source_pgid": "0.0", 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "ready_epoch": 0, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_started": 0, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_epoch_clean": 0, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "source_version": "0'0", 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "target_version": "0'0" 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_change": "968", 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend": "0", 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-31T00:00:36.364 INFO:teuthology.orchestra.run.smithi106.stdout: "last_force_op_resend_preluminous": "966", 2024-08-31T00:00:36.365 INFO:teuthology.orchestra.run.smithi106.stdout: "auid": 0, 2024-08-31T00:00:36.365 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_mode": "selfmanaged", 2024-08-31T00:00:36.365 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_seq": 0, 2024-08-31T00:00:36.367 INFO:teuthology.orchestra.run.smithi106.stdout: "snap_epoch": 0, 2024-08-31T00:00:36.367 INFO:teuthology.orchestra.run.smithi106.stdout: "pool_snaps": [], 2024-08-31T00:00:36.367 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps": "[]", 2024-08-31T00:00:36.368 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_bytes": 0, 2024-08-31T00:00:36.368 INFO:teuthology.orchestra.run.smithi106.stdout: "quota_max_objects": 0, 2024-08-31T00:00:36.368 INFO:teuthology.orchestra.run.smithi106.stdout: "tiers": [], 2024-08-31T00:00:36.368 INFO:teuthology.orchestra.run.smithi106.stdout: "tier_of": -1, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "read_tier": -1, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "write_tier": -1, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_mode": "none", 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_bytes": 0, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "target_max_objects": 0, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_flush_age": 0, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "cache_min_evict_age": 0, 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profile": "", 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_params": { 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "none" 2024-08-31T00:00:36.369 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_period": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_count": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "use_gmt_hitset": true, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "min_read_recency_for_promote": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "min_write_recency_for_promote": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_grade_decay_rate": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "hit_set_search_last_n": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "grade_table": [], 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "stripe_width": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "expected_num_objects": 0, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "fast_read": false, 2024-08-31T00:00:36.370 INFO:teuthology.orchestra.run.smithi106.stdout: "options": {}, 2024-08-31T00:00:36.371 INFO:teuthology.orchestra.run.smithi106.stdout: "application_metadata": { 2024-08-31T00:00:36.371 INFO:teuthology.orchestra.run.smithi106.stdout: "rados": {} 2024-08-31T00:00:36.371 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.371 INFO:teuthology.orchestra.run.smithi106.stdout: "read_balance": { 2024-08-31T00:00:36.371 INFO:teuthology.orchestra.run.smithi106.stdout: "score_type": "Fair distribution", 2024-08-31T00:00:36.371 INFO:teuthology.orchestra.run.smithi106.stdout: "score_acting": 2.2200000286102295, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "score_stable": 2.2200000286102295, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "optimal_score": 1.1000000238418579, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_acting": 2.440000057220459, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "raw_score_stable": 2.440000057220459, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity_weighted": 0.68016940355300903, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity": 0.81999999284744263, 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "average_primary_affinity_weighted": 0.68000000715255737 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.372 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "48096b59-4ae9-43da-a0c2-a689ad9fe2e9", 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 8, 2024-08-31T00:00:36.373 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 800, 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6802", 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6803", 2024-08-31T00:00:36.374 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6804", 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.375 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6805", 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6808", 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.376 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6809", 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.377 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6806", 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6807", 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 3938533896 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6803/3938533896", 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6805/3938533896", 2024-08-31T00:00:36.378 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6809/3938533896", 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6807/3938533896", 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0b7f3c0d-489c-48b4-91ca-fa8da6baef53", 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.379 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.3292388916015625, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 13, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 368, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 525, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 998, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 369, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6810", 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.380 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6811", 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.381 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6812", 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6813", 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.382 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6816", 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.383 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.384 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.384 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6817", 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6814", 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.385 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.387 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.387 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6815", 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2928503139 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6811/2928503139", 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6813/2928503139", 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6817/2928503139", 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6815/2928503139", 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.388 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b2453c8c-444c-4a5c-9c37-36f6f0213219", 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0, 2024-08-31T00:00:36.390 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 18, 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 863, 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.391 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6818", 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6819", 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.392 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6820", 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6821", 2024-08-31T00:00:36.393 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6824", 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6825", 2024-08-31T00:00:36.394 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6822", 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.395 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6823", 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2791969052 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6819/2791969052", 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6821/2791969052", 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6825/2791969052", 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6823/2791969052", 2024-08-31T00:00:36.396 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "d1778ea7-0e45-4646-8fd2-b0f7c9e3086f", 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.397 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 26, 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 1001, 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6826", 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.398 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6827", 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6828", 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.399 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.400 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6829", 2024-08-31T00:00:36.400 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.400 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.400 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.400 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.401 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.401 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.401 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.401 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.401 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6832", 2024-08-31T00:00:36.401 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6833", 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.402 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6830", 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.106:6831", 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2233735435 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.106:6827/2233735435", 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.106:6829/2233735435", 2024-08-31T00:00:36.403 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.106:6833/2233735435", 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.106:6831/2233735435", 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "0335aeb0-87b1-4cff-9351-c489ad188729", 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.404 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 32, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 957, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6800", 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.405 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6801", 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6802", 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6803", 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6806", 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6807", 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6804", 2024-08-31T00:00:36.408 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6805", 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2130223341 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6801/2130223341", 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6803/2130223341", 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6807/2130223341", 2024-08-31T00:00:36.409 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6805/2130223341", 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dc2279d3-fcec-4934-8a75-9befec1bbbe3", 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 566, 2024-08-31T00:00:36.410 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 593, 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 649, 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 997, 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 594, 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6808", 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.411 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6809", 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6810", 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.412 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6811", 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6814", 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.413 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6815", 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6812", 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.414 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6813", 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2739515404 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6809/2739515404", 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6811/2739515404", 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6815/2739515404", 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6813/2739515404", 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.415 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "dddb0cd9-8f57-4836-96fd-f66925b36c84", 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 45, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 996, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.416 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6816", 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6817", 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6818", 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.418 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6819", 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6822", 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.419 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6823", 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6820", 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.420 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6821", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2133537865 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6817/2133537865", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6819/2133537865", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6823/2133537865", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6821/2133537865", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.421 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "b9894fb4-6f34-47cd-81cd-9e7d9758360b", 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 0, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.1631317138671875, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 54, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 879, 2024-08-31T00:00:36.422 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 903, 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6824", 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6825", 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.423 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6826", 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.424 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6827", 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.425 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6830", 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6831", 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.426 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6828", 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.136:6829", 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 1537958513 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.136:6825/1537958513", 2024-08-31T00:00:36.427 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.136:6827/1537958513", 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.136:6831/1537958513", 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.136:6829/1537958513", 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "exists" 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "046e019e-1e33-403c-9577-757c7d546f83", 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.428 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.9682159423828125, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 62, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 994, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6800", 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.429 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6801", 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6802", 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6803", 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6806", 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6807", 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6804", 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.432 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6805", 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 526702876 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6801/526702876", 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6803/526702876", 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6807/526702876", 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6805/526702876", 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.433 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "ea9ca538-e1da-4687-8b4f-542c0fc3737c", 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 0.701446533203125, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 70, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 998, 2024-08-31T00:00:36.434 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6808", 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6809", 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.435 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6810", 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6811", 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.436 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6814", 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6815", 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.437 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6812", 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6813", 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 478092222 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.438 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.439 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.439 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6809/478092222", 2024-08-31T00:00:36.439 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6811/478092222", 2024-08-31T00:00:36.439 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6815/478092222", 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6813/478092222", 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.440 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "085c32cd-dde4-4a54-bb0d-9182075040c2", 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 78, 2024-08-31T00:00:36.441 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 998, 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6816", 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6817", 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.442 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6818", 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6819", 2024-08-31T00:00:36.443 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6822", 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.444 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6823", 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6820", 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.445 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6821", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 225763808 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6817/225763808", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6819/225763808", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6823/225763808", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6821/225763808", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.446 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "uuid": "6bd9ce25-ce61-4998-af79-26c4ebe2ce3b", 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "up": 1, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "in": 1, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "weight": 1, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_affinity": 1, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_begin": 0, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "last_clean_end": 0, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "up_from": 86, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "up_thru": 1002, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "down_at": 0, 2024-08-31T00:00:36.447 INFO:teuthology.orchestra.run.smithi106.stdout: "lost_at": 0, 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addrs": { 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6824", 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6825", 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.448 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addrs": { 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6826", 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6827", 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.449 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addrs": { 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6830", 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6831", 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.450 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addrs": { 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "addrvec": [ 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v2", 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6828", 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "type": "v1", 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "addr": "172.21.15.173:6829", 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: "nonce": 2219423084 2024-08-31T00:00:36.451 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: "public_addr": "172.21.15.173:6825/2219423084", 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: "cluster_addr": "172.21.15.173:6827/2219423084", 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_back_addr": "172.21.15.173:6831/2219423084", 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: "heartbeat_front_addr": "172.21.15.173:6829/2219423084", 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: "state": [ 2024-08-31T00:00:36.452 INFO:teuthology.orchestra.run.smithi106.stdout: "exists", 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "up" 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_xinfo": [ 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 0, 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.453 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:59:52.922686+0000", 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 1, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:59:48.805323+0000", 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 368 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.454 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 2, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:22.851688+0000", 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 3, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.455 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:22.422393+0000", 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 4, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:27.912415+0000", 2024-08-31T00:00:36.456 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.457 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.457 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.457 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 5, 2024-08-31T00:00:36.457 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.457 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.457 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:59:31.602114+0000", 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 593 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 6, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.458 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:22.416070+0000", 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 7, 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:55:09.475333+0000", 2024-08-31T00:00:36.459 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 902 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 8, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:24.952803+0000", 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.460 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 9, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:10.442044+0000", 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 10, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.461 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-30T23:59:18.808620+0000", 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "osd": 11, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "down_stamp": "0.000000", 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_probability": 0, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "laggy_interval": 0, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "features": 4540719139924082687, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "old_weight": 0, 2024-08-31T00:00:36.462 INFO:teuthology.orchestra.run.smithi106.stdout: "last_purged_snaps_scrub": "2024-08-31T00:00:11.060064+0000", 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: "dead_epoch": 0 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap": [ 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.0", 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: 7, 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: 11 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.463 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1", 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.f", 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: 9 2024-08-31T00:00:36.464 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_items": [], 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_upmap_primaries": [], 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: "pg_temp": [ 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.0", 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: 11, 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: 9 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.465 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.4", 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 11, 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 8, 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 6 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "1.1e", 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 11, 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 8, 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-31T00:00:36.466 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.5", 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: 11, 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: 5, 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: 1 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.467 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.10", 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: 4, 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: 1 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.11", 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: 3, 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: 6, 2024-08-31T00:00:36.468 INFO:teuthology.orchestra.run.smithi106.stdout: 9 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.15", 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: 8, 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: 4, 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: 5 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.18", 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.469 INFO:teuthology.orchestra.run.smithi106.stdout: 10, 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: 5, 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: 9 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.1a", 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: 3, 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: 2, 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: 1 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.470 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.1e", 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: 5, 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: 0, 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: 10 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: "pgid": "9.22", 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: "osds": [ 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: 9, 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: 4, 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: 1 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:36.471 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: ], 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "primary_temp": [], 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "blocklist": { 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3708473409": "2024-08-31T23:18:51.239661+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1031615522": "2024-08-31T23:18:51.239661+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3817969167": "2024-08-31T23:18:51.239661+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3688603346": "2024-08-31T23:18:27.701240+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6801/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1186392683": "2024-08-31T23:18:51.239661+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/3790397394": "2024-08-31T23:18:27.701240+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:6800/1549137855": "2024-08-31T23:18:27.701240+0000", 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: "172.21.15.106:0/1787333807": "2024-08-31T23:18:27.701240+0000" 2024-08-31T00:00:36.472 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "range_blocklist": {}, 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "erasure_code_profiles": { 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "default": { 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "crush-failure-domain": "osd", 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "k": "2", 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "m": "1", 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "plugin": "jerasure", 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "technique": "reed_sol_van" 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "removed_snaps_queue": [], 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "new_removed_snaps": [], 2024-08-31T00:00:36.473 INFO:teuthology.orchestra.run.smithi106.stdout: "new_purged_snaps": [], 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "crush_node_flags": {}, 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "device_class_flags": {}, 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode": { 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_enabled": false, 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_bucket_count": 0, 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "degraded_stretch_mode": 0, 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "recovering_stretch_mode": 0, 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: "stretch_mode_bucket": 0 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:36.474 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-31T00:00:36.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:36.497 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:36.513 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:36.514 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:36.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:36 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1764/43047 objects degraded (4.098%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:36.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:36 smithi106 ceph-mon[16603]: osdmap e1027: 12 total, 11 up, 12 in 2024-08-31T00:00:36.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3353309317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:36.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:36 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2274742647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-31T00:00:36.535 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:36.535 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:36.556 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:36.556 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:36.596 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:36.596 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:36.599 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:36.625 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:36.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:36.749 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:36 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1764/43047 objects degraded (4.098%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:36 smithi136 ceph-mon[20207]: osdmap e1027: 12 total, 11 up, 12 in 2024-08-31T00:00:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3353309317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:36 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2274742647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-31T00:00:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:36 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1764/43047 objects degraded (4.098%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:36 smithi173 ceph-mon[20554]: osdmap e1027: 12 total, 11 up, 12 in 2024-08-31T00:00:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3353309317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:36 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2274742647' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-08-31T00:00:36.827 INFO:teuthology.orchestra.run.smithi173.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:36.828 INFO:teuthology.orchestra.run.smithi173.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:36.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.874 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:36.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:36.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:36.998 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:36.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:37.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:37.126 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:37.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:37.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-31T00:00:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.256 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:37.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:37.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:37.381 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:37.483 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:37.513 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:37.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:37 smithi106 ceph-mon[16603]: pgmap v2268: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 1403/32562 objects degraded (4.309%); 2629/32562 objects misplaced (8.074%) 2024-08-31T00:00:37.616 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.649 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:37.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:37.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:37.782 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.784 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:37.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:37 smithi136 ceph-mon[20207]: pgmap v2268: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 1403/32562 objects degraded (4.309%); 2629/32562 objects misplaced (8.074%) 2024-08-31T00:00:37.803 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1027, 'fsid': 'eb936aa2-6725-11ef-bcd4-c7b262605968', 'created': '2024-08-30T23:18:02.086618+0000', 'modified': '2024-08-31T00:00:36.104712+0000', 'last_up_change': '2024-08-30T23:55:58.153661+0000', 'last_in_change': '2024-08-30T23:59:13.167369+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 29, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-30T23:21:09.548123+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1005', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '877', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6100000143051147, 'score_stable': 1.6100000143051147, 'optimal_score': 1.100000023841858, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.680169403553009, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.6800000071525574}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-08-30T23:57:35.187734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '968', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '966', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2200000286102295, 'score_stable': 2.2200000286102295, 'optimal_score': 1.100000023841858, 'raw_score_acting': 2.440000057220459, 'raw_score_stable': 2.440000057220459, 'primary_affinity_weighted': 0.680169403553009, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.6800000071525574}}], 'osds': [{'osd': 0, 'uuid': '48096b59-4ae9-43da-a0c2-a689ad9fe2e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 800, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6802', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6803', 'nonce': 3938533896}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6804', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6805', 'nonce': 3938533896}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6808', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6809', 'nonce': 3938533896}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6806', 'nonce': 3938533896}, {'type': 'v1', 'addr': '172.21.15.106:6807', 'nonce': 3938533896}]}, 'public_addr': '172.21.15.106:6803/3938533896', 'cluster_addr': '172.21.15.106:6805/3938533896', 'heartbeat_back_addr': '172.21.15.106:6809/3938533896', 'heartbeat_front_addr': '172.21.15.106:6807/3938533896', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '0b7f3c0d-489c-48b4-91ca-fa8da6baef53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3292388916015625, 'last_clean_begin': 13, 'last_clean_end': 368, 'up_from': 525, 'up_thru': 998, 'down_at': 369, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6810', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6811', 'nonce': 2928503139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6812', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6813', 'nonce': 2928503139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6816', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6817', 'nonce': 2928503139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6814', 'nonce': 2928503139}, {'type': 'v1', 'addr': '172.21.15.106:6815', 'nonce': 2928503139}]}, 'public_addr': '172.21.15.106:6811/2928503139', 'cluster_addr': '172.21.15.106:6813/2928503139', 'heartbeat_back_addr': '172.21.15.106:6817/2928503139', 'heartbeat_front_addr': '172.21.15.106:6815/2928503139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b2453c8c-444c-4a5c-9c37-36f6f0213219', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 863, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6818', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6819', 'nonce': 2791969052}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6820', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6821', 'nonce': 2791969052}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6824', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6825', 'nonce': 2791969052}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6822', 'nonce': 2791969052}, {'type': 'v1', 'addr': '172.21.15.106:6823', 'nonce': 2791969052}]}, 'public_addr': '172.21.15.106:6819/2791969052', 'cluster_addr': '172.21.15.106:6821/2791969052', 'heartbeat_back_addr': '172.21.15.106:6825/2791969052', 'heartbeat_front_addr': '172.21.15.106:6823/2791969052', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'd1778ea7-0e45-4646-8fd2-b0f7c9e3086f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 1001, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6826', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6827', 'nonce': 2233735435}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6828', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6829', 'nonce': 2233735435}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 2233735435}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6830', 'nonce': 2233735435}, {'type': 'v1', 'addr': '172.21.15.106:6831', 'nonce': 2233735435}]}, 'public_addr': '172.21.15.106:6827/2233735435', 'cluster_addr': '172.21.15.106:6829/2233735435', 'heartbeat_back_addr': '172.21.15.106:6833/2233735435', 'heartbeat_front_addr': '172.21.15.106:6831/2233735435', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0335aeb0-87b1-4cff-9351-c489ad188729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 957, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6801', 'nonce': 2130223341}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6802', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6803', 'nonce': 2130223341}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6807', 'nonce': 2130223341}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6804', 'nonce': 2130223341}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2130223341}]}, 'public_addr': '172.21.15.136:6801/2130223341', 'cluster_addr': '172.21.15.136:6803/2130223341', 'heartbeat_back_addr': '172.21.15.136:6807/2130223341', 'heartbeat_front_addr': '172.21.15.136:6805/2130223341', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'dc2279d3-fcec-4934-8a75-9befec1bbbe3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 566, 'last_clean_end': 593, 'up_from': 649, 'up_thru': 997, 'down_at': 594, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6808', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 2739515404}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6811', 'nonce': 2739515404}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 2739515404}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6812', 'nonce': 2739515404}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 2739515404}]}, 'public_addr': '172.21.15.136:6809/2739515404', 'cluster_addr': '172.21.15.136:6811/2739515404', 'heartbeat_back_addr': '172.21.15.136:6815/2739515404', 'heartbeat_front_addr': '172.21.15.136:6813/2739515404', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'dddb0cd9-8f57-4836-96fd-f66925b36c84', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 996, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 2133537865}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 2133537865}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 2133537865}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 2133537865}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 2133537865}]}, 'public_addr': '172.21.15.136:6817/2133537865', 'cluster_addr': '172.21.15.136:6819/2133537865', 'heartbeat_back_addr': '172.21.15.136:6823/2133537865', 'heartbeat_front_addr': '172.21.15.136:6821/2133537865', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'b9894fb4-6f34-47cd-81cd-9e7d9758360b', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.1631317138671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 879, 'down_at': 903, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 1537958513}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 1537958513}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 1537958513}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 1537958513}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 1537958513}]}, 'public_addr': '172.21.15.136:6825/1537958513', 'cluster_addr': '172.21.15.136:6827/1537958513', 'heartbeat_back_addr': '172.21.15.136:6831/1537958513', 'heartbeat_front_addr': '172.21.15.136:6829/1537958513', 'state': ['exists']}, {'osd': 8, 'uuid': '046e019e-1e33-403c-9577-757c7d546f83', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9682159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 994, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6800', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6801', 'nonce': 526702876}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6802', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6803', 'nonce': 526702876}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6806', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6807', 'nonce': 526702876}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6804', 'nonce': 526702876}, {'type': 'v1', 'addr': '172.21.15.173:6805', 'nonce': 526702876}]}, 'public_addr': '172.21.15.173:6801/526702876', 'cluster_addr': '172.21.15.173:6803/526702876', 'heartbeat_back_addr': '172.21.15.173:6807/526702876', 'heartbeat_front_addr': '172.21.15.173:6805/526702876', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'ea9ca538-e1da-4687-8b4f-542c0fc3737c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.701446533203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 998, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6808', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6809', 'nonce': 478092222}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6810', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6811', 'nonce': 478092222}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6814', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6815', 'nonce': 478092222}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6812', 'nonce': 478092222}, {'type': 'v1', 'addr': '172.21.15.173:6813', 'nonce': 478092222}]}, 'public_addr': '172.21.15.173:6809/478092222', 'cluster_addr': '172.21.15.173:6811/478092222', 'heartbeat_back_addr': '172.21.15.173:6815/478092222', 'heartbeat_front_addr': '172.21.15.173:6813/478092222', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '085c32cd-dde4-4a54-bb0d-9182075040c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 78, 'up_thru': 998, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6816', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6817', 'nonce': 225763808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6818', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6819', 'nonce': 225763808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6822', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6823', 'nonce': 225763808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6820', 'nonce': 225763808}, {'type': 'v1', 'addr': '172.21.15.173:6821', 'nonce': 225763808}]}, 'public_addr': '172.21.15.173:6817/225763808', 'cluster_addr': '172.21.15.173:6819/225763808', 'heartbeat_back_addr': '172.21.15.173:6823/225763808', 'heartbeat_front_addr': '172.21.15.173:6821/225763808', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '6bd9ce25-ce61-4998-af79-26c4ebe2ce3b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 86, 'up_thru': 1002, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6824', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6825', 'nonce': 2219423084}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6826', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6827', 'nonce': 2219423084}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6830', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6831', 'nonce': 2219423084}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6828', 'nonce': 2219423084}, {'type': 'v1', 'addr': '172.21.15.173:6829', 'nonce': 2219423084}]}, 'public_addr': '172.21.15.173:6825/2219423084', 'cluster_addr': '172.21.15.173:6827/2219423084', 'heartbeat_back_addr': '172.21.15.173:6831/2219423084', 'heartbeat_front_addr': '172.21.15.173:6829/2219423084', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:59:52.922686+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:59:48.805323+0000', 'dead_epoch': 368}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:22.851688+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:22.422393+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:27.912415+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:59:31.602114+0000', 'dead_epoch': 593}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:22.416070+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:55:09.475333+0000', 'dead_epoch': 902}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:24.952803+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:10.442044+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-30T23:59:18.808620+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-08-31T00:00:11.060064+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [6, 7, 11]}, {'pgid': '1.1', 'osds': [2, 0, 10]}, {'pgid': '1.f', 'osds': [0, 6, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.0', 'osds': [6, 11, 9]}, {'pgid': '1.4', 'osds': [11, 8, 6]}, {'pgid': '1.1e', 'osds': [11, 8, 10]}, {'pgid': '9.5', 'osds': [11, 5, 1]}, {'pgid': '9.10', 'osds': [4, 0, 1]}, {'pgid': '9.11', 'osds': [3, 6, 9]}, {'pgid': '9.15', 'osds': [8, 4, 5]}, {'pgid': '9.18', 'osds': [10, 5, 9]}, {'pgid': '9.1a', 'osds': [3, 2, 1]}, {'pgid': '9.1e', 'osds': [5, 0, 10]}, {'pgid': '9.22', 'osds': [9, 4, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.106:6801/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3708473409': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/1031615522': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3817969167': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3688603346': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6801/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1186392683': '2024-08-31T23:18:51.239661+0000', '172.21.15.106:0/3790397394': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:6800/1549137855': '2024-08-31T23:18:27.701240+0000', '172.21.15.106:0/1787333807': '2024-08-31T23:18:27.701240+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-08-31T00:00:37.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-31T00:00:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:37 smithi173 ceph-mon[20554]: pgmap v2268: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.21k op/s; 1403/32562 objects degraded (4.309%); 2629/32562 objects misplaced (8.074%) 2024-08-31T00:00:37.885 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:37.912 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:37.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:37.983 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-31T00:00:38.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.064 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:38.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:38.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:38.202 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:38.293 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:38.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:38.325 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:38.339 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:38.346 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.c/config 2024-08-31T00:00:38.354 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:38.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:38.467 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:38.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.598 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:38.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:38.699 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:38.725 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:38.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:38.851 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:38.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:38.902 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-08-31T00:00:38.902 INFO:teuthology.orchestra.run.smithi173.stdout: "size": 20, 2024-08-31T00:00:38.902 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 600, 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "ops": [ 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37699 8.d 8:b57c4cbd:::benchmark_data_smithi173_143816_object37698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121522+0000", 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77863734699997, 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93994113199999996, 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.903 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37699 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121522+0000", 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.904 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121523+0000", 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3480000000000001e-06 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121550+0000", 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7135e-05 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121561+0000", 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.081e-05 2024-08-31T00:00:38.905 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121569+0000", 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6990000000000003e-06 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3838999999999996e-05 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.906 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121654+0000", 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.1416999999999999e-05 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061408+0000", 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.93975412199999997 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061463+0000", 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4762000000000002e-05 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.907 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37724 8.d 8:bb898ecc:::benchmark_data_smithi173_143816_object37723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121591+0000", 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.778568312, 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94419919900000004, 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.908 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.909 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37724 2024-08-31T00:00:38.909 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.909 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.909 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.912 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.912 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-31T00:00:38.912 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.912 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121591+0000", 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.1699999999999998e-07 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.913 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121616+0000", 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4661999999999999e-05 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121625+0000", 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6009999999999995e-06 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121631+0000", 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.4500000000000001e-06 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.914 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121691+0000", 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9901000000000001e-05 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121695+0000", 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6849999999999999e-06 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061884+0000", 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94018891100000002 2024-08-31T00:00:38.915 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.061957+0000", 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2855999999999995e-05 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.062054+0000", 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.7165999999999998e-05 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.916 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065642+0000", 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0035878989999999999 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065656+0000", 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4899000000000001e-05 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065728+0000", 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1124999999999994e-05 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.917 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065764+0000", 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6154e-05 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065790+0000", 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6273000000000001e-05 2024-08-31T00:00:38.918 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37735 8.d 8:bd387e6f:::benchmark_data_smithi173_143816_object37734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121643+0000", 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77851572899999, 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94428094500000004, 2024-08-31T00:00:38.919 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37735 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.920 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121643+0000", 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121644+0000", 2024-08-31T00:00:38.921 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9500000000000002e-07 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121680+0000", 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6387000000000002e-05 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121688+0000", 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.5839999999999997e-06 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.922 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121694+0000", 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.3119999999999999e-06 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.923 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121745+0000", 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0837999999999997e-05 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121749+0000", 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6150000000000001e-06 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.924 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063145+0000", 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94139648399999998 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063233+0000", 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8199e-05 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063310+0000", 2024-08-31T00:00:38.925 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6536e-05 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065662+0000", 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023518760000000001 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065685+0000", 2024-08-31T00:00:38.926 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2708999999999999e-05 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065841+0000", 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015618500000000001 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065890+0000", 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.888e-05 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.927 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065924+0000", 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4745e-05 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37745 8.d 8:b0c83413:::benchmark_data_smithi173_143816_object37744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121721+0000", 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77843788400003, 2024-08-31T00:00:38.928 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94438678600000003, 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37745 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.929 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121721+0000", 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121722+0000", 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.6700000000000003e-07 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.930 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121746+0000", 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4036000000000002e-05 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121752+0000", 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2570000000000004e-06 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121756+0000", 2024-08-31T00:00:38.931 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9620000000000004e-06 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121787+0000", 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0587999999999999e-05 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121790+0000", 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3519999999999998e-06 2024-08-31T00:00:38.932 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.063974+0000", 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94218355200000004 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064046+0000", 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1983999999999994e-05 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.933 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064119+0000", 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3266e-05 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065799+0000", 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.001680186 2024-08-31T00:00:38.934 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065979+0000", 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000179552 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066033+0000", 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4085999999999997e-05 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.935 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066077+0000", 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4382000000000003e-05 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066108+0000", 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1016000000000003e-05 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.936 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37783 8.d 8:bd0ef9a3:::benchmark_data_smithi173_143816_object37782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121764+0000", 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.778394822, 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94672526099999998, 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.937 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.938 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37783 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121764+0000", 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.939 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121765+0000", 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4799999999999999e-07 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121789+0000", 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.4697000000000001e-05 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.940 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121795+0000", 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.1429999999999997e-06 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121798+0000", 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.472e-06 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121834+0000", 2024-08-31T00:00:38.941 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5528000000000003e-05 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121842+0000", 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3159999999999997e-06 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064918+0000", 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.943076154 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.942 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.064992+0000", 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.3893999999999994e-05 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065076+0000", 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.3681000000000001e-05 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.065991+0000", 2024-08-31T00:00:38.943 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00091558099999999995 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068378+0000", 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0023872390000000002 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068428+0000", 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9329999999999997e-05 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.944 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068464+0000", 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5917999999999997e-05 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068490+0000", 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5959999999999999e-05 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.945 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37794 8.d 8:b33c5833:::benchmark_data_smithi173_143816_object37793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121806+0000", 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77835316400001, 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94686707599999997, 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37794 2024-08-31T00:00:38.946 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.947 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121806+0000", 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5900000000000003e-07 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121850+0000", 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3831000000000001e-05 2024-08-31T00:00:38.948 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121856+0000", 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2670000000000003e-06 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121861+0000", 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8799999999999999e-06 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.949 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121892+0000", 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1179000000000001e-05 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121896+0000", 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3780000000000001e-06 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066144+0000", 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94424862499999995 2024-08-31T00:00:38.950 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066245+0000", 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010058800000000001 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.066373+0000", 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00012802199999999999 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.951 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068538+0000", 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021646870000000002 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068591+0000", 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3541000000000001e-05 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068604+0000", 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2607e-05 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.952 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068648+0000", 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3781e-05 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068673+0000", 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.5330999999999999e-05 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.953 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37798 8.d 8:b32c1d3f:::benchmark_data_smithi173_143816_object37797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121869+0000", 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77828984500002, 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96212161299999999, 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.954 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37798 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121869+0000", 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.955 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121870+0000", 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0800000000000001e-07 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121910+0000", 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0589000000000001e-05 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121916+0000", 2024-08-31T00:00:38.956 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.981e-06 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121920+0000", 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5719999999999999e-06 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121956+0000", 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.6510000000000001e-05 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121970+0000", 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3285000000000001e-05 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067392+0000", 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94542252100000002 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067488+0000", 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5859000000000005e-05 2024-08-31T00:00:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.067570+0000", 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.2109999999999998e-05 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068681+0000", 2024-08-31T00:00:38.959 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0011110720000000001 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074538+0000", 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0058571530000000004 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.960 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083860+0000", 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0093212599999999996 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083936+0000", 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6007999999999995e-05 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.961 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083991+0000", 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5384999999999999e-05 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37806 8.d 8:b567e58a:::benchmark_data_smithi173_143816_object37805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.121928+0000", 2024-08-31T00:00:38.962 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77823064900002, 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96731055799999999, 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37806 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.963 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121928+0000", 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.121929+0000", 2024-08-31T00:00:38.964 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.22e-07 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122024+0000", 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.5063999999999996e-05 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122031+0000", 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2060000000000001e-06 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.965 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122035+0000", 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7960000000000002e-06 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122067+0000", 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2122999999999997e-05 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.966 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122072+0000", 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.5140000000000003e-06 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068710+0000", 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.94663852100000001 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068855+0000", 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014504599999999999 2024-08-31T00:00:38.967 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.068990+0000", 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000135177 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074497+0000", 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0055070910000000004 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.968 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074576+0000", 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.9109000000000004e-05 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089138+0000", 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.014561274000000001 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089193+0000", 2024-08-31T00:00:38.969 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4864000000000002e-05 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089239+0000", 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6451e-05 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37807 8.d 8:bd227052:::benchmark_data_smithi173_143816_object37806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.970 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122043+0000", 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77811600400003, 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96736628899999999, 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37807 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.971 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.972 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122043+0000", 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1399999999999998e-07 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122079+0000", 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5790000000000001e-05 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.973 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122085+0000", 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9170000000000003e-06 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122088+0000", 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.084e-06 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122122+0000", 2024-08-31T00:00:38.974 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3738999999999997e-05 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.2359999999999998e-06 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.072882+0000", 2024-08-31T00:00:38.975 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95075315999999999 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073020+0000", 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013803900000000001 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.073142+0000", 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122043 2024-08-31T00:00:38.976 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077477+0000", 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0043347180000000004 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077486+0000", 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0759999999999992e-06 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.977 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089319+0000", 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.011832666 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089368+0000", 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.9446000000000001e-05 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.978 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089409+0000", 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1060999999999999e-05 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37812 8.d 8:bad38160:::benchmark_data_smithi173_143816_object37811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122094+0000", 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77806537100003, 2024-08-31T00:00:38.979 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.967454445, 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37812 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.980 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122094+0000", 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8599999999999999e-07 2024-08-31T00:00:38.981 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122123+0000", 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8960000000000001e-05 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122129+0000", 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.5520000000000004e-06 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.982 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122132+0000", 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.5379999999999998e-06 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122161+0000", 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9351000000000001e-05 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.983 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122166+0000", 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.3440000000000003e-06 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074606+0000", 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95243996799999997 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074746+0000", 2024-08-31T00:00:38.984 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00014037899999999999 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.074867+0000", 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000120969 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077539+0000", 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0026718330000000002 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.985 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078510+0000", 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00097061800000000002 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089459+0000", 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010949181000000001 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089509+0000", 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0547999999999998e-05 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.986 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089548+0000", 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8918000000000002e-05 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37816 8.d 8:bceb9fa7:::benchmark_data_smithi173_143816_object37815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122138+0000", 2024-08-31T00:00:38.987 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77802133900002, 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96754795299999996, 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37816 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.988 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122138+0000", 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6e-07 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.989 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122168+0000", 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9663e-05 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122172+0000", 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.155e-06 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122176+0000", 2024-08-31T00:00:38.990 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2400000000000001e-06 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122205+0000", 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9184999999999999e-05 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122212+0000", 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.1849999999999998e-06 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.991 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076169+0000", 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95395610799999997 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076291+0000", 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000122633 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.076448+0000", 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015706699999999999 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.992 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079517+0000", 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003068777 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079533+0000", 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.5656000000000001e-05 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089594+0000", 2024-08-31T00:00:38.993 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010061151000000001 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089648+0000", 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4110999999999997e-05 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.089686+0000", 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7762000000000002e-05 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:38.994 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37819 8.d 8:b27be7f0:::benchmark_data_smithi173_143816_object37818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122182+0000", 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77797754199997, 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96865621300000004, 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:38.995 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37819 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.996 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122182+0000", 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.72e-07 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122214+0000", 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1964000000000001e-05 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122218+0000", 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1649999999999999e-06 2024-08-31T00:00:38.997 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122222+0000", 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.7179999999999998e-06 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122251+0000", 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8810999999999999e-05 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122254+0000", 2024-08-31T00:00:38.998 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8410000000000002e-06 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077587+0000", 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95533246100000002 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077667+0000", 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0438000000000005e-05 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:38.999 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.077751+0000", 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.4217999999999999e-05 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079574+0000", 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0018224580000000001 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080426+0000", 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00085253 2024-08-31T00:00:39.000 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090743+0000", 2024-08-31T00:00:39.001 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010316967 2024-08-31T00:00:39.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090798+0000", 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4866e-05 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.090838+0000", 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9504e-05 2024-08-31T00:00:39.003 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37827 8.d 8:bc586ec9:::benchmark_data_smithi173_143816_object37826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122228+0000", 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77793074099998, 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.96986781300000002, 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:39.004 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37827 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.005 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122228+0000", 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122229+0000", 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8799999999999998e-07 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122263+0000", 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3946999999999999e-05 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.006 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122268+0000", 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3000000000000001e-06 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122271+0000", 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1879999999999998e-06 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122300+0000", 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8824e-05 2024-08-31T00:00:39.007 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122303+0000", 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4249999999999998e-06 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078551+0000", 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95624797399999995 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.008 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078639+0000", 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.7868000000000006e-05 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.078716+0000", 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.6747000000000002e-05 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081587+0000", 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0028711209999999999 2024-08-31T00:00:39.009 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081602+0000", 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4864e-05 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.091996+0000", 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.010393823 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092053+0000", 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.7574000000000001e-05 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092096+0000", 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2870000000000001e-05 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37830 8.d 8:b56bd01c:::benchmark_data_smithi173_143816_object37829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:39.011 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122278+0000", 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77788145199997, 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97002787499999998, 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37830 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.012 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122278+0000", 2024-08-31T00:00:39.013 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8500000000000002e-07 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122306+0000", 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.8231000000000001e-05 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122311+0000", 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2179999999999998e-06 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.014 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122314+0000", 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.9299999999999999e-06 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122348+0000", 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3624999999999997e-05 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122359+0000", 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.0685e-05 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.015 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079614+0000", 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95725496799999998 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079729+0000", 2024-08-31T00:00:39.016 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000114969 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.079807+0000", 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.8337000000000006e-05 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083762+0000", 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0039546900000000003 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.017 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.083812+0000", 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.0781000000000001e-05 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092177+0000", 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083641789999999994 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092265+0000", 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.8883999999999998e-05 2024-08-31T00:00:39.018 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092306+0000", 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0092999999999997e-05 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37833 8.d 8:bd5be583:::benchmark_data_smithi173_143816_object37832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122321+0000", 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77783764399999, 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97017067000000001, 2024-08-31T00:00:39.019 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37833 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.020 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122321+0000", 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122322+0000", 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.021 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122349+0000", 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.7472999999999999e-05 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122356+0000", 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.7889999999999997e-06 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122360+0000", 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.8330000000000004e-06 2024-08-31T00:00:39.022 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122389+0000", 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.879e-05 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122393+0000", 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1980000000000003e-06 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.023 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080473+0000", 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95808044999999997 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080581+0000", 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00010725700000000001 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.080712+0000", 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013142199999999999 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.024 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084054+0000", 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.003342434 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084069+0000", 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.4487e-05 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092412+0000", 2024-08-31T00:00:39.025 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0083428129999999993 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092458+0000", 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6440999999999998e-05 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092492+0000", 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.4019000000000001e-05 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.026 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:37856 8.d 8:b1ef2347:::benchmark_data_smithi173_143816_object37855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:28.122366+0000", 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 370.77779311799998, 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.97027297800000001, 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 37856 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.027 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122366+0000", 2024-08-31T00:00:39.028 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.6399999999999998e-07 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122397+0000", 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.0994000000000002e-05 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122401+0000", 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1439999999999996e-06 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.029 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122406+0000", 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2579999999999997e-06 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122438+0000", 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.2008000000000003e-05 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:28.122445+0000", 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.0049999999999998e-06 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.030 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081647+0000", 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.95920211399999999 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081737+0000", 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 9.0278999999999994e-05 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 2,7", 2024-08-31T00:00:39.031 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.081881+0000", 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000144036 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084081+0000", 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0021993759999999998 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.084140+0000", 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.9562999999999999e-05 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.032 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092544+0000", 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0084042449999999994 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092598+0000", 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.3677999999999997e-05 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:29.092639+0000", 2024-08-31T00:00:39.033 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.1013999999999997e-05 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45699 8.7 8:eae4016d:::benchmark_data_smithi173_143816_object45698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001109+0000", 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 361.89905042800001, 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64772118199999995, 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:39.034 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45699 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.035 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001109+0000", 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9100000000000003e-07 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001164+0000", 2024-08-31T00:00:39.036 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4604000000000001e-05 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001173+0000", 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.9269999999999994e-06 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001177+0000", 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4950000000000002e-06 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.037 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001209+0000", 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.1758999999999999e-05 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001213+0000", 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.9600000000000002e-06 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642148+0000", 2024-08-31T00:00:39.038 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64093464099999997 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642302+0000", 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015428999999999999 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.642473+0000", 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000170843 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.039 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646115+0000", 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0036422680000000002 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648167+0000", 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.002051911 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648704+0000", 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00053669899999999997 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.040 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648748+0000", 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4098999999999999e-05 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648830+0000", 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.1995000000000004e-05 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.041 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.27853.0:45725 8.7 8:e95f37b8:::benchmark_data_smithi173_143816_object45724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:54:37.001187+0000", 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 361.89897226199997, 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64797183599999997, 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.27853", 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 45725 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.042 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001187+0000", 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001188+0000", 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.9999999999999997e-07 2024-08-31T00:00:39.043 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001240+0000", 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.2120999999999998e-05 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001248+0000", 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.6430000000000001e-06 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001253+0000", 2024-08-31T00:00:39.044 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.4989999999999996e-06 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001286+0000", 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 3.3328999999999999e-05 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.001290+0000", 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.0090000000000001e-06 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.045 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.643899+0000", 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.64260842600000001 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644052+0000", 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00015375600000000001 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for subops from 1,10", 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.644192+0000", 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.00013943899999999999 2024-08-31T00:00:39.046 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "op_commit", 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.646441+0000", 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0022493819999999999 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.648867+0000", 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.0024262860000000002 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649035+0000", 2024-08-31T00:00:39.047 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000167052 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "commit_sent", 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649081+0000", 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.6409999999999998e-05 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:54:37.649159+0000", 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 7.7783999999999999e-05 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.048 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "osd_op(client.28405.0:19279 9.13 9.d9822fb3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:54.641293+0000", 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 164.258866582, 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.434407932, 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "reached pg", 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_info": { 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client": "client.28405", 2024-08-31T00:00:39.049 INFO:teuthology.orchestra.run.smithi173.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "tid": 19279 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641293+0000", 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641293+0000", 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.050 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641294+0000", 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2240000000000001e-06 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641316+0000", 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.2464e-05 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641330+0000", 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.3390000000000001e-05 2024-08-31T00:00:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641336+0000", 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.2639999999999997e-06 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641385+0000", 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.8779000000000002e-05 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "waiting for peered", 2024-08-31T00:00:39.052 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:54.641391+0000", 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 6.426e-06 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.075688+0000", 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.43429733300000001 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.075700+0000", 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.2052000000000001e-05 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.053 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "description": "replica_scrub(pg: 9.9,from:0'0,to:949'1135,epoch:950/948,start:9:96404f94:::benchmark_data_smithi173_160189_object9101:0,end:9:97883b83:::benchmark_data_smithi173_160189_object15908:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "initiated_at": "2024-08-30T23:57:55.017414+0000", 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "age": 163.88274544800001, 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78106588499999996, 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "continuous": false, 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "type_data": { 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "flag_point": "started", 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "events": [ 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "initiated", 2024-08-31T00:00:39.054 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017414+0000", 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "header_read", 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017414+0000", 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "throttled", 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017414+0000", 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 8.0500000000000002e-07 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.055 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "all_read", 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.017420+0000", 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 5.4809999999999997e-06 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "dispatched", 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798214+0000", 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.78079391099999995 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "queued_for_pg", 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798234+0000", 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 2.0608999999999999e-05 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.056 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "reached_pg", 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798420+0000", 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 0.000185353 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "started", 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798437+0000", 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 1.6782999999999998e-05 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: { 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "event": "done", 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "time": "2024-08-30T23:57:55.798480+0000", 2024-08-31T00:00:39.057 INFO:teuthology.orchestra.run.smithi173.stdout: "duration": 4.2942999999999999e-05 2024-08-31T00:00:39.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.058 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.058 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-08-31T00:00:39.058 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-08-31T00:00:39.058 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-08-31T00:00:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.059 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:39.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:39.161 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:39.177 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-31T00:00:39.178 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-31T00:00:39.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.198 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:39.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:39.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:39.323 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:39.425 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:39.449 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:39.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:39.576 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:39.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.705 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:39.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:39 smithi136 ceph-mon[20207]: pgmap v2270: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 543 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1098/26052 objects degraded (4.215%); 2119/26052 objects misplaced (8.134%) 2024-08-31T00:00:39.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:39.807 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:39 smithi173 ceph-mon[20554]: pgmap v2270: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 543 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1098/26052 objects degraded (4.215%); 2119/26052 objects misplaced (8.134%) 2024-08-31T00:00:39.828 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:39.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-31T00:00:39.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:39 smithi106 ceph-mon[16603]: pgmap v2270: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 543 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.08k op/s; 1098/26052 objects degraded (4.215%); 2119/26052 objects misplaced (8.134%) 2024-08-31T00:00:39.931 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:39.952 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:39.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:40.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.079 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:40.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:40.180 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:40.206 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.207 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:40.293 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-31T00:00:40.308 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:40.333 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:40.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:40.458 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:40.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:40.587 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:40.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:40 smithi106 ceph-mon[16603]: from='client.28652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:00:40.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:40.711 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:40 smithi136 ceph-mon[20207]: from='client.28652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:00:40.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:40 smithi173 ceph-mon[20554]: from='client.28652 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:00:40.839 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:40.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:40.972 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:40.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:41.073 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.095 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:41.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:41.197 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.222 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:41.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:41.324 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:41.351 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:41.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.485 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:41.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:41.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.620 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:41.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:41.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.752 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:41.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:41.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:41 smithi106 ceph-mon[16603]: pgmap v2271: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 389 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 813/18627 objects degraded (4.365%); 1562/18627 objects misplaced (8.386%) 2024-08-31T00:00:41.787 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:41 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 813/18627 objects degraded (4.365%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:41.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:41 smithi136 ceph-mon[20207]: pgmap v2271: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 389 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 813/18627 objects degraded (4.365%); 1562/18627 objects misplaced (8.386%) 2024-08-31T00:00:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:41 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 813/18627 objects degraded (4.365%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:41 smithi173 ceph-mon[20554]: pgmap v2271: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 389 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.20k op/s; 813/18627 objects degraded (4.365%); 1562/18627 objects misplaced (8.386%) 2024-08-31T00:00:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:41 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 813/18627 objects degraded (4.365%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:41.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:41.878 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:41.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:41.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:42.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.004 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:42.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:42.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:42.133 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:42.235 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:42.267 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:42.368 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:42.392 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:42.420 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-31T00:00:42.494 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:42.518 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:42.620 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:42.645 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:42.747 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:42.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/144890314' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:42.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.779 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:42.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:42.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/144890314' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:42.791 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2270,"stamp":"2024-08-31T00:00:37.891303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569581429,"num_objects":8684,"num_object_clones":0,"num_object_copies":26052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1098,"num_objects_misplaced":2119,"num_objects_unfound":0,"num_objects_dirty":8684,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":147298,"num_write_kb":3544115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5301,"num_bytes_recovered":347490483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237697,"ondisk_log_size":237697,"up":177,"acting":188,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":202,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11190356,"kb_used_data":2325356,"kb_used_omap":109,"kb_used_meta":8864530,"kb_avail":1113751468,"statfs":{"total":1151940427776,"available":1140481503232,"internally_reserved":0,"allocated":2381164544,"data_stored":2368826363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":111976,"internal_metadata":9077279384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-565575680,"num_objects":-8630,"num_object_clones":0,"num_object_copies":-25890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1049,"num_objects_misplaced":-2051,"num_objects_unfound":0,"num_objects_dirty":-8630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8630,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002376"},"pg_stats":[{"pgid":"9.23","version":"1027'4340","reported_seq":7164,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:37.206888+0000","last_change":"2024-08-31T00:00:20.517080+0000","last_active":"2024-08-31T00:00:37.206888+0000","last_peered":"2024-08-31T00:00:37.206888+0000","last_clean":"2024-08-31T00:00:37.206888+0000","last_became_active":"2024-08-30T23:59:01.084823+0000","last_became_peered":"2024-08-30T23:59:01.084823+0000","last_unstale":"2024-08-31T00:00:37.206888+0000","last_undegraded":"2024-08-31T00:00:37.206888+0000","last_fullsized":"2024-08-31T00:00:37.206888+0000","mapping_epoch":984,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":985,"parent":"0.0","parent_split_bits":6,"last_scrub":"1019'4094","last_scrub_stamp":"2024-08-31T00:00:20.517005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-31T00:00:20.517005+0000","objects_scrubbed":369,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:46.529893+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7798784,"num_objects":119,"num_object_clones":0,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1431,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.22","version":"1025'4212","reported_seq":6084,"reported_epoch":1026,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:34.402462+0000","last_change":"2024-08-31T00:00:00.816265+0000","last_active":"2024-08-31T00:00:34.402462+0000","last_peered":"2024-08-31T00:00:34.402462+0000","last_clean":"2024-08-31T00:00:34.402462+0000","last_became_active":"2024-08-30T23:58:34.338653+0000","last_became_peered":"2024-08-30T23:58:34.338653+0000","last_unstale":"2024-08-31T00:00:34.402462+0000","last_undegraded":"2024-08-31T00:00:34.402462+0000","last_fullsized":"2024-08-31T00:00:34.402462+0000","mapping_epoch":972,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":6,"last_scrub":"1013'3646","last_scrub_stamp":"2024-08-31T00:00:00.815879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-31T00:00:00.815879+0000","objects_scrubbed":697,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:25.410929+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":131,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1886,"num_write_kb":34496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4,1],"avail_no_missing":["9","1","4"],"object_location_counts":[{"shards":"1,4,9","objects":131}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.21","version":"1026'4499","reported_seq":8155,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.510099+0000","last_change":"2024-08-30T23:58:43.086940+0000","last_active":"2024-08-31T00:00:34.510099+0000","last_peered":"2024-08-31T00:00:34.510099+0000","last_clean":"2024-08-31T00:00:34.510099+0000","last_became_active":"2024-08-30T23:58:42.354339+0000","last_became_peered":"2024-08-30T23:58:42.354339+0000","last_unstale":"2024-08-31T00:00:34.510099+0000","last_undegraded":"2024-08-31T00:00:34.510099+0000","last_fullsized":"2024-08-31T00:00:34.510099+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":6,"last_scrub":"975'2511","last_scrub_stamp":"2024-08-30T23:58:43.086862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:43.086862+0000","objects_scrubbed":1135,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9568256,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":32000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.20","version":"1026'4403","reported_seq":7931,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:35.616224+0000","last_change":"2024-08-30T23:59:54.164396+0000","last_active":"2024-08-31T00:00:35.616224+0000","last_peered":"2024-08-31T00:00:35.616224+0000","last_clean":"2024-08-31T00:00:35.616224+0000","last_became_active":"2024-08-30T23:58:42.350889+0000","last_became_peered":"2024-08-30T23:58:42.350889+0000","last_unstale":"2024-08-31T00:00:35.616224+0000","last_undegraded":"2024-08-31T00:00:35.616224+0000","last_fullsized":"2024-08-31T00:00:35.616224+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":6,"last_scrub":"1013'3639","last_scrub_stamp":"2024-08-30T23:59:54.164286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:54.164286+0000","objects_scrubbed":911,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:06.645753+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9502720,"num_objects":145,"num_object_clones":0,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1869,"num_write_kb":28608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,3],"acting":[8,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.1c","version":"1027'7648","reported_seq":14472,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.291514+0000","last_change":"2024-08-30T23:59:27.914302+0000","last_active":"2024-08-31T00:00:36.291514+0000","last_peered":"2024-08-31T00:00:36.291514+0000","last_clean":"2024-08-31T00:00:36.291514+0000","last_became_active":"2024-08-30T23:59:27.099132+0000","last_became_peered":"2024-08-30T23:59:27.099132+0000","last_unstale":"2024-08-31T00:00:36.291514+0000","last_undegraded":"2024-08-31T00:00:36.291514+0000","last_fullsized":"2024-08-31T00:00:36.291514+0000","mapping_epoch":1002,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":5,"last_scrub":"1003'5120","last_scrub_stamp":"2024-08-30T23:59:27.914208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:27.914208+0000","objects_scrubbed":2770,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:47.433680+0000","scrub_duration":610,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2539,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,3],"acting":[11,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"23'92","reported_seq":1228,"reported_epoch":1026,"state":"active+undersized","last_fresh":"2024-08-31T00:00:34.402547+0000","last_change":"2024-08-30T23:55:59.174852+0000","last_active":"2024-08-31T00:00:34.402547+0000","last_peered":"2024-08-31T00:00:34.402547+0000","last_clean":"2024-08-30T23:55:57.155206+0000","last_became_active":"2024-08-30T23:55:59.174852+0000","last_became_peered":"2024-08-30T23:55:59.174852+0000","last_unstale":"2024-08-31T00:00:34.402547+0000","last_undegraded":"2024-08-31T00:00:34.402547+0000","last_fullsized":"2024-08-30T23:55:59.171288+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":867,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:54:34.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:54:34.892604+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":85,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.122351+0000","last_change":"2024-08-30T23:59:14.299069+0000","last_active":"2024-08-31T00:00:36.122351+0000","last_peered":"2024-08-31T00:00:36.122351+0000","last_clean":"2024-08-31T00:00:36.122351+0000","last_became_active":"2024-08-30T23:59:14.186459+0000","last_became_peered":"2024-08-30T23:59:14.186459+0000","last_unstale":"2024-08-31T00:00:36.122351+0000","last_undegraded":"2024-08-31T00:00:36.122351+0000","last_fullsized":"2024-08-31T00:00:36.122351+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:14.299023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:14.299023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:37.195731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.3","version":"1025'4401","reported_seq":4658,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.402705+0000","last_change":"2024-08-30T23:59:45.974821+0000","last_active":"2024-08-31T00:00:34.402705+0000","last_peered":"2024-08-31T00:00:34.402705+0000","last_clean":"2024-08-31T00:00:34.402705+0000","last_became_active":"2024-08-30T23:58:23.229111+0000","last_became_peered":"2024-08-30T23:58:23.229111+0000","last_unstale":"2024-08-31T00:00:34.402705+0000","last_undegraded":"2024-08-31T00:00:34.402705+0000","last_fullsized":"2024-08-31T00:00:34.402705+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"1008'3523","last_scrub_stamp":"2024-08-30T23:59:45.974730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:45.974730+0000","objects_scrubbed":1018,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:48.509388+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8847360,"num_objects":135,"num_object_clones":0,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":45952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1289,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.402557+0000","last_change":"2024-08-30T23:59:23.416907+0000","last_active":"2024-08-31T00:00:34.402557+0000","last_peered":"2024-08-31T00:00:34.402557+0000","last_clean":"2024-08-31T00:00:34.402557+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-31T00:00:34.402557+0000","last_undegraded":"2024-08-31T00:00:34.402557+0000","last_fullsized":"2024-08-31T00:00:34.402557+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:23.416839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:23.416839+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"1025'4407","reported_seq":4655,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-08-31T00:00:33.950740+0000","last_change":"2024-08-31T00:00:19.628898+0000","last_active":"2024-08-31T00:00:33.950740+0000","last_peered":"2024-08-31T00:00:33.950740+0000","last_clean":"2024-08-31T00:00:33.950740+0000","last_became_active":"2024-08-30T23:58:23.225981+0000","last_became_peered":"2024-08-30T23:58:23.225981+0000","last_unstale":"2024-08-31T00:00:33.950740+0000","last_undegraded":"2024-08-31T00:00:33.950740+0000","last_fullsized":"2024-08-31T00:00:33.950740+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'4141","last_scrub_stamp":"2024-08-31T00:00:19.628768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-31T00:00:19.628768+0000","objects_scrubbed":411,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:21.720200+0000","scrub_duration":380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437184,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2207,"num_write_kb":46976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1136,"reported_epoch":1026,"state":"active+undersized","last_fresh":"2024-08-31T00:00:34.402480+0000","last_change":"2024-08-30T23:55:59.174783+0000","last_active":"2024-08-31T00:00:34.402480+0000","last_peered":"2024-08-31T00:00:34.402480+0000","last_clean":"2024-08-30T23:39:14.863777+0000","last_became_active":"2024-08-30T23:55:59.174783+0000","last_became_peered":"2024-08-30T23:55:59.174783+0000","last_unstale":"2024-08-31T00:00:34.402480+0000","last_undegraded":"2024-08-31T00:00:34.402480+0000","last_fullsized":"2024-08-30T23:55:59.171115+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.159710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.159710+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-08-31T00:00:36.704877+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":72126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2],"acting":[9,2],"avail_no_missing":["9","2"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.1","version":"1025'4466","reported_seq":4712,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.007731+0000","last_change":"2024-08-31T00:00:16.398502+0000","last_active":"2024-08-31T00:00:34.007731+0000","last_peered":"2024-08-31T00:00:34.007731+0000","last_clean":"2024-08-31T00:00:34.007731+0000","last_became_active":"2024-08-30T23:58:23.226284+0000","last_became_peered":"2024-08-30T23:58:23.226284+0000","last_unstale":"2024-08-31T00:00:34.007731+0000","last_undegraded":"2024-08-31T00:00:34.007731+0000","last_fullsized":"2024-08-31T00:00:34.007731+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'4141","last_scrub_stamp":"2024-08-31T00:00:16.398255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-31T00:00:16.398255+0000","objects_scrubbed":477,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:31.011447+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9961472,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":49216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":539,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.399370+0000","last_change":"2024-08-30T23:56:43.187954+0000","last_active":"2024-08-31T00:00:34.399370+0000","last_peered":"2024-08-31T00:00:34.399370+0000","last_clean":"2024-08-31T00:00:34.399370+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-31T00:00:34.399370+0000","last_undegraded":"2024-08-31T00:00:34.399370+0000","last_fullsized":"2024-08-31T00:00:34.399370+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:56:43.187912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:56:43.187912+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"1027'4531","reported_seq":7077,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:37.327549+0000","last_change":"2024-08-30T23:59:50.472551+0000","last_active":"2024-08-31T00:00:37.327549+0000","last_peered":"2024-08-31T00:00:37.327549+0000","last_clean":"2024-08-31T00:00:37.327549+0000","last_became_active":"2024-08-30T23:59:16.112131+0000","last_became_peered":"2024-08-30T23:59:16.112131+0000","last_unstale":"2024-08-31T00:00:37.327549+0000","last_undegraded":"2024-08-31T00:00:37.327549+0000","last_fullsized":"2024-08-31T00:00:37.327549+0000","mapping_epoch":996,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"1010'3661","last_scrub_stamp":"2024-08-30T23:59:50.472482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:50.472482+0000","objects_scrubbed":976,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:51.264247+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6750208,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2307,"num_write_kb":48192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1135,"num_bytes_recovered":74383360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1367,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.126432+0000","last_change":"2024-08-30T23:59:23.211405+0000","last_active":"2024-08-31T00:00:36.126432+0000","last_peered":"2024-08-31T00:00:36.126432+0000","last_clean":"2024-08-31T00:00:36.126432+0000","last_became_active":"2024-08-30T23:57:36.158625+0000","last_became_peered":"2024-08-30T23:57:36.158625+0000","last_unstale":"2024-08-31T00:00:36.126432+0000","last_undegraded":"2024-08-31T00:00:36.126432+0000","last_fullsized":"2024-08-31T00:00:36.126432+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":939,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:23.211350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:23.211350+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:43.117426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,9],"acting":[6,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"1027'6711","reported_seq":6907,"reported_epoch":1027,"state":"active+undersized+degraded","last_fresh":"2024-08-31T00:00:37.032519+0000","last_change":"2024-08-30T23:57:53.077387+0000","last_active":"2024-08-31T00:00:37.032519+0000","last_peered":"2024-08-31T00:00:37.032519+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:57:53.077387+0000","last_became_peered":"2024-08-30T23:57:53.077387+0000","last_unstale":"2024-08-31T00:00:37.032519+0000","last_undegraded":"2024-08-30T23:57:53.068813+0000","last_fullsized":"2024-08-30T23:57:53.068689+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-31T00:00:38.447240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6175,"num_write_kb":204864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3],"acting":[10,3],"avail_no_missing":["10","3"],"object_location_counts":[{"shards":"3,10","objects":227}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":1075,"reported_epoch":1027,"state":"active+undersized","last_fresh":"2024-08-31T00:00:36.109961+0000","last_change":"2024-08-30T23:55:59.166873+0000","last_active":"2024-08-31T00:00:36.109961+0000","last_peered":"2024-08-31T00:00:36.109961+0000","last_clean":"2024-08-30T23:55:57.155710+0000","last_became_active":"2024-08-30T23:55:59.166873+0000","last_became_peered":"2024-08-30T23:55:59.166873+0000","last_unstale":"2024-08-31T00:00:36.109961+0000","last_undegraded":"2024-08-31T00:00:36.109961+0000","last_fullsized":"2024-08-30T23:55:59.164440+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:02.465926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:02.465926+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.6","version":"1026'6601","reported_seq":6814,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.736651+0000","last_change":"2024-08-30T23:57:55.058608+0000","last_active":"2024-08-31T00:00:34.736651+0000","last_peered":"2024-08-31T00:00:34.736651+0000","last_clean":"2024-08-31T00:00:34.736651+0000","last_became_active":"2024-08-30T23:57:53.081131+0000","last_became_peered":"2024-08-30T23:57:53.081131+0000","last_unstale":"2024-08-31T00:00:34.736651+0000","last_undegraded":"2024-08-31T00:00:34.736651+0000","last_fullsized":"2024-08-31T00:00:34.736651+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"950'1118","last_scrub_stamp":"2024-08-30T23:57:55.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:55.058465+0000","objects_scrubbed":594,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":990,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":165760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":1143,"reported_epoch":1027,"state":"active+undersized","last_fresh":"2024-08-31T00:00:36.110153+0000","last_change":"2024-08-30T23:55:59.167520+0000","last_active":"2024-08-31T00:00:36.110153+0000","last_peered":"2024-08-31T00:00:36.110153+0000","last_clean":"2024-08-30T23:55:57.155918+0000","last_became_active":"2024-08-30T23:55:59.167520+0000","last_became_peered":"2024-08-30T23:55:59.167520+0000","last_unstale":"2024-08-31T00:00:36.110153+0000","last_undegraded":"2024-08-31T00:00:36.110153+0000","last_fullsized":"2024-08-30T23:55:59.164366+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:01.489470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:01.489470+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:37.484319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"1027'6573","reported_seq":12389,"reported_epoch":1027,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:36.359498+0000","last_change":"2024-08-30T23:59:44.575417+0000","last_active":"2024-08-31T00:00:36.359498+0000","last_peered":"2024-08-31T00:00:36.359498+0000","last_clean":"2024-08-31T00:00:36.359498+0000","last_became_active":"2024-08-30T23:59:15.187163+0000","last_became_peered":"2024-08-30T23:59:15.187163+0000","last_unstale":"2024-08-31T00:00:36.359498+0000","last_undegraded":"2024-08-31T00:00:36.359498+0000","last_fullsized":"2024-08-31T00:00:36.359498+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":1006,"parent":"0.0","parent_split_bits":0,"last_scrub":"1008'4725","last_scrub_stamp":"2024-08-30T23:59:44.575194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:44.575194+0000","objects_scrubbed":2082,"log_size":6573,"log_dups_size":0,"ondisk_log_size":6573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:49.747737+0000","scrub_duration":444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":228,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1854,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5,1],"avail_no_missing":["11","1","5"],"object_location_counts":[{"shards":"1,5,11","objects":228}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"23'92","reported_seq":1171,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-08-31T00:00:29.381122+0000","last_change":"2024-08-30T23:59:43.546775+0000","last_active":"2024-08-31T00:00:29.381122+0000","last_peered":"2024-08-31T00:00:29.381122+0000","last_clean":"2024-08-31T00:00:29.381122+0000","last_became_active":"2024-08-30T23:55:01.288430+0000","last_became_peered":"2024-08-30T23:55:01.288430+0000","last_unstale":"2024-08-31T00:00:29.381122+0000","last_undegraded":"2024-08-31T00:00:29.381122+0000","last_fullsized":"2024-08-31T00:00:29.381122+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:43.546701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:43.546701+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:47.459935+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"1027'6486","reported_seq":6732,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.938346+0000","last_change":"2024-08-30T23:59:43.014777+0000","last_active":"2024-08-31T00:00:36.938346+0000","last_peered":"2024-08-31T00:00:36.938346+0000","last_clean":"2024-08-31T00:00:36.938346+0000","last_became_active":"2024-08-30T23:57:53.070921+0000","last_became_peered":"2024-08-30T23:57:53.070921+0000","last_unstale":"2024-08-31T00:00:36.938346+0000","last_undegraded":"2024-08-31T00:00:36.938346+0000","last_fullsized":"2024-08-31T00:00:36.938346+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":949,"parent":"0.0","parent_split_bits":0,"last_scrub":"1008'4577","last_scrub_stamp":"2024-08-30T23:59:43.014614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:43.014614+0000","objects_scrubbed":2160,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:52.109682+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5416,"num_write_kb":163008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":514,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.126003+0000","last_change":"2024-08-30T23:59:18.160031+0000","last_active":"2024-08-31T00:00:36.126003+0000","last_peered":"2024-08-31T00:00:36.126003+0000","last_clean":"2024-08-31T00:00:36.126003+0000","last_became_active":"2024-08-30T23:52:07.970265+0000","last_became_peered":"2024-08-30T23:52:07.970265+0000","last_unstale":"2024-08-31T00:00:36.126003+0000","last_undegraded":"2024-08-31T00:00:36.126003+0000","last_fullsized":"2024-08-31T00:00:36.126003+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":800,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:18.159957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:18.159957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:46.265398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"1026'7947","reported_seq":8202,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:35.653436+0000","last_change":"2024-08-30T23:59:53.391858+0000","last_active":"2024-08-31T00:00:35.653436+0000","last_peered":"2024-08-31T00:00:35.653436+0000","last_clean":"2024-08-31T00:00:35.653436+0000","last_became_active":"2024-08-30T23:58:23.233769+0000","last_became_peered":"2024-08-30T23:58:23.233769+0000","last_unstale":"2024-08-31T00:00:35.653436+0000","last_undegraded":"2024-08-31T00:00:35.653436+0000","last_fullsized":"2024-08-31T00:00:35.653436+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"1013'6323","last_scrub_stamp":"2024-08-30T23:59:53.391691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:53.391691+0000","objects_scrubbed":1879,"log_size":7947,"log_dups_size":0,"ondisk_log_size":7947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:58.480640+0000","scrub_duration":333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16580608,"num_objects":253,"num_object_clones":0,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4525,"num_write_kb":97920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":2173,"reported_epoch":1027,"state":"active+undersized","last_fresh":"2024-08-31T00:00:36.109788+0000","last_change":"2024-08-30T23:55:59.166425+0000","last_active":"2024-08-31T00:00:36.109788+0000","last_peered":"2024-08-31T00:00:36.109788+0000","last_clean":"2024-08-30T23:55:57.155995+0000","last_became_active":"2024-08-30T23:55:59.166425+0000","last_became_peered":"2024-08-30T23:55:59.166425+0000","last_unstale":"2024-08-31T00:00:36.109788+0000","last_undegraded":"2024-08-31T00:00:36.109788+0000","last_fullsized":"2024-08-30T23:55:59.164395+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:31.643486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:31.643486+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.a","version":"1026'7757","reported_seq":7952,"reported_epoch":1026,"state":"active+undersized+degraded","last_fresh":"2024-08-31T00:00:34.856071+0000","last_change":"2024-08-30T23:58:23.233914+0000","last_active":"2024-08-31T00:00:34.856071+0000","last_peered":"2024-08-31T00:00:34.856071+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.233914+0000","last_became_peered":"2024-08-30T23:58:23.233914+0000","last_unstale":"2024-08-31T00:00:34.856071+0000","last_undegraded":"2024-08-30T23:58:23.221431+0000","last_fullsized":"2024-08-30T23:58:23.221296+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":7757,"log_dups_size":0,"ondisk_log_size":7757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-31T00:00:37.028289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19791872,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":302}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":694,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.108416+0000","last_change":"2024-08-30T23:59:41.633746+0000","last_active":"2024-08-31T00:00:36.108416+0000","last_peered":"2024-08-31T00:00:36.108416+0000","last_clean":"2024-08-31T00:00:36.108416+0000","last_became_active":"2024-08-30T23:48:30.880551+0000","last_became_peered":"2024-08-30T23:48:30.880551+0000","last_unstale":"2024-08-31T00:00:36.108416+0000","last_undegraded":"2024-08-31T00:00:36.108416+0000","last_fullsized":"2024-08-31T00:00:36.108416+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":711,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:41.633645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:41.633645+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:43.807564+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.9","version":"1027'6818","reported_seq":9098,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:37.187044+0000","last_change":"2024-08-30T23:59:45.295381+0000","last_active":"2024-08-31T00:00:37.187044+0000","last_peered":"2024-08-31T00:00:37.187044+0000","last_clean":"2024-08-31T00:00:37.187044+0000","last_became_active":"2024-08-30T23:59:18.091017+0000","last_became_peered":"2024-08-30T23:59:18.091017+0000","last_unstale":"2024-08-31T00:00:37.187044+0000","last_undegraded":"2024-08-31T00:00:37.187044+0000","last_fullsized":"2024-08-31T00:00:37.187044+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":0,"last_scrub":"1008'4874","last_scrub_stamp":"2024-08-30T23:59:45.295260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:45.295260+0000","objects_scrubbed":2163,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:54.083393+0000","scrub_duration":776,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":168064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1008,"num_bytes_recovered":66060288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1356,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.126440+0000","last_change":"2024-08-30T23:59:44.346734+0000","last_active":"2024-08-31T00:00:36.126440+0000","last_peered":"2024-08-31T00:00:36.126440+0000","last_clean":"2024-08-31T00:00:36.126440+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-31T00:00:36.126440+0000","last_undegraded":"2024-08-31T00:00:36.126440+0000","last_fullsized":"2024-08-31T00:00:36.126440+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:44.346663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:44.346663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:45.433345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"1026'7734","reported_seq":7960,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.828349+0000","last_change":"2024-08-30T23:58:26.120565+0000","last_active":"2024-08-31T00:00:34.828349+0000","last_peered":"2024-08-31T00:00:34.828349+0000","last_clean":"2024-08-31T00:00:34.828349+0000","last_became_active":"2024-08-30T23:58:23.234078+0000","last_became_peered":"2024-08-30T23:58:23.234078+0000","last_unstale":"2024-08-31T00:00:34.828349+0000","last_undegraded":"2024-08-31T00:00:34.828349+0000","last_fullsized":"2024-08-31T00:00:34.828349+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"967'3422","last_scrub_stamp":"2024-08-30T23:58:26.120459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:26.120459+0000","objects_scrubbed":1763,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726336,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4312,"num_write_kb":91200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":709,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.108661+0000","last_change":"2024-08-30T23:59:19.405781+0000","last_active":"2024-08-31T00:00:36.108661+0000","last_peered":"2024-08-31T00:00:36.108661+0000","last_clean":"2024-08-31T00:00:36.108661+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-31T00:00:36.108661+0000","last_undegraded":"2024-08-31T00:00:36.108661+0000","last_fullsized":"2024-08-31T00:00:36.108661+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:19.405731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:19.405731+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:45.706371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1027'8067","reported_seq":8263,"reported_epoch":1027,"state":"active+undersized+degraded","last_fresh":"2024-08-31T00:00:37.083072+0000","last_change":"2024-08-30T23:58:23.240351+0000","last_active":"2024-08-31T00:00:37.083072+0000","last_peered":"2024-08-31T00:00:37.083072+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.240351+0000","last_became_peered":"2024-08-30T23:58:23.240351+0000","last_unstale":"2024-08-31T00:00:37.083072+0000","last_undegraded":"2024-08-30T23:58:23.225188+0000","last_fullsized":"2024-08-30T23:58:23.224931+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-31T00:00:39.463209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17104896,"num_objects":261,"num_object_clones":0,"num_object_copies":783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6361,"num_write_kb":211904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1],"acting":[10,1],"avail_no_missing":["10","1"],"object_location_counts":[{"shards":"1,10","objects":261}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":735,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.108399+0000","last_change":"2024-08-30T23:59:42.474383+0000","last_active":"2024-08-31T00:00:36.108399+0000","last_peered":"2024-08-31T00:00:36.108399+0000","last_clean":"2024-08-31T00:00:36.108399+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-31T00:00:36.108399+0000","last_undegraded":"2024-08-31T00:00:36.108399+0000","last_fullsized":"2024-08-31T00:00:36.108399+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:42.474318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:42.474318+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:55.750097+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.d","version":"1026'7497","reported_seq":7692,"reported_epoch":1026,"state":"active+undersized+degraded","last_fresh":"2024-08-31T00:00:34.837725+0000","last_change":"2024-08-30T23:58:23.224228+0000","last_active":"2024-08-31T00:00:34.837725+0000","last_peered":"2024-08-31T00:00:34.837725+0000","last_clean":"2024-08-30T23:57:35.791607+0000","last_became_active":"2024-08-30T23:58:23.224228+0000","last_became_peered":"2024-08-30T23:58:23.224228+0000","last_unstale":"2024-08-31T00:00:34.837725+0000","last_undegraded":"2024-08-30T23:58:23.222413+0000","last_fullsized":"2024-08-30T23:58:23.221376+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.791607+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-08-31T00:00:36.060481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":307,"num_object_clones":0,"num_object_copies":921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5875,"num_write_kb":197856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":307}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":379,"reported_epoch":1027,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:36.122633+0000","last_change":"2024-08-30T23:59:49.114557+0000","last_active":"2024-08-31T00:00:36.122633+0000","last_peered":"2024-08-31T00:00:36.122633+0000","last_clean":"2024-08-31T00:00:36.122633+0000","last_became_active":"2024-08-30T23:59:15.177663+0000","last_became_peered":"2024-08-30T23:59:15.177663+0000","last_unstale":"2024-08-31T00:00:36.122633+0000","last_undegraded":"2024-08-31T00:00:36.122633+0000","last_fullsized":"2024-08-31T00:00:36.122633+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:49.114430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:49.114430+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:12.047745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8,6],"avail_no_missing":["11","6","8"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.c","version":"1025'7618","reported_seq":7866,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-08-31T00:00:33.928480+0000","last_change":"2024-08-30T23:59:59.070045+0000","last_active":"2024-08-31T00:00:33.928480+0000","last_peered":"2024-08-31T00:00:33.928480+0000","last_clean":"2024-08-31T00:00:33.928480+0000","last_became_active":"2024-08-30T23:58:23.224459+0000","last_became_peered":"2024-08-30T23:58:23.224459+0000","last_unstale":"2024-08-31T00:00:33.928480+0000","last_undegraded":"2024-08-31T00:00:33.928480+0000","last_fullsized":"2024-08-31T00:00:33.928480+0000","mapping_epoch":938,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":967,"parent":"0.0","parent_split_bits":0,"last_scrub":"1013'6361","last_scrub_stamp":"2024-08-30T23:59:59.068207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:59.068207+0000","objects_scrubbed":1569,"log_size":7618,"log_dups_size":0,"ondisk_log_size":7618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:08.016114+0000","scrub_duration":741,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19726336,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":92992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":961,"reported_epoch":1027,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:36.125838+0000","last_change":"2024-08-31T00:00:27.368692+0000","last_active":"2024-08-31T00:00:36.125838+0000","last_peered":"2024-08-31T00:00:36.125838+0000","last_clean":"2024-08-31T00:00:36.125838+0000","last_became_active":"2024-08-30T23:59:15.180603+0000","last_became_peered":"2024-08-30T23:59:15.180603+0000","last_unstale":"2024-08-31T00:00:36.125838+0000","last_undegraded":"2024-08-31T00:00:36.125838+0000","last_fullsized":"2024-08-31T00:00:36.125838+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:00:27.368599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:00:27.368599+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:38.997698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11,9],"avail_no_missing":["6","9","11"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1026'6695","reported_seq":6959,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.402458+0000","last_change":"2024-08-30T23:59:18.580672+0000","last_active":"2024-08-31T00:00:34.402458+0000","last_peered":"2024-08-31T00:00:34.402458+0000","last_clean":"2024-08-31T00:00:34.402458+0000","last_became_active":"2024-08-30T23:59:15.179731+0000","last_became_peered":"2024-08-30T23:59:15.179731+0000","last_unstale":"2024-08-31T00:00:34.402458+0000","last_undegraded":"2024-08-31T00:00:34.402458+0000","last_fullsized":"2024-08-31T00:00:34.402458+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":938,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":0,"last_scrub":"999'3824","last_scrub_stamp":"2024-08-30T23:59:18.580580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:18.580580+0000","objects_scrubbed":3161,"log_size":6695,"log_dups_size":0,"ondisk_log_size":6695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5538,"num_write_kb":167360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":516,"reported_epoch":1026,"state":"active+undersized","last_fresh":"2024-08-31T00:00:34.399496+0000","last_change":"2024-08-30T23:55:59.170281+0000","last_active":"2024-08-31T00:00:34.399496+0000","last_peered":"2024-08-31T00:00:34.399496+0000","last_clean":"2024-08-30T23:55:57.164091+0000","last_became_active":"2024-08-30T23:55:59.170281+0000","last_became_peered":"2024-08-30T23:55:59.170281+0000","last_unstale":"2024-08-31T00:00:34.399496+0000","last_undegraded":"2024-08-31T00:00:34.399496+0000","last_fullsized":"2024-08-30T23:55:59.168051+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:06.914406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:06.914406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10],"acting":[5,10],"avail_no_missing":["5","10"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.18","version":"1027'6789","reported_seq":8271,"reported_epoch":1027,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:37.236508+0000","last_change":"2024-08-30T23:59:47.072318+0000","last_active":"2024-08-31T00:00:37.236508+0000","last_peered":"2024-08-31T00:00:37.236508+0000","last_clean":"2024-08-31T00:00:37.236508+0000","last_became_active":"2024-08-30T23:58:07.098389+0000","last_became_peered":"2024-08-30T23:58:07.098389+0000","last_unstale":"2024-08-31T00:00:37.236508+0000","last_undegraded":"2024-08-31T00:00:37.236508+0000","last_fullsized":"2024-08-31T00:00:37.236508+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"1009'5016","last_scrub_stamp":"2024-08-30T23:59:47.072044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:47.072044+0000","objects_scrubbed":2045,"log_size":6789,"log_dups_size":0,"ondisk_log_size":6789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:03.551095+0000","scrub_duration":618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384000,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":250,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5191,"num_write_kb":140544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5],"acting":[10,5,9],"avail_no_missing":["10","5","9"],"object_location_counts":[{"shards":"5,9,10","objects":250}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":1157,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.396340+0000","last_change":"2024-08-30T23:59:28.049854+0000","last_active":"2024-08-31T00:00:34.396340+0000","last_peered":"2024-08-31T00:00:34.396340+0000","last_clean":"2024-08-31T00:00:34.396340+0000","last_became_active":"2024-08-30T23:50:25.934418+0000","last_became_peered":"2024-08-30T23:50:25.934418+0000","last_unstale":"2024-08-31T00:00:34.396340+0000","last_undegraded":"2024-08-31T00:00:34.396340+0000","last_fullsized":"2024-08-31T00:00:34.396340+0000","mapping_epoch":761,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":762,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:28.049794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:28.049794+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:36.962058+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.19","version":"1025'6604","reported_seq":8547,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.011199+0000","last_change":"2024-08-30T23:59:51.250807+0000","last_active":"2024-08-31T00:00:34.011199+0000","last_peered":"2024-08-31T00:00:34.011199+0000","last_clean":"2024-08-31T00:00:34.011199+0000","last_became_active":"2024-08-30T23:58:04.122431+0000","last_became_peered":"2024-08-30T23:58:04.122431+0000","last_unstale":"2024-08-31T00:00:34.011199+0000","last_undegraded":"2024-08-31T00:00:34.011199+0000","last_fullsized":"2024-08-31T00:00:34.011199+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"1011'5026","last_scrub_stamp":"2024-08-30T23:59:51.250614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:51.250614+0000","objects_scrubbed":1906,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:03.808350+0000","scrub_duration":869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":310,"num_object_clones":0,"num_object_copies":930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":149056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":528,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.399521+0000","last_change":"2024-08-30T23:57:35.093839+0000","last_active":"2024-08-31T00:00:34.399521+0000","last_peered":"2024-08-31T00:00:34.399521+0000","last_clean":"2024-08-31T00:00:34.399521+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-31T00:00:34.399521+0000","last_undegraded":"2024-08-31T00:00:34.399521+0000","last_fullsized":"2024-08-31T00:00:34.399521+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:57:35.093787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:57:35.093787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"1025'7597","reported_seq":12841,"reported_epoch":1025,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:33.988223+0000","last_change":"2024-08-30T23:59:13.562000+0000","last_active":"2024-08-31T00:00:33.988223+0000","last_peered":"2024-08-31T00:00:33.988223+0000","last_clean":"2024-08-31T00:00:33.988223+0000","last_became_active":"2024-08-30T23:58:55.439550+0000","last_became_peered":"2024-08-30T23:58:55.439550+0000","last_unstale":"2024-08-31T00:00:33.988223+0000","last_undegraded":"2024-08-31T00:00:33.988223+0000","last_fullsized":"2024-08-31T00:00:33.988223+0000","mapping_epoch":982,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":983,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'4765","last_scrub_stamp":"2024-08-30T23:59:13.561894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:13.561894+0000","objects_scrubbed":3127,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:43.313485+0000","scrub_duration":338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":295,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2832,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":85,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.122480+0000","last_change":"2024-08-30T23:59:51.108929+0000","last_active":"2024-08-31T00:00:36.122480+0000","last_peered":"2024-08-31T00:00:36.122480+0000","last_clean":"2024-08-31T00:00:36.122480+0000","last_became_active":"2024-08-30T23:59:14.207545+0000","last_became_peered":"2024-08-30T23:59:14.207545+0000","last_unstale":"2024-08-31T00:00:36.122480+0000","last_undegraded":"2024-08-31T00:00:36.122480+0000","last_fullsized":"2024-08-31T00:00:36.122480+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:51.108850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:51.108850+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:13.153640+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.1b","version":"1025'7807","reported_seq":13319,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.008165+0000","last_change":"2024-08-31T00:00:29.389914+0000","last_active":"2024-08-31T00:00:34.008165+0000","last_peered":"2024-08-31T00:00:34.008165+0000","last_clean":"2024-08-31T00:00:34.008165+0000","last_became_active":"2024-08-30T23:59:18.147982+0000","last_became_peered":"2024-08-30T23:59:18.147982+0000","last_unstale":"2024-08-31T00:00:34.008165+0000","last_undegraded":"2024-08-31T00:00:34.008165+0000","last_fullsized":"2024-08-31T00:00:34.008165+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":5,"last_scrub":"999'4958","last_scrub_stamp":"2024-08-30T23:59:18.736685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:18.736685+0000","objects_scrubbed":199,"log_size":7807,"log_dups_size":0,"ondisk_log_size":7807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:50.528587+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20119552,"num_objects":307,"num_object_clones":0,"num_object_copies":921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1d","version":"1026'7730","reported_seq":13146,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.834228+0000","last_change":"2024-08-30T23:59:12.818297+0000","last_active":"2024-08-31T00:00:34.834228+0000","last_peered":"2024-08-31T00:00:34.834228+0000","last_clean":"2024-08-31T00:00:34.834228+0000","last_became_active":"2024-08-30T23:58:51.078720+0000","last_became_peered":"2024-08-30T23:58:51.078720+0000","last_unstale":"2024-08-31T00:00:34.834228+0000","last_undegraded":"2024-08-31T00:00:34.834228+0000","last_fullsized":"2024-08-31T00:00:34.834228+0000","mapping_epoch":978,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":979,"parent":"0.0","parent_split_bits":5,"last_scrub":"993'4811","last_scrub_stamp":"2024-08-30T23:59:12.818133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:12.818133+0000","objects_scrubbed":3206,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21233688,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2939,"num_write_kb":1793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,0],"acting":[5,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":259,"reported_epoch":1026,"state":"active+undersized","last_fresh":"2024-08-31T00:00:34.401338+0000","last_change":"2024-08-30T23:55:59.170252+0000","last_active":"2024-08-31T00:00:34.401338+0000","last_peered":"2024-08-31T00:00:34.401338+0000","last_clean":"2024-08-30T23:23:05.641834+0000","last_became_active":"2024-08-30T23:55:59.170252+0000","last_became_peered":"2024-08-30T23:55:59.170252+0000","last_unstale":"2024-08-31T00:00:34.401338+0000","last_undegraded":"2024-08-31T00:00:34.401338+0000","last_fullsized":"2024-08-30T23:55:59.167476+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:04.142845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:04.142845+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":["5","8"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1e","version":"1026'7933","reported_seq":13477,"reported_epoch":1026,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:34.851758+0000","last_change":"2024-08-30T23:59:10.753672+0000","last_active":"2024-08-31T00:00:34.851758+0000","last_peered":"2024-08-31T00:00:34.851758+0000","last_clean":"2024-08-31T00:00:34.851758+0000","last_became_active":"2024-08-30T23:58:53.399195+0000","last_became_peered":"2024-08-30T23:58:53.399195+0000","last_unstale":"2024-08-31T00:00:34.851758+0000","last_undegraded":"2024-08-31T00:00:34.851758+0000","last_fullsized":"2024-08-31T00:00:34.851758+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":983,"parent":"0.0","parent_split_bits":5,"last_scrub":"992'4895","last_scrub_stamp":"2024-08-30T23:59:10.753461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:10.753461+0000","objects_scrubbed":3197,"log_size":7933,"log_dups_size":0,"ondisk_log_size":7933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":326,"num_object_clones":0,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":326,"num_objects_unfound":0,"num_objects_dirty":326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":5504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,10],"avail_no_missing":["5","0","10"],"object_location_counts":[{"shards":"0,5,10","objects":326}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":342,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.123353+0000","last_change":"2024-08-30T23:59:45.452154+0000","last_active":"2024-08-31T00:00:36.123353+0000","last_peered":"2024-08-31T00:00:36.123353+0000","last_clean":"2024-08-31T00:00:36.123353+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-31T00:00:36.123353+0000","last_undegraded":"2024-08-31T00:00:36.123353+0000","last_fullsized":"2024-08-31T00:00:36.123353+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:45.452094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:45.452094+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:07.963543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1f","version":"1027'7923","reported_seq":13299,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.699907+0000","last_change":"2024-08-31T00:00:18.592172+0000","last_active":"2024-08-31T00:00:36.699907+0000","last_peered":"2024-08-31T00:00:36.699907+0000","last_clean":"2024-08-31T00:00:36.699907+0000","last_became_active":"2024-08-30T23:58:53.399542+0000","last_became_peered":"2024-08-30T23:58:53.399542+0000","last_unstale":"2024-08-31T00:00:36.699907+0000","last_undegraded":"2024-08-31T00:00:36.699907+0000","last_fullsized":"2024-08-31T00:00:36.699907+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":966,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":5,"last_scrub":"1019'7276","last_scrub_stamp":"2024-08-31T00:00:18.592072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-31T00:00:18.592072+0000","objects_scrubbed":868,"log_size":7923,"log_dups_size":0,"ondisk_log_size":7923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:43.423933+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":340,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.396583+0000","last_change":"2024-08-30T23:59:51.083387+0000","last_active":"2024-08-31T00:00:34.396583+0000","last_peered":"2024-08-31T00:00:34.396583+0000","last_clean":"2024-08-31T00:00:34.396583+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-31T00:00:34.396583+0000","last_undegraded":"2024-08-31T00:00:34.396583+0000","last_fullsized":"2024-08-31T00:00:34.396583+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:51.083308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:51.083308+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:18.443736+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"23'92","reported_seq":545,"reported_epoch":1026,"state":"active+undersized+degraded","last_fresh":"2024-08-31T00:00:34.399317+0000","last_change":"2024-08-30T23:55:59.169756+0000","last_active":"2024-08-31T00:00:34.399317+0000","last_peered":"2024-08-31T00:00:34.399317+0000","last_clean":"2024-08-30T23:55:57.161346+0000","last_became_active":"2024-08-30T23:55:59.169756+0000","last_became_peered":"2024-08-30T23:55:59.169756+0000","last_unstale":"2024-08-31T00:00:34.399317+0000","last_undegraded":"2024-08-30T23:55:59.167597+0000","last_fullsized":"2024-08-30T23:55:59.167501+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.977391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.977391+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.10","version":"1027'6680","reported_seq":8675,"reported_epoch":1027,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:36.695329+0000","last_change":"2024-08-30T23:58:05.325967+0000","last_active":"2024-08-31T00:00:36.695329+0000","last_peered":"2024-08-31T00:00:36.695329+0000","last_clean":"2024-08-31T00:00:36.695329+0000","last_became_active":"2024-08-30T23:58:04.118964+0000","last_became_peered":"2024-08-30T23:58:04.118964+0000","last_unstale":"2024-08-31T00:00:36.695329+0000","last_undegraded":"2024-08-31T00:00:36.695329+0000","last_fullsized":"2024-08-31T00:00:36.695329+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'1457","last_scrub_stamp":"2024-08-30T23:58:05.325770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:05.325770+0000","objects_scrubbed":883,"log_size":6680,"log_dups_size":0,"ondisk_log_size":6680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":263,"num_objects_unfound":0,"num_objects_dirty":263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":147648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0,1],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":263}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":334,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.108212+0000","last_change":"2024-08-30T23:59:23.045008+0000","last_active":"2024-08-31T00:00:36.108212+0000","last_peered":"2024-08-31T00:00:36.108212+0000","last_clean":"2024-08-31T00:00:36.108212+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-31T00:00:36.108212+0000","last_undegraded":"2024-08-31T00:00:36.108212+0000","last_fullsized":"2024-08-31T00:00:36.108212+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:23.044946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:23.044946+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:44.560457+0000","scrub_duration":612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.11","version":"1025'6550","reported_seq":6795,"reported_epoch":1025,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:33.991124+0000","last_change":"2024-08-30T23:59:56.873328+0000","last_active":"2024-08-31T00:00:33.991124+0000","last_peered":"2024-08-31T00:00:33.991124+0000","last_clean":"2024-08-31T00:00:33.991124+0000","last_became_active":"2024-08-30T23:57:56.118862+0000","last_became_peered":"2024-08-30T23:57:56.118862+0000","last_unstale":"2024-08-31T00:00:33.991124+0000","last_undegraded":"2024-08-31T00:00:33.991124+0000","last_fullsized":"2024-08-31T00:00:33.991124+0000","mapping_epoch":951,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":952,"parent":"0.0","parent_split_bits":5,"last_scrub":"1013'5226","last_scrub_stamp":"2024-08-30T23:59:56.873134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:56.873134+0000","objects_scrubbed":1679,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:00.342001+0000","scrub_duration":621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":338,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5404,"num_write_kb":163648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,9],"avail_no_missing":["3","6","9"],"object_location_counts":[{"shards":"3,6,9","objects":338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":324,"reported_epoch":1027,"state":"active+undersized","last_fresh":"2024-08-31T00:00:36.109726+0000","last_change":"2024-08-30T23:55:59.167291+0000","last_active":"2024-08-31T00:00:36.109726+0000","last_peered":"2024-08-31T00:00:36.109726+0000","last_clean":"2024-08-30T23:55:57.155629+0000","last_became_active":"2024-08-30T23:55:59.167291+0000","last_became_peered":"2024-08-30T23:55:59.167291+0000","last_unstale":"2024-08-31T00:00:36.109726+0000","last_undegraded":"2024-08-31T00:00:36.109726+0000","last_fullsized":"2024-08-30T23:55:59.164310+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:55:03.433907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:55:03.433907+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:36.470726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"1026'6460","reported_seq":8513,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.829354+0000","last_change":"2024-08-30T23:58:04.315602+0000","last_active":"2024-08-31T00:00:34.829354+0000","last_peered":"2024-08-31T00:00:34.829354+0000","last_clean":"2024-08-31T00:00:34.829354+0000","last_became_active":"2024-08-30T23:58:04.118740+0000","last_became_peered":"2024-08-30T23:58:04.118740+0000","last_unstale":"2024-08-31T00:00:34.829354+0000","last_undegraded":"2024-08-31T00:00:34.829354+0000","last_fullsized":"2024-08-31T00:00:34.829354+0000","mapping_epoch":957,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":958,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'1428","last_scrub_stamp":"2024-08-30T23:58:04.315539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:58:04.315539+0000","objects_scrubbed":909,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5032,"num_write_kb":141120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":348,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.396371+0000","last_change":"2024-08-31T00:00:22.994210+0000","last_active":"2024-08-31T00:00:34.396371+0000","last_peered":"2024-08-31T00:00:34.396371+0000","last_clean":"2024-08-31T00:00:34.396371+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-31T00:00:34.396371+0000","last_undegraded":"2024-08-31T00:00:34.396371+0000","last_fullsized":"2024-08-31T00:00:34.396371+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:00:22.994155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:00:22.994155+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:36.635741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"1027'6573","reported_seq":10671,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.389080+0000","last_change":"2024-08-30T23:59:52.499193+0000","last_active":"2024-08-31T00:00:36.389080+0000","last_peered":"2024-08-31T00:00:36.389080+0000","last_clean":"2024-08-31T00:00:36.389080+0000","last_became_active":"2024-08-30T23:59:17.091165+0000","last_became_peered":"2024-08-30T23:59:17.091165+0000","last_unstale":"2024-08-31T00:00:36.389080+0000","last_undegraded":"2024-08-31T00:00:36.389080+0000","last_fullsized":"2024-08-31T00:00:36.389080+0000","mapping_epoch":997,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":998,"parent":"0.0","parent_split_bits":5,"last_scrub":"1012'5003","last_scrub_stamp":"2024-08-30T23:59:52.499119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:52.499119+0000","objects_scrubbed":1803,"log_size":6573,"log_dups_size":0,"ondisk_log_size":6573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:01:13.861912+0000","scrub_duration":446,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":145536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,5],"acting":[11,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":340,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:34.396427+0000","last_change":"2024-08-30T23:59:16.147143+0000","last_active":"2024-08-31T00:00:34.396427+0000","last_peered":"2024-08-31T00:00:34.396427+0000","last_clean":"2024-08-31T00:00:34.396427+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-31T00:00:34.396427+0000","last_undegraded":"2024-08-31T00:00:34.396427+0000","last_fullsized":"2024-08-31T00:00:34.396427+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:16.147096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:16.147096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:36.010828+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.14","version":"1027'6753","reported_seq":10492,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.895390+0000","last_change":"2024-08-30T23:59:43.860427+0000","last_active":"2024-08-31T00:00:36.895390+0000","last_peered":"2024-08-31T00:00:36.895390+0000","last_clean":"2024-08-31T00:00:36.895390+0000","last_became_active":"2024-08-30T23:59:18.148949+0000","last_became_peered":"2024-08-30T23:59:18.148949+0000","last_unstale":"2024-08-31T00:00:36.895390+0000","last_undegraded":"2024-08-31T00:00:36.895390+0000","last_fullsized":"2024-08-31T00:00:36.895390+0000","mapping_epoch":998,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":999,"parent":"0.0","parent_split_bits":5,"last_scrub":"1008'4744","last_scrub_stamp":"2024-08-30T23:59:43.860120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:43.860120+0000","objects_scrubbed":2247,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:46.422926+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5365,"num_write_kb":152064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":57278464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":85,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.122455+0000","last_change":"2024-08-30T23:59:15.264826+0000","last_active":"2024-08-31T00:00:36.122455+0000","last_peered":"2024-08-31T00:00:36.122455+0000","last_clean":"2024-08-31T00:00:36.122455+0000","last_became_active":"2024-08-30T23:59:14.206483+0000","last_became_peered":"2024-08-30T23:59:14.206483+0000","last_unstale":"2024-08-31T00:00:36.122455+0000","last_undegraded":"2024-08-31T00:00:36.122455+0000","last_fullsized":"2024-08-31T00:00:36.122455+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:15.264564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:15.264564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.15","version":"1026'6715","reported_seq":8921,"reported_epoch":1026,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:35.593649+0000","last_change":"2024-08-30T23:59:20.137229+0000","last_active":"2024-08-31T00:00:35.593649+0000","last_peered":"2024-08-31T00:00:35.593649+0000","last_clean":"2024-08-31T00:00:35.593649+0000","last_became_active":"2024-08-30T23:58:03.116440+0000","last_became_peered":"2024-08-30T23:58:03.116440+0000","last_unstale":"2024-08-31T00:00:35.593649+0000","last_undegraded":"2024-08-31T00:00:35.593649+0000","last_fullsized":"2024-08-31T00:00:35.593649+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"1000'3865","last_scrub_stamp":"2024-08-30T23:59:20.136835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:20.136835+0000","objects_scrubbed":3160,"log_size":6715,"log_dups_size":0,"ondisk_log_size":6715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:39.001209+0000","scrub_duration":1007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":288,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":148928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4],"acting":[8,4,5],"avail_no_missing":["8","4","5"],"object_location_counts":[{"shards":"4,5,8","objects":288}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":307,"reported_epoch":1027,"state":"active+clean+remapped","last_fresh":"2024-08-31T00:00:36.122502+0000","last_change":"2024-08-30T23:59:15.183405+0000","last_active":"2024-08-31T00:00:36.122502+0000","last_peered":"2024-08-31T00:00:36.122502+0000","last_clean":"2024-08-31T00:00:36.122502+0000","last_became_active":"2024-08-30T23:59:15.181247+0000","last_became_peered":"2024-08-30T23:59:15.181247+0000","last_unstale":"2024-08-31T00:00:36.122502+0000","last_undegraded":"2024-08-31T00:00:36.122502+0000","last_fullsized":"2024-08-31T00:00:36.122502+0000","mapping_epoch":995,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":996,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-30T23:59:13.150214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-30T23:59:13.150214+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:38.671130+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8],"acting":[11,8,10],"avail_no_missing":["11","8","10"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.16","version":"1027'6755","reported_seq":11894,"reported_epoch":1027,"state":"active+clean","last_fresh":"2024-08-31T00:00:36.392782+0000","last_change":"2024-08-30T23:59:43.915598+0000","last_active":"2024-08-31T00:00:36.392782+0000","last_peered":"2024-08-31T00:00:36.392782+0000","last_clean":"2024-08-31T00:00:36.392782+0000","last_became_active":"2024-08-30T23:59:19.090116+0000","last_became_peered":"2024-08-30T23:59:19.090116+0000","last_unstale":"2024-08-31T00:00:36.392782+0000","last_undegraded":"2024-08-31T00:00:36.392782+0000","last_fullsized":"2024-08-31T00:00:36.392782+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":1000,"parent":"0.0","parent_split_bits":5,"last_scrub":"1008'4773","last_scrub_stamp":"2024-08-30T23:59:43.915494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:43.915494+0000","objects_scrubbed":2244,"log_size":6755,"log_dups_size":0,"ondisk_log_size":6755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:45.338460+0000","scrub_duration":742,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16252928,"num_objects":248,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5287,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":82116608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,2],"acting":[11,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.17","version":"1026'6565","reported_seq":8909,"reported_epoch":1026,"state":"active+clean","last_fresh":"2024-08-31T00:00:35.653128+0000","last_change":"2024-08-30T23:59:50.349233+0000","last_active":"2024-08-31T00:00:35.653128+0000","last_peered":"2024-08-31T00:00:35.653128+0000","last_clean":"2024-08-31T00:00:35.653128+0000","last_became_active":"2024-08-30T23:58:05.121618+0000","last_became_peered":"2024-08-30T23:58:05.121618+0000","last_unstale":"2024-08-31T00:00:35.653128+0000","last_undegraded":"2024-08-31T00:00:35.653128+0000","last_fullsized":"2024-08-31T00:00:35.653128+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":948,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"1010'4930","last_scrub_stamp":"2024-08-30T23:59:50.349053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:57:35.791607+0000","last_clean_scrub_stamp":"2024-08-30T23:59:50.349053+0000","objects_scrubbed":1883,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:00:53.940785+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16187392,"num_objects":247,"num_object_clones":0,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":142464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":36,"stat_sum":{"num_bytes":568950808,"num_objects":8682,"num_object_clones":0,"num_object_copies":26046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1097,"num_objects_misplaced":2119,"num_objects_unfound":0,"num_objects_dirty":8682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147290,"num_write_kb":3543969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5301,"num_bytes_recovered":347406336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1631006720,"data_stored":1630994504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234845,"ondisk_log_size":234845,"up":96,"acting":104,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630621,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":84147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":81,"acting":84,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855612,"kb_used_data":120548,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92889540,"statfs":{"total":95995035648,"available":95118888960,"internally_reserved":0,"allocated":123441152,"data_stored":122388526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":752679364},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449531,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912888,"kb_used_data":155556,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92832264,"statfs":{"total":95995035648,"available":95060238336,"internally_reserved":0,"allocated":159289344,"data_stored":158236718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":775420352},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711159,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107360,"kb_used_data":205712,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92637792,"statfs":{"total":95995035648,"available":94861099008,"internally_reserved":0,"allocated":210649088,"data_stored":209600562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":923269568},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972803,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":659116,"kb_used_data":86488,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93086036,"statfs":{"total":95995035648,"available":95320100864,"internally_reserved":0,"allocated":88563712,"data_stored":87515172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,5,6,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234382,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1304480,"kb_used_data":579144,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":92440672,"statfs":{"total":95995035648,"available":94659248128,"internally_reserved":0,"allocated":593043456,"data_stored":592258535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":742717898},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528780,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":642616,"kb_used_data":79716,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93102536,"statfs":{"total":95995035648,"available":95336996864,"internally_reserved":0,"allocated":81629184,"data_stored":80576558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,1,2,3,4,5,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775283,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844904,"kb_used_data":348376,"kb_used_omap":5,"kb_used_meta":496506,"kb_avail":92900248,"statfs":{"total":95995035648,"available":95129853952,"internally_reserved":0,"allocated":356737024,"data_stored":355684946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":508422218},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953943,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":842172,"kb_used_data":75172,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92902980,"statfs":{"total":95995035648,"available":95132651520,"internally_reserved":0,"allocated":76976128,"data_stored":75923502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150168,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202396,"kb_used_data":196620,"kb_used_omap":1,"kb_used_meta":1005758,"kb_avail":92542756,"statfs":{"total":95995035648,"available":94763782144,"internally_reserved":0,"allocated":201338880,"data_stored":200294426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1029896647},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411806,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766052,"kb_used_data":58708,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92979100,"statfs":{"total":95995035648,"available":95210598400,"internally_reserved":0,"allocated":60116992,"data_stored":59068986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830623,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":962680,"kb_used_data":235044,"kb_used_omap":87,"kb_used_meta":727528,"kb_avail":92782472,"statfs":{"total":95995035648,"available":95009251328,"internally_reserved":0,"allocated":240685056,"data_stored":239632430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":744988831},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090080,"kb_used_data":184272,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92655072,"statfs":{"total":95995035648,"available":94878793728,"internally_reserved":0,"allocated":188694528,"data_stored":187646002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":171905024,"data_stored":171900952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228065280,"data_stored":228065280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":55443456,"data_stored":55443456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180944896,"data_stored":180944896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":72417280,"data_stored":72417280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":325193728,"data_stored":325189656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":68878336,"data_stored":68878336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":79233024,"data_stored":79233024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":189206528,"data_stored":189202456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":144506880,"data_stored":144506880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":115212288,"data_stored":115212288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T00:00:42.798 INFO:tasks.thrashosds.thrasher:Setting 1.c to [7, 4, 0, 5, 6, 8] 2024-08-31T00:00:42.798 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.c', '7', '4', '0', '5', '6', '8'] 2024-08-31T00:00:42.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.c 7 4 0 5 6 8 2024-08-31T00:00:42.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/144890314' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:42.880 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:42.911 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:42.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:43.004 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:43.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.072 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:43.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:43.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.204 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:43.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:43.305 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:43.336 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:43.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:43.461 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:43.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:43.585 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:43.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:43 smithi106 ceph-mon[16603]: pgmap v2272: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 302 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 813/14469 objects degraded (5.619%); 1061/14469 objects misplaced (7.333%) 2024-08-31T00:00:43.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:43 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:43.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:43 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:43.676 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:43 smithi106 ceph-mon[16603]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:43.677 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:43 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/144890314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:43.677 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:43 smithi106 ceph-mon[16603]: osdmap e1028: 12 total, 11 up, 12 in 2024-08-31T00:00:43.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:43.708 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.709 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:43 smithi136 ceph-mon[20207]: pgmap v2272: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 302 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 813/14469 objects degraded (5.619%); 1061/14469 objects misplaced (7.333%) 2024-08-31T00:00:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:43 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:43 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:43 smithi136 ceph-mon[20207]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:43 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/144890314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:43.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:43 smithi136 ceph-mon[20207]: osdmap e1028: 12 total, 11 up, 12 in 2024-08-31T00:00:43.810 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:43 smithi173 ceph-mon[20554]: pgmap v2272: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 302 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 813/14469 objects degraded (5.619%); 1061/14469 objects misplaced (7.333%) 2024-08-31T00:00:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:43 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:43 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:43 smithi173 ceph-mon[20554]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:43 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/144890314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:43.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:43 smithi173 ceph-mon[20554]: osdmap e1028: 12 total, 11 up, 12 in 2024-08-31T00:00:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.834 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:43.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:43.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:43.957 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:43.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:44.041 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-31T00:00:44.041 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [], 2024-08-31T00:00:44.041 INFO:teuthology.orchestra.run.smithi106.stdout: "num_ops": 0 2024-08-31T00:00:44.041 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-31T00:00:44.058 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:44.080 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:44.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:44.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:44.203 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:44.305 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.328 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:44.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:44.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:44.434 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Removed 102001 objects 2024-08-31T00:00:44.435 INFO:tasks.radosbench.radosbench.2.smithi173.stdout:Clean up completed and total clean up time :89.4382 2024-08-31T00:00:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.453 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:44.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:44.554 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:44.575 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:44.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:44 smithi106 ceph-mon[16603]: 9.15 scrub starts 2024-08-31T00:00:44.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:44 smithi106 ceph-mon[16603]: 9.15 scrub ok 2024-08-31T00:00:44.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:44 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-31T00:00:44.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:44 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-31T00:00:44.677 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:44.700 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:44.709 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:44.738 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-31T00:00:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:44 smithi136 ceph-mon[20207]: 9.15 scrub starts 2024-08-31T00:00:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:44 smithi136 ceph-mon[20207]: 9.15 scrub ok 2024-08-31T00:00:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:44 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-31T00:00:44.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:44 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-31T00:00:44.802 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:44 smithi173 ceph-mon[20554]: 9.15 scrub starts 2024-08-31T00:00:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:44 smithi173 ceph-mon[20554]: 9.15 scrub ok 2024-08-31T00:00:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:44 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-31T00:00:44.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:44 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-31T00:00:44.843 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:44.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:44.976 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:44.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:45.077 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:45.098 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:45.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:45.174 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:45.174 INFO:teuthology.orchestra.run.smithi106.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-31T00:00:45.198 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:45.198 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:45.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:45.225 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:45.225 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:45.226 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:45.238 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:45.250 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:45.250 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:45.270 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:45.274 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:45.275 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:45.297 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:45.297 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:45.318 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:45.319 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:45.319 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:45.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:45.340 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:45.341 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:45.354 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:45.361 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:45.361 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:45.383 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:45.383 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:45.405 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:45.405 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:45.456 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:45.478 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:45.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:45 smithi106 ceph-mon[16603]: pgmap v2274: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 214 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 449/10266 objects degraded (4.374%); 870/10266 objects misplaced (8.475%) 2024-08-31T00:00:45.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:45 smithi106 ceph-mon[16603]: 9.1a scrub starts 2024-08-31T00:00:45.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:45 smithi106 ceph-mon[16603]: 9.1a scrub ok 2024-08-31T00:00:45.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:45 smithi106 ceph-mon[16603]: 9.8 scrub starts 2024-08-31T00:00:45.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:45 smithi106 ceph-mon[16603]: 9.8 scrub ok 2024-08-31T00:00:45.572 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2389933697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:45.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:45.602 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-31T00:00:45.605 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:45.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:45.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:45.733 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:45 smithi136 ceph-mon[20207]: pgmap v2274: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 214 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 449/10266 objects degraded (4.374%); 870/10266 objects misplaced (8.475%) 2024-08-31T00:00:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:45 smithi136 ceph-mon[20207]: 9.1a scrub starts 2024-08-31T00:00:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:45 smithi136 ceph-mon[20207]: 9.1a scrub ok 2024-08-31T00:00:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:45 smithi136 ceph-mon[20207]: 9.8 scrub starts 2024-08-31T00:00:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:45 smithi136 ceph-mon[20207]: 9.8 scrub ok 2024-08-31T00:00:45.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2389933697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:45 smithi173 ceph-mon[20554]: pgmap v2274: 67 pgs: 11 active+clean+remapped, 8 active+undersized, 5 active+undersized+degraded, 43 active+clean; 214 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s; 449/10266 objects degraded (4.374%); 870/10266 objects misplaced (8.475%) 2024-08-31T00:00:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:45 smithi173 ceph-mon[20554]: 9.1a scrub starts 2024-08-31T00:00:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:45 smithi173 ceph-mon[20554]: 9.1a scrub ok 2024-08-31T00:00:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:45 smithi173 ceph-mon[20554]: 9.8 scrub starts 2024-08-31T00:00:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:45 smithi173 ceph-mon[20554]: 9.8 scrub ok 2024-08-31T00:00:45.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2389933697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:45.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:45.859 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:45.957 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-08-31T00:00:45.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-08-31T00:00:45.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:45.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:45.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:45.988 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:46.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:46.115 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:46.217 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:46.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.244 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:46.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:46.345 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:46.371 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:46.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:46.494 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:46.567 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:46 smithi106 ceph-mon[16603]: 1.1c scrub starts 2024-08-31T00:00:46.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:46 smithi106 ceph-mon[16603]: 1.1c scrub ok 2024-08-31T00:00:46.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:46 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 449/10266 objects degraded (4.374%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:46.568 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:46 smithi106 ceph-mon[16603]: osdmap e1029: 12 total, 11 up, 12 in 2024-08-31T00:00:46.595 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:46.616 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:46.692 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:46.718 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:46.739 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:46 smithi136 ceph-mon[20207]: 1.1c scrub starts 2024-08-31T00:00:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:46 smithi136 ceph-mon[20207]: 1.1c scrub ok 2024-08-31T00:00:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:46 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 449/10266 objects degraded (4.374%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:46.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:46 smithi136 ceph-mon[20207]: osdmap e1029: 12 total, 11 up, 12 in 2024-08-31T00:00:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:46 smithi173 ceph-mon[20554]: 1.1c scrub starts 2024-08-31T00:00:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:46 smithi173 ceph-mon[20554]: 1.1c scrub ok 2024-08-31T00:00:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:46 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 449/10266 objects degraded (4.374%), 5 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:46.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:46 smithi173 ceph-mon[20554]: osdmap e1029: 12 total, 11 up, 12 in 2024-08-31T00:00:46.841 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:46.867 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:46.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:46.998 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:46.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:47.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9 2024-08-31T00:00:47.125 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.9' 2024-08-31T00:00:47.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:47.252 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:47.354 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:47.376 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:47.444 INFO:teuthology.orchestra.run.smithi106.stderr:set 1.c pg_upmap_items mapping to [7->4,0->5,6->8] 2024-08-31T00:00:47.445 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-31T00:00:47.475 INFO:teuthology.orchestra.run.smithi106.stdout:ERROR: (22) Invalid argument 2024-08-31T00:00:47.475 INFO:teuthology.orchestra.run.smithi106.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-31T00:00:47.477 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.500 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:47.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:47.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:47.628 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:47.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: pgmap v2275: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 9 active+undersized, 4 active+undersized+degraded, 42 active+clean; 56 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 161/2661 objects degraded (6.050%); 224/2661 objects misplaced (8.418%) 2024-08-31T00:00:47.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-31T00:00:47.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-31T00:00:47.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/905624281' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [7, 4, 0, 5, 6, 8]} : dispatch 2024-08-31T00:00:47.690 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-31T00:00:47.690 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-31T00:00:47.690 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:47 smithi106 ceph-mon[16603]: from='client.19563 172.21.15.106:0/31762082' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:47.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:47.756 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: pgmap v2275: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 9 active+undersized, 4 active+undersized+degraded, 42 active+clean; 56 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 161/2661 objects degraded (6.050%); 224/2661 objects misplaced (8.418%) 2024-08-31T00:00:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-31T00:00:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-31T00:00:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/905624281' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [7, 4, 0, 5, 6, 8]} : dispatch 2024-08-31T00:00:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-31T00:00:47.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-31T00:00:47.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:47 smithi136 ceph-mon[20207]: from='client.19563 172.21.15.106:0/31762082' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: pgmap v2275: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 9 active+undersized, 4 active+undersized+degraded, 42 active+clean; 56 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 161/2661 objects degraded (6.050%); 224/2661 objects misplaced (8.418%) 2024-08-31T00:00:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-31T00:00:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-31T00:00:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/905624281' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [7, 4, 0, 5, 6, 8]} : dispatch 2024-08-31T00:00:47.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-31T00:00:47.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-31T00:00:47.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:47 smithi173 ceph-mon[20554]: from='client.19563 172.21.15.106:0/31762082' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:00:47.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:47.880 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:47.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:47.982 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:48.003 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:48.106 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:48.127 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:48.229 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:48.244 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-31T00:00:48.255 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:48.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:48.383 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:48.485 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:48.506 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:48.565 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:48 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-31T00:00:48.565 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/905624281' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [7, 4, 0, 5, 6, 8]}]': finished 2024-08-31T00:00:48.565 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:48 smithi106 ceph-mon[16603]: from='client.19563 172.21.15.106:0/31762082' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:48.565 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:48 smithi106 ceph-mon[16603]: osdmap e1030: 12 total, 11 up, 12 in 2024-08-31T00:00:48.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-31T00:00:48.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:48.634 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:48.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.762 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:48.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:48 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-31T00:00:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/905624281' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [7, 4, 0, 5, 6, 8]}]': finished 2024-08-31T00:00:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:48 smithi136 ceph-mon[20207]: from='client.19563 172.21.15.106:0/31762082' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:48.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:48 smithi136 ceph-mon[20207]: osdmap e1030: 12 total, 11 up, 12 in 2024-08-31T00:00:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:48 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-31T00:00:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/905624281' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [7, 4, 0, 5, 6, 8]}]': finished 2024-08-31T00:00:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:48 smithi173 ceph-mon[20554]: from='client.19563 172.21.15.106:0/31762082' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:00:48.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:48 smithi173 ceph-mon[20554]: osdmap e1030: 12 total, 11 up, 12 in 2024-08-31T00:00:48.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:48.888 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:48.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:48.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:49.018 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:49.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:49.145 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:49.247 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:49.268 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:49.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.397 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:49.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:49.498 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.523 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:49.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:49.541 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:49.580 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:49.583 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:49.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:49.651 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:49.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:49.771 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: pgmap v2278: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 42 active+clean; 3.7 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 50/153 objects degraded (32.680%) 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: osdmap e1031: 12 total, 11 up, 12 in 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: 9.3 scrub starts 2024-08-31T00:00:49.772 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:49 smithi106 ceph-mon[16603]: 9.3 scrub ok 2024-08-31T00:00:49.778 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: pgmap v2278: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 42 active+clean; 3.7 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 50/153 objects degraded (32.680%) 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: osdmap e1031: 12 total, 11 up, 12 in 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: 9.3 scrub starts 2024-08-31T00:00:49.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:49 smithi136 ceph-mon[20207]: 9.3 scrub ok 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: pgmap v2278: 67 pgs: 1 active+clean+scrubbing, 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 42 active+clean; 3.7 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s; 50/153 objects degraded (32.680%) 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: osdmap e1031: 12 total, 11 up, 12 in 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: 9.3 scrub starts 2024-08-31T00:00:49.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:49 smithi173 ceph-mon[20554]: 9.3 scrub ok 2024-08-31T00:00:49.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:49.905 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:49.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:50.007 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.030 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:50.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:50.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.154 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:50.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:50.214 INFO:teuthology.orchestra.run.smithi106.stdout:osd.0: {} 2024-08-31T00:00:50.235 INFO:teuthology.orchestra.run.smithi106.stdout:osd.1: {} 2024-08-31T00:00:50.236 INFO:teuthology.orchestra.run.smithi106.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-31T00:00:50.255 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:50.256 INFO:teuthology.orchestra.run.smithi106.stdout:osd.2: {} 2024-08-31T00:00:50.257 INFO:teuthology.orchestra.run.smithi106.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-31T00:00:50.278 INFO:teuthology.orchestra.run.smithi106.stdout:osd.3: {} 2024-08-31T00:00:50.279 INFO:teuthology.orchestra.run.smithi106.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-31T00:00:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.280 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:50.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:50.300 INFO:teuthology.orchestra.run.smithi106.stdout:osd.4: {} 2024-08-31T00:00:50.301 INFO:teuthology.orchestra.run.smithi106.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-31T00:00:50.322 INFO:teuthology.orchestra.run.smithi106.stdout:osd.5: {} 2024-08-31T00:00:50.322 INFO:teuthology.orchestra.run.smithi106.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-31T00:00:50.344 INFO:teuthology.orchestra.run.smithi106.stdout:osd.6: {} 2024-08-31T00:00:50.345 INFO:teuthology.orchestra.run.smithi106.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-31T00:00:50.345 INFO:teuthology.orchestra.run.smithi106.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-31T00:00:50.367 INFO:teuthology.orchestra.run.smithi106.stdout:osd.8: {} 2024-08-31T00:00:50.367 INFO:teuthology.orchestra.run.smithi106.stderr:osd.8: osd_enable_op_tracker = '' 2024-08-31T00:00:50.382 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3 2024-08-31T00:00:50.388 INFO:teuthology.orchestra.run.smithi106.stdout:osd.9: {} 2024-08-31T00:00:50.388 INFO:teuthology.orchestra.run.smithi106.stderr:osd.9: osd_enable_op_tracker = '' 2024-08-31T00:00:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.403 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:50.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.3' 2024-08-31T00:00:50.410 INFO:teuthology.orchestra.run.smithi106.stdout:osd.10: {} 2024-08-31T00:00:50.410 INFO:teuthology.orchestra.run.smithi106.stderr:osd.10: osd_enable_op_tracker = '' 2024-08-31T00:00:50.432 INFO:teuthology.orchestra.run.smithi106.stdout:osd.11: {} 2024-08-31T00:00:50.433 INFO:teuthology.orchestra.run.smithi106.stderr:osd.11: osd_enable_op_tracker = '' 2024-08-31T00:00:50.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:50.531 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:50.633 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2 2024-08-31T00:00:50.655 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.2' 2024-08-31T00:00:50.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:50 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-31T00:00:50.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:50 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-31T00:00:50.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:50 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-31T00:00:50.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:50 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-31T00:00:50.748 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3393942387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:50.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:50.784 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.785 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:50 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-31T00:00:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:50 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-31T00:00:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:50 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-31T00:00:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:50 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-31T00:00:50.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3393942387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:50 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-31T00:00:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:50 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-31T00:00:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:50 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-31T00:00:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:50 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-31T00:00:50.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3393942387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-31T00:00:50.847 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-08-31T00:00:50.847 INFO:teuthology.orchestra.run.smithi106.stdout: "size": 20, 2024-08-31T00:00:50.847 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 600, 2024-08-31T00:00:50.847 INFO:teuthology.orchestra.run.smithi106.stdout: "ops": [ 2024-08-31T00:00:50.847 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e833,800 7:e3e0581a:::benchmark_data_smithi173_129296_object20938:head,781'6741)", 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:38.907308+0000", 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 431.93854708399999, 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56565642400000005, 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.848 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.849 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.850 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.850 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.850 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907308+0000", 2024-08-31T00:00:50.850 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.92e-07 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:38.907313+0000", 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5199999999999999e-06 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472540+0000", 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.56522729500000002 2024-08-31T00:00:50.851 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472548+0000", 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9780000000000005e-06 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472616+0000", 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7745999999999998e-05 2024-08-31T00:00:50.852 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.854 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.854 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.854 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:39.472964+0000", 2024-08-31T00:00:50.854 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00034819300000000001 2024-08-31T00:00:50.854 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,800 7:e6a7a338:::benchmark_data_smithi173_129296_object26093:head,783'7057)", 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:41.361662+0000", 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 429.48419247099997, 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29360009100000001, 2024-08-31T00:00:50.856 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361662+0000", 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.857 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361662+0000", 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361663+0000", 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4499999999999997e-07 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.858 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.361666+0000", 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.579e-06 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654885+0000", 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.29321898600000001 2024-08-31T00:00:50.859 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654896+0000", 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.0654e-05 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.654958+0000", 2024-08-31T00:00:50.860 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.2126999999999997e-05 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:41.655262+0000", 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00030420000000000002 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.861 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,800 7:e80dc891:::benchmark_data_smithi173_129296_object31110:head,789'7374)", 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:53:43.075101+0000", 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 427.770754092, 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70067806200000005, 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-31T00:00:50.862 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-31T00:00:50.863 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075101+0000", 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9999999999999997e-07 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.864 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.075106+0000", 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.6249999999999998e-06 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775466+0000", 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.70036012700000005 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775475+0000", 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.6789999999999994e-06 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775526+0000", 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1054000000000003e-05 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.866 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:53:43.775779+0000", 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00025287699999999998 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:36528 8.7 8:ed4e2712:::benchmark_data_smithi173_143816_object36527:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e862)", 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:25.367630+0000", 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 385.47822481399999, 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46017543799999999, 2024-08-31T00:00:50.867 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 36528 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367630+0000", 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.868 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367630+0000", 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367631+0000", 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.548e-06 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367685+0000", 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.4001e-05 2024-08-31T00:00:50.869 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367695+0000", 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.6150000000000003e-06 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.367700+0000", 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5650000000000003e-06 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368384+0000", 2024-08-31T00:00:50.870 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00068396600000000004 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368520+0000", 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00013615799999999999 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 10,11", 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.368643+0000", 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00012345699999999999 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.871 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.370618+0000", 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001974632 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.373944+0000", 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0033259520000000001 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.826917+0000", 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.452973338 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.872 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.826991+0000", 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.3951999999999999e-05 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:25.827805+0000", 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00081425400000000004 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.873 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37734 8.1 8:87d95214:::benchmark_data_smithi173_143816_object37733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.155654+0000", 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 383.69020068700002, 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33538572700000002, 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37734 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.874 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155654+0000", 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2300000000000001e-07 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.875 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155716+0000", 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.1576999999999997e-05 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155723+0000", 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8770000000000004e-06 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.155733+0000", 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.9280000000000005e-06 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.876 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.156800+0000", 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0010672489999999999 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.156972+0000", 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000171516 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.157064+0000", 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.2851999999999998e-05 2024-08-31T00:00:50.877 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.157928+0000", 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00086371099999999997 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.164297+0000", 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0063691900000000003 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490984+0000", 2024-08-31T00:00:50.878 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.32668678800000001 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491017+0000", 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3030000000000001e-05 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491040+0000", 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.2486000000000001e-05 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.879 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37734 8.1 e864/840 ondisk, result = 0)", 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.158056+0000", 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 383.68779886800002, 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33298887799999999, 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.880 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158056+0000", 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8500000000000002e-07 2024-08-31T00:00:50.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.158060+0000", 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3620000000000002e-06 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490938+0000", 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33287868399999998 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.882 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490943+0000", 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.481e-06 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490976+0000", 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3006999999999998e-05 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.490981+0000", 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0710000000000002e-06 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.883 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.884 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.491045+0000", 2024-08-31T00:00:50.884 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.3788000000000007e-05 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:37770 8.1 8:8a7dfc05:::benchmark_data_smithi173_143816_object37769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e864)", 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.228037+0000", 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 383.61781728099999, 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.393374581, 2024-08-31T00:00:50.885 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 37770 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228037+0000", 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.886 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228037+0000", 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228039+0000", 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2550000000000001e-06 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228154+0000", 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000115809 2024-08-31T00:00:50.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228166+0000", 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1505000000000001e-05 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228171+0000", 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.0499999999999999e-06 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.888 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228250+0000", 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.9462000000000006e-05 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228340+0000", 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.9864000000000006e-05 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-31T00:00:50.889 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.228442+0000", 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000101697 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229583+0000", 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0011410179999999999 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.890 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.264158+0000", 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.034575084999999998 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621315+0000", 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.35715717000000002 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.891 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621360+0000", 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4866e-05 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621412+0000", 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1799999999999999e-05 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:37770 8.1 e864/840 ondisk, result = 0)", 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:27.229920+0000", 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 383.61593466900001, 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39149838599999998, 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-31T00:00:50.893 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229920+0000", 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229920+0000", 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229921+0000", 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.9800000000000003e-07 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.229924+0000", 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3330000000000001e-06 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.895 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621242+0000", 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.39131752400000003 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621251+0000", 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4860000000000003e-06 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.896 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621303+0000", 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2337999999999999e-05 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621311+0000", 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4320000000000002e-06 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.897 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:27.621418+0000", 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000107575 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(8.f 864/854 [PushReplyOp(8:ff509c29:::benchmark_data_smithi173_143816_object7126:head)])", 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:28.072822+0000", 2024-08-31T00:00:50.898 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 382.773032661, 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46479312699999997, 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072822+0000", 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072822+0000", 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072823+0000", 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4580000000000001e-06 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.900 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.072831+0000", 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1890000000000001e-06 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536520+0000", 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46368978100000002 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536534+0000", 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3192e-05 2024-08-31T00:00:50.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.536618+0000", 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.4377999999999997e-05 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:28.537615+0000", 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00099712900000000003 2024-08-31T00:00:50.902 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.27853.0:45729 8.1 8:842ca331:::benchmark_data_smithi173_143816_object45728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:36.750932+0000", 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.09492262999999, 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.51209708499999995, 2024-08-31T00:00:50.903 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.27853", 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 45729 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750932+0000", 2024-08-31T00:00:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750932+0000", 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750934+0000", 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6789999999999999e-06 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.905 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750972+0000", 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.8312000000000002e-05 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750988+0000", 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6388e-05 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.906 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.750996+0000", 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.4989999999999997e-06 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751056+0000", 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.9786e-05 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.907 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751197+0000", 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000140879 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 2,9", 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.751340+0000", 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00014324499999999999 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752873+0000", 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00153348 2024-08-31T00:00:50.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.753371+0000", 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00049803800000000002 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262925+0000", 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50955419099999999 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262977+0000", 2024-08-31T00:00:50.909 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1261999999999998e-05 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.263029+0000", 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.2326e-05 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.910 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.27853.0:45729 8.1 e869/840 ondisk, result = 0)", 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:54:36.752989+0000", 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 374.09286577300003, 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.51004729900000001, 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.911 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752989+0000", 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752989+0000", 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.912 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752990+0000", 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0100000000000004e-07 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:36.752993+0000", 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.619e-06 2024-08-31T00:00:50.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262804+0000", 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.50981123900000003 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262811+0000", 2024-08-31T00:00:50.914 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6780000000000003e-06 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262906+0000", 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4585000000000003e-05 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.262918+0000", 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2184000000000001e-05 2024-08-31T00:00:50.915 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:54:37.263036+0000", 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000118293 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:19261 9.4 9:27013f3f:::benchmark_data_smithi173_160189_object19260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.083444+0000", 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.762410303, 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46293038199999997, 2024-08-31T00:00:50.916 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 19261 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083444+0000", 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.917 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083444+0000", 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083446+0000", 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.215e-06 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.918 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083483+0000", 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.7916000000000001e-05 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083495+0000", 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.199e-05 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.919 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083501+0000", 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1229999999999999e-06 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083540+0000", 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.9263e-05 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.920 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083624+0000", 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3987999999999996e-05 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 4,8", 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.083719+0000", 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.4710000000000006e-05 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084202+0000", 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00048355400000000003 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.921 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.138227+0000", 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.054025380999999997 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546203+0000", 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.407975486 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546303+0000", 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00010033400000000001 2024-08-31T00:00:50.922 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546375+0000", 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1421999999999995e-05 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop_reply(client.28405.0:19261 9.4 e950/948 ondisk, result = 0)", 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.084652+0000", 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.76120214900001, 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46173567799999998, 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.923 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084652+0000", 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084652+0000", 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.924 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084653+0000", 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.1500000000000005e-07 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.084656+0000", 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.118e-06 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546041+0000", 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.46138474899999998 2024-08-31T00:00:50.925 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546057+0000", 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.662e-05 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546176+0000", 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00011835 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546193+0000", 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6986999999999998e-05 2024-08-31T00:00:50.926 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.546388+0000", 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00019533899999999999 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.28405.0:19270 9.f e950/938 9:f74eb61c:::benchmark_data_smithi173_160189_object19269:head v 950'1246, mlcod=949'1244)", 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.328695+0000", 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.51715989300001, 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31213600899999999, 2024-08-31T00:00:50.927 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328695+0000", 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328695+0000", 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.928 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328696+0000", 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.536e-06 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.328785+0000", 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.8706000000000001e-05 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639878+0000", 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31109302799999999 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.929 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639884+0000", 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.4629999999999998e-06 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639941+0000", 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6628000000000001e-05 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639961+0000", 2024-08-31T00:00:50.930 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.944e-05 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.640802+0000", 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00084156699999999999 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.640831+0000", 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8640999999999999e-05 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.931 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "replica_scrub(pg: 9.6,from:0'0,to:949'1111,epoch:950/948,start:9:6299959b:::benchmark_data_smithi173_160189_object13888:0,end:9:63fa1766:::benchmark_data_smithi173_160189_object12833:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:57:54.639844+0000", 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 176.206011034, 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.306055256, 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "started", 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.932 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639844+0000", 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639844+0000", 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639845+0000", 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.540000000000001e-07 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.933 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.639852+0000", 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.0559999999999999e-06 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945749+0000", 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.30589699599999998 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945758+0000", 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0739999999999999e-06 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.934 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945829+0000", 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.1110000000000002e-05 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945845+0000", 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.6690000000000001e-05 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:57:54.945899+0000", 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3375999999999999e-05 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.935 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.28405.0:57585 9.22 e970/969 9:447b4f71:::benchmark_data_smithi173_160189_object57584:head v 970'2231, mlcod=953'1283)", 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:28.894879+0000", 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 141.95097540099999, 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38515502400000001, 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.936 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894879+0000", 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894879+0000", 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894880+0000", 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.5099999999999998e-07 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.937 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:28.894961+0000", 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.1088999999999996e-05 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.276804+0000", 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.38184307299999998 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.276811+0000", 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6200000000000001e-06 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.938 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.277218+0000", 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.000407484 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.277243+0000", 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.4284999999999999e-05 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.279951+0000", 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0027079740000000001 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.939 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:29.280034+0000", 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.3647999999999996e-05 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "MOSDPGPushReply(9.20 973/969 [PushReplyOp(9:05c2bc44:::benchmark_data_smithi173_160189_object27957:head)])", 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:58:34.746380+0000", 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 136.09947496500001, 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31719325700000001, 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.940 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "reached pg", 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746380+0000", 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746380+0000", 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.941 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746380+0000", 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.2099999999999998e-07 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:34.746383+0000", 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.8459999999999999e-06 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.062098+0000", 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31571540199999998 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.942 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.062107+0000", 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.1300000000000007e-06 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.062815+0000", 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00070730900000000004 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:58:35.063573+0000", 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00075824900000000001 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.943 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:102698 9.1f 9:fe82fe08:::benchmark_data_smithi173_160189_object694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996)", 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:15.773116+0000", 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 95.072739103000004, 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31421399300000002, 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:50.944 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 102698 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773116+0000", 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773116+0000", 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.945 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773117+0000", 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.1739999999999999e-06 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773121+0000", 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.0940000000000001e-06 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773134+0000", 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.3213999999999999e-05 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.946 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773142+0000", 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.5510000000000002e-06 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773185+0000", 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3074000000000001e-05 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773263+0000", 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 7.8102999999999994e-05 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.947 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,10", 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.773329+0000", 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.6154000000000004e-05 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.774707+0000", 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.001377992 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:15.775166+0000", 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.00045898800000000002 2024-08-31T00:00:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:16.087267+0000", 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.31210154499999998 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:16.087303+0000", 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.5782999999999999e-05 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:16.087330+0000", 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 2.6321000000000001e-05 2024-08-31T00:00:50.949 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_repop(client.28405.0:103639 9.1a e999/982 9:5fb158be:::benchmark_data_smithi173_160189_object1635:head v 999'4810, mlcod=999'4807)", 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-30T23:59:18.739569+0000", 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 92.106285322999994, 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.34070336600000001, 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.950 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:18.739569+0000", 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:18.739569+0000", 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:18.739570+0000", 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.6700000000000003e-07 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.951 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:18.739574+0000", 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.4839999999999998e-06 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:19.078667+0000", 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.33909238400000002 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:19.078671+0000", 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.1300000000000003e-06 2024-08-31T00:00:50.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:19.078724+0000", 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 5.3544000000000001e-05 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:19.078744+0000", 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.9428000000000001e-05 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:19.080239+0000", 2024-08-31T00:00:50.953 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0014956209999999999 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-30T23:59:19.080273+0000", 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 3.3207999999999998e-05 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "description": "osd_op(client.28405.0:196161 9.1f 9:f8a0b88a:::benchmark_data_smithi173_160189_object94157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1027)", 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "initiated_at": "2024-08-31T00:00:36.468075+0000", 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "age": 14.377779156000001, 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.15251644, 2024-08-31T00:00:50.954 INFO:teuthology.orchestra.run.smithi106.stdout: "continuous": false, 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "type_data": { 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "client_info": { 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "client": "client.28405", 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "tid": 196161 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "events": [ 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "initiated", 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468075+0000", 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.955 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "header_read", 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468075+0000", 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "throttled", 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468077+0000", 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.4389999999999999e-06 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "all_read", 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468081+0000", 2024-08-31T00:00:50.956 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.3320000000000002e-06 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "dispatched", 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468094+0000", 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 1.2882e-05 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "queued_for_pg", 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468101+0000", 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.8070000000000002e-06 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "reached_pg", 2024-08-31T00:00:50.957 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468147+0000", 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.5691e-05 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "started", 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468227+0000", 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 8.0273999999999997e-05 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "waiting for subops from 6,10", 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.468295+0000", 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 6.7735000000000001e-05 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "op_commit", 2024-08-31T00:00:50.958 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.473525+0000", 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.0052301819999999999 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.480170+0000", 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.006644896 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.620453+0000", 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 0.140283459 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "commit_sent", 2024-08-31T00:00:50.959 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.620544+0000", 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 9.0585000000000001e-05 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: { 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: "event": "done", 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: "time": "2024-08-31T00:00:36.620592+0000", 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: "duration": 4.8158000000000002e-05 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-08-31T00:00:50.960 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-08-31T00:00:50.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:50.987 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:50.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:51.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:51.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-08-31T00:00:51.114 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:51.216 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:51.240 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:51.341 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:51.367 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.368 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:51.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.497 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:51.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:51.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:51.624 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:51.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:51.753 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: pgmap v2280: 67 pgs: 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 43 active+clean; 3.7 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 50/153 objects degraded (32.680%); 53 KiB/s, 0 objects/s recovering 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: 1.1d scrub starts 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: 1.1d scrub ok 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: 9.14 scrub starts 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: 9.14 scrub ok 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: 9.10 scrub starts 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: 9.10 scrub ok 2024-08-31T00:00:51.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:51 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 50/153 objects degraded (32.680%), 3 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: pgmap v2280: 67 pgs: 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 43 active+clean; 3.7 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 50/153 objects degraded (32.680%); 53 KiB/s, 0 objects/s recovering 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: 1.1d scrub starts 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: 1.1d scrub ok 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: 9.14 scrub starts 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: 9.14 scrub ok 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: 9.10 scrub starts 2024-08-31T00:00:51.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: 9.10 scrub ok 2024-08-31T00:00:51.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:51 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 50/153 objects degraded (32.680%), 3 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:51.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5 2024-08-31T00:00:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: pgmap v2280: 67 pgs: 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 43 active+clean; 3.7 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 564 op/s; 50/153 objects degraded (32.680%); 53 KiB/s, 0 objects/s recovering 2024-08-31T00:00:51.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: 1.1d scrub starts 2024-08-31T00:00:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: 1.1d scrub ok 2024-08-31T00:00:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: 9.14 scrub starts 2024-08-31T00:00:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: 9.14 scrub ok 2024-08-31T00:00:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: 9.10 scrub starts 2024-08-31T00:00:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: 9.10 scrub ok 2024-08-31T00:00:51.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:51 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 50/153 objects degraded (32.680%), 3 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:51.883 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:51.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.5' 2024-08-31T00:00:51.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-31T00:00:51.984 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.012 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:52.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:52.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1 2024-08-31T00:00:52.138 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:52.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.1' 2024-08-31T00:00:52.192 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:52.239 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:52.261 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:52.363 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:52.384 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:52.461 INFO:teuthology.orchestra.run.smithi106.stderr:pool 'unique_pool_7' removed 2024-08-31T00:00:52.486 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11 2024-08-31T00:00:52.512 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.11' 2024-08-31T00:00:52.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:52.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:52 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-31T00:00:52.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:52 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-31T00:00:52.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:52 smithi136 ceph-mon[20207]: 9.0 scrub starts 2024-08-31T00:00:52.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:52 smithi136 ceph-mon[20207]: 9.0 scrub ok 2024-08-31T00:00:52.622 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:52 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2753182224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-08-31T00:00:52.640 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:52.732 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:00:52.732 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-31T00:00:52.732 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-31T00:00:52.732 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:00:52.741 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10 2024-08-31T00:00:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.770 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:52.770 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.10' 2024-08-31T00:00:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:52 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-31T00:00:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:52 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-31T00:00:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:52 smithi173 ceph-mon[20554]: 9.0 scrub starts 2024-08-31T00:00:52.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:52 smithi173 ceph-mon[20554]: 9.0 scrub ok 2024-08-31T00:00:52.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:52 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2753182224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-08-31T00:00:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:52 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-31T00:00:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:52 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-31T00:00:52.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:52 smithi106 ceph-mon[16603]: 9.0 scrub starts 2024-08-31T00:00:52.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:52 smithi106 ceph-mon[16603]: 9.0 scrub ok 2024-08-31T00:00:52.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:52 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2753182224' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-08-31T00:00:52.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8 2024-08-31T00:00:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:52.897 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: docker: command not found 2024-08-31T00:00:52.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi173 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.8' 2024-08-31T00:00:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-31T00:00:52.947 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-31T00:00:52.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6 2024-08-31T00:00:53.021 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:53.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.6' 2024-08-31T00:00:53.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:53.144 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:53.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:53.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:53.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4 2024-08-31T00:00:53.268 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: docker: command not found 2024-08-31T00:00:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:53.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi136 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.4' 2024-08-31T00:00:53.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0 2024-08-31T00:00:53.392 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: docker: command not found 2024-08-31T00:00:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:00:53.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi106 with status 1: 'sudo docker kill -s 1 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd.0' 2024-08-31T00:00:53.426 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-08-31T00:00:53.476 INFO:tasks.thrashosds:joining thrashosds 2024-08-31T00:00:53.477 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:53.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-31T00:00:53.561 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: pgmap v2281: 67 pgs: 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 43 active+clean; 3.7 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 50/153 objects degraded (32.680%); 53 KiB/s, 0 objects/s recovering 2024-08-31T00:00:53.561 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: 9.4 scrub starts 2024-08-31T00:00:53.561 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: 9.4 scrub ok 2024-08-31T00:00:53.562 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: 1.19 scrub starts 2024-08-31T00:00:53.562 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: 1.19 scrub ok 2024-08-31T00:00:53.562 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2753182224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-08-31T00:00:53.562 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:53 smithi136 ceph-mon[20207]: osdmap e1032: 12 total, 11 up, 12 in 2024-08-31T00:00:53.688 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: pgmap v2281: 67 pgs: 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 43 active+clean; 3.7 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 50/153 objects degraded (32.680%); 53 KiB/s, 0 objects/s recovering 2024-08-31T00:00:53.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: 9.4 scrub starts 2024-08-31T00:00:53.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: 9.4 scrub ok 2024-08-31T00:00:53.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: 1.19 scrub starts 2024-08-31T00:00:53.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: 1.19 scrub ok 2024-08-31T00:00:53.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2753182224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-08-31T00:00:53.689 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:53 smithi106 ceph-mon[16603]: osdmap e1032: 12 total, 11 up, 12 in 2024-08-31T00:00:53.789 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: pgmap v2281: 67 pgs: 11 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 43 active+clean; 3.7 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 140 op/s; 50/153 objects degraded (32.680%); 53 KiB/s, 0 objects/s recovering 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: 9.4 scrub starts 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: 9.4 scrub ok 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: 1.19 scrub starts 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: 1.19 scrub ok 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2753182224' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-08-31T00:00:53.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:53 smithi173 ceph-mon[20554]: osdmap e1032: 12 total, 11 up, 12 in 2024-08-31T00:00:54.289 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:00:54.289 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-31T00:00:54.289 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-08-31T00:00:54.289 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-08-31T00:00:54.289 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:00:54.471 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is set 2024-08-31T00:00:54.497 INFO:tasks.thrashosds.thrasher:reviving osd 2024-08-31T00:00:54.497 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-08-31T00:00:54.497 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-08-31T00:00:54.497 DEBUG:teuthology.orchestra.run.smithi136:osd.7> sudo journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7.service 2024-08-31T00:00:54.500 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl start ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7 2024-08-31T00:00:54.764 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:54 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/868015939' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:54.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:54 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/868015939' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:54.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:54 smithi136 systemd[1]: Starting Ceph osd.7 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:00:54.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:54 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/868015939' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-31T00:00:54.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-31T00:00:55.279 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 podman[170801]: 2024-08-31 00:00:55.033141896 +0000 UTC m=+0.016394555 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:00:55.279 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 podman[170801]: 2024-08-31 00:00:55.152183754 +0000 UTC m=+0.135436411 container create 6cae4c56e4168bc670ebe3f323e2412d31985a85635a75723f6f134bc48d7636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-08-31T00:00:55.521 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:55.537 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 podman[170801]: 2024-08-31 00:00:55.396872266 +0000 UTC m=+0.380124927 container init 6cae4c56e4168bc670ebe3f323e2412d31985a85635a75723f6f134bc48d7636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-08-31T00:00:55.538 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 podman[170801]: 2024-08-31 00:00:55.400328479 +0000 UTC m=+0.383581137 container start 6cae4c56e4168bc670ebe3f323e2412d31985a85635a75723f6f134bc48d7636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate, GIT_CLEAN=True, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-08-31T00:00:55.538 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 podman[170801]: 2024-08-31 00:00:55.488411851 +0000 UTC m=+0.471664522 container attach 6cae4c56e4168bc670ebe3f323e2412d31985a85635a75723f6f134bc48d7636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-08-31T00:00:55.538 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: pgmap v2283: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: 1.1e scrub starts 2024-08-31T00:00:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: 1.1e scrub ok 2024-08-31T00:00:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:55.539 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/868015939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-mon[20207]: osdmap e1033: 12 total, 11 up, 12 in 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.792 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.793 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-31T00:00:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: pgmap v2283: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: 1.1e scrub starts 2024-08-31T00:00:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: 1.1e scrub ok 2024-08-31T00:00:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:55.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:55.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/868015939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:55.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:55 smithi173 ceph-mon[20554]: osdmap e1033: 12 total, 11 up, 12 in 2024-08-31T00:00:55.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: pgmap v2283: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: 1.1e scrub starts 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: 1.1e scrub ok 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/868015939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-31T00:00:55.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:55 smithi106 ceph-mon[16603]: osdmap e1033: 12 total, 11 up, 12 in 2024-08-31T00:00:56.100 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-31T00:00:56.100 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-31T00:00:56.101 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-08-31T00:00:56.101 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:55 smithi136 bash[170801]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-7 --no-mon-config 2024-08-31T00:00:56.208 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:00:56.215 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-08-31T00:00:56.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(7.0 795/788 [PushOp(7:0859067a:::benchmark_data_smithi173_129296_object41869:head, version: 739'2606, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:0859067a:::benchmark_data_smithi173_129296_object41869:head@739'2606, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:52:02.814175+0000", 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 533.39176748299997, 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.2782153, 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814175+0000", 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814176+0000", 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7199999999999998e-07 2024-08-31T00:00:56.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:02.814204+0000", 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7526e-05 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090173+0000", 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.27596965699999998 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090195+0000", 2024-08-31T00:00:56.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2075000000000001e-05 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090304+0000", 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010874699999999999 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.090321+0000", 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7149000000000001e-05 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:52:03.092391+0000", 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002069474 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21763 8.4 8:2f5d79f6:::benchmark_data_smithi173_143816_object21762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e852)", 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.077936+0000", 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 406.12800691799998, 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30212884200000001, 2024-08-31T00:00:56.220 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21763 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077936+0000", 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077938+0000", 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0480000000000001e-06 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.077983+0000", 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5318999999999998e-05 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078009+0000", 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5554999999999998e-05 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078016+0000", 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4769999999999997e-06 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078078+0000", 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1748999999999994e-05 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078236+0000", 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000157434 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.078373+0000", 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013750000000000001 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079802+0000", 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001428653 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.079818+0000", 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6597999999999999e-05 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379984+0000", 2024-08-31T00:00:56.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30016611799999998 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380032+0000", 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7268000000000002e-05 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380065+0000", 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3123000000000001e-05 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21763 8.4 e852/840 ondisk, result = 0)", 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.083496+0000", 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 406.122446642, 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.296575003, 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083496+0000", 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083497+0000", 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1500000000000004e-07 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.083505+0000", 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1580000000000005e-06 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379828+0000", 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.29632308899999998 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379844+0000", 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5665999999999999e-05 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379970+0000", 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012600999999999999 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.379979+0000", 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9770000000000006e-06 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.380071+0000", 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2387999999999997e-05 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.230 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21778 8.8 8:10e57910:::benchmark_data_smithi173_143816_object21777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.227323+0000", 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 405.97861993999999, 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28839689299999999, 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:56.231 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21778 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227323+0000", 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227333+0000", 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5489999999999995e-06 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227385+0000", 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2265999999999997e-05 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227427+0000", 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2255000000000001e-05 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227452+0000", 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4443999999999998e-05 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227520+0000", 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8916000000000002e-05 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.227528+0000", 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6550000000000004e-06 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228639+0000", 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0011110810000000001 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228736+0000", 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7001000000000002e-05 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3,9", 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.228817+0000", 2024-08-31T00:00:56.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1038000000000006e-05 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.229951+0000", 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001133319 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230356+0000", 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040563399999999999 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515551+0000", 2024-08-31T00:00:56.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28519461400000001 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515669+0000", 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117731 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515720+0000", 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1390000000000001e-05 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21778 8.8 e853/840 ondisk, result = 0)", 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.230373+0000", 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 405.97557034599998, 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28535590100000002, 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230373+0000", 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0399999999999996e-07 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.230376+0000", 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2480000000000001e-06 2024-08-31T00:00:56.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515346+0000", 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.28496944699999999 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515361+0000", 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5573000000000002e-05 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515505+0000", 2024-08-31T00:00:56.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515533+0000", 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7696e-05 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.515728+0000", 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019533300000000001 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:21807 8.4 8:2833fcc7:::benchmark_data_smithi173_143816_object21806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e853)", 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.602847+0000", 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 405.60309597200001, 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34692570299999997, 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-31T00:00:56.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 21807 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602847+0000", 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602848+0000", 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009e-06 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602878+0000", 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0278999999999999e-05 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602891+0000", 2024-08-31T00:00:56.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2371000000000001e-05 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602897+0000", 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9009999999999999e-06 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.602941+0000", 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4617000000000003e-05 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603031+0000", 2024-08-31T00:00:56.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0060000000000002e-05 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603115+0000", 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3387000000000006e-05 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.603890+0000", 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00077577699999999996 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604297+0000", 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040658900000000001 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949697+0000", 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34539998399999999 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949739+0000", 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1758000000000002e-05 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949773+0000", 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3970999999999997e-05 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.27853.0:21807 8.4 e853/840 ondisk, result = 0)", 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:10.604230+0000", 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 405.60171257899998, 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34559199699999998, 2024-08-31T00:00:56.249 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604230+0000", 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604231+0000", 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7599999999999999e-07 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.604232+0000", 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.105e-06 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949585+0000", 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.34535339199999998 2024-08-31T00:00:56.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949593+0000", 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.644e-06 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949684+0000", 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1680000000000003e-05 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949693+0000", 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.354e-06 2024-08-31T00:00:56.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:10.949822+0000", 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129646 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.27853.0:61503 8.4 8:29953d46:::benchmark_data_smithi173_143816_object61502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:54:50.261288+0000", 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 365.94465543799998, 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24835605799999999, 2024-08-31T00:00:56.253 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.27853", 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/3904007387", 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 61503 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261288+0000", 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9299999999999997e-07 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261343+0000", 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5149999999999999e-05 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261352+0000", 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261356+0000", 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1420000000000003e-06 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261393+0000", 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899e-05 2024-08-31T00:00:56.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261516+0000", 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123299 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 9,10", 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.261655+0000", 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000139195 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263257+0000", 2024-08-31T00:00:56.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016018880000000001 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.263968+0000", 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00071132499999999996 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509535+0000", 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24556666499999999 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509591+0000", 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6273000000000001e-05 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:54:50.509644+0000", 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2148999999999998e-05 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "rep_scrubmap(9.6 e948 from shard 10)", 2024-08-31T00:00:56.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:54.088793+0000", 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 182.117150436, 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.544203887, 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088793+0000", 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088793+0000", 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088793+0000", 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.088831+0000", 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7805000000000001e-05 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632788+0000", 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.54395688799999997 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632793+0000", 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.853e-06 2024-08-31T00:00:56.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632843+0000", 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0306000000000003e-05 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632856+0000", 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3033999999999999e-05 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.632996+0000", 2024-08-31T00:00:56.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000140381 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:19270 9.f 9:f74eb61c:::benchmark_data_smithi173_160189_object19269:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e950)", 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:54.328258+0000", 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 181.87768489499999, 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.312871391, 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-31T00:00:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19270 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328258+0000", 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328258+0000", 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328260+0000", 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7859999999999999e-06 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328294+0000", 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4258999999999998e-05 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328314+0000", 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9814e-05 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328324+0000", 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5959999999999993e-06 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328407+0000", 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328512+0000", 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104817 2024-08-31T00:00:56.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,9", 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.328611+0000", 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8474999999999995e-05 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.329444+0000", 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00083315400000000001 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.330021+0000", 2024-08-31T00:00:56.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00057678899999999997 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.641045+0000", 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31102477299999998 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.641086+0000", 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0747000000000001e-05 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:54.641129+0000", 2024-08-31T00:00:56.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3408999999999997e-05 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "replica_scrub(pg: 9.9,from:0'0,to:949'1127,epoch:951/948,start:9:97883b83:::benchmark_data_smithi173_160189_object15908:0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:57:55.808503+0000", 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 180.397440292, 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31569409599999998, 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808503+0000", 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808503+0000", 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808505+0000", 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.424e-06 2024-08-31T00:00:56.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:55.808521+0000", 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.562e-05 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.122578+0000", 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31405760300000002 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.122590+0000", 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1260000000000001e-05 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.124149+0000", 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00155949 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.124163+0000", 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3582e-05 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:57:56.124197+0000", 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4116999999999999e-05 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(9.21 971/969 [PushOp(9:85fa522a:::benchmark_data_smithi173_160189_object19232:head, version: 949'1108, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:85fa522a:::benchmark_data_smithi173_160189_object19232:head@949'1108, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:33.089878+0000", 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 143.11606512399999, 2024-08-31T00:00:56.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25101715099999999, 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089878+0000", 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089878+0000", 2024-08-31T00:00:56.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089878+0000", 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.75e-07 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.089926+0000", 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7298e-05 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340259+0000", 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25033364699999999 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340268+0000", 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1679999999999993e-06 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340345+0000", 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7075000000000003e-05 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340359+0000", 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3258000000000001e-05 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340895+0000", 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00053622999999999995 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:62298 9.19 e971/957 9:9be56da5:::benchmark_data_smithi173_160189_object62297:head v 971'2500, mlcod=971'2497)", 2024-08-31T00:00:56.278 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:33.094019+0000", 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 143.11192348500001, 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.24765731099999999, 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094019+0000", 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094019+0000", 2024-08-31T00:00:56.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094020+0000", 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3799999999999997e-07 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.094123+0000", 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102896 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340587+0000", 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.246463975 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340591+0000", 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6320000000000001e-06 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340632+0000", 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1198000000000001e-05 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.340652+0000", 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0171999999999999e-05 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.341651+0000", 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00099860500000000007 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:33.341677+0000", 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6194999999999999e-05 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.282 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:74093 9.21 9:865d22da:::benchmark_data_smithi173_160189_object74092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e976)", 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:44.666003+0000", 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.53993964099999, 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25998222799999998, 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 74093 2024-08-31T00:00:56.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666003+0000", 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666003+0000", 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666004+0000", 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666058+0000", 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3695e-05 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666069+0000", 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.079e-05 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666081+0000", 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.222e-05 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666131+0000", 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0492999999999999e-05 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666239+0000", 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108045 2024-08-31T00:00:56.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1,2", 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.666365+0000", 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012594900000000001 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667262+0000", 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00089650099999999996 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.669369+0000", 2024-08-31T00:00:56.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021067350000000002 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925924+0000", 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25655586000000002 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925959+0000", 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4552999999999999e-05 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925986+0000", 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6519000000000001e-05 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.28405.0:74093 9.21 e976/974 ondisk, result = 0)", 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:58:44.667959+0000", 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.53798372599999, 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25803844799999998, 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-08-31T00:00:56.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667959+0000", 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667959+0000", 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667960+0000", 2024-08-31T00:00:56.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0200000000000002e-07 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.667964+0000", 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0720000000000001e-06 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925769+0000", 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.25780540000000002 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925781+0000", 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1717e-05 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925896+0000", 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114911 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925921+0000", 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5085000000000001e-05 2024-08-31T00:00:56.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:58:44.925998+0000", 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6661000000000003e-05 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.28405.0:91166 9.1d 9:b9e063a0:::benchmark_data_smithi173_160189_object91165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e984)", 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:00.451819+0000", 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 115.754123752, 2024-08-31T00:00:56.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.63616116899999997, 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.28405", 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.173:0/234065", 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 91166 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451819+0000", 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451819+0000", 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451820+0000", 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451875+0000", 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4614000000000003e-05 2024-08-31T00:00:56.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451886+0000", 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1785e-05 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.451895+0000", 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6950000000000006e-06 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.453708+0000", 2024-08-31T00:00:56.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018123639999999999 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.453830+0000", 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012297299999999999 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0,9", 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.454044+0000", 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021353 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.455288+0000", 2024-08-31T00:00:56.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012440859999999999 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:00.470029+0000", 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014740985999999999 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.087909+0000", 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.61788033799999997 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.087947+0000", 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7463999999999999e-05 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:01.087980+0000", 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3458999999999999e-05 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(9.1b 994/969 [PushReplyOp(9:db66c045:::benchmark_data_smithi173_160189_object46533:head)])", 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:13.482460+0000", 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.72348338400001, 2024-08-31T00:00:56.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35726268799999999, 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482460+0000", 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482460+0000", 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482460+0000", 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4499999999999996e-07 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.482466+0000", 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3029999999999999e-06 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839039+0000", 2024-08-31T00:00:56.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.35657290000000003 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839049+0000", 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0334e-05 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839141+0000", 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2153999999999999e-05 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:13.839722+0000", 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00058115200000000004 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPushReply(9.1b 995/969 [PushReplyOp(9:da33c270:::benchmark_data_smithi173_160189_object52147:head)])", 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:14.534516+0000", 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.67142713, 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.45652447000000002, 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-08-31T00:00:56.303 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534516+0000", 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534516+0000", 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534516+0000", 2024-08-31T00:00:56.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6199999999999998e-07 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.534520+0000", 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6239999999999999e-06 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.990657+0000", 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.456136879 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.990670+0000", 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3463e-05 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.990766+0000", 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5520999999999996e-05 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:14.991040+0000", 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027442100000000001 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:103252 9.13 e998/997 9:cf2b091d:::benchmark_data_smithi173_160189_object1248:head v 998'3726, mlcod=998'3723)", 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:17.297722+0000", 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 98.908220665000002, 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37625371200000002, 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.307 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297722+0000", 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297722+0000", 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297723+0000", 2024-08-31T00:00:56.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9899999999999999e-07 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.297729+0000", 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672211+0000", 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.37448169399999998 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672220+0000", 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3950000000000007e-06 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672285+0000", 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4584000000000004e-05 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.672310+0000", 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5624999999999999e-05 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.673937+0000", 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016269050000000001 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.673976+0000", 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8821e-05 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop(client.28405.0:103268 9.19 e998/957 9:9c47d45a:::benchmark_data_smithi173_160189_object1264:head v 998'3776, mlcod=998'3774)", 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-08-30T23:59:17.320130+0000", 2024-08-31T00:00:56.311 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 98.885812740999995, 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.73841249399999997, 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320130+0000", 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320130+0000", 2024-08-31T00:00:56.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320132+0000", 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5579999999999999e-06 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:17.320140+0000", 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9019999999999999e-06 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-08-31T00:00:56.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.056952+0000", 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.73681268300000002 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.056958+0000", 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9229999999999999e-06 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.057012+0000", 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3801000000000002e-05 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-08-31T00:00:56.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.057029+0000", 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6574000000000001e-05 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.058491+0000", 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0014624760000000001 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-08-30T23:59:18.058543+0000", 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1576999999999998e-05 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-08-31T00:00:56.316 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-08-31T00:00:56.316 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 bash[170801]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-7/block 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 bash[170801]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 bash[170801]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 bash[170801]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-31T00:00:56.316 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate[170827]: --> ceph-volume lvm activate successful for osd ID: 7 2024-08-31T00:00:56.317 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 bash[170801]: --> ceph-volume lvm activate successful for osd ID: 7 2024-08-31T00:00:56.317 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:56 smithi136 podman[170801]: 2024-08-31 00:00:56.20766533 +0000 UTC m=+1.190917988 container died 6cae4c56e4168bc670ebe3f323e2412d31985a85635a75723f6f134bc48d7636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-31T00:00:56.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:56 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:56.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:56 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:56.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:56 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-31T00:00:57.201 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-08-31T00:00:57.453 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:57.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:57.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:57.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:57 smithi106 ceph-mon[16603]: pgmap v2285: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:57.574 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:57 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:00:57.575 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:57 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:00:57.575 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:57 smithi106 ceph-mon[16603]: osdmap e1034: 12 total, 11 up, 12 in 2024-08-31T00:00:57.751 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:57.752 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:57.752 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:57 smithi136 ceph-mon[20207]: pgmap v2285: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:57.752 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:57 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:00:57.752 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:57 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:00:57.752 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:57 smithi136 ceph-mon[20207]: osdmap e1034: 12 total, 11 up, 12 in 2024-08-31T00:00:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:57 smithi173 ceph-mon[20554]: pgmap v2285: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:57 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:00:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:57 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:00:57.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:57 smithi173 ceph-mon[20554]: osdmap e1034: 12 total, 11 up, 12 in 2024-08-31T00:00:58.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:57 smithi136 podman[170986]: 2024-08-31 00:00:57.751767441 +0000 UTC m=+1.587240200 container remove 6cae4c56e4168bc670ebe3f323e2412d31985a85635a75723f6f134bc48d7636 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-08-31T00:00:58.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:57 smithi136 podman[171063]: 2024-08-31 00:00:57.859524624 +0000 UTC m=+0.015431302 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:00:58.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:57 smithi136 podman[171063]: 2024-08-31 00:00:57.986291965 +0000 UTC m=+0.142198685 container create cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-08-31T00:00:58.355 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:58 smithi136 podman[171063]: 2024-08-31 00:00:58.222157847 +0000 UTC m=+0.378064528 container init cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-31T00:00:58.355 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:58 smithi136 podman[171063]: 2024-08-31 00:00:58.225656009 +0000 UTC m=+0.381562687 container start cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-31T00:00:58.355 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:58 smithi136 ceph-osd[171082]: -- 172.21.15.136:0/2585592873 <== mon.2 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cb447ba40 con 0x564cb5248000 2024-08-31T00:00:58.355 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:58 smithi136 bash[171063]: cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed 2024-08-31T00:00:58.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-31T00:00:58.476 INFO:teuthology.orchestra.run.smithi106.stderr:noscrub is unset 2024-08-31T00:00:58.609 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:00:58.729 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/455805633' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:58.729 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:58 smithi136 systemd[1]: Started Ceph osd.7 for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-31T00:00:58.771 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/455805633' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:58.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/455805633' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:no valid command found; 10 closest matches: 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:0 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:1 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:2 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:abort 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:assert 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:config diff 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:config diff get 2024-08-31T00:00:59.134 INFO:teuthology.orchestra.run.smithi136.stderr:config get 2024-08-31T00:00:59.135 INFO:teuthology.orchestra.run.smithi136.stderr:config help [] 2024-08-31T00:00:59.135 INFO:teuthology.orchestra.run.smithi136.stderr:config set ... 2024-08-31T00:00:59.135 INFO:teuthology.orchestra.run.smithi136.stderr:admin_socket: invalid command 2024-08-31T00:00:59.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-31T00:00:59.460 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:00:59.728 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:59 smithi106 ceph-mon[16603]: pgmap v2287: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:59.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:59 smithi106 ceph-mon[16603]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-08-31T00:00:59.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/455805633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:59.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:00:59 smithi106 ceph-mon[16603]: osdmap e1035: 12 total, 11 up, 12 in 2024-08-31T00:00:59.775 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:59 smithi136 ceph-mon[20207]: pgmap v2287: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:59.775 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:59 smithi136 ceph-mon[20207]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-08-31T00:00:59.776 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/455805633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:59.776 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:00:59 smithi136 ceph-mon[20207]: osdmap e1035: 12 total, 11 up, 12 in 2024-08-31T00:00:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:59 smithi173 ceph-mon[20554]: pgmap v2287: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:00:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:59 smithi173 ceph-mon[20554]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-08-31T00:00:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/455805633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-31T00:00:59.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:00:59 smithi173 ceph-mon[20554]: osdmap e1035: 12 total, 11 up, 12 in 2024-08-31T00:00:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-31T00:00:59.943 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-08-31T00:01:00.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:00:59 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:00:59.774+0000 7f54cc3bb640 -1 Falling back to public interface 2024-08-31T00:01:00.478 INFO:teuthology.orchestra.run.smithi106.stderr:nodeep-scrub is unset 2024-08-31T00:01:00.770 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:00 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-31T00:01:00.771 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:00 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-31T00:01:00.771 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:00 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4162092994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:01:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:00 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-31T00:01:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:00 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-31T00:01:00.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:00 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4162092994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:01:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:00 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-31T00:01:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:00 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-31T00:01:00.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:00 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4162092994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-31T00:01:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:01 smithi136 ceph-mon[20207]: pgmap v2289: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4162092994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:01:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:01 smithi136 ceph-mon[20207]: osdmap e1036: 12 total, 11 up, 12 in 2024-08-31T00:01:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:01 smithi136 ceph-mon[20207]: osdmap e1037: 12 total, 11 up, 12 in 2024-08-31T00:01:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:01 smithi173 ceph-mon[20554]: pgmap v2289: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4162092994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:01:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:01 smithi173 ceph-mon[20554]: osdmap e1036: 12 total, 11 up, 12 in 2024-08-31T00:01:01.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:01 smithi173 ceph-mon[20554]: osdmap e1037: 12 total, 11 up, 12 in 2024-08-31T00:01:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:01 smithi106 ceph-mon[16603]: pgmap v2289: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4162092994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-31T00:01:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:01 smithi106 ceph-mon[16603]: osdmap e1036: 12 total, 11 up, 12 in 2024-08-31T00:01:01.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:01 smithi106 ceph-mon[16603]: osdmap e1037: 12 total, 11 up, 12 in 2024-08-31T00:01:03.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:01:03.297+0000 7f54cc3bb640 -1 osd.7 902 log_to_monitors true 2024-08-31T00:01:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-mon[20207]: pgmap v2292: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-31T00:01:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-31T00:01:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-mon[20207]: from='osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-31T00:01:03.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-31T00:01:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:03 smithi173 ceph-mon[20554]: pgmap v2292: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:03 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-31T00:01:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:03 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-31T00:01:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:03 smithi173 ceph-mon[20554]: from='osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-31T00:01:03.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:03 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-31T00:01:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:03 smithi106 ceph-mon[16603]: pgmap v2292: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:03.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:03 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-31T00:01:03.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:03 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-31T00:01:03.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:03 smithi106 ceph-mon[16603]: from='osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-31T00:01:03.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:03 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-31T00:01:04.122 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:03 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:01:03.823+0000 7f54c015d640 -1 osd.7 1038 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-31T00:01:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:04 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-31T00:01:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:04 smithi136 ceph-mon[20207]: from='osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-31T00:01:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:04 smithi136 ceph-mon[20207]: osdmap e1038: 12 total, 11 up, 12 in 2024-08-31T00:01:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:04 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-31T00:01:04.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:04 smithi136 ceph-mon[20207]: from='osd.7 ' entity='osd.7' 2024-08-31T00:01:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:04 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-31T00:01:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:04 smithi173 ceph-mon[20554]: from='osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-31T00:01:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:04 smithi173 ceph-mon[20554]: osdmap e1038: 12 total, 11 up, 12 in 2024-08-31T00:01:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:04 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-31T00:01:04.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:04 smithi173 ceph-mon[20554]: from='osd.7 ' entity='osd.7' 2024-08-31T00:01:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:04 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-31T00:01:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:04 smithi106 ceph-mon[16603]: from='osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-31T00:01:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:04 smithi106 ceph-mon[16603]: osdmap e1038: 12 total, 11 up, 12 in 2024-08-31T00:01:04.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:04 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi136", "root=default"]} : dispatch 2024-08-31T00:01:04.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:04 smithi106 ceph-mon[16603]: from='osd.7 ' entity='osd.7' 2024-08-31T00:01:04.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-31T00:01:05.206 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.b/config 2024-08-31T00:01:05.609 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:05 smithi136 ceph-mon[20207]: pgmap v2294: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:05.609 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:05 smithi136 ceph-mon[20207]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-31T00:01:05.609 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:05 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-31T00:01:05.609 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:05 smithi136 ceph-mon[20207]: osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111] boot 2024-08-31T00:01:05.609 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:05 smithi136 ceph-mon[20207]: osdmap e1039: 12 total, 12 up, 12 in 2024-08-31T00:01:05.710 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-08-31T00:01:05.711 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-08-31T00:01:05.711 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-08-31T00:01:05.711 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-08-31T00:01:05.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:05 smithi173 ceph-mon[20554]: pgmap v2294: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:05.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:05 smithi173 ceph-mon[20554]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-31T00:01:05.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:05 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-31T00:01:05.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:05 smithi173 ceph-mon[20554]: osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111] boot 2024-08-31T00:01:05.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:05 smithi173 ceph-mon[20554]: osdmap e1039: 12 total, 12 up, 12 in 2024-08-31T00:01:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:05 smithi106 ceph-mon[16603]: pgmap v2294: 31 pgs: 3 active+clean+remapped, 8 active+undersized, 1 active+undersized+degraded, 19 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1/6 objects degraded (16.667%) 2024-08-31T00:01:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:05 smithi106 ceph-mon[16603]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-31T00:01:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:05 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-31T00:01:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:05 smithi106 ceph-mon[16603]: osd.7 [v2:172.21.15.136:6824/907316111,v1:172.21.15.136:6825/907316111] boot 2024-08-31T00:01:05.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:05 smithi106 ceph-mon[16603]: osdmap e1039: 12 total, 12 up, 12 in 2024-08-31T00:01:06.408 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-08-31T00:01:06.675 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: osdmap e1040: 12 total, 12 up, 12 in 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: osdmap e1041: 12 total, 12 up, 12 in 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: 1.1a scrub starts 2024-08-31T00:01:07.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:06 smithi173 ceph-mon[20554]: 1.1a scrub ok 2024-08-31T00:01:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:07.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:07.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: osdmap e1040: 12 total, 12 up, 12 in 2024-08-31T00:01:07.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: osdmap e1041: 12 total, 12 up, 12 in 2024-08-31T00:01:07.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-31T00:01:07.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: 1.1a scrub starts 2024-08-31T00:01:07.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:06 smithi106 ceph-mon[16603]: 1.1a scrub ok 2024-08-31T00:01:07.132 INFO:teuthology.orchestra.run.smithi106.stdout:mon.a: {} 2024-08-31T00:01:07.133 INFO:teuthology.orchestra.run.smithi106.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-31T00:01:07.147 INFO:teuthology.orchestra.run.smithi106.stdout:mon.c: {} 2024-08-31T00:01:07.147 INFO:teuthology.orchestra.run.smithi106.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-31T00:01:07.161 INFO:teuthology.orchestra.run.smithi106.stdout:mon.b: {} 2024-08-31T00:01:07.162 INFO:teuthology.orchestra.run.smithi106.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-31T00:01:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:07.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: osdmap e1040: 12 total, 12 up, 12 in 2024-08-31T00:01:07.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: osdmap e1041: 12 total, 12 up, 12 in 2024-08-31T00:01:07.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-31T00:01:07.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: 1.1a scrub starts 2024-08-31T00:01:07.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:06 smithi136 ceph-mon[20207]: 1.1a scrub ok 2024-08-31T00:01:07.873 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-08-31T00:01:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: pgmap v2297: 31 pgs: 2 peering, 3 active+clean+remapped, 7 active+undersized, 19 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:08.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-31T00:01:08.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: Health check update: Degraded data redundancy: 7 pgs undersized (PG_DEGRADED) 2024-08-31T00:01:08.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: 1.10 scrub starts 2024-08-31T00:01:08.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: 1.10 scrub ok 2024-08-31T00:01:08.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1546477067' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-31T00:01:08.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: 1.15 scrub starts 2024-08-31T00:01:08.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:07 smithi173 ceph-mon[20554]: 1.15 scrub ok 2024-08-31T00:01:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: pgmap v2297: 31 pgs: 2 peering, 3 active+clean+remapped, 7 active+undersized, 19 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-31T00:01:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: Health check update: Degraded data redundancy: 7 pgs undersized (PG_DEGRADED) 2024-08-31T00:01:08.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: 1.10 scrub starts 2024-08-31T00:01:08.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: 1.10 scrub ok 2024-08-31T00:01:08.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1546477067' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-31T00:01:08.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: 1.15 scrub starts 2024-08-31T00:01:08.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:07 smithi106 ceph-mon[16603]: 1.15 scrub ok 2024-08-31T00:01:08.117 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: pgmap v2297: 31 pgs: 2 peering, 3 active+clean+remapped, 7 active+undersized, 19 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: Health check update: Degraded data redundancy: 7 pgs undersized (PG_DEGRADED) 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: 1.10 scrub starts 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: 1.10 scrub ok 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1546477067' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: 1.15 scrub starts 2024-08-31T00:01:08.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:07 smithi136 ceph-mon[20207]: 1.15 scrub ok 2024-08-31T00:01:08.798 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.0 to 1 (10000) 2024-08-31T00:01:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:08 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-31T00:01:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:08 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-31T00:01:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:08 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-31T00:01:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:08 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-31T00:01:09.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:08 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/388228355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-31T00:01:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:08 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-31T00:01:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:08 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-31T00:01:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:08 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-31T00:01:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:08 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-31T00:01:09.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:08 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/388228355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-31T00:01:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:08 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-31T00:01:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:08 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-31T00:01:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:08 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-31T00:01:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:08 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-31T00:01:09.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:08 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/388228355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-31T00:01:09.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 0 1 2024-08-31T00:01:09.699 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: pgmap v2299: 31 pgs: 2 peering, 3 active+undersized, 26 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/388228355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: osdmap e1042: 12 total, 12 up, 12 in 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: 1.18 scrub starts 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: 1.18 scrub ok 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: 1.e scrub starts 2024-08-31T00:01:10.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:09 smithi173 ceph-mon[20554]: 1.e scrub ok 2024-08-31T00:01:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: pgmap v2299: 31 pgs: 2 peering, 3 active+undersized, 26 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/388228355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-31T00:01:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: osdmap e1042: 12 total, 12 up, 12 in 2024-08-31T00:01:10.082 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-31T00:01:10.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: 1.18 scrub starts 2024-08-31T00:01:10.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: 1.18 scrub ok 2024-08-31T00:01:10.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: 1.e scrub starts 2024-08-31T00:01:10.083 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:09 smithi106 ceph-mon[16603]: 1.e scrub ok 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: pgmap v2299: 31 pgs: 2 peering, 3 active+undersized, 26 active+clean; 616 KiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/388228355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: osdmap e1042: 12 total, 12 up, 12 in 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: 1.18 scrub starts 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: 1.18 scrub ok 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: 1.e scrub starts 2024-08-31T00:01:10.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:09 smithi136 ceph-mon[20207]: 1.e scrub ok 2024-08-31T00:01:10.814 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-08-31T00:01:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:10 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-31T00:01:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:10 smithi173 ceph-mon[20554]: 1.16 scrub starts 2024-08-31T00:01:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:10 smithi173 ceph-mon[20554]: 1.16 scrub ok 2024-08-31T00:01:11.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:10 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/542052428' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-31T00:01:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:10 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-31T00:01:11.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:10 smithi106 ceph-mon[16603]: 1.16 scrub starts 2024-08-31T00:01:11.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:10 smithi106 ceph-mon[16603]: 1.16 scrub ok 2024-08-31T00:01:11.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:10 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/542052428' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-31T00:01:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:10 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-31T00:01:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:10 smithi136 ceph-mon[20207]: 1.16 scrub starts 2024-08-31T00:01:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:10 smithi136 ceph-mon[20207]: 1.16 scrub ok 2024-08-31T00:01:11.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:10 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/542052428' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-31T00:01:11.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-08-31T00:01:11.807 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:12.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:11 smithi106 ceph-mon[16603]: pgmap v2301: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:12.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:11 smithi106 ceph-mon[16603]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3 pgs undersized) 2024-08-31T00:01:12.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:11 smithi106 ceph-mon[16603]: Cluster is now healthy 2024-08-31T00:01:12.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:11 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/542052428' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-31T00:01:12.063 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:11 smithi106 ceph-mon[16603]: osdmap e1043: 12 total, 12 up, 12 in 2024-08-31T00:01:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:11 smithi173 ceph-mon[20554]: pgmap v2301: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:11 smithi173 ceph-mon[20554]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3 pgs undersized) 2024-08-31T00:01:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:11 smithi173 ceph-mon[20554]: Cluster is now healthy 2024-08-31T00:01:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:11 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/542052428' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-31T00:01:12.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:11 smithi173 ceph-mon[20554]: osdmap e1043: 12 total, 12 up, 12 in 2024-08-31T00:01:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:11 smithi136 ceph-mon[20207]: pgmap v2301: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:11 smithi136 ceph-mon[20207]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3 pgs undersized) 2024-08-31T00:01:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:11 smithi136 ceph-mon[20207]: Cluster is now healthy 2024-08-31T00:01:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:11 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/542052428' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-31T00:01:12.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:11 smithi136 ceph-mon[20207]: osdmap e1043: 12 total, 12 up, 12 in 2024-08-31T00:01:12.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-31T00:01:12.498+0000 7f4820c0b640 0 --2- 172.21.15.106:0/3130722604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f481c107d80 0x7f481c10e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-31T00:01:12.820 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.2 to 1 (10000) 2024-08-31T00:01:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:12 smithi173 ceph-mon[20554]: osdmap e1044: 12 total, 12 up, 12 in 2024-08-31T00:01:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:12 smithi173 ceph-mon[20554]: 1.8 scrub starts 2024-08-31T00:01:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:12 smithi173 ceph-mon[20554]: 1.8 scrub ok 2024-08-31T00:01:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:12 smithi173 ceph-mon[20554]: 1.2 scrub starts 2024-08-31T00:01:13.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:12 smithi173 ceph-mon[20554]: 1.2 scrub ok 2024-08-31T00:01:13.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:12 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4029263732' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-31T00:01:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:12 smithi106 ceph-mon[16603]: osdmap e1044: 12 total, 12 up, 12 in 2024-08-31T00:01:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:12 smithi106 ceph-mon[16603]: 1.8 scrub starts 2024-08-31T00:01:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:12 smithi106 ceph-mon[16603]: 1.8 scrub ok 2024-08-31T00:01:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:12 smithi106 ceph-mon[16603]: 1.2 scrub starts 2024-08-31T00:01:13.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:12 smithi106 ceph-mon[16603]: 1.2 scrub ok 2024-08-31T00:01:13.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:12 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4029263732' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-31T00:01:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:12 smithi136 ceph-mon[20207]: osdmap e1044: 12 total, 12 up, 12 in 2024-08-31T00:01:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:12 smithi136 ceph-mon[20207]: 1.8 scrub starts 2024-08-31T00:01:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:12 smithi136 ceph-mon[20207]: 1.8 scrub ok 2024-08-31T00:01:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:12 smithi136 ceph-mon[20207]: 1.2 scrub starts 2024-08-31T00:01:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:12 smithi136 ceph-mon[20207]: 1.2 scrub ok 2024-08-31T00:01:13.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:12 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4029263732' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-31T00:01:13.758 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 2 1 2024-08-31T00:01:14.052 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: pgmap v2304: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4029263732' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-31T00:01:14.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: osdmap e1045: 12 total, 12 up, 12 in 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: 1.12 scrub starts 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: 1.12 scrub ok 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:13 smithi106 ceph-mon[16603]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: pgmap v2304: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4029263732' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-31T00:01:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: osdmap e1045: 12 total, 12 up, 12 in 2024-08-31T00:01:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: 1.12 scrub starts 2024-08-31T00:01:14.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: 1.12 scrub ok 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:13 smithi136 ceph-mon[20207]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: pgmap v2304: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4029263732' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: osdmap e1045: 12 total, 12 up, 12 in 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: 1.12 scrub starts 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: 1.12 scrub ok 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:14.326 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:13 smithi173 ceph-mon[20554]: from='mgr.14151 ' entity='mgr.y' 2024-08-31T00:01:15.107 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:15.107 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:15.108 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280845]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:15.108 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280845]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:15.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:01:15.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:15.108 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:15.109 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:14 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2947078299' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:15.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:01:15.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:01:15.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:15.291 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:14 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2947078299' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:15.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:15.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:01:15.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:01:15.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:01:15.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:15.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:14 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2947078299' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-31T00:01:15.325 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-08-31T00:01:15.607 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:15.607 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:15.608 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280848]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:15.608 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:01:15 smithi106 sudo[280848]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:16.034 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-08-31T00:01:16.283 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:16.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:16 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2947078299' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-31T00:01:16.584 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:16 smithi106 ceph-mon[16603]: osdmap e1046: 12 total, 12 up, 12 in 2024-08-31T00:01:16.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:16 smithi106 ceph-mon[16603]: osdmap e1047: 12 total, 12 up, 12 in 2024-08-31T00:01:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:16 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2947078299' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-31T00:01:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:16 smithi136 ceph-mon[20207]: osdmap e1046: 12 total, 12 up, 12 in 2024-08-31T00:01:16.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:16 smithi136 ceph-mon[20207]: osdmap e1047: 12 total, 12 up, 12 in 2024-08-31T00:01:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:16 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2947078299' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-31T00:01:16.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:16 smithi173 ceph-mon[20554]: osdmap e1046: 12 total, 12 up, 12 in 2024-08-31T00:01:16.825 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:16 smithi173 ceph-mon[20554]: osdmap e1047: 12 total, 12 up, 12 in 2024-08-31T00:01:17.331 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.3 to 1 (10000) 2024-08-31T00:01:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:17 smithi106 ceph-mon[16603]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:17 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-31T00:01:17.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:17 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-31T00:01:17.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/852576667' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-31T00:01:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:17 smithi136 ceph-mon[20207]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:17 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-31T00:01:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:17 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-31T00:01:17.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/852576667' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-31T00:01:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:17 smithi173 ceph-mon[20554]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:17 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-31T00:01:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:17 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-31T00:01:17.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/852576667' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-31T00:01:18.040 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 3 1 2024-08-31T00:01:18.283 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:18.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/852576667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-31T00:01:18.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:18 smithi106 ceph-mon[16603]: osdmap e1048: 12 total, 12 up, 12 in 2024-08-31T00:01:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/852576667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-31T00:01:18.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:18 smithi136 ceph-mon[20207]: osdmap e1048: 12 total, 12 up, 12 in 2024-08-31T00:01:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/852576667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-31T00:01:18.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:18 smithi173 ceph-mon[20554]: osdmap e1048: 12 total, 12 up, 12 in 2024-08-31T00:01:19.344 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-08-31T00:01:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:19 smithi106 ceph-mon[16603]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:19 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:19.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:19 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:19.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:19 smithi106 ceph-mon[16603]: osdmap e1049: 12 total, 12 up, 12 in 2024-08-31T00:01:19.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2252842662' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-31T00:01:19.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:19 smithi136 ceph-mon[20207]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:19.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:19 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:19.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:19 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:19.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:19 smithi136 ceph-mon[20207]: osdmap e1049: 12 total, 12 up, 12 in 2024-08-31T00:01:19.792 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2252842662' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-31T00:01:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:19 smithi173 ceph-mon[20554]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:19 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:19 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:19 smithi173 ceph-mon[20554]: osdmap e1049: 12 total, 12 up, 12 in 2024-08-31T00:01:19.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2252842662' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-31T00:01:20.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-08-31T00:01:20.282 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:20.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:20 smithi106 ceph-mon[16603]: 1.13 scrub starts 2024-08-31T00:01:20.705 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:20 smithi106 ceph-mon[16603]: 1.13 scrub ok 2024-08-31T00:01:20.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:20 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2252842662' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-31T00:01:20.706 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:20 smithi106 ceph-mon[16603]: osdmap e1050: 12 total, 12 up, 12 in 2024-08-31T00:01:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:20 smithi136 ceph-mon[20207]: 1.13 scrub starts 2024-08-31T00:01:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:20 smithi136 ceph-mon[20207]: 1.13 scrub ok 2024-08-31T00:01:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:20 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2252842662' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-31T00:01:20.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:20 smithi136 ceph-mon[20207]: osdmap e1050: 12 total, 12 up, 12 in 2024-08-31T00:01:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:20 smithi173 ceph-mon[20554]: 1.13 scrub starts 2024-08-31T00:01:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:20 smithi173 ceph-mon[20554]: 1.13 scrub ok 2024-08-31T00:01:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:20 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2252842662' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-31T00:01:20.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:20 smithi173 ceph-mon[20554]: osdmap e1050: 12 total, 12 up, 12 in 2024-08-31T00:01:21.119 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.4 to 1 (10000) 2024-08-31T00:01:21.464 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:21 smithi106 ceph-mon[16603]: pgmap v2314: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:21.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/242594435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-31T00:01:21.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/242594435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-31T00:01:21.465 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:21 smithi106 ceph-mon[16603]: osdmap e1051: 12 total, 12 up, 12 in 2024-08-31T00:01:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:21 smithi136 ceph-mon[20207]: pgmap v2314: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/242594435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-31T00:01:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/242594435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-31T00:01:21.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:21 smithi136 ceph-mon[20207]: osdmap e1051: 12 total, 12 up, 12 in 2024-08-31T00:01:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:21 smithi173 ceph-mon[20554]: pgmap v2314: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/242594435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-31T00:01:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/242594435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-31T00:01:21.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:21 smithi173 ceph-mon[20554]: osdmap e1051: 12 total, 12 up, 12 in 2024-08-31T00:01:21.840 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-31T00:01:22.082 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:22.489 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:22 smithi106 ceph-mon[16603]: 1.17 scrub starts 2024-08-31T00:01:22.489 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:22 smithi106 ceph-mon[16603]: 1.17 scrub ok 2024-08-31T00:01:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:22 smithi136 ceph-mon[20207]: 1.17 scrub starts 2024-08-31T00:01:22.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:22 smithi136 ceph-mon[20207]: 1.17 scrub ok 2024-08-31T00:01:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:22 smithi173 ceph-mon[20554]: 1.17 scrub starts 2024-08-31T00:01:22.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:22 smithi173 ceph-mon[20554]: 1.17 scrub ok 2024-08-31T00:01:23.366 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-31T00:01:23.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:23 smithi106 ceph-mon[16603]: pgmap v2316: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:23.700 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:23 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/921542949' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-31T00:01:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:23 smithi136 ceph-mon[20207]: pgmap v2316: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:23 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/921542949' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-31T00:01:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:23 smithi173 ceph-mon[20554]: pgmap v2316: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:23.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:23 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/921542949' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-31T00:01:24.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-08-31T00:01:24.359 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:24.633 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:24 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/921542949' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-31T00:01:24.634 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:24 smithi106 ceph-mon[16603]: osdmap e1052: 12 total, 12 up, 12 in 2024-08-31T00:01:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:24 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/921542949' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-31T00:01:24.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:24 smithi136 ceph-mon[20207]: osdmap e1052: 12 total, 12 up, 12 in 2024-08-31T00:01:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:24 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/921542949' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-31T00:01:24.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:24 smithi173 ceph-mon[20554]: osdmap e1052: 12 total, 12 up, 12 in 2024-08-31T00:01:25.377 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.6 to 1 (10000) 2024-08-31T00:01:25.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:25 smithi106 ceph-mon[16603]: pgmap v2318: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:25.669 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:25 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1240173625' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-31T00:01:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:25 smithi136 ceph-mon[20207]: pgmap v2318: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:25 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1240173625' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-31T00:01:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:25 smithi173 ceph-mon[20554]: pgmap v2318: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:25.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:25 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1240173625' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-31T00:01:25.957 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 6 1 2024-08-31T00:01:26.205 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:26.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1240173625' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-31T00:01:26.589 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:26 smithi106 ceph-mon[16603]: osdmap e1053: 12 total, 12 up, 12 in 2024-08-31T00:01:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1240173625' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-31T00:01:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:26 smithi136 ceph-mon[20207]: osdmap e1053: 12 total, 12 up, 12 in 2024-08-31T00:01:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1240173625' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-31T00:01:26.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:26 smithi173 ceph-mon[20554]: osdmap e1053: 12 total, 12 up, 12 in 2024-08-31T00:01:27.393 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-08-31T00:01:27.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:27 smithi106 ceph-mon[16603]: pgmap v2320: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:27.709 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:27 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2036407193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-31T00:01:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:27 smithi136 ceph-mon[20207]: pgmap v2320: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:27 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2036407193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-31T00:01:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:27 smithi173 ceph-mon[20554]: pgmap v2320: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:27.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:27 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2036407193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-31T00:01:28.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-08-31T00:01:28.406 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:28.654 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:28 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:28.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:28 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:28.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:28 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2036407193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-31T00:01:28.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:28 smithi106 ceph-mon[16603]: osdmap e1054: 12 total, 12 up, 12 in 2024-08-31T00:01:28.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:28 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:28.655 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:28 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:28 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:28 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:28 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2036407193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-31T00:01:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:28 smithi136 ceph-mon[20207]: osdmap e1054: 12 total, 12 up, 12 in 2024-08-31T00:01:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:28 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:28.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:28 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:28 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:28 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:28 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2036407193' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-31T00:01:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:28 smithi173 ceph-mon[20554]: osdmap e1054: 12 total, 12 up, 12 in 2024-08-31T00:01:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:28 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:28.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:28 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:29.411 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.8 to 1 (10000) 2024-08-31T00:01:29.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:29 smithi106 ceph-mon[16603]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:29.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:29 smithi106 ceph-mon[16603]: osdmap e1055: 12 total, 12 up, 12 in 2024-08-31T00:01:29.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:29 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:29.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:29 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:29.718 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2028776671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-31T00:01:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:29 smithi136 ceph-mon[20207]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:29 smithi136 ceph-mon[20207]: osdmap e1055: 12 total, 12 up, 12 in 2024-08-31T00:01:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:29 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:29 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2028776671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-31T00:01:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:29 smithi173 ceph-mon[20554]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:29 smithi173 ceph-mon[20554]: osdmap e1055: 12 total, 12 up, 12 in 2024-08-31T00:01:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:29 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:29 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:29.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2028776671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-31T00:01:30.143 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 8 1 2024-08-31T00:01:30.387 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:30.672 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:30 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2028776671' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-31T00:01:30.672 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:30 smithi106 ceph-mon[16603]: osdmap e1056: 12 total, 12 up, 12 in 2024-08-31T00:01:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:30 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2028776671' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-31T00:01:30.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:30 smithi136 ceph-mon[20207]: osdmap e1056: 12 total, 12 up, 12 in 2024-08-31T00:01:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:30 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2028776671' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-31T00:01:30.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:30 smithi173 ceph-mon[20554]: osdmap e1056: 12 total, 12 up, 12 in 2024-08-31T00:01:31.121 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-08-31T00:01:31.767 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:31 smithi106 ceph-mon[16603]: pgmap v2325: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:31.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1360079989' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-08-31T00:01:31.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:31 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1360079989' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-31T00:01:31.768 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:31 smithi106 ceph-mon[16603]: osdmap e1057: 12 total, 12 up, 12 in 2024-08-31T00:01:31.789 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:31 smithi136 ceph-mon[20207]: pgmap v2325: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1360079989' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-08-31T00:01:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:31 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1360079989' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-31T00:01:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:31 smithi136 ceph-mon[20207]: osdmap e1057: 12 total, 12 up, 12 in 2024-08-31T00:01:31.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-08-31T00:01:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:31 smithi173 ceph-mon[20554]: pgmap v2325: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1360079989' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-08-31T00:01:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:31 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1360079989' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-08-31T00:01:31.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:31 smithi173 ceph-mon[20554]: osdmap e1057: 12 total, 12 up, 12 in 2024-08-31T00:01:32.038 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:33.428 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.9 to 1 (10000) 2024-08-31T00:01:33.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:33 smithi106 ceph-mon[16603]: pgmap v2327: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:33.749 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:33 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3773981775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:33 smithi136 ceph-mon[20207]: pgmap v2327: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:33 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3773981775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:33 smithi173 ceph-mon[20554]: pgmap v2327: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:33.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:33 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3773981775' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:34.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 9 1 2024-08-31T00:01:34.470 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:34.590 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:34 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3773981775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-31T00:01:34.591 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:34 smithi106 ceph-mon[16603]: osdmap e1058: 12 total, 12 up, 12 in 2024-08-31T00:01:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:34 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3773981775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-31T00:01:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:34 smithi136 ceph-mon[20207]: osdmap e1058: 12 total, 12 up, 12 in 2024-08-31T00:01:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:34 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3773981775' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-31T00:01:34.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:34 smithi173 ceph-mon[20554]: osdmap e1058: 12 total, 12 up, 12 in 2024-08-31T00:01:35.443 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-08-31T00:01:35.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:35 smithi106 ceph-mon[16603]: pgmap v2329: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:35.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/4055754551' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:35.757 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:35 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:35 smithi136 ceph-mon[20207]: pgmap v2329: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/4055754551' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:35 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:35.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:35 smithi173 ceph-mon[20554]: pgmap v2329: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/4055754551' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:35.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:35 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-08-31T00:01:36.137 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-08-31T00:01:36.381 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:36.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:36 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-08-31T00:01:36.736 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:36 smithi106 ceph-mon[16603]: osdmap e1059: 12 total, 12 up, 12 in 2024-08-31T00:01:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:36 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-08-31T00:01:36.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:36 smithi136 ceph-mon[20207]: osdmap e1059: 12 total, 12 up, 12 in 2024-08-31T00:01:36.823 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:36 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-08-31T00:01:36.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:36 smithi173 ceph-mon[20554]: osdmap e1059: 12 total, 12 up, 12 in 2024-08-31T00:01:37.453 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.10 to 1 (10000) 2024-08-31T00:01:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:37 smithi136 ceph-mon[20207]: pgmap v2331: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:37 smithi136 ceph-mon[20207]: osdmap e1060: 12 total, 12 up, 12 in 2024-08-31T00:01:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:37 smithi136 ceph-mon[20207]: 1.9 scrub starts 2024-08-31T00:01:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:37 smithi136 ceph-mon[20207]: 1.9 scrub ok 2024-08-31T00:01:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:37 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/599034435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-31T00:01:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:37 smithi173 ceph-mon[20554]: pgmap v2331: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:37 smithi173 ceph-mon[20554]: osdmap e1060: 12 total, 12 up, 12 in 2024-08-31T00:01:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:37 smithi173 ceph-mon[20554]: 1.9 scrub starts 2024-08-31T00:01:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:37 smithi173 ceph-mon[20554]: 1.9 scrub ok 2024-08-31T00:01:37.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:37 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/599034435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-31T00:01:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:37 smithi106 ceph-mon[16603]: pgmap v2331: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:37 smithi106 ceph-mon[16603]: osdmap e1060: 12 total, 12 up, 12 in 2024-08-31T00:01:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:37 smithi106 ceph-mon[16603]: 1.9 scrub starts 2024-08-31T00:01:37.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:37 smithi106 ceph-mon[16603]: 1.9 scrub ok 2024-08-31T00:01:37.858 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:37 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/599034435' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-31T00:01:38.309 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 10 1 2024-08-31T00:01:38.557 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:38 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/599034435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-31T00:01:38.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:38 smithi106 ceph-mon[16603]: osdmap e1061: 12 total, 12 up, 12 in 2024-08-31T00:01:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:38 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/599034435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-31T00:01:38.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:38 smithi136 ceph-mon[20207]: osdmap e1061: 12 total, 12 up, 12 in 2024-08-31T00:01:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:38 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/599034435' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-31T00:01:38.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:38 smithi173 ceph-mon[20554]: osdmap e1061: 12 total, 12 up, 12 in 2024-08-31T00:01:39.466 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-08-31T00:01:39.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:39 smithi106 ceph-mon[16603]: 1.1b scrub starts 2024-08-31T00:01:39.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:39 smithi106 ceph-mon[16603]: 1.1b scrub ok 2024-08-31T00:01:39.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:39 smithi106 ceph-mon[16603]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:39.782 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:39 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2060902993' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-08-31T00:01:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:39 smithi136 ceph-mon[20207]: 1.1b scrub starts 2024-08-31T00:01:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:39 smithi136 ceph-mon[20207]: 1.1b scrub ok 2024-08-31T00:01:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:39 smithi136 ceph-mon[20207]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:39.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:39 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2060902993' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-08-31T00:01:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:39 smithi173 ceph-mon[20554]: 1.1b scrub starts 2024-08-31T00:01:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:39 smithi173 ceph-mon[20554]: 1.1b scrub ok 2024-08-31T00:01:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:39 smithi173 ceph-mon[20554]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:39 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2060902993' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-08-31T00:01:40.233 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-08-31T00:01:40.480 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:40.729 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:40 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2060902993' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-08-31T00:01:40.730 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:40 smithi106 ceph-mon[16603]: osdmap e1062: 12 total, 12 up, 12 in 2024-08-31T00:01:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:40 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2060902993' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-08-31T00:01:40.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:40 smithi136 ceph-mon[20207]: osdmap e1062: 12 total, 12 up, 12 in 2024-08-31T00:01:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:40 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2060902993' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-08-31T00:01:40.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:40 smithi173 ceph-mon[20554]: osdmap e1062: 12 total, 12 up, 12 in 2024-08-31T00:01:41.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:41 smithi106 ceph-mon[16603]: pgmap v2336: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-08-31T00:01:41.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:41 smithi106 ceph-mon[16603]: 1.0 scrub starts 2024-08-31T00:01:41.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:41 smithi106 ceph-mon[16603]: 1.0 scrub ok 2024-08-31T00:01:41.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:41 smithi106 ceph-mon[16603]: osdmap e1063: 12 total, 12 up, 12 in 2024-08-31T00:01:41.780 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:41 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/528415668' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-31T00:01:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:41 smithi136 ceph-mon[20207]: pgmap v2336: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-08-31T00:01:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:41 smithi136 ceph-mon[20207]: 1.0 scrub starts 2024-08-31T00:01:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:41 smithi136 ceph-mon[20207]: 1.0 scrub ok 2024-08-31T00:01:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:41 smithi136 ceph-mon[20207]: osdmap e1063: 12 total, 12 up, 12 in 2024-08-31T00:01:41.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:41 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/528415668' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-31T00:01:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:41 smithi173 ceph-mon[20554]: pgmap v2336: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-08-31T00:01:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:41 smithi173 ceph-mon[20554]: 1.0 scrub starts 2024-08-31T00:01:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:41 smithi173 ceph-mon[20554]: 1.0 scrub ok 2024-08-31T00:01:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:41 smithi173 ceph-mon[20554]: osdmap e1063: 12 total, 12 up, 12 in 2024-08-31T00:01:41.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:41 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/528415668' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-31T00:01:42.048 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282335]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:42.048 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282335]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.048 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282335]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282335]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282329]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282329]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282329]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282329]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282332]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282332]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282332]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.049 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:01:41 smithi106 sudo[282332]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.128 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.11 to 1 (10000) 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282338]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282338]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282338]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282338]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282341]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282341]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282341]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.358 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282341]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.359 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282356]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:42.359 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282356]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.359 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282356]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.359 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:01:42 smithi106 sudo[282356]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.816 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:42 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:42.816 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:42 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/528415668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-31T00:01:42.816 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:42 smithi136 ceph-mon[20207]: osdmap e1064: 12 total, 12 up, 12 in 2024-08-31T00:01:42.816 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:42.816 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.817 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172023]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.817 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172023]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.817 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:42.817 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:42.817 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172026]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:42.817 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172026]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:42 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:42 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/528415668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-31T00:01:42.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:42 smithi106 ceph-mon[16603]: osdmap e1064: 12 total, 12 up, 12 in 2024-08-31T00:01:42.969 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 11 1 2024-08-31T00:01:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:42 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:42 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/528415668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-31T00:01:43.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:42 smithi173 ceph-mon[20554]: osdmap e1064: 12 total, 12 up, 12 in 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172035]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172035]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172029]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.095 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172029]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.096 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:43.096 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.096 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172032]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.096 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:01:42 smithi136 sudo[172032]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.211 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172038]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172038]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172041]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172041]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.540 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172041]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.541 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172041]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.541 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172044]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:43.541 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172044]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.541 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172044]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.541 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:01:43 smithi136 sudo[172044]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171875]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171875]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171875]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171875]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171878]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:43.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:01:43 smithi173 sudo[171878]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:43 smithi136 ceph-mon[20207]: pgmap v2338: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-08-31T00:01:44.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:43 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:43 smithi173 ceph-mon[20554]: pgmap v2338: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-08-31T00:01:44.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:43 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:43 smithi106 ceph-mon[16603]: pgmap v2338: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-08-31T00:01:44.107 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:43 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:44.433 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:44.433 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:44.433 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171881]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:44.434 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171881]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:44.434 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171884]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:44.434 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171884]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:44.434 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171884]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:44.434 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171884]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:44.727 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171893]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:44.727 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171893]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:44.727 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171893]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:44.727 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171893]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:44.727 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171887]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:44.727 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171887]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:44.728 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171887]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:44.728 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171887]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:44.728 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171890]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:44.728 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171890]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:44.728 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171890]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:44.728 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171890]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:44.750 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/537776055' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172047]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172047]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172047]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172047]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172050]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172050]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:44 smithi136 sudo[172050]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/537776055' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282462]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282462]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282462]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282462]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282477]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:45.057 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282477]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.058 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282477]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.058 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:44 smithi106 sudo[282477]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171896]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:45.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171896]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171896]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171896]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/537776055' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-08-31T00:01:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:01:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:45.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171899]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171899]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171899]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171899]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171902]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171902]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171902]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-31T00:01:45.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:44 smithi173 sudo[171902]: pam_unix(sudo:session): session closed for user root 2024-08-31T00:01:45.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-08-31T00:01:45.707 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: pgmap v2340: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-08-31T00:01:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/537776055' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-08-31T00:01:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: osdmap e1065: 12 total, 12 up, 12 in 2024-08-31T00:01:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.040 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.041 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:45 smithi136 ceph-mon[20207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: pgmap v2340: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/537776055' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: osdmap e1065: 12 total, 12 up, 12 in 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.074 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.075 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:45 smithi173 ceph-mon[20554]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: pgmap v2340: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/537776055' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: osdmap e1065: 12 total, 12 up, 12 in 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.078 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:01:46.079 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:45 smithi106 ceph-mon[16603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-31T00:01:47.137 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.1 to 1 (10000) 2024-08-31T00:01:47.456 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:47 smithi106 ceph-mon[16603]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 164 KiB/s wr, 11 op/s 2024-08-31T00:01:47.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:47 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:47.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:47 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:47.457 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:47 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3257550723' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-31T00:01:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:47 smithi136 ceph-mon[20207]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 164 KiB/s wr, 11 op/s 2024-08-31T00:01:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:47 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:47 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:47.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:47 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3257550723' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-31T00:01:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:47 smithi173 ceph-mon[20554]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 164 KiB/s wr, 11 op/s 2024-08-31T00:01:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:47 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:47 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:47.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:47 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3257550723' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-31T00:01:47.833 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 1 1 2024-08-31T00:01:48.088 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:48.394 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:48 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3257550723' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-31T00:01:48.394 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:48 smithi106 ceph-mon[16603]: osdmap e1066: 12 total, 12 up, 12 in 2024-08-31T00:01:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:48 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3257550723' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-31T00:01:48.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:48 smithi136 ceph-mon[20207]: osdmap e1066: 12 total, 12 up, 12 in 2024-08-31T00:01:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:48 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3257550723' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-31T00:01:48.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:48 smithi173 ceph-mon[20554]: osdmap e1066: 12 total, 12 up, 12 in 2024-08-31T00:01:49.154 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-08-31T00:01:49.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:49 smithi106 ceph-mon[16603]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 164 KiB/s wr, 11 op/s 2024-08-31T00:01:49.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:49 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:49.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:49 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:49.515 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:49 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/466909155' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-31T00:01:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:49 smithi136 ceph-mon[20207]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 164 KiB/s wr, 11 op/s 2024-08-31T00:01:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:49 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:49 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:49.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:49 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/466909155' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-31T00:01:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:49 smithi173 ceph-mon[20554]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 164 KiB/s wr, 11 op/s 2024-08-31T00:01:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:49 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:49 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:49.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:49 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/466909155' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-31T00:01:49.919 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-08-31T00:01:50.171 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:50.416 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:50 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/466909155' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-31T00:01:50.417 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:50 smithi106 ceph-mon[16603]: osdmap e1067: 12 total, 12 up, 12 in 2024-08-31T00:01:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:50 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/466909155' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-31T00:01:50.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:50 smithi136 ceph-mon[20207]: osdmap e1067: 12 total, 12 up, 12 in 2024-08-31T00:01:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:50 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/466909155' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-31T00:01:50.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:50 smithi173 ceph-mon[20554]: osdmap e1067: 12 total, 12 up, 12 in 2024-08-31T00:01:51.127 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.5 to 1 (10000) 2024-08-31T00:01:51.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:51 smithi106 ceph-mon[16603]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 240 KiB/s wr, 16 op/s 2024-08-31T00:01:51.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:51 smithi106 ceph-mon[16603]: osdmap e1068: 12 total, 12 up, 12 in 2024-08-31T00:01:51.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/902963414' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:51.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:51 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/902963414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-31T00:01:51.490 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:51 smithi106 ceph-mon[16603]: osdmap e1069: 12 total, 12 up, 12 in 2024-08-31T00:01:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:51 smithi136 ceph-mon[20207]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 240 KiB/s wr, 16 op/s 2024-08-31T00:01:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:51 smithi136 ceph-mon[20207]: osdmap e1068: 12 total, 12 up, 12 in 2024-08-31T00:01:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/902963414' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:51 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/902963414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-31T00:01:51.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:51 smithi136 ceph-mon[20207]: osdmap e1069: 12 total, 12 up, 12 in 2024-08-31T00:01:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:51 smithi173 ceph-mon[20554]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 240 KiB/s wr, 16 op/s 2024-08-31T00:01:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:51 smithi173 ceph-mon[20554]: osdmap e1068: 12 total, 12 up, 12 in 2024-08-31T00:01:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/902963414' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:51 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/902963414' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-31T00:01:51.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:51 smithi173 ceph-mon[20554]: osdmap e1069: 12 total, 12 up, 12 in 2024-08-31T00:01:51.958 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 5 1 2024-08-31T00:01:52.205 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:52.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:52 smithi106 ceph-mon[16603]: 1.1d scrub starts 2024-08-31T00:01:52.326 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:52 smithi106 ceph-mon[16603]: 1.1d scrub ok 2024-08-31T00:01:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:52 smithi136 ceph-mon[20207]: 1.1d scrub starts 2024-08-31T00:01:52.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:52 smithi136 ceph-mon[20207]: 1.1d scrub ok 2024-08-31T00:01:52.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:52 smithi173 ceph-mon[20554]: 1.1d scrub starts 2024-08-31T00:01:52.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:52 smithi173 ceph-mon[20554]: 1.1d scrub ok 2024-08-31T00:01:53.173 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-08-31T00:01:53.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:53 smithi106 ceph-mon[16603]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-08-31T00:01:53.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:53 smithi106 ceph-mon[16603]: 1.1 scrub starts 2024-08-31T00:01:53.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:53 smithi106 ceph-mon[16603]: 1.1 scrub ok 2024-08-31T00:01:53.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:53 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1277827984' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:53.514 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:53 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:53 smithi136 ceph-mon[20207]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-08-31T00:01:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:53 smithi136 ceph-mon[20207]: 1.1 scrub starts 2024-08-31T00:01:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:53 smithi136 ceph-mon[20207]: 1.1 scrub ok 2024-08-31T00:01:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:53 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1277827984' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:53.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:53 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:53 smithi173 ceph-mon[20554]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-08-31T00:01:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:53 smithi173 ceph-mon[20554]: 1.1 scrub starts 2024-08-31T00:01:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:53 smithi173 ceph-mon[20554]: 1.1 scrub ok 2024-08-31T00:01:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:53 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1277827984' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:53.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:53 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-31T00:01:53.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-08-31T00:01:54.187 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:54.446 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:54 smithi106 ceph-mon[16603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-31T00:01:54.446 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:54 smithi106 ceph-mon[16603]: osdmap e1070: 12 total, 12 up, 12 in 2024-08-31T00:01:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:54 smithi136 ceph-mon[20207]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-31T00:01:54.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:54 smithi136 ceph-mon[20207]: osdmap e1070: 12 total, 12 up, 12 in 2024-08-31T00:01:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:54 smithi173 ceph-mon[20554]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-31T00:01:54.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:54 smithi173 ceph-mon[20554]: osdmap e1070: 12 total, 12 up, 12 in 2024-08-31T00:01:55.186 INFO:teuthology.orchestra.run.smithi106.stderr:reweighted osd.7 to 1 (10000) 2024-08-31T00:01:55.505 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:55 smithi106 ceph-mon[16603]: pgmap v2351: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 215 B/s rd, 28 KiB/s wr, 1 op/s 2024-08-31T00:01:55.506 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:55 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2561668511' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-31T00:01:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:55 smithi136 ceph-mon[20207]: pgmap v2351: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 215 B/s rd, 28 KiB/s wr, 1 op/s 2024-08-31T00:01:55.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:55 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2561668511' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-31T00:01:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:55 smithi173 ceph-mon[20554]: pgmap v2351: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 215 B/s rd, 28 KiB/s wr, 1 op/s 2024-08-31T00:01:55.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:55 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2561668511' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-31T00:01:55.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-31T00:01:56.130 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:56.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:56 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2561668511' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-31T00:01:56.493 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:56 smithi106 ceph-mon[16603]: osdmap e1071: 12 total, 12 up, 12 in 2024-08-31T00:01:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:56 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2561668511' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-31T00:01:56.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:56 smithi136 ceph-mon[20207]: osdmap e1071: 12 total, 12 up, 12 in 2024-08-31T00:01:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:56 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2561668511' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-31T00:01:56.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:56 smithi173 ceph-mon[20554]: osdmap e1071: 12 total, 12 up, 12 in 2024-08-31T00:01:57.199 INFO:teuthology.orchestra.run.smithi106.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-31T00:01:57.513 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:57 smithi106 ceph-mon[16603]: pgmap v2353: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:57.513 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:57 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3787920387' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-31T00:01:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:57 smithi136 ceph-mon[20207]: pgmap v2353: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:57.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:57 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3787920387' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-31T00:01:57.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:57 smithi173 ceph-mon[20554]: pgmap v2353: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:57.675 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:57 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3787920387' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-31T00:01:57.939 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-08-31T00:01:57.939 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-08-31T00:01:57.939 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-08-31T00:01:57.939 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-08-31T00:01:57.940 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-31T00:01:57.940 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-31T00:01:58.185 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:01:58.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:58 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:01:58.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:58 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:01:58.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:58 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3787920387' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-31T00:01:58.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:58 smithi106 ceph-mon[16603]: osdmap e1072: 12 total, 12 up, 12 in 2024-08-31T00:01:58.460 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:58 smithi106 ceph-mon[16603]: 1.19 scrub starts 2024-08-31T00:01:58.461 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:58 smithi106 ceph-mon[16603]: 1.19 scrub ok 2024-08-31T00:01:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:58 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:01:58.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:58 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:01:58.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:58 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3787920387' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-31T00:01:58.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:58 smithi136 ceph-mon[20207]: osdmap e1072: 12 total, 12 up, 12 in 2024-08-31T00:01:58.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:58 smithi136 ceph-mon[20207]: 1.19 scrub starts 2024-08-31T00:01:58.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:58 smithi136 ceph-mon[20207]: 1.19 scrub ok 2024-08-31T00:01:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:58 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:01:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:58 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:01:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:58 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3787920387' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-31T00:01:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:58 smithi173 ceph-mon[20554]: osdmap e1072: 12 total, 12 up, 12 in 2024-08-31T00:01:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:58 smithi173 ceph-mon[20554]: 1.19 scrub starts 2024-08-31T00:01:58.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:58 smithi173 ceph-mon[20554]: 1.19 scrub ok 2024-08-31T00:01:58.877 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-31T00:01:58.877 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1073,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","created":"2024-08-30T23:18:02.086618+0000","modified":"2024-08-31T00:01:58.199581+0000","last_up_change":"2024-08-31T00:01:04.790539+0000","last_in_change":"2024-08-30T23:59:13.167369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T23:21:09.548123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1005","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"877","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5499999523162842,"score_stable":1.5499999523162842,"optimal_score":1,"raw_score_acting":1.5499999523162842,"raw_score_stable":1.5499999523162842,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48096b59-4ae9-43da-a0c2-a689ad9fe2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1043,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6803","nonce":3938533896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6805","nonce":3938533896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6809","nonce":3938533896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6807","nonce":3938533896}]},"public_addr":"172.21.15.106:6803/3938533896","cluster_addr":"172.21.15.106:6805/3938533896","heartbeat_back_addr":"172.21.15.106:6809/3938533896","heartbeat_front_addr":"172.21.15.106:6807/3938533896","state":["exists","up"]},{"osd":1,"uuid":"0b7f3c0d-489c-48b4-91ca-fa8da6baef53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":368,"up_from":525,"up_thru":1067,"down_at":369,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6811","nonce":2928503139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6813","nonce":2928503139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6817","nonce":2928503139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6815","nonce":2928503139}]},"public_addr":"172.21.15.106:6811/2928503139","cluster_addr":"172.21.15.106:6813/2928503139","heartbeat_back_addr":"172.21.15.106:6817/2928503139","heartbeat_front_addr":"172.21.15.106:6815/2928503139","state":["exists","up"]},{"osd":2,"uuid":"b2453c8c-444c-4a5c-9c37-36f6f0213219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1046,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6819","nonce":2791969052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6821","nonce":2791969052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6825","nonce":2791969052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6823","nonce":2791969052}]},"public_addr":"172.21.15.106:6819/2791969052","cluster_addr":"172.21.15.106:6821/2791969052","heartbeat_back_addr":"172.21.15.106:6825/2791969052","heartbeat_front_addr":"172.21.15.106:6823/2791969052","state":["exists","up"]},{"osd":3,"uuid":"d1778ea7-0e45-4646-8fd2-b0f7c9e3086f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1030,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6827","nonce":2233735435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6829","nonce":2233735435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6833","nonce":2233735435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6831","nonce":2233735435}]},"public_addr":"172.21.15.106:6827/2233735435","cluster_addr":"172.21.15.106:6829/2233735435","heartbeat_back_addr":"172.21.15.106:6833/2233735435","heartbeat_front_addr":"172.21.15.106:6831/2233735435","state":["exists","up"]},{"osd":4,"uuid":"0335aeb0-87b1-4cff-9351-c489ad188729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":1039,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6801","nonce":2130223341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6803","nonce":2130223341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6807","nonce":2130223341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6805","nonce":2130223341}]},"public_addr":"172.21.15.136:6801/2130223341","cluster_addr":"172.21.15.136:6803/2130223341","heartbeat_back_addr":"172.21.15.136:6807/2130223341","heartbeat_front_addr":"172.21.15.136:6805/2130223341","state":["exists","up"]},{"osd":5,"uuid":"dc2279d3-fcec-4934-8a75-9befec1bbbe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":566,"last_clean_end":593,"up_from":649,"up_thru":1039,"down_at":594,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6809","nonce":2739515404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6811","nonce":2739515404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6815","nonce":2739515404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6813","nonce":2739515404}]},"public_addr":"172.21.15.136:6809/2739515404","cluster_addr":"172.21.15.136:6811/2739515404","heartbeat_back_addr":"172.21.15.136:6815/2739515404","heartbeat_front_addr":"172.21.15.136:6813/2739515404","state":["exists","up"]},{"osd":6,"uuid":"dddb0cd9-8f57-4836-96fd-f66925b36c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6817","nonce":2133537865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6819","nonce":2133537865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6823","nonce":2133537865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6821","nonce":2133537865}]},"public_addr":"172.21.15.136:6817/2133537865","cluster_addr":"172.21.15.136:6819/2133537865","heartbeat_back_addr":"172.21.15.136:6823/2133537865","heartbeat_front_addr":"172.21.15.136:6821/2133537865","state":["exists","up"]},{"osd":7,"uuid":"b9894fb4-6f34-47cd-81cd-9e7d9758360b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":902,"up_from":1039,"up_thru":1072,"down_at":903,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6825","nonce":907316111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6827","nonce":907316111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6831","nonce":907316111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6829","nonce":907316111}]},"public_addr":"172.21.15.136:6825/907316111","cluster_addr":"172.21.15.136:6827/907316111","heartbeat_back_addr":"172.21.15.136:6831/907316111","heartbeat_front_addr":"172.21.15.136:6829/907316111","state":["exists","up"]},{"osd":8,"uuid":"046e019e-1e33-403c-9577-757c7d546f83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6801","nonce":526702876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6803","nonce":526702876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6807","nonce":526702876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6805","nonce":526702876}]},"public_addr":"172.21.15.173:6801/526702876","cluster_addr":"172.21.15.173:6803/526702876","heartbeat_back_addr":"172.21.15.173:6807/526702876","heartbeat_front_addr":"172.21.15.173:6805/526702876","state":["exists","up"]},{"osd":9,"uuid":"ea9ca538-e1da-4687-8b4f-542c0fc3737c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":1059,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6809","nonce":478092222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6811","nonce":478092222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6815","nonce":478092222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6813","nonce":478092222}]},"public_addr":"172.21.15.173:6809/478092222","cluster_addr":"172.21.15.173:6811/478092222","heartbeat_back_addr":"172.21.15.173:6815/478092222","heartbeat_front_addr":"172.21.15.173:6813/478092222","state":["exists","up"]},{"osd":10,"uuid":"085c32cd-dde4-4a54-bb0d-9182075040c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":998,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6817","nonce":225763808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6819","nonce":225763808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6823","nonce":225763808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6821","nonce":225763808}]},"public_addr":"172.21.15.173:6817/225763808","cluster_addr":"172.21.15.173:6819/225763808","heartbeat_back_addr":"172.21.15.173:6823/225763808","heartbeat_front_addr":"172.21.15.173:6821/225763808","state":["exists","up"]},{"osd":11,"uuid":"6bd9ce25-ce61-4998-af79-26c4ebe2ce3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6825","nonce":2219423084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6827","nonce":2219423084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6831","nonce":2219423084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6829","nonce":2219423084}]},"public_addr":"172.21.15.173:6825/2219423084","cluster_addr":"172.21.15.173:6827/2219423084","heartbeat_back_addr":"172.21.15.173:6831/2219423084","heartbeat_front_addr":"172.21.15.173:6829/2219423084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:00:55.907841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:00:54.132294+0000","dead_epoch":368},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:44.323947+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:27.300198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:46.337023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:00:55.763198+0000","dead_epoch":593},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:28.428871+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:04.342789+0000","dead_epoch":902},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:48.594235+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:27.841702+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:56.958276+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:18.171317+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[6,7,11]},{"pgid":"1.1","osds":[2,0,10]},{"pgid":"1.f","osds":[0,6,9]}],"pg_upmap_items":[{"pgid":"1.c","mappings":[{"from":0,"to":5},{"from":6,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3708473409":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/1031615522":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3817969167":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3688603346":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6801/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3790397394":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:0/1787333807":"2024-08-31T23:18:27.701240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T00:01:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:59 smithi136 ceph-mon[20207]: pgmap v2355: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:59 smithi136 ceph-mon[20207]: osdmap e1073: 12 total, 12 up, 12 in 2024-08-31T00:01:59.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:01:59 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2079435419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-31T00:01:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:59 smithi173 ceph-mon[20554]: pgmap v2355: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:59 smithi173 ceph-mon[20554]: osdmap e1073: 12 total, 12 up, 12 in 2024-08-31T00:01:59.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:01:59 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2079435419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-31T00:01:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:59 smithi106 ceph-mon[16603]: pgmap v2355: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:01:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:59 smithi106 ceph-mon[16603]: osdmap e1073: 12 total, 12 up, 12 in 2024-08-31T00:01:59.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:01:59 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2079435419' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-31T00:01:59.729 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-31T00:01:59.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-08-31T00:01:59.975 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:00 smithi106 ceph-mon[16603]: 1.d scrub starts 2024-08-31T00:02:00.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:00 smithi106 ceph-mon[16603]: 1.d scrub ok 2024-08-31T00:02:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:00 smithi136 ceph-mon[20207]: 1.d scrub starts 2024-08-31T00:02:00.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:00 smithi136 ceph-mon[20207]: 1.d scrub ok 2024-08-31T00:02:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:00 smithi173 ceph-mon[20554]: 1.d scrub starts 2024-08-31T00:02:00.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:00 smithi173 ceph-mon[20554]: 1.d scrub ok 2024-08-31T00:02:00.628 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-31T00:02:00.628 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1073,"fsid":"eb936aa2-6725-11ef-bcd4-c7b262605968","created":"2024-08-30T23:18:02.086618+0000","modified":"2024-08-31T00:01:58.199581+0000","last_up_change":"2024-08-31T00:01:04.790539+0000","last_in_change":"2024-08-30T23:59:13.167369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-30T23:21:09.548123+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1005","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"877","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5499999523162842,"score_stable":1.5499999523162842,"optimal_score":1,"raw_score_acting":1.5499999523162842,"raw_score_stable":1.5499999523162842,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48096b59-4ae9-43da-a0c2-a689ad9fe2e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1043,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6803","nonce":3938533896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6805","nonce":3938533896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6809","nonce":3938533896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3938533896},{"type":"v1","addr":"172.21.15.106:6807","nonce":3938533896}]},"public_addr":"172.21.15.106:6803/3938533896","cluster_addr":"172.21.15.106:6805/3938533896","heartbeat_back_addr":"172.21.15.106:6809/3938533896","heartbeat_front_addr":"172.21.15.106:6807/3938533896","state":["exists","up"]},{"osd":1,"uuid":"0b7f3c0d-489c-48b4-91ca-fa8da6baef53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":368,"up_from":525,"up_thru":1067,"down_at":369,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6811","nonce":2928503139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6813","nonce":2928503139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6817","nonce":2928503139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2928503139},{"type":"v1","addr":"172.21.15.106:6815","nonce":2928503139}]},"public_addr":"172.21.15.106:6811/2928503139","cluster_addr":"172.21.15.106:6813/2928503139","heartbeat_back_addr":"172.21.15.106:6817/2928503139","heartbeat_front_addr":"172.21.15.106:6815/2928503139","state":["exists","up"]},{"osd":2,"uuid":"b2453c8c-444c-4a5c-9c37-36f6f0213219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1046,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6819","nonce":2791969052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6821","nonce":2791969052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6825","nonce":2791969052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":2791969052},{"type":"v1","addr":"172.21.15.106:6823","nonce":2791969052}]},"public_addr":"172.21.15.106:6819/2791969052","cluster_addr":"172.21.15.106:6821/2791969052","heartbeat_back_addr":"172.21.15.106:6825/2791969052","heartbeat_front_addr":"172.21.15.106:6823/2791969052","state":["exists","up"]},{"osd":3,"uuid":"d1778ea7-0e45-4646-8fd2-b0f7c9e3086f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1030,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6827","nonce":2233735435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6829","nonce":2233735435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6833","nonce":2233735435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":2233735435},{"type":"v1","addr":"172.21.15.106:6831","nonce":2233735435}]},"public_addr":"172.21.15.106:6827/2233735435","cluster_addr":"172.21.15.106:6829/2233735435","heartbeat_back_addr":"172.21.15.106:6833/2233735435","heartbeat_front_addr":"172.21.15.106:6831/2233735435","state":["exists","up"]},{"osd":4,"uuid":"0335aeb0-87b1-4cff-9351-c489ad188729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":1039,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6801","nonce":2130223341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6803","nonce":2130223341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6807","nonce":2130223341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":2130223341},{"type":"v1","addr":"172.21.15.136:6805","nonce":2130223341}]},"public_addr":"172.21.15.136:6801/2130223341","cluster_addr":"172.21.15.136:6803/2130223341","heartbeat_back_addr":"172.21.15.136:6807/2130223341","heartbeat_front_addr":"172.21.15.136:6805/2130223341","state":["exists","up"]},{"osd":5,"uuid":"dc2279d3-fcec-4934-8a75-9befec1bbbe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":566,"last_clean_end":593,"up_from":649,"up_thru":1039,"down_at":594,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6809","nonce":2739515404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6811","nonce":2739515404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6815","nonce":2739515404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2739515404},{"type":"v1","addr":"172.21.15.136:6813","nonce":2739515404}]},"public_addr":"172.21.15.136:6809/2739515404","cluster_addr":"172.21.15.136:6811/2739515404","heartbeat_back_addr":"172.21.15.136:6815/2739515404","heartbeat_front_addr":"172.21.15.136:6813/2739515404","state":["exists","up"]},{"osd":6,"uuid":"dddb0cd9-8f57-4836-96fd-f66925b36c84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6817","nonce":2133537865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6819","nonce":2133537865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6823","nonce":2133537865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":2133537865},{"type":"v1","addr":"172.21.15.136:6821","nonce":2133537865}]},"public_addr":"172.21.15.136:6817/2133537865","cluster_addr":"172.21.15.136:6819/2133537865","heartbeat_back_addr":"172.21.15.136:6823/2133537865","heartbeat_front_addr":"172.21.15.136:6821/2133537865","state":["exists","up"]},{"osd":7,"uuid":"b9894fb4-6f34-47cd-81cd-9e7d9758360b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":54,"last_clean_end":902,"up_from":1039,"up_thru":1072,"down_at":903,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6825","nonce":907316111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6827","nonce":907316111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6831","nonce":907316111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":907316111},{"type":"v1","addr":"172.21.15.136:6829","nonce":907316111}]},"public_addr":"172.21.15.136:6825/907316111","cluster_addr":"172.21.15.136:6827/907316111","heartbeat_back_addr":"172.21.15.136:6831/907316111","heartbeat_front_addr":"172.21.15.136:6829/907316111","state":["exists","up"]},{"osd":8,"uuid":"046e019e-1e33-403c-9577-757c7d546f83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6801","nonce":526702876}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6803","nonce":526702876}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6807","nonce":526702876}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":526702876},{"type":"v1","addr":"172.21.15.173:6805","nonce":526702876}]},"public_addr":"172.21.15.173:6801/526702876","cluster_addr":"172.21.15.173:6803/526702876","heartbeat_back_addr":"172.21.15.173:6807/526702876","heartbeat_front_addr":"172.21.15.173:6805/526702876","state":["exists","up"]},{"osd":9,"uuid":"ea9ca538-e1da-4687-8b4f-542c0fc3737c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":1059,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6809","nonce":478092222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6811","nonce":478092222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6815","nonce":478092222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":478092222},{"type":"v1","addr":"172.21.15.173:6813","nonce":478092222}]},"public_addr":"172.21.15.173:6809/478092222","cluster_addr":"172.21.15.173:6811/478092222","heartbeat_back_addr":"172.21.15.173:6815/478092222","heartbeat_front_addr":"172.21.15.173:6813/478092222","state":["exists","up"]},{"osd":10,"uuid":"085c32cd-dde4-4a54-bb0d-9182075040c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":998,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6817","nonce":225763808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6819","nonce":225763808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6823","nonce":225763808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":225763808},{"type":"v1","addr":"172.21.15.173:6821","nonce":225763808}]},"public_addr":"172.21.15.173:6817/225763808","cluster_addr":"172.21.15.173:6819/225763808","heartbeat_back_addr":"172.21.15.173:6823/225763808","heartbeat_front_addr":"172.21.15.173:6821/225763808","state":["exists","up"]},{"osd":11,"uuid":"6bd9ce25-ce61-4998-af79-26c4ebe2ce3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":86,"up_thru":1040,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6825","nonce":2219423084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6827","nonce":2219423084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6831","nonce":2219423084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":2219423084},{"type":"v1","addr":"172.21.15.173:6829","nonce":2219423084}]},"public_addr":"172.21.15.173:6825/2219423084","cluster_addr":"172.21.15.173:6827/2219423084","heartbeat_back_addr":"172.21.15.173:6831/2219423084","heartbeat_front_addr":"172.21.15.173:6829/2219423084","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:00:55.907841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:00:54.132294+0000","dead_epoch":368},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:44.323947+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:27.300198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:46.337023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:00:55.763198+0000","dead_epoch":593},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:28.428871+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:04.342789+0000","dead_epoch":902},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:48.594235+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:27.841702+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:56.958276+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-08-31T00:01:18.171317+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[6,7,11]},{"pgid":"1.1","osds":[2,0,10]},{"pgid":"1.f","osds":[0,6,9]}],"pg_upmap_items":[{"pgid":"1.c","mappings":[{"from":0,"to":5},{"from":6,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.106:6801/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3708473409":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/1031615522":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3817969167":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3688603346":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6801/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1186392683":"2024-08-31T23:18:51.239661+0000","172.21.15.106:0/3790397394":"2024-08-31T23:18:27.701240+0000","172.21.15.106:6800/1549137855":"2024-08-31T23:18:27.701240+0000","172.21.15.106:0/1787333807":"2024-08-31T23:18:27.701240+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-31T00:02:01.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-31T00:02:01.217 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-31T00:02:01.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-31T00:02:01.218 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-31T00:02:01.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-31T00:02:01.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-31T00:02:01.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-31T00:02:01.220 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-31T00:02:01.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-31T00:02:01.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-31T00:02:01.221 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-31T00:02:01.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-31T00:02:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:01 smithi136 ceph-mon[20207]: pgmap v2357: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:01.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:01 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1588420910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-31T00:02:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:01 smithi173 ceph-mon[20554]: pgmap v2357: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:01.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:01 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1588420910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-31T00:02:01.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:01 smithi106 ceph-mon[16603]: pgmap v2357: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:01.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:01 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1588420910' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-31T00:02:01.849 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.199 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.200 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.205 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.209 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.209 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.210 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.210 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.213 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.214 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.225 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:02.481 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:03.494 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:03 smithi106 ceph-mon[16603]: pgmap v2358: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:03.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:03 smithi136 ceph-mon[20207]: pgmap v2358: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:03.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:03 smithi173 ceph-mon[20554]: pgmap v2358: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:04.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:04 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:02:04.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:04 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:02:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:04 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:02:04.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:04 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:02:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:04 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:02:04.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:04 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:02:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:05 smithi136 ceph-mon[20207]: pgmap v2359: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:05.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:05 smithi136 ceph-mon[20207]: osdmap e1074: 12 total, 12 up, 12 in 2024-08-31T00:02:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:05 smithi173 ceph-mon[20554]: pgmap v2359: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:05.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:05 smithi173 ceph-mon[20554]: osdmap e1074: 12 total, 12 up, 12 in 2024-08-31T00:02:05.577 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:05 smithi106 ceph-mon[16603]: pgmap v2359: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:05.578 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:05 smithi106 ceph-mon[16603]: osdmap e1074: 12 total, 12 up, 12 in 2024-08-31T00:02:05.815 INFO:teuthology.orchestra.run.smithi106.stdout:300647711178 2024-08-31T00:02:05.815 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-31T00:02:06.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: 1.b scrub starts 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: 1.b scrub ok 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: 1.f scrub starts 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: 1.f scrub ok 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: 1.1c scrub starts 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: 1.1c scrub ok 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:02:06.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:06 smithi106 ceph-mon[16603]: osdmap e1075: 12 total, 12 up, 12 in 2024-08-31T00:02:06.420 INFO:teuthology.orchestra.run.smithi106.stdout:77309411825 2024-08-31T00:02:06.420 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: 1.b scrub starts 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: 1.b scrub ok 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: 1.f scrub starts 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: 1.f scrub ok 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: 1.1c scrub starts 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: 1.1c scrub ok 2024-08-31T00:02:06.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:02:06.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:02:06.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:06 smithi136 ceph-mon[20207]: osdmap e1075: 12 total, 12 up, 12 in 2024-08-31T00:02:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: 1.b scrub starts 2024-08-31T00:02:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: 1.b scrub ok 2024-08-31T00:02:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: 1.f scrub starts 2024-08-31T00:02:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: 1.f scrub ok 2024-08-31T00:02:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: 1.1c scrub starts 2024-08-31T00:02:06.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: 1.1c scrub ok 2024-08-31T00:02:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:02:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:02:06.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:06 smithi173 ceph-mon[20554]: osdmap e1075: 12 total, 12 up, 12 in 2024-08-31T00:02:07.322 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:07 smithi106 ceph-mon[16603]: pgmap v2361: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:07.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:07 smithi136 ceph-mon[20207]: pgmap v2361: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:07.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:07 smithi173 ceph-mon[20554]: pgmap v2361: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:08.553 INFO:teuthology.orchestra.run.smithi106.stdout:137438953961 2024-08-31T00:02:08.554 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-31T00:02:09.375 INFO:teuthology.orchestra.run.smithi106.stdout:2787433775301 2024-08-31T00:02:09.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-31T00:02:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:09 smithi136 ceph-mon[20207]: pgmap v2363: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:09 smithi136 ceph-mon[20207]: 1.5 scrub starts 2024-08-31T00:02:09.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:09 smithi136 ceph-mon[20207]: 1.5 scrub ok 2024-08-31T00:02:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:09 smithi173 ceph-mon[20554]: pgmap v2363: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:09 smithi173 ceph-mon[20554]: 1.5 scrub starts 2024-08-31T00:02:09.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:09 smithi173 ceph-mon[20554]: 1.5 scrub ok 2024-08-31T00:02:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:09 smithi106 ceph-mon[16603]: pgmap v2363: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:09 smithi106 ceph-mon[16603]: 1.5 scrub starts 2024-08-31T00:02:09.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:09 smithi106 ceph-mon[16603]: 1.5 scrub ok 2024-08-31T00:02:10.054 INFO:teuthology.orchestra.run.smithi106.stdout:335007449549 2024-08-31T00:02:10.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-31T00:02:10.153 INFO:teuthology.orchestra.run.smithi106.stdout:369367187910 2024-08-31T00:02:10.153 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-31T00:02:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:10 smithi106 ceph-mon[16603]: 1.11 scrub starts 2024-08-31T00:02:10.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:10 smithi106 ceph-mon[16603]: 1.11 scrub ok 2024-08-31T00:02:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:10 smithi136 ceph-mon[20207]: 1.11 scrub starts 2024-08-31T00:02:10.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:10 smithi136 ceph-mon[20207]: 1.11 scrub ok 2024-08-31T00:02:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:10 smithi173 ceph-mon[20554]: 1.11 scrub starts 2024-08-31T00:02:10.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:10 smithi173 ceph-mon[20554]: 1.11 scrub ok 2024-08-31T00:02:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:11 smithi136 ceph-mon[20207]: pgmap v2364: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:11 smithi136 ceph-mon[20207]: 1.a scrub starts 2024-08-31T00:02:11.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:11 smithi136 ceph-mon[20207]: 1.a scrub ok 2024-08-31T00:02:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:11 smithi173 ceph-mon[20554]: pgmap v2364: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:11 smithi173 ceph-mon[20554]: 1.a scrub starts 2024-08-31T00:02:11.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:11 smithi173 ceph-mon[20554]: 1.a scrub ok 2024-08-31T00:02:11.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:11 smithi106 ceph-mon[16603]: pgmap v2364: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:11.592 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:11 smithi106 ceph-mon[16603]: 1.a scrub starts 2024-08-31T00:02:11.593 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:11 smithi106 ceph-mon[16603]: 1.a scrub ok 2024-08-31T00:02:12.165 INFO:teuthology.orchestra.run.smithi106.stdout:4462471020558 2024-08-31T00:02:12.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-31T00:02:12.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:12 smithi106 ceph-mon[16603]: 1.3 scrub starts 2024-08-31T00:02:12.492 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:12 smithi106 ceph-mon[16603]: 1.3 scrub ok 2024-08-31T00:02:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:12 smithi136 ceph-mon[20207]: 1.3 scrub starts 2024-08-31T00:02:12.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:12 smithi136 ceph-mon[20207]: 1.3 scrub ok 2024-08-31T00:02:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:12 smithi173 ceph-mon[20554]: 1.3 scrub starts 2024-08-31T00:02:12.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:12 smithi173 ceph-mon[20554]: 1.3 scrub ok 2024-08-31T00:02:12.890 INFO:teuthology.orchestra.run.smithi106.stdout:193273528799 2024-08-31T00:02:12.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-31T00:02:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:13 smithi136 ceph-mon[20207]: pgmap v2365: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:13 smithi136 ceph-mon[20207]: 1.1a scrub starts 2024-08-31T00:02:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:13 smithi136 ceph-mon[20207]: 1.1a scrub ok 2024-08-31T00:02:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:13 smithi136 ceph-mon[20207]: 1.7 scrub starts 2024-08-31T00:02:13.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:13 smithi136 ceph-mon[20207]: 1.7 scrub ok 2024-08-31T00:02:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:13 smithi173 ceph-mon[20554]: pgmap v2365: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:13 smithi173 ceph-mon[20554]: 1.1a scrub starts 2024-08-31T00:02:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:13 smithi173 ceph-mon[20554]: 1.1a scrub ok 2024-08-31T00:02:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:13 smithi173 ceph-mon[20554]: 1.7 scrub starts 2024-08-31T00:02:13.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:13 smithi173 ceph-mon[20554]: 1.7 scrub ok 2024-08-31T00:02:13.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:13 smithi106 ceph-mon[16603]: pgmap v2365: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:13 smithi106 ceph-mon[16603]: 1.1a scrub starts 2024-08-31T00:02:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:13 smithi106 ceph-mon[16603]: 1.1a scrub ok 2024-08-31T00:02:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:13 smithi106 ceph-mon[16603]: 1.7 scrub starts 2024-08-31T00:02:13.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:13 smithi106 ceph-mon[16603]: 1.7 scrub ok 2024-08-31T00:02:14.172 INFO:teuthology.orchestra.run.smithi106.stdout:34359738877 2024-08-31T00:02:14.172 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-31T00:02:14.269 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:14 smithi106 ceph-mon[16603]: 1.14 scrub starts 2024-08-31T00:02:14.269 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:14 smithi106 ceph-mon[16603]: 1.14 scrub ok 2024-08-31T00:02:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:14 smithi136 ceph-mon[20207]: 1.14 scrub starts 2024-08-31T00:02:14.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:14 smithi136 ceph-mon[20207]: 1.14 scrub ok 2024-08-31T00:02:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:14 smithi173 ceph-mon[20554]: 1.14 scrub starts 2024-08-31T00:02:14.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:14 smithi173 ceph-mon[20554]: 1.14 scrub ok 2024-08-31T00:02:14.701 INFO:teuthology.orchestra.run.smithi106.stdout:111669150187 2024-08-31T00:02:14.702 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-31T00:02:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: pgmap v2366: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.359 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:15.360 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:15 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:15.538 INFO:teuthology.orchestra.run.smithi106.stdout:2254857830642 2024-08-31T00:02:15.539 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: pgmap v2366: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:02:15.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:15.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:15 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: pgmap v2366: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:15.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:15.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-31T00:02:15.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-31T00:02:15.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-31T00:02:15.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:15.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:15.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:15.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:15.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:15.576 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:15 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:15.993 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:16.001 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:16.046 INFO:teuthology.orchestra.run.smithi106.stdout:266287972823 2024-08-31T00:02:16.046 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-31T00:02:16.606 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:16.606 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:16.763 INFO:teuthology.orchestra.run.smithi106.stdout:335007449551 2024-08-31T00:02:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:17 smithi136 ceph-mon[20207]: 1.4 scrub starts 2024-08-31T00:02:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:17 smithi136 ceph-mon[20207]: 1.4 scrub ok 2024-08-31T00:02:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:17 smithi136 ceph-mon[20207]: pgmap v2367: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:17.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:17 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3507378016' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-31T00:02:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:17 smithi173 ceph-mon[20554]: 1.4 scrub starts 2024-08-31T00:02:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:17 smithi173 ceph-mon[20554]: 1.4 scrub ok 2024-08-31T00:02:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:17 smithi173 ceph-mon[20554]: pgmap v2367: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:17.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:17 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3507378016' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-31T00:02:17.585 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:17 smithi106 ceph-mon[16603]: 1.4 scrub starts 2024-08-31T00:02:17.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:17 smithi106 ceph-mon[16603]: 1.4 scrub ok 2024-08-31T00:02:17.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:17 smithi106 ceph-mon[16603]: pgmap v2367: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:17.586 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:17 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3507378016' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-31T00:02:17.976 INFO:teuthology.orchestra.run.smithi106.stdout:300647711181 2024-08-31T00:02:18.156 INFO:teuthology.orchestra.run.smithi106.stdout:2787433775304 2024-08-31T00:02:18.522 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:18 smithi106 ceph-mon[16603]: 1.c scrub starts 2024-08-31T00:02:18.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:18 smithi106 ceph-mon[16603]: 1.c scrub ok 2024-08-31T00:02:18.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3692153308' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-31T00:02:18.523 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:18 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/1626532719' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-31T00:02:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:18 smithi136 ceph-mon[20207]: 1.c scrub starts 2024-08-31T00:02:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:18 smithi136 ceph-mon[20207]: 1.c scrub ok 2024-08-31T00:02:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3692153308' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-31T00:02:18.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:18 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/1626532719' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-31T00:02:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:18 smithi173 ceph-mon[20554]: 1.c scrub starts 2024-08-31T00:02:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:18 smithi173 ceph-mon[20554]: 1.c scrub ok 2024-08-31T00:02:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3692153308' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-31T00:02:18.575 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:18 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/1626532719' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-31T00:02:18.639 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:18.640 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:18.808 INFO:teuthology.orchestra.run.smithi106.stdout:4462471020560 2024-08-31T00:02:19.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647711178 got 300647711181 for osd.9 2024-08-31T00:02:19.129 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:19 smithi106 ceph-mon[16603]: pgmap v2368: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:19.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:19 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2086199076' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-31T00:02:19.391 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:19.395 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:19.395 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:19 smithi136 ceph-mon[20207]: pgmap v2368: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:19.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:19 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2086199076' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-31T00:02:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:19 smithi173 ceph-mon[20554]: pgmap v2368: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:19.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:19 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2086199076' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-31T00:02:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:20 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:02:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:20 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:02:20.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:20 smithi106 ceph-mon[16603]: osdmap e1076: 12 total, 12 up, 12 in 2024-08-31T00:02:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:20 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:02:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:20 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:02:20.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:20 smithi136 ceph-mon[20207]: osdmap e1076: 12 total, 12 up, 12 in 2024-08-31T00:02:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:20 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:02:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:20 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:02:20.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:20 smithi173 ceph-mon[20554]: osdmap e1076: 12 total, 12 up, 12 in 2024-08-31T00:02:20.852 INFO:teuthology.orchestra.run.smithi106.stderr:2024-08-31T00:02:20.851+0000 7fb0fb7fe640 0 --2- 172.21.15.106:0/954644978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0fc1073f0 0x7fb0fc107830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-31T00:02:21.021 INFO:teuthology.orchestra.run.smithi106.stdout:193273528801 2024-08-31T00:02:21.045 INFO:teuthology.orchestra.run.smithi106.stdout:34359738880 2024-08-31T00:02:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:21 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:02:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:21 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:02:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:21 smithi136 ceph-mon[20207]: pgmap v2370: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:21.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:21 smithi136 ceph-mon[20207]: osdmap e1077: 12 total, 12 up, 12 in 2024-08-31T00:02:21.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/860375416' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-31T00:02:21.541 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:21 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/822727849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-31T00:02:21.559 INFO:teuthology.orchestra.run.smithi106.stdout:369367187912 2024-08-31T00:02:21.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:21 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:02:21.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:21 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:02:21.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:21 smithi106 ceph-mon[16603]: pgmap v2370: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:21.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:21 smithi106 ceph-mon[16603]: osdmap e1077: 12 total, 12 up, 12 in 2024-08-31T00:02:21.559 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/860375416' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-31T00:02:21.560 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:21 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/822727849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-31T00:02:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:21 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:02:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:21 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:02:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:21 smithi173 ceph-mon[20554]: pgmap v2370: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:21 smithi173 ceph-mon[20554]: osdmap e1077: 12 total, 12 up, 12 in 2024-08-31T00:02:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/860375416' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-31T00:02:21.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:21 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/822727849' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-31T00:02:21.790 INFO:teuthology.orchestra.run.smithi106.stdout:111669150190 2024-08-31T00:02:22.438 INFO:tasks.cephadm.ceph_manager.ceph:need seq 369367187910 got 369367187912 for osd.11 2024-08-31T00:02:22.438 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:22 smithi136 ceph-mon[20207]: 1.15 scrub starts 2024-08-31T00:02:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:22 smithi136 ceph-mon[20207]: 1.15 scrub ok 2024-08-31T00:02:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/967462245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-31T00:02:22.540 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:22 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/618384700' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-31T00:02:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:22 smithi173 ceph-mon[20554]: 1.15 scrub starts 2024-08-31T00:02:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:22 smithi173 ceph-mon[20554]: 1.15 scrub ok 2024-08-31T00:02:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/967462245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-31T00:02:22.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:22 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/618384700' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-31T00:02:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:22 smithi106 ceph-mon[16603]: 1.15 scrub starts 2024-08-31T00:02:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:22 smithi106 ceph-mon[16603]: 1.15 scrub ok 2024-08-31T00:02:22.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/967462245' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-31T00:02:22.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:22 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/618384700' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-31T00:02:22.979 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4462471020558 got 4462471020560 for osd.7 2024-08-31T00:02:22.979 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:23 smithi173 ceph-mon[20554]: 1.1e scrub starts 2024-08-31T00:02:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:23 smithi173 ceph-mon[20554]: 1.1e scrub ok 2024-08-31T00:02:23.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:23 smithi173 ceph-mon[20554]: pgmap v2372: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:23 smithi106 ceph-mon[16603]: 1.1e scrub starts 2024-08-31T00:02:23.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:23 smithi106 ceph-mon[16603]: 1.1e scrub ok 2024-08-31T00:02:23.608 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:23 smithi106 ceph-mon[16603]: pgmap v2372: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:23.715 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528799 got 193273528801 for osd.6 2024-08-31T00:02:23.715 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:23 smithi136 ceph-mon[20207]: 1.1e scrub starts 2024-08-31T00:02:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:23 smithi136 ceph-mon[20207]: 1.1e scrub ok 2024-08-31T00:02:23.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:23 smithi136 ceph-mon[20207]: pgmap v2372: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:24.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449549 got 335007449551 for osd.10 2024-08-31T00:02:24.163 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:25.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150187 got 111669150190 for osd.3 2024-08-31T00:02:25.129 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:25.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:25 smithi106 ceph-mon[16603]: pgmap v2373: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:25.523 INFO:teuthology.orchestra.run.smithi106.stdout:137438953965 2024-08-31T00:02:25.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:25 smithi173 ceph-mon[20554]: pgmap v2373: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:25.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:25 smithi136 ceph-mon[20207]: pgmap v2373: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:26.379 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:26.413 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2787433775301 got 2787433775304 for osd.5 2024-08-31T00:02:26.414 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:26.481 INFO:teuthology.orchestra.run.smithi106.stdout:77309411829 2024-08-31T00:02:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:26 smithi173 ceph-mon[20554]: 1.6 scrub starts 2024-08-31T00:02:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:26 smithi173 ceph-mon[20554]: 1.6 scrub ok 2024-08-31T00:02:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/3408544743' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-31T00:02:26.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:26 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2289173469' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-31T00:02:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:26 smithi106 ceph-mon[16603]: 1.6 scrub starts 2024-08-31T00:02:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:26 smithi106 ceph-mon[16603]: 1.6 scrub ok 2024-08-31T00:02:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/3408544743' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-31T00:02:26.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:26 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2289173469' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-31T00:02:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:26 smithi136 ceph-mon[20207]: 1.6 scrub starts 2024-08-31T00:02:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:26 smithi136 ceph-mon[20207]: 1.6 scrub ok 2024-08-31T00:02:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/3408544743' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-31T00:02:26.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:26 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2289173469' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-31T00:02:27.113 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411825 got 77309411829 for osd.2 2024-08-31T00:02:27.114 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:27.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:27 smithi106 ceph-mon[16603]: pgmap v2374: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:27 smithi106 ceph-mon[16603]: 1.16 scrub starts 2024-08-31T00:02:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:27 smithi106 ceph-mon[16603]: 1.16 scrub ok 2024-08-31T00:02:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:27 smithi106 ceph-mon[16603]: purged_snaps scrub starts 2024-08-31T00:02:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:27 smithi106 ceph-mon[16603]: purged_snaps scrub ok 2024-08-31T00:02:27.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:27 smithi106 ceph-mon[16603]: osdmap e1078: 12 total, 12 up, 12 in 2024-08-31T00:02:27.510 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:27.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738877 got 34359738880 for osd.0 2024-08-31T00:02:27.544 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:27 smithi173 ceph-mon[20554]: pgmap v2374: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:27 smithi173 ceph-mon[20554]: 1.16 scrub starts 2024-08-31T00:02:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:27 smithi173 ceph-mon[20554]: 1.16 scrub ok 2024-08-31T00:02:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:27 smithi173 ceph-mon[20554]: purged_snaps scrub starts 2024-08-31T00:02:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:27 smithi173 ceph-mon[20554]: purged_snaps scrub ok 2024-08-31T00:02:27.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:27 smithi173 ceph-mon[20554]: osdmap e1078: 12 total, 12 up, 12 in 2024-08-31T00:02:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:27 smithi136 ceph-mon[20207]: pgmap v2374: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:27 smithi136 ceph-mon[20207]: 1.16 scrub starts 2024-08-31T00:02:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:27 smithi136 ceph-mon[20207]: 1.16 scrub ok 2024-08-31T00:02:27.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:27 smithi136 ceph-mon[20207]: purged_snaps scrub starts 2024-08-31T00:02:27.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:27 smithi136 ceph-mon[20207]: purged_snaps scrub ok 2024-08-31T00:02:27.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:27 smithi136 ceph-mon[20207]: osdmap e1078: 12 total, 12 up, 12 in 2024-08-31T00:02:27.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953961 got 137438953965 for osd.4 2024-08-31T00:02:27.945 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:28.795 INFO:teuthology.orchestra.run.smithi106.stdout:2254857830647 2024-08-31T00:02:29.014 INFO:teuthology.orchestra.run.smithi106.stdout:266287972826 2024-08-31T00:02:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:29 smithi173 ceph-mon[20554]: pgmap v2376: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/589609524' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-31T00:02:29.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:29 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/693557274' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-31T00:02:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:29 smithi106 ceph-mon[16603]: pgmap v2376: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/589609524' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-31T00:02:29.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:29 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/693557274' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-31T00:02:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:29 smithi136 ceph-mon[20207]: pgmap v2376: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/589609524' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-31T00:02:29.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:29 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/693557274' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-31T00:02:29.882 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2254857830642 got 2254857830647 for osd.1 2024-08-31T00:02:29.882 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:30.445 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972823 got 266287972826 for osd.8 2024-08-31T00:02:30.446 DEBUG:teuthology.parallel:result is None 2024-08-31T00:02:30.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-31T00:02:30.446 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-31T00:02:30.712 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:31.379 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-31T00:02:31.380 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:31 smithi106 ceph-mon[16603]: pgmap v2377: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:31.381 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-31T00:02:31.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:31 smithi173 ceph-mon[20554]: pgmap v2377: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:31.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:31 smithi136 ceph-mon[20207]: pgmap v2377: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:32.120 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2377,"stamp":"2024-08-31T00:02:29.915962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":69,"num_write_kb":1438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2913,"ondisk_log_size":2913,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8317400,"kb_used_data":29832,"kb_used_omap":112,"kb_used_meta":8284687,"kb_avail":1116624424,"statfs":{"total":1151940427776,"available":1143423410176,"internally_reserved":0,"allocated":30547968,"data_stored":20088858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114790,"internal_metadata":8483520410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.639878"},"pg_stats":[{"pgid":"1.14","version":"23'92","reported_seq":1431,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286554+0000","last_change":"2024-08-31T00:02:13.483773+0000","last_active":"2024-08-31T00:02:27.286554+0000","last_peered":"2024-08-31T00:02:27.286554+0000","last_clean":"2024-08-31T00:02:27.286554+0000","last_became_active":"2024-08-31T00:01:16.126186+0000","last_became_peered":"2024-08-31T00:01:16.126186+0000","last_unstale":"2024-08-31T00:02:27.286554+0000","last_undegraded":"2024-08-31T00:02:27.286554+0000","last_fullsized":"2024-08-31T00:02:27.286554+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:13.483718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:13.483718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:36.574657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":202,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.469517+0000","last_change":"2024-08-31T00:02:04.774134+0000","last_active":"2024-08-31T00:02:20.469517+0000","last_peered":"2024-08-31T00:02:20.469517+0000","last_clean":"2024-08-31T00:02:20.469517+0000","last_became_active":"2024-08-30T23:59:14.186459+0000","last_became_peered":"2024-08-30T23:59:14.186459+0000","last_unstale":"2024-08-31T00:02:20.469517+0000","last_undegraded":"2024-08-31T00:02:20.469517+0000","last_fullsized":"2024-08-31T00:02:20.469517+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:04.774069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:04.774069+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:23.425949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1410,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287350+0000","last_change":"2024-08-31T00:02:10.615650+0000","last_active":"2024-08-31T00:02:27.287350+0000","last_peered":"2024-08-31T00:02:27.287350+0000","last_clean":"2024-08-31T00:02:27.287350+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-31T00:02:27.287350+0000","last_undegraded":"2024-08-31T00:02:27.287350+0000","last_fullsized":"2024-08-31T00:02:27.287350+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:10.615582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:10.615582+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:28.414564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1902,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145912+0000","last_change":"2024-08-31T00:01:58.209720+0000","last_active":"2024-08-31T00:02:27.145912+0000","last_peered":"2024-08-31T00:02:27.145912+0000","last_clean":"2024-08-31T00:02:27.145912+0000","last_became_active":"2024-08-31T00:01:58.209318+0000","last_became_peered":"2024-08-31T00:01:58.209318+0000","last_unstale":"2024-08-31T00:02:27.145912+0000","last_undegraded":"2024-08-31T00:02:27.145912+0000","last_fullsized":"2024-08-31T00:02:27.145912+0000","mapping_epoch":1072,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1073,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:36.660608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:36.660608+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:48.143019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":651,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286885+0000","last_change":"2024-08-31T00:01:12.021758+0000","last_active":"2024-08-31T00:02:27.286885+0000","last_peered":"2024-08-31T00:02:27.286885+0000","last_clean":"2024-08-31T00:02:27.286885+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-31T00:02:27.286885+0000","last_undegraded":"2024-08-31T00:02:27.286885+0000","last_fullsized":"2024-08-31T00:02:27.286885+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:12.021699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:12.021699+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:41.434591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1251,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.144965+0000","last_change":"2024-08-31T00:02:05.163306+0000","last_active":"2024-08-31T00:02:27.144965+0000","last_peered":"2024-08-31T00:02:27.144965+0000","last_clean":"2024-08-31T00:02:27.144965+0000","last_became_active":"2024-08-31T00:01:11.823620+0000","last_became_peered":"2024-08-31T00:01:11.823620+0000","last_unstale":"2024-08-31T00:02:27.144965+0000","last_undegraded":"2024-08-31T00:02:27.144965+0000","last_fullsized":"2024-08-31T00:02:27.144965+0000","mapping_epoch":1043,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1044,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:05.163253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:05.163253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:06.578807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":1190,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.277363+0000","last_change":"2024-08-31T00:01:09.585977+0000","last_active":"2024-08-31T00:02:20.277363+0000","last_peered":"2024-08-31T00:02:20.277363+0000","last_clean":"2024-08-31T00:02:20.277363+0000","last_became_active":"2024-08-31T00:01:05.803505+0000","last_became_peered":"2024-08-31T00:01:05.803505+0000","last_unstale":"2024-08-31T00:02:20.277363+0000","last_undegraded":"2024-08-31T00:02:20.277363+0000","last_fullsized":"2024-08-31T00:02:20.277363+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:09.585910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:09.585910+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:36.926732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":710,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145892+0000","last_change":"2024-08-31T00:01:59.142315+0000","last_active":"2024-08-31T00:02:27.145892+0000","last_peered":"2024-08-31T00:02:27.145892+0000","last_clean":"2024-08-31T00:02:27.145892+0000","last_became_active":"2024-08-31T00:01:58.209620+0000","last_became_peered":"2024-08-31T00:01:58.209620+0000","last_unstale":"2024-08-31T00:02:27.145892+0000","last_undegraded":"2024-08-31T00:02:27.145892+0000","last_fullsized":"2024-08-31T00:02:27.145892+0000","mapping_epoch":1072,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1073,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:59.142217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:59.142217+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:14.878010+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"1065'112","reported_seq":1333,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.620642+0000","last_change":"2024-08-31T00:02:17.141848+0000","last_active":"2024-08-31T00:02:27.620642+0000","last_peered":"2024-08-31T00:02:27.620642+0000","last_clean":"2024-08-31T00:02:27.620642+0000","last_became_active":"2024-08-31T00:00:48.962813+0000","last_became_peered":"2024-08-31T00:00:48.962813+0000","last_unstale":"2024-08-31T00:02:27.620642+0000","last_undegraded":"2024-08-31T00:02:27.620642+0000","last_fullsized":"2024-08-31T00:02:27.620642+0000","mapping_epoch":1030,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1031,"parent":"0.0","parent_split_bits":4,"last_scrub":"1065'112","last_scrub_stamp":"2024-08-31T00:02:17.141790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:17.141790+0000","objects_scrubbed":1,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:42.086429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":33,"num_write":28,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":254,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.293732+0000","last_change":"2024-08-31T00:02:11.026313+0000","last_active":"2024-08-31T00:02:20.293732+0000","last_peered":"2024-08-31T00:02:20.293732+0000","last_clean":"2024-08-31T00:02:20.293732+0000","last_became_active":"2024-08-31T00:01:50.166411+0000","last_became_peered":"2024-08-31T00:01:50.166411+0000","last_unstale":"2024-08-31T00:02:20.293732+0000","last_undegraded":"2024-08-31T00:02:20.293732+0000","last_fullsized":"2024-08-31T00:02:20.293732+0000","mapping_epoch":1067,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1068,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:11.026261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:11.026261+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:29.941758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1737,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145910+0000","last_change":"2024-08-31T00:01:58.209911+0000","last_active":"2024-08-31T00:02:27.145910+0000","last_peered":"2024-08-31T00:02:27.145910+0000","last_clean":"2024-08-31T00:02:27.145910+0000","last_became_active":"2024-08-31T00:01:58.209541+0000","last_became_peered":"2024-08-31T00:01:58.209541+0000","last_unstale":"2024-08-31T00:02:27.145910+0000","last_undegraded":"2024-08-31T00:02:27.145910+0000","last_fullsized":"2024-08-31T00:02:27.145910+0000","mapping_epoch":1072,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1073,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:12.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:12.430934+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:40.589591+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":321,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286419+0000","last_change":"2024-08-31T00:01:52.609129+0000","last_active":"2024-08-31T00:02:27.286419+0000","last_peered":"2024-08-31T00:02:27.286419+0000","last_clean":"2024-08-31T00:02:27.286419+0000","last_became_active":"2024-08-31T00:01:16.128896+0000","last_became_peered":"2024-08-31T00:01:16.128896+0000","last_unstale":"2024-08-31T00:02:27.286419+0000","last_undegraded":"2024-08-31T00:02:27.286419+0000","last_fullsized":"2024-08-31T00:02:27.286419+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:52.609052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:52.609052+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:10.095274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,10],"acting":[2,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1475,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286912+0000","last_change":"2024-08-31T00:02:12.584408+0000","last_active":"2024-08-31T00:02:27.286912+0000","last_peered":"2024-08-31T00:02:27.286912+0000","last_clean":"2024-08-31T00:02:27.286912+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-31T00:02:27.286912+0000","last_undegraded":"2024-08-31T00:02:27.286912+0000","last_fullsized":"2024-08-31T00:02:27.286912+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:12.584349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:12.584349+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:17.730204+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":826,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:25.328130+0000","last_change":"2024-08-31T00:02:25.328130+0000","last_active":"2024-08-31T00:02:25.328130+0000","last_peered":"2024-08-31T00:02:25.328130+0000","last_clean":"2024-08-31T00:02:25.328130+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-31T00:02:25.328130+0000","last_undegraded":"2024-08-31T00:02:25.328130+0000","last_fullsized":"2024-08-31T00:02:25.328130+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:25.328066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:25.328066+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:28.176621+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":852,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.278425+0000","last_change":"2024-08-31T00:02:08.356084+0000","last_active":"2024-08-31T00:02:20.278425+0000","last_peered":"2024-08-31T00:02:20.278425+0000","last_clean":"2024-08-31T00:02:20.278425+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-31T00:02:20.278425+0000","last_undegraded":"2024-08-31T00:02:20.278425+0000","last_fullsized":"2024-08-31T00:02:20.278425+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:08.356015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:08.356015+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:36.295152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":501,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.469563+0000","last_change":"2024-08-31T00:02:15.871629+0000","last_active":"2024-08-31T00:02:20.469563+0000","last_peered":"2024-08-31T00:02:20.469563+0000","last_clean":"2024-08-31T00:02:20.469563+0000","last_became_active":"2024-08-31T00:01:06.123520+0000","last_became_peered":"2024-08-31T00:01:06.123520+0000","last_unstale":"2024-08-31T00:02:20.469563+0000","last_undegraded":"2024-08-31T00:02:20.469563+0000","last_fullsized":"2024-08-31T00:02:20.469563+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:15.871562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:15.871562+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:42.563245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":1078,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286620+0000","last_change":"2024-08-31T00:01:40.304766+0000","last_active":"2024-08-31T00:02:27.286620+0000","last_peered":"2024-08-31T00:02:27.286620+0000","last_clean":"2024-08-31T00:02:27.286620+0000","last_became_active":"2024-08-31T00:01:06.121871+0000","last_became_peered":"2024-08-31T00:01:06.121871+0000","last_unstale":"2024-08-31T00:02:27.286620+0000","last_undegraded":"2024-08-31T00:02:27.286620+0000","last_fullsized":"2024-08-31T00:02:27.286620+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:40.304704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:40.304704+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:56.694767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":636,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286899+0000","last_change":"2024-08-31T00:01:07.067872+0000","last_active":"2024-08-31T00:02:27.286899+0000","last_peered":"2024-08-31T00:02:27.286899+0000","last_clean":"2024-08-31T00:02:27.286899+0000","last_became_active":"2024-08-31T00:01:05.803448+0000","last_became_peered":"2024-08-31T00:01:05.803448+0000","last_unstale":"2024-08-31T00:02:27.286899+0000","last_undegraded":"2024-08-31T00:02:27.286899+0000","last_fullsized":"2024-08-31T00:02:27.286899+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:07.067811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:07.067811+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:31.616225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":660,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.144840+0000","last_change":"2024-08-31T00:02:09.252746+0000","last_active":"2024-08-31T00:02:27.144840+0000","last_peered":"2024-08-31T00:02:27.144840+0000","last_clean":"2024-08-31T00:02:27.144840+0000","last_became_active":"2024-08-31T00:01:11.821126+0000","last_became_peered":"2024-08-31T00:01:11.821126+0000","last_unstale":"2024-08-31T00:02:27.144840+0000","last_undegraded":"2024-08-31T00:02:27.144840+0000","last_fullsized":"2024-08-31T00:02:27.144840+0000","mapping_epoch":1043,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1044,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:09.252691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:09.252691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:26.700107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":641,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286740+0000","last_change":"2024-08-31T00:01:13.041420+0000","last_active":"2024-08-31T00:02:27.286740+0000","last_peered":"2024-08-31T00:02:27.286740+0000","last_clean":"2024-08-31T00:02:27.286740+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-31T00:02:27.286740+0000","last_undegraded":"2024-08-31T00:02:27.286740+0000","last_fullsized":"2024-08-31T00:02:27.286740+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:13.041361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:13.041361+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:39.411119+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":84,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.293740+0000","last_change":"2024-08-31T00:01:50.166859+0000","last_active":"2024-08-31T00:02:20.293740+0000","last_peered":"2024-08-31T00:02:20.293740+0000","last_clean":"2024-08-31T00:02:20.293740+0000","last_became_active":"2024-08-31T00:01:50.166462+0000","last_became_peered":"2024-08-31T00:01:50.166462+0000","last_unstale":"2024-08-31T00:02:20.293740+0000","last_undegraded":"2024-08-31T00:02:20.293740+0000","last_fullsized":"2024-08-31T00:02:20.293740+0000","mapping_epoch":1067,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1068,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:18.985020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:18.985020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:39.575916+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":125,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.146050+0000","last_change":"2024-08-31T00:02:20.579730+0000","last_active":"2024-08-31T00:02:27.146050+0000","last_peered":"2024-08-31T00:02:27.146050+0000","last_clean":"2024-08-31T00:02:27.146050+0000","last_became_active":"2024-08-31T00:01:05.803951+0000","last_became_peered":"2024-08-31T00:01:05.803951+0000","last_unstale":"2024-08-31T00:02:27.146050+0000","last_undegraded":"2024-08-31T00:02:27.146050+0000","last_fullsized":"2024-08-31T00:02:27.146050+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:20.579668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:20.579668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:46.720695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":458,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:26.007325+0000","last_change":"2024-08-31T00:02:26.007325+0000","last_active":"2024-08-31T00:02:26.007325+0000","last_peered":"2024-08-31T00:02:26.007325+0000","last_clean":"2024-08-31T00:02:26.007325+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-31T00:02:26.007325+0000","last_undegraded":"2024-08-31T00:02:26.007325+0000","last_fullsized":"2024-08-31T00:02:26.007325+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:26.007274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:26.007274+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:54.700927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":451,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.579972+0000","last_change":"2024-08-31T00:01:21.156065+0000","last_active":"2024-08-31T00:02:20.579972+0000","last_peered":"2024-08-31T00:02:20.579972+0000","last_clean":"2024-08-31T00:02:20.579972+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-31T00:02:20.579972+0000","last_undegraded":"2024-08-31T00:02:20.579972+0000","last_fullsized":"2024-08-31T00:02:20.579972+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:21.156001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:21.156001+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:41.622643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"1065'133","reported_seq":710,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286676+0000","last_change":"2024-08-31T00:01:09.047659+0000","last_active":"2024-08-31T00:02:27.286676+0000","last_peered":"2024-08-31T00:02:27.286676+0000","last_clean":"2024-08-31T00:02:27.286676+0000","last_became_active":"2024-08-31T00:01:05.803494+0000","last_became_peered":"2024-08-31T00:01:05.803494+0000","last_unstale":"2024-08-31T00:02:27.286676+0000","last_undegraded":"2024-08-31T00:02:27.286676+0000","last_fullsized":"2024-08-31T00:02:27.286676+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:09.047579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:09.047579+0000","objects_scrubbed":1,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:30.675181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":41,"num_write_kb":734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":451,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.278576+0000","last_change":"2024-08-31T00:01:57.300797+0000","last_active":"2024-08-31T00:02:20.278576+0000","last_peered":"2024-08-31T00:02:20.278576+0000","last_clean":"2024-08-31T00:02:20.278576+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-31T00:02:20.278576+0000","last_undegraded":"2024-08-31T00:02:20.278576+0000","last_fullsized":"2024-08-31T00:02:20.278576+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:57.300734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:57.300734+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:05.030381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":447,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.277306+0000","last_change":"2024-08-31T00:02:12.532565+0000","last_active":"2024-08-31T00:02:20.277306+0000","last_peered":"2024-08-31T00:02:20.277306+0000","last_clean":"2024-08-31T00:02:20.277306+0000","last_became_active":"2024-08-31T00:01:05.805296+0000","last_became_peered":"2024-08-31T00:01:05.805296+0000","last_unstale":"2024-08-31T00:02:20.277306+0000","last_undegraded":"2024-08-31T00:02:20.277306+0000","last_fullsized":"2024-08-31T00:02:20.277306+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:12.532500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:12.532500+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:27.911645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":458,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.579791+0000","last_change":"2024-08-31T00:01:37.732190+0000","last_active":"2024-08-31T00:02:20.579791+0000","last_peered":"2024-08-31T00:02:20.579791+0000","last_clean":"2024-08-31T00:02:20.579791+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-31T00:02:20.579791+0000","last_undegraded":"2024-08-31T00:02:20.579791+0000","last_fullsized":"2024-08-31T00:02:20.579791+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:37.732119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:37.732119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:43.532665+0000","scrub_duration":522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":458,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.579836+0000","last_change":"2024-08-31T00:02:05.241394+0000","last_active":"2024-08-31T00:02:20.579836+0000","last_peered":"2024-08-31T00:02:20.579836+0000","last_clean":"2024-08-31T00:02:20.579836+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-31T00:02:20.579836+0000","last_undegraded":"2024-08-31T00:02:20.579836+0000","last_fullsized":"2024-08-31T00:02:20.579836+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:05.241327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:05.241327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:34.143688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":91,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.293760+0000","last_change":"2024-08-31T00:01:51.046833+0000","last_active":"2024-08-31T00:02:20.293760+0000","last_peered":"2024-08-31T00:02:20.293760+0000","last_clean":"2024-08-31T00:02:20.293760+0000","last_became_active":"2024-08-31T00:01:50.165444+0000","last_became_peered":"2024-08-31T00:01:50.165444+0000","last_unstale":"2024-08-31T00:02:20.293760+0000","last_undegraded":"2024-08-31T00:02:20.293760+0000","last_fullsized":"2024-08-31T00:02:20.293760+0000","mapping_epoch":1067,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1068,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:51.046785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:51.046785+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:57.262221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":430,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:21.792384+0000","last_change":"2024-08-31T00:02:21.792384+0000","last_active":"2024-08-31T00:02:21.792384+0000","last_peered":"2024-08-31T00:02:21.792384+0000","last_clean":"2024-08-31T00:02:21.792384+0000","last_became_active":"2024-08-31T00:01:06.121875+0000","last_became_peered":"2024-08-31T00:01:06.121875+0000","last_unstale":"2024-08-31T00:02:21.792384+0000","last_undegraded":"2024-08-31T00:02:21.792384+0000","last_fullsized":"2024-08-31T00:02:21.792384+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:21.792313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:21.792313+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:47.361345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":69,"num_write_kb":1438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2913,"ondisk_log_size":2913,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187914,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":746184,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92998968,"statfs":{"total":95995035648,"available":95230943232,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":761199044},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773592,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92971560,"statfs":{"total":95995035648,"available":95202877440,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":789641664},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899944,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92845208,"statfs":{"total":95995035648,"available":95073492992,"internally_reserved":0,"allocated":2326528,"data_stored":1486630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":919009728},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579912,"kb_used_data":2760,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93165240,"statfs":{"total":95995035648,"available":95401205760,"internally_reserved":0,"allocated":2826240,"data_stored":1924729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1039,"seq":4462471020563,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78704,"kb_used_data":2472,"kb_used_omap":4,"kb_used_meta":76027,"kb_avail":93666448,"statfs":{"total":95995035648,"available":95914442752,"internally_reserved":0,"allocated":2531328,"data_stored":1687878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":77852364},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553224,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93191928,"statfs":{"total":95995035648,"available":95428534272,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":563608006},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":525872,"kb_used_data":2792,"kb_used_omap":5,"kb_used_meta":522874,"kb_avail":93219280,"statfs":{"total":95995035648,"available":95456542720,"internally_reserved":0,"allocated":2859008,"data_stored":2015558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":535423050},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953966,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773960,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92971192,"statfs":{"total":95995035648,"available":95202500608,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150192,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022440,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":92722712,"statfs":{"total":95995035648,"available":94948057088,"internally_reserved":0,"allocated":2654208,"data_stored":1814310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1044117959},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411830,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697456,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93047696,"statfs":{"total":95995035648,"available":95280840704,"internally_reserved":0,"allocated":2531328,"data_stored":1687878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830647,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":743960,"kb_used_data":2440,"kb_used_omap":87,"kb_used_meta":741416,"kb_avail":93001192,"statfs":{"total":95995035648,"available":95233220608,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":759210143},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":922152,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92823000,"statfs":{"total":95995035648,"available":95050752000,"internally_reserved":0,"allocated":2326528,"data_stored":1486630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T00:02:32.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-08-31T00:02:32.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:32 smithi106 ceph-mon[16603]: from='client.19917 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:02:32.365 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:32.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:32 smithi173 ceph-mon[20554]: from='client.19917 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:02:32.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:32 smithi136 ceph-mon[20207]: from='client.19917 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:02:32.999 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-08-31T00:02:33.000 INFO:teuthology.orchestra.run.smithi106.stderr:dumped all 2024-08-31T00:02:33.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:33 smithi173 ceph-mon[20554]: pgmap v2378: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:33.607 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:33 smithi106 ceph-mon[16603]: pgmap v2378: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:33.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:33 smithi136 ceph-mon[20207]: pgmap v2378: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:33.822 INFO:teuthology.orchestra.run.smithi106.stdout:{"pg_ready":true,"pg_map":{"version":2378,"stamp":"2024-08-31T00:02:31.916466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":69,"num_write_kb":1438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2913,"ondisk_log_size":2913,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8316344,"kb_used_data":29328,"kb_used_omap":112,"kb_used_meta":8284687,"kb_avail":1116625480,"statfs":{"total":1151940427776,"available":1143424491520,"internally_reserved":0,"allocated":30031872,"data_stored":19757601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":114790,"internal_metadata":8483520410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002144"},"pg_stats":[{"pgid":"1.14","version":"23'92","reported_seq":1431,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286554+0000","last_change":"2024-08-31T00:02:13.483773+0000","last_active":"2024-08-31T00:02:27.286554+0000","last_peered":"2024-08-31T00:02:27.286554+0000","last_clean":"2024-08-31T00:02:27.286554+0000","last_became_active":"2024-08-31T00:01:16.126186+0000","last_became_peered":"2024-08-31T00:01:16.126186+0000","last_unstale":"2024-08-31T00:02:27.286554+0000","last_undegraded":"2024-08-31T00:02:27.286554+0000","last_fullsized":"2024-08-31T00:02:27.286554+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:13.483718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:13.483718+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:36.574657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"23'92","reported_seq":204,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287281+0000","last_change":"2024-08-31T00:02:04.774134+0000","last_active":"2024-08-31T00:02:27.287281+0000","last_peered":"2024-08-31T00:02:27.287281+0000","last_clean":"2024-08-31T00:02:27.287281+0000","last_became_active":"2024-08-30T23:59:14.186459+0000","last_became_peered":"2024-08-30T23:59:14.186459+0000","last_unstale":"2024-08-31T00:02:27.287281+0000","last_undegraded":"2024-08-31T00:02:27.287281+0000","last_fullsized":"2024-08-31T00:02:27.287281+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:04.774069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:04.774069+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:23.425949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"23'92","reported_seq":1410,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287350+0000","last_change":"2024-08-31T00:02:10.615650+0000","last_active":"2024-08-31T00:02:27.287350+0000","last_peered":"2024-08-31T00:02:27.287350+0000","last_clean":"2024-08-31T00:02:27.287350+0000","last_became_active":"2024-08-30T23:55:01.286283+0000","last_became_peered":"2024-08-30T23:55:01.286283+0000","last_unstale":"2024-08-31T00:02:27.287350+0000","last_undegraded":"2024-08-31T00:02:27.287350+0000","last_fullsized":"2024-08-31T00:02:27.287350+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:10.615582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:10.615582+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:28.414564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.9","version":"23'92","reported_seq":1902,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145912+0000","last_change":"2024-08-31T00:01:58.209720+0000","last_active":"2024-08-31T00:02:27.145912+0000","last_peered":"2024-08-31T00:02:27.145912+0000","last_clean":"2024-08-31T00:02:27.145912+0000","last_became_active":"2024-08-31T00:01:58.209318+0000","last_became_peered":"2024-08-31T00:01:58.209318+0000","last_unstale":"2024-08-31T00:02:27.145912+0000","last_undegraded":"2024-08-31T00:02:27.145912+0000","last_fullsized":"2024-08-31T00:02:27.145912+0000","mapping_epoch":1072,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1073,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:36.660608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:36.660608+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:48.143019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"23'92","reported_seq":651,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286885+0000","last_change":"2024-08-31T00:01:12.021758+0000","last_active":"2024-08-31T00:02:27.286885+0000","last_peered":"2024-08-31T00:02:27.286885+0000","last_clean":"2024-08-31T00:02:27.286885+0000","last_became_active":"2024-08-30T23:55:01.288971+0000","last_became_peered":"2024-08-30T23:55:01.288971+0000","last_unstale":"2024-08-31T00:02:27.286885+0000","last_undegraded":"2024-08-31T00:02:27.286885+0000","last_fullsized":"2024-08-31T00:02:27.286885+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:12.021699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:12.021699+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:41.434591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"23'92","reported_seq":1251,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.144965+0000","last_change":"2024-08-31T00:02:05.163306+0000","last_active":"2024-08-31T00:02:27.144965+0000","last_peered":"2024-08-31T00:02:27.144965+0000","last_clean":"2024-08-31T00:02:27.144965+0000","last_became_active":"2024-08-31T00:01:11.823620+0000","last_became_peered":"2024-08-31T00:01:11.823620+0000","last_unstale":"2024-08-31T00:02:27.144965+0000","last_undegraded":"2024-08-31T00:02:27.144965+0000","last_fullsized":"2024-08-31T00:02:27.144965+0000","mapping_epoch":1043,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1044,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:05.163253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:05.163253+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:06.578807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"23'92","reported_seq":1192,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145670+0000","last_change":"2024-08-31T00:01:09.585977+0000","last_active":"2024-08-31T00:02:27.145670+0000","last_peered":"2024-08-31T00:02:27.145670+0000","last_clean":"2024-08-31T00:02:27.145670+0000","last_became_active":"2024-08-31T00:01:05.803505+0000","last_became_peered":"2024-08-31T00:01:05.803505+0000","last_unstale":"2024-08-31T00:02:27.145670+0000","last_undegraded":"2024-08-31T00:02:27.145670+0000","last_fullsized":"2024-08-31T00:02:27.145670+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:09.585910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:09.585910+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:36.926732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"23'92","reported_seq":710,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145892+0000","last_change":"2024-08-31T00:01:59.142315+0000","last_active":"2024-08-31T00:02:27.145892+0000","last_peered":"2024-08-31T00:02:27.145892+0000","last_clean":"2024-08-31T00:02:27.145892+0000","last_became_active":"2024-08-31T00:01:58.209620+0000","last_became_peered":"2024-08-31T00:01:58.209620+0000","last_unstale":"2024-08-31T00:02:27.145892+0000","last_undegraded":"2024-08-31T00:02:27.145892+0000","last_fullsized":"2024-08-31T00:02:27.145892+0000","mapping_epoch":1072,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1073,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:59.142217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:59.142217+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:14.878010+0000","scrub_duration":787,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"1065'112","reported_seq":1333,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.620642+0000","last_change":"2024-08-31T00:02:17.141848+0000","last_active":"2024-08-31T00:02:27.620642+0000","last_peered":"2024-08-31T00:02:27.620642+0000","last_clean":"2024-08-31T00:02:27.620642+0000","last_became_active":"2024-08-31T00:00:48.962813+0000","last_became_peered":"2024-08-31T00:00:48.962813+0000","last_unstale":"2024-08-31T00:02:27.620642+0000","last_undegraded":"2024-08-31T00:02:27.620642+0000","last_fullsized":"2024-08-31T00:02:27.620642+0000","mapping_epoch":1030,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1031,"parent":"0.0","parent_split_bits":4,"last_scrub":"1065'112","last_scrub_stamp":"2024-08-31T00:02:17.141790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:17.141790+0000","objects_scrubbed":1,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:42.086429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":33,"num_write":28,"num_write_kb":704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,8],"acting":[3,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"23'92","reported_seq":254,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.293732+0000","last_change":"2024-08-31T00:02:11.026313+0000","last_active":"2024-08-31T00:02:20.293732+0000","last_peered":"2024-08-31T00:02:20.293732+0000","last_clean":"2024-08-31T00:02:20.293732+0000","last_became_active":"2024-08-31T00:01:50.166411+0000","last_became_peered":"2024-08-31T00:01:50.166411+0000","last_unstale":"2024-08-31T00:02:20.293732+0000","last_undegraded":"2024-08-31T00:02:20.293732+0000","last_fullsized":"2024-08-31T00:02:20.293732+0000","mapping_epoch":1067,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1068,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:11.026261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:11.026261+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:29.941758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"23'92","reported_seq":1737,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145910+0000","last_change":"2024-08-31T00:01:58.209911+0000","last_active":"2024-08-31T00:02:27.145910+0000","last_peered":"2024-08-31T00:02:27.145910+0000","last_clean":"2024-08-31T00:02:27.145910+0000","last_became_active":"2024-08-31T00:01:58.209541+0000","last_became_peered":"2024-08-31T00:01:58.209541+0000","last_unstale":"2024-08-31T00:02:27.145910+0000","last_undegraded":"2024-08-31T00:02:27.145910+0000","last_fullsized":"2024-08-31T00:02:27.145910+0000","mapping_epoch":1072,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1073,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:12.430934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:12.430934+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:40.589591+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"23'92","reported_seq":321,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286419+0000","last_change":"2024-08-31T00:01:52.609129+0000","last_active":"2024-08-31T00:02:27.286419+0000","last_peered":"2024-08-31T00:02:27.286419+0000","last_clean":"2024-08-31T00:02:27.286419+0000","last_became_active":"2024-08-31T00:01:16.128896+0000","last_became_peered":"2024-08-31T00:01:16.128896+0000","last_unstale":"2024-08-31T00:02:27.286419+0000","last_undegraded":"2024-08-31T00:02:27.286419+0000","last_fullsized":"2024-08-31T00:02:27.286419+0000","mapping_epoch":1046,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1047,"parent":"0.0","parent_split_bits":4,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:52.609052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:52.609052+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:10.095274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,10],"acting":[2,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"23'92","reported_seq":1475,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286912+0000","last_change":"2024-08-31T00:02:12.584408+0000","last_active":"2024-08-31T00:02:27.286912+0000","last_peered":"2024-08-31T00:02:27.286912+0000","last_clean":"2024-08-31T00:02:27.286912+0000","last_became_active":"2024-08-30T23:55:01.288877+0000","last_became_peered":"2024-08-30T23:55:01.288877+0000","last_unstale":"2024-08-31T00:02:27.286912+0000","last_undegraded":"2024-08-31T00:02:27.286912+0000","last_fullsized":"2024-08-31T00:02:27.286912+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:12.584349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:12.584349+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:17.730204+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"23'92","reported_seq":826,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:25.328130+0000","last_change":"2024-08-31T00:02:25.328130+0000","last_active":"2024-08-31T00:02:25.328130+0000","last_peered":"2024-08-31T00:02:25.328130+0000","last_clean":"2024-08-31T00:02:25.328130+0000","last_became_active":"2024-08-30T23:55:01.287255+0000","last_became_peered":"2024-08-30T23:55:01.287255+0000","last_unstale":"2024-08-31T00:02:25.328130+0000","last_undegraded":"2024-08-31T00:02:25.328130+0000","last_fullsized":"2024-08-31T00:02:25.328130+0000","mapping_epoch":710,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:25.328066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:25.328066+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:28.176621+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.5","version":"23'92","reported_seq":852,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.278425+0000","last_change":"2024-08-31T00:02:08.356084+0000","last_active":"2024-08-31T00:02:20.278425+0000","last_peered":"2024-08-31T00:02:20.278425+0000","last_clean":"2024-08-31T00:02:20.278425+0000","last_became_active":"2024-08-30T23:55:01.288622+0000","last_became_peered":"2024-08-30T23:55:01.288622+0000","last_unstale":"2024-08-31T00:02:20.278425+0000","last_undegraded":"2024-08-31T00:02:20.278425+0000","last_fullsized":"2024-08-31T00:02:20.278425+0000","mapping_epoch":799,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:08.356015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:08.356015+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:36.295152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"23'92","reported_seq":503,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287278+0000","last_change":"2024-08-31T00:02:15.871629+0000","last_active":"2024-08-31T00:02:27.287278+0000","last_peered":"2024-08-31T00:02:27.287278+0000","last_clean":"2024-08-31T00:02:27.287278+0000","last_became_active":"2024-08-31T00:01:06.123520+0000","last_became_peered":"2024-08-31T00:01:06.123520+0000","last_unstale":"2024-08-31T00:02:27.287278+0000","last_undegraded":"2024-08-31T00:02:27.287278+0000","last_fullsized":"2024-08-31T00:02:27.287278+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":3,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:15.871562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:15.871562+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:42.563245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"23'92","reported_seq":1078,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286620+0000","last_change":"2024-08-31T00:01:40.304766+0000","last_active":"2024-08-31T00:02:27.286620+0000","last_peered":"2024-08-31T00:02:27.286620+0000","last_clean":"2024-08-31T00:02:27.286620+0000","last_became_active":"2024-08-31T00:01:06.121871+0000","last_became_peered":"2024-08-31T00:01:06.121871+0000","last_unstale":"2024-08-31T00:02:27.286620+0000","last_undegraded":"2024-08-31T00:02:27.286620+0000","last_fullsized":"2024-08-31T00:02:27.286620+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:40.304704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:40.304704+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:56.694767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"23'92","reported_seq":636,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286899+0000","last_change":"2024-08-31T00:01:07.067872+0000","last_active":"2024-08-31T00:02:27.286899+0000","last_peered":"2024-08-31T00:02:27.286899+0000","last_clean":"2024-08-31T00:02:27.286899+0000","last_became_active":"2024-08-31T00:01:05.803448+0000","last_became_peered":"2024-08-31T00:01:05.803448+0000","last_unstale":"2024-08-31T00:02:27.286899+0000","last_undegraded":"2024-08-31T00:02:27.286899+0000","last_fullsized":"2024-08-31T00:02:27.286899+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:07.067811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:07.067811+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:31.616225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"23'92","reported_seq":660,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.144840+0000","last_change":"2024-08-31T00:02:09.252746+0000","last_active":"2024-08-31T00:02:27.144840+0000","last_peered":"2024-08-31T00:02:27.144840+0000","last_clean":"2024-08-31T00:02:27.144840+0000","last_became_active":"2024-08-31T00:01:11.821126+0000","last_became_peered":"2024-08-31T00:01:11.821126+0000","last_unstale":"2024-08-31T00:02:27.144840+0000","last_undegraded":"2024-08-31T00:02:27.144840+0000","last_fullsized":"2024-08-31T00:02:27.144840+0000","mapping_epoch":1043,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1044,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:09.252691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:09.252691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:26.700107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"23'92","reported_seq":641,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286740+0000","last_change":"2024-08-31T00:01:13.041420+0000","last_active":"2024-08-31T00:02:27.286740+0000","last_peered":"2024-08-31T00:02:27.286740+0000","last_clean":"2024-08-31T00:02:27.286740+0000","last_became_active":"2024-08-30T23:51:42.950278+0000","last_became_peered":"2024-08-30T23:51:42.950278+0000","last_unstale":"2024-08-31T00:02:27.286740+0000","last_undegraded":"2024-08-31T00:02:27.286740+0000","last_fullsized":"2024-08-31T00:02:27.286740+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:13.041361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:13.041361+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:39.411119+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"23'92","reported_seq":84,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.293740+0000","last_change":"2024-08-31T00:01:50.166859+0000","last_active":"2024-08-31T00:02:20.293740+0000","last_peered":"2024-08-31T00:02:20.293740+0000","last_clean":"2024-08-31T00:02:20.293740+0000","last_became_active":"2024-08-31T00:01:50.166462+0000","last_became_peered":"2024-08-31T00:01:50.166462+0000","last_unstale":"2024-08-31T00:02:20.293740+0000","last_undegraded":"2024-08-31T00:02:20.293740+0000","last_fullsized":"2024-08-31T00:02:20.293740+0000","mapping_epoch":1067,"log_start":"0'0","ondisk_log_start":"0'0","created":507,"last_epoch_clean":1068,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:18.985020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:18.985020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:39.575916+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"23'92","reported_seq":125,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.146050+0000","last_change":"2024-08-31T00:02:20.579730+0000","last_active":"2024-08-31T00:02:27.146050+0000","last_peered":"2024-08-31T00:02:27.146050+0000","last_clean":"2024-08-31T00:02:27.146050+0000","last_became_active":"2024-08-31T00:01:05.803951+0000","last_became_peered":"2024-08-31T00:01:05.803951+0000","last_unstale":"2024-08-31T00:02:27.146050+0000","last_undegraded":"2024-08-31T00:02:27.146050+0000","last_fullsized":"2024-08-31T00:02:27.146050+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:20.579668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:20.579668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:46.720695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.16","version":"23'92","reported_seq":458,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:26.007325+0000","last_change":"2024-08-31T00:02:26.007325+0000","last_active":"2024-08-31T00:02:26.007325+0000","last_peered":"2024-08-31T00:02:26.007325+0000","last_clean":"2024-08-31T00:02:26.007325+0000","last_became_active":"2024-08-30T23:55:03.314093+0000","last_became_peered":"2024-08-30T23:55:03.314093+0000","last_unstale":"2024-08-31T00:02:26.007325+0000","last_undegraded":"2024-08-31T00:02:26.007325+0000","last_fullsized":"2024-08-31T00:02:26.007325+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:26.007274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:26.007274+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:54.700927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"23'92","reported_seq":453,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287706+0000","last_change":"2024-08-31T00:01:21.156065+0000","last_active":"2024-08-31T00:02:27.287706+0000","last_peered":"2024-08-31T00:02:27.287706+0000","last_clean":"2024-08-31T00:02:27.287706+0000","last_became_active":"2024-08-30T23:55:03.319161+0000","last_became_peered":"2024-08-30T23:55:03.319161+0000","last_unstale":"2024-08-31T00:02:27.287706+0000","last_undegraded":"2024-08-31T00:02:27.287706+0000","last_fullsized":"2024-08-31T00:02:27.287706+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:21.156001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:21.156001+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:41.622643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.18","version":"1065'133","reported_seq":710,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.286676+0000","last_change":"2024-08-31T00:01:09.047659+0000","last_active":"2024-08-31T00:02:27.286676+0000","last_peered":"2024-08-31T00:02:27.286676+0000","last_clean":"2024-08-31T00:02:27.286676+0000","last_became_active":"2024-08-31T00:01:05.803494+0000","last_became_peered":"2024-08-31T00:01:05.803494+0000","last_unstale":"2024-08-31T00:02:27.286676+0000","last_undegraded":"2024-08-31T00:02:27.286676+0000","last_fullsized":"2024-08-31T00:02:27.286676+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:09.047579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:09.047579+0000","objects_scrubbed":1,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:30.675181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":41,"num_write_kb":734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.19","version":"23'92","reported_seq":451,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.278576+0000","last_change":"2024-08-31T00:01:57.300797+0000","last_active":"2024-08-31T00:02:20.278576+0000","last_peered":"2024-08-31T00:02:20.278576+0000","last_clean":"2024-08-31T00:02:20.278576+0000","last_became_active":"2024-08-30T23:55:03.317626+0000","last_became_peered":"2024-08-30T23:55:03.317626+0000","last_unstale":"2024-08-31T00:02:20.278576+0000","last_undegraded":"2024-08-31T00:02:20.278576+0000","last_fullsized":"2024-08-31T00:02:20.278576+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:57.300734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:57.300734+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:05.030381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1a","version":"23'92","reported_seq":449,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.145651+0000","last_change":"2024-08-31T00:02:12.532565+0000","last_active":"2024-08-31T00:02:27.145651+0000","last_peered":"2024-08-31T00:02:27.145651+0000","last_clean":"2024-08-31T00:02:27.145651+0000","last_became_active":"2024-08-31T00:01:05.805296+0000","last_became_peered":"2024-08-31T00:01:05.805296+0000","last_unstale":"2024-08-31T00:02:27.145651+0000","last_undegraded":"2024-08-31T00:02:27.145651+0000","last_fullsized":"2024-08-31T00:02:27.145651+0000","mapping_epoch":1039,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1040,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:12.532500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:12.532500+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:27.911645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1b","version":"23'92","reported_seq":460,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287729+0000","last_change":"2024-08-31T00:01:37.732190+0000","last_active":"2024-08-31T00:02:27.287729+0000","last_peered":"2024-08-31T00:02:27.287729+0000","last_clean":"2024-08-31T00:02:27.287729+0000","last_became_active":"2024-08-30T23:55:03.317287+0000","last_became_peered":"2024-08-30T23:55:03.317287+0000","last_unstale":"2024-08-31T00:02:27.287729+0000","last_undegraded":"2024-08-31T00:02:27.287729+0000","last_fullsized":"2024-08-31T00:02:27.287729+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:37.732119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:37.732119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:43.532665+0000","scrub_duration":522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1c","version":"23'92","reported_seq":460,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287751+0000","last_change":"2024-08-31T00:02:05.241394+0000","last_active":"2024-08-31T00:02:27.287751+0000","last_peered":"2024-08-31T00:02:27.287751+0000","last_clean":"2024-08-31T00:02:27.287751+0000","last_became_active":"2024-08-30T23:55:03.309042+0000","last_became_peered":"2024-08-30T23:55:03.309042+0000","last_unstale":"2024-08-31T00:02:27.287751+0000","last_undegraded":"2024-08-31T00:02:27.287751+0000","last_fullsized":"2024-08-31T00:02:27.287751+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:05.241327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:05.241327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:34.143688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"23'92","reported_seq":91,"reported_epoch":1077,"state":"active+clean","last_fresh":"2024-08-31T00:02:20.293760+0000","last_change":"2024-08-31T00:01:51.046833+0000","last_active":"2024-08-31T00:02:20.293760+0000","last_peered":"2024-08-31T00:02:20.293760+0000","last_clean":"2024-08-31T00:02:20.293760+0000","last_became_active":"2024-08-31T00:01:50.165444+0000","last_became_peered":"2024-08-31T00:01:50.165444+0000","last_unstale":"2024-08-31T00:02:20.293760+0000","last_undegraded":"2024-08-31T00:02:20.293760+0000","last_fullsized":"2024-08-31T00:02:20.293760+0000","mapping_epoch":1067,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1068,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:01:51.046785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:01:51.046785+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:02:57.262221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"23'92","reported_seq":432,"reported_epoch":1078,"state":"active+clean","last_fresh":"2024-08-31T00:02:27.287449+0000","last_change":"2024-08-31T00:02:21.792384+0000","last_active":"2024-08-31T00:02:27.287449+0000","last_peered":"2024-08-31T00:02:27.287449+0000","last_clean":"2024-08-31T00:02:27.287449+0000","last_became_active":"2024-08-31T00:01:06.121875+0000","last_became_peered":"2024-08-31T00:01:06.121875+0000","last_unstale":"2024-08-31T00:02:27.287449+0000","last_undegraded":"2024-08-31T00:02:27.287449+0000","last_fullsized":"2024-08-31T00:02:27.287449+0000","mapping_epoch":1040,"log_start":"0'0","ondisk_log_start":"0'0","created":877,"last_epoch_clean":1041,"parent":"0.0","parent_split_bits":5,"last_scrub":"23'92","last_scrub_stamp":"2024-08-31T00:02:21.792313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-30T23:21:10.293975+0000","last_clean_scrub_stamp":"2024-08-31T00:02:21.792313+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-31T00:03:47.361345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":69,"num_write_kb":1438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2913,"ondisk_log_size":2913,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":86,"seq":369367187915,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745832,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92999320,"statfs":{"total":95995035648,"available":95231303680,"internally_reserved":0,"allocated":2326528,"data_stored":1486630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":761199044},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":78,"seq":335007449554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773592,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92971560,"statfs":{"total":95995035648,"available":95202877440,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":789641664},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":70,"seq":300647711184,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899944,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92845208,"statfs":{"total":95995035648,"available":95073492992,"internally_reserved":0,"allocated":2326528,"data_stored":1486630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":919009728},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579560,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93165592,"statfs":{"total":95995035648,"available":95401566208,"internally_reserved":0,"allocated":2654208,"data_stored":1814310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":1039,"seq":4462471020563,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78704,"kb_used_data":2472,"kb_used_omap":4,"kb_used_meta":76027,"kb_avail":93666448,"statfs":{"total":95995035648,"available":95914442752,"internally_reserved":0,"allocated":2531328,"data_stored":1687878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4404,"internal_metadata":77852364},"hb_peers":[0,1,2,3,4,5,6,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":553224,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93191928,"statfs":{"total":95995035648,"available":95428534272,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":563608006},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":649,"seq":2787433775307,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":525872,"kb_used_data":2792,"kb_used_omap":5,"kb_used_meta":522874,"kb_avail":93219280,"statfs":{"total":95995035648,"available":95456542720,"internally_reserved":0,"allocated":2859008,"data_stored":2015558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6070,"internal_metadata":535423050},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953967,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773608,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92971544,"statfs":{"total":95995035648,"available":95202861056,"internally_reserved":0,"allocated":2326528,"data_stored":1486630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150192,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022440,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":92722712,"statfs":{"total":95995035648,"available":94948057088,"internally_reserved":0,"allocated":2654208,"data_stored":1814310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1044117959},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411830,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697456,"kb_used_data":2472,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93047696,"statfs":{"total":95995035648,"available":95280840704,"internally_reserved":0,"allocated":2531328,"data_stored":1687878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":525,"seq":2254857830647,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":743960,"kb_used_data":2440,"kb_used_omap":87,"kb_used_meta":741416,"kb_avail":93001192,"statfs":{"total":95995035648,"available":95233220608,"internally_reserved":0,"allocated":2498560,"data_stored":1597049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":89953,"internal_metadata":759210143},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738882,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":922152,"kb_used_data":2272,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92823000,"statfs":{"total":95995035648,"available":95050752000,"internally_reserved":0,"allocated":2326528,"data_stored":1486630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-31T00:02:33.823 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-31T00:02:33.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec shell --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-08-31T00:02:34.068 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/mon.a/config 2024-08-31T00:02:34.469 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:34 smithi106 ceph-mon[16603]: from='client.19923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:02:34.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:34 smithi173 ceph-mon[20554]: from='client.19923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:02:34.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:34 smithi136 ceph-mon[20207]: from='client.19923 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-31T00:02:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: pgmap v2379: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:35.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2225049452' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-08-31T00:02:35.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: from='client.? 172.21.15.106:0/2225049452' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-08-31T00:02:35.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:35.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:35.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:35.358 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 ceph-mon[16603]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:35.487 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-31T00:02:35.507 INFO:tasks.cephadm:Teardown begin 2024-08-31T00:02:35.508 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-31T00:02:35.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-31T00:02:35.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: pgmap v2379: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2225049452' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: from='client.? 172.21.15.106:0/2225049452' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:35.574 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:35 smithi173 ceph-mon[20554]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:35.619 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-31T00:02:35.619 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-31T00:02:35.646 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-31T00:02:35.676 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-31T00:02:35.743 INFO:tasks.cephadm:Stopping all daemons... 2024-08-31T00:02:35.743 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-31T00:02:35.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: pgmap v2379: 31 pgs: 31 active+clean; 577 KiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2225049452' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: from='client.? 172.21.15.106:0/2225049452' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-31T00:02:35.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:35 smithi136 ceph-mon[20207]: from='mgr.14151 172.21.15.106:0/3419632988' entity='mgr.y' 2024-08-31T00:02:36.062 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:35 smithi106 systemd[1]: Stopping Ceph mon.a for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:36 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-31T00:02:36.060+0000 7f04cff01640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:02:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:36 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a[16599]: 2024-08-31T00:02:36.061+0000 7f04cff01640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-08-31T00:02:36.357 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:36 smithi106 podman[286617]: 2024-08-31 00:02:36.194016211 +0000 UTC m=+0.239758020 container died dc9bafd601de2e62d8690b245c3158e7348976e75b83dd7f4346ef1c9682602e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:02:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:36 smithi106 podman[286617]: 2024-08-31 00:02:36.762142544 +0000 UTC m=+0.807884353 container cleanup dc9bafd601de2e62d8690b245c3158e7348976e75b83dd7f4346ef1c9682602e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-08-31T00:02:36.857 INFO:journalctl@ceph.mon.a.smithi106.stdout:Aug 31 00:02:36 smithi106 bash[286617]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-a 2024-08-31T00:02:37.191 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.a.service' 2024-08-31T00:02:37.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:37.229 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-08-31T00:02:37.229 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-08-31T00:02:37.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.b 2024-08-31T00:02:37.534 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:37 smithi136 systemd[1]: Stopping Ceph mon.b for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:37 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-b[20203]: 2024-08-31T00:02:37.533+0000 7fb8a40e3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:02:37.790 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:37 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-b[20203]: 2024-08-31T00:02:37.533+0000 7fb8a40e3640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-08-31T00:02:37.791 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:37 smithi136 podman[172324]: 2024-08-31 00:02:37.616308617 +0000 UTC m=+0.192389280 container died ad6978186c00d2d833e60b4073232dca6f8a114d6400b6736902ad18bdc8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-08-31T00:02:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:38 smithi136 podman[172324]: 2024-08-31 00:02:38.159831119 +0000 UTC m=+0.735911824 container cleanup ad6978186c00d2d833e60b4073232dca6f8a114d6400b6736902ad18bdc8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-31T00:02:38.290 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:38 smithi136 bash[172324]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-b 2024-08-31T00:02:38.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.b.service' 2024-08-31T00:02:38.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:38 smithi136 podman[172337]: 2024-08-31 00:02:38.344888398 +0000 UTC m=+0.721638550 container remove ad6978186c00d2d833e60b4073232dca6f8a114d6400b6736902ad18bdc8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-31T00:02:38.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:38 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.b.service: Deactivated successfully. 2024-08-31T00:02:38.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:38 smithi136 systemd[1]: Stopped Ceph mon.b for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-31T00:02:38.626 INFO:journalctl@ceph.mon.b.smithi136.stdout:Aug 31 00:02:38 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.b.service: Consumed 31.940s CPU time. 2024-08-31T00:02:38.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:38.641 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-08-31T00:02:38.641 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-08-31T00:02:38.641 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.c 2024-08-31T00:02:38.998 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:38 smithi173 systemd[1]: Stopping Ceph mon.c for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:39.324 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-c[20549]: 2024-08-31T00:02:38.997+0000 7f683cb13640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:02:39.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-c[20549]: 2024-08-31T00:02:38.997+0000 7f683cb13640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-08-31T00:02:39.325 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:39 smithi173 podman[172177]: 2024-08-31 00:02:39.212029148 +0000 UTC m=+0.333225897 container died b491d30aec041063a9307243dcc21d00de94850ff2457576d17687258ef5fa16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-08-31T00:02:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:39 smithi173 podman[172177]: 2024-08-31 00:02:39.698166032 +0000 UTC m=+0.819362824 container cleanup b491d30aec041063a9307243dcc21d00de94850ff2457576d17687258ef5fa16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-31T00:02:39.824 INFO:journalctl@ceph.mon.c.smithi173.stdout:Aug 31 00:02:39 smithi173 bash[172177]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mon-c 2024-08-31T00:02:40.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mon.c.service' 2024-08-31T00:02:40.227 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:40.228 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-08-31T00:02:40.228 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-08-31T00:02:40.228 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y 2024-08-31T00:02:40.554 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 31 00:02:40 smithi106 systemd[1]: Stopping Ceph mgr.y for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:40.857 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 31 00:02:40 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-31T00:02:40.553+0000 7f22a1026640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:02:40.857 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 31 00:02:40 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y[16831]: 2024-08-31T00:02:40.553+0000 7f22a1026640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-08-31T00:02:40.858 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 31 00:02:40 smithi106 podman[286742]: 2024-08-31 00:02:40.585683088 +0000 UTC m=+0.154973712 container died 7eb86b7aeb2eecc76a6e1da77889e041c914e104ea188afa27a84ce26e4cf655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2) 2024-08-31T00:02:41.187 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 31 00:02:41 smithi106 podman[286742]: 2024-08-31 00:02:41.167187633 +0000 UTC m=+0.736478259 container cleanup 7eb86b7aeb2eecc76a6e1da77889e041c914e104ea188afa27a84ce26e4cf655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:24432b660539783e58f890fcff372342c034f6ec, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:02:41.187 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Aug 31 00:02:41 smithi106 bash[286742]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-y 2024-08-31T00:02:41.591 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.y.service' 2024-08-31T00:02:41.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:41.625 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-08-31T00:02:41.625 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-08-31T00:02:41.625 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.x 2024-08-31T00:02:41.994 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:41 smithi136 systemd[1]: Stopping Ceph mgr.x for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:42.248 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:41 smithi136 podman[172443]: 2024-08-31 00:02:41.99417611 +0000 UTC m=+0.172260424 container died 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-08-31T00:02:42.533 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 podman[172443]: 2024-08-31 00:02:42.339922078 +0000 UTC m=+0.518006382 container cleanup 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-31T00:02:42.533 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 bash[172443]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x 2024-08-31T00:02:42.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.x.service' 2024-08-31T00:02:42.783 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 podman[172456]: 2024-08-31 00:02:42.533201763 +0000 UTC m=+0.534307713 container remove 123c700415741a5844b4c718d04602fa8f8be23eef84d28c1651f878ffb164d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-mgr-x, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-08-31T00:02:42.784 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-08-31T00:02:42.784 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-08-31T00:02:42.784 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 systemd[1]: Stopped Ceph mgr.x for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-31T00:02:42.784 INFO:journalctl@ceph.mgr.x.smithi136.stdout:Aug 31 00:02:42 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@mgr.x.service: Consumed 7.904s CPU time. 2024-08-31T00:02:42.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:42.797 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-08-31T00:02:42.797 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-08-31T00:02:42.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.0 2024-08-31T00:02:43.107 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:42 smithi106 systemd[1]: Stopping Ceph osd.0 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:43.607 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:43 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0[30291]: 2024-08-31T00:02:43.129+0000 7f4bc84f7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:02:43.607 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:43 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0[30291]: 2024-08-31T00:02:43.129+0000 7f4bc84f7640 -1 osd.0 1079 *** Got signal Terminated *** 2024-08-31T00:02:43.608 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:43 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0[30291]: 2024-08-31T00:02:43.129+0000 7f4bc84f7640 -1 osd.0 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:02:48.858 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:48 smithi106 podman[286862]: 2024-08-31 00:02:48.582767967 +0000 UTC m=+5.597817338 container died 506f3392360145e58d3b29a6c7b842f98bf720a178c77c14460cd9e5e96a90d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:02:49.357 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:49 smithi106 podman[286862]: 2024-08-31 00:02:49.047407551 +0000 UTC m=+6.062456926 container cleanup 506f3392360145e58d3b29a6c7b842f98bf720a178c77c14460cd9e5e96a90d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-08-31T00:02:49.358 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:49 smithi106 bash[286862]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0 2024-08-31T00:02:49.807 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:49 smithi106 podman[286879]: 2024-08-31 00:02:49.407518388 +0000 UTC m=+0.818494532 container remove 506f3392360145e58d3b29a6c7b842f98bf720a178c77c14460cd9e5e96a90d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-31T00:02:50.086 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:49 smithi106 podman[286967]: 2024-08-31 00:02:49.710336613 +0000 UTC m=+0.016435852 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:02:50.086 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:49 smithi106 podman[286967]: 2024-08-31 00:02:49.833400799 +0000 UTC m=+0.139500022 container create 9adb0cf551daef0fb021fd92cba77007fd4c3a955cd40e59d1a13b2a18817898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-31T00:02:50.086 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:50 smithi106 podman[286967]: 2024-08-31 00:02:50.086142326 +0000 UTC m=+0.392241556 container init 9adb0cf551daef0fb021fd92cba77007fd4c3a955cd40e59d1a13b2a18817898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-08-31T00:02:50.341 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:50 smithi106 podman[286967]: 2024-08-31 00:02:50.089945233 +0000 UTC m=+0.396044464 container start 9adb0cf551daef0fb021fd92cba77007fd4c3a955cd40e59d1a13b2a18817898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-31T00:02:50.341 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:50 smithi106 podman[286967]: 2024-08-31 00:02:50.177774536 +0000 UTC m=+0.483873775 container attach 9adb0cf551daef0fb021fd92cba77007fd4c3a955cd40e59d1a13b2a18817898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-31T00:02:50.599 INFO:journalctl@ceph.osd.0.smithi106.stdout:Aug 31 00:02:50 smithi106 podman[286967]: 2024-08-31 00:02:50.341750572 +0000 UTC m=+0.647849797 container died 9adb0cf551daef0fb021fd92cba77007fd4c3a955cd40e59d1a13b2a18817898 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-31T00:02:51.013 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.0.service' 2024-08-31T00:02:51.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:51.051 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-08-31T00:02:51.051 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-31T00:02:51.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1 2024-08-31T00:02:51.357 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:51 smithi106 systemd[1]: Stopping Ceph osd.1 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:02:51.857 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:51 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[165229]: 2024-08-31T00:02:51.387+0000 7f83ad307640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:02:51.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:51 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[165229]: 2024-08-31T00:02:51.387+0000 7f83ad307640 -1 osd.1 1079 *** Got signal Terminated *** 2024-08-31T00:02:51.858 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:51 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1[165229]: 2024-08-31T00:02:51.387+0000 7f83ad307640 -1 osd.1 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:02:57.202 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:56 smithi106 podman[287069]: 2024-08-31 00:02:56.919659443 +0000 UTC m=+5.651007015 container died 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-08-31T00:02:57.469 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:57 smithi106 podman[287069]: 2024-08-31 00:02:57.448822473 +0000 UTC m=+6.180170043 container cleanup 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-08-31T00:02:57.470 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:57 smithi106 bash[287069]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1 2024-08-31T00:02:58.066 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:57 smithi106 podman[287086]: 2024-08-31 00:02:57.65324809 +0000 UTC m=+0.737155149 container remove 74d2b61476fb2a7473323457387e8a51cd9433281a41143f898eef92a2f46025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-08-31T00:02:58.357 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:58 smithi106 podman[287172]: 2024-08-31 00:02:57.968760378 +0000 UTC m=+0.016027400 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:02:58.357 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:58 smithi106 podman[287172]: 2024-08-31 00:02:58.087410685 +0000 UTC m=+0.134677693 container create 24e09000e750bcde58a833d71ed746b8edfa78a6cc09ed54e98797a6fe51564a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:02:58.358 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:58 smithi106 podman[287172]: 2024-08-31 00:02:58.306578558 +0000 UTC m=+0.353845579 container init 24e09000e750bcde58a833d71ed746b8edfa78a6cc09ed54e98797a6fe51564a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:02:58.358 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:58 smithi106 podman[287172]: 2024-08-31 00:02:58.310032362 +0000 UTC m=+0.357299390 container start 24e09000e750bcde58a833d71ed746b8edfa78a6cc09ed54e98797a6fe51564a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-31T00:02:58.842 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:58 smithi106 podman[287172]: 2024-08-31 00:02:58.414975912 +0000 UTC m=+0.462242935 container attach 24e09000e750bcde58a833d71ed746b8edfa78a6cc09ed54e98797a6fe51564a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-08-31T00:02:58.843 INFO:journalctl@ceph.osd.1.smithi106.stdout:Aug 31 00:02:58 smithi106 podman[287172]: 2024-08-31 00:02:58.498447892 +0000 UTC m=+0.545714911 container died 24e09000e750bcde58a833d71ed746b8edfa78a6cc09ed54e98797a6fe51564a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:02:59.251 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.1.service' 2024-08-31T00:02:59.295 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:02:59.295 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-31T00:02:59.295 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-31T00:02:59.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.2 2024-08-31T00:02:59.607 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:02:59 smithi106 systemd[1]: Stopping Ceph osd.2 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:00.107 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:02:59 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2[41630]: 2024-08-31T00:02:59.663+0000 7f9300487640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:00.108 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:02:59 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2[41630]: 2024-08-31T00:02:59.663+0000 7f9300487640 -1 osd.2 1079 *** Got signal Terminated *** 2024-08-31T00:03:00.108 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:02:59 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2[41630]: 2024-08-31T00:02:59.663+0000 7f9300487640 -1 osd.2 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:05.608 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:05 smithi106 podman[287276]: 2024-08-31 00:03:05.284060211 +0000 UTC m=+5.754837851 container died 8fd2b08584af648ded1fcea609dcad6f79186423a5b8e58f3fc12e1ba939ff98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:03:05.998 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:05 smithi106 podman[287276]: 2024-08-31 00:03:05.73001551 +0000 UTC m=+6.200793139 container cleanup 8fd2b08584af648ded1fcea609dcad6f79186423a5b8e58f3fc12e1ba939ff98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:03:05.998 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:05 smithi106 bash[287276]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2 2024-08-31T00:03:06.347 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:05 smithi106 podman[287292]: 2024-08-31 00:03:05.998133687 +0000 UTC m=+0.716008222 container remove 8fd2b08584af648ded1fcea609dcad6f79186423a5b8e58f3fc12e1ba939ff98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:06.607 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:06 smithi106 podman[287384]: 2024-08-31 00:03:06.250727348 +0000 UTC m=+0.015908437 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:06.608 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:06 smithi106 podman[287384]: 2024-08-31 00:03:06.35725327 +0000 UTC m=+0.122434350 container create 77d763fe7fc459ff12c84c92d969c0ec523b3cc614ba22e623a7cc9b5bcc4cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-31T00:03:06.608 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:06 smithi106 podman[287384]: 2024-08-31 00:03:06.58475565 +0000 UTC m=+0.349936730 container init 77d763fe7fc459ff12c84c92d969c0ec523b3cc614ba22e623a7cc9b5bcc4cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:06.608 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:06 smithi106 podman[287384]: 2024-08-31 00:03:06.587943386 +0000 UTC m=+0.353124466 container start 77d763fe7fc459ff12c84c92d969c0ec523b3cc614ba22e623a7cc9b5bcc4cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-08-31T00:03:07.054 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:06 smithi106 podman[287384]: 2024-08-31 00:03:06.685131489 +0000 UTC m=+0.450312569 container attach 77d763fe7fc459ff12c84c92d969c0ec523b3cc614ba22e623a7cc9b5bcc4cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-31T00:03:07.054 INFO:journalctl@ceph.osd.2.smithi106.stdout:Aug 31 00:03:06 smithi106 podman[287384]: 2024-08-31 00:03:06.775072877 +0000 UTC m=+0.540253977 container died 77d763fe7fc459ff12c84c92d969c0ec523b3cc614ba22e623a7cc9b5bcc4cf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:07.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.2.service' 2024-08-31T00:03:07.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:07.491 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-31T00:03:07.491 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-31T00:03:07.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.3 2024-08-31T00:03:07.809 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:07 smithi106 systemd[1]: Stopping Ceph osd.3 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:08.107 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:07 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-31T00:03:07.807+0000 7ff371737640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:08.107 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:07 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-31T00:03:07.807+0000 7ff371737640 -1 osd.3 1079 *** Got signal Terminated *** 2024-08-31T00:03:08.107 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:07 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-31T00:03:07.807+0000 7ff371737640 -1 osd.3 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:09.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:09.129+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:10.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:10 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:10.172+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:10.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:10 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:10.505+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:11.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:10 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:10.823+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:11.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:11.175+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:11.775 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:11 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:11.486+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:11.858 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:11 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-31T00:03:11.574+0000 7ff36dd4c640 -1 osd.3 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.222654+0000 front 2024-08-31T00:02:46.222797+0000 (oldest deadline 2024-08-31T00:03:11.522789+0000) 2024-08-31T00:03:12.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:11 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:11.815+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:12.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:11 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:11.775+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:12.492 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:12.199+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:12.759 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:12.440+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:12.760 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:12.553+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:12.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:12.491+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:12.857 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:12 smithi106 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3[47333]: 2024-08-31T00:03:12.579+0000 7ff36dd4c640 -1 osd.3 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.222654+0000 front 2024-08-31T00:02:46.222797+0000 (oldest deadline 2024-08-31T00:03:11.522789+0000) 2024-08-31T00:03:13.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:12.852+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:13.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:12 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:12.759+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:13.509 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:13.223+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:13.509 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:13.075+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:13.575 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:13 smithi106 podman[287486]: 2024-08-31 00:03:13.306199463 +0000 UTC m=+5.591698517 container died 3ce6d91da8c21cea42c76c4c89ccb1f62a1c6fa5c43848229176f4a684e0a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:13.748 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:13.527+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:13.749 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:13.487+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:13.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:13.508+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:13.857 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:13 smithi106 podman[287486]: 2024-08-31 00:03:13.758530117 +0000 UTC m=+6.044029170 container cleanup 3ce6d91da8c21cea42c76c4c89ccb1f62a1c6fa5c43848229176f4a684e0a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-31T00:03:13.858 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:13 smithi106 bash[287486]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3 2024-08-31T00:03:14.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:13.856+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:14.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:13 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:13.747+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:14.357 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287504]: 2024-08-31 00:03:14.03486109 +0000 UTC m=+0.723658238 container remove 3ce6d91da8c21cea42c76c4c89ccb1f62a1c6fa5c43848229176f4a684e0a71e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:14.479 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:14 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:14.101+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:14.481 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:14 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:14.185+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:14.613 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287589]: 2024-08-31 00:03:14.360565109 +0000 UTC m=+0.112533266 container create 0770b61bca609df847b7df8c90ab0d3562826cbb045bdc30b0bb2d293cf3f279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:14.613 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287589]: 2024-08-31 00:03:14.263847629 +0000 UTC m=+0.015815797 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:14.613 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287589]: 2024-08-31 00:03:14.612806266 +0000 UTC m=+0.364774433 container init 0770b61bca609df847b7df8c90ab0d3562826cbb045bdc30b0bb2d293cf3f279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:14.768 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:14 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:14.472+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:14.768 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:14 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:14.495+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:14.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:14 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:14.478+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:15.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:14 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:14.812+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:15.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:14 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:14.767+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:15.082 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287589]: 2024-08-31 00:03:14.616284035 +0000 UTC m=+0.368252192 container start 0770b61bca609df847b7df8c90ab0d3562826cbb045bdc30b0bb2d293cf3f279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:03:15.082 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287589]: 2024-08-31 00:03:14.696497765 +0000 UTC m=+0.448465947 container attach 0770b61bca609df847b7df8c90ab0d3562826cbb045bdc30b0bb2d293cf3f279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:15.082 INFO:journalctl@ceph.osd.3.smithi106.stdout:Aug 31 00:03:14 smithi106 podman[287589]: 2024-08-31 00:03:14.802965865 +0000 UTC m=+0.554934021 container died 0770b61bca609df847b7df8c90ab0d3562826cbb045bdc30b0bb2d293cf3f279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-31T00:03:15.441 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.3.service' 2024-08-31T00:03:15.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:15.481 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-31T00:03:15.482 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-08-31T00:03:15.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.4 2024-08-31T00:03:15.526 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:15.235+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:15.527 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:15.084+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:15.755 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:15 smithi136 systemd[1]: Stopping Ceph osd.4 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:15.756 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:15.540+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:15.757 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:15.498+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:15.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:15.525+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:16.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:15.754+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:16.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:15.804+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:16.041 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:15.906+0000 7fee2c8f2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:16.041 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:15.906+0000 7fee2c8f2640 -1 osd.4 1079 *** Got signal Terminated *** 2024-08-31T00:03:16.041 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:15 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:15.906+0000 7fee2c8f2640 -1 osd.4 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:16.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:16.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:16.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:16.231+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:16.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:16 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:16.528+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:16.791 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:16 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:16.491+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:16.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:16.540+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:17.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:16 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:16.799+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:17.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:16 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:16.803+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:17.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:17.255+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:17.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:17.080+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:17.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:17.499+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:17.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:17.489+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:18.041 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:17.575+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:18.041 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:17.575+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:18.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:17.825+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:18.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:17 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:17.814+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:18.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:18 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:18.040+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:18.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:18 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:18.208+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:18.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:18 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:18.208+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:18.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:18.522+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:18.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:18.529+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:18.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:18.529+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:18.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:18 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:18.566+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:18.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:18 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:18.566+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:19.182 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:18 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:18.844+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:19.183 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:19.087+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:19.183 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:19.087+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:19.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:18.801+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:19.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:18.801+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.684436+0000 front 2024-08-31T00:02:55.684480+0000 (oldest deadline 2024-08-31T00:03:18.584378+0000) 2024-08-31T00:03:19.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:18.820+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:19.291 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:18 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:18.820+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:19.570 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:19.181+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:19.571 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:19.181+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:19.783 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:19.497+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:19.783 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:19.497+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:19.784 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:19.497+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:19.784 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:19.497+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:19.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:19.570+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:19.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:19.570+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:20.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:19.818+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:20.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:19.818+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.684436+0000 front 2024-08-31T00:02:55.684480+0000 (oldest deadline 2024-08-31T00:03:18.584378+0000) 2024-08-31T00:03:20.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:19.782+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:20.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:19 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:19.782+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:20.130 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:19 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:19.877+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:20.572 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:20.153+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:20.573 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:20.153+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:20.573 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:20.129+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:20.573 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:20.129+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:20.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:20.464+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:20.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:20.464+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:20.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:20.531+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:20.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:20.531+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:20.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:20.572+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:20.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:20.572+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:21.138 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:20 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:20.839+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:21.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:20.829+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.283418+0000 front 2024-08-31T00:02:46.283429+0000 (oldest deadline 2024-08-31T00:03:10.383322+0000) 2024-08-31T00:03:21.290 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4[24704]: 2024-08-31T00:03:20.829+0000 7fee28f07640 -1 osd.4 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.684436+0000 front 2024-08-31T00:02:55.684480+0000 (oldest deadline 2024-08-31T00:03:18.584378+0000) 2024-08-31T00:03:21.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:20.826+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:21.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:20 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:20.826+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:21.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:21.137+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:21.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:21.137+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:21.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:21.163+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:21.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:21.163+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:21.790 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:21 smithi136 podman[172566]: 2024-08-31 00:03:21.416757224 +0000 UTC m=+5.745794871 container died 6793248a9f99b5e39decef43483e1705afb9f953c8f1e6fc06248e14715120fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:03:21.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:21.466+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:21.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:21.466+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:21.791 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:21.552+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:21.791 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:21.552+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:22.074 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:21.803+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:22.074 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:21.600+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:22.074 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:21 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:21.600+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:22.075 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:21 smithi136 podman[172566]: 2024-08-31 00:03:21.817014951 +0000 UTC m=+6.146052638 container cleanup 6793248a9f99b5e39decef43483e1705afb9f953c8f1e6fc06248e14715120fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-31T00:03:22.075 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:21 smithi136 bash[172566]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4 2024-08-31T00:03:22.075 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:21.790+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:22.075 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:21 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:21.790+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:22.425 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172581]: 2024-08-31 00:03:22.07466201 +0000 UTC m=+0.639867300 container remove 6793248a9f99b5e39decef43483e1705afb9f953c8f1e6fc06248e14715120fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-08-31T00:03:22.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:22.197+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:22.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:22.197+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:22.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:22.115+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:22.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:22.115+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:22.744 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:22.510+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:22.745 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:22.510+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:22.745 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:22.591+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:22.745 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:22.591+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:22.745 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172671]: 2024-08-31 00:03:22.328065947 +0000 UTC m=+0.015748780 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:22.745 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172671]: 2024-08-31 00:03:22.425558137 +0000 UTC m=+0.113240963 container create 0002d427507c2a94e5f418c7836bd83757cb2ebe55da975421a92092e20ff166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-08-31T00:03:22.745 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172671]: 2024-08-31 00:03:22.661501572 +0000 UTC m=+0.349184400 container init 0002d427507c2a94e5f418c7836bd83757cb2ebe55da975421a92092e20ff166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:22.746 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172671]: 2024-08-31 00:03:22.665276219 +0000 UTC m=+0.352959042 container start 0002d427507c2a94e5f418c7836bd83757cb2ebe55da975421a92092e20ff166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-31T00:03:22.746 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172671]: 2024-08-31 00:03:22.744670517 +0000 UTC m=+0.432353353 container attach 0002d427507c2a94e5f418c7836bd83757cb2ebe55da975421a92092e20ff166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-08-31T00:03:23.040 INFO:journalctl@ceph.osd.4.smithi136.stdout:Aug 31 00:03:22 smithi136 podman[172671]: 2024-08-31 00:03:22.86264221 +0000 UTC m=+0.550325037 container died 0002d427507c2a94e5f418c7836bd83757cb2ebe55da975421a92092e20ff166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-08-31T00:03:23.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:22.772+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:23.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:22 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:22.772+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:23.074 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:22.801+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:23.074 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:22.639+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:23.074 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:22 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:22.639+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:23.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.4.service' 2024-08-31T00:03:23.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:23.240+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:23.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:23.240+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:23.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:23.165+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:23.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:23.165+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:23.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:23.592 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-08-31T00:03:23.592 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-31T00:03:23.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5 2024-08-31T00:03:23.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:23.601+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:23.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:23.601+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:23.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:23 smithi136 systemd[1]: Stopping Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:23.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:23.747+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:23.791 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:23.747+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:23.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:23.499+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:23.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:23.499+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:24.074 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:23.832+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:24.074 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:23.590+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:24.074 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:23 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:23.590+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:24.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:23.909+0000 7fc7fbe89640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:24.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:23.909+0000 7fc7fbe89640 -1 osd.5 1079 *** Got signal Terminated *** 2024-08-31T00:03:24.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:23 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:23.909+0000 7fc7fbe89640 -1 osd.5 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:24.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:24.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:24.558 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:24.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:24.558 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:24.173+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:24.559 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:24.173+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:24.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:24 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:24.702+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:24.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:24 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:24.702+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:24.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:24 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:24.618+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:24.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:24 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:24.618+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:24.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:24 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:24.521+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:24.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:24 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:24.522+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:24.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:24.557+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:24.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:24.557+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:25.183 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:24 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:24.850+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:25.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:25.212+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:25.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:25.212+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:25.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:25.183+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:25.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:25.183+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:25.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:25 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:25.667+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:25.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:25 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:25.667+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:25.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:25 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:25.600+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:25.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:25 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:25.600+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:25.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:25 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:25.497+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:25.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:25 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:25.497+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:25.878 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:25.585+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:25.878 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:25.585+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:26.204 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:25 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:25.877+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:26.513 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:26.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:26.514 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:26.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:26.514 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:26.203+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:26.514 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:26.203+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:26.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:26.692+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:26.790 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:26.692+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:26.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:26.560+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:26.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:26.560+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:26.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:26.505+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:26.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:26.505+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:26.791 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:26 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:26.505+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:26.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:26.609+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:26.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:26.609+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:27.173 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:26 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:26.866+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:27.530 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:27.172+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:27.531 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:27.172+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:27.531 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:27.214+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:27.531 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:27.214+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:27.531 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:27.214+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:27.718 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:27.559+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:27.718 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:27.559+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:27.718 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:27.463+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:27.718 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:27.463+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:27.718 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:27.463+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:27.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:27.583+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:27.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:27.583+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:27.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:27.583+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:28.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:27.717+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:28.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:27.717+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:28.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:27 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:27.717+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.412445+0000 front 2024-08-31T00:03:00.412529+0000 (oldest deadline 2024-08-31T00:03:26.813028+0000) 2024-08-31T00:03:28.179 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:27 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:27.910+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:28.179 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:28.158+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:28.179 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:28.158+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:28.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:28.178+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:28.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:28.178+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:28.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:28.178+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:28.737 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:28.426+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:28.737 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:28.426+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:28.737 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:28.426+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:28.737 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:28.602+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:28.737 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:28.602+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:28.737 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:28.602+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:28.931 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:28.603+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:28.931 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:28.603+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:28.931 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:28.603+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:29.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:28.736+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:46.509689+0000 front 2024-08-31T00:02:46.509763+0000 (oldest deadline 2024-08-31T00:03:11.209597+0000) 2024-08-31T00:03:29.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:28.736+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.211283+0000 front 2024-08-31T00:02:55.211390+0000 (oldest deadline 2024-08-31T00:03:18.711339+0000) 2024-08-31T00:03:29.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:28 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5[118079]: 2024-08-31T00:03:28.736+0000 7fc7f849e640 -1 osd.5 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.412445+0000 front 2024-08-31T00:03:00.412529+0000 (oldest deadline 2024-08-31T00:03:26.813028+0000) 2024-08-31T00:03:29.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:29.176+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:29.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:29.176+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:29.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:28.930+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:29.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:28 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:28.930+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:29.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:29.135+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:29.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:29.135+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:29.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:29.135+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:29.744 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:29 smithi136 podman[172773]: 2024-08-31 00:03:29.474496637 +0000 UTC m=+5.664709989 container died be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-08-31T00:03:29.744 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:29 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:29.451+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:29.744 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:29 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:29.451+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:29.744 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:29 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:29.451+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:29.745 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:29 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:29.647+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:29.745 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:29 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:29.647+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:29.745 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:29 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:29.647+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:29.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:29.572+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:29.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:29.572+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:29.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:29.572+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:30.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:29 smithi136 podman[172773]: 2024-08-31 00:03:29.935224678 +0000 UTC m=+6.125438073 container cleanup be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:30.040 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:29 smithi136 bash[172773]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5 2024-08-31T00:03:30.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:30.169+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:30.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:30.169+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:30.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:29.964+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:30.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:29 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:29.964+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:30.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:30.146+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:30.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:30.146+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:30.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:30.146+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:30.654 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:30 smithi136 podman[172803]: 2024-08-31 00:03:30.312358065 +0000 UTC m=+0.832278861 container remove be0503bc9be4319ea60bad8055166436b395523527ee9ab2960e7524d962fdbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-31T00:03:30.654 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:30 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:30.427+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:30.654 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:30 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:30.427+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:30.654 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:30 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:30.427+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:30.926 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:30 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:30.653+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:30.926 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:30 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:30.653+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:30.926 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:30 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:30.653+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:30.926 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:30 smithi136 podman[172890]: 2024-08-31 00:03:30.59417169 +0000 UTC m=+0.015840656 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:30.926 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:30 smithi136 podman[172890]: 2024-08-31 00:03:30.707110967 +0000 UTC m=+0.128779923 container create 6a2ad9cb1a01f987ad9b2b45dc5486948ee70110a0d96fe88a337823399dccad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:30.927 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:30 smithi136 podman[172890]: 2024-08-31 00:03:30.926251883 +0000 UTC m=+0.347920848 container init 6a2ad9cb1a01f987ad9b2b45dc5486948ee70110a0d96fe88a337823399dccad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-31T00:03:30.985 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:30.590+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:30.986 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:30.590+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:30.986 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:30.590+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:31.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:30 smithi136 podman[172890]: 2024-08-31 00:03:30.929625402 +0000 UTC m=+0.351294359 container start 6a2ad9cb1a01f987ad9b2b45dc5486948ee70110a0d96fe88a337823399dccad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:03:31.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:31 smithi136 podman[172890]: 2024-08-31 00:03:31.067936982 +0000 UTC m=+0.489605959 container attach 6a2ad9cb1a01f987ad9b2b45dc5486948ee70110a0d96fe88a337823399dccad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:03:31.290 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:31 smithi136 podman[172890]: 2024-08-31 00:03:31.114653231 +0000 UTC m=+0.536322197 container died 6a2ad9cb1a01f987ad9b2b45dc5486948ee70110a0d96fe88a337823399dccad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:03:31.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:31.195+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:31.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:31.195+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:31.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:30.985+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:31.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:30 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:30.985+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:31.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:31.173+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:31.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:31.173+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:31.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:31.173+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:31.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service' 2024-08-31T00:03:31.719 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:31 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:31.467+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:31 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:31.467+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:31 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:31.467+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:31 smithi136 podman[172913]: 2024-08-31 00:03:31.679666018 +0000 UTC m=+0.559000638 container remove 6a2ad9cb1a01f987ad9b2b45dc5486948ee70110a0d96fe88a337823399dccad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:31 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service: Deactivated successfully. 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:31 smithi136 systemd[1]: Stopped Ceph osd.5 for eb936aa2-6725-11ef-bcd4-c7b262605968. 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.5.smithi136.stdout:Aug 31 00:03:31 smithi136 systemd[1]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.5.service: Consumed 11min 28.480s CPU time. 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:31 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:31.682+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:31.720 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:31 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:31.682+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:31.721 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:31 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:31.682+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:31.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:31.736 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-31T00:03:31.736 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-31T00:03:31.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.6 2024-08-31T00:03:31.983 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:31.622+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:31.984 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:31.622+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:31.984 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:31.622+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:32.040 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:31 smithi136 systemd[1]: Stopping Ceph osd.6 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:32.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:32.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:32.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:32.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:32.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:31.983+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:32.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:31 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:31.983+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:32.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:32.128+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:32.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:32.128+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:32.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:32.128+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:32.479 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:32.065+0000 7ff63aa3b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:32.479 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:32.065+0000 7ff63aa3b640 -1 osd.6 1079 *** Got signal Terminated *** 2024-08-31T00:03:32.479 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:32.065+0000 7ff63aa3b640 -1 osd.6 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:32.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:32.478+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:32.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:32.478+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:32.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:32.478+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:32.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:32.660+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:32.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:32.660+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:32.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:32 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:32.660+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:32.943 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:32.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:32.943 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:32.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:32.943 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:32.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:33.230 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:33.132+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:33.230 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:33.132+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:33.230 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:33.132+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:33.230 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:32.942+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:33.230 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:32 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:32.942+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:33.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:33.229+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:33.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:33.229+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:33.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:33 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:33.429+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:33.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:33 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:33.429+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:33.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:33 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:33.429+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:33.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:33 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:33.638+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:33.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:33 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:33.638+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:33.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:33 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:33.638+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:33.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:33.691+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:33.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:33.691+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:33.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:33.691+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:34.242 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:33.968+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:34.242 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:33 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:33.968+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:34.242 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:34.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:34.242 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:34.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:34.242 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:34.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:34.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:34.241+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:34.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:34.241+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:34.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:34 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:34.456+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:34.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:34 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:34.456+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:34.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:34 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:34.456+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:34.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:34 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:34.674+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:34.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:34 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:34.674+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:34.791 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:34 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:34.674+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:34.971 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:34.660+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:34.971 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:34.660+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:34.971 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:34.660+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:35.255 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:34.971+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:35.256 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:34 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:34.971+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:35.256 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:35.154+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:35.256 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:35.154+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:35.256 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:35.154+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:35.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:35.255+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:35.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:35.255+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:35.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:35.255+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:35.505+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:35.505+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:35.505+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:35.715+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:35.715+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:35.715+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:35.790 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:35 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:35.715+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:35.971 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:35.638+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:35.971 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:35.638+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:35.971 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:35.638+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:36.269 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:35.971+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:36.270 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:35.971+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:36.270 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:35 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:35.971+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:36.270 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:36.112+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:36.270 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:36.112+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:36.270 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:36.112+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:36.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:36.269+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:36.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:36.269+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:36.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:36.269+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:36.761 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:36.509+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.906555+0000 front 2024-08-31T00:02:47.906670+0000 (oldest deadline 2024-08-31T00:03:12.506956+0000) 2024-08-31T00:03:36.761 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:36.509+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:48.407114+0000 front 2024-08-31T00:02:48.407183+0000 (oldest deadline 2024-08-31T00:03:17.807495+0000) 2024-08-31T00:03:36.762 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:36.509+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:00.608923+0000 front 2024-08-31T00:03:00.608874+0000 (oldest deadline 2024-08-31T00:03:25.710064+0000) 2024-08-31T00:03:36.762 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6[36338]: 2024-08-31T00:03:36.509+0000 7ff633048640 -1 osd.6 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.011043+0000 front 2024-08-31T00:03:11.011058+0000 (oldest deadline 2024-08-31T00:03:35.710898+0000) 2024-08-31T00:03:36.983 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:36.664+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:36.983 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:36.664+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:36.983 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:36.664+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:37.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:36.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:37.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:36.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:37.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:36.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:37.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:36 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:36.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:37.271 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:36.982+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:37.271 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:36.982+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:37.272 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:36 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:36.982+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:37.272 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:37.076+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:37.272 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:37.076+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:37.272 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:37.076+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:37.272 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:37.076+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:37.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:37.271+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:37.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:37.271+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:37.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:37.271+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:37.883 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:37 smithi136 podman[172993]: 2024-08-31 00:03:37.602950222 +0000 UTC m=+5.636994717 container died fbb915b0e43b377ff36011f905f57ad0ff02f774d50ea1b39a3a36525cee70c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:37.883 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:37 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:37.757+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:37.884 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:37 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:37.757+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:37.884 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:37 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:37.757+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:37.884 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:37 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:37.757+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:38.014 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:37.659+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:38.014 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:37.659+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:38.014 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:37 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:37.659+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:38.272 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:38.014+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:38.272 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:38.014+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:38.272 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:38.014+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:38.273 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:38.044+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:38.273 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:38.044+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:38.273 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:38.044+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:38.273 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:38.044+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:38.290 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:38 smithi136 podman[172993]: 2024-08-31 00:03:38.032195676 +0000 UTC m=+6.066240155 container cleanup fbb915b0e43b377ff36011f905f57ad0ff02f774d50ea1b39a3a36525cee70c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-08-31T00:03:38.290 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:38 smithi136 bash[172993]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6 2024-08-31T00:03:38.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:38.272+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:38.575 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:38.272+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:38.575 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:38.272+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:38.742 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:38 smithi136 podman[173009]: 2024-08-31 00:03:38.308955746 +0000 UTC m=+0.707344111 container remove fbb915b0e43b377ff36011f905f57ad0ff02f774d50ea1b39a3a36525cee70c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-31T00:03:38.968 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:38.666+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:38.968 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:38.666+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:38.969 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:38.666+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:39.029 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:38 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:38.742+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:39.029 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:38 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:38.742+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:39.029 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:38 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:38.742+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:39.029 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:38 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:38.742+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:39.030 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:38 smithi136 podman[173097]: 2024-08-31 00:03:38.689556056 +0000 UTC m=+0.015775854 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:39.030 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:38 smithi136 podman[173097]: 2024-08-31 00:03:38.809967823 +0000 UTC m=+0.136187649 container create a7ac0c927909a4a3ef0be08add2cbd1a911c106d8b38568f9791d2a27caa17cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True) 2024-08-31T00:03:39.030 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:39 smithi136 podman[173097]: 2024-08-31 00:03:39.029220778 +0000 UTC m=+0.355440567 container init a7ac0c927909a4a3ef0be08add2cbd1a911c106d8b38568f9791d2a27caa17cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:39.243 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:38.968+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:39.243 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:38.968+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:39.243 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:38 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:38.968+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:39.244 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:39.055+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:39.244 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:39.055+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:39.244 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:39.055+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:39.244 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:39.055+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:39.290 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:39 smithi136 podman[173097]: 2024-08-31 00:03:39.032760496 +0000 UTC m=+0.358980284 container start a7ac0c927909a4a3ef0be08add2cbd1a911c106d8b38568f9791d2a27caa17cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-08-31T00:03:39.290 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:39 smithi136 podman[173097]: 2024-08-31 00:03:39.11225014 +0000 UTC m=+0.438469926 container attach a7ac0c927909a4a3ef0be08add2cbd1a911c106d8b38568f9791d2a27caa17cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD) 2024-08-31T00:03:39.290 INFO:journalctl@ceph.osd.6.smithi136.stdout:Aug 31 00:03:39 smithi136 podman[173097]: 2024-08-31 00:03:39.222637419 +0000 UTC m=+0.548857203 container died a7ac0c927909a4a3ef0be08add2cbd1a911c106d8b38568f9791d2a27caa17cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-08-31T00:03:39.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:39.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:39.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:39.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:39.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:39.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:39.794 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.6.service' 2024-08-31T00:03:39.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:39.835 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-31T00:03:39.835 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-31T00:03:39.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7 2024-08-31T00:03:40.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:39.684+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:40.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:39.684+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:40.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:39 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:39.684+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:40.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:39 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:39.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:40.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:39 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:39.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:40.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:39 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:39.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:40.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:39 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:39.760+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:40.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:39 smithi136 systemd[1]: Stopping Ceph osd.7 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:40.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:40.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:40.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:40.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:40.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:40.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:40.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:40.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:40.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:40.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:40.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:40.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:40.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:40.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:40.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:40.013+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:40.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:40.013+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:40.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:40.013+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:40.540 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.151+0000 7f54c9353640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:40.540 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.151+0000 7f54c9353640 -1 osd.7 1079 *** Got signal Terminated *** 2024-08-31T00:03:40.540 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.151+0000 7f54c9353640 -1 osd.7 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:41.007 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:40.699+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:41.008 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:40.699+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:41.008 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:40 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:40.699+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:41.174 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.804+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:41.174 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.804+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:41.174 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.804+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:41.174 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:40 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:40.804+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:41.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:41.230+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:41.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:41.230+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:41.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:41.230+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:41.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:41.106+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:41.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:41.106+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:41.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:41.106+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:41.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:41.106+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:41.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:41.007+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:41.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:41.007+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:41.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:41.007+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:41.966 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:41.672+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:41.966 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:41.672+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:41.966 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:41.672+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:42.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:41.781+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:42.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:41.781+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:42.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:41.781+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:42.040 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:41 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:41.781+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:42.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:42.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:42.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:42.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:42.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:42.194+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:42.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:42.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:42.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:42.114+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:41.966+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:41.966+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:42.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:41 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:41.966+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:42.938 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:42.659+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:42.939 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:42.659+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:42.939 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:42.659+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:43.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:42 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:42.811+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:43.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:42 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:42.811+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:43.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:42 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:42.811+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:43.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:42 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:42.811+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:43.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:42 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:42.811+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.826375+0000 front 2024-08-31T00:03:20.826448+0000 (oldest deadline 2024-08-31T00:03:41.926439+0000) 2024-08-31T00:03:43.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:43.187+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:43.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:43.187+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:43.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:43.187+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:43.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:43.116+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:43.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:43.116+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:43.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:43.116+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:43.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:43.116+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:43.326 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:42.938+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:43.326 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:42.938+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:43.326 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:42 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:42.938+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:43.980 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:43.625+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:43.980 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:43.625+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:43.980 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:43.625+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:43.980 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:43.625+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:44.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:43 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:43.795+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:44.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:43 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:43.795+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:44.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:43 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:43.795+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:44.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:43 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:43.795+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:44.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:43 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:43.795+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.826375+0000 front 2024-08-31T00:03:20.826448+0000 (oldest deadline 2024-08-31T00:03:41.926439+0000) 2024-08-31T00:03:44.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:44.188+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:44.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:44.188+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:44.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:44.188+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:44.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:44.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:44.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:44.085+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:43.979+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:43.979+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:44.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:43 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:43.979+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:45.027 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:44.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:45.027 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:44.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:45.027 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:44.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:45.027 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:44 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:44.649+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:45.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:44 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:44.844+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:48.022390+0000 front 2024-08-31T00:02:48.022306+0000 (oldest deadline 2024-08-31T00:03:09.722280+0000) 2024-08-31T00:03:45.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:44 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:44.844+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:53.823400+0000 front 2024-08-31T00:02:53.823382+0000 (oldest deadline 2024-08-31T00:03:19.123402+0000) 2024-08-31T00:03:45.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:44 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:44.844+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:02:59.124126+0000 front 2024-08-31T00:02:59.124059+0000 (oldest deadline 2024-08-31T00:03:27.924468+0000) 2024-08-31T00:03:45.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:44 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:44.844+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:07.925065+0000 front 2024-08-31T00:03:07.925082+0000 (oldest deadline 2024-08-31T00:03:34.925481+0000) 2024-08-31T00:03:45.291 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:44 smithi136 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7[171078]: 2024-08-31T00:03:44.844+0000 7f54c1960640 -1 osd.7 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.826375+0000 front 2024-08-31T00:03:20.826448+0000 (oldest deadline 2024-08-31T00:03:41.926439+0000) 2024-08-31T00:03:45.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:45.222+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:45.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:45.222+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:45.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:45.222+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:45.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:45.038+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:45.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:45.038+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:45.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:45.038+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:45.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:45.038+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:45.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:45.027+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:45.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:45.027+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:45.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:45.027+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:45.877 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:45 smithi136 podman[173199]: 2024-08-31 00:03:45.595310753 +0000 UTC m=+5.538795537 container died cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-31T00:03:46.061 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:45.627+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:46.061 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:45.627+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:46.061 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:45.627+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:46.062 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:45 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:45.627+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:46.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173199]: 2024-08-31 00:03:46.081097058 +0000 UTC m=+6.024581838 container cleanup cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True) 2024-08-31T00:03:46.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 bash[173199]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7 2024-08-31T00:03:46.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:46.061+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:46.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:46.061+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:46.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:46.061+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:46.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:46.061+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:46.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:46.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:46.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:46.242+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:46.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:46.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:46.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:46.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:46.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:46.612 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173214]: 2024-08-31 00:03:46.302628037 +0000 UTC m=+0.709166829 container remove cd1f7dd71ca6c86deb8bcce498c88566d8085866ae7a44ef1736955c80da47ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-08-31T00:03:46.872 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173301]: 2024-08-31 00:03:46.515402618 +0000 UTC m=+0.015754023 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:46.873 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173301]: 2024-08-31 00:03:46.620017932 +0000 UTC m=+0.120369330 container create 0930a163c62966e5234bcfd92a2b007b5c3c3407b3cc3c9735d2a98a36732283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-31T00:03:46.873 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173301]: 2024-08-31 00:03:46.872592956 +0000 UTC m=+0.372944351 container init 0930a163c62966e5234bcfd92a2b007b5c3c3407b3cc3c9735d2a98a36732283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-08-31T00:03:47.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:46.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:47.014 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:46.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:47.014 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:46.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:47.014 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:46 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:46.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:47.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173301]: 2024-08-31 00:03:46.875945373 +0000 UTC m=+0.376296769 container start 0930a163c62966e5234bcfd92a2b007b5c3c3407b3cc3c9735d2a98a36732283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True) 2024-08-31T00:03:47.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:46 smithi136 podman[173301]: 2024-08-31 00:03:46.947433476 +0000 UTC m=+0.447784882 container attach 0930a163c62966e5234bcfd92a2b007b5c3c3407b3cc3c9735d2a98a36732283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-31T00:03:47.290 INFO:journalctl@ceph.osd.7.smithi136.stdout:Aug 31 00:03:47 smithi136 podman[173301]: 2024-08-31 00:03:47.062830671 +0000 UTC m=+0.563182067 container died 0930a163c62966e5234bcfd92a2b007b5c3c3407b3cc3c9735d2a98a36732283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-7-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-31T00:03:47.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:47.109+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:47.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:47.109+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:47.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:47.109+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:47.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:47.109+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:47.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.249+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.249+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.249+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.249+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:47.012+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:47.012+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:47.012+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:47.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:47.012+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:47.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.7.service' 2024-08-31T00:03:47.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:47.688 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-31T00:03:47.688 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-08-31T00:03:47.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.8 2024-08-31T00:03:47.991 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 systemd[1]: Stopping Ceph osd.8 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:47.991 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:47.614+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:47.991 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:47.614+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:47.991 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:47.614+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:47.991 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:47.614+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:48.266 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.990+0000 7f71d42e9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:48.266 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.990+0000 7f71d42e9640 -1 osd.8 1079 *** Got signal Terminated *** 2024-08-31T00:03:48.266 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:47 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:47.990+0000 7f71d42e9640 -1 osd.8 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:48.266 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:48.070+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:48.070+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:48.070+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:48.070+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:48.034+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:48.034+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:48.034+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:48.267 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:48.034+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:48.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:48.265+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:48.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:48.265+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:48.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:48.265+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:48.574 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:48.265+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:49.036 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:48.640+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:49.036 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:48.640+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:49.037 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:48.640+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:49.037 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:48 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:48.640+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:49.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:49.070+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:49.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:49.070+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:49.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:49.070+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:49.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:49.070+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:49.225+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:49.225+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:49.225+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:49.225+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.036+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.036+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.036+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:49.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.036+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:49.997 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:49.637+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:49.997 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:49.637+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:49.997 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:49.637+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:49.997 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:49.637+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:50.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:50.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:50.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:50.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:50.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:50.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:50.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:50.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:50.215+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.996+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.996+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.996+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:50.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:49 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:49.996+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:50.985 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:50.645+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:50.985 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:50.645+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:50.985 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:50.645+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:50.985 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:50.645+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:50.985 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:50.645+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:51.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:51.233+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:51.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:51.233+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:51.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:51.233+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:51.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:51.233+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:51.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:51.233+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.533807+0000 front 2024-08-31T00:03:25.533658+0000 (oldest deadline 2024-08-31T00:03:50.833592+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.984+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.984+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.984+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.984+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:50 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:50.984+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:51.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:51.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:51.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:51.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:51.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:51.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:51.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:51.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:52.005 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:51.646+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:52.006 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:51.646+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:52.006 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:51.646+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:52.006 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:51.646+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:52.006 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:51 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:51.646+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:52.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:52.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:52.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:52.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:52.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:52.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:52.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:52.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:52.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:52.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:52.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:52.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:52.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:52.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:52.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:52.328 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:52.191+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.726844+0000 front 2024-08-31T00:02:47.727077+0000 (oldest deadline 2024-08-31T00:03:08.826900+0000) 2024-08-31T00:03:52.328 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:52.191+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.128117+0000 front 2024-08-31T00:02:54.128278+0000 (oldest deadline 2024-08-31T00:03:17.628110+0000) 2024-08-31T00:03:52.328 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:52.191+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:09.131497+0000 front 2024-08-31T00:03:09.131390+0000 (oldest deadline 2024-08-31T00:03:35.031353+0000) 2024-08-31T00:03:52.328 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:52.191+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:20.233059+0000 front 2024-08-31T00:03:20.233079+0000 (oldest deadline 2024-08-31T00:03:45.533077+0000) 2024-08-31T00:03:52.328 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8[25345]: 2024-08-31T00:03:52.191+0000 7f71cc8f6640 -1 osd.8 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.533807+0000 front 2024-08-31T00:03:25.533658+0000 (oldest deadline 2024-08-31T00:03:50.833592+0000) 2024-08-31T00:03:52.981 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:52.596+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:52.981 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:52.596+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:52.981 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:52.596+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:52.981 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:52.596+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:52.981 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:52.596+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:53.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:53.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:53.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:53.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:53.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:53.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:53.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:53.061+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.979+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.979+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.979+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.979+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:53.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:52 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:52.979+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:53.875 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:53 smithi173 podman[172316]: 2024-08-31 00:03:53.595927126 +0000 UTC m=+5.714338006 container died 0dbe15962604926963061a1d0d0df46b49c484e7a1bf2eeb400cae5c48aa0e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-31T00:03:53.875 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:53.610+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:53.875 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:53.610+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:53.876 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:53.610+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:53.876 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:53.610+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:53.876 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:53 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:53.610+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:54.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:54 smithi173 podman[172316]: 2024-08-31 00:03:54.057629142 +0000 UTC m=+6.176040015 container cleanup 0dbe15962604926963061a1d0d0df46b49c484e7a1bf2eeb400cae5c48aa0e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD) 2024-08-31T00:03:54.324 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:54 smithi173 bash[172316]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8 2024-08-31T00:03:54.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:54.063+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:54.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:54.063+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:54.063+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:54.063+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:54.063+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:54.021+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:54.021+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:54.021+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:54.021+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:54.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:54.021+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:54.824 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:54 smithi173 podman[172330]: 2024-08-31 00:03:54.426409466 +0000 UTC m=+0.824486518 container remove 0dbe15962604926963061a1d0d0df46b49c484e7a1bf2eeb400cae5c48aa0e0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2) 2024-08-31T00:03:54.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:54.574+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:54.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:54.574+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:54.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:54.574+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:54.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:54.574+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:54.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:54 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:54.574+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:55.079 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:55.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:55.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:55.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:55.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:55.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:54 smithi173 podman[172419]: 2024-08-31 00:03:54.827208007 +0000 UTC m=+0.107030086 container create d4ae7f3300cd1263d4f8d1eec398c534e566d7c3674e1af8ff3c61cf9b1c7c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:54 smithi173 podman[172419]: 2024-08-31 00:03:54.736182428 +0000 UTC m=+0.016004473 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:03:55.080 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:55 smithi173 podman[172419]: 2024-08-31 00:03:55.079913323 +0000 UTC m=+0.359735359 container init d4ae7f3300cd1263d4f8d1eec398c534e566d7c3674e1af8ff3c61cf9b1c7c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-08-31T00:03:55.081 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:55.066+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:55.081 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:55.066+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:55.081 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:55.066+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:55.081 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:55.066+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:55.081 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:55.066+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:55.375 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:55 smithi173 podman[172419]: 2024-08-31 00:03:55.083534901 +0000 UTC m=+0.363356932 container start d4ae7f3300cd1263d4f8d1eec398c534e566d7c3674e1af8ff3c61cf9b1c7c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:55.375 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:55 smithi173 podman[172419]: 2024-08-31 00:03:55.154908869 +0000 UTC m=+0.434730920 container attach d4ae7f3300cd1263d4f8d1eec398c534e566d7c3674e1af8ff3c61cf9b1c7c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-08-31T00:03:55.683 INFO:journalctl@ceph.osd.8.smithi173.stdout:Aug 31 00:03:55 smithi173 podman[172419]: 2024-08-31 00:03:55.37563259 +0000 UTC m=+0.655454626 container died d4ae7f3300cd1263d4f8d1eec398c534e566d7c3674e1af8ff3c61cf9b1c7c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-8-deactivate, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:03:55.683 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:55.595+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:55.683 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:55.595+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:55.683 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:55.595+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:55.683 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:55.595+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:55.683 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:55 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:55.595+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:55.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.8.service' 2024-08-31T00:03:55.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:03:55.990 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-08-31T00:03:55.991 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-08-31T00:03:55.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.9 2024-08-31T00:03:56.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.043+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:56.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.043+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.043+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.043+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.043+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 systemd[1]: Stopping Ceph osd.9 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:56.026+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:56.026+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:56.026+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:56.026+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:56.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:56.026+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:56.602 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.396+0000 7f5697ed8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:03:56.603 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.396+0000 7f5697ed8640 -1 osd.9 1079 *** Got signal Terminated *** 2024-08-31T00:03:56.603 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:56.396+0000 7f5697ed8640 -1 osd.9 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:03:57.038 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:56.602+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:57.038 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:56.602+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:57.038 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:56.602+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:57.039 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:56.602+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:57.039 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:56 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:56.602+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:57.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:57.055+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:57.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:57.055+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:57.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:57.055+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:57.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:57.055+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:57.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:57.055+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:57.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:57.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:57.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:57.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:57.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:57.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:57.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:57.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:57.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:57.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:58.018 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:57.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:58.018 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:57.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:58.018 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:57.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:58.019 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:57.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:58.019 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:57 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:57.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:58.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:58.019+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:58.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:58.019+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:58.019+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:58.019+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:58.019+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:58.017+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:58.017+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:58.017+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:58.017+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:58.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:58.017+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:03:59.009 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:58.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:03:59.009 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:58.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:03:59.009 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:58.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:03:59.009 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:58.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:03:59.009 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:58 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:58.631+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:03:59.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:59.020+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:03:59.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:59.020+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:03:59.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:59.020+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:03:59.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:59.020+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:03:59.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:03:59.020+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:03:59.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:59.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:03:59.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:59.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:03:59.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:59.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:03:59.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:59.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:03:59.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:03:59.008+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:00.000 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:59.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:00.001 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:59.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:00.001 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:59.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:00.001 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:59.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:00.001 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:59.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:00.001 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:03:59 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:03:59.598+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:00.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:00.000+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:04:00.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:00.000+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:04:00.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:00.000+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:04:00.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:00.000+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:04:00.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:00.000+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:04:00.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:00.022+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:00.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:00.022+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:00.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:00.022+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:00.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:00.022+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:00.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:00.022+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:00.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:00.568+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:00.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:00.568+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:00.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:00.568+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:00.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:00.568+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:00.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:00.568+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:00.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:00 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:00.568+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:01.324 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:01.025+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:54.370369+0000 front 2024-08-31T00:02:54.370311+0000 (oldest deadline 2024-08-31T00:03:17.870104+0000) 2024-08-31T00:04:01.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:01.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:03.171287+0000 front 2024-08-31T00:03:03.171404+0000 (oldest deadline 2024-08-31T00:03:28.471066+0000) 2024-08-31T00:04:01.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:01.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:11.173096+0000 front 2024-08-31T00:03:11.173241+0000 (oldest deadline 2024-08-31T00:03:35.773410+0000) 2024-08-31T00:04:01.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:01.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.874535+0000 front 2024-08-31T00:03:19.874384+0000 (oldest deadline 2024-08-31T00:03:45.774503+0000) 2024-08-31T00:04:01.325 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9[31010]: 2024-08-31T00:04:01.026+0000 7f56904e5640 -1 osd.9 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:28.557515+0000 front 2024-08-31T00:03:28.476729+0000 (oldest deadline 2024-08-31T00:03:49.576525+0000) 2024-08-31T00:04:01.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.004+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:01.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.004+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:01.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.004+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:01.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.004+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:01.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.004+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:01.326 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.004+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:01.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:01.529+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:01.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:01.529+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:01.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:01.529+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:01.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:01.529+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:01.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:01.529+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:01.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:01.529+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:02.254 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:01 smithi173 podman[172533]: 2024-08-31 00:04:01.961650327 +0000 UTC m=+5.697406509 container died 088facdb8f9ece42bf177fbd96eba8abf81a55884b8036efe16567357602ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:04:02.254 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.990+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:02.254 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.990+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:02.254 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.990+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:02.255 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.990+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:02.255 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.990+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:02.255 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:01 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:01.990+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:02.560 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:02 smithi173 podman[172533]: 2024-08-31 00:04:02.445582257 +0000 UTC m=+6.181338433 container cleanup 088facdb8f9ece42bf177fbd96eba8abf81a55884b8036efe16567357602ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-31T00:04:02.560 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:02 smithi173 bash[172533]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:02 smithi173 podman[172554]: 2024-08-31 00:04:02.697162193 +0000 UTC m=+0.737328971 container remove 088facdb8f9ece42bf177fbd96eba8abf81a55884b8036efe16567357602ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:02 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:02.559+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:02 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:02.559+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:02 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:02.559+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:02 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:02.559+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:02 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:02.559+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:02.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:02 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:02.559+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:03.284 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:03 smithi173 podman[172643]: 2024-08-31 00:04:02.909339268 +0000 UTC m=+0.015897656 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:04:03.284 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:03 smithi173 podman[172643]: 2024-08-31 00:04:03.040118333 +0000 UTC m=+0.146676714 container create fc0d3e502575bec670280be260547671e145ff75e8144b3b0c71cb3822216f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9-deactivate, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-31T00:04:03.284 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:03 smithi173 podman[172643]: 2024-08-31 00:04:03.28418722 +0000 UTC m=+0.390745601 container init fc0d3e502575bec670280be260547671e145ff75e8144b3b0c71cb3822216f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:04:03.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:03.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:03.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:03.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:03.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:03.285 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:03.574 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:03 smithi173 podman[172643]: 2024-08-31 00:04:03.287687501 +0000 UTC m=+0.394245890 container start fc0d3e502575bec670280be260547671e145ff75e8144b3b0c71cb3822216f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-31T00:04:03.574 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:03 smithi173 podman[172643]: 2024-08-31 00:04:03.375823891 +0000 UTC m=+0.482382285 container attach fc0d3e502575bec670280be260547671e145ff75e8144b3b0c71cb3822216f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:04:03.574 INFO:journalctl@ceph.osd.9.smithi173.stdout:Aug 31 00:04:03 smithi173 podman[172643]: 2024-08-31 00:04:03.476216843 +0000 UTC m=+0.582775232 container died fc0d3e502575bec670280be260547671e145ff75e8144b3b0c71cb3822216f56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-9-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6) 2024-08-31T00:04:03.574 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:03.524+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:03.574 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:03.524+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:03.575 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:03.524+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:03.575 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:03.524+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:03.575 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:03.524+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:03.575 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:03.524+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:04.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.974+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:04.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.974+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:04.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.974+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:04.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.974+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:04.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.974+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:04.074 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:03 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:03.974+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:04.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.9.service' 2024-08-31T00:04:04.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:04:04.254 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-08-31T00:04:04.254 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-08-31T00:04:04.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.10 2024-08-31T00:04:04.566 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 systemd[1]: Stopping Ceph osd.10 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:04:04.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.565+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.565+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.565+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.565+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.565+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.565+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.607+0000 7f601fc0c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.607+0000 7f601fc0c640 -1 osd.10 1079 *** Got signal Terminated *** 2024-08-31T00:04:04.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:04 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:04.607+0000 7f601fc0c640 -1 osd.10 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:04:05.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:05.024+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:05.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:05.024+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:05.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:05.024+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:05.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:05.024+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:05.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:05.024+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:05.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:05.024+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:05.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:05.551+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:05.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:05.551+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:05.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:05.551+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:05.826 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:05.551+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:05.826 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:05.551+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:05.826 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:05 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:05.551+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:06.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:06.060+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:06.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:06.060+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:06.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:06.060+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:06.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:06.060+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:06.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:06.060+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:06.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:06.060+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:06.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:06.510+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:06.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:06.510+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:06.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:06.510+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:06.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:06.510+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:06.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:06.510+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:06.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:06 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:06.510+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:07.464 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:07.081+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:07.464 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:07.081+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:07.464 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:07.081+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:07.465 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:07.081+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:07.465 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:07.081+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:07.465 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:07.081+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:07.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:07.463+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:07.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:07.463+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:07.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:07.463+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:07.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:07.463+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:07.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:07.463+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:07.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:07 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:07.463+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:08.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:08.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:08.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:08.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:08.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:08.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:08.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:08.042+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:08.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:08.496+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:08.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:08.496+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:08.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:08.496+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:08.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:08.496+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:08.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:08.496+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:08.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:08 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:08.496+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:09.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.030+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:09.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:44.277320+0000 front 2024-08-31T00:02:44.277371+0000 (oldest deadline 2024-08-31T00:03:12.477617+0000) 2024-08-31T00:04:09.824 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.978824+0000 front 2024-08-31T00:02:55.978997+0000 (oldest deadline 2024-08-31T00:03:17.079000+0000) 2024-08-31T00:04:09.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:02.980086+0000 front 2024-08-31T00:03:02.980120+0000 (oldest deadline 2024-08-31T00:03:27.079914+0000) 2024-08-31T00:04:09.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6804 osd.4 since back 2024-08-31T00:03:19.782925+0000 front 2024-08-31T00:03:19.782863+0000 (oldest deadline 2024-08-31T00:03:43.282620+0000) 2024-08-31T00:04:09.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:24.983624+0000 front 2024-08-31T00:03:24.983647+0000 (oldest deadline 2024-08-31T00:03:49.683613+0000) 2024-08-31T00:04:09.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.885514+0000 front 2024-08-31T00:03:34.885434+0000 (oldest deadline 2024-08-31T00:03:59.585282+0000) 2024-08-31T00:04:09.825 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10[36866]: 2024-08-31T00:04:09.486+0000 7f601c221640 -1 osd.10 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:44.186983+0000 front 2024-08-31T00:03:44.186890+0000 (oldest deadline 2024-08-31T00:04:09.486885+0000) 2024-08-31T00:04:10.324 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:10 smithi173 podman[172743]: 2024-08-31 00:04:10.071880957 +0000 UTC m=+5.591474658 container died 01d053ed54d8e9d43f08ce581977942d81709003a73b7ac3acd7ff266d531406 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:10.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:09 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:09.988+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:10.701 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:10 smithi173 podman[172743]: 2024-08-31 00:04:10.449425415 +0000 UTC m=+5.969019101 container cleanup 01d053ed54d8e9d43f08ce581977942d81709003a73b7ac3acd7ff266d531406 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-31T00:04:10.701 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:10 smithi173 bash[172743]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10 2024-08-31T00:04:11.024 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:10 smithi173 podman[172762]: 2024-08-31 00:04:10.700700336 +0000 UTC m=+0.624895224 container remove 01d053ed54d8e9d43f08ce581977942d81709003a73b7ac3acd7ff266d531406 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-08-31T00:04:11.324 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:11 smithi173 podman[172851]: 2024-08-31 00:04:11.051806344 +0000 UTC m=+0.109488548 container create 4a5caaafafbc6f18cec272ab0532736313c2eff5988aca6b52b9da10f87aefe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-31T00:04:11.324 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:11 smithi173 podman[172851]: 2024-08-31 00:04:10.958077031 +0000 UTC m=+0.015759199 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:04:11.324 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:11 smithi173 podman[172851]: 2024-08-31 00:04:11.254501712 +0000 UTC m=+0.312183872 container init 4a5caaafafbc6f18cec272ab0532736313c2eff5988aca6b52b9da10f87aefe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_CLEAN=True, ceph=True) 2024-08-31T00:04:11.324 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:11 smithi173 podman[172851]: 2024-08-31 00:04:11.257856698 +0000 UTC m=+0.315538860 container start 4a5caaafafbc6f18cec272ab0532736313c2eff5988aca6b52b9da10f87aefe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:11.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:11 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:11.023+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:11.732 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:11 smithi173 podman[172851]: 2024-08-31 00:04:11.341763911 +0000 UTC m=+0.399446083 container attach 4a5caaafafbc6f18cec272ab0532736313c2eff5988aca6b52b9da10f87aefe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-31T00:04:11.732 INFO:journalctl@ceph.osd.10.smithi173.stdout:Aug 31 00:04:11 smithi173 podman[172851]: 2024-08-31 00:04:11.448815972 +0000 UTC m=+0.506498132 container died 4a5caaafafbc6f18cec272ab0532736313c2eff5988aca6b52b9da10f87aefe3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-08-31T00:04:12.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.10.service' 2024-08-31T00:04:12.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:04:12.073 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-08-31T00:04:12.073 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-08-31T00:04:12.073 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.11 2024-08-31T00:04:12.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.037+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:12.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 systemd[1]: Stopping Ceph osd.11 for eb936aa2-6725-11ef-bcd4-c7b262605968... 2024-08-31T00:04:12.824 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.427+0000 7fcfbcd96640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-31T00:04:12.824 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.427+0000 7fcfbcd96640 -1 osd.11 1079 *** Got signal Terminated *** 2024-08-31T00:04:12.824 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:12 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:12.427+0000 7fcfbcd96640 -1 osd.11 1079 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-31T00:04:13.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:13.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:13.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:13.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:13.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:13.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:13.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.012+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:14.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:14.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:14.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:14.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:14.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:14.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:14.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:13 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:13.997+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:15.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:15 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:15.002+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:16.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:16.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:16.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:16.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:16.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:16.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:16.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:16.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:16 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:16.005+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.173:6804 osd.8 since back 2024-08-31T00:03:51.684178+0000 front 2024-08-31T00:03:51.684249+0000 (oldest deadline 2024-08-31T00:04:15.184186+0000) 2024-08-31T00:04:17.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6806 osd.0 since back 2024-08-31T00:02:47.073627+0000 front 2024-08-31T00:02:47.073807+0000 (oldest deadline 2024-08-31T00:03:12.373592+0000) 2024-08-31T00:04:17.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6814 osd.1 since back 2024-08-31T00:02:55.275012+0000 front 2024-08-31T00:02:55.275062+0000 (oldest deadline 2024-08-31T00:03:18.174807+0000) 2024-08-31T00:04:17.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6822 osd.2 since back 2024-08-31T00:03:01.676071+0000 front 2024-08-31T00:03:01.676171+0000 (oldest deadline 2024-08-31T00:03:26.376203+0000) 2024-08-31T00:04:17.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.106:6830 osd.3 since back 2024-08-31T00:03:12.177884+0000 front 2024-08-31T00:03:12.177983+0000 (oldest deadline 2024-08-31T00:03:36.277882+0000) 2024-08-31T00:04:17.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-08-31T00:03:25.579895+0000 front 2024-08-31T00:03:25.579957+0000 (oldest deadline 2024-08-31T00:03:50.279762+0000) 2024-08-31T00:04:17.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6820 osd.6 since back 2024-08-31T00:03:34.781933+0000 front 2024-08-31T00:03:34.781914+0000 (oldest deadline 2024-08-31T00:04:00.582304+0000) 2024-08-31T00:04:17.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.136:6828 osd.7 since back 2024-08-31T00:03:41.683418+0000 front 2024-08-31T00:03:41.683476+0000 (oldest deadline 2024-08-31T00:04:07.583325+0000) 2024-08-31T00:04:17.325 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:17 smithi173 ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11[42885]: 2024-08-31T00:04:17.041+0000 7fcfb53a3640 -1 osd.11 1079 heartbeat_check: no reply from 172.21.15.173:6804 osd.8 since back 2024-08-31T00:03:51.684178+0000 front 2024-08-31T00:03:51.684249+0000 (oldest deadline 2024-08-31T00:04:15.184186+0000) 2024-08-31T00:04:18.324 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:18 smithi173 podman[172955]: 2024-08-31 00:04:18.012318953 +0000 UTC m=+5.719526131 container died 9519ca99c99014b8275ba74f11e185b13959055637d0bf39f949cd07ad2b5f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-08-31T00:04:18.824 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:18 smithi173 podman[172955]: 2024-08-31 00:04:18.33570465 +0000 UTC m=+6.042911814 container cleanup 9519ca99c99014b8275ba74f11e185b13959055637d0bf39f949cd07ad2b5f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-31T00:04:18.824 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:18 smithi173 bash[172955]: ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11 2024-08-31T00:04:18.824 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:18 smithi173 podman[172970]: 2024-08-31 00:04:18.520202069 +0000 UTC m=+0.510571709 container remove 9519ca99c99014b8275ba74f11e185b13959055637d0bf39f949cd07ad2b5f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2) 2024-08-31T00:04:19.231 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:18 smithi173 podman[173058]: 2024-08-31 00:04:18.837388602 +0000 UTC m=+0.016396578 image pull fb225243d408dc6fa8fbba3d1b600d3ee6f4903cc02563d5d2cf79f544ab1c60 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8 2024-08-31T00:04:19.232 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:18 smithi173 podman[173058]: 2024-08-31 00:04:18.946048098 +0000 UTC m=+0.125056100 container create 526bd9a2203abaa04dd90558f943d859f6d7fbcf2cc236ecc8e62406e675466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-08-31T00:04:19.232 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:19 smithi173 podman[173058]: 2024-08-31 00:04:19.148296845 +0000 UTC m=+0.327304810 container init 526bd9a2203abaa04dd90558f943d859f6d7fbcf2cc236ecc8e62406e675466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-08-31T00:04:19.232 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:19 smithi173 podman[173058]: 2024-08-31 00:04:19.151429815 +0000 UTC m=+0.330437776 container start 526bd9a2203abaa04dd90558f943d859f6d7fbcf2cc236ecc8e62406e675466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-31T00:04:19.232 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:19 smithi173 podman[173058]: 2024-08-31 00:04:19.231833196 +0000 UTC m=+0.410841190 container attach 526bd9a2203abaa04dd90558f943d859f6d7fbcf2cc236ecc8e62406e675466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-08-31T00:04:19.574 INFO:journalctl@ceph.osd.11.smithi173.stdout:Aug 31 00:04:19 smithi173 podman[173058]: 2024-08-31 00:04:19.343325414 +0000 UTC m=+0.522333391 container died 526bd9a2203abaa04dd90558f943d859f6d7fbcf2cc236ecc8e62406e675466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9ce7ea81fdcbadee7b108bc3cbb05cabc1790a17f0dd31a0b9908b2f99b5aff8, name=ceph-eb936aa2-6725-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-bharath1-testing-2024-08-30-1604-24432b6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-08-31T00:04:19.920 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb936aa2-6725-11ef-bcd4-c7b262605968@osd.11.service' 2024-08-31T00:04:19.960 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-31T00:04:19.960 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-08-31T00:04:19.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --force --keep-logs 2024-08-31T00:04:20.163 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-31T00:04:22.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --force --keep-logs 2024-08-31T00:04:22.233 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-31T00:04:24.177 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --force --keep-logs 2024-08-31T00:04:24.379 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-31T00:04:25.843 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-31T00:04:25.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-31T00:04:25.909 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-31T00:04:25.935 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-31T00:04:25.936 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi106/crash 2024-08-31T00:04:25.937 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash -- . 2024-08-31T00:04:25.984 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-08-31T00:04:25.984 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-08-31T00:04:25.986 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi136/crash 2024-08-31T00:04:25.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash -- . 2024-08-31T00:04:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-08-31T00:04:26.026 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-08-31T00:04:26.028 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi173/crash 2024-08-31T00:04:26.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash -- . 2024-08-31T00:04:26.068 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-08-31T00:04:26.069 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-08-31T00:04:26.070 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-31T00:04:26.070 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-31T00:04:26.163 INFO:teuthology.orchestra.run.smithi106.stdout:2024-08-30T23:40:00.000117+0000 mon.a (mon.0) 2249 : cluster [WRN] Health detail: HEALTH_WARN 2 osds down; Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-31T00:04:26.164 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-08-31T00:04:26.164 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[SEC\]' /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-31T00:04:26.200 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[ERR\]' /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-31T00:04:26.276 DEBUG:teuthology.orchestra.run.smithi106:> sudo egrep '\[WRN\]' /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-08-31T00:04:26.351 INFO:teuthology.orchestra.run.smithi106.stdout:2024-08-30T23:40:00.000117+0000 mon.a (mon.0) 2249 : cluster [WRN] Health detail: HEALTH_WARN 2 osds down; Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized 2024-08-31T00:04:26.351 INFO:tasks.cephadm:Compressing logs... 2024-08-31T00:04:26.351 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T00:04:26.394 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T00:04:26.397 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-31T00:04:26.429 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-31T00:04:26.431 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.0.log 2024-08-31T00:04:26.434 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.1.log 2024-08-31T00:04:26.434 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/objectstore_tool.2.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.a.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.2.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mgr.y.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.2.log: -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.3.log 2024-08-31T00:04:26.435 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-31T00:04:26.437 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-31T00:04:26.437 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/objectstore_tool.2.log 2024-08-31T00:04:26.437 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.7.log 2024-08-31T00:04:26.437 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log 2024-08-31T00:04:26.437 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.4.log 2024-08-31T00:04:26.437 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log 2024-08-31T00:04:26.439 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-08-31T00:04:26.439 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.6.log 2024-08-31T00:04:26.439 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.5.log 2024-08-31T00:04:26.439 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log 2024-08-31T00:04:26.440 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mgr.x.log 2024-08-31T00:04:26.440 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.b.log 2024-08-31T00:04:26.440 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-31T00:04:26.445 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 86.1% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log.gz 2024-08-31T00:04:26.462 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-31T00:04:26.462 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.10.log 2024-08-31T00:04:26.464 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.9.log 2024-08-31T00:04:26.464 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log 2024-08-31T00:04:26.464 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.10.log: /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log 2024-08-31T00:04:26.464 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.c.log 2024-08-31T00:04:26.464 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-08-31T00:04:26.465 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log: /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.11.log 2024-08-31T00:04:26.465 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log 2024-08-31T00:04:26.465 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.8.log 2024-08-31T00:04:26.465 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-31T00:04:26.486 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 81.9% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-08-31T00:04:26.486 INFO:teuthology.orchestra.run.smithi106.stderr: 91.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-08-31T00:04:26.488 INFO:teuthology.orchestra.run.smithi136.stderr: 93.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mgr.x.log.gz 2024-08-31T00:04:26.499 INFO:teuthology.orchestra.run.smithi106.stderr: 82.3% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-08-31T00:04:26.520 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log: /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.8.log: /var/log/ceph/cephadm.log: 81.7% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-08-31T00:04:26.522 INFO:teuthology.orchestra.run.smithi106.stderr: 89.7% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-08-31T00:04:26.569 INFO:teuthology.orchestra.run.smithi173.stderr: 95.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-31T00:04:26.610 INFO:teuthology.orchestra.run.smithi106.stderr: 95.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-31T00:04:26.650 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-08-31T00:04:26.668 INFO:teuthology.orchestra.run.smithi136.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-31T00:04:26.670 INFO:teuthology.orchestra.run.smithi136.stderr: 91.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-08-31T00:04:26.703 INFO:teuthology.orchestra.run.smithi136.stderr: 86.1% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log.gz 2024-08-31T00:04:26.703 INFO:teuthology.orchestra.run.smithi173.stderr: 89.8% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-08-31T00:04:26.738 INFO:teuthology.orchestra.run.smithi106.stderr: 94.7% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/objectstore_tool.2.log.gz 2024-08-31T00:04:26.888 INFO:teuthology.orchestra.run.smithi173.stderr: 86.1% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph.log.gz 2024-08-31T00:04:26.982 INFO:teuthology.orchestra.run.smithi173.stderr: 91.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-08-31T00:04:27.771 INFO:teuthology.orchestra.run.smithi136.stderr: 94.5% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/objectstore_tool.2.log.gz 2024-08-31T00:04:33.128 INFO:teuthology.orchestra.run.smithi106.stderr: 91.5% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mgr.y.log.gz 2024-08-31T00:04:43.675 INFO:teuthology.orchestra.run.smithi106.stderr: 90.0% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.a.log.gz 2024-08-31T00:04:45.904 INFO:teuthology.orchestra.run.smithi136.stderr: 91.0% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.b.log.gz 2024-08-31T00:04:50.340 INFO:teuthology.orchestra.run.smithi173.stderr: 91.1% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-mon.c.log.gz 2024-08-31T00:07:54.181 INFO:teuthology.orchestra.run.smithi136.stderr: 91.7% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.6.log.gz 2024-08-31T00:18:10.109 INFO:teuthology.orchestra.run.smithi136.stderr: 92.5% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.7.log.gz 2024-08-31T00:18:37.011 INFO:teuthology.orchestra.run.smithi136.stderr: 92.4% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.4.log.gz 2024-08-31T00:18:40.399 INFO:teuthology.orchestra.run.smithi136.stderr: 92.3% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.5.log.gz 2024-08-31T00:18:40.401 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-08-31T00:18:40.401 INFO:teuthology.orchestra.run.smithi136.stderr:real 14m13.989s 2024-08-31T00:18:40.401 INFO:teuthology.orchestra.run.smithi136.stderr:user 7m9.618s 2024-08-31T00:18:40.401 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m19.136s 2024-08-31T00:31:16.462 INFO:teuthology.orchestra.run.smithi173.stderr: 92.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.9.log.gz 2024-08-31T00:32:05.406 INFO:teuthology.orchestra.run.smithi173.stderr: 91.8% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.8.log.gz 2024-08-31T00:33:39.558 INFO:teuthology.orchestra.run.smithi106.stderr: 91.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.2.log.gz 2024-08-31T00:33:39.964 INFO:teuthology.orchestra.run.smithi173.stderr: 91.8% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.10.log.gz 2024-08-31T00:33:49.160 INFO:teuthology.orchestra.run.smithi173.stderr: 92.3% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.11.log.gz 2024-08-31T00:33:49.162 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-08-31T00:33:49.162 INFO:teuthology.orchestra.run.smithi173.stderr:real 29m22.751s 2024-08-31T00:33:49.162 INFO:teuthology.orchestra.run.smithi173.stderr:user 9m13.747s 2024-08-31T00:33:49.162 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m27.356s 2024-08-31T00:39:15.215 INFO:teuthology.orchestra.run.smithi106.stderr: 92.0% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.3.log.gz 2024-08-31T00:39:15.676 INFO:teuthology.orchestra.run.smithi106.stderr: 91.6% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.1.log.gz 2024-08-31T00:39:43.082 INFO:teuthology.orchestra.run.smithi106.stderr: 91.2% -- replaced with /var/log/ceph/eb936aa2-6725-11ef-bcd4-c7b262605968/ceph-osd.0.log.gz 2024-08-31T00:39:43.122 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-08-31T00:39:43.122 INFO:teuthology.orchestra.run.smithi106.stderr:real 35m16.715s 2024-08-31T00:39:43.122 INFO:teuthology.orchestra.run.smithi106.stderr:user 11m13.960s 2024-08-31T00:39:43.122 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m33.135s 2024-08-31T00:39:43.123 INFO:tasks.cephadm:Archiving logs... 2024-08-31T00:39:43.124 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi106/log 2024-08-31T00:39:43.125 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T00:41:44.271 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi136/log 2024-08-31T00:41:44.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T00:42:59.793 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi173/log 2024-08-31T00:42:59.794 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-31T00:44:35.639 INFO:tasks.cephadm:Removing cluster... 2024-08-31T00:44:35.640 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --force 2024-08-31T00:44:36.341 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-31T00:44:37.302 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --force 2024-08-31T00:44:37.601 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-31T00:44:38.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb936aa2-6725-11ef-bcd4-c7b262605968 --force 2024-08-31T00:44:38.552 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: eb936aa2-6725-11ef-bcd4-c7b262605968 2024-08-31T00:44:39.392 INFO:tasks.cephadm:Removing cephadm ... 2024-08-31T00:44:39.392 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-31T00:44:39.426 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-31T00:44:39.445 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-31T00:44:39.461 INFO:tasks.cephadm:Teardown complete 2024-08-31T00:44:39.462 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-31T00:44:39.471 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-31T00:44:39.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T00:44:39.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T00:44:39.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-31T00:44:39.785 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-31T00:44:39.793 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-31T00:44:39.793 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T00:44:39.796 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T00:44:39.799 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-31T00:44:39.811 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-08-31T00:44:39.814 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-08-31T00:44:39.814 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-08-31T00:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T00:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-08-31T00:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +563us[ +547us] +/- 43ms 2024-08-31T00:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 -1267us[-1283us] +/- 51ms 2024-08-31T00:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +604us[ +588us] +/- 64ms 2024-08-31T00:44:39.839 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T00:44:39.846 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T00:44:39.846 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-08-31T00:44:39.846 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +534us[ +563us] +/- 43ms 2024-08-31T00:44:39.846 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1323us[-1323us] +/- 51ms 2024-08-31T00:44:39.846 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +531us[ +556us] +/- 63ms 2024-08-31T00:44:39.846 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T00:44:39.873 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-31T00:44:39.873 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-08-31T00:44:39.873 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv01.front.sepia.ceph.com 3 7 377 69 +567us[ +577us] +/- 42ms 2024-08-31T00:44:39.873 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 65 -1250us[-1250us] +/- 66ms 2024-08-31T00:44:39.873 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +589us[ +589us] +/- 64ms 2024-08-31T00:44:39.873 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-31T00:44:39.887 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-31T00:44:39.896 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-31T00:44:39.896 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-31T00:44:39.904 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-31T00:44:40.075 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-31T00:44:40.167 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-31T00:44:40.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-31T00:44:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:44:40.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-31T00:44:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:44:40.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-08-31T00:44:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:44:40.328 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-31T00:44:40.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-31T00:44:40.348 INFO:teuthology.task.internal:Duration was 5482.196934 seconds 2024-08-31T00:44:40.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-31T00:44:40.357 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-31T00:44:40.357 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T00:44:40.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T00:44:40.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-31T00:44:40.425 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T00:44:40.430 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T00:44:40.430 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-31T00:44:40.852 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-31T00:44:40.853 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-08-31T00:44:40.853 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T00:44:40.875 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-08-31T00:44:40.875 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T00:44:40.904 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-08-31T00:44:40.905 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-31T00:44:40.932 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-31T00:44:40.932 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T00:44:40.934 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T00:44:40.948 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-31T00:44:40.999 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-31T00:44:40.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T00:44:41.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T00:44:41.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-31T00:44:43.738 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-31T00:44:44.598 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-31T00:44:44.598 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T00:44:44.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T00:44:44.662 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-31T00:44:44.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-31T00:44:44.782 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T00:44:44.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T00:44:44.787 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-31T00:44:44.825 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-08-31T00:44:44.827 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-08-31T00:44:44.833 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-08-31T00:44:44.871 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T00:44:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:44:44.888 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T00:44:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:44:44.905 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-31T00:44:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-31T00:44:44.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-31T00:44:44.932 INFO:teuthology.task.internal:Transferring archived files... 2024-08-31T00:44:44.933 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi106 2024-08-31T00:44:44.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T00:44:45.015 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi136 2024-08-31T00:44:45.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T00:44:45.062 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-08-30_17:51:23-rados-wip-bharath1-testing-2024-08-30-1604-distro-default-smithi/7882173/remote/smithi173 2024-08-31T00:44:45.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-31T00:44:45.109 INFO:teuthology.task.internal:Removing archive directory... 2024-08-31T00:44:45.109 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T00:44:45.112 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T00:44:45.114 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-31T00:44:45.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-31T00:44:45.170 INFO:teuthology.task.internal:Not uploading archives. 2024-08-31T00:44:45.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-31T00:44:45.179 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-31T00:44:45.179 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T00:44:45.182 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T00:44:45.184 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-31T00:44:45.195 INFO:teuthology.orchestra.run.smithi106.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 00:44 /home/ubuntu/cephtest 2024-08-31T00:44:45.199 INFO:teuthology.orchestra.run.smithi136.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 00:44 /home/ubuntu/cephtest 2024-08-31T00:44:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 31 00:44 /home/ubuntu/cephtest 2024-08-31T00:44:45.217 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-31T00:44:45.227 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-31T00:44:45.250 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} duration: 5482.196934223175 failure_reason: '"2024-08-30T23:40:00.000117+0000 mon.a (mon.0) 2249 : cluster [WRN] Health detail: HEALTH_WARN 2 osds down; Degraded data redundancy: 34942/222120 objects degraded (15.731%), 7 pgs degraded, 2 pgs undersized" in cluster log' flavor: default owner: scheduled_skanta@teuthology success: false 2024-08-31T00:44:45.250 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-31T00:44:45.340 INFO:teuthology.run:FAIL